Create a robots.txt file

By Daniel Scocco

The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.

A robots.txt is a simple text file that can be created with Notepad. If you are using WordPress a sample robots.txt file would be:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

“User-agent: *” means that all the search bots (from Google, Yahoo, MSN and so on) should use those instructions to crawl your website. Unless your website is complex you will not need to set different instructions for different spiders.

“Disallow: /wp-” will make sure that the search engines will not crawl the WordPress files. This line will exclude all files and foldes starting with “wp-“ from the indexation, avoiding duplicated content and admin files.

If you are not using WordPress just substitute the Disallow lines with files or folders on your website that should not be crawled, for instance:

User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /any other folder to be excluded/

After you created the robots.txt file just upload it to your root directory and you are done!



Share

98 Responses to “Create a robots.txt file”

  • Alan

    Hi,

    My only issue is that WP has a virtual robots.txt file. This file, for example, is used with the xml sitemap plugin to list the Sitemap.

    So as not to have a duplicate Sitemap, I would imagine that I should tick the xml sitemap plugin option NOT to show the Sitemap in the virtual robots.txt file & put it the the new actual robots.txt file instead.

    Does this sound right? Thanks for the post!

  • Saifa Raz

    This Article is Great. thanks for the Information.
    but r these really useful for indexing by Google ?

  • Keith Davis

    Hi Dan
    That’s my next step for my site… creating a robots.txt file.

    I’ve always understood the problem of duplicate content, but the penny has only just dropped about how easy it is to create duplicate content with a wordpress site.

    Great solution, to use a robots.txt file.

  • Daily Good Tips

    Create robot.txt just can be created in wordpress or website with wordpress but I think robot.txt creating can not be applied in blogspot template. We just talk google if we want to change the robot.txt.

  • Kitchener Waterloo SEO

    I like your idea of blocking whole directories instead of mentioning specific files (which I’ve seen done). For example:

    User agent: *

    Disallow: /my_passwords_file.html

    Robot.txt are considered best practice, but they can leave your website open to exploits if your not careful.

  • blogging posts

    I real want to know, how can we do with the image, dis or allow in robots.txt.

  • Kamal Hasa

    Thank you for this post. But askapache has got a better topic on the same subject..

  • Werbeagentur Berlin

    Great Tutorial, thank you. This has me very helped.

  • leon

    great tips … wonder if we can disallow : /wp-register.php ?

  • KoolDesigning USA

    Really great post, thanks for sharing. I am new here and agree with you that robot.txt file is more important when you did not allowed the search engine robots the specific page or folder at your web site.

    Thanks again and Best regards,

    KoolDesigning

  • Live4pk Dot Com

    I have a website on dot com, it is simple website which updated monthly, do i need to creat? please help me clearly or give a sample text file which we can edit and use who dont have experience to do.

  • Bang Kritikus

    Google don’t like duplicate content so you just allowed one.

  • seo

    Nice post. Great reminder of how you can easily protect folders and files on your server.

  • drift80

    Optimizing robots.txt file will help prevent Google penalizing you for duplicate content.

  • Full pc utilities downloads

    Great tutorial, thanks for sharing this with us.. I’ve learn a lot.

  • Webagentur

    Thank you, this Example has me very helped. But why was not already a robots.txt during the installation process?

  • Krystyl

    Is there a way to do this on blogger?

  • Richard

    I just found this article as I searched for robots.txt wordpress SEO. I did this search after checking my google webmaster tools. Go gave me a bund of warnings about unfound pages. These specifically included trackback and feed pages which you exclude in your robots.txt file. Thanks for the valuable help,
    Richard

  • Michael Aulia

    It should be under your public_html, as that’s the folder of your root files under your domain


    Michael Aulia
    http://www.michaelaulia.com

  • BusySphere

    Thanks for the post. While uploading the robots.txt file, it has been kept under “/” and “/public_html” folders. Which is the correct root directory?

  • Dough Roller

    Sue, yes you create the robots.txt file in Notepad and upload it to your server. The robots.txt is a simple text file. You can also create it in the text editing function that is probably available within your file manager on the server that hosts your site. As for the text you can check out http://www.doughroller.net/robots.txt to see what I did (I run WordPress). I took this code from ask apache. Good luck.

  • Sue Huss

    I’m confused can someone just give me the correct text for a wordpress blog. Will uploading it from notepad work then?

  • Gr.Zhang

    喜欢,借鉴,学习

Comments are closed.