Create a robots.txt file

By Daniel Scocco

The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.

A robots.txt is a simple text file that can be created with Notepad. If you are using WordPress a sample robots.txt file would be:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

“User-agent: *” means that all the search bots (from Google, Yahoo, MSN and so on) should use those instructions to crawl your website. Unless your website is complex you will not need to set different instructions for different spiders.

“Disallow: /wp-” will make sure that the search engines will not crawl the WordPress files. This line will exclude all files and foldes starting with “wp-“ from the indexation, avoiding duplicated content and admin files.

If you are not using WordPress just substitute the Disallow lines with files or folders on your website that should not be crawled, for instance:

User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: /any other folder to be excluded/

After you created the robots.txt file just upload it to your root directory and you are done!



Share

98 Responses to “Create a robots.txt file”

  • Mac Utopia

    I agree that robots.txt files are important, however I disagree with blocking the images folder. Some sites achieve some pretty good traffic numbers from google image search, blocking this directory will block your images from showing up in these results.

    • Kamal Hasa

      Well the images folder that is being mentioned could be the ones like the icons and miscellaneous images.

      as far as the normal images are concerned it should and needs to be crawled by Google. My blog receives more than 30% of the traffic from Google Images presently.

  • egon

    Same with WordPress. If you disallow “/wp-” then it’s not going to index any of your uploads like images since they are in your wp-content folder. I get quite a bit of traffic from Google Image Search.

  • Daniel

    I am not sure if the GoogleImage bot tracks down images from the /images/ folder or directly from the posts where the images where inserted.

    In fact if I think the latter option is true, because the content of the page, the keywords and the alt tag play an important role on the image search algorithm.

    In that case Disallowing the image folder should not affect your incoming traffic from image searches.

  • Dawud Miracle

    Nice post. Great reminder of how you can easily protect folders and files on your server.

  • Thilak

    I guess disalowing “wp-” will not affect Google Image bot from crawling your images because it crawls them from the post and not from the directory

  • Bes Z

    Thilak, unless I am mistaken, wouldn’t the Google Image Bot be following the rules of the robots.txt file regardless of where it starts crawling from?

  • Daniel

    Bes, if I am not the wrong the Google Image Bot will not need to crawl your image folder at all. It will crawl your pages, and it will index all the images on those pages (i.e. posts).

  • engtech

    One other thing to consider blocking is any duplicate content on your side. WordPress gives you about three thousand ways to access content (/page, /tag, direct links, etc). Blocking some of them might be a good idea.

    But I’m not an SEO and I have no idea what should be blocked.

  • Ajay

    Daniel, that is not the case, rules of robots.txt are always followed no matter where the index is followed from.

    Excluding wp- and wp-content will block out indexing of your images.

    You will need to seperately allow the folder you want.

    I suggest using Webmaster Tools to analyze your robots.txt to give you a proper understanding of what is allowed / blocked.

    Also I am not sure if /trackback/ will disallow everything with /trackback/ can you get this confirmed?

  • Daniel

    Ajay /trackback/ will disallow all the trackback pages, and I also think we should disallow comments, but I am not sure if /comment/ is the right attribute for that.

    Secondly, regarding the indexing of images, it is true that with the /-wp the images on your upload folder will not be indexed, but they do not have to. I still think that Google indexes images through web pages, regardless of whether it crawls the image folder or not. I will research more about it though.

  • Mr Apache

    Just to let you know there is an example robots.txt for wordpress at http://www.askapache.com/2007/seo/seo-with-robotstxt.html

    Its full of nice tips.. like some robots can recognize wildcards like /wp-* but some can’t.

  • SEO

    Great article mate. I’m still unsure what needs to be excluded though. But there is some stuff that needs to be blocked to avoid duplicate content. Cheers

  • John T. Pratt

    Thanks for the post, very helpful. Is there any problem with doing it like this:
    Disallow: */feed/
    Disallow: */trackback/

    Also, when you do:
    Disallow: /wp-

    don’t you need a star, like:
    Disallow: wp-*

    or is it better to just do this:
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Also, now with sitemap inclusion, you should consider updating this to have:
    Sitemap: http://www.site.com/sitemap.xml

    in there for those using the google sitemap plugin.

    thanks again!

  • Daniel

    John, regarding the first question: not all search bots recognize the * attribute. Some people argue that the Google Bot specifically does not interpret the * as a joker, it just ignores it. That is why I avoid using it. Plus, it should not be necessary to add it before a folder like */feed/.

    Secondly, I am also not sure about the /wp- working for all folders that begin with that. In fact, on my latest robots.txt file I added all the folders that were to be excluded individually like /wp-admin/ and /wp-includes/.

  • rtemp

    How do I upload the robots.txt to the root directory using WordPress?

  • John T. Pratt

    you can’t upload robots.txt to the root directory using WordPress – it doesn’t have an FTP function. The only way it would be possible is if someone associated “robots.txt” as a theme associated file, and you could edit and save it in “theme editor” under the “presentation” tab. You wouldn’t think it would be too difficult to associate a file with a theme by modifying a little code, but presently I don’t know how to do it.

    Otherwise, you have to do it with an FTP program.

  • Scibiz

    The particual robot.txt file is an important choice from the SEO point of view. Thanks for the original approach to the problem!

    I’m going to check some other txt…

  • derek

    I was curious, my blog isnt installed in the root but instead a folder on my side.

    so for instance instead of /wp-admin I would have /blogfolder/wp-admin

    Does that change things, should I set it up differently, could you give me a hand?

  • Daniel

    If I am not wrong, even if the blog in on a sub-folder, the robots.txt file should still go on the root directory since it is the first thing a search bot will look for.

  • Derek

    Thanks Daniel I know that the robots.txt file should still go in the root, I just want to know if I have to change the robots file to look inside my new blog installation directory
    like
    disallow: /blogfolder/wp-admin

    for instance

  • Daniel

    Derek, I see what you are asking now.

    I think both

    disallow: /blogfolder/wp-admin/

    and

    disallow: /wp-admin/

    would work fine.

  • John Doe

    So many people with so many ideas!!! I like the ideas in general to block robots to avoid duplicate contents. But in my opinion duplicate contents should be avoided at the url level. Dont allow your cms to generate more then one url of the same post.

    I have posted my thought on my blog. Have a look.

  • SEO Thailand

    Yes Great Post
    But can you give me tips regarding how to use sitemap effectively

  • Harsha

    I think there is no harm in allowing the Bots to assecc your images folders. It can bring traffic through the Google image search

  • AskApache

    Its nice that people are beginning to think about controlling robots with robots.txt.. You may want to look at my updated wordpress robots.txt file on AskApache, especially regarding the digg mirror, way back archiver, etc..

    @ http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html

  • karim

    thanks for info

  • Bollywood Actress

    Although this post didn’t discuss why and how a certain robots.txt file with some certain entries is best, you made sure people should understand what those codes mean. This is something I have never seen on any other site on my quest to find the best robots.txt file for better SEO. Thank you very much for that.

    I hope you would take the pain to come with a robots.txt file and explain why and how it is better, which would really help a lot of people like me. Thanks again.

  • Jony

    Thanks for the post, very helpful.

  • Jony

    gracias justo lo que necesitaba

  • Rafi

    I feel robots are much more important to be placed on the root of the site directoy
    it had really inceared my ranking

    http://www.ehotbid.com
    gem and jewelry auction starting from $2

  • Gr.Zhang

    喜欢,借鉴,学习

  • Sue Huss

    I’m confused can someone just give me the correct text for a wordpress blog. Will uploading it from notepad work then?

  • Dough Roller

    Sue, yes you create the robots.txt file in Notepad and upload it to your server. The robots.txt is a simple text file. You can also create it in the text editing function that is probably available within your file manager on the server that hosts your site. As for the text you can check out http://www.doughroller.net/robots.txt to see what I did (I run WordPress). I took this code from ask apache. Good luck.

  • BusySphere

    Thanks for the post. While uploading the robots.txt file, it has been kept under “/” and “/public_html” folders. Which is the correct root directory?

  • Michael Aulia

    It should be under your public_html, as that’s the folder of your root files under your domain


    Michael Aulia
    http://www.michaelaulia.com

  • Richard

    I just found this article as I searched for robots.txt wordpress SEO. I did this search after checking my google webmaster tools. Go gave me a bund of warnings about unfound pages. These specifically included trackback and feed pages which you exclude in your robots.txt file. Thanks for the valuable help,
    Richard

  • Krystyl

    Is there a way to do this on blogger?

  • Webagentur

    Thank you, this Example has me very helped. But why was not already a robots.txt during the installation process?

  • Full pc utilities downloads

    Great tutorial, thanks for sharing this with us.. I’ve learn a lot.

  • drift80

    Optimizing robots.txt file will help prevent Google penalizing you for duplicate content.

  • seo

    Nice post. Great reminder of how you can easily protect folders and files on your server.

  • Bang Kritikus

    Google don’t like duplicate content so you just allowed one.

  • Live4pk Dot Com

    I have a website on dot com, it is simple website which updated monthly, do i need to creat? please help me clearly or give a sample text file which we can edit and use who dont have experience to do.

  • KoolDesigning USA

    Really great post, thanks for sharing. I am new here and agree with you that robot.txt file is more important when you did not allowed the search engine robots the specific page or folder at your web site.

    Thanks again and Best regards,

    KoolDesigning

  • leon

    great tips … wonder if we can disallow : /wp-register.php ?

  • Werbeagentur Berlin

    Great Tutorial, thank you. This has me very helped.

  • Kamal Hasa

    Thank you for this post. But askapache has got a better topic on the same subject..

  • blogging posts

    I real want to know, how can we do with the image, dis or allow in robots.txt.

  • Kitchener Waterloo SEO

    I like your idea of blocking whole directories instead of mentioning specific files (which I’ve seen done). For example:

    User agent: *

    Disallow: /my_passwords_file.html

    Robot.txt are considered best practice, but they can leave your website open to exploits if your not careful.

  • Daily Good Tips

    Create robot.txt just can be created in wordpress or website with wordpress but I think robot.txt creating can not be applied in blogspot template. We just talk google if we want to change the robot.txt.

  • Keith Davis

    Hi Dan
    That’s my next step for my site… creating a robots.txt file.

    I’ve always understood the problem of duplicate content, but the penny has only just dropped about how easy it is to create duplicate content with a wordpress site.

    Great solution, to use a robots.txt file.

  • Saifa Raz

    This Article is Great. thanks for the Information.
    but r these really useful for indexing by Google ?

  • Alan

    Hi,

    My only issue is that WP has a virtual robots.txt file. This file, for example, is used with the xml sitemap plugin to list the Sitemap.

    So as not to have a duplicate Sitemap, I would imagine that I should tick the xml sitemap plugin option NOT to show the Sitemap in the virtual robots.txt file & put it the the new actual robots.txt file instead.

    Does this sound right? Thanks for the post!

Comments are closed.