Comments on: Collection of Robots.txt Files http://www.dailyblogtips.com/collection-of-robotstxt-files/ Thu, 09 Feb 2012 07:21:17 +0000 hourly 1 By: Dan from Lawn Mower Reviews http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-1291030 Dan from Lawn Mower Reviews Fri, 25 Mar 2011 05:00:40 +0000 http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-1291030 Hi, I use the All in one SEO plugin on most of my sites. Will this automatically insert a robot.txt file for me? I have heard about this before, but to be honest I kinda overlooked it and didn't think it was too important, however now I am a little worried that I am missing a vital trick here. I would guess that wordpress automatically create the robot.txt file for you, but just double checking? Thanks Hi,
I use the All in one SEO plugin on most of my sites. Will this automatically insert a robot.txt file for me? I have heard about this before, but to be honest I kinda overlooked it and didn’t think it was too important, however now I am a little worried that I am missing a vital trick here.

I would guess that wordpress automatically create the robot.txt file for you, but just double checking?

Thanks

]]> By: shabi http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-1231876 shabi Mon, 16 Aug 2010 12:56:03 +0000 http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-1231876 hi...nice to see this...but can anyone tell me...whatis the function of these lines in robots.txt... "User-Agent: Googlebot Disallow: /index.xml$ Disallow: /excerpts.xml$" i have seen these lines used gizmodo and other bloggers.. should these 2 lines help in removing duplicate content?? pls i want the answer.. thanks hi…nice to see this…but can anyone tell me…whatis the function of these lines in robots.txt…
“User-Agent: Googlebot
Disallow: /index.xml$
Disallow: /excerpts.xml$”
i have seen these lines used gizmodo and other bloggers..
should these 2 lines help in removing duplicate content??
pls i want the answer..
thanks

]]> By: Free Web Directory http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-885514 Free Web Directory Wed, 03 Jun 2009 08:17:11 +0000 http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-885514 nice stuff, you make ideas for me to change my robots.txt nice stuff, you make ideas for me to change my robots.txt

]]> By: jo http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-871778 jo Mon, 18 May 2009 13:03:57 +0000 http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-871778 what kind of robot.txt should be used for this site: http://alphapal.net/fee-calculator.php it is a paypal fee calculator what kind of robot.txt should be used for this site:

http://alphapal.net/fee-calculator.php

it is a paypal fee calculator

]]> By: Elle http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-866055 Elle Sun, 10 May 2009 20:54:15 +0000 http://www.dailyblogtips.com/collection-of-robotstxt-files/#comment-866055 Should the /wp-content folder be included as well...where all the themes and plugins reside? I've not noticed it listed on any robot.txt file anywhere.. Thanks. Should the /wp-content folder be included as well…where all the themes and plugins reside? I’ve not noticed it listed on any robot.txt file anywhere..

Thanks.

]]>