Comments on: Yahoo’s Robots-Nocontent Tag

By: Bang Kritikus

Bang Kritikus — Fri, 30 Jan 2009 04:14:09 +0000

How to edit robots.txt in blogspot ?

By: Stephen

Stephen — Fri, 25 May 2007 14:48:11 +0000

To answer your question, Keith, yes it would make it harder. Since the robots-nocontent directive is relatively new, however, and hasn’t been widely adopted (primarily because Google doesn’t support it), Google might decide it’s better for them to make their own standard.

On that note, if a competing directive is going to pop up, I predict it will pop up sooner rather than later. The longer Google and MSN let Yahoo have the stage, the less likely they’ll be able to compete with Yahoo’s mindshare.

By: Daniel Scocco

Daniel Scocco — Thu, 24 May 2007 23:38:54 +0000

Keith, that is true. In fact sometimes it is a good thing that Google has such a giant market share of the search industry.

This allow them to create standards for the whole industry.

By: Keith Dsouza

Keith Dsouza — Thu, 24 May 2007 23:34:13 +0000

If different search engines start having different rules wouldn’t be harder for publishers like us to create content.

It would be hard if another search engine robot like that of microsoft come up with their own rules.

By: Stephen Ward

Stephen Ward — Thu, 24 May 2007 18:11:27 +0000

Glad to be here, Daniel. To answer your question, there's no saying when Google might adopt robots-nocontent, if at all. Google has a strong tendency to be the search industry leader. For example, Google invented the sitemaps protocol that MSN and Yahoo had to then start supporting because of its popularity. Google may decide ignore it completely or even produce their own alternative in lieu of yielding their covetted thought leadership to Yahoo. Only time will tell.