The Google Blog On Robots.txt
The Google blog has a nice ongoing set of tutorials on how to use the Robots Exclusion Protocol rules to control how and what search engines index on your site. The first part was published last month and this afternoon they posted a sequel.
Most of the information in the little tutorials applies to all search engines that follow robots.txt, though a couple of things are specific to Google.
And even if you think you know everything about robots.txt already there still might be a few surprises for you in these tutorials. For instance I never knew that it was possible to stop Google from displaying the little summary text snippets below the results links. I still can’t think of a situation where that would be helpful, but it’s good to know should the need arise.
Today’s post promises at least one more short tutorial detailing common exclusion problems that and how to solve them so stay tuned. Also worth checking out is Google’s overall guide to the Robots Exclusion Protocol as well as the more search engine neutral guidelines at robotstxt.org.
