Google Spiders to Start Crawling The ‘Deep’ Web
Google recently announced it will soon begin indexing the so-called “deep” web, those pages hiding behind HTML forms and other inadvertently spider-blocking HTML elements. The move will potentially open up a whole new range of webpages that were previously invisible to the search engine.
Among the possible wins for Google users is the ability to find pages within sites based on searches of those site. As the Google Webmaster blog explains:
For text boxes, our computers automatically choose words from the site that has the form; for select menus, check boxes, and radio buttons on the form, we choose from among the values of the HTML. Having chosen the values for each input, we generate and then try to crawl URLs that correspond to a possible query a user may have made
The results of those crawls would then show up in your Google search results, potentially offering a faster, more direct way to reach the information you’re searching for.
Before any webmasters out there freak out about the possibility that Google will index pages you don’t want indexed, the Google spiders will still obey any robots.txt, nofollow, and noindex rules. However, if you have a site you don’t want crawled and you’ve been relying on a form as a means of blocking spiders, it’s time to break out the robots.txt file and specifically disallow your pages.
Another fairly humorous scenario mentioned on Hacker News serves as a reminder that using GET to modify content is very bad idea. One poor webmaster discovered the Google spider accidentally deleted his whole site by following GET-based delete URLs — don’t be that guy.
Google says that the new form-filling spiders will only be crawling certain sites, though it doesn’t offer any details about which sites it will hit.
We’ll have to wait a while to see how well this experiment works, but if it does, it could potentially open up a whole new wealth of information.
[via Slashdot]
See Also:
