Wikia Buys Grub to Help Power New Search Project
During a keynote address at the O’Reilly Open Source Convention in Portland, Oregon, Wikia co-founder Jimmy Wales announced that his company has acquired the distributed web crawler Grub. The company will use Grub, which it purchased from LookSmart for an undisclosed sum, to build the index of web pages for its open and transparent search engine project currently under development.
Grub is a web crawling engine that uses a distributed computing model to index web pages. If you download and install the client, it will run whenever your computer is idle. It works like SETI@Home, except instead of sniffing for intelligent life, Grub indexes URLs. It reports back what it finds to a central location,m where a database is assembled.
Wikia plans to release the code for Grub under an open source license in the hopes that developers will polish up the tool. Between 2000 and 2005, when the project was active, it was criticized
for ignoring some robots.txt files, which are placed on sites by
webmasters in order to instruct web crawlers how to properly index
their site’s content. Wikia has always emphasized the use of open source tools to build it’s transparent search engine. In fact, Wales and Wikia CEO Gil Penchina have said that their goal is to create a search engine equivalent of the LAMP stack that anyone can use, hack and improve upon. Also coming soon for Grub: a Linux client.
