As we said in our initial report on this debacle, if you’re a developer looking for data to use in a mashup, think twice about Craigslist. The site has a wealth of data, but it guards it jealously and has no qualms about blocking even major players like Yahoo.
Thinking of building a mashup using Craigslist data? You might want to look elsewhere for classified listings.
Craigslist has a long history of being openly hostile to outside developers, but the site took its walled garden mentality to new levels with a recent decision to block Yahoo’s popular Pipes tool from accessing the site’s data.
According to developer Romy Maxwell, Craigslist has blocked Yahoo Pipes primarily to stop his mapping mashup, Flippity. The Flippity mashup was using Yahoo Pipes to gather Craigslist posts from its public RSS feeds and plot them on a map. This apparently rubbed Craiglist the wrong way, but rather than just block the offending Pipe, Craigslist decided to block the whole service — effectively killing thousands of Yahoo Pipes-based mashups.
However, unlike ListPic, which Craigslist claimed was using resources excessively, the latest ban doesn’t seem to be about traffic.
In his blog post, Maxwell claims his application complies with Craigslist’s terms of service and doesn’t use excessive bandwidth. In fact, according his post, the Yahoo Pipe in question is private and only runs once every fifteen minutes. Couple that with the fact that Yahoo Pipes caches the data it scrapes, and one could conclude that bandwidth concerns should not be a factor in the shutdown.
As Wired magazine reported in its cover story about Craigslist earlier this year, the company has been quick to ban third-party search services which “[subvert] Craigslist’s mission to enable local, face-to-face transactions” or services which “increase the risk of scams and can be exploited to snatch up bargains, giving technically sophisticated users an advantage over casual browsers.” Flippity doesn’t appear to have been introducing any feature that would enable such behavior.
So, perhaps Flippty was in violation of the TOS? Maxwell asked founder Craig Newmark about the app and Newmark told him, “as a rule of thumb, [it's] okay to use RSS feeds for noncommercial purposes.”
If it’s unlikely to be about bandwidth and there’s no overt violation of the TOS, why ban Yahoo Pipes?
Frankly, we’re not sure. Craigslist isn’t talking — we’ve contacted the company and asked for an explanation, but as of this writing, nobody has gotten back to us, nor has any public statement been issued.
One thing that is clear though, if you’re thinking of adding Craigslist data to your application, think again.
For its part, Maxwell says Flippity will continue, but plans to use data from developer-friendly sites like eBay and Oodle.
Yahoo’s data plumbing application, Pipes, lets you turn RSS feeds and output from APIs into data you can use. It’s extremely powerful.
One cool feature I hadn’t noticed is that it has the ability to output to a web service. So, you can set up a page on your own server to accept input from Pipes when the result changes. While I love the RSS output option, if you’re building an app on top of Pipes, you’d need to ping the RSS feed often. The web service feature means always having the latest data available to Pipes.
The Pipes team posted a brief tutorial showing how to use Pipes with AppJet and Google App Engine. In these cases, you don’t even need to have your own server to accept the output–and do something–with Pipes data.