All posts tagged ‘pipes’

File Under: APIs, Web Services

Craigslist Reverses Yahoo Pipes Ban, But Developers Have Already Moved On

Earlier this month, Craigslist blocked Yahoo Pipes from accessing any Craigslist page. As a result, mashups all over the web were suddenly without data.

After not responding to inquires, Craigslist CEO Jim Buckmaster eventually posted a terse message on the company’s blog saying that Yahoo Pipes mashups were using “a disproportionate amount of server/bandwidth resources.”Now it would seem that Craigslist has ended its ban of Yahoo Pipes, but for developers we imagine the damage has been done. For its part, the startup Flippity, which was the first to notice its Yahoo Pipe had been blocked, says it has moved on to friendlier sources, rewriting Flippity to use the eBay API.

It wasn’t the first time Craigslist has shutdown outsiders trying to improve on the site’s famously antiquated tools — the site previous blocked ListPic, a tool designed to help Craigslist users browse by images, and a tool to search all the Craigslist sites at once.

As we said in our initial report on this debacle, if you’re a developer looking for data to use in a mashup, think twice about Craigslist. The site has a wealth of data, but it guards it jealously and has no qualms about blocking even major players like Yahoo.

See Also:

File Under: Web Services

Craigslist to Developers: Look Elsewhere For Mashup Data

Thinking of building a mashup using Craigslist data? You might want to look elsewhere for classified listings.

Craigslist has a long history of being openly hostile to outside developers, but the site took its walled garden mentality to new levels with a recent decision to block Yahoo’s popular Pipes tool from accessing the site’s data.

According to developer Romy Maxwell, Craigslist has blocked Yahoo Pipes primarily to stop his mapping mashup, Flippity. The Flippity mashup was using Yahoo Pipes to gather Craigslist posts from its public RSS feeds and plot them on a map. This apparently rubbed Craiglist the wrong way, but rather than just block the offending Pipe, Craigslist decided to block the whole service — effectively killing thousands of Yahoo Pipes-based mashups.

This isn’t the first time Craigslist has shutdown outsiders trying to improve on the site’s famously antiquated tools — the site previous blocked ListPic, a tool designed to help Craigslist users browse by images and a tool to search all the Craigslist sites at once.

However, unlike ListPic, which Craigslist claimed was using resources excessively, the latest ban doesn’t seem to be about traffic.

In his blog post, Maxwell claims his application complies with Craigslist’s terms of service and doesn’t use excessive bandwidth. In fact, according his post, the Yahoo Pipe in question is private and only runs once every fifteen minutes. Couple that with the fact that Yahoo Pipes caches the data it scrapes, and one could conclude that bandwidth concerns should not be a factor in the shutdown.

As Wired magazine reported in its cover story about Craigslist earlier this year, the company has been quick to ban third-party search services which “[subvert] Craigslist’s mission to enable local, face-to-face transactions” or services which “increase the risk of scams and can be exploited to snatch up bargains, giving technically sophisticated users an advantage over casual browsers.” Flippity doesn’t appear to have been introducing any feature that would enable such behavior.

So, perhaps Flippty was in violation of the TOS? Maxwell asked founder Craig Newmark about the app and Newmark told him, “as a rule of thumb, [it's] okay to use RSS feeds for noncommercial purposes.”

If it’s unlikely to be about bandwidth and there’s no overt violation of the TOS, why ban Yahoo Pipes?

Frankly, we’re not sure. Craigslist isn’t talking — we’ve contacted the company and asked for an explanation, but as of this writing, nobody has gotten back to us, nor has any public statement been issued.

One thing that is clear though, if you’re thinking of adding Craigslist data to your application, think again.

For its part, Maxwell says Flippity will continue, but plans to use data from developer-friendly sites like eBay and Oodle.

See Also:

File Under: Uncategorized

Build an App With Yahoo Pipes

Yahoo PipesYahoo’s data plumbing application, Pipes, lets you turn RSS feeds and output from APIs into data you can use. It’s extremely powerful.

One cool feature I hadn’t noticed is that it has the ability to output to a web service. So, you can set up a page on your own server to accept input from Pipes when the result changes. While I love the RSS output option, if you’re building an app on top of Pipes, you’d need to ping the RSS feed often. The web service feature means always having the latest data available to Pipes.

The Pipes team posted a brief tutorial showing how to use Pipes with AppJet and Google App Engine. In these cases, you don’t even need to have your own server to accept the output–and do something–with Pipes data.

If you’re brand new to Pipes, be sure to check out my Pipes tutorial.

[via Amber Case]

See also: