All posts tagged ‘RSS’

Scaling on a Shoestring, Lessons from NewsBlur

NewsBlur survives a traffic surge after news of Google Reader’s pending demise gets around.
Image: NewsBlur.

One of the more interesting stories to emerge from the demise of Google Reader is that of NewsBlur, a previously small, but very nice, open source alternative RSS reader.

NewsBlur is a one-man operation that was humming along quite nicely, but when Google announced Reader would shutdown, NewsBlur saw a massive traffic spike — in a few short days NewsBlur more than doubled its user base. How NewsBlur developer Samuel Clay handled the influx of new users should be required reading for anyone working on a small site without loads of funding and armies of developers.

“I was able to handle the 1,500 users who were using the service everyday,” writes Clay, “but when 50,000 users hit an uncachable and resource intensive backend, unless you’ve done your homework and load tested the living crap out of your entire stack, there’s going to be trouble brewing.”

Having tested NewsBlur a few times right after Google announced Reader was closing, I can vouch for the fact that there were times when the site was reduced to a crawl, but it came back to life remarkably quickly for a one-man operation.

In his postmortem, Clay details the moves he had to make to keep NewsBlur functioning under the heavy load — switching to new servers, adding a new mailing service (which then accidentally mailed Clay 250,000 error reports) and other moments of rapid, awkward growth.

It’s also worth noting that Clay credits the ability to scale to his premium subscription model, writing that, “the immediate benefits of revenue have been very clear over the past few days.”

As for the future, Clay says he plans to work on “scaling, scaling, scaling,” launching a visual refresh (which you can preview at and listening to feedback from the service’s host of new users.

If you’re looking for a Google Reader replacement, give NewsBlur a try. There’s a free version you can test out (the number of feeds is limited). A premium account runs $24/year and you can also host NewsBlur on your own server if you prefer.

File Under: APIs, JavaScript, Web Basics

RSS in JSON, for Real?

Photo: Kevin Conboy/Flickr

A short while ago Twitter said they were going to move to JSON over XML, without much explanation other than they like JSON and not XML so much these days, etc. I’m a big believer that everyone has the right to support whatever they want when they want for whatever reason, whether they say the truth or not. Because of that belief, I take with a grain of salt every bit of support for every format and protocol. I assume that just because someone supports it today doesn’t tell you for sure that they will support it tomorrow. Though the penalty is usually pretty high for removing support for interfaces people depend on. They tend to remember it next time you ask for their trust. All that is fair game too.

So anyway, this got me thinking again about the possibility that JSON might take over from XML. What then? Should we give up all the interop we get from RSS just because it uses XML and not JSON? And it’s because of all that interop that that day will never come. A transition may happen over a long period of time, and before it’s complete there will be something after JSON. Because smart people see that, they tend to be conservative about switching just for the sake of switching. It’s why the web, which is entirely an XML application, will keep XML support everywhere for the forseeable future.

In other words, I’d bet with virtual 100 percent certainty that it’s safe to keep producing XML-based RSS feeds.

But people like JSON, there’s no denying that. And a JSONified RSS can totally co-exist with the original XML. So let’s have RSS in JSON? That’s a question that seems worth asking about, at this time.

Turns out it is a very straightforward thing to do. I of course have an RSS feed for Scripting News, the blog you’re reading right now. I wrote a script that maintains JSON and JSONP versions of the same content, automatically. When the RSS is built so are the JSON formats. and

I learned a long time ago to embrace change. It’s why there is a RSS today that is derived from the RSS that Netscape shipped in 1999 and has features of my scriptingNews format shipped in 1997. If the world wants to go to JSON, help it get there in a way that benefits from all we learned in the evolution of RSS from 1997 through 2002. It’s stood up pretty well over the years. And there’s wide support for it, and lots of understanding of how it works. If there is to be a JSON-based syndication standard, we can cut years off the development process by simply accommodating it.

So I put together an invitation to discuss this.

If you find this interesting, give it some thought, and if you have something to say, write a blog post of your own, or write a comment on that page. Obviously there’s no moderation for what goes on your blog, but there will be moderation of the comments. Be aware of that. One feature of the past are personal attacks which are totally pointless and subtract from the discourse, and we should not carry that practice forward. That’s why the moderation. :-)

Otherwise, I totally look forward to hearing what people think.


This post first appeared on Scripting News.

Dave Winer, a former researcher at NYU and Harvard, pioneered the development of weblogs, syndication (RSS), podcasting, outlining, and web content management software. A former contributing editor at Wired magazine, Dave won the Wired Tech Renegade award in 2001.
Follow @davewiner on Twitter.
File Under: APIs, Social, Web Basics

One Foot on the Platform…

There’s an old and wonderful Little Feat song.

Lowell George’s girlfriend can’t make up her mind. How he describes it is what’s so cool. “She’s got one foot on the platform, the other on the train.”

And that’s the best strategy, right now, for a reporter or blogger using Twitter.

You can’t get off the platform, that’s where everyone is. But you need a Plan B, just in case you have to get off the platform. That’s the train.

You need a tool that allows you to publish to Twitter, and at the same time publish to an open system that can be connected to other open systems. So users can create their own Twitter, the same way they use Twitter to follow many sources, without having to go through Twitter.

Twitter is the platform. The feed is the train.

It might sound complicated, but it’s not.

If Twitter were to cancel my account, I would keep posting, and people who followed me on the train (following the analogy) would continue to get my updates. The people on the platform, however — would not.

It’s how we develop strength, and the power to choose, without leaving Twitter.

If Twitter Corp plans on being nice to us, then they should not have a problem with this approach. Their API permits it. It’s consistent with Dick Costolo’s edict that we should put stuff into Twitter, but not take stuff out of it.

It’s a way to preserve journalistic integrity even if Twitter hasn’t yet figured out if it’s in the business of providing a platform for journalism.

This post first appeared on Scripting News.

Dave Winer, a former researcher at NYU and Harvard, pioneered the development of weblogs, syndication (RSS), podcasting, outlining, and web content management software. A former contributing editor at Wired magazine, Dave won the Wired Tech Renegade award in 2001.
Follow @davewiner on Twitter.
File Under: APIs, Web Services

It’s Time to Build a Twitter-Free Twitter

Image: Twitter.

Twitter dropped a bombshell on third-party application developers last Friday — the social network built on the backs of third-party developers and clever, innovative clients has decided it no longer needs them.

Twitter’s blog post is short on specific details, but the gist of it is that Twitter is tightening up its API access for third-party developers. The company has long viewed third-party apps as unnecessary and previously warned developers not to “build client apps that mimic or reproduce the mainstream Twitter consumer client experience.” But thus far it hasn’t enforced that guideline. Now it seems it will.

In a post on the Twitter developer blog titled Delivering a consistent Twitter experience, Michael Sippey, Twitter’s director of product, seems to say that the company wants its official clients — and only its clients — to be the way people use Twitter. Instead of building clients that pull out of Twitter, the company wants developers to “build into Twitter.” In other words, kiss your Tweetbot, Twicca or Hibari goodbye and get ready for some embedded widgets instead of good ol’ tweets.

Much digital ink was spilled over the weekend denouncing Twitter’s policy change or lamenting the potential loss of alternative Twitter clients. Of course Twitter is in charge of Twitter and when you use its service — or build apps on its API — you must suffer its whims.

But Twitter’s decision to start enforcing its API restrictions “more thoroughly” could end up a great thing if it inspires developers to take the essence of what makes Twitter great — succinct, timely messages to and from your friends — and free it from Twitter the company.

An independent and decentralized equivalent to Twitter is certainly not a new idea. The basic building blocks you’d need to build such a system have been with us for many years — a combination RSS, OPML and perhaps PubSubHubbub would cover of most of it — but until now there hasn’t been widespread client developer support for such a system. After all, why go to all the trouble of building a decentralized network on top of open web standards when using the Twitter API is so much easier?

Twitter’s third-party developers now have the answer to that question — because you can’t be locked out of the open web.

Developer Brent Simmons, perhaps best known for creating the Mac-based RSS reading app NetNewsWire, has a basic outline of how Twitter app developers could band together and make something that not only sidesteps Twitter’s coming API restrictions, but the service itself.

“The interesting (to geeks like us) part,” writes Simmons on his blog, is “what system that works like Twitter could exist without a company behind it?”

Simmons then proceeds to break Twitter down to its essentials: “under the hood, following somebody is really just subscribing to a feed of their statuses. Posting is really just updating a feed of your own statuses. So you standardize on a feed format. RSS would work great, of course, and there’s a ton of RSS reading and writing code out there already.”

Instead of Twitter clients, what you’d really be building is a real-time RSS client. That’s not a far-fetched idea. Dave Winer, the forefather of blogging and creator of RSS, has been building one for years. (He’s also been telling everyone to build a distributed Twitter-like publishing system for years.)

Simmons doesn’t address it directly, but it’s worth noting that building such a system doesn’t preclude using Twitter. It’s not either/or, it can be both. In this scenario you’d write a post in a client like Tweetbot and Tweetbot could automatically send it Twitter and to your own feed. Start with both and then a migration away from Twitter would be smoother. Those that want to dump Twitter immediately could do so, but still keep posting to anyone with a client that supports the open structure. Then, if Twitter really does cut out third-party apps completely, the infrastructure necessary to support an open alternative is already up and running.

Simmons has more details for developers on his blog and in a follow-up post that delves more into the logistical complexities, but the basic message to developers is simple: Twitter’s changes means you need to find a better network for your clients to use.

The better network is the one that’s always been there — the web. The advantage for app developers feeling threatened by Twitter’s API changes is obvious. As Simmons writes, “there’s a practical reason to use the open web: your app can’t be shut down.”

The question is, if there were an open alternative would disgruntled Twitter users embrace it? The main argument against any alternative is the so-called network effect: Everyone I know is on Twitter; why would I go somewhere else? But not too long ago no one used Twitter and everyone used Myspace. Everyone used Friendster. Everyone use AOL. People change; networks move. A distributed version of Twitter sans Twitter might well be the web to Twitter’s AOL, but there’s one certainty: We’ll never know until we build it.

File Under: Web Services

Google Reader Gets a ‘+1’ Makeover

Google Reader's new, minimalist interface.

Google Reader, the company’s popular RSS feed reading application, is the latest Google app to integrate with the Google Plus social network. Google Reader, which hasn’t been updated in over a year, also received a facelift to match the new, minimalist interface that’s already available for Google Apps, Gmail and other Google services.

The new Google Plus functionality replaces Google Reader’s built-in sharing tools. The old Google Reader “Like” button has been replaced with a “+1″ button, and the “Share” and “Share with Note” features have been replaced with an option to share items with your Google Plus circles.

The sharing changes are good news if you’ve embraced Google Plus. They’re bad news if you haven’t because Google hasn’t just integrated Google Plus into Reader, it’s removed Reader’s functionality in favor of Google Plus.

Google Reader no longer offers friending, following, shared items or comments. Google’s message is pretty simple: The conversation that used to happen on Google Reader will now happen on Google Plus.

The Google Blog says that killing off Google Reader’s original sharing features “helps [Google] focus on fewer areas, and build an even better experience across all of Google.” In the Google universe all sharing will now happen on Google Plus. Google Plus’ primacy is also reflected in the new Reader interface where the “+1″ button is prominently located right next to the star button, while the Twitter and Facebook sharing tools are buried out of sight in the “Send to” drop-down menu.

It’s a smart move for Google — Google Plus needs more content and shared items from Reader means more content on Google Plus — but one that may leave some users in the lurch.

If you were a heavy user of Google Reader’s sharing capabilities, using it, for example, to follow friends and comment on their shared items, the revamped design is going to make you unhappy. To make matters worse Google does not offer an easy way to migrate your data over to Google Plus. That data — your list of friends and followers — is simply gone. Also gone is the list of all the items you ever shared or liked via Google Reader.

There is an option to export your shared and liked items, along with a list of friends and followers, on Google Reader’s settings page, but it comes with a big catch — the export format. There are two options for exporting your old sharing items, a JSON Activity Stream or a custom Google Reader JSON format. Neither format will do you much good. One was made up for Google Reader and the other is not widely used, meaning there isn’t much software out there that can read your exported data. Google likes to pride itself on its data portability, but in this case there’s nowhere to take your data, making Google’s export efforts disingenuous at best.

Of course Google Reader still exports OPML files, so it’s not hard to dump your subscriptions and move to another feed reader if the revamped Google Reader leaves you wanting. In fact, Google even acknowledges that many users may want to do this, reminding you that “if you decide that the product is no longer for you, then please do take advantage of Reader’s subscription export feature.” In other words, if you aren’t jumping on the Google Plus train, Google is no longer interested in you.

See Also: