Twitter Asks for Scalability Help From Community
If you’re a Twitter user, chances are you’ve probably noticed — or even Twittered about — your favorite microblogging platform’s inconsistent stability.
Turns out, the speed at which Twitter has been growing has caused a frustrating amount of downtime for the beleaguered service. Twitter’s developer’s blog opened the door Thursday to community discussion in a blog post outlining the formidable scaling challenge it faces.
Twitter engineer Alex Payne poses the problem as such: “Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency’s sake, Twitter was built with technologies and practices that are more appropriate to a content management system.”
Sounds messy. Payne alludes to an architectural problem you can’t solve by just “(throwing) more machines at it.”
The post goes on to claim that the Twitter team has hacked its system to behave like a messaging service. It describes the task ahead as a component-by-component rewrite of the service — in other words, a ground-up redesign.
This begs the question from commenters over Twitter’s use of Ruby on Rails, a web development platform known for fast development. Ruby’s relative newfangledness cause many to believe it lends itself to poor scalability.
Members of the tech community feel for Twitter. TechCrunch has spent some time on the matter, and have responded by condemning the problem but highlighting its frequency among social networks, pointing at MySpace for evidence.
A lengthy post by Eran Hammer-Lahav, who blogs about Twitter and other microblogging topics at Hueniverse.com, brings some perspective to the discussion by dividing scalability issues between delivery and retrieval. Hammer-Lahav suggests delivery is easier to scale than retrieval and calls for caching and mirroring methods akin to e-mail. The trade-off would be a mammoth amount of disk space.
The basis for concern lays firmly in a problem of convenience: the service has gotten too popular. Of course, any start-up would kill to have this problem. Twitter has taken a different tack than usual by engaging the community for advice. So let’s hear it. How would you fix Twitter?
