All posts tagged ‘Amazon’

File Under: Web Services
Apr
29
2011

Amazon Autopsy Reveals Causes of Cloud Death

By

Amazon has apologized to customers affected by last week’s EC2 outage and offered a detailed post mortem about exactly what went wrong. The short answer is that a network update shifted traffic to the wrong router, which then wrecked havoc on Amazon’s US East Region Availability Zone.

In addition to apologizing, Amazon is giving affected customers “a 10 day credit equal to 100 percent of their usage of EBS Volumes, EC2 Instances and RDS database instances that were running in the affected Availability Zone.”

Amazon is also promising to improve its communication with customers when things go wrong, but as we pointed out earlier, the real problem is not necessarily Amazon. While Amazon’s services unquestionably failed, those sites that had a true distributed system in place (e.g. Netflix, SmugMug, SimpleGeo) were not affected.

In the end it depends how you were using EC2. If you were simply using it as a scalable web hosting service, your site went down. If you were using EC2 as a platform to build your own cloud architecture, then your services did not go down. The later is a very complex thing to do, and it’s telling that the sites that survived unaffected were all large companies with entire engineering teams dedicated to creating reliable EC2-based systems.

That may be the real lesson of Amazon’s failure — EC2 is no substitute for quality engineers.

See Also:

Tags: ,
File Under: Programming, Web Apps, Web Services
Apr
25
2011

Lessons From a Cloud Failure: It’s Not Amazon, It’s You

By

Chaos Monkey will eat your cloud.

Amazon’s cloud-hosted Web Services experienced a catastrophic failure last week, knocking hundreds of sites off the web. Some developers saw the AWS outage as a warning about what happens when we rely too much on the cloud. But the real failure of Amazon’s downtime is not AWS, but the sites that use it.

The problem for those sites that were brought down by the AWS outage is the sites’ own failure to implement the one key design principle of the cloud: Design with failure in mind.

That’s not to say that Amazon didn’t fail rather spectacularly, taking out huge sites like Quora, Reddit, FourSquare and Everyblock, but as Paul Smith of Everyblock admits, while Amazon bears some of the responsibility, Everyblock failed as well:

Frankly, we screwed up. AWS explicitly advises that developers should design a site’s architecture so that it is resilient to occasional failures and outages such as what occurred yesterday, and we did not follow that advice

But perhaps the most instructive lesson comes from those sites that were not affected, notably Netflix, SimpleGeo and SmugMug. Netflix published a look at how it uses AWS last year and, by all appearances, those lessons served the company well, because Netflix remained unaffected by the recent failure.

Among Netflix’s suggestions is to always design for failure: “We’ve sometimes referred to the Netflix software architecture in AWS as our Rambo Architecture. Each system has to be able to succeed, no matter what, even all on its own.”

To ensure that each system can stand on its own, Netflix uses something it calls the Chaos Monkey (no relation). The Chaos Monkey is a set of scripts that run through Netflix’s AWS process and randomly shuts them down to ensure that the rest of the system is able to keep running. Think of it as a system where the parts are greater than the whole.

The photo sharing site SmugMug has also detailed its approach to designing for failure and why SmugMug was largely unaffected by the recent AWS outage. SmugMug co-founder and CEO Don MacAskill, echos Netflix’s redundancy mantra, writing, “Each component (EC2 instance, etc.) should be able to die without affecting the whole system as much as possible. Your product or design may make that hard or impossible to do 100 percent — but I promise large portions of your system can be designed that way.”

MacAskill also has strong words for those who think the recent AWS outage is a good argument for sticking with your own data center: “[SmugMug's] data-center related outages have all been far worse … we’re working hard to get our remaining services out of our control and into Amazon’s.”

“Cloud computing is just a tool,” writes MacAskill, “Some companies, like Netflix and SimpleGeo, likely understand the tool better.”

If you’d like to learn more about how designing for cloud services differs from traditional data-center setups, check out this excellent post on O’Reilly. Also, be sure to read Netflix’s advice and learn from Everyblock’s downtime by following the guidelines in Amazon’s own documentation.

Photo: Technically not a monkey. (DBoy/Flickr/CC)

See Also:

Tags:
File Under: Multimedia, Web Services
Mar
29
2011

Amazon’s New ‘Cloud Drive’: Your Music, Everywhere You Go

By

Apple and Google are both rumored to be working on music streaming services, but the first real competitor to enter the streaming music battle is Amazon.com.

The company has announced Amazon Cloud Drive, a web-based backup service where you can store your documents, photos music and other files. To go along with Cloud Drive, Amazon is offering Cloud Player, which can stream your music library to any web browser or Android mobile device. Cloud Player also allows you to download files and create playlists through its web-based interface. Note, however, that Amazon is using Flash Player to upload and stream music which means it won’t work on Apple’s iOS devices.

Amazon is offering 5GB of Cloud Drive storage for free to anyone with an Amazon account. That number can be bumped to 20GB with the purchase of an Amazon MP3 album. Additional storage works out to $1 per GB, with plans at 20GB, 50GB, 100GB, 200GB and 1000GB.

At the high end 1000GB will set you back $1000 per year, which is enough to buy roughly 10 TB worth of external hard drives every year. Of course your external hard drives don’t live in the cloud and won’t let you listen to music wherever you are, but if it’s just backups you want, clearly there are cheaper alternatives.

The real appeal of Cloud Drive only comes into play when it’s used with Cloud Player. Bring the two together and you can stream your music library to any web browser or Android mobile device.

While an online music locker capable of sending your music to any device feels almost inevitable at this point, Amazon’s offering is, thus far, disappointing. The interface is awkward and looks a bit like Hotmail did when it first launched — primitive. For example, using the web interface, there’s no way to download more than one file at a time.

Any music you already own must be manually uploaded, there is no automatic syncing like you’ll find in file backup services like Dropbox. The Amazon MP3 Uploader can scan your iTunes library and makes uploading a bit smoother, but if you’ve got a sizable music collection — several hundred gigabytes of music — you’re looking at days, if not weeks, to upload everything to Cloud Drive.

Cloud Drive is much better if you’re buying all your music from the Amazon MP3 Store, and clearly this is what Amazon would like you to do. To encourage that, songs purchased from the Amazon MP3 store and saved directly to your Amazon Cloud Drive never count toward your storage limit and are free to store forever. Couple that with rumors of Amazon working on an Android device and it isn’t hard to see how Cloud Drive, though basic at the moment, might one day become a serious iTunes competitor.

But there are possible legal problems Cloud Drive may have to overcome. The record industry still believes that you should buy a new copy of every song for every device you own, and Amazon has no way to determine whether you actually purchased your existing music files. MP3tunes, which also offers to store and stream your music library, was sued by music labels. Amazon, however, doesn’t believe licensing will be a problem. “We don’t need a license to store music,” Craig Pape, director of music at Amazon, tells the New York Times. “The functionality is the same as an external hard drive.”

Amazon may be first to the market among its sizable competitors like Apple and Google, but Cloud Drive in its current form isn’t particularly innovative, nor does it offer many of the features long-standing competitors like Dropbox or MP3Tunes already have. Still, Amazon’s Cloud Drive already has Google and Apple beat on one count — it exist.

See Also:

Tags:
File Under: Browsers
Mar
8
2010

Amazon Is Building a Better Browser for Kindle

By

Browsing the web on one of Amazon’s Kindle e-readers is like taking a step backwards in time. It’s clunky and has only limited support for web standards, and bare-bones JavaScript capabilities.

But now Amazon may be looking to add browser engineers to the Kindle team, according to the job listings on the company’s website.

A job posting for a browser engineer at Lab126, the division of Amazon that develops the Kindle, indicates the company is looking for somebody to develop “an innovative embedded web browser” for a consumer product.

The role at Lab126 includes designing new features for a new browser while supporting the existing code. Job requirements include familiarity with current web standards and web rendering engines, as well as experience with Java and embedded Linux, both of which the Kindle runs.

The Kindle’s current browsing experience is notably subpar. It’s good enough to check your e-mail, post to Twitter or read Wikipedia, but it doesn’t handle images or more complex web apps particularly well. It certainly doesn’t live up to the same vision of the mobile web being outlined by the iPhone, or Android phones like the Droid or Nexus One. And with the coming of the Apple iPad and other threats to Amazon’s dominant e-reader, which should behave on the web about as well as (if not better than) the iPhone, the Kindle had better improve its browser if the device is going to continue to compete with these more capable devices.

Amazon recently launched a beta program for third-party app developers who want to build software for the Kindle.

Apparently, the job listing has been up for a month, but I only became aware of it once CNET’s Stephen Shankland tweeted about it.

Calls to Lab126 and Amazon on Monday morning went unreturned. I’ll update this post if and when I get more information from Amazon or anyone else.

Meanwhile, if you have any advice about improving the Kindle’s browsing mojo, leave it in the comments.

Photo: Charlie Sorrel/Wired.com

Tags: ,
File Under: Business
Sep
3
2008

Amazon Contest Eyes AWS Developers

By

Amazon Web ServicesAmazon Web Services today launched a contest for developers building their web business off of services like EC2 and S3. The Startup Challenge will award one winner $50,000 in cash and $50,000 in AWS credits, plus potential investment from Amazon.

New startups are commonly using one or more of these web services available from Amazon:

  • EC2 hosts web applications. Our tutorial helps you Get Started With Amazon Cloud Computing.
  • S3 is the “simple storage solution” used by even big name startups, like Twitter.
  • EBS provides persistent storage to EC2.
  • SimpleDB is in beta and provides access to structured data.

In early October Amazon will pick five finalists in the contest, which the public can vote on. A panel of judges will determine the eventual winner. The contest application form is straightforward, with seven long form questions to answer, including the problem being addressed and target customers. Anyone with a qualified entry (I’m assuming this means a site that uses AWS services) receives $25 in AWS credits.

Need some inspiration? Amazon has a list of AWS case studies that show how sites are using their services.

See also:

Tags: , , , , , ,
File Under: Software & Tools
Aug
21
2008

Amazon Expands Elastic Computing With New Storage Service

By

Amazon Web ServicesAmazon Web Services has released a new storage service that, at first, may look a lot like their S3 offering. The new service, Elastic Block Storage (EBS), is meant to increase the usefulness of their EC2 computing cloud.

In case you’re confused by all these services, here’s how Amazon describes their latest addition:

EBS gives you persistent, high-performance, high-availability block-level storage which you can attach to a running instance of EC2. You can format it and mount it as a file system, or you can access the raw storage directly. You can, of course, host a database on an EBS volume.

While S3 is great for storage, EBS is more flexible with its uses. EBS is used in tandem with EC2 instances. But normally, when an EC2 instance goes away, its storage disappears, too. EBS is, as Amazon says, persistent. It sticks around.

Cloud Computing providers RightScale say EBS opens up Amazon’s services to many new customers. Applications not written directly for Amazon’s other offerings are easier to incorporate with EBS. Amazon even points to a tutorial for running MySQL on EBS.

The costs are similar to other Amazon Web Services, which charge by usage. Storage is 10 cents a GB per month. I/O requests are 10 cents per million. There’s a AWS calculator to help you figure out your own costs.

See also:

Tags: , , , ,
File Under: Software & Tools
Jul
30
2008

Amazon’s Mechanical Turk Simplifies Automation

By

Amazon’s automated human intelligence service, Mechanical Turk, has become more useful to a general audience. To use the service in the past required advanced programming skills. Now Amazon has created an interface to bulk load tasks that require human eyes.

Mechanical Turk answer options

Mechanical Turk can help with data collection and correction. Many are already using the service for filtering out obscene photos/comments, or appropriately categorizing and tagging items. Most pay just a few pennies for the answers.

The site is equally catering itself to the workers. There seems to be plenty of interest in the work, called Human Intelligence Tasks, or HITS. I accidentally posted my test HIT, and received eight responses before I could take it offline.

One of the recent changes to Turk are the HIT templates that give requestors an example to start with. Then Amazon provides a comma-separated file to fill in with your data and re-upload. Previously this had to interface with your database via web services.

The name Mechanical Turk comes from an 18th century hoax. A chess-playing machine invented to impress a royal was later revealed to be controlled by a human. As such, Amazon calls its service “artificial artificial intelligence,” because HITs are automated, yet completed by a human.

See also:

Tags: ,
File Under: Other
Jul
21
2008

S3 Outage Makes Developers Consider Redundancy

By

Amazon Web ServicesAt least the downtime gods picked a notoriously low traffic day to punish Amazon’s S3 storage service. The darling of many web apps, including popular Twitter, was down for eight hours Sunday.

Amazon’s service provides storage and transfer of data for a small fee, so that developers can let their own servers focus on more important issues. For small upstart sites, it’s cheaper to buy their data a tiny slice at a time than to invest in a lot of hardware. S3 is especially popular for image hosting, which can be large files in comparison to trim HTML. Also, there are often many images per web page, putting extra strain on a server.

Among the revelations for some developers after a third of a day without S3 is that no single service can be counted on for 100% uptime. Of course, there’s oodles of redundancy built in to Amazon’s service. Yet, still it can go down, leaving many sites that count on it with a single point of failure.

Dave Winer sees a business opportunity in S3 redundancy:

“It would be easy to hook up an external service to S3, and for a fee, keep a mirror on another server. Then it would be a matter of redirecting domains to point at the other server when S3 goes down.”

Developers could achieve the same result Winer mentions on their own. Robert Accettura notes how WordPress.com weathered the S3 outage gracefully:

“They have (slower) back up’s in house for when S3 is down and can failover if S3 has a problem. This means they can leverage S3 to their advantage, but aren’t down because of S3.”

As many have noted, true web scalability and redundancy can be a tough sheep to sheer. While Larry Dignan questions if S3 is too complicated, I think the larger issue is that it is too simple. Too many have viewed it as a silver bullet, with Amazon doing the dirty work for them. This outage (and another back in February) has shown that S3 and services like it can help us a lot, but we still need to do our own work.

See also:

Tags: ,
File Under: Software & Tools
May
19
2008

Run Ajax Apps on Bare S3

By

S3Amazon’s Simple Storage Service is designed to host data, not entire applications. It’s made to just serve as the backend to an app server — typically in Amazon’s cloud.

Tom Evslin has worked out a way to host dynamic Ajax applications directly in the storage, with no intermediate server. Very cool.

See Also:

Tags: , ,