Webmonkey » Databases

Behind the Scenes at Instagram: Tools for Building Reliable Web Services

Scott Gilbertson — Tue, 10 Apr 2012 16:58:59 +0000

In case you missed it, yesterday Facebook acquired Instagram, a photo-sharing service with some 30 million users and hundreds of millions of images on its servers.

The reported sale price of one billion dollars no doubt has many developers dreaming of riches, but how do you build a service and scale it to the size and success of Instagram? At least part of the answer lies in choosing your tools wisely.

Fortunately for outside developers, Instagram’s devs have been documenting the tools they used all along. The company’s engineering blog outlined its development stack last year and has further detailed how it uses several of the tools it’s chosen.

Instagram uses an interesting mashup of tried-and-true technologies alongside more cutting-edge tools, mixing SQL databases with NoSQL tools like Redis, and chosing to host its traditional Ubuntu servers in Amazon’s cloud.

In a blog post last year Instagram outlined its core principles when it comes to chosing tools, writing, “keep it very simple, don’t reinvent the wheel [and] go with proven and solid technologies when you can.”

In other words, go with the boring stuff that just works.

For Instagram that means a Django-based stack that runs on Ubuntu 11.04 servers and uses PostgreSQL for storage. There are several additional layers for load balancing, push notifications, queues and other tasks, but overwhelmingly Instagram’s stack consists of stolid, proven tools.

Among the newer stuff is Instagram’s use of Redis to store hundreds of millions of key-value pairs for fast feeds, and Gunicorn instead of Apache as a web server.

All in all it’s a very impressive setup that has, thus far, helped Instagram avoid the down time that has plague many similar services hit with the same kind of exponential growth. (Twitter, I’m looking at you.) For more details on how Instagram looks behind the scenes and which tools the company uses, be sure to check out the blog post as well as the archives.

OpenStreetBlock Gives Geodata the Human Touch

Scott Gilbertson — Thu, 07 Apr 2011 17:27:18 +0000

Location-based web services are all the rage right now, but for most of us the actual geographic location isn’t very interesting — do you know where “40.737813,-73.997887″ is off the top of your head? No? How about “West 14th Street bet. 6th Ave. and 7th Ave?”

For the geographic web to become useful geodata has to be converted into something humans actually understand. Enter OpenStreetBlock.

OpenStreetBlock is a new web service that takes geographic coordinates (latitude/longitude pairs) and turns them into an actual city block description. The result is textual information which, in many cases, will be even more meaningful to your users than the ubiquitous pin on a map.

If you’d like to play around with a sampling of data from New York, head over to OpenStreetBlock and try out the New York demos.

If you’ve ever wanted to build your own version of EveryBlock — which pinpoints events, news stories and public data at the city-block level — OpenStreetBlock will go a long way toward getting you there. So long as you can pull geo coordinates out of your source data, OpenStreetBlock can turn that into more meaningful information.

Under the hood OpenStreetBlock relies on OpenStreetMap data and uses PHP in conjunction with a geographic database to turn your coordinates into block descriptions.

As cool as OpenStreetBlock is, getting it up and running on your own site will require a bit of work. Luckily, there are some good tutorials available that will walk you through the process of installing and setting up many of the prerequisites like PostgreSQL and PostGIS (I’ll assume you already have an Apache server with PHP installed).

To get started with OpenStreetBlock, grab the code from GitHub. The next thing you’ll need is a PostgreSQL database with all the PostGIS tools installed. Luckily those are also prerequisites for GeoDjango, so head over to the GeoDjango installation page, skip the Django-specific parts and just follow the Postgres and PostGIS installation instructions.

Next you’ll need to download Osmosis and Osm2pgsql to convert OpenStreetMap data into something Postgres can handle. Head over to OpenStreetMap, zoom into an area you’d like to query with OpenStreetBlock and then choose “export.” Select the OpenStreetMap XML Data option and save the file.

From there you can check out the guide to importing the OpenStreetMap XML Data in the OpenStreetBlock read me.

See Also:

Open Data’s Access Problem, and How to Solve it

Scott Gilbertson — Wed, 22 Sep 2010 19:05:34 +0000

The recent Gov 2.0 summit in Washington D.C. saw several promising new announcements which will help government agencies share code and best practices for making public data available to developers.

The idea behind new projects like Challenge.gov, the FCC’s new developer tools and the Civic Commons is that by giving developers access to data previously stored in dusty filing cabinets, they can create tools to give ordinary citizens greater access to that data.

Unfortunately, not everything open data project leads to good things. It is critical that if open data is made available on the web, it must be accompanied by some effort to ensure everyone can access it.

We’ve seen an explosion in creative hacks that use this newly available data to provide excellent online resources. Public data sites like EveryBlock, or the Sunlight Foundation’s Design for America contest have highlighted some of the amazing ways open data can make our lives better. Whether it’s finding out crime stats, real estate values, health hazards and business license statuses in your neighborhood, or visualizing how the government is spending your tax dollars through innovative maps, open data and what you can do with it is the current hotness among web developers.

Most of the benefits are close to home — in the U.S., just about everyone has access to online government resources thanks to web-enabled computers in free public libraries.

But extend that argument to the rest of the world and the number of people that really have access to the data drops significantly. If you don’t have an easy way to get online, you can’t benefit from open data.

Michael Gurstein, Executive Director of the Center for Community Informatics Research, recently highlighted some of the problems with open data accessibility.

Gurstein points out a number of assumptions about open data that are often overlooked by those most enthusiastic about making such data publicly available.

Worse, he shows how such data can be used against you.

Gurstein’s example of the dark side of open data is Bangalore, India’s digitization of land records, which gives every citizen a way to see who owns what in Bangalgore. On the surface, it seems like a good thing, but the upper classes and corporations have been using the land records data to gain ownership of land from the unknowing poor.

The data, writes Gurstein, allowed the well-to-do to instruct surveyors and lawyers how to most effectively “challenge titles, exploit gaps in title, take advantage of mistakes in documentation, identify opportunities and targets for bribery” among other things. Details are in this PDF.

It isn’t necessary to go all the way to India to find examples of open data leading to unintended consequences.

In an e-mail exchange, Gurstein told me of a similar case in Nova Scotia where efforts to make titles, deeds and other land data led to very same situation — companies pouring over 19th century deeds, ancient maps and other newly available data, finding oversights, misfiled papers and other means to seize land from owners.

Of course unintended consequences aren’t a reason to stop making data available. For Gurstein, the solution is to make sure that open data isn’t just thrown onto the web, but that universal accessibility is built in so it can really benefit everyone.

How that is done will vary considerably by location and the type of data in question, but without such efforts Gurstein worries that “the outcome of ‘open data’ may be quite the opposite to that which is anticipated (and presumably desired) by its strongest proponents.”

It might come as a shock to some of the more enthusiastic open data proponents, but there is more to open data than just dredging it out of the Indiana Jones-style warehouses where it currently gathers dust. Putting it online for “anyone” access and just walking away isn’t necessarily a recipe for good things.

Gurstien also pointed out several solutions to me, which he lists in a follow-up blog post. These solutions would help ensure that what happened in Nova Scotia and Bangalore won’t happen elsewhere. Among the things he believes governments and other data providers need to take into account are:

Advocacy — Perhaps the most important of Gurstein’s guidelines is to ensure that everyone knows the data is available, making sure that a community’s resources are sufficient for turning the data into some kind of project with local benefits.
Internet access — Especially a concern in rural areas, the level of internet access is the cornerstone to open data. Just because data is on the web does not mean everyone can get to it. And if not everyone has access, then your data isn’t “open.”
Content and formatting — If the data just a raw GIS database that most people won’t understand, then even internet access doesn’t matter because only those with specific skills (or the money to hire them) will be able to do anything with the data.
Computer/software skills — Similar to content and formatting issues is having access to GIS tools and other specialty software. As Gurstein says “techies know how to do the visualization stuff, university and professional types know how to use the analytical software but ordinary community people might not know how to do either.”

It’s also worth pointing out that Gurstein has several examples of open data being used in constructive ways. He isn’t arguing that we shouldn’t put government and other data online, just that we should keep in mind that the data isn’t necessarily useful to everyone in its most raw forms.

As Tim O’Reilly notes in conjunction with Gurstein’s post, “we need to think deeply about the future” — to consider all the consequences of open data, not just the ones we’d like to see.

Punchcard scan by Steve Collins/Flickr/CC

See Also:

Big Data in the Deep Freeze: John Jacobsen of IceCube

Michael Calore — Fri, 23 Jul 2010 21:16:51 +0000

John Jacobsen works for the IceCube telescope project, the world’s largest neutrino detector, located at the South Pole. The project’s mission is to search for the radioactive sub-atomic particles that have been generated by violent astrophysical events: “exploding stars, gamma ray bursts, and cataclysmic phenomena involving black holes and neutron stars,” according to the project website.

Jacobsen is one of the people in charge of handling the massive amounts of data collected by IceCube. In the video, shot this week at the O’Reilly OSCON 2010 conference in Portland, Oregon, John explains how they collect a terabyte of raw data per hour, then send everything to IceCube’s remote research and backup facilities using a finicky satellite hook-up.

Antarctica is one of the least accommodating places on Earth to perform scientific research with computers. It’s the driest spot on the planet — atmospheric humidity hovers around zero — and bursts of static electricity threaten the integrity of IceCube’s data stores. The lack of humidity causes the server clusters’ cooling systems to break down. And if something fails, a spare might take six months to arrive.

Sunlight Labs Offering $5K for Best Government Data Mashups

Scott Gilbertson — Mon, 29 Mar 2010 18:30:30 +0000

Artists, web developers and data visualization geniuses, here’s a chance to strut your stuff, serve your country and win some serious money in the process.

Sunlight Foundation, a non-profit organization that provides tools to make government data more transparent, has announced a new contest called Design for America. Billed as a “design and data visualization extravaganza,” Sunlight is encouraging the public to create and publish data visualizations that help make complex government data easier for people to digest and interact with.

There are several different categories open for submission, including: visualizations of Recovery.gov data that shows how the stimulus money is being spent, visualizations showing how a bill becomes a law, a redesign of a .gov website, and a redesign of any government form. Top prize in each category is a cool $5,000.

Creations can be in any form — a website, a game, a poster, a sculpture, whatever — though we suspect most of the entries will be either posters or interactive Flash graphics.

The contest is being run by Sunlight Labs, the skunkworks wing of the larger Sunlight Foundation. The Sunlight group spends most of its energy collecting government data, organizing it into publicly accessible databases, then creating tools that make it easier for ordinary people to access that data. The non-profit works with organizations like OpenCongress, MapLight, FollowTheMoney and USASpending.gov. Sunlight also maintains a list of APIs developers can use to access the data.

The Design for America contest encourages participants to sift through the vast datasets available from all of these organizations, as well as the datasets maintained by Sunlight Foundation and any raw government data that’s available. As the Sunlight Labs blog says, the goal of the contest is to “tell interesting stories” that go beyond what can be an overwhelming amount of unfiltered data.

Visualizations can be in any medium, not just the web, so if you’re a video or infographic specialist, you can still enter the contest. The main criteria for judging are the visual quality of the artwork and how well the underlying information is conveyed.

Design for America follows in the footsteps of Sunlight’s Apps for America contests, which sought to bring transparency and accountability to government using open web applications to mine government data sites. Apps for America generated a number of useful apps — like RSS feeds for the House Committee schedule, GovPulse and the Earmark Watch tool, among others. With any luck, the Design for America contest will generate some equally useful and enlightening visualizations of government data.

If you’d like to test your mashup and design chops, head over to Sunlight Labs and have a look at the full contest rules. One important thing for designers to keep in mind — your entry needs to be licensed under a Creative Commons By-Attribution or Creative Commons 0 license, or an OSI approved license for those submitting code.

The contest judges vary by category, but the list includes some big names in the online design world — Nicholas Feltron and Nathan Yau are both on the list, as are various members of Sunlight Labs and other design gurus.

Design for America submissions are due by May 17 and the winners will be announced at the Gov 2.0 Expo in Washington, D.C. on May 27.

See Also:

PHP and MySQL Tutorial – Lesson 3

Webmonkey Staff — Tue, 16 Feb 2010 01:45:47 +0000

Welcome to the third and final lesson for this tutorial. If you’ve gone through Lesson 1 and Lesson 2, you already know the essentials for installing and writing useful scripts with MySQL and PHP. We’re going to look at some useful PHP functions that should make your life a lot easier. First, let’s look at include files.

We all know the basics of includes, right? Contents of an external file are referenced and imported into the main file. It’s pretty easy:You call a file and it’s included. When we do this in PHP there are two functions we need to talk about:include() and require(). The difference between these two functions is subtle but important, so let’s take a closer look. The require() function works in a XSSI-like way; files are included as part of the original document as soon as that file is parsed, regardless of its location in the script. So if you decide to place a require() function inside a conditional loop, the external file will be included even if that part of the conditional loop is false.

A Place for Everything

The include() function imports the referenced file each time it is encountered. If it’s not encountered, PHP won’t bother with it. This means that you can use include in loops and conditional statements, and they’ll work exactly as planned.

Finally, if you use require() and the file you’re including does not exist, your script will halt and produce an error. If you use include(), your script will generate a warning, but carry on. You can test this yourself by trying the following script. Run the script, then replace include() with require() and compare the results.

I like to use the suffix .inc with my include files so I can separate them from normal PHP scripts. If you do this, make sure that you set your Web server configuration file to parse .inc files as PHP files. Otherwise, hackers might be able to guess the name of your include files and display them through the browser as text files. This could be bad if you’ve got sensitive information – such as database passwords – contained in the includes.

So what are you going to do with include files? Simple! Place information common to all pages inside them. Things like HTML headers, footers, database connection code, and user-defined functions are all good candidates. Paste this text into a file called header.inc.









<?php echo $title ?>

Then create another file called footer.txt that contains some appropriate closing text and tags.

Now let’s create a third file containing the actual PHP script. Try the following code, making sure that your MySQL server is running.

n";

echo "NamePositionn";

while ($myrow = mysql_fetch_row($result)) {

  printf("%s %s%sn", $myrow[1], $myrow[2], $myrow[3]);

}

echo "n";

include("footer.inc");

?>

See what happens? The include files are tossed into the main file and then the whole thing is executed by PHP. Notice how the variable $title was defined before header.inc is referenced. Its value is made available to the code in header.inc; hence, the title of the page is changed. You can now use header.inc across all your PHP pages, and all you’ll have to do is change the value of $title from page to page.

Using a combination of includes, HTML, conditional statements, and loops, you can create complex variations from page to page with an absolute minimum of code. Includes become especially useful when used with functions, as we’ll see down the road.On to the exciting world of data validation.

Simple Validation

Imagine for a moment that we’ve got our database nicely laid out and we’re now requesting information from users that will be inserted into the database. Further, let’s imagine that you have a field in your database waiting for some numeric input, such as a price. Finally, imagine your application falling over in a screaming heap because some smart aleck put text in that field. MySQL doesn’t want to see text in that portion of your SQL statement – and it complains bitterly.

What to do? Time to validate.

Validation simply means that we’ll examine a piece of data, usually from an HTML form, and check to make sure that it fits a certain model. This can range from ensuring that a element is not blank to validating that an element meets certain criteria (for example, that a numeric value is stipulated or that an email address contains an @ for an email address).

Validation can be done on the server side or on the client side. PHP is used for server-side validation, while JavaScript or another client-based scripting language can provide client-side validation. This article is about PHP, so we’re going to concentrate on the server end of things.

Let’s ignore our database for the moment and concentrate on PHP validation. If you wish, you can add additional fields to our employee database quite simply by using the MySQL ALTER statement – that is, if you want to commit to the values that we’ll validate.

There are several useful PHP functions we can use to validate our data, and they range from simple to highly complex. A simple function we could use might be strlen(), which tells us the length of the variable.

A more complex function would be ereg(), which uses full regular expression handling for complex queries. I won’t delve into the complexities of regex here, as entire books have been written on the subject, but I will provide some examples on the next page.

Let’s start with a simple example. We’ll check to see whether a variable does or does not exist.

The keys to this script are the nested conditional statements. The first checks to see whether the Submit button has been pressed. If it has, it goes on to check that both the variables $first and $last exist. The || symbol means “or” and the ! symbol means “not.” We could also rewrite the statement to say, “If $first does not exist or $last does not exist, then set $error to the following.”

Next, let’s extend things a little by checking to see whether a string is a certain length. This would be ideal for passwords, since you don’t want some lazy user entering a password of only one or two letters. You’d rather it be, say, six or more characters.

The function for this is, as you already know, strlen(). It simply returns a number equal to the number of characters in the variable being tested. Here, I modified the script above to check the length of $first and $last.

Run this script and try entering six or fewer letters to see what happens. It’s simple yet quite effective.

Not-So-Simple Validation

Let’s talk a bit about using regular expressions with the ereg() and eregi() functions. As I said earlier, these can be either quite complex or very simple, depending on what you need.

Using regular expressions, you can examine a string and intelligently search for patterns and variations to see whether they match the criteria you set. The most common of these involves checking whether an email address is valid (although, of course, there’s no fail-safe way of doing this).

Rather than delve into the mysteries of regular expressions, I’ll provide some examples. You can use the same form we created on the previous page – just paste in the lines below to see how they work.

First, let’s make sure that text only has been entered into a form element. This regular expression tests true if the user has entered one or more lowercase characters, from a to z. No numbers are allowed:

 if (!ereg("[a-Z]", $first) || !ereg("[a-Z]", $last)) {

Now, let’s extend this expression to check whether the string is four to six characters in length. Using alpha: is an easy way to check for valid alphabetic characters. The numbers in the braces check for the number of occurrences. And note that the ^ and $ indicate the beginning and end of the string.

 if (!ereg("^[[:alpha:]]{4,6}$", $first) || !ereg("^[[:alpha:]]{4,6}$", $last)) {

Finally, let’s build a regular expression that will check an email address’ validity. There’s been plenty of discussion about the effectiveness of checking for email addresses in this way. Nothing’s completely foolproof, but what I have below works pretty well.

I took this gem from the PHP mailing list. It’s a great resource – use it. And yes, this is as scary as it looks.

	if (!ereg('^[-!#$%&'*+./0-9=?A-Z^_`a-z{|}~]+'.

'@'.

'[-!#$%&'*+/0-9=?A-Z^_`a-z{|}~]+.'.

'[-!#$%&'*+./0-9=?A-Z^_`a-z{|}~]+$', $last)) {

Don’t spend too much time looking at this. Just move on to the next bit.

Functions

Enjoy that last regex expression? Fun, wasn’t it? Wouldn’t it be even more fun to enter that chunk on a dozen different pages that need to process email addresses?! Think about the joy of finding a typo in that mess – and doing it a dozen times no less. But of course, there’s a better way.

Remember when we talked about include files earlier in this lesson? They’ll allow us to create a piece of code like the email checker and include it multiple times across several pages. This way, when we want to change the code, we need edit only one file, not many.

But if we want to get this done, we’ll have to use functions.

We’ve already used functions plenty of times. Every time we query the database or check the length of a string we’re using functions. These functions are built into PHP. If you’re a keen coder, you can extend PHP with your own customized functions. But that’s a bit advanced for this tutorial. Instead we’ll create functions that will reside within our PHP script.

A function is simply a block of code that we pass one or more values to. The function then processes the information and returns a value. The function can be as simple or complex as we like, but as long as we can pass a value in and get one out, we don’t really care how complex it is. That’s the beauty of functions.

Functions in PHP behave similarly to functions in C. When we define the functions, we must specify what values the function can expect to receive. It’s tricky to get a handle on at first, but it prevents weird things from happening down the road. This is done because the variables inside a function are known as private variables. That is, they exist only inside the function. You may, for instance, have a variable in your script called $myname. If you created a function and expected to use the same $myname variable (with the same value), it wouldn’t work. Alternatively, you could have the variable $myname in your script and also create another variable called $myname in your function, and the two would co-exist quite happily with separate values. I do not recommend doing this, however! When you come back and edit it six months later, you’ll be breaking things left and right. There are exceptions to this rule as with all things, but that’s outside the scope of this article.

So let’s create a function. We’ll start simply. We need to give the function a name and tell it what variables to expect. We also need to define the function before we call it.

That’s it! First, we created our function. Notice how we defined two new variables, called $first and $second. When we call the function, each variable is assigned a value based on the order in which it appears in the list – 4 goes to $first, 5 to $second. Then we simply added the two numbers together and returned the result. “Return” here simply means to send the result back. At the end of the script we print the number 9.

Let’s create something that’s more useful to our database application. How about something that gracefully handles errors? Try this:

 
 

 ";

 	echo "The reported error was $error.n
";

 	echo "Best you get hold of the site admin and let her know.";

 	die;
 }

 if (!$db = @mysql_connect("localhost","user", "password")) {

 	$db_error = "Could not connect to MySQL Server";

 	do_error($db_error);

 }

 ?>

Before running this, try shutting down MySQL or using a bogus username or password. You’ll get a nice, useful error message. Observant readers will notice the @ symbol in front of mysql_connect(). This suppresses error messages so that you get the information only from the function. You’ll also see we were able to pass a variable into the function, which was defined elsewhere.

Remember that I said functions use their own private variables? That was a little white lie. In fact, you can make variables outside of a function accessible to the function. You might create a function to query a database and display a set of results over several pages. You don’t want to have to pass the database connection identifier into the function every time. So in this situation, you can make connection code available as a global variable. For example:

This is a basic function, but the point is that you don’t need to send $db through when you call the function – you can make it available using the word global. You can define other variables as global in this statement, just separate the variable names by a comma.

Finally, you can look like a real pro by using optional function variables. Here, the key is to define the variable to some default in the function, then when you call the function without specifying a value for the variable, the default will be adopted. But if you do specify a value, it will take precedence.

Confused? For example, when you connect to a database, you nearly always connect to the same server and you’ll likely use the same username and password. But sometimes you’ll need to connect to a different database. Let’s take a look.

Isn’t that cool? The variables used inside the function were defined when the function was defined. The first time the function is called, the defaults are used. The second time, we connect to a new host, but with the same username and password. Great stuff!

Think about where you could use other functions in your code. You could use them for data checking, performing routine tasks, and so on. I use them a lot when processing text for display on a Web page. I can check, parse, and modify the text to add new lines and escape HTML characters in one fell swoop.

Now all that’s left to do is to impart some words of wisdom.

Closing Advice

When it comes to databasing, there’s a lot to learn. If you haven’t done it already, find a good book about database design and learn to put together a solid database – on any platform. It’s an invaluable skill and it will save you plenty of time and headache in the long run. Also, learn about MySQL. It’s a complex but interesting database with a wealth of useful documentation. Learn about table structure, data types, and SQL. You can actually achieve some pretty impressive stuff if you know enough SQL.

Finally, there’s PHP. The PHP Web site has nearly everything you need, from a comprehensive manual to mailing-list archives to code repositories. An excellent way to learn about PHP is to study the examples used in the manual and to check out the code archives. Many of the posted scripts consist of functions or classes that you can use for free in your own scripts without having to reinvent the wheel. Additionally, the mailing list is an excellent spot to check out if you get stuck. The developers themselves read the list and there are plenty of knowledgeable people there who can help you along the way.

Good luck and good coding!

Cake and Trax for Beginners

Webmonkey Staff — Tue, 16 Feb 2010 01:45:47 +0000

In my last article, Building With Ajax and Rails I made a faintly disparaging joke about some new web frameworks that have been created in fond imitation of Rails. I got a lot of feedback about that joke. I’m not allowed to comment here about the pending lawsuits, but I would ask that the drive-by eggings of my house and threats to my family please cease. (They’ve been relocated to a secret Webmonkey farm anyway.)

Today we’re going to take a look at a couple of those frameworks for PHP:Trax and Cake. Both attempt to bring the quick, easy, helpful tools and easily understood, easily maintained structure of Rails to PHP — a boon to web developers who know PHP and perhaps have some keeper code in that language, but can’t resist the Rails buzz. Both Trax and Cake use the same model-view-controller pattern and Active Record ways of addressing data that Rails does. Makes one curious, no? I don’t have time to get deeply into them today, but both stress “rapid development,” so let’s see if I, your average not-too-bright web developer, can get a little app off the ground before the end of this article.

Cutting into Cake
Makin’ Trax
Hardly the End of the Story
Helpful Links

Cutting into Cake

This is 100 percent true:At lunch yesterday, my fortune cookie said “A nice cake is waiting for you.” So, I am going to start with Cake first and try Trax later.

I go to the Cake site and download the latest stable release (discouragingly called 1.1.19.6305) onto my Debian web server running Apache and PHP 5. Unzipping the tarchive, I immediately see some names that look familiar from Rails:app, config and something called dispatcher.php. The app directory looks even friendlier. It contains views, controllers, model – just like Rails!

There’s also a webroot directory, which I assume is going to be the root of my application. Indeed, look at that! If I browse to http://my.web.server/trax_and_cake_playground/cake/app/webroot, I see a friendly boilerplate intro page. Later, if I decide I want to use Cake full time, I’ll add a line to my Apache configuration to set that directory as the whole server’s root directory. At the moment, the page is extolling Cake’s coolness and warning me, “Your database configuration file is not present.”

Correct, it’s not. I find a sample database configuration in app/config/database.php.default. I’ll copy that sample to an actual working file, database.php, in the same directory. The file includes an array of settings, which I change to suit my own environment:

var $default = array('driver'   => 'mysql',



                  'connect'  => 'mysql_pconnect',



                  'host'     => 'localhost',



                  'login'    => 'paul',



                  'password' => 'passw0rd',



                  'database' => 'bookmarker' );

Now, when I reload the intro page, it tells me, “Your database configuration file is present. Cake is able to connect to the database.” Yay!

Now, I’ve told it I have a MySQL database called “bookmarker,” but that’s not true yet. I still have to create it. I’m eager to get rolling, so I’ll just recycle the code from Webmonkey’s Ruby on Rails for Beginners article in which I built a bookmarking app. This time I also want to populate the table with a little starter data, to give us something to work with. So I type:

 $mysql



 mysql> CREATE DATABASE bookmarker;



 mysql> USE bookmarker;



 mysql> CREATE TABLE bookmarks (



    ->  id SMALLINT (5) NOT NULL auto_increment,



    ->  url VARCHAR (255),



    ->  name VARCHAR (100),



    ->  description VARCHAR (255),



    ->  PRIMARY KEY (id)



    -> );



 mysql> INSERT INTO bookmarks (url, name, description) VALUES



 ('http://webmonkey.com/', 'Webmonkey', 'Not the only monkey that makes



 websites, but the best!');



 mysql> INSERT INTO bookmarks (url, name, description) VALUES



 ('http://ubuntu.com/', 'Ubuntu', 'Linux for human beings (no monkeys



 please!)');



 mysql> exit;

At this point in developing a Rails app, I’d run the built-in scaffold-generating tool. Here in Cakeland, I see a file called bake.php in cake/scripts, and instantly I pick up on the metaphor – bake a cake, get it? Turns out that’s a false start, as bake.php doesn’t seem to work in the pre-beta. It’s giving me all sorts of fatal errors and files not found. I’m going to have to create my model, view and controller by hand-editing code.

The app/models directory contains an empty file called put_models_here. So, I do just that, creating a file in that directory called bookmark.php:

 set('data',$this->Bookmark->findAll());



     }



 }



 ?>

That puts the data from the table in $data, using the built-in set method. You might want to read along in the Cake API documentation – I certainly am!

Then I create a controller, app/controllers/bookmarks_controller.php:

I will also need to create a view. In the app/views directory, I create a subdirectory called bookmarks and a file called index.thtml inside of the subdirectory. The “t” in the file extension is for template, I assume. Note that I have had to scratch around in some slightly opaque documentation to glean all this – not as polished as Rails, not yet at least! In index.thtml I just put together a little foreach loop to display my content:

";



 foreach ($data as $bookmark):



      echo "";



 endforeach;



 echo "";



      echo $html->link($bookmark['Bookmark']['name'],



 $bookmark['Bookmark']['url']);



      echo " ";



      echo $bookmark['Bookmark']['description'];



      echo "
";



 ?>

Finally, I need to add routing in order to tell Cake where my little app is and what to do with it. In app/config/routes.php I add a line:

 $Route->connect ('/bookmarks/*', array('controller'=>'bookmarks',



 'action'=>'display'));

Note that the action, “display”, corresponds to the method I defined in my controller.

Now all the pieces are in place at last. I browse to http://my.web.server/trax_and_cake_playground/cake/index.php/ bookmarks (I can clean up that ugly URL later) and lo! The boilerplate page is now displaying the preliminary bookmarks I put in my table! I am up and running. In Rails, at this point, I would have some nice auto-generated methods to add and delete bookmarks. Hopefully Cake’s auto-bake feature will come together soon.

Next, let’s take a look at Trax.

Makin’ Trax

Quick now, we know the drill. Go to the Trax site, download the archive, untar it. Like all of these prima-donna frameworks, Trax wants to be the top level of my web hierarchy, or rather it wants its public directory to be, and its trax directory to be parallel at the same level, out of sight of prying eyes. If my web server were publicly accessible, I’d put it up there, or find a way around that limitation. For now I’ll just unzip it in my playground (so to speak). Whoo!

Already now, if I go to http://my.web.server/trax_and_cake_playground/trax/public/, I have a page telling me “Congratulations, you’ve put PHP on Trax!” I feel good.

Trax does have a working auto-generate function, which is very nice, because I am getting tired of hand-editing all these files. If I wanted to do that, I’d be writing Perl or something. Still, there are a few more steps. I need to install the PEAR modules that Trax requires, which I do on the command-line:

 pear install Mail_Mime

I’ll use the same database and table I created back at the beginning, so long ago. I need to configure trax/config/database.ini to tell the application where they are:

    phptype = mysql



    database = bookmarker



    hostspec = localhost



    username = paul



    password = passw0rd



    persistent = 1

And then I need to edit public/.htaccess to tell the application where in my file system its own configuration is. The start of that file reads:

 # php include path for Trax ( Pear & Trax config dir )



 php_value include_path



 .:/usr/local/lib/php:/home//trax/config

I change it to suit my Debian setup:

 # php include path for Trax ( Pear & Trax config dir )



 php_value include_path .:/usr/share/php:/var/www/html/trax/trax/config

Now I am ready to — make Trax? Lay down Trax? There doesn’t seem to be a catchy verb for auto-generating models and controllers and views here. I, um, look both ways and prepare to cross the Trax.

The script that does the work is trax/script/generate.php. First, I need to edit the first line of that file to point to where my PHP executable actually is. Not /usr/local/bin/php, as the default has it. Mine is at /usr/bin/php. Also, the script seems to be happiest if I add my path information there as well, so toward the beginning of the file I enter the following:

 ini_set("include_path", ".:/usr/share/php:/var/www/html/trax/trax/config");

While I’m at it, I’ll add the same line to public/dispatch.php.

Then I need to make generate.php executable with a deft chmod a+x.

Now I’m ready to run the generate script, which I do from the command line. If your server doesn’t like PHP scripts to be run from the command line, you’ll have to open the script in a browser. I type this:

 ./trax/script/generate.php model Bookmark

and Trax replies:

 created /var/www/html/trax/trax/app/models/bookmark.php

Then I type:

 ./trax/script/generate.php controller bookmark display add

and Trax replies:

created /var/www/html/trax/trax/app/controllers/bookmark_controller.php



 created /var/www/html/trax/trax/app/helpers/bookmark_helper.php



 created /var/www/html/trax/trax/app/views/bookmark/display.phtml



 created /var/www/html/trax/trax/app/views/bookmark/add.phtml

“Display” and “add” are the initial views I want. Finally (I hope), I just need to add routing, so I can view my views in a browser. I add the following to trax/config/routes.php:

 $router->connect( "", array(":controller" => "bookmark") );

and voila! I can see a page! True, it’s blank, which means that I need to flesh out my actions and views. But it is a page! Let me just put in a couple of basics.

In trax/app/controllers/bookmark_controller.php, there’s an empty definition of the view() method. I’ll put this in:

 function view() {



        $bookmark = new Bookmark();



        $this->bookmarks = $bookmark->find_all();



    }

and in app/views/bookmark/display.phtml I put a PHP table similar to the one I used for Cake, to loop through the bookmarks table and display each one.

Whew. Two starter semi-apps built in as many hours. Not too bad.

Hardly the End of the Story

Would I do it again? Are either of the frameworks are 100 percent ready for prime time? Documentation is getting better every day, but neither framework has anything like the whopping user community of Rails.

I was able to iron out my own snags — several had to do with permissions, Apache directory overrides, and stuff like that – but if I couldn’t, it would be nice to have more resources to turn to. Conversely, PHP has a considerably larger support community on the Web than Ruby does, although that tide may slowly be turning.

Trax and Cake aren’t the end of the PHP dev framework story:there’s Symfony; Zend has a framework and a quick trawl of the web turns up things like PHP2Go, Qcodo and doubtless many more.

Rails is a pleasure to use, while Trax and Cake combine pleasure with a bit of effort. If you’re a PHP wiz, though, the fact that they’re based in a language you know should more than outweigh any difficulties brought on by the frameworks’ immaturity. It’s easier to debug a stuck app than to learn a whole new language, even one as perspicuous and enjoyable as Ruby.

So tune in next time when we cover Brails, the new web programming framework for the visually impaired! That’s a joke. Don’t sue.

Helpful Links

Visit CakePHP.org
Visit The Bakery
Visit the CakePHP forum at Nabble

Visit the Trax forum

Manage Transactions in MySQL

Webmonkey Staff — Tue, 16 Feb 2010 01:45:47 +0000

Overview

In this tell-all tutorial, Jay Greenspan, author of MySQL Weekend Crash Course and co-author of MySQL/PHP Database Applications, starts with a tour of the basics: He answers the age-old Q: “What’s the big deal with Transactions?”; investigates the four properties that a database must have to be considered transaction-capable; takes a closer look at locking mechanisms; and finishes up with a look at MyISAM tables, the lesser cousin of fully transaction-capable tables.

Once you have a taste of the limitations of MyISAM tables, you’ll be hungry for the real deal. In Lesson 2, Jay satiates that hunger with a thorough introduction to MySQL’s different transactional table types: BDB, Gemini, and InnoDB.

Get started: Lesson 1

Manage Transactions in MySQL – Lesson 1

Webmonkey Staff — Tue, 16 Feb 2010 01:45:47 +0000

(Note: This article is adapted from Jay’s new book, MySQL Weekend Crash Course look for it at a store near you. A book store! -Ed.)

MySQL has become the database of choice for many Web developers over the last few years and for good reason. It’s fast, free, easy to use, and has great community support.

But many experienced developers refused to touch MySQL because, they complained, the product didn’t implement features that were absolutely critical in an SQL server. MySQL’s most egregious omission, according to some, was its lack of transaction support. But thanks to recent developments in MySQL land, that’s no longer the case.

When it first hit the cyber-street, MySQL offered only one table type for data storage, the ISAM table now upgraded to the MyISAM type for all recent versions of MySQL. But MyISAM tables were limited. Very limited.

Then the folks from Sleepycat Software came into the picture. Sleepycat creates and sells a database storage engine which is used mostly with embedded devices. The storage engine comes with an API that allows developers to integrate Sleepycat’s data storage software into their products. And that’s just what the folks at MySQL did, they integrated the Berkeley DB (or BDB) table from Sleepycat. This was the first transactional table type included available to MySQL users.

Berkeley DB tables were followed shortly by two other transactional table types: InnoDB and Gemini. Gemini tables are adopted from another embedded storage mechanism; this one from NuSphere, a Progress Software property. InnoDB tables were designed specifically for MySQL.

But we’re getting ahead of ourselves here. Before we take a closer look at each of these different options, we need to start at square one: Why you want transaction support in the first place.

Getting in on the Transactions
Are You on ACID?
Lockdown!
Using MyISAM Tables

Getting in on the Transactions

If you’re relatively new to databases, or if MySQL has been the database on which you’ve cut your teeth, you may not even know what a transaction is. Put simply, a transaction is a series of SQL statements that are executed as a single unit; either all the statements are executed completely or none are executed at all.

Why are transactions so important? Consider a fairly typical Web application: a shopping cart. In a simple shopping cart application, you’re likely to have tables that look a little something like this (only less simplified, and with real info instead of the dummy data):

 products

 +------------+-------------------+---------------------------+---------------+

 | product_id | product_name      | product_description       | product_price |

 +------------+-------------------+---------------------------+---------------+

 |          1 | Old Pair of Socks | Plenty of holes           |          4.25 |

 |          2 | Old T-shirt       | Ketchup stains everywhere |          6.75 |

 +------------+-------------------+---------------------------+---------------+



 inventory

 +------------+----------+

 | product_id | in_stock |

 +------------+----------+

 |          1 |       25 |

 |          2 |       12 |

 +------------+----------+



 buyers

 +----------+-------+---------+------------+------------+

 | buyer_id | fname | lname   | phone      | fax        |

 +----------+-------+---------+------------+------------+

 |        1 | John  | Doe     | 7185551414 | NULL       |

 |        2 | Jane  | Johnson | 7185551414 | 2126667777 |

 +----------+-------+---------+------------+------------+



 orders

 +----------+----------+-------------+

 | order_id | buyer_id | order_price |

 +----------+----------+-------------+

 |        1 |        1 |       11.00 |

 +----------+----------+-------------+



 order_items

 +----------+------------+---------------+

 | order_id | product_id | product_price |

 +----------+------------+---------------+

 |        1 |          1 |          4.25 |

 |        1 |          2 |          6.75 |

 +----------+------------+---------------+

You can see how these tables fit together in this diagram:

When it comes time to process an order, you have to run several SQL statements within a script (written in PHP, Perl, Java, or whatever language you prefer). In the script, you want to take a look at what items the buyer wants, see if there’s adequate inventory to complete the order, and if there is adequate inventory, you want to complete the order. In pseudo-code, the script used to complete the order would look something like this:

 get buyer data and shopping cart data from web forms

 insert buyer data into buyer table

 start order by creating row in orders table

 get current order_id

 for each item desired

    check available inventory

    if inventory is available

       write item to order_items table

       decrement inventory

    endif

 end for loop

 get total for items for the order

 update orders table

I simplified this listing so that it’s easier to spot a potential problem. Consider what would happen if the power failed on the machine hosting the database just as it was in the middle of checking the inventory of the first item in the order. You’d restart your machine to find a row in the orders table without any child rows in the order_items table. It’s quite possible you’d be left with data that was, to a large extent, incomprehensible. You wouldn’t know what orders had been placed unless your customers sent you email, wondering just when you planned to send them the old pairs of socks they had requested not the best way to run a business.

If you had transaction support in place, however, you could treat this group of statements as a single unit. If for any reason the database failed to complete all of the statements in their entirety, the data would revert (or roll back) to the condition it was in prior to the first statement’s execution.

Of course there’s a bit more to transactions. A transaction-capable database must implement four specific properties. Wanna know what they are? Turn the page.

Are You on ACID?

Transaction-capable databases must implement four properties, collectively known by the mnemonic acronym ACID.

Atomicity: Transactions must be singular operations. Either all statements are executed as a single (atomic) unit or none are executed at all.
Consistency: In a transactional database, data will move from one consistent state to another. There will never be a point during a transaction when data is partly processed.
Isolation: The dealings of one transaction will not be visible to other clients until the transaction is completed successfully or rolled back. You can be sure that the data available to one transaction is accurate because it is isolated from changes other clients might make.
Durability: When a transaction completes successfully, the changes are permanent. Nothing, not even a disk crash or power failure, will erase the changes made by a successfully completed transaction.

SQL servers that allow transactions make use of several keywords: BEGIN WORK, COMMIT, and ROLLBACK. The phrase BEGIN WORK lets the SQL server know that the SQL statements that follow are part of a transaction. The transaction is not completed until either a COMMIT or ROLLBACK statement is executed. COMMIT writes the changes to the database. Once a transaction has been successfully COMMITed, only another successfully committed SQL statement can alter the data. No crashes or concurrently run SQL statements will effect the data. The ROLLBACK command tells the database that all of the statements within the transaction should be ignored and the database should revert to the point it was in prior to the start of the transactions. In the case of a crash, all transactions that were not expressly committed are automatically rolled back.

You can now revisit the pseudo-code first presented on the previous page. I’ve improved the listing by incorporating a transaction.



get buyer data and shopping cart data from web forms

insert buyer data into buyer table

BEGIN WORK

start order by creating row in orders table

get current order_id

for each item desired

   check available inventory

   if inventory is available

      write item to order_items table

      decrement inventory

   else

       set error variable to true

   endif

end for loop

if error variable is true

	ROLLBACK

else

	get total for items for the order

	update orders table

	COMMIT

Notice the ROLLBACK that I’ve added. This ensures that if inventory isn’t available to complete even a portion of a user’s request, the entire order is ignored. No row will be written to the orders or order_items table. Are you beginning to see how good transactions can be?

In a Web environment, you can expect multiple users (database clients or threads) to be accessing the script simultaneously. Therefore, the inventory of a given item will be changing continually. But when using transactions, you don’t have to worry that a user will complete an order for an item that is actually out of stock. That’s because of the I (for Isolation) in ACID. In a transactional environment, each transaction is isolated from the other, so one transaction cannot see the working of another transaction until the first one is complete.

The last sentence was a bit of a simplification, but it’s pretty close, and it’s good enough for now. In order to isolate one transaction from another, database systems must implement some sort of locking scheme. That is, the database needs a way for one client (or thread) to lock out (or isolate itself) from all other clients.

Locking is a key element of transactions. I’ll be talking about this subject extensively throughout this tutorial, starting on the next page.

Lockdown!

Locking makes some portion of data the property of a single client. That client says, in effect, “this data here is mine, and the rest of you can only do what I expressly permit.” Locks can have one of two effects: A lock may prevent other clients from altering data (with UPDATE or DELETE statements) or a lock may prevent all access to some data preventing UPDATEs, DELETEs, and even SELECTs.

To understand locking mechanisms in MySQL, you first need to recognize that MySQL is an unusual product. It isn’t really a single, unified piece of software. Rather, it uses technology from several different sources, and the way you implement your transactions in MySQL largely depends on the table type you use. Each table-type uses a different method of locking, and the differences in those locking mechanisms will effect how you write your code.

MyISAM tables are very fast for SELECTs, but they have some serious drawbacks when it comes to locking. These shortcomings are what prevented MySQL from implementing some key database features, including transactions.

Looking at the way MyISAM struggles with locking, you really begin to appreciate the power and value of actual transactions. So before we get into BDB, Gemini, and InnoDB, let’s first take a look at the limitations of MyISAM’s table-level locking.

Using MyISAM Tables

I already mentioned that MyISAM tables (usually MySQL’s default table type) don’t support transactions. This is largely because MyISAM tables offer only table-level locking, which means that locks can only be placed on entire tables. So if you want to prevent a single row in a table from being changed, you need to prevent all rows in the table from being changed.

Take our inventory table as an example. If one client is buying an item, you’ll want to check the inventory, and if the item is there in sufficient quantity, you’ll decrement the number available after the sale. To make sure the quantity doesn’t change between the time you check the availability and the time your change the inventory, you’ll want to put a lock on that row. But because MyISAM offers only table-level locking, you’ll have to cut off access to all other rows in the table.

MyISAM offers two types of table-level locks, a read lock and a write lock. When a read lock is initiated by a client, all other clients are prevented both from making changes to the table via INSERTs, DELETEs, or UPDATES.

To see how a read-level lock works on a MyISAM table, open up two copies of the MySQL command-line client. Then create a table and insert some data with the statements below

 create table inventory (

     product_id int not null  primary key,

     in_stock int not null,

     index index_on_in_stock(in_stock)

 )type=myisam;



 INSERT INTO inventory (product_id, in_stock) VALUES(1,25), (2,12);

Now in one client, place a read lock on the inventory table with the following command:

 LOCK TABLES inventory READ;

Now in the second copy of the client run a SELECT * FROM inventory. You’ll see that the command executes just fine. However, if you try to run an UPDATE, it’s a different story. Try the following command with the lock in place.

UPDATE inventory set in_stock=24 where product_id=1;

You’ll see that this copy of the client does not respond. It’s locked out and can’t execute the command. Only the client that placed the lock can change the table. Now go back to the first copy of the command-line client and release the lock with:

 UNLOCK TABLES;

Once the lock is released the second client will be free to run the UPATE command and change the row.

A write lock prevents other clients from even running SELECTs on the locked table. You can place a write lock with the following command:

 LOCK TABLES inventory WRITE;

Go back and run the previous experiment but this time issue a WRITE lock. You’ll see that the other client is prevented from doing anything, even reading from the locked table with a SELECT.

So now you know all the locks that are possible with MyISAM tables. As I mentioned earlier, the folks at MySQL AB (who run MySQL development) often argue that transactions really aren’t necessary. They say that by properly applying locks and writing clever SQL, you should be able to avoid the need for transactions. Below I’ve written some pseudo-code that adds transaction-like abilities to the shopping cart pseudo-code using MyISAM tables.

 INSERT query into buyers table.

 run last_insert_id() to get buyer_id

 run INSERT into orders table

 run last_insert_id() to get order_id

 get write lock on inventory table

 for each of the items in the order

     get quantity from the inventory table

     if quantity is > 0

         insert into order_items table

         update inventory table subtracting the ordered item

     elseif quantity = 0

         delete all items from order_items with current order_id

         delete item from orders table with current order_id

         update inventory table to replenish items previously subtracted

         set error variable to true

         break from for loop

 if error variable is not true

     update orders table with the current order_id, adding the order_total

 else

     output error

I might be able to clean this up a bit and remove a few of the SQL statements, but I think you see the way you’d need to go writing transaction-like code with MyISAM tables. While this code might add a degree of isolation (remember the I in ACID) by way of locks, there are other ACID properties missing here. Most notable is the lack of consistency: The data moves through several inconsistent states before the script is done. If the power goes out during one these inconsistent phases, you’ve got problems.

While ISAM locks are better than nothing, they are really no substitute for true ACID transactions. In Lesson 2 of this tutorial, I’ll discuss locking and transactions in BDB, InnoDB, and Gemini tables.

Come on back, ya’here.

Next: Lesson2

Manage Transactions in MySQL – Lesson 2

Webmonkey Staff — Tue, 16 Feb 2010 01:45:47 +0000

In Lesson 1 of this tutorial, you leaned how to approximate transaction-like behavior with MyISAM tables. In this lesson, you’ll see how you can achieve real transactions with each of MySQL’s three transactional table types: BDB, InnoDB, and Gemini. But before I get to specifics surrounding each table, I need to revisit the notion of locking.

Remember that MyISAM supports only table-level locks of two types: read locks and write locks. The transactional tables types offer more granular locking mechanisms: They can place locks on specific subsets of data. You’ll read more about each table’s locking mechanisms as I get to them in the following pages.

Using BDB Tables
Working with Gemini Tables
Using InnoDB Tables
The Right Choice for You

Working with MySQL’s Transactional Table Types

Transactional table types can apply two types of locks. These are known as shared locks and exclusive locks. A shared lock is similar to the MyISAM read lock. When a client places a shared lock on data, other clients cannot alter that data with UPDATE or DELETE statements, but they can read the locked data via SELECTs. When a client gains an exclusive lock, other clients can neither alter the locked data via UPDATEs and DELETEs, nor can they read the data with SELECTs.

Before we get to the actual code, there’s a MySQL setting I need to mention. MySQL contains an environment variable called autocommit. By default, autocommit is set to 1. When autocommit is set to 1, you cannot run transactions with any of the MySQL table types. In this mode, MySQL treats every SQL statement as its own transaction. To use groups of statements as transactions you need to turn autocommit mode off. So in all the code snippets that follow, you’ll notice that I start each with the line:

SET autocommit=0;

OK, now on to the rest of the code, starting with BDB tables.

Using BDB Tables

To check if you have BDB tables available, run the following query:

 mysql> show variables like '%have%';

 +---------------+-------+

 | Variable_name | Value |

 +---------------+-------+

 | have_bdb      | YES   |

 | have_innodb   | YES   |

 | have_isam     | YES   |

 | have_raid     | NO    |

 | have_symlink  | YES   |

 | have_openssl  | NO    |

 +---------------+-------+

 6 rows in set (0.00 sec)

If the first row in the listing says “no” you don’t have BDB installed. If you’ve got either InnoDB or Gemini, that’s fine — you can just use one of those, and I’ll talk about these tables soon (but please stick around, what you learn here will be useful there). If you decide to work with BDB tables, you can either install a MySQL-Max binary or recompile MySQL from source using the --with-bdb flag. BDB tables are available for most operating systems, including Windows, Linux, and Mac OS X.

To create BDB tables, you can include type=bdb at the end of the CREATE statement. For example:


 create table inventory(

     product_id int not null primary key,

     in_stock int not null

 )type=bdb;

Or, if you wish to move data from a MyISAM to a BDB table, you can run an ALTER statement, like so:



ALTER table table_name type=bdb;

Note that before you run an ALTER statement like this, you should backup the table. This type of ALTER statement actually dumps the table data, creates a new BDB table, and then loads the dumped data into the table. It’s an involved process that can fail, and if it does fail you want to be protected. Also be prepared to wait a while. If you’ve got large table, the dump-and-reload process can take a bit of time.

BDB tables offer page-level locking. What’s a page? It’s a group of rows. The exact number of rows in a page varies depending on the number of rows in your tables, the operating system in use, and other factors. During a transaction, MySQL will apply a lock to a page worth of data. All other rows are unaffected by locks and ready to process whatever queries you throw at them.

Page-level locking is a vast improvement over the table-level locking used by MyISAM tables. If only groups of rows — small subsets of whole tables — are locked by a single client, that means many clients can access different portions of the table simultaneously. In fact, it’s acceptable, even commonplace, to have many clients applying various locks to different portions of a table at the same time. The simultaneous access of data by different clients is usually referred to as concurrency. In database land (where the men have hairless backs and the women shapely legs) the greater the concurrency the better.

One of the great things about transactions is that for the most part you don’t need to worry about manually applying the locks. Just start a transaction, run your SQL statements as you normally would, and the clever MySQL engine will mostly take care of the locking for you. Neat-o!

Take a look at a simple transaction in a BDB table working with the following table:

mysql> select * from inventory;

 +------------+----------+

 | product_id | in_stock |

 +------------+----------+

 |          1 |       24 |

 |          2 |       47 |

 +------------+----------+

 2 rows in set (0.05 sec)

To see the effect of a transaction, open up two copies of the command-line client. In one, start a transaction and run a SELECT on the first row:

 mysql> set autocommit=0;

 mysql> begin work;

 mysql> select * from inventory where product_id=1;

Now in the second client, start a transaction and try to update the row 1.

 mysql> set autocommit=0;

 mysql> begin work;

 mysql> update inventory set in_stock=23 where product_id=1;

You’ll see that the second client, the one trying to do the UPDATE, doesn’t respond. That’s because the first client has a share lock on that row. To release the lock, commit the transaction in the first client:

mysql> commit;

You’ll notice that the second client is immediately freed to run the update operation.

Take a look at the following improvement to the pseudo-code I used in Lesson 1 of the tutorial. Note the use of the BEGIN WORK-COMMIT/ROLLBACK syntax.

 set autocommit=0

 INSERT query into buyers table.

 run last_insert_id() to get user_id

 BEGIN WORK

 run INSERT into orders table

 run last_insert_id() to get order_id

 for each of the items in the order

     get quantity from the inventory table

     if quantity is > 0

         insert into order_items table

         update inventory table subtracting the ordered item

     elseif quantity = 0

         set error variable

 if error variable is not set

     update orders table with the current order_id,

     adding the order_total



     COMMIT

 else

     ROLLBACK

     output error

Pretty nice, eh? Much cleaner than the MyISAM transaction-equivalent, I think. Keep in mind that you get all the ACID properties with this code if you’re using BDB tables. There is only one line of this code that doesn’t really work well and needs to be improved.

Notice that for each item in the order, I first check the quantity from the inventory table. That quantity will come from a SELECT statement, something like SELECT in_stock FROM inventory WHERE product_id=1. But in most cases, the inventory is going to change just after this SELECT is run. If there’s sufficient quantity in stock, this person is going to buy the item and therefore the quantity listed in the table will decrement by one when the transaction is complete. You’d hate to see other clients get data that will be inaccurate in a matter of milliseconds.

In a case like this, you’d be much better off putting an exclusive lock on a page containing the row in question until the transaction is complete. Remember, with an exclusive lock, all other clients are forbidden from even reading the locked rows. But unfortunately, BDB tables have no way to execute exclusive locks on pages of data. Only by instituting a table-wide write lock can you safeguard the data in the inventory table.

When revised with the table-wide write lock the pseudo-code looks like this:

 set autocommit=0

 INSERT query into buyers table.

 run last_insert_id() to get user_id

 BEGIN WORK

 run INSERT into orders table

 run last_insert_id() to get order_id

 LOCK TABLES inventory WRITE

 for each of the items in the order

     get quantity from the inventory table

     if quantity is > 0

         insert into order_items table

         update inventory table subtracting the ordered item

     elseif quantity = 0

         set error variable

 UNLOCK TABLES;

 if error variable is not set

     update orders table with the current order_id,

     adding the order_total



     COMMIT

 else

     ROLLBACK

The LOCK TABLES command here is a bit messy. But with the other two MySQL tables you won’t have to worry about this kind of lock.

Working with Gemini Tables

It’s difficult to talk about Gemini tables without touching on the controversy surrounding the maker of the Gemini table, NuSphere. The NuSphere corporation is currently in a legal dispute with MySQL AB, the folks who code the MySQL core engine. The issues include copyright infringement and abuse of the GNU GPL. You may want to read further about these issues before choosing the Gemini table type.

Gemini offers a more granular level of locking than BDB. With Gemini (and InnoDB, which I’ll discuss momentarily), you get the advantage of row-level locking. With Gemini, you can apply both share and exclusive locks to one row at a time. A row-level locking scheme will generally achieve higher concurrency than page-level locking, and much greater concurrency than table-level locking.

You can check if you have Gemini installed by running show variables like '%have%'. And if you don’t have Gemini, you can get an installer that includes MySQL, PHP, mod_perl and other goodies from the NuSphere website. Note that the most current release of the Gemini table seems to have some bugs running on Windows 98, so if any of the following experiments crash your NuSphere MySQL installation on Windows 98, don’t blame me.

To create Gemini tables include the type=gemini syntax at the end of the CREATE statement, and to transfer a current table to Gemini, run an ALTER TABLE query. The same caveats I mentioned for BDB apply here. For example:

	 create table inventory(

     product_id int not null primary key,

     in_stock int not null

 )type=gemini;





 ALTER table table_name type=gemini;

Unlike BDB, Gemini tables implement both shared and exclusive locks. If a thread runs an UPDATE or DELETE statement on a row, Gemini will place an exclusive lock on that row, preventing other threads from either reading or altering the locked row until the transaction is complete. If you think about it, this makes perfect sense. If a row is about to change, other threads should be protected from seeing potentially inaccurate data.

If a thread runs a SELECT statement on a row, a shared lock will be placed on that row so other threads can read the row with SELECTs but they won’t be able to alter the row with UPDATE or DELETE statements.

You can do some quick experiments with Gemini tables to see how this locking works. Assume you have the following Gemini table.

mysql> select * from inventory;

 +------------+----------+

 | product_id | in_stock |

 +------------+----------+

 |          1 |       24 |

 |          2 |       47 |

 +------------+----------+

 2 rows in set (0.05 sec)

Open two copies of the command-line client. In the first start a transaction, then run a SELECT on row 1.

 mysql> set autocommit=0;

 mysql> begin work;

 mysql> SELECT * FROM inventory WHERE product_id=1;

Now in the second client, start a transaction and try two statements, a SELECT and an UPDATE.

 mysql> set autocommit=0;

 mysql> begin work;

 mysql> SELECT * FROM inventory WHERE product_id=1;

 mysql> UPDATE inventory SET in_stock=23 WHERE product_id=1;

You’ll see that the SELECT executes immediately, but the UPDATE doesn’t execute. The UPDATE won’t execute until you COMMIT or ROLLBACK the transaction started in the first client, which releases the shared lock.

You can re-run this experiment trying an UPDATE in the first client. You’ll notice that the second client is prevented from reading the locked row with a SELECT. When a client runs an UPDATE statement, it gets an exclusive lock on that row.

When I discussed BDB tables, I pointed out that at times when inventory numbers are changing fast, you need a SELECT to obtain an exclusive lock so your users access the most up-to-date information. When you run a SELECT on the inventory table to find out the number of items currently in stock, you’re better off locking out all other threads from reading that row because that number is likely to change very quickly — an UPDATE will almost always follow that SELECT.

With Gemini tables, you can get a exclusive lock while running a SELECT by using the FOR UPDATE keywords. For example.

SELECT * FROM inventory WHERE product_id=1 FOR UPDATE

Using this added power, the shopping cart pseudo-code can be re-written like so:

 set autocommit=0

 INSERT query into buyers table.

 run last_insert_id() to get user_id

 BEGIN WORK

 run INSERT into orders table

 run last_insert_id() to get order_id

 for each of the items in the order

     get quantity from the inventory

     table with SELECT ... FOR UPDATE



     if quantity is > 0

         insert into order_items table

         update inventory table subtracting the ordered item

     elseif quantity = 0

         set error variable

 if error variable is not set

     update orders table with the current order_id,

     adding the order_total



     COMMIT

 else

     ROLLBACK

This is much cleaner, and much safer, than non-transactional code. And you can do something similar with InnoDB tables.

Using InnoDB Tables

InnoDB offers row-level locking and works on a wide variety of operating systems, including most UNIX variants, MacOS X, and Windows. This table type is now included by default in most MySQL installations, so if you get current binary from mysql.com, you’ll have InnoDB tables. You can check if you have InnoDB installed by running SHOW VARIABLES LIKE '%have%';.

You can create InnoDB tables by using type=innodb in the CREATE statement or by running an ALTER TABLE command. For example:

 create table inventory(

     product_id int not null primary key,

     in_stock int not null

 )type=innodb;





 ALTER table table_name type=innodb;

The InnoDB transaction model is not quite as straightforward as the Gemini model. If you don’t understand how InnoDB implements transactions, you could end up with some serious problems with your data.

InnoDB uses what’s called a multi-versioning concurrency model. Essentially, this means that each transaction is locked to a view of the data at a specific point in time. When a thread initiates a transaction, InnoDB gets a snapshot of the data based on the exact moment the transaction started. To understand how this works, take a look at the following example. You have a simple InnoDB table:

 mysql> select * from inventory;

 +------------+----------+

 | product_id | in_stock |

 +------------+----------+

 |          1 |       24 |

 |          2 |       47 |

 +------------+----------+

In one copy of the command line client, you start a transaction:

 mysql> set AUTOCOMMIT=0;

 mysql> begin work;

This thread now has a snapshot of the table seen above. If you were to then open another copy of the client and start another transaction, that second client would have the same snapshot of the same table. At this point, the first client runs an UPDATE statement and commits the transaction.

mysql> UPDATE inventory set in_stock=23 WHERE product_id=1

mysql> commit;

Even after this COMMIT, the second client will not see the results of the UPDATE. The second client receives a snapshot of the data at the moment its transaction began. Until the second client finishes its transaction (with a COMMIT or ROLLBACK), it will not be able to view the changes implemented by other threads that took place while the transaction was active.

Things get really sticky if the second client was also running an UPDATE on the same row. If the first and second clients start their transactions at the same time, they would get the same snapshots of data. The first client would run an update, WHERE order_id=3, but the second client would be unaware of the change. The second client could then run its own update, UPDATE WHERE order_id=3. After both of these clients commit their transactions, only one of the values can survive. As it turns out, the last one to COMMIT wins.

As you can see, this is potentially a dangerous situation. If you’re not careful, some of your threads could see out-of-date information.

But there are steps you can take to make sure the rows obtained by your transactions are up to date. It may not surprise you that staying current involves locking. With InnoDB, threads can apply the same types of locks that were available in Gemini.

In Gemini tables, when you run a SELECT within a transaction, a share lock is placed on the rows read by the SELECT. However, by default, InnoDB does not place any locks on rows during a SELECT. But you can let InnoDB know that you want to obtain a share lock by the adding the clause LOCK IN SHARE MODE at the end of a SELECT.

When a thread issues a SELECT with a LOCK IN SHARE MODE clause, InnoDB goes beyond the snapshot of data taken at the start of the transaction and looks for the latest committed data. If another thread is operating on that data (via an UPDATE, DELETE, or another LOCK IN SHARE MODE), the first thread won’t be able to obtain the share lock and will wait until the other thread finishes its transaction and releases its locks. By attempting to obtain the share lock and receiving the latest committed data, the LOCK IN SHARE MODE clause ensures that the data read by a SELECT will be accurate and up to date.

If you’d like to see how this works, try the following, using the same table shown above. In one client, start a transaction and then run an UPDATE statement:

 mysql> set autocommit=0;

 mysql> begin work;

 mysql> update inventory set in_stock=22 where product_id=1;

Now in the second client, run two different SELECTs. The first you will run without the LOCK IN SHARE MODE clause. Try SELECT * FROM inventory. This will execute immediately, but the data is potentially inaccurate because there’s no telling if the first client will COMMIT or ROLLBACK the transaction. If you alter that SELECT statement to include the LOCK IN SHARE MODE clause, the client will not respond until the first client issues a COMMIT or ROLLBACK.

Threads can obtain exclusive locks in SELECT statements by using the same FOR UPDATE clause that is used with Gemini.

The pseudo-code shown for Gemini will also work for InnoDB tables. But be very careful when writing your SELECT statements for InnoDB. You want to make sure you’re getting the latest, most accurate data.

The Right Choice for You

At this point, InnoDB is the most logical choice for most. It offers row-level locking, is available on most every operating system, and is getting a lot of support from MySQL AB. InnoDB is also the first table type to enforce foreign key constraints. But if you’re working with an installation that has support of one of the other transactional tables, both of those will work just fine.

When you decide on the table type you wish to work with, you’ll have more technical matters to consider. You should read everything the MySQL manual has to say on the table type you end up using.

Pay especially careful attention to your logs. The transactional tables use logs to help recover from crashes. If a hard-disk failure causes a crash on your system, you could be in trouble if your logs are on the same hard disk as your MySQL data.

If you’re unhappy with the types of locks initiated by specific statements, you can adjust the locking behavior of Gemini and BDB table with the SET TRANSACTION command. I won’t cover it in detail here because the defaults discussed in this tutorial are usually preferable. Note that InnoDB tables are unaffected by SET TRANSACTION command.

Transactions are a great improvement to the MySQL database. If you’re not using them now, consider using them in the future. Because when that backhoe works it way through your power cord, you’ll be awfully glad you did.