The “horrible thing” in developer Erik Rose’s talk from this year’s PyCon is the Mediawiki syntax, but that’s just a jumping off point for one of the best overviews of data parsing that I’ve run across. If you’ve got a project that involves parsing, or are, like me, considering one, this talk is a must-watch.

This is PyCon, so much of the talk focuses on parsing in Python, but there’s plenty of broader, dare I say, “parsing philosophy” that make it well worth a watch even if you don’t end up using the specific Python parsing libraries Rose mentions.

The scripting language Python is named after the Monty Python British Comedy series, although the language takes after its reptilian namesake as well — it’s simple by design, yet flexible and powerful. It can’t exactly swallow large rodents whole, but it can be used for a wide variety of applications. Whether you need to automate some part of you desktop workflow, create a website or build a full-fledged desktop application, Python is a strong candidate for the job.

What Is Python?

Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools, comes with extensive standard libraries, and can be learned in a few days. Many Python programmers report substantial productivity gains and feel the language encourages the development of higher quality, more maintainable code. [source: python.org]

Python was created by Guido van Rossum, who now works for Google. As a result, the company uses Python extensively in its various web services, most recently App Engine. Even though we joked around about Python exhibiting some likeness to its slithering namesake, the language takes its name from Monty Python’s Flying Circus, which is why many Python tutorials feature shameless Monty Python puns. This one is no exception.

Advantages

Simple Clean Syntax
Python has an extremely simple syntax. It has comparatively few keywords and well written code in Python tends to remain extremely readable years after it was written. Much has been made about using white space to denote code blocks in Python, however users of the language will note this adds to readability and standardizes code formatting across the Python community.

Object Oriented
Python has advantages over other object-oriented languages. In Python everything is object. In some object oriented languages like Java, almost everything is object, but there almost always seems to be an exception. This creates some rather clunky syntactical maneuvering. There is no such problem in Python. In Python, the so-called primitive types are objects, functions are objects and code blocks are objects — this allows for a more natural programming style.

No One Style Enforced
What if object orientation isn’t really your thing? Maybe you only write 50-line text processing scripts and have no need to write complex objects. Well, you will find Python perfectly suited for your needs. While Python has advanced object oriented design elements, it doesn’t force you to use them. You can write large programs in the procedural style in Python and never be bothered to learn the object oriented way of doing things. Python also supports some functional programming constructs like anonymous one-line functions (using the Lambda keyword) and cool features like list comprehension.

Standard Library
Python’s biggest advantage is perhaps its large all inclusive standard library. Unlike many other open-source languages, Python has almost everything you need to write almost any type of application included. So whether you are writing a database-backed web application or a cross platform networked console application, be sure that the standard library has you covered. If for some reason you need more than is provided by the standard library, there are many, many third party libraries out there. They cover everything from industry standard encryption libraries to 3D graphics frameworks.

Uses

Python is a compiled application that can be installed in a variety of environments — your desktop, a web server or embedded devices like your phone. Python can run on all the major platforms and comes pre-installed on Mac OS X and most Linux distros.

There are two main ways to use Python — either by starting it directly from the command line or by calling an external script. Obviously, the latter method is much more flexible unless you’re just doing a quick one-time program.

If you’re running Python on a web server, you can serve your scripts from a cgi-bin folder just as you would with a Perl script. There are also faster, more robust options like mod_python or mod_wsgi, two Apache modules that call on the Python interpreter to process your pages.

Before we dig in, you should know about a site called Python.org. Python is an open-source language, and Python.org is its control center, with extensive reference material, additional tutorials and a complete library of all the elements available in Python.

Installing Python

Python code can be written as scripts and saved in text files with the .py extension. There’s also a shell interpreter that makes it very easy to get started just by typing python into your shell prompt. For now, that’s what we’ll be using to show some of the basic language principles.

Go ahead and fire up a terminal window and type python. If you have Python installed, you’ll get a message like this:

Python 2.4.4 (#1, Oct 18 2006, 10:34:39)

[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>>

Tip: The three angle brackets are the standard Python command prompt which lets you know you’re now using the Python interpreter. Any time you see code snippets written with the >>> you know the author is referring to the Python shell.

If typing python in your shell didn’t load the interpreter, you’ll need to install Python. You can download the source from the Python website or use one of the many package installers available for most platforms.

As of this writing, the latest version of Python is 2.5.2. Mac OS X and most Linux distros ship with Python 2.5. Windows users may need to install the latest version.

Although Python 2.5 offers some nice new features over previous versions, for backwards compatibility reasons we’ll stick to concepts and tools that will work with Python 2.3 and greater.

Python Differences

For the most part, Python behaves much like PHP, Perl, Ruby and other languages you may be familiar with. However, there are some important and noteworthy differences.

Perhaps the most obvious (and Python aficionados would argue, important) is that line breaks and indentions in your code have meaning in Python. Whereas PHP and others use a semicolon or other mark to designate the end of a line, Python sees all new lines as, well, new lines.

Also where PHP and others use curly brackets to enclose code blocks, Python merely wants the code indented.

Python forces you to properly indent code blocks and eschews end-of-line punctuation like PHP’s semicolons in favor of simple line breaks.

This has some import consequences. First and foremost, it makes Python code much easier to read. The structure of a Python script is a snap to figure out at first glance. Even if you have no idea what the code is doing, you can tell how it does it just by glancing at it.

Python forces neat, well structured code, but it also forces you pay more attention to how you write your code. Consider the following two code snippets, which do very different things:

def myfunction():

    if x == y:

        return True

def myfunction():

    if x == y:

    return True

In the first code block our return statement is indented, and therefore within the if statement. In the second code block we didn’t indent the return statement so that function always returns true, regardless of our if test. Technically, that second function would generate an error because Python expects an indented block after the colon.

Spaces versus Tabs

As the joke goes, the most popular way to write Python code is to indent with spaces. The second most popular way to write Python is with tabs. Most good text editors have an “entab/detab” function which can convert tabs to spaces and vice versa.

The important thing is to be consistent. Don’t mix tab and space indenting in the same script! Doing so will cause Python to throw an error and your code won’t execute.

Getting Started

Assuming you’ve got Python installed, fire up a shell window and type python to start the interpreter. Here’s a simple script. Just type print 'hello world' and hit return.

>>> print 'hello world'

hello world

>>>

Any time you want feedback from Python, use the print statement. As with any language, Python has built-in tools for doing common things, like in this case, printing something out.

Let’s create some variable assignments and play around with some of Python’s built-in types. We’ll start by creating a string and playing around with it:

>>> x = 'spam'

>>> x[0]

's'

>>> x[:1]

's'



>>> x[1:]

'pam'

The first line just assigns x the value of “spam.” Python is dynamically typed language, that is, there’s no need to tell Python that x is going to be a string, it will figure that out at run time.

It’s also worth noting that Python is a strongly typed language, which means that “types” are always enforced. If you try to treat x as a number after you’ve already assigned it a string value, Python will throw an error. In order to use x as a number, you’d have the explicitly recast x as a number first.

The next line x[0] shows how Python’s treats strings — much like a list (array) with each letter being like an element of the list. The x[:1] is an example of Python’s slicing methods. The basic syntax is variable[start:end]. Slice indices always start with 0 and by default, omitting first index defaults that value to zero, while omitting second index defaults to the size of the string/list.

Here are few more operators:

>>> x = 'spam'



>>> len(x)

4

>>> y = x

>>> x = 'knights'

>>> y

'spam'

>>>

Note that last sequence, we set a new variable y to the value of x and then change the value of x, but y stays the same. In other words, y doesn’t become a reference to x, it’s a new variable with the value of x. Just because x changes doesn’t mean y will as well.

More Strings, Formatting and Comments

We already know that line endings and indentions matter in Python, but what about within Python strings? For instance:

>>> long_string = "Python is the best language ever, but I willn

... keep that to myself and not start flame wars on Slashdot.n

... Just becausen

...     I think it's the best doesn't mean everyone needs to"

>>> long_string

"Python is the best language ever, but I willnkeep that to myself and not start flame wars on Slashdot.nJust becausentI think it's the best doesn't mean everyone needs to"

>>> print long_string

Python is the best language ever, but I will

keep that to myself and not start flame wars on Slashdot.

Just because

        I think it's the best doesn't mean everyone needs to

>>>

So what’s going on here? How did we write multiple lines if line breaks mean the end of a line? The trick is the which tells Python to ignore the end of the line. Also note the difference between calling our variable directly and printing it using the print statement.

Escaping line breaks with a backslash also works within normal Python code blocks as well, not just within strings. In some cases it can help make your code more readable.

The other Python concept that might look a bit funny to those coming from other languages is the string formatting tools. In other languages the common way to add data to a string is to just concatenate the string like this:

"the beginning of a sentence" + variable_data + "the end of a sentence"

While this will work in Python as well, there is a far more elegant way to write it using the %s operator.

>>> b = 'beginning of a sentence'

>>> e = 'end of a sentence'

>>> v = 'variable data'

>>> '%s, %s, %s' %(b, v, e)

'beginning of a sentence, variable data, end of a sentence'

The other nice thing about using %s is that it will force the value to a string, whereas straight concatenation won’t.

Note: There is a similar %f for inserting numbers into a string.

One last thing about strings (aside from them being super-absorbant and perfect for doing away with floods and tidal waves).

To create comments in Python you have several options. Like Perl, the hash mark can be used for inline comments, like so:

x = 'spam' #initial value

The other way to comment your code is with doc-strings:

def superfunction(params):

   """ The super function can do things

       you've only dreamed of"""

   print 'Spam!'

A doc-string is an unassigned string at the beginning of a function definition. In this example, we’ve used a tripple-quoted string which can span several lines but a normal, single-quoted string is also OK.

A doc-string isn’t executed but is stored with the function to describe its purpose. Doc-strings can be accessed in your code – a feature which has been used to produce self-documenting and self-testing programs.

Lists, Dictionaries and Tuples

Arrays are one of the most useful constructs in a language. They allow you to store and manipulate compound data. Python has three distinct objects for handling compound data types: lists, dictionaries and tuples.

Lists

Perhaps the most useful is the list, which is pretty straightforward; it’s a list of data:

>> l = ['spam', 'ham', 314, 23]

>>> l

['spam', 'ham', 314, 23]

>>> l[0]

'spam'

As you can see, we construct a list using square brackets and assign it to the variable l. From there, we can get the whole list by calling it directly or access individual list members using a zero-based index, just like we did earlier with the strings.

Also note that lists can mix together any data-type you want. Here, we used both numbers and strings, but you could add anything you want — even other lists.

Lists can do some other neat tricks. For instance, let’s try replacing some items:

>>> l[0] = 'monkey'

>>> l

['monkey', 'ham', 314, 23]



>>> # slightly more complex replace method:

>>> l[0:2] = ['banana', 18]

>>> l

['banana', 18, 314, 23]

>>> # we can even insert the list itself:

>>> l[:0] = l

>>> l

['banana', 18, 314, 23, 'banana', 18, 314, 23]

As you can see, lists are mutable. You can do pretty much anything you want with them. As you start getting more comfortable with Python, you’ll find yourself using lists all the time. Be sure to check out the Python docs for a complete list of list methods.

Dictionaries

Another useful compound data-type is the dictionary. A Python dictionary is created using curly brackets, like so:

>>> d = {"monkey":"banana", "spam":"eggs"}

>>> d

{'monkey': 'banana', 'spam': 'eggs'}

The name:value pairs are the dictionaries keys and values. To access dictionary data you just call the key using the square bracket syntax like lists, but instead of a number you’ll use a key:

>>> d['monkey']

'banana'

>>> #try to call something that doesn't exist and you'll get a KeyError.



>>> #this means that you can't get keys from values:

>>> d['banana']

Traceback (most recent call last):

  File "<stdin>", line 1, in ?

KeyError: 'monkey'

Like lists, dictionaries are mutable, which means you can add key:value pairs whenever you want. Just keep in mind two things: you can’t have duplicate keys, as the second will overwrite the first, and key names are case sensitive.

>>> #Start with an empty dictionary

>>> d = {}

>>> # add two key:value pairs

>>> d['monkey'] = 'banana'



>>> d['spam'] = 'eggs'

>>> d

{'monkey': 'banana', 'spam': 'eggs'}

>>> #try to add another monkey key

>>> d['monkey'] = 'wrench'

>>> # oops, we overwrote the original value:

>>> d

{'monkey': 'wrench', 'spam': 'eggs'}

As with lists, dictionaries can store any type of data you want. Even dictionary keys can be a variety of data-types, so long as the data-type is immutable (i.e. strings, integers, tuples, et cetera). And you can mix and match data-types within the same dictionary.

For more details on what sort of manipulations you can perform with dictionaries, check out the Python documentation page.

Tuples

What the heck is a tuple? It’s got a funny name, but it’s just an immutable list. Once you stick data in a tuple you can’t change it. Unlike lists, tuples have no methods.

Tuples are created using parentheses, and data in a tuple can be accessed just as we did with a list. Often functions that operate on lists work on tuples, too.

>>> t = ('spam', 'ham', 314, 23)

>>> t

('spam', 'ham', 314, 23)

>>> t[0]

'spam'

>>> t[0] = 'eggs'

Traceback (most recent call last):

  File "<stdin>", line 1, in ?

TypeError: object does not support item assignment



>>> len(t)

4

>>> max(t)

'spam'

In many contexts, you don’t need the brackets to define a list. The comma alone is sufficient.

>>> t = 'spam', 'ham', 314, 23

>>> t

('spam', 'ham', 314, 23)

So you might be wondering, what’s the point of a tuple if it behaves like a crippled list? The answer is that tuples are faster than lists and Python uses them for holding some static sequences as we’ll see later. If you have a sequence of data that you know isn’t going to change, put it in a tuple.

Tuples are also handy for data that should not need to be changed, like in a private class method. Using a tuple instead of a list prevents outside tampering. Tuples can be keys in a dictionary but lists cannot.

Assigning Multiple Values at Once

One of the cool tricks in Python is ability to assign multiple variables values at the same time. For instance:

>>> m,s = 'monkey','spam'

>>> m

'monkey'

>>> s

'spam'

But this gets much more powerful with more complex data. For instance we can also do this:

>>> a = 'b','c','d'

>>> x, y, z = a

>>> x

'b'

This is called ‘list unpacking’. Both sides of the assignment must have the same number of elements.

Another variation on this theme is the built-in range function. Consider this:

>>> range(7)

[0, 1, 2, 3, 4, 5, 6]

>>> Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday = range(7)

>>> Sunday

0

The Python for Loop

Of all the things that will have you scratching your head while learning Python, the for loop is probably the most obvious difference from other languages. Rather than iterating over a progression of numbers, or allowing you to define both the iteration step and halting condition (as in C), Python’s for statement iterates over the items of any sequence (a list or a string), in the order that they appear in the sequence.

The best way to understand how it works is to look at an example. In this case, we’ll create a list and then loop through the values.

>>> a = 'spam', 'ham', 314, 23

>>> for b in a:

...     print b

...

spam

ham

314

23

Note that the variable b is totally arbitrary. You can name the iterator whatever you would like, preferably picking whatever makes the most sense. For instance, if your list holds a bunch of URLs, you might write for url in urls:.

We’d use the same technique to loop through a tuple, a string and other data-types. But what about a dictionary? That’s a little bit different, but we can borrow some of the multiple values techniques we mentioned above to help out.

>>> mydict = {'guido':'python','larry':'perl','rasmus':'php'}

>>> for key in mydict:

...     print key, 'created', mydict[key]

...

larry created perl

rasmus created php

guido created python

It’s worth noting that while Python’s for loops are just as capable as those in other language, the chances are that you won’t use them as much. One reason is that Python offers something called a list comprehension which does something similar but usually more elegantly .

For instance suppose you have sequence of numbers and you want to perform a multiplication function on them. You might be tempted to do something like this:

>>> new_list = []

>>> for num in 0, 1, 1, 2, 3, 5:

...     new_list += [num * 2]

>>> new_list

[0, 2, 2, 4, 6, 10]

But in Python, it’s much easier to just use a list comprehension. Here’s the same function in one line:

>>> [num * 2 for num in 0, 1, 1, 2, 3, 5]

[0, 2, 2, 4, 6, 10]

>>> # or if you want to store the results:

>>> new_li = [num * 2 for num in 0, 1, 1, 2, 3, 5]

>>> new_li

[0, 2, 2, 4, 6, 10]

It may take a while to wrap your head around, but list comprehensions are very very powerful and well worth spending some time with so you understand not just how they work, but when and where to use them.

Have a look at the official Python tutorial for more information.

Suggested readings

• Learning Python, 3rd Edition, by Mark Lutz (O’Reilly)

• Also worth reading through is Mark Pilgrim’s seminal Dive into Python which covers some of the same ground we went over here, but also hits on some other areas of interest like parsing XML with Python’s powerful DOM tools and attempting to demystify Unicode text handling in Python.

Conclusion

So now you know how to say “nee!” and perform some basic operations using Python. If you’d like some more practice head over to the Cheese Shop (we did warn you that the Monty Python puns were thick in the Python community), a site dedicated to Python scripts, and try downloading some things that look useful.

Next up we’ll take a look at how to write a script you can run from the command line and how to open and read data from a web service. In the mean time we’re off to return this parrot which seems almost like it’s, um, dead.

The microblogging site with the funny name is one of the hottest web services around.

Twitter is one of those websites with very little room for functional nuance. Its limit of 140 characters per post forces users to be succinct, something that makes many people feel over-constrained and leads them to view the service as too simple to actually be useful. Others see unbridled freedom inside such a unique limitation and embrace it like a poetic device. The lesson: You either get Twitter or you don’t.

Regardless of how you feel about it, if you’re looking to try out an API for the first time, Twitter is a great place to start.

Introduction

Like the service itself, the Twitter API is simple and easy to use. The only thing to keep in mind is that Twitter limits you to 70 requests per 60 sixty minute interval, so remember to cache or otherwise store your results. Otherwise, you may find yourself banned.

If you end up building something that needs to make requests more frequently, you can always e-mail Twitter and ask for permission.

Getting Started

Twitter’s API is REST-based and will return results as either XML or JSON, as well as both RSS and ATOM feed formats.

For a very simple look at a typical data response fire up a terminal window and type:

curl http://twitter.com/statuses/public_timeline.rss

That line will give you the latest 20 tweets from the public timeline in the form of an RSS feed. To get the same results in JSON just change the extension to “.json”.

Public timelines can be accessed by any client, but all other Twitter methods require authentication.

Authenticating with Twitter

To authenticate yourself to Twitter you need to send a username and password. The basic format is:

curl -u email:password http://twitter.com/statuses/friends_timeline.xml

Most client libraries offer easy ways to pass in a username/password pair, making it very simple to authenticate with Twitter.

Client Libraries

There are already some pretty good libraries out there for accessing Twitter, and there’s no need to re-invent the wheel. Whatever you’re looking for, Twitter’s got it: ActionScript, Python, Ruby, PHP and many more.

Using Python with Twitter

For the sake of example, we’ll use the Python library. Download and install the library and fire up a terminal window.

Let’s grab your last twenty tweets. Fire up Python and type the following line, replacing “youusername” with your username:

>>> import twitter

>>> client = twitter.Api()

>>> latest_posts = client.GetUserTimeline("yourusername")

>>> print [s.text for s in latest_posts]

The last line prints out just the text of your posts. To get at things like date pasted and other useful methods, have a read through the Python library documentation.

To go the other direction (posting something to Twitter) we’ll need to authenticate. Recreate our Twitter client, but this time pass in your username and password:

>>> client = twitter.Api(username='yourusername', password='yourpassword')

>>> update = client.PostUpdate('The Twitter API is easy')

Head over to your Twitter page and you should see the update.

Mashups, Apps and More

To get some idea of what you can do with the Twitter API, visit the fan wiki and check out some of the applications and mashups created by Twitterheads.

Hashtags

One area of particular usefulness is the hashtags concept. Hashtags involve inserting a # sign to denote keywords or other data like location. Although not everyone is a fan of hashtags (they tend to make your tweets less readable) parsing Twitter streams for hashtags can yield a wealth of useful data.

Some of the client libraries include functions to parse hashtags, but in many case you may have to write your own functions.

Full API documentation

You can check out the development talk for the full api documentation.

Django is a web framework designed to help you build complex web applications simply and quickly. It’s written in the Python programming language.

Django takes it name from the early jazz guitarist Django Reinhardt, a gypsy savant who managed to play dazzling and electrifying runs on his instrument even though two of the fingers on his left hand were paralyzed in an accident when he was young.

Thus, it’s a fitting name for the framework: Django can do some very complex things with less code and a simpler execution than you’d expect. It doesn’t take a heavy hand to build with Django. The framework does the repetitive work for you, allowing you to get a working website up quickly and easily.

Django’s DRY pledge

Django was designed from the ground up to handle two common web developer challenges: intensive deadlines and strict adherence to the Don’t Repeat Yourself (DRY) principle. DRY sounds exactly like what it is: Why write the same code over and over again?

The result is a fast framework, nimble and capable of generating full site mockups in a very short time. Django’s slogan captures its essence quite nicely: “The web framework for perfectionists with deadlines.”

But quick doesn’t mean sloppy. Django comes with a slick built-in section for administering sites. The admin section supports a variety of caching options (including Memcached) and a host of other stable, scalable tools.

Perhaps the best part about Django is its outstanding documentation. Unlike many open source projects, Django has very thorough and readable docs available online. If you have any problems with our tutorials, head over to the Django site for additional reading. Also consider joining the django-users Google Group which is full of helpful folks who can point you in the right direction if you run into trouble.

Background

Before we dive in, it’s worth pausing for a moment to understand where Django comes from, which has influenced both what it is and what it is not.

Django was developed over a two year period by programmers working for an online news operation, the Lawrence Journal-World Online in Lawrence, Kansas. Eventually, the team realized it had a real framework on its hands and released the code to the public under an open-source BSD license.

Once it became a community project, the development took off and Django began to pop up all over the web. For a complete list of Django sites, check out Django Sites. Notable examples include Pownce, The Washington Post and Everyblock.

Perhaps the most common misconception about Django is that it’s a content management system. It’s not. It’s a web framework. It is a tool in which to build a CMS, like Drupal or other systems, but not one in itself.

Django is designed to work in a slightly modified Model-View-Controller (MVC) pattern. MVC is a software engineering architecture concept describing how a program can change the visual aspect and the business aspect without affecting one another. The original Django developers refer to it as a model, template and view (MTV) framework.

However, in Django’s vision of MVC development, the “view” refers to the data presented to user. Mind you, not how data looks, but which data. In other words, to quote the Django documentation, “the view describes which data you see, not how you see it.”

It leaves the “how” to an additional element found in Django: the template.

Why isn’t Django a 1.0 release?

It is! Django was released as 1.0 on September 3rd, 2008. Django 1.1 was released on July 29, 2009.

Structural overview

Django was designed to make web development fast and easy — dare we say, even fun. And by fast, they mean fast. Once you’re comfortable with Django, it’s not hard to go from a simple wireframe to a working demo site in an hour or so.

For development purposes, Django is entirely self-contained. It includes a command line interface, a web server and everything you need to get up and running without installing anything other than Django.

That said, the web server included with Django is not intended to be used in a production environment. The preferred method is to run Django through mod_python or mod_wsgi. Don’t worry, we’ll get to mod_python and mod_wsgi in our more advanced tutorial. First, let’s look at how you might go about building a Django application.

As it turns out, building a Django app actually takes much less time than explaining how to build it.

Before we start building sites, it’s worth asking: What do you mean by building a Django app? Django’s official documentation and many of its built-in tools are set up to create projects, or containers for your whole site. Within projects you have many apps, or sections of your site.

Apps don’t need to be in projects, and in many cases it’s better not to put them there.

For the sake of example, here’s a very high-level overview of how a Django app is built:

First, create a project using the built in command line tool (python django-admin.py startproject projectname). This will generate a new folder containing the project’s base settings file which you can use to specify database connections, template locations and more. Also in the folder is urls.py, which defines your base URLs and manage.py, which contains a standard set of command line tools.

Next, create an app using the built in command line tool python manage.py startapp appname.

Once the app folder is in place, look inside. You’ll see three files: models.py, urls.py and views.py. These are the files you’ll use to actually build the app:

• models.py to design your model
• urls.py to write your URLs
• views.py to create your views

Django includes its own template language, but as with many elements of Django, using it entirely optional. You can drop in another template language if you like, but you might want to give Django’s a try first. It’s simple and fast, and it’s already there.

The other thing to keep in mind is that Django is written in Python and requires Python 2.3 or higher. Mac OS X and most Linux distros ship with Python 2.5, so you shouldn’t have any problems. Windows users, however, may need to install Python separately.

So how does Django work?

The simplest way to look at Django is to break it down into its component parts. First off, there’s a models.py file which defines your data model by extrapolating your single lines of code into full database tables and adding a pre-built (totally optional) administration section to manage content.

The next element is the urls.py file which uses regular expressions to capture URL patterns for processing.

The actual processing happens in your views which, if you haven’t seen the pattern yet, live in views.py. This is really the meat of Django, since views are where you grab the data you’re presenting to the visitor.

Here’s what happens when a visitor lands on your Django page:

1. First, Django consults the various URL patterns you’ve created and uses the information to retrieve a view.

2. The view then processes the request, querying your database if necessary.

3. The view passes the requested information on to your template.

4. The template then renders the data in a layout you’ve created and displays the page.

Dive in

Now you have at least some idea of how Django works, you’re ready to dive in and get your hands dirty. It’s really the best way to learn. For the sake of example, we’ll be building a blog-type site. It makes a nice intro to Django and takes advantage of some of Django’s handy shortcuts and other features.

We aren’t going to just build a blog, we’ll also walk through adding a contact form, static pages (like an “about” page) and even integrate it with del.icio.us web services to display all your del.icio.us bookmarks on your new Django blog.

So head on over to part two of our Django series, Install Django and Build Your First App

Last time around, we installed Django and started building a blog application. We got Django’s built-in admin system up and running and explored some third-party libraries like the django-tagging project.

So far we have some cool administrative tools, but no website for the rest of the world to see. This time around, we’ll work on displaying our content to the world by building the URL patterns and constructing some “views” — a term with a very specific meaning within the Django framework.

Everything we’re going to do will make more sense if you understand how Django processes your visitor’s request. We went over some of this in our introduction, but here’s a quick refresher course.

How content is presented

The Django flow goes something like this:

1) Visitor’s browser asks for a URL.
2) Django matches the request against its urls.py files.
3) If a match is found, Django moves on to the view that’s associated with the URL. Views are generally found inside each app in the views.py file.
4) The view generally handles all the database manipulation. It grabs data and passes it on.
5) A template (specified in the view) then displays that data.

Designer and coder Jeff Croft has put together a visual representation of this flow that makes it even easier to understand:

See the full size original (along with explanatory text) at Flickr.

With that in mind, let’s start building our public site by creating some URL patterns.

Working with urls.py

Remember the urls.py file where we set up our admin URLs in the last lesson? Open that file in your favorite text editor.

Now, we could define all our URLs in this file. But then what happens if we want to reuse this blog in an entirely separate project? We’d get a mess.

A far better approach is to define all your app-specific URLs in a file that lives inside the app itself. In this case, we’re going to use a file inside the blog app, which will also be named urls.py.

However, before we start with that file, we need to tell Django about it. So add this line to the project-wide urls.py file, just below the line that defines the admin URLs, like so:

from django.conf.urls.defaults import *

from django.contrib import admin



urlpatterns = patterns('',

    (r'^admin/(.*)', admin.site.root),

	(r'^blog/', include('djangoblog.blog.urls')),

)

OK, now head into the blog app folder and create a new urls.py file, which we’ll use to hold all the URLs related to our blog application.

Thoughts on URLs

One of the nice things about Django is that it forces you to think about your URL designs, something many people don’t spend much time considering. If, perchance, you’ve never spent too much time thinking about URLs, now is good time to read the W3C guide on the subject.

As the W3C points out, good URLs never change. In fact, even bad URLs never change — people change them. But when you change your URLs, you break everyone’s bookmarks and inbound links. So, spend a bit of time designing a good URL scheme from the beginning and you shouldn’t need to change things down the road.

I would actually add one item to the W3Cs guidelines: good URLs are hackable. What do I mean by hackable? Let’s say our blog has URLs like:

http://mysite.com/blog/2008/jul/08/post-slug/

That URL would display the blog post with the slug named “post-slug” which was published on July 8, 2008.

Note: “Slug” is an old newspaper term. An article about fires in Nevada would probably be slugged “nv-fires” for easy identification. In this context, the slug refers to the last bit of the URL and can contain letters and dashes rather than spaces.

Ideally, if the user heads up to their browser’s location bar and chops off the post-slug/ bit, they would see all the blog entries from July 8, 2008. If they were to chop off 08/, they would see all the posts from July 2008, and so on.

In other words, the URL is hackable. Now, most people probably won’t do that. But in addition to making your site easier to navigate for the hands-on types, this rule also enforces an easy-to-use structure around which to build your site. In this case, the date-based structure was probably already obvious. But what about tags?

http://mysite.com/blog/tags/tag-slug/

This URL structure uses the same idea, but one-ups it. Not only can you hack the URL to get to a list of all tags (provided you create such a page), it should be obvious that you could plug just about any word into the “tag-slug” part of the URL and it effectively functions like a tag-based search engine.

So, how do we actually build the URLs?

Being Generic is Good

Let’s get started. Paste this code into your blog/urls.py file:

from django.conf.urls.defaults import *

from djangoblog.blog.models import Entry

from tagging.views import tagged_object_list



info_dict = {

	'queryset': Entry.objects.filter(status=1),

	'date_field': 'pub_date',

}



urlpatterns = patterns('django.views.generic.date_based',

	(r'(?P<year>d{4})/(?P<month>[a-z]{3})/(?P<day>w{1,2})/(?P<slug>[-w]+)/$', 'object_detail', dict(info_dict, slug_field='slug',template_name='blog/detail.html')),

	(r'^(?P<year>d{4})/(?P<month>[a-z]{3})/(?P<day>w{1,2})/(?P<slug>[-w]+)/$', 'object_detail', dict(info_dict, template_name='blog/list.html')),

	(r'^(?P<year>d{4})/(?P<month>[a-z]{3})/(?P<day>w{1,2})/$','archive_day',dict(info_dict,template_name='blog/list.html')),

	(r'^(?P<year>d{4})/(?P<month>[a-z]{3})/$','archive_month', dict(info_dict, template_name='blog/list.html')),

	(r'^(?P<year>d{4})/$','archive_year', dict(info_dict, template_name='blog/list.html')),

	(r'^$','archive_index', dict(info_dict, template_name='blog/list.html')),

)

Now, remember when I said that the URL patterns determine which view Django will use to grab data from our database? In that scheme, we would write our regular expressions and then point each pattern to a function in views.py.

But we’re cheating a little bit here by taking advantage of some built in views that Django provides. These are known as “generic views.”

Django’s developers wisely figured that date-based archives were likely to be a common problem that just about every site has at least some use for, so they baked in some generic data-based views.

What we’ve done here is take advantage of the built in views to construct our URLs.

Let’s start with info_dict, which is just a Python dictionary that holds two things: a queryset containing all of our public blog entries and the name of our date field in the database.

It’s important to realize that Django querysets are lazy — Django only hits the database when the queryset is evaluated, so there’s no performance penalty for defining a queryset that looks for everything, then filtering it on a case-by-case basis. This happens to be essentially what we’ve just done.

Passing the queryset to the generic view enables Django to automatically do whatever date sorting we need, for instance, to show all the entries from a single month or year. For more info on querysets, check out the database API docs on the Django site.

That’s all the URL patterns list is: a regular expression that looks at the URL and figures out what view to use. The view then determines which entry or list of entries to show.

Let’s break it down and go through each part of the URL pattern. We’ll use the first line as an example:

(r'(?P<year>d{4})/(?P<month>[a-z]{3})/(?P<day>w{1,2})/(?P<slug>[-w]+)/$', 'object_detail', dict(info_dict, slug_field='slug',template_name='blog/detail.html')),

The first bit:

(?P<year>d{4})/(?P<month>[a-z]{3})/(?P<day>w{1,2})/(?P<slug>[-w]+)/$

is the regular expression. In this case, the expression will match our permalink URLs and capture the year, month, day and slug of a particular entry. That information will then be passed to the next bit, object_detail, which is the name of the generic view that will pull out a single entry.

The full path to object_detail is actually django.views.generic.date_based.object_detail, but since we started our urlpattern definition by including the django.views.generic.date_based bit, there’s no need to retype it every time, we just need to call the individual function, in this case object_detail.

After we grab the URL info and pass it to the object_detail function, we also pass along a dictionary of data. Most of the time you can just pass info_dict here. The object_detail generic view is something of an exception because it needs to pass along the slug_field variable as well.

I wanted to show some of the other data you can include as well, so I wrapped it in the dict your see above. In this case, we’ve passed info_dict, the slug_field and the name of the template Django should use to display the data.

The rest of the patterns just work their way back up the URL using ever-shortening regular expressions until we get to nothing, which would be the URL: http:/mysite.com/blog/.

We’ll be using that URL as the location of our archive page. So, I guess you can think of this as a tumblelog rather than a traditional blog, which would probably have separate archive and home pages. Naturally, you can tweak the URL patterns to fit whatever design you’d like.

Django’s generic views are incredibly powerful, and there are quite a few other options beyond just the date-based ones we’ve used here. There’s also a super-handy built-in pagination system for some generic views. Be sure to read through the documentation on the Django website and also have a look at James Bennett’s excellent guide to the various ways you can wrap and extend generic views.

Go your own way

Django’s generic views can save you quite a bit of time, but you will probably encounter some situations where they don’t quite do exactly what you want. When that happens, it’s time to write your own views.

Fear not, writing a custom view isn’t hard.

We’ve pretty much covered our blogs URLs, from date-based archives to the detail pages, but what about the pages that display our entries by tag?

The tagging application actually includes some views that we could use. But for the sake of example, we’ll write some custom views. Rather than overwriting what’s already in the tagging application, we’re just going to create a views file that lives on its own in our main project folder.

So, inside the “djangoblog” folder create a new file named tag_views.py. Remember, before we get started there, we need to tell Django about the tag_views file, so open up djangoblog/urls.py and add the last line below what’s already there:

urlpatterns = patterns('',

	(r'^admin/(.*)', admin.site.root),

	(r'^blog/', include('djangoblog.blog.urls')),

	(r'^tags/(?P<slug>[a-zA-Z0-9_.-]+)/$', 'djangoblog.tag_views.tag_detail'),

)

Here, we haven’t included another url.py file like we did with the lines above. You could argue that we should, but just to show that you don’t have to, we’ll just point directly to our tag_views.py file which will soon have a function named tag_detail. Note that in the tag URL, we’re capturing the slug paramater. We’ll use that in just a minute to filter our blog entries by tag.

Now it’s time to create the tag_detail function in the tag_views.py file. Open up that file in your text editor and paste in this code:

from django.views.generic.list_detail import object_detail



from tagging.models import Tag,TaggedItem

from blog.models import Entry



def tag_detail(request, slug):

	unslug = slug.replace('-', ' ')

	tag = Tag.objects.get(name=unslug)

	qs = TaggedItem.objects.get_by_model(Entry, tag)

	return object_list(request, queryset=qs, extra_context={'tag':slug}, template_name='tags/detail.html')

What’s going on here? Well, ignore the first line for now, we’ll get to that in a minute. We import all the things we need — in this case, the Tag and TaggedItem classes from django tagging and then our own Entry class. Then we define the tag_detail function, which is just an ordinary Python function that takes two parameters. The first is request which Django passes to all view functions, and the second is the slug paramater we defined in our URL pattern above.

Now because we’re using a slug for our tag URLs, but strings of words with spaces for our tags, we need to get rid of the dashes and replace them with spaces (Remember, a slug can contain letters and dashes, but not spaces).

Because our slug parameter is just a string, we can use the normal Python string function to make that replacement.

In the next line, we look up our tag name in the database using the objects manager. Then we take advantage of django-tagging’s built in function get_by_model to grab all the entries with the given tag.

The last step is to return something so that Django can load our template and display the entries to our visitor. To do that we’re going to use another of Django’s generic view functions — object_detail from the generic list views.

Object_detail requires a few things. First is the request object, then the queryset of results. After that, I’ve added an extra context variable named tag, so our template will be aware not just what entries to display, but also the current tag. Finally, the last item simply tells Django which template to use.

We haven’t yet created a URL for http://mysite.com/blog/tags/ to list all our tags, but that’s a good stepping stone to practice writing a view function on your own. Here’s a hint: you can use pretty much the same code we used for the tag_detail function, but you don’t need to worry about the slug param. And instead of looking up TaggedItems, just grab all the tags (i.e. qs = Tag.objects.all())

Conclusion

And there you have it, a quick and dirty overview of how URL patterns and view work in Django.

If you point your browser to our development URL (http://127.0.0.1:8000/blog/) you should see a Django error page complaining that the template blog/list.html does not exist. This is true, since we haven’t created it yet (visiting the tag pages will give you “list index out of range” error, also due to the missing templates).

But don’t worry — in the next lesson, we’ll dive into Django’s template system and explore all the cool things we can do, including how to write custom template filters and more.

See you then!

This is part 4 of Webmonkey’s introductory Django tutorial. If you’re arriving here to learn about getting started with Django, start back at the beginning with Lesson 1.

When we left off last time, we had defined some URLs for our blog and constructed a custom view to handle displaying posts by tag. If you point your browser to our development URL at this point, (http://127.0.0.1:8000/blog/) you’ll still see a Django error page complaining that the template blog/list.html does not exist. Don’t panic, it’s true — we haven’t created it yet.

It’s time to tackle the last aspect of Django, the template syntax.

Create some templates

If you look back through the code we’ve written so far, you’ll find that we’ve point Django to a number of templates (check out the [Tutorial:Use URL Patterns and Views in Django | urlpatterns code] and the custom view we wrote).

The templates we’ve defined need to be created. We’re going to use the following names within this directory structure:

djangoblog

	- templates

		-blog

			list.html

			detail.html

		-tags

			list.html

You can go ahead and create that directory structure — just create a folder in your main “djangoblog” folder and name it “templates.” Inside that, create two folders named “blog” and “tags.”

Then create your list.html and detail.html files.

Note: The .html extension is totally optional — I prefer it because it turns on the syntax highlighting feature in my text editor, but you can use any extension you like.

Inheritance

If we step back for a minute and think about our blog and what our templates need to display, the first thing that jumps out at you is that there’s a whole bunch of stuff that’s common to every page — a header, site navigation, sidebar, footer, and so on.

It would be silly (and a egregious violation of the DRY principle) if we wrote that code more than once. Luckily, like most good template languages, Django provides a way to extend a single template file. We can define our site-wide components once and then simply inherit from that file, adding in the aspects of the site that do change.

So before we dive into the detail pages, let’s first create a base file. Being the creative type, I generally call the file base.html. So inside the templates folder you created above, add a new file called base.html.

A simple template

Open that file in your text editor. Let’s sketch out some of the site-wide HTML we might need. For the sake of this tutorial, I’ve kept things pretty simple, but feel free to get as fancy as you want with your HTML. Here’s a basic starting point:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

<html lang="en">

	<head>

		<title>My Site - {%block pagetitle %}{% endblock %}</title>

	</head>



	<body>

		<div id="header">My Site</div>

		<div id="nav">

			<ul>

				<li><a href="/">home</a></li>



				<li><a href="/blog/">Blog</a></li>

				<li><a href="/links/">Links</a></li>

				<li><a href="/about/">About</a></li>



			</ul>

		</div>

		<div id="content">

			<div id="primary">

				<h1>{% block title %}{% endblock %}</h1>



				{% block primary %}{% endblock %}

			</div>

			<div id="secondary">

				<h3>Recent Entries:</h3>

			<div>



		</div>

	</body>

</html>

That should be pretty self explanatory, save perhaps the curious things enclosed in curly brackets, the {% %} bits. What up with that stuff?

That is Django’s template syntax for creating blocks which other templates can plug data into. In this case we’re creating a base template with a few blocks (think of them as holes) and our other templates will extend this and fill in the holes.

To see what I mean let’s create the blog/list.html template. Add a new blog folder in the templates folder (if you haven’t already) and create a file list.html. Now open that up and past in the line:

{% extends 'base.html' %}

Now, if you revisit http://127.0.0.1:8000/blog/ the template not found error should be gone. In its place, you should see everything we just put in base.html.

Not much, I’m afraid. But fear not — now that we’re extending base.html, we can start plugging in some values for those blocks we created earlier.

Add this below the extends statement:

{% block pagetitle %}Tumblelog{% endblock %}

{% block title %}My Page Headline{% endblock %}

{% block primary %}

{% for object in latest %}

	<h2>{{ object.title }}</h2>



	<p>{{ object.pub_date }}</p>

	 {{ object.body_html|truncatewords_html:"20"|safe }}

	<p>tags: {% for tag in object.get_tags%}<a href="/blog/tags/{{tag.name|slugify}}/">{{tag}}</a>{% endfor %}</p>

	<p><a href="{{object.get_absolute_url}}">read more</a></p>



{% endfor %}

{% endblock %}

The first thing we do is fill in our page title block in the head tags. Then we do the same for the displayed title. The next block we fill in is the primary content block. Here’s where we display the data that our generic view grabbed for us.

The Django template syntax can be considered fairly “Pythonic,” in that your define loops using the same for x in dataset syntax as Python. In this case, the generic view function object_detail passes in a variable named latest. By default, this displays the fifteen latest entries, though you can go back to the urls.py file and increase that number using the num_latest param.

All we do is construct a loop using the latest variable. Then within that loop we pull out some of our data — again, accessing an objects attributes uses the Python-like dot accessor methods.

Using built-in filters

The only part that requires additional explaining is the object.body_html section where we’ve applied two built-in Django template filters, truncatewords_html and safe.

Truncatewords_html should be fairly obvious — this clips the body of our post after twenty words, but also preserves the structure of the HTML by appending any closing tags to make sure the HTML remains intact.

The safe filter simply tells Django that it’s OK to display HTML. Without the safe filter, Django will automatically escape all HTML entities and tags. Autoescaping is a nice feature for avoiding nefarious XSS attacks and the like, but in this case, since we trust the source, we’ll let the HTML through.

Displaying tags

All fine so far, but what about tags? We have tags on our entries, so we might as well display them. This is what we do in the next line.

You’ll notice that in that line, we have a loop within a loop. Remember when we created our models and we added a get_tags method to return all the tags for each entry? Well, here it is in action.

That will loop through all the tags for each entry and display them along with a link to that tag’s permalink page. And note that we’ve used the slugify filter to make sure than any multiword tags will be hyphenated in the URL

Note: If you remember back when we wrote our custom tag view, we used a string replace function to “unslugify” the URL for lookup in the datebase.

Finishing up the template

The last line calls the get_absolute_url function that we defined when we built our model in the last lesson. This provides a link to the permalink page for each entry in the list.

{{object.title}}

The code might look something like this:

{% block pagetitle %}{{object.title}}{% endblock %}

{% block title %}{{object.title}}{% endblock %}

{% block primary %}

	<ul class="page-nav">

		{% if object.get_previous_published%}

		<li>

			<a href="{{ object.get_previous_published.get_absolute_url }}" title=" {{object.get_previous_published.title}}">« previous</a>



		</li>

		{%endif%}

		{% if object.get_next_published%}

			<li>

				<a href="{{ object.get_next_published.get_absolute_url }}" title=" {{object.get_next_published.title}}">next »</a>

			</li>

		{%endif%}

    </ul>



	<h2>{{ object.title }}</h2>

	<p>{{ object.pub_date }}</p>

	 {{ object.body_html|safe }}

	<p>tags: {% for tag in object.get_tags%}<a href="/blog/tags/{{tag.name|slugify}}/">{{tag}}</a>{% endfor %}</p>



	<p><a href="{{object.get_absolute_url}}">read more</a></p>

{% endblock %}

Note that I’ve made use of those get_next_published and get_previous_published functions that we defined way back when wrote our models. That way, users have some handy next and previous links for navigating through your permalink pages.

Naturally you can get much more sophisticated with your HTML than this simple example.

To create the templates for the tag pages, you’ll essentially do the same thing. In our custom tag view we returned a list of all the entries in an object named object_list. So in order to display them, just loop through object_list like we did with the latest variable above.

More about built-in template filters

Before we move on, a slight digression.

It’s worth paying a visit to the Django documentation on template filters and tags. There’s a whole bunch of useful stuff built in to Django. You can use {% if %} tags to narrow and filter results, and there’s also {% else %}, {% ifequal var1 var2 %}, {% ifnotequal var1 var2 %} and most other common programming structures.

Then there’s a host of template filters, like the truncatewords_html and safe filters we used above. For instance, there’s a handy date filter if you’d like to display your post date in something a little sexier than a UNIX timestamp.

Here’s what that would look like using the “date” filter:

{{object.pub_date|date:"D d M Y"}}

Another great filter is the timesince filter which will automatically convert a date into syntax like “1 week, 2 days ago”.

There are also filters for escaping ampersands, escaping JavaScript, adding line breaks, removing HTML, converting to upper/lowercase and dozens more. In fact in two and a half years of building sites with Django I’ve only needed to write a custom filter once. Naturally your mileage may vary somewhat.

Roll your own template tags

One thing I do frequently do is write custom template “tags.” Template tags are perhaps one of the more confusing aspects of Django, since they still have a little bit of “magic” in them. But luckily they aren’t too hard to work with.

Template tags are a way of extending the Django template system to use it in project specific ways. For instance, custom template tags are a perfect way to handle things that don’t make sense in a view, but do require some database logic.

Perhaps the best example is something like a sidebar. So far ours is empty, but we can easily add a list of our most recent blog posts.

We could write a filter that specifically fetches blog entries, but then what happens when we add links in the next lesson and want to display the most recent links? Write another template filter? That’s not very DRY! Instead, let’s just write a filter that fetches the most recent entries from any model with a date field.

In fact we don’t even need to really write it. James Bennett has already written some great reusable code so we’ll just use that. I strongly recommend that you have read through James’ tutorial so you can see how and why this code works.

Open a new file and paste in this code:

from django.template import Library, Node

from django.db.models import get_model



register = Library()



class LatestContentNode(Node):

    def __init__(self, model, num, varname):

        self.num, self.varname = num, varname

        self.model = get_model(*model.split('.'))



    def render(self, context):

        context[self.varname] = self.model._default_manager.all()[:self.num]

        return ''



def get_latest(parser, token):

    bits = token.contents.split()

    if len(bits) != 5:

        raise TemplateSyntaxError, "get_latest tag takes exactly four arguments"

    if bits[3] != 'as':

        raise TemplateSyntaxError, "third argument to get_latest tag must be 'as'"

    return LatestContentNode(bits[1], bits[2], bits[4])



get_latest = register.tag(get_latest)

Now save that file as “get_latest.py” in a new folder within the blog app named “templatetags”. The folder name and location are important since Django only looks up template tags in specific locations. You also need to create an empty file named “__init__.py” inside the “templatetags” folder.

One thing to note about this code, if you look closely you’ll notice that our template tag is going to fetch all entries, not just the public ones. In other words, our model allows for draft posts, but our template tag doesn’t.

This is the line in question:

self.model._default_manager.all()

There are two ways around this, one is quick and dirty — just change that line to filter only published entries. In other words:

self.model._default_manager.filter(status=1)

The better (and cleaner) way to do it would be overwrite the default manager for our Entry model. However, that’s a little beyond the scope of this article, so for now we’ll just use the quick and dirty method.

Now open up base.html again and add this line at the top:

{% load get_latest %}

That tells Django to load the template tag, but then we need to grab the actual data. So, head down to the sidebar section and replace it with this code:

<div id="secondary">

	<h3>Recent Entries:</h3>

	{% get_latest blog.Entry 5 as recent_posts %}

    <ul class="archive">



    	{% for obj in recent_posts %}

		<li>

			<a href="{{ obj.get_absolute_url }}" title="Permanent Link to {{ obj.title}}">{{ obj.title}}</a>

		</li>

    	{% endfor %}

	</ul>



<div>

If you want to override that in templates that are inheriting from base.html, just wrap that code in a {% block %} tag and then replace it with something else in the new template.

Links and other resources

The Django template system is quite vast in terms of capabilities, and we’ve really just scratched the surface here.

Make sure you read through the documentation for Python programmers as well as the documentation for template authors and familiarize yourself with all the various built in tags and filters. Another great article is Jeff Croft’s Django Templates: An Introduction.

It’s worth noting that there’s an extensive collection of useful filters and template tags on the Django Snippets site. If you ever find yourself needing a custom tag or filter, have a look through there and see if anyone has already written something that works for your project.

I should also point out that if you just don’t for whatever reason like Django’s template system, it’s possible drop in another template language, like Cheetah or others.

Be sure to stop by for our next installment when we’ll tackle a plan to import, store and display our del.icio.us bookmarks. See you then!

Welcome back! If you’ve been following along our entire series of tutorials on building sites with Django, you’ll (by now) have built a blog website with date-based archives and some nice extras such as tagging and Markdown support.

Along the way, we also ported our app over to the new Newforms Admin version of Django so that we’ll be all ready to go when Django hits version 1.0. If you haven’t done that yet, be sure to do it before we continue.

Delicious data

Now let’s branch out a little bit. Blogs are fine and dandy, but what about some of the cool web services we all use — Flickr, Delicious, Upcoming and the like. Wouldn’t it be cool if you could pull custom data from those services and display it on our site?

I’m going to walk you through integrating data from Delicious, the popular social bookmarking site, to power your own link blog. I’ve chosen Delicious because you’ll easily be able to take the basic outline we’ll use and apply it to any service with the decent API.

The way our setup will work is whenever we want to post a link to our site, we’ll take advantage of the nice Delicious front-end tools like browser add-ons and bookmarklets. We’ll then use the Delicious API to pull that info into our own site.

Before we get started, you might be wondering why? What’s the point when there are dead simple cut-and-paste JavaScript widgets that can already do that for us?

Well it’s true, there are some easier ways to integrate Delicious data, but JavaScript widgets don’t allow us to store the data locally on our own server, and that’s the win. There are two reasons local data is better. First, if Delicious goes down or the server stops responding for some reason, our local app is unaffected and keeps humming along as usual. Second, if Delicious ever goes away permanently or if you simply find another service you like better, it isn’t hard to add in the new service without affecting the existing data.

Convinced? OK let’s get started.

Building the model

The first thing we’ll do is create a new app. Why not just add a bookmark to our blog app? Well you could, but I’ve broken it out on its own for portability and for easy reusability. So, create a new folder in our djangoblog project and name it “links.”

Now in the links folder, create a new file called “models.py” (and don’t forget to add an __init__.py file in the links folder as well, so Python can access it). Open models.py in your text editor and add these lines:

from django.db import models



from tagging.fields import TagField

from tagging.models import Tag



class Link(models.Model):

    title = models.CharField(max_length=200)

    description = models.TextField()

    url = models.CharField(max_length=400)

    date = models.DateTimeField('Date published')

    tags = TagField()

    status = models.IntegerField(default=1)



    class Meta:

        ordering = ('-date',)

        get_latest_by = 'date'



    def __unicode__(self):

        return u'%s' %(self.title)



    def get_absolute_url(self):

        return "/link/%s/" %(self.id)



    def get_tags(self):

        return Tag.objects.get_for_object(self)

Look familiar? It should. It’s basically a very simplified version of our blog model. Now let’s add our Link model to the admin page. Create a new file in the links folder called “admin.py” and paste in this code:

from django.contrib import admin



from djangoblog.links.models import Link



class LinkAdmin(admin.ModelAdmin):

    pass



admin.site.register(Link, LinkAdmin)

Note that I haven’t customized anything here, but feel free to add some list_display options or filters for easier sorting in the Admin interface.

Now, open up the admin.py file in your djangoblog folder and change it so that it matches this code:

from django.contrib import admin

from djangoblog.blog.models import Entry

from djangoblog.blog.admin import EntryAdmin



from djangoblog.links.models import Link

from djangoblog.links.admin import LinkAdmin



class AdminSite(admin.AdminSite):

    pass



site = AdminSite()

site.register(Entry, EntryAdmin)

site.register(Link, LinkAdmin)

The last step is to let Django know about our new Links app, so open up your settings.py file and add “links” to your list of installed apps.

Now run syncdb in the shell:

python manage.py syncdb

And you’ll see that Django has created the required database tables. Fire up the development server and refresh the admin page to see your new links model.

Building a Delicious client

Having Django set up the backend is pretty cool, but we don’t actually want to administer our links — we want them to be automatically created when we bookmark something on Delicious.com.

To do that we’re going to need to send our user name and password over to Delicious which will the return a list of our most recent bookmarks. Then we need to parse through the XML, grab the data we need and store it in our new Link model.

It might sound very complex, but it really isn’t, and once you have it set up you’ll never have to bother with it again, so let’s get started.

What we need is a very simple client program to interact with Delicious.com. Luckily, there is a Python library that does just that. However, I find its dependancies make it more of a hassle than it needs to be. So I wrote my own client, which is dead simple, lacks many features of the other, but works for our purposes.

Now you might be wondering, where should we store this? There isn’t a hard and fast rule that I know of, but I generally stick these sorts of things in the “utils.py” file within my app. So, create a new file in the links directory and name it utils.py.

Here’s the code (which was sort of mashed up from several sources):

import urllib

import time

import dateutil.parser

import dateutil.tz

import xml.etree.cElementTree as xml_parser

from django.utils.encoding import smart_unicode



from djangoblog.links.models import Link



class DeliciousClient(object):

    interval = 0



    def __init__(self, username, password):

        self.username, self.password = username, password



    def fetch(self, **params):

        delta = time.time() - DeliciousClient.interval

        if delta < 2:

            time.sleep(2 - delta)

        DeliciousClient.interval = time.time()

        url = 'https://%s:%s@api.del.icio.us/v1/posts/recent' % (self.username, self.password)

        return self.fetch_xml(url)



    def fetch_xml(self, url):

        u = urllib.FancyURLopener(None)

        usock = u.open(url)

        rawdata = usock.read()

        usock.close()

        return xml_parser.fromstring(rawdata)





    def __getattr__(self, method):

        return DeliciousClient(self.username, self.password, '%s/%s' % (self.method, method))



    def __repr__(self):

        return "<DeliciousClient>"





def create_link(data):

    for post in data.findall('post'):

        info = dict((k, smart_unicode(post.get(k))) for k in post.keys())

        b, created = Link.objects.get_or_create(

            url = info['href'],

            description = info['extended'],

            tags = info.get('tag', ''),

            date = parsedate(info['time']),

            title = info['description']

        )





def parsedate(s):

	dt = dateutil.parser.parse(s)

	if dt.tzinfo:

	    dt = dt.astimezone(dateutil.tz.tzlocal()).replace(tzinfo=None)

	return dt

There are many ways we could do this. Granted, my particular take is not the best, but I wanted to avoid dependancies, which this does. The only exception is that the ElementTree library is only present starting in Python 2.5. If you’re using an earlier version you’ll need to download and install ElementTree separately if you want to use this code.

So what’s going on here? Well we have a class DeliciousClient which then has several methods. The main method of importance is “fetch,” which, when called, will return an ElementTree object with the 15 most recent links from our Delicious account.

One thing to note: one of the rules of the Delicious API is that you can’t ping it more frequently than once per second. The code at the top of the fetch function ensures that we don’t exceed that limit.

After the class, we have a small helper function that simply loops through the returned ElementTree object, pulls out the data nodes we need and then creates a new link in our database if one doesn’t already exist.

The last function is just a generic date parser that converts a string into a date object, since that’s what our Django model is looking for.

Using our new client

So how does it work? Well, in order to demonstrate this app and run it on your own site, you’ll need to have a Delicious account with some links in it. So once you’re set up, follow along and get ready to import some links.

Fire up your terminal and start the Django python client:

djangoblog sng$ python manage.py shell

Now let’s import our new tools and put them to use:

>>> from djangoblog.links import utils

>>> client = utils.DeliciousClient('username','passwd')

OK, now it’s time to fetch the data and save it to the database:

>>> data = client.fetch()

>>> utils.create_link(data)

With any luck, your fifteen most recent Delicious links should now be visible in the admin section. Start the dev server and refresh the page.

It’s all in the timing

So far, we’re feeling pretty proud of ourselves. But we don’t want to have to log in to the shell every time we want to update our displayed list of bookmarks. In fact, we want our site to automatically update itself. To do that, we’re going to write a quick python script and then run it through a cron job.

The only problem with that solution is that cron will run our script through the normal python interpreter which isn’t aware of our Django settings file and the like. Now worries though — we just need to tell it about it.

Create a new file named sync_link.py and paste in this code:

import sys, os



sys.path.append('/path/to/django')

sys.path.append('/path/to/djangoblog')

os.environ['DJANGO_SETTINGS_MODULE'] = 'djangoblog.settings'



from djangoblog.links import utils

client = utils.DeliciousClient('username','passwd')

data = client.fetch()

utils.create_link(data)

Now open your crontab and create a new entry that reads:

0,15,30,45 * * * * /path/to/python2.5 /path/to/sync_links.py 2>&1

That script will update your site with a new list every fifteen minutes. Naturally, you can adjust that list to suit your needs. If you’re a less-active bookmarker, once or twice per day should be sufficient.

Displaying links

To keep things simple, we’re not going to create permalink pages for our Delicious posts. If you’d like to do that, refer back to lesson 3, Use URL Patterns and Views in Django.

Instead, we’ll just put our links in the sidebar of our blog. Open up the base.html template and add this code in the sidebar section:

<h3>Recent Delicious Links</h3>

{% get_latest links.Link 5 as recent_links %}



<ul class="archive">

	{% for obj in recent_links %}

	<li>

		<a href="{{ obj.url }}" title="{{ obj.title}}">{{ obj.title}}</a>{{obj.description}} Posted on {{obj.pub_date|date:"D d M Y"}}

	</li>

	{% endfor %}



</ul>

See, I told you that template tag would come in handy! We’ve used the very same template tag we created in the last lesson to show recent blog entries to display the links. Very nice.

Conclusion

And there you have it, we’ve successfully integrated delicious into our blog. Hopefully, we’ve also shown how you might do something similar with other web services as well — Flickr, Twitter, Yelp, or anything with an API, really.

Keep in mind that our Delicious client is just a quick and dirty hack. There are a number of ways you could improve it, such as adding functionality to check for updates to links that already exist. Or perhaps even abstract it some more so that it interacts with any of the many other Delicious API methods. The limits are… limitless.

In the final installment of our Django Tutorial, we’ll set up our blog with a “river” of links and posts — a riff on the model popularized by FriendFeed and Tumblr — and add some RSS so people can subscribe to the streams of awesome content sure to follow.

See you then.

Thus far in our introductory Django tutorial, we’ve installed the open-source Django framework, set up a blog and beefed it up by adding some extras like semantic content tags, some handy template tags and a list of our bookmarks from delicious.com. If you haven’t been following along, now would be a good time to go back to Lesson 1 and catch up.

However, what we’ve created is not much different than what one could do with WordPress or another out-of-the-box blogging tool. That’s OK for a learning project. But now we’re getting close to being experts, we are going to explore some territory beyond what we can do with pre-built tools.

Let’s build something a little more advanced. Let’s build a microblog.

Let’s get ready to Tumble

These days, many of us follow our friends on services like FriendFeed or Facebook, where the concept of a “blog post” is much looser, smaller and faster.

Whether you want to call it a newsfeed, lifestream, microblog, tumblelog or a dozen other clever portmanteaus or neologisms, the concept is the same: a site to log snippets of your life and its discoveries in one place.

FriendFeed is a favorite. The service combines tweets from Twitter, photos from Flickr, links from Delicious, updates from Facebook and other sundry data and displays them all on one page. The page itself also has an RSS feed, giving you (or any of your many fans) a way to follow your every move.

Let’s build a version of Friendfeed using Django. We’ll build a page displaying both our links and blog posts together in a single, time-based “river.” As a bonus we’ll add in some RSS sugar so our friends can follow along without having to visit the site.

Laying the groundwork

Before we get started, let’s think for a bit about what needs to be done. We will post data two separate using two models: the blog Entry model and the Link model. We need to query them both at the same time and then sort the feed into reverse chronological order.

The key to making it work is by creating a way to normalize our data. The database table for each of these models are significantly different. Instead of querying both models, wouldn’t it be better if we just query against one table?

Let’s do it.

How it works

There’s one thing all models will have in common: a publication date. What we’re going to do is create a kind of meta model which stores the date as well as what kind of data it is (i.e. blog entry, link, etc).

Django ships with two tools which actually make it very easy to build our meta model. The first is the content types framework. As the docs explain, “instances of ContentType represent and store information about the models installed in your project, and new instances of ContentType are automatically created whenever new models are installed.”

Huh? Basically, what the Django Project is telling us is just by referencing the content type, we will always know what type of content we’re storing. Storing the content type is half the battle.

The other half of the battle will be won by using a built-in field known as a GenericForeignKey. You may have noticed a ForeignKey field in the Django docs. ForeignKey is similar to what we’re going to use. Where ForeignKey refers to a single model, our GenericForeignKey can refer to any model by referencing the content_type instead.

If that sounds confusing, don’t worry. It’ll make more sense when you see it in action.

Let’s start writing some code. I’m going to store our meta app in a folder named “tumblelog” so create the folder and create a new models.py file inside it (don’t forget the __init__.py file as well, Python’s never-ending “gotcha”). Open up models.py in your text editor and paste in this code:

from django.db import models

from django.contrib.contenttypes import generic



class TumbleItem(models.Model):

    content_type = models.ForeignKey(ContentType)

    object_id = models.PositiveIntegerField()

    pub_date = models.DateTimeField()

    content_object = generic.GenericForeignKey('content_type', 'object_id')



    class Meta:

        ordering = ('-pub_date',)



    def __unicode__(self):

        return self.content_type.name

If you’re familiar with our previous tutorials (or Django in general), by now this should look somewhat familiar. We have a regular ForeignKey to the ContentType as we discussed above. Then we have an id field, which we need to pass to the GenericForeignKey field (along with content_type). Finally we add a datetime field for easy sorting. If you look at the GenericForeignKey documentation you’ll notice this is pretty much the same code used to demonstrate GenericForeignKeys. The only difference is we’ve added a date field.

Now we have a basic script, but we certainly don’t want to update this by hand every time we post something. Especially since our delicious.com import script runs automatically.

Well, it turns out there’s a very powerful feature baked into Django which can handle the task for us.

Django includes an internal “dispatcher” which allows objects to listen for signals from other objects. In our case, our tumblelog app is going to “listen” to our Entry and Link models. Every time a new Entry or Link is saved, those models will send out a signal. When the tumblelog app gets the signal it will automatically update itself.

Sweet. How do we do it?

Listen for the Signals

The first thing we need to do is import the signals framework. Add this to the top of your model.py file:

from django.db.models import signals

from django.contrib.contenttypes.models import ContentType

from django.dispatch import dispatcher

from blog.models import Entry

from links.models import Link

OK, now move to the bottom of the file and add these lines:

for modelname in [Entry, Link]:

	dispatcher.connect(create_tumble_item, signal=signals.post_save, sender=modelname)

In Django 1.1 (and 1.0 I think) the signal have changed. Change the import to:

from django.db.models import signals

from django.contrib.contenttypes.models import ContentType

from django.db.models.signals import post_save

from blog.models import Entry

from links.models import Link

and change the dispatcher line to

post_save.connect(create_tweet_item, sender=Entry)

post_save.connect(create_tweet_item, sender=Link)

So what’s it do? Well it tells the dispatcher to listen for the post_save signal from out Entry and Link models and whenever it gets the signal, it will fire off our create_tumble_item function.

But wait, we don’t have a create_tumble_item function do we?

No, so we better write one. Just above the dispatcher function add this code:

def create_tumble_item(sender, instance, signal, *args, **kwargs):

	if 'created' in kwargs:

		if kwargs['created']:

			create = True

			ctype = ContentType.objects.get_for_model(instance)

			if ctype.name == 'link':

				pub_date = instance.date

			else:

				pub_date = instance.pub_date

			if create:

				t = TumbleItem.objects.get_or_create(content_type=ctype, object_id=instance.id, pub_date=pub_date)

What’s this function doing? For the most part it’s just taking the data sent by the dispatcher and using it to create a new TumbleItem.

The first line looks to see if a variable, created, has been passed by the dispatcher. The post_save signal is sent everytime an object is saved, so it’s possible we’ve just updated an existing item rather than creating a new one. In that case, we don’t want to create a new TumbleItem, so the function checks to make sure this is, in fact, a new object.

If the dispatcher has passed the variable created, and its true, then we set our own create flag and get the content_type of the passed instance. This way, we know whether it’s a Link or an Entry.

If it’s a Link we need to find the datetime info. When we built our Link model, we called the field date, so we set our normalizing pub_date field equal to the instance.date field.

If the instance is a blog Entry we set pub_date to the value of the instance’s pub_date field since, it shares the name we used in our Entry model.

Then we create a new TumbleItem using the built-in get_or_create method. We also pass in a content_type, id and pub_date<code> to their respective fields.

And there you have it. Any time you create a blog entry or the script adds a new link, our new TumbleItem model will automatically update.

There’s a slight problem though. What about the data we’ve already got in there?

Well to make sure it gets added, we’re going to have to comment out the following lines:

	if 'created' in kwargs:

		if kwargs['created']:

Comment it out and save the file. Then fire up the terminal and enter this code one line at a time:

>>> from blog.models import Entry

>>> from link.models import Link

>>> entries = Entry.objects.all()

>>> for entry in entries:

...     entry.save()

...



>>> links = Link.objects.all()

>>> for link in Link:

...     link.save()

...

>>>

What did it do? We just called the <code>save method and dispatcher passed its signal along to the TumbleItem model which then updated itself. Because we commented out the lines that check to see if an item is new, the function ran regardless of the instance’s created variable.

Now uncomment those lines and save the file.

Before we move on I should point out post_save isn’t the only signal Django can send. There are in fact a whole bunch of useful signals like request_started, pre_save and many more. Check out the Django wiki for more details.

Writing the URLs and views

Now we need to add a tumblelog URL and pull out the data. Open up your project level urls.py file and add this line:

(r'^tumblelog/', 'tumblelog.views.tumbler'),

Now create a new views.py file inside your tumblelog folder and open it up to paste in this code:

from tumblelog.models import TumbleItem

from django.shortcuts import render_to_response



def tumbler(request):

	context = {

		'tumble_item_list': TumbleItem.objects.all().order_by('-pub_date')

	}



	return render_to_response('tumblelog/list.html', context)

What’s going on here? Well first we grab the tumblelog items and then we use a Django shortcut function to return to pass the list on to template named list.html.

Let’s create the template. Add a new folder “tumblelog” inside your templates folder and create the list.html file. We’ll start by extending our base template and filling in the blocks we created in an earlier lesson:

{% block title %}My Tumblelog{% endblock %}

{% block primary %}

{% for object in object_list %}

	{%if object 'link'%}

		<a href="{{ obj.url }}" title="{{ obj.title}}">{{ obj.title}}</a>{{object.description}} Posted on {{object.pub_date|date:"D d M Y"}}

	{% endif %}



	{%if object 'entry'%}

		<h2>{{ object.title }}</h2>



		<p>{{ object.pub_date }}</p>

		{{ object.body_html|truncatewords_html:"20"|safe }}

		<p><a href="{{object.get_absolute_url}}">more</a></p>

	{% endif %}

{% endfor %}

{% endblock %}

If you start up the development server and head to http://127.0.0.1:8000/tumblelog/ you should see your blog entries and links ordered by date.

Where do we go from here?

We’re looking pretty good up until we get to the template stage. The if statements aren’t so bad when we’re only sorting two content types, but if you start pulling in tons of different types of data you’ll quickly end up with spaghetti code.

A far better idea would be to write a template tag which takes the content type as an argument, uses it to call a template file, renders it and passes back to the html as a string.

We’ll leave it up to you as an exercise (hint: read up on Django’s render_to_string method).

The other thing you might be wondering about is pagination. If you save dozens of links a day and blog like a madman, this page is going to get really large really fast. Django ships with some built-in pagination tools (see the docs) and there’s also a slick django-pagination app available from Eric Florenzano. Eric even has a very nice screencast on how to set things up.

RSS feeds

We’re almost done. Let’s hook up some RSS feeds for our new site.

It turns out, Django ships with a very nice syndication framework. All we need to do is turn it on and hook it up.

Let’s start by going into our TumbleItem models.py file and setting up the framework. Paste in this code at the bottom of the file:

from django.contrib.syndication.feeds import Feed

class LatestItems(Feed):

    title = "My Tumblelog: Links"

    link = "/tumblelog/"

    description = "Latest Items posted to mysite.com"

    description_template = 'feeds/description.html'



    def items(self):

        return TumbleItems.objects.all.order_by('-pub_date')[:10]

Here we’ve imported the syndication class Feed and then defined our own feed class to fill in some basic data. The last step is returning the items using a normal objects query.

We need to create the URLs first. There are several places you could do this, I generally opt for the main project URLs.py file. Open it up and paste this code just below the other import statements:

from tumblelog.models import LatestItems

feeds = {

    'tumblelog': LatestItems

}

Now add this line to your urlpattern function:

(r'^feeds/(?P<url>.*)/$', 'django.contrib.syndication.views.feed', {'feed_dict': feeds}),

The last step is creating the description template in a new folder we’ll lable “feeds.” Just plug in the same code from the tumblelog template, but delete the forloop tag since we’re only passing a single object this time.

And there you have it. Fire up the development server and point it to http://127.0.0.1:8000/feeds/tumblelog/. If you want to add separate feeds for links or just blog entries, all you need to do is repeat the process, creating new feed classes for each.

Conclusion

Whew! It’s been a long journey and we wrote a lot of code, but hopefully you learned a good bit about how Django works. You not only have a blog/linklog/tumblelog, but hopefully some ideas of how to approach other projects.

Speaking of projects, although we used one, there’s really no need to organize your code this way. Sometimes the projects come in handy, but other times they make your PYTHONPATH into a snarled mess. For some other ideas on how organize you code, check out James Bennet’s article on the subject.

If you ever have questions about Django, jump on the #Django IRC channel, there’s load of friendly and helpful developers who can point you in the right direction, and be sure to join the mailing list. Happy coding!

If you were a young programmer trying to pick a language to learn, based on these two websites, which would you choose?

Python or Java?