Integrate Web APIs into Your Django Site

Welcome back! If you've been following along our entire series of tutorials on building sites with Django, you'll (by now) have built a blog website with date-based archives and some nice extras such as tagging and Markdown support.

Along the way, we also ported our app over to the new Newforms Admin version of Django so that we'll be all ready to go when Django hits version 1.0. If you haven't done that yet, be sure to do it before we continue.

Delicious data

Now let's branch out a little bit. Blogs are fine and dandy, but what about some of the cool web services we all use -- Flickr, Delicious, Upcoming and the like. Wouldn't it be cool if you could pull custom data from those services and display it on our site?

I'm going to walk you through integrating data from Delicious, the popular social bookmarking site, to power your own link blog. I've chosen Delicious because you'll easily be able to take the basic outline we'll use and apply it to any service with the decent API.

The way our setup will work is whenever we want to post a link to our site, we'll take advantage of the nice Delicious front-end tools like browser add-ons and bookmarklets. We'll then use the Delicious API to pull that info into our own site.

Before we get started, you might be wondering why? What's the point when there are dead simple cut-and-paste JavaScript widgets that can already do that for us?

Well it's true, there are some easier ways to integrate Delicious data, but JavaScript widgets don't allow us to store the data locally on our own server, and that's the win. There are two reasons local data is better. First, if Delicious goes down or the server stops responding for some reason, our local app is unaffected and keeps humming along as usual. Second, if Delicious ever goes away permanently or if you simply find another service you like better, it isn't hard to add in the new service without affecting the existing data.

Convinced? OK let's get started.

Building the model

The first thing we'll do is create a new app. Why not just add a bookmark to our blog app? Well you could, but I've broken it out on its own for portability and for easy reusability. So, create a new folder in our djangoblog project and name it "links."

Now in the links folder, create a new file called "models.py" (and don't forget to add an __init__.py file in the links folder as well, so Python can access it). Open models.py in your text editor and add these lines:

from django.db import models

from tagging.fields import TagField
from tagging.models import Tag

class Link(models.Model):
    title = models.CharField(max_length=200)
    description = models.TextField()
    url = models.CharField(max_length=400)
    date = models.DateTimeField('Date published')
    tags = TagField()
    status = models.IntegerField(default=1)
    
    class Meta:
        ordering = ('-date',)
        get_latest_by = 'date'

    def __unicode__(self):
        return u'%s' %(self.title)

    def get_absolute_url(self):
        return "/link/%s/" %(self.id)
    
    def get_tags(self):
        return Tag.objects.get_for_object(self)

Look familiar? It should. It's basically a very simplified version of our blog model. Now let's add our Link model to the admin page. Create a new file in the links folder called "admin.py" and paste in this code:

from django.contrib import admin

from djangoblog.links.models import Link

class LinkAdmin(admin.ModelAdmin):
    pass

admin.site.register(Link, LinkAdmin)

Note that I haven't customized anything here, but feel free to add some list_display options or filters for easier sorting in the Admin interface.

Now, open up the admin.py file in your djangoblog folder and change it so that it matches this code:

from django.contrib import admin
from djangoblog.blog.models import Entry
from djangoblog.blog.admin import EntryAdmin

from djangoblog.links.models import Link
from djangoblog.links.admin import LinkAdmin

class AdminSite(admin.AdminSite):
    pass
    
site = AdminSite()
site.register(Entry, EntryAdmin)
site.register(Link, LinkAdmin)

The last step is to let Django know about our new Links app, so open up your settings.py file and add "links" to your list of installed apps.

Now run syncdb in the shell:

python manage.py syncdb

And you'll see that Django has created the required database tables. Fire up the development server and refresh the admin page to see your new links model.

Building a Delicious client

Having Django set up the backend is pretty cool, but we don't actually want to administer our links -- we want them to be automatically created when we bookmark something on Delicious.com.

To do that we're going to need to send our user name and password over to Delicious which will the return a list of our most recent bookmarks. Then we need to parse through the XML, grab the data we need and store it in our new Link model.

It might sound very complex, but it really isn't, and once you have it set up you'll never have to bother with it again, so let's get started.

What we need is a very simple client program to interact with Delicious.com. Luckily, there is a Python library that does just that. However, I find its dependancies make it more of a hassle than it needs to be. So I wrote my own client, which is dead simple, lacks many features of the other, but works for our purposes.

Now you might be wondering, where should we store this? There isn't a hard and fast rule that I know of, but I generally stick these sorts of things in the "utils.py" file within my app. So, create a new file in the links directory and name it utils.py.

Here's the code (which was sort of mashed up from several sources):

import urllib 
import time
import dateutil.parser
import dateutil.tz
import xml.etree.cElementTree as xml_parser
from django.utils.encoding import smart_unicode

from djangoblog.links.models import Link

class DeliciousClient(object):
    interval = 0

    def __init__(self, username, password):
        self.username, self.password = username, password
    
    def fetch(self, **params):
        delta = time.time() - DeliciousClient.interval
        if delta < 2:
            time.sleep(2 - delta)
        DeliciousClient.interval = time.time()
        url = 'https://%s:%s@api.del.icio.us/v1/posts/recent' % (self.username, self.password)
        return self.fetch_xml(url)
    
    def fetch_xml(self, url):
        u = urllib.FancyURLopener(None)
        usock = u.open(url)
        rawdata = usock.read()
        usock.close()
        return xml_parser.fromstring(rawdata)

        
    def __getattr__(self, method):
        return DeliciousClient(self.username, self.password, '%s/%s' % (self.method, method))
        
    def __repr__(self):
        return "<DeliciousClient>"
        

def create_link(data):
    for post in data.findall('post'):
        info = dict((k, smart_unicode(post.get(k))) for k in post.keys())
        b, created = Link.objects.get_or_create(
            url = info['href'],
            description = info['extended'],
            tags = info.get('tag', ''),
            date = parsedate(info['time']),
            title = info['description']
        )


def parsedate(s):
	dt = dateutil.parser.parse(s)
	if dt.tzinfo:
	    dt = dt.astimezone(dateutil.tz.tzlocal()).replace(tzinfo=None)
	return dt

There are many ways we could do this. Granted, my particular take is not the best, but I wanted to avoid dependancies, which this does. The only exception is that the ElementTree library is only present starting in Python 2.5. If you're using an earlier version you'll need to download and install ElementTree separately if you want to use this code.

So what's going on here? Well we have a class DeliciousClient which then has several methods. The main method of importance is "fetch," which, when called, will return an ElementTree object with the 15 most recent links from our Delicious account.

One thing to note: one of the rules of the Delicious API is that you can't ping it more frequently than once per second. The code at the top of the fetch function ensures that we don't exceed that limit.

After the class, we have a small helper function that simply loops through the returned ElementTree object, pulls out the data nodes we need and then creates a new link in our database if one doesn't already exist.

The last function is just a generic date parser that converts a string into a date object, since that's what our Django model is looking for.

Using our new client

So how does it work? Well, in order to demonstrate this app and run it on your own site, you'll need to have a Delicious account with some links in it. So once you're set up, follow along and get ready to import some links.

Fire up your terminal and start the Django python client:

djangoblog sng$ python manage.py shell

Now let's import our new tools and put them to use:

>>> from djangoblog.links import utils
>>> client = utils.DeliciousClient('username','passwd')

OK, now it's time to fetch the data and save it to the database:

>>> data = client.fetch()
>>> utils.create_link(data)

With any luck, your fifteen most recent Delicious links should now be visible in the admin section. Start the dev server and refresh the page.

It's all in the timing

So far, we're feeling pretty proud of ourselves. But we don't want to have to log in to the shell every time we want to update our displayed list of bookmarks. In fact, we want our site to automatically update itself. To do that, we're going to write a quick python script and then run it through a cron job.

The only problem with that solution is that cron will run our script through the normal python interpreter which isn't aware of our Django settings file and the like. Now worries though -- we just need to tell it about it.

Create a new file named sync_link.py and paste in this code:

import sys, os

sys.path.append('/path/to/django')
sys.path.append('/path/to/djangoblog')
os.environ['DJANGO_SETTINGS_MODULE'] = 'djangoblog.settings'

from djangoblog.links import utils
client = utils.DeliciousClient('username','passwd')
data = client.fetch()
utils.create_link(data)

Now open your crontab and create a new entry that reads:

0,15,30,45 * * * * /path/to/python2.5 /path/to/sync_links.py 2>&1

That script will update your site with a new list every fifteen minutes. Naturally, you can adjust that list to suit your needs. If you're a less-active bookmarker, once or twice per day should be sufficient.

Displaying links

To keep things simple, we're not going to create permalink pages for our Delicious posts. If you'd like to do that, refer back to lesson 3, Use URL Patterns and Views in Django.

Instead, we'll just put our links in the sidebar of our blog. Open up the base.html template and add this code in the sidebar section:

<h3>Recent Delicious Links</h3>
{% get_latest links.Link 5 as recent_links %}
<ul class="archive">
	{% for obj in recent_links %}
	<li>
		<a href="{{ obj.url }}" title="{{ obj.title}}">{{ obj.title}}</a>{{obj.description}} Posted on {{obj.pub_date|date:"D d M Y"}}
	</li>
	{% endfor %}
</ul>

See, I told you that template tag would come in handy! We've used the very same template tag we created in the last lesson to show recent blog entries to display the links. Very nice.

Conclusion

And there you have it, we've successfully integrated delicious into our blog. Hopefully, we've also shown how you might do something similar with other web services as well -- Flickr, Twitter, Yelp, or anything with an API, really.

Keep in mind that our Delicious client is just a quick and dirty hack. There are a number of ways you could improve it, such as adding functionality to check for updates to links that already exist. Or perhaps even abstract it some more so that it interacts with any of the many other Delicious API methods. The limits are... limitless.

In the final installment of our Django Tutorial, we'll set up our blog with a "river" of links and posts -- a riff on the model popularized by FriendFeed and Tumblr -- and add some RSS so people can subscribe to the streams of awesome content sure to follow.

See you then.

• This page was last modified 11:22, 11 April 2009.