Back Up Disqus Comments

There are a lot of ways to integrate forums and comments into a web page. Disqus just happens to be one of the best.

The only problem with using Disqus is there's no on-site backup -- all your comments are stored and managed through Disqus. If Disqus is down for some reason, or you decide to stop using it in the future? Well, you're screwed.

Luckily for us, the company recently launched a new public API, which offers some ways to retrieve your comment data and store it wherever you like. Double-click your favorite code editor and we'll dig into to how the Disqus API works.

Introduction

If you've never heard of Disqus, it is a very clever and simple way to add comments to just about any page. Thanks to the handy plugins for various blogging platforms (WordPress, Movable Type, Blogger and more) it's easy to integrate into your current publishing system.

For those of us using a custom blogging engine there's even some nice JavaScript code that can get Disqus comments on your site in no time.

WordPress users are lucky. There is an available plug-in bloggers can use to automatically generate local backups of Disqus comments. The rest of us are stuck with the API. Let's get started.

Getting Started

Bear with us. The Disqus API is slightly convoluted at first, but with a little work we can get what we're after. The first step is grabbing your Disqus API key (note that you should log into Disqus to get the API key).

Now, if you scan over the documentation you'll notice there's actually two API keys we need. The first is we linked to earlier. We'll also need a key for each Disqus forum we're going to access. Luckily we can query for the second key.

If the word "forum" is a little confusing, it's because it is. The terminology, we suspect, originates from Disqus' infancy when it was a forum software project. Here's the low down.

Each Disqus set-up is what Disqus refers to as a "forum." Within each site, or forum, you have message threads. The threads correspond to your blog posts or the page where you embed Disqus.

Then within each thread are the "posts," or comments, people have left on your site.

So the API process works like this:

1. First you fetch a list of forums (if you only have one site, there's just one forum_id to worry about)
2. Then you fetch a list of "threads" in each forum
3. then you request the comments for each thread

Dive in

To help get you started, I've written a quick and dirty Disqus API Client for Python. Because the Disqus API is young and subject to change, I didn't mirror it. Rather, I created a generic wrapper function which can handle all the current (and future) methods.

Here's the code:

import urllib

import simplejson


BASE_PATH = 'http://disqus.com/api/'
DEBUG = True


class DisqusError(Exception):
    def __init__(self, code, message):
        self.code, self.message = code, message
    def __str__(self):
        return 'DisqusError %s: %s' % (self.code, self.message)


class DisqusAPIClient():
    def __init__(self):
       """instantiate"""
    
    def __getattr__(self, method):
        def method(_self=self, _method=method, **params):
            url = "%s%s/?&%s" %  (BASE_PATH, _method, urllib.urlencode(params))
            if DEBUG: print url
            data = self.fetch(url)
            return data
        
        return method
    
    def fetch(self, url):
        data = simplejson.load(urllib.urlopen(url))
        if data.get("code", "") != "ok":
            raise DisqusError(data["code"], data["message"])
        return data['message']
        
    def __repr__(self):
        return "<DisqusClient: %s>" % self.method

Copy, paste and save the above code in a new file, named disqus.py, somewhere on your PythonPath. Be sure to note that we're using the Python simplejson library, so you'll need to download and install it if you haven't already.

Using the Disqus API

Okay, now we have something we can use to access Disqus and return nice, native Python objects. So do we go about using it?

Here's an example using the Python command line interface:

>>> from disqus import DisqusAPIClient
>>> client = DisqusAPIClient()
>>> API_KEY = 'XXXXXXXXXXXXXX'
>>> fkey = client.get_forum_list(user_api_key=API_KEY)
>>> fkey
[{u'created_at': u'2008-08-29 18:33:26.560284', u'shortname': u'luxagraf', u'name': u'luxagraf', u'id': u'00000000'}]

So the first thing we do is import our client. Then, we define our API key. The next step is fetching our list of forums using the Disqus method get_forum_list, which requires a parameter user_api_key along with the actual key.

As you can see, the result is a Python list containing, among other data, our forum ID. Now we can plug the forum ID into the function that will retrieve our forum key.

Here's how it works:

>>> forum_key = client.get_forum_api_key(user_api_key=API_KEY, forum_id=fkey[0]['id'])
>>> forum_key
u'u@E6KnR....'

All we've done here is call the get_forum_api_key method, passing it the user API key and the forum id, which we already extracted from our earlier call.

Now we have the forum key, we can actually retrieve a list of threads (all the posts where we have Disqus comments running).

Once we have a list of threads, we can query all comments on each thread:

>>> comments = []
>>> posts = client.get_thread_list(forum_api_key=forum_key)
>>> for post in posts:
...     comments.append(client.get_thread_posts(forum_api_key=forum_key, thread_id=post['id']))

What we've done here is create a list object to store all our comments and then queried for the list of threads. Once we have the thread we loop through each one and call get_thread_posts which returns the actual comments.

Then we just append those to our comments list.

Now we have all the comments that have been posted on each entry in a single Python list. From here all we need to do loop through the comments and store them in database that matches to all the kinds of data Disqus stores -- comment, commenter name, avatar and so on.

Because there are any number of ways you can do that, different databases etc, we'll leave that as an exercise for the reader. That said, to access the individual comments and associated data you would just need to loop through our comments list like so:

>>> for c in comments:
...     if c:
...         print c[0]['message']... etc

Rather than simply printing out the data, just call a function that writes the data to the database and you'll have a local backup of Disqus comments.

Conclusion

When it comes to sending comments to Disqus, you'll have to stick with the JavaScript forms that the company provides -- at least for now. Although, the API docs do note that Disqus is looking into other ways of submitting comments.

Still, despite being one-way, the Disqus API makes it relatively easy to store a local backup of all comments that have been submitted to your site through the Disqus service.