GitHub Changelog

It turns out the guys at GitHub publish the commit messages from their work on GitHub itself. If you really have to keep up you can even subscribe to an Atom feed.

OK, so this might not be hugely useful. But it is funny, and a just a nice sign of a small company being open in an interesting way. A few of my favourites:

mmm pizza… guard against bad SHAs in Walker#commit

really need to get a handle on this insane testing setup

I guess ff3 linux is the worst browser ever made

dont care

Epydoc Ant task

I quite like Epydoc for generating Python API documentation, even if the interface looks a little dated and could do with a lick of paint.

For most project I use an Ant build script to generate documentation when needed. You could wrap the basic commands in a make file or a bash script if you prefer that sort of thing though. The only trick is to make sure you have everything you need on your Python Path. In Django projects, or App Engine projects, you’ll probably find the default runner script plays with the path somewhat. The following example adds the current directory to the path along with an ext directory where I stash external modules, which should see it run cleanly without any import errors.

pre.

Example of using XMPP on App Engine (via IMified)

As I mentioned before, App Engine is getting an XMPP API at some point soon. But if you just can’t wait to start adding IM interfaces to your applications then you can do it now, by using a nifty third party in IMified.

IMified provide an incredibly simple HTTP API for interacting with your own IM bot. If we want to be buzz word compliant we might even call it a webhook. It’s also currently a free service while they work through the beta. The documentation is short and to the point but only contains examples in PHP. It supports multiple step conversations as well as authentication.

So, armed with a little time on the train over the last few days I got to work knocking together a quick demo application as a proof of concept. You can find the site on imified-demo.appspot.com and if you want to chat with the bot you can add [email protected] to your contacts. The bot uses the Jabber protocol so is available over Jabber or GTalk. IMified make it easy to use MSN or Yahoo IM accounts as well, which is something the App Engine API might very well not do I would imagine.

As always you can find the code on GitHub. Most of the code is actually just the site itself or settings to make local development easier. The following is a slightly edited version of the live code (logging and caching removed to make it easier to follow). All we need to do is accept a HTTP Post request with a list of arguments and return a plain text response. All being well the response is sent as a IM message to the sender.

pre. class IMified(webapp.RequestHandler): “This is the endpoint for the message from IMified” def post(self): “We recieve post data from IMified” userkey = self.request.get(‘userkey’) network = self.request.get(‘network’) msg = self.request.get(‘msg’) step = self.request.get(‘step’) try: # we try and create the message message = Message( userkey = userkey, network = network, msg = msg, step = int(step) ) message.put() # the response is send as an IM message to the sender self.response.out.write(‘Message saved’) except: self.response.out.write(‘An error ocured, message not saved’)

IMified can obviously be used outside App Engine as well, and in fact it’s not just about working around limitations in existing systems. Running the long running processes required for bots, and potentially even running your own XMPP server, is fiddly at times and requires at least some setup, monitoring and configuration to get working. Not having that as a barrier for entry for simple experiments or applications is a good thing.

Aral spoke at the last DJUGL about App Engine and mentioned a wide range of third party services that you can use to get around current limitations. IMified definitely fits into this group of support services very nicely indeed. I’d love to see them do really well as it really makes it much easier to get started with XMPP applications, even if what you can do is limited to a few simple APIs. I’d love to hear about other services that people are using in this way to build these distributed applications.

Python REST Client

I’m working on a small project involving using RESTful APIs and wanted a simple HTTP client, something that sat a little higher in the stack than httplib2 or similar. I turned initially to the Django Test Client which now supports all the required methods but it turned out that I’d have to unpick it from django a little.

With a little bit of looking around I found the python-rest-client which certainly sounded like it would do what I wanted. It lets you make HTTP requests in as straight forward a manner as possible and fitted the bill perfectly.

pre. from restful_lib import Connection conn = Connection(“http://morethanseven.net") response = conn.request_get(“/”)

It supports authentication, nice helper methods and gives you the response in a nice format.

pre. conn = Connection(“http://example.org", username=“XXX”, password=“XXX”) conn.request_delete(‘/items/11232344’)

As an added benefit it also comes with a Google App Engine compatible version.

Let you Sys Admin Override your Django Settings

The previous Django settings tip seemed to go down well so I thought I’d jot down a few more over the next few weeks. Most of these have come out of working with a decent sized Django team at Global so I can’t take credit for anything but writing them down for the most part. For this example I think Alex Knowles did the original version.

I was talking with out friendly sys admins on Friday about a new application and whether they were happy with some application specific logging (using the Python logging module) I’d build in. Nothing fancy, just application logging to a rotated log file for system events. Their answer was yes, as long as they could control where the log files ended up and the maximum file size, ideally without having to play around in the code or to redeploy the application if they wanted to move the files elsewhere.

These things were already specified in the settings file rather than hardcode into the application but that only gets us half way. The standard Linux way of setting this sort of thing is with a configuration file stored in the /etc directory. So we ended up with the following snippet of code in our settings.py file.

pre. # we’re going to allow overriding of settings via a yml file.

1. This makes live nicer for anyone managing the box
2. and means settings can be overloaded without redeploying the site SETTINGS_OVERIDE = “/etc/application_name.yml” try: file = open(SETTINGS_OVERIDE) for key, value in yaml.load(file).items(): globals()[key]=value except: # we don’t always have the file around or need the setting # defined so best to be quite if things go wrong pass

Then in the yaml file you can simply clobber any of the settings using a simple name value pair approach.

pre. LOG_FILE: “/var/log/application_name.log”

It lets us keep production paths that might change out of the developers code, at the same time as giving the sys admins a familiar way of managing the production environment.

It does have one downside, if you’re not aware of it’s presence then it can make debugging settings related issues a pain. With that in mind you could wrap it so as to only work this way when DEBUG is False, or take the approach here which is to leave extensive comments.

Looking for a New Job in June

As I think a few people are aware I’ll be leaving Global at the end of May.

It’s going to have been a pretty good year all told. I got to work with some great people and we built some pretty good stuff from the group up. The Capital Radio website was part of it but it’s the systems behind the scenes where all the interestingness lies. But as the team as a whole was just getting started the economy got bored of growing each year and went south. Global is first and foremost a radio company that makes money from advertising and, in need of cash, it decided to cut lots of the interactive department. All of that meant lots of redundancies and less interesting work and I decided I’d rather go now than wait around.

It’s an odd situation to be in but I’m not going to dwell on that and I’m not bitter about any of the goings on. I got to play with Django full time in what, I think, was the biggest Django development team in the UK. I got to work, and argue, with smart people every day. We had two internal hackdays where I got to play with XMPP and messaging apps. And I’m definitely a far better programmer than a year ago.

I’ll be sticking around at Global for the next three months to finish off a number of projects that I’ve been involved in and tidy up a few loose ends, as well as to work out what to do next. Which is where you (might) come in. Think of it like a job vacancy in reverse.

• I’m looking for a job to start in June or thereabouts.
• I live in Cambridge and have been working in London for the last 9 months, so I don’t mind a commute. I’d also consider something that had me working remotely and traveling further afield as needed.
• My main focus is on finding something where I get to build interesting things. By that I mean a web development role that also involves bit of system design and other bits thrown in.
• I’d like something that tallies with my current interests. This blog, Slideshare and my GitHub profile should give you an idea about what they are but to name but a few: testing, REST, APIs, messaging, content distribution, aggregation and development team tools.
• Technology wise I’d be more than happy with something using Python or Ruby. I’d love to do more than just play around at home with Erlang as well, or the opportunity to pick up Scala. Having said all that if it’s interesting enough I don’t mind too much, I learn new tricks for fun and have 3 months of free time to kill.
• Big company or small? It depends on the company, and probably depends more on the people. I’d choose smart people over a fancy company name any day of the week. I also don’t mind the idea of getting involved with a startup where some salary is offset against equity - but only if the idea is good enough.
• I’ve got around 8 years of wide ranging professional experience, I write articles, speak at conferences and get things done. I know roughly what the going rate for good professional web developers is as well.
• Ask the internet about me if you want to know more.

So if anyone knows of something that would be up my street I’d be grateful. Even better, if you or your company are looking for someone at the moment and don’t mind waiting a few months then I might be what you’re looking for. For those that don’t already have an email address you can find me on [email protected]

(There is obviously a chance that you might be a recruiter, rather than someone who wants me to work with or for them. I won’t ignore you out of hand but please don’t try and ring me unless I ask you to. Email is both quicker and easier for me to deal with. Also if you just want my CV for your files, have only a vague job description, or you want to tell me about a job in a company you can’t tell me the name of please don’t be offended if I don’t get back to you.)

Django Settings Tip - Setting Relative Paths

Django settings files are pretty interesting. Rather than being written in some sort of purely declarative markup they just use Python. This brings both lots of power as well as the ability to do things in the settings file that you probably shouldn’t do.

One area where I find this capability particularly useful is when specifying file system paths. Lots of the settings concern where Django can find templates, images, or stylesheets for instance. The examples given in the default settings file are all of the form /this/directory/structure/. If you plan on only working on your own, and never running your applications anywhere except your local machine this is probably fine. The moment you want to deploy your application, or want to collaborate with others this becomes a problem. You either have to agree upon a fixed directory structure between all developers (annoying) or have distinct settings files for each machine, which probably means them being outside source control (also annoying).

A better approach is to have those paths dynamically ascertained at runtime. It makes the application much more portable, making local development and production use easier. Using the standard library os module we can do just that.

pre. import os import django

1. calculated paths for django and the site
2. used as starting points for various other paths DJANGO_ROOT = os.path.dirname(os.path.realpath(django.file)) SITE_ROOT = os.path.dirname(os.path.realpath(file))

Here we set a couple of useful constants, one is the path to the site folder and the other the path to where django is stored on this machine. settings.py contains a number of places where these constants are useful. For instance the MEDIA_ROOT settings which specifies the file system location for assets like images or stylesheets. The default settings file even comes with an instruction and example showing a non portable path.

pre. # Absolute path to the directory that holds media.

1. Example: “/home/media/media.lawrence.com/” MEDIA_ROOT = os.path.join(SITE_ROOT, ‘assets’)

Other examples include setting the path for a SQLite database:

pre. DATABASE_ENGINE = ‘sqlite3’ DATABASE_NAME = os.path.join(SITE_ROOT, ‘db’) + ‘/development.db’

Or specifying directories in which we can place templates.

pre. TEMPLATE_DIRS = ( os.path.join(SITE_ROOT, ‘templates’) )

I actually think this should probably make it’s way into the default settings file. I might very well be missing something but I can’t see when it’s not much better to do things this way.

Append slashes to URLs in Django

Quick Django pop quiz. Can anyone spot the deliberate mistake in the following url definition? We’re trying to define a view called log_viewer and instructing a specific url pattern to render it.

pre. urlpatterns = patterns(“, (r’^log/?$‘, log_viewer), )

In this case our regex matches /log or /log/ using the /? optional pattern. This is because even if we only link to one format we know people will probably visit both, either by entering the URL manually or by linking from an external source.

As far as HTTP is concerned though /log and /log/ are separate URLs, even if they display the same content. The main reason this matters for public facing websites is that our friendly search engine spiders are likely to index both separately, leading to splitting the page rank as well as accusations of duplicate content which might see further erosion of rankings.

The solution is generally to issue a 301 redirect from one format to the other. This tells search engines and people alike that the canonical location for the requested content is elsewhere. You could specify the redirect manually, but this is going to get irritating quickly once you have a few more definitions.

pre. urlpatterns = patterns(“, (r’^log$‘, redirect_to, {‘url’: ‘/log/’}), (r’^log/$‘, log_viewer), )

Handily Django provides a mechanism to do exactly what we want to do by setting APPEND_SLASH to True in your settings file. Even better it’s switched on by default. So if you don’t know much about the intricacies of HTTP you still get the correct behavior. That is unless you specify your URL patterns in the format above.

You see APPEND_SLASH only works if the URL doesn’t match a specified pattern. If no pattern match is found it appends a trailing slash and checks for a match again. Because the above pattern matches the pattern without the trailing slash (/log) the desired behavior is never triggered, and the view is rendered at both URLs. So although we want to catch /log and /log/ on the front end, our urls.py definition should actually be:

pre. urlpatterns = patterns(“, (r’^log/$‘, log_viewer), )

Django has lots of useful bits of magic for doing the right thing, but unless you know what they actually do you either end up recreating functionality yourself, or find features don’t work in quite the way you thought. It’s a good argument for keeping frameworks small whenever possible, and for developers to at least know their way around the code of their respective framework.

New Version of Radiant CMS Out Today

Via tagging a new release on GitHub I see a new version of Radiant (0.7) has been released. Radiant is a really nice CMS for smaller projects, it’s used on the official Ruby site and I used it here at one point.

Go check out the blog post for the full list of the changes.

PDB and AppEngine

It turns out App Engine breaks the default behaviour of the Python debugger PDB by sending STDOUT to the browser. But with a little bit of python you can put it back in.

pre. import sys import pdb for attr in (‘stdin’, ‘stdout’, ‘stderr’): setattr(sys, attr, getattr(sys, ‘s’ attr)) pdb.set_trace()