Quick dmigrations Update

I’ve finally been getting round to doing a bit of work on dmigrations, the Python based database migration tool we developed at work.

dmigrations offers a simple but flexible way of managing changes to the database in your Django projects. It is a replacement for Django’s built in syncdb command.

I still have some work to do to merge in a number of modifications we’ve been running internally for a while but I’ve triaged the issues list that had been left alone for too long and merged in a fantastic collection of patches from a few people including Alexander Robbins and Chris Lamb. Thanks hugely to everyone for these.

Everything is available in trunk in SVN at the moment ahead of me cleaning a few thinks up and updating the download version. I’ve added a setup.py file as well ahead of hopefully adding it to the Python Package Index.

I obviously wasn’t the only one who had decided to do some work on the project in the last couple of days. msaelices has just submitted a patch which adds a PostgreSQL backend. I don’t tend to use PostgreSQL personally so if anyone does want to have a look at running dmigrations with the patch I’d love to hear about it.

HTTP Debugging Server in Django

I’ve been busy building and playing with various HTTP clients recently, mainly due to more playing with RESTful web services. I took a couple of hours out to build something to make my life easier - namely a very simple logging HTTP server in Django.

All the application does is accepts HTTP requests and log the results to a file. I’ve been using it to make sure the requests I’m sending from elsewhere are correct, before pointing the client at a web service that actually does something useful. It supports POST, GET, PUT, DELETE, etc. So far so simple.

screenshot showing us tailing the logs

I ended up using Django mainly because I did most of the work on the train, and I know my way around it pretty well by now. I did originally thing it might have been better to use something simpler but I did end up learning a few new tricks that I’ll use for future projects.

  • In theory Django doesn’t need a database. While this is true for the most part, the in-built test runner does require a database. One thing I’ve talked about before is the usefulness of creating your own custom test runner. If anyone else runs into this specific problem have a look at the nodatabase.py file.
  • The logging is handled via Python’s inbuilt logging module. It has some nifty features (like log rotation, maximum file sizes and different logging levels) but is a little fiddly to get working correctly. Tales abound of log messages appearing multiple times, or permission errors causing problems. At least for this project I think I have everything working. I’ll try using this module on a couple of other projects and maybe spin it off into a small reusable app.

For something that took only a few hours to write it was both good fun and a useful learning tool. And Django proved itself more useful that I though for smaller services like this.

Startup Hubs in the UK - Where To Put One?

I always enjoy reading things by Paul Graham, even if I’d like to be able to disagree with his conclusions on occasion. It’s not just that I love Hackers and Painters, but also he often says things like the following that make me smile:

hackers are much more constrained by gentlemen’s agreements than regulations. If they shake your hand on a promise, they’ll keep it. But show them a lock and their first thought is how to pick it.

His latest short essay concerns startup hubs again, something that I’m actually pretty interested in, end up talking to people about, but hardly ever writing anything on. Mike over at Techcrunch Uk elicited some good comments last year with a post on the subject. As usual Graham focuses on the US, but what about translating his idea of buying a Startup Hub and applying it to the UK? He sets several criteria for success:

  • Do you have good weather?
  • Do people live downtown, or have they abandoned the center for the suburbs?
  • Would the city be described as “hip” and “tolerant,” or as reflecting “traditional values?”
  • Are there good universities nearby? Are there walkable neighborhoods?
  • Would nerds feel at home?

Lets assume for the moment that the idea would work. Take £15-20million and invest in 30 or so well chosen startups on the condition they move into town. The question is then: where in the UK would this be most likely to work? And are their any UK specific criteria that would increase the success rate?

Some people might be thinking “but no UK city is going to invest £20million in a scheme to encourage startups”. Leeds is doing just that. Or rather Yorkshire Forward are putting up £35million along with Leeds University putting up £31milion. What are they doing with all that money? They are going to build a big building with it - which is exactly what Graham says won’t work.

GitHub Changelog

It turns out the guys at GitHub publish the commit messages from their work on GitHub itself. If you really have to keep up you can even subscribe to an Atom feed.

OK, so this might not be hugely useful. But it is funny, and a just a nice sign of a small company being open in an interesting way. A few of my favourites:

mmm pizza… guard against bad SHAs in Walker#commit

really need to get a handle on this insane testing setup

I guess ff3 linux is the worst browser ever made

dont care

Epydoc Ant task

I quite like Epydoc for generating Python API documentation, even if the interface looks a little dated and could do with a lick of paint.

For most project I use an Ant build script to generate documentation when needed. You could wrap the basic commands in a make file or a bash script if you prefer that sort of thing though. The only trick is to make sure you have everything you need on your Python Path. In Django projects, or App Engine projects, you’ll probably find the default runner script plays with the path somewhat. The following example adds the current directory to the path along with an ext directory where I stash external modules, which should see it run cleanly without any import errors.

pre.

Example of using XMPP on App Engine (via IMified)

As I mentioned before, App Engine is getting an XMPP API at some point soon. But if you just can’t wait to start adding IM interfaces to your applications then you can do it now, by using a nifty third party in IMified.

IMified provide an incredibly simple HTTP API for interacting with your own IM bot. If we want to be buzz word compliant we might even call it a webhook. It’s also currently a free service while they work through the beta. The documentation is short and to the point but only contains examples in PHP. It supports multiple step conversations as well as authentication.

So, armed with a little time on the train over the last few days I got to work knocking together a quick demo application as a proof of concept. You can find the site on imified-demo.appspot.com and if you want to chat with the bot you can add [email protected] to your contacts. The bot uses the Jabber protocol so is available over Jabber or GTalk. IMified make it easy to use MSN or Yahoo IM accounts as well, which is something the App Engine API might very well not do I would imagine.

Screengrab of the IMified App Engine site

As always you can find the code on GitHub. Most of the code is actually just the site itself or settings to make local development easier. The following is a slightly edited version of the live code (logging and caching removed to make it easier to follow). All we need to do is accept a HTTP Post request with a list of arguments and return a plain text response. All being well the response is sent as a IM message to the sender.

pre. class IMified(webapp.RequestHandler): “This is the endpoint for the message from IMified” def post(self): “We recieve post data from IMified” userkey = self.request.get(‘userkey’) network = self.request.get(‘network’) msg = self.request.get(‘msg’) step = self.request.get(‘step’) try: # we try and create the message message = Message( userkey = userkey, network = network, msg = msg, step = int(step) ) message.put() # the response is send as an IM message to the sender self.response.out.write(‘Message saved’) except: self.response.out.write(‘An error ocured, message not saved’)

IMified can obviously be used outside App Engine as well, and in fact it’s not just about working around limitations in existing systems. Running the long running processes required for bots, and potentially even running your own XMPP server, is fiddly at times and requires at least some setup, monitoring and configuration to get working. Not having that as a barrier for entry for simple experiments or applications is a good thing.

Aral spoke at the last DJUGL about App Engine and mentioned a wide range of third party services that you can use to get around current limitations. IMified definitely fits into this group of support services very nicely indeed. I’d love to see them do really well as it really makes it much easier to get started with XMPP applications, even if what you can do is limited to a few simple APIs. I’d love to hear about other services that people are using in this way to build these distributed applications.

Python REST Client

I’m working on a small project involving using RESTful APIs and wanted a simple HTTP client, something that sat a little higher in the stack than httplib2 or similar. I turned initially to the Django Test Client which now supports all the required methods but it turned out that I’d have to unpick it from django a little.

With a little bit of looking around I found the python-rest-client which certainly sounded like it would do what I wanted. It lets you make HTTP requests in as straight forward a manner as possible and fitted the bill perfectly.

pre. from restful_lib import Connection conn = Connection(“http://morethanseven.net") response = conn.request_get(“/”)

It supports authentication, nice helper methods and gives you the response in a nice format.

pre. conn = Connection(“http://example.org", username=“XXX”, password=“XXX”) conn.request_delete(‘/items/11232344’)

As an added benefit it also comes with a Google App Engine compatible version.

Let you Sys Admin Override your Django Settings

The previous Django settings tip seemed to go down well so I thought I’d jot down a few more over the next few weeks. Most of these have come out of working with a decent sized Django team at Global so I can’t take credit for anything but writing them down for the most part. For this example I think Alex Knowles did the original version.

I was talking with out friendly sys admins on Friday about a new application and whether they were happy with some application specific logging (using the Python logging module) I’d build in. Nothing fancy, just application logging to a rotated log file for system events. Their answer was yes, as long as they could control where the log files ended up and the maximum file size, ideally without having to play around in the code or to redeploy the application if they wanted to move the files elsewhere.

These things were already specified in the settings file rather than hardcode into the application but that only gets us half way. The standard Linux way of setting this sort of thing is with a configuration file stored in the /etc directory. So we ended up with the following snippet of code in our settings.py file.

pre. # we’re going to allow overriding of settings via a yml file.

  1. This makes live nicer for anyone managing the box
  2. and means settings can be overloaded without redeploying the site SETTINGS_OVERIDE = “/etc/application_name.yml” try: file = open(SETTINGS_OVERIDE) for key, value in yaml.load(file).items(): globals()[key]=value except: # we don’t always have the file around or need the setting # defined so best to be quite if things go wrong pass

Then in the yaml file you can simply clobber any of the settings using a simple name value pair approach.

pre. LOG_FILE: “/var/log/application_name.log”

It lets us keep production paths that might change out of the developers code, at the same time as giving the sys admins a familiar way of managing the production environment.

It does have one downside, if you’re not aware of it’s presence then it can make debugging settings related issues a pain. With that in mind you could wrap it so as to only work this way when DEBUG is False, or take the approach here which is to leave extensive comments.

Looking for a New Job in June

As I think a few people are aware I’ll be leaving Global at the end of May.

It’s going to have been a pretty good year all told. I got to work with some great people and we built some pretty good stuff from the group up. The Capital Radio website was part of it but it’s the systems behind the scenes where all the interestingness lies. But as the team as a whole was just getting started the economy got bored of growing each year and went south. Global is first and foremost a radio company that makes money from advertising and, in need of cash, it decided to cut lots of the interactive department. All of that meant lots of redundancies and less interesting work and I decided I’d rather go now than wait around.

It’s an odd situation to be in but I’m not going to dwell on that and I’m not bitter about any of the goings on. I got to play with Django full time in what, I think, was the biggest Django development team in the UK. I got to work, and argue, with smart people every day. We had two internal hackdays where I got to play with XMPP and messaging apps. And I’m definitely a far better programmer than a year ago.

I’ll be sticking around at Global for the next three months to finish off a number of projects that I’ve been involved in and tidy up a few loose ends, as well as to work out what to do next. Which is where you (might) come in. Think of it like a job vacancy in reverse.

  • I’m looking for a job to start in June or thereabouts.
  • I live in Cambridge and have been working in London for the last 9 months, so I don’t mind a commute. I’d also consider something that had me working remotely and traveling further afield as needed.
  • My main focus is on finding something where I get to build interesting things. By that I mean a web development role that also involves bit of system design and other bits thrown in.
  • I’d like something that tallies with my current interests. This blog, Slideshare and my GitHub profile should give you an idea about what they are but to name but a few: testing, REST, APIs, messaging, content distribution, aggregation and development team tools.
  • Technology wise I’d be more than happy with something using Python or Ruby. I’d love to do more than just play around at home with Erlang as well, or the opportunity to pick up Scala. Having said all that if it’s interesting enough I don’t mind too much, I learn new tricks for fun and have 3 months of free time to kill.
  • Big company or small? It depends on the company, and probably depends more on the people. I’d choose smart people over a fancy company name any day of the week. I also don’t mind the idea of getting involved with a startup where some salary is offset against equity - but only if the idea is good enough.
  • I’ve got around 8 years of wide ranging professional experience, I write articles, speak at conferences and get things done. I know roughly what the going rate for good professional web developers is as well.
  • Ask the internet about me if you want to know more.

So if anyone knows of something that would be up my street I’d be grateful. Even better, if you or your company are looking for someone at the moment and don’t mind waiting a few months then I might be what you’re looking for. For those that don’t already have an email address you can find me on [email protected]

(There is obviously a chance that you might be a recruiter, rather than someone who wants me to work with or for them. I won’t ignore you out of hand but please don’t try and ring me unless I ask you to. Email is both quicker and easier for me to deal with. Also if you just want my CV for your files, have only a vague job description, or you want to tell me about a job in a company you can’t tell me the name of please don’t be offended if I don’t get back to you.)

Django Settings Tip - Setting Relative Paths

Django settings files are pretty interesting. Rather than being written in some sort of purely declarative markup they just use Python. This brings both lots of power as well as the ability to do things in the settings file that you probably shouldn’t do.

One area where I find this capability particularly useful is when specifying file system paths. Lots of the settings concern where Django can find templates, images, or stylesheets for instance. The examples given in the default settings file are all of the form /this/directory/structure/. If you plan on only working on your own, and never running your applications anywhere except your local machine this is probably fine. The moment you want to deploy your application, or want to collaborate with others this becomes a problem. You either have to agree upon a fixed directory structure between all developers (annoying) or have distinct settings files for each machine, which probably means them being outside source control (also annoying).

A better approach is to have those paths dynamically ascertained at runtime. It makes the application much more portable, making local development and production use easier. Using the standard library os module we can do just that.

pre. import os import django

  1. calculated paths for django and the site
  2. used as starting points for various other paths DJANGO_ROOT = os.path.dirname(os.path.realpath(django.file)) SITE_ROOT = os.path.dirname(os.path.realpath(file))

Here we set a couple of useful constants, one is the path to the site folder and the other the path to where django is stored on this machine. settings.py contains a number of places where these constants are useful. For instance the MEDIA_ROOT settings which specifies the file system location for assets like images or stylesheets. The default settings file even comes with an instruction and example showing a non portable path.

pre. # Absolute path to the directory that holds media.

  1. Example: “/home/media/media.lawrence.com/” MEDIA_ROOT = os.path.join(SITE_ROOT, ‘assets’)

Other examples include setting the path for a SQLite database:

pre. DATABASE_ENGINE = ‘sqlite3’ DATABASE_NAME = os.path.join(SITE_ROOT, ‘db’) + ‘/development.db’

Or specifying directories in which we can place templates.

pre. TEMPLATE_DIRS = ( os.path.join(SITE_ROOT, ‘templates’) )

I actually think this should probably make it’s way into the default settings file. I might very well be missing something but I can’t see when it’s not much better to do things this way.