RewiredState was awesome. 100 or so geeks plus a smattering of government types gathered in the shiny new Guardian offices in Kings Cross on Saturday to hack (the Government).

Some events like this are more productive than others, and the end of day demos included some realy impressive stuff. See for your self on the projects page

My own little project even won a prize (an invisible bottle of Champagne no less). My complain was that if you want to report an issue about the over 7000 government websites you have to do it per site. All of the sites do their own thing, which might be a nice contact form, maybe an email address or in some cases a postal address hidden on a page that’s not linked to from anywhere.

My solution was pretty simple - a centralised issue reporting and navigating tool. So you go along to the site you have an issue with and hit a bookmarklet (or a badge on the site if the site in question have been nice enough to add one). You fill in a very simple form which appears before your eyes and everything is tracked on a nice shiny website.

The advantage of all that is transparency. You can see which sites people have issue with, and also ideally which issues get addresses or at least acknowledged by the support staff for the site in question. The hack had comments so others could follow up on individual issues. It would be simple enough to have league tables and the like as well, or add tagging for a little bit of categorisation - I did only have a few hours though.

screenshot of FeedbackGov website

After a few nice comments I’m going to have to finish it off and get it up somewhere I think. All I really need to do is clean up the bookmarklet code (which was a hack in more ways than one) and add a bit of sanity checking to data entered into the system.

A massive well done to everyone involved is in order as well, especially James who I remember talking to the idea about ages ago. Congratulations all - and hopefully be back next year.

Content to Markup ratio bookmarklet

Stoyan Stefanov just released an excellent little bookmarklet to calculate a content to markup ratio

It’s interesting browsing around a few sites and comparing ratios:

App Engine for Python Developers

I gave a short presentation about App Engine last night at the Cambridge Python User Group and as always it’s available on Slideshare.

I find App Engine a great way of just writing (Python) code and putting it online quickly, and the presentation was aimed at being something of a fanboy ode. During questions afterwards we talked about some of the limitations and cases where App Engine isn’t as well suited, and at least a few people were heading off planning on having a play.

The event was good fun and I picked up a few Python related tips and tricks from the other presentations and general conversation. Thanks for everyone who organised and attended the event and to RealVNC for hosting the talks.

Quick dmigrations Update

I’ve finally been getting round to doing a bit of work on dmigrations, the Python based database migration tool we developed at work.

dmigrations offers a simple but flexible way of managing changes to the database in your Django projects. It is a replacement for Django’s built in syncdb command.

I still have some work to do to merge in a number of modifications we’ve been running internally for a while but I’ve triaged the issues list that had been left alone for too long and merged in a fantastic collection of patches from a few people including Alexander Robbins and Chris Lamb. Thanks hugely to everyone for these.

Everything is available in trunk in SVN at the moment ahead of me cleaning a few thinks up and updating the download version. I’ve added a file as well ahead of hopefully adding it to the Python Package Index.

I obviously wasn’t the only one who had decided to do some work on the project in the last couple of days. msaelices has just submitted a patch which adds a PostgreSQL backend. I don’t tend to use PostgreSQL personally so if anyone does want to have a look at running dmigrations with the patch I’d love to hear about it.

HTTP Debugging Server in Django

I’ve been busy building and playing with various HTTP clients recently, mainly due to more playing with RESTful web services. I took a couple of hours out to build something to make my life easier - namely a very simple logging HTTP server in Django.

All the application does is accepts HTTP requests and log the results to a file. I’ve been using it to make sure the requests I’m sending from elsewhere are correct, before pointing the client at a web service that actually does something useful. It supports POST, GET, PUT, DELETE, etc. So far so simple.

screenshot showing us tailing the logs

I ended up using Django mainly because I did most of the work on the train, and I know my way around it pretty well by now. I did originally thing it might have been better to use something simpler but I did end up learning a few new tricks that I’ll use for future projects.

  • In theory Django doesn’t need a database. While this is true for the most part, the in-built test runner does require a database. One thing I’ve talked about before is the usefulness of creating your own custom test runner. If anyone else runs into this specific problem have a look at the file.
  • The logging is handled via Python’s inbuilt logging module. It has some nifty features (like log rotation, maximum file sizes and different logging levels) but is a little fiddly to get working correctly. Tales abound of log messages appearing multiple times, or permission errors causing problems. At least for this project I think I have everything working. I’ll try using this module on a couple of other projects and maybe spin it off into a small reusable app.

For something that took only a few hours to write it was both good fun and a useful learning tool. And Django proved itself more useful that I though for smaller services like this.

Startup Hubs in the UK - Where To Put One?

I always enjoy reading things by Paul Graham, even if I’d like to be able to disagree with his conclusions on occasion. It’s not just that I love Hackers and Painters, but also he often says things like the following that make me smile:

hackers are much more constrained by gentlemen’s agreements than regulations. If they shake your hand on a promise, they’ll keep it. But show them a lock and their first thought is how to pick it.

His latest short essay concerns startup hubs again, something that I’m actually pretty interested in, end up talking to people about, but hardly ever writing anything on. Mike over at Techcrunch Uk elicited some good comments last year with a post on the subject. As usual Graham focuses on the US, but what about translating his idea of buying a Startup Hub and applying it to the UK? He sets several criteria for success:

  • Do you have good weather?
  • Do people live downtown, or have they abandoned the center for the suburbs?
  • Would the city be described as “hip” and “tolerant,” or as reflecting “traditional values?”
  • Are there good universities nearby? Are there walkable neighborhoods?
  • Would nerds feel at home?

Lets assume for the moment that the idea would work. Take £15-20million and invest in 30 or so well chosen startups on the condition they move into town. The question is then: where in the UK would this be most likely to work? And are their any UK specific criteria that would increase the success rate?

Some people might be thinking “but no UK city is going to invest £20million in a scheme to encourage startups”. Leeds is doing just that. Or rather Yorkshire Forward are putting up £35million along with Leeds University putting up £31milion. What are they doing with all that money? They are going to build a big building with it - which is exactly what Graham says won’t work.

GitHub Changelog

It turns out the guys at GitHub publish the commit messages from their work on GitHub itself. If you really have to keep up you can even subscribe to an Atom feed.

OK, so this might not be hugely useful. But it is funny, and a just a nice sign of a small company being open in an interesting way. A few of my favourites:

mmm pizza… guard against bad SHAs in Walker#commit

really need to get a handle on this insane testing setup

I guess ff3 linux is the worst browser ever made

dont care

Epydoc Ant task

I quite like Epydoc for generating Python API documentation, even if the interface looks a little dated and could do with a lick of paint.

For most project I use an Ant build script to generate documentation when needed. You could wrap the basic commands in a make file or a bash script if you prefer that sort of thing though. The only trick is to make sure you have everything you need on your Python Path. In Django projects, or App Engine projects, you’ll probably find the default runner script plays with the path somewhat. The following example adds the current directory to the path along with an ext directory where I stash external modules, which should see it run cleanly without any import errors.


Example of using XMPP on App Engine (via IMified)

As I mentioned before, App Engine is getting an XMPP API at some point soon. But if you just can’t wait to start adding IM interfaces to your applications then you can do it now, by using a nifty third party in IMified.

IMified provide an incredibly simple HTTP API for interacting with your own IM bot. If we want to be buzz word compliant we might even call it a webhook. It’s also currently a free service while they work through the beta. The documentation is short and to the point but only contains examples in PHP. It supports multiple step conversations as well as authentication.

So, armed with a little time on the train over the last few days I got to work knocking together a quick demo application as a proof of concept. You can find the site on and if you want to chat with the bot you can add [email protected] to your contacts. The bot uses the Jabber protocol so is available over Jabber or GTalk. IMified make it easy to use MSN or Yahoo IM accounts as well, which is something the App Engine API might very well not do I would imagine.

Screengrab of the IMified App Engine site

As always you can find the code on GitHub. Most of the code is actually just the site itself or settings to make local development easier. The following is a slightly edited version of the live code (logging and caching removed to make it easier to follow). All we need to do is accept a HTTP Post request with a list of arguments and return a plain text response. All being well the response is sent as a IM message to the sender.

pre. class IMified(webapp.RequestHandler): “This is the endpoint for the message from IMified” def post(self): “We recieve post data from IMified” userkey = self.request.get(‘userkey’) network = self.request.get(‘network’) msg = self.request.get(‘msg’) step = self.request.get(‘step’) try: # we try and create the message message = Message( userkey = userkey, network = network, msg = msg, step = int(step) ) message.put() # the response is send as an IM message to the sender self.response.out.write(‘Message saved’) except: self.response.out.write(‘An error ocured, message not saved’)

IMified can obviously be used outside App Engine as well, and in fact it’s not just about working around limitations in existing systems. Running the long running processes required for bots, and potentially even running your own XMPP server, is fiddly at times and requires at least some setup, monitoring and configuration to get working. Not having that as a barrier for entry for simple experiments or applications is a good thing.

Aral spoke at the last DJUGL about App Engine and mentioned a wide range of third party services that you can use to get around current limitations. IMified definitely fits into this group of support services very nicely indeed. I’d love to see them do really well as it really makes it much easier to get started with XMPP applications, even if what you can do is limited to a few simple APIs. I’d love to hear about other services that people are using in this way to build these distributed applications.

Python REST Client

I’m working on a small project involving using RESTful APIs and wanted a simple HTTP client, something that sat a little higher in the stack than httplib2 or similar. I turned initially to the Django Test Client which now supports all the required methods but it turned out that I’d have to unpick it from django a little.

With a little bit of looking around I found the python-rest-client which certainly sounded like it would do what I wanted. It lets you make HTTP requests in as straight forward a manner as possible and fitted the bill perfectly.

pre. from restful_lib import Connection conn = Connection(“") response = conn.request_get(“/”)

It supports authentication, nice helper methods and gives you the response in a nice format.

pre. conn = Connection(“", username=“XXX”, password=“XXX”) conn.request_delete(‘/items/11232344’)

As an added benefit it also comes with a Google App Engine compatible version.