Django Testing Presentation at DJUGL

The second Django User Group (DJUGL) meeting was last night and was a great success as far as I could tell. Our boardroom at work was chock full of budding Django developers and interested parties - 70 people or so in all. Good work by Rob to get everything back on track at the start of the year and the next event is already planned for the back end of February so I hear. More news soon on the mailing list hopefully.

Rather unfortunately Simon had to pull out at the last minute and I stepped in to do a quick 20 minutes or so talk on writing tests for your Django applications.

I was rather surprised that only about a third of the people coming along were using Django in some work related capacity, and when I asked who was writing tests for their code a few more hands went down. Very few of the people not using Django at work are writing tests which is unfortunate (and contrasts rather interestingly with my experiences of the Ruby community).

For a presentation done in less than a day while at work I felt it went OK. testing is a hard sell at the best of times and trying to talk to both those without experience of writing tests and at the same time trying to get in a few of the more interesting things we’ve done at work was maybe a bit of an ask.

It not being a blog post round here these days unless I include a link to GitHub I did mention a small project last night in the talk - Django Test Extensions. It’s more just a collection of useful common testing code: additional assertions, custom test runners, etc. Thanks to Ross for the last minute patch with a couple of bug fixes as well.

The other two presentations went great. Andrew talking about the South migrations system and Aral talking about real world Google App Engine were both up my street of things I’m interested in. We have video of all the talks as well so hopefully that will make it out into the wild at some point once it’s been edited.

A Simple Python Jabber Bot

I’ve been knocking together various little instant messaging bots recently, partly as a way to play around with XMPP. As well as using the low level xmppy and XMPP4R-Simple libraries I’ve been having lots of fun with the JabberBot framework.

Jabberbot lets you write simple bots incredibly quickly, using simple conventions to determine what commands the bot exposes. It’s easier to explain with a simple example. The following bot lets you send the command time to it and returns the current time on the server on which the bot is running. The magic is in the name of the method bot_time. Any methods that start bot_ are automatically exposed as commands for the bot to accept.

You’ll need an xmpp server for the bot to connect to but you could always just register an additional google account and use it over gtalk if you wanted to. Personally I’m running ejabberd on a local Ubuntu VM as well for testing.

pre. from jabberbot import JabberBot import datetime class SystemInfoJabberBot(JabberBot): def bot_time( self, mess, args): “”“Displays current server time”“” return str(datetime.datetime.now()) username = ‘[email protected]’ password = ‘my-password’ bot = SystemInfoJabberBot(username,password) bot.serve_forever()

The JabberBot site has a few more examples as well with fancier features. At the recent Last.fm hackday I spent a bit of time knocking together a bot which talks to the Last.fm API (using the PyLast library for the API backwards and forwards.) I did this mainly as a demonstration of how simple it can be to create a useful command line interface to your API using an instant messaging client.

The code for LastBot is on GitHub. It has a few limitations and doesn’t intent to cover anywhere near all the API. When up and running you should be able to talk to it with your IM client. Simply send the user specified in the settings file a message like so:

pre. search {query}

So if you wanted to search for “astley” you would type:

pre. search astley

Which would probably give you:

pre. you probably mean Never Gonna Give You Up by Rick Astley http://www.last.fm/music/Rick_Astley/_/Never_Gonna_Give_You_Up Richard Paul Astley (born February 6, 1966) is an English dance-pop singer, songwriter and musician. He was born in Newton-le-Willows, St Helens, Lancashire, England. Astley currently resides in Richmond, Surrey with his Danish girlfriend, Lene Bausager, and their daughter, Emilie. In 1985, Astley was playing the club circuit as a singer with a soul band named FBI, when he was seen by the record producer Pete Waterman and persuaded to come to London to work at the PWL recording studio.

If the first result that comes back isn’t the one you wanted you can ask for the next result by simply sending next in another message. You can use prev as well to come back through the set.

pre. next

You can always send it a call for help at any time which should return the instructions to you via an IM message.

pre. help

Which should return something like:

pre. Personal LastFM search bot. Useful for looking for tracks you can’t remember the full name of. Or for findind out who sung a track. search: do a search for tracks prev: get details about the previous track in the list next: get details about the next track in the list

More instructions, and the code behind the bot, can be found in the README.

As developers we spend a lot of time using command line interfaces - mainly for speed and because you can cram a lot of functionality into a small amount of screen real estate. We’re increasingly spending time debugging API calls as well and exposing your API calls for use by instant messaging clients has the potential to make development easier. Think of the Python interactive shell or of IRB, but for APIs.

Are XMPP and offline processing coming to Google App Engine?

Google just announced that, once the port of Jaiku to App Engine is complete they will be Open Sourcing the code and stopping official development. I only used Jaiku sporadically in the heady days when everyone had to sign up for a new web 2.0 service every week or be mocked by their colleagues. What really interests me about this move though is what it means for App Engine.

If memory serves Jaiku had an instant message interface. Does that mean App Engine is going to get an XMPP interface?

I’m also presuming the original Jaiku application had at least some features powered by offline processing or used a message based architecture behind the scenes? Or used scheduled batch jobs for reporting or data mungling?

All of these issues are high up on the App Engine issues list and solutions to these two outstanding problems would move App Engine, in my opinion, to being very cool to being very useful. I’ve been impressed with the simple, well designed and well documented current set of APIs and I’d love to see a few more added for XMPP and offline processing. If the Jaiku code is open source but relies on Google only APIs it would seem a little odd.

So does anyone have any more information on this? Any passing Googlers who can get us an answer?

Localbuilder on GitHub

One of several little projects I have up and running on GitHub at present is LocalBuilder. It’s a pretty simple little script which watches for changes in a given directory and when they occur runs a given command. I knocked it together to use to trigger the running of a test suite each time I save files in a project. It’s written in Python but you could use it to run commands in any language you like.

It’s all pretty simple Python really and is hopefully reasonably commented and tested so if you’re interested you can follow along with the code. If you just want to use the damn thing then:

<code>./localbuilder --path /path/to/watch --command /command/to/execute</code>

It turns out Ruby has a much nicer tool to do this and more in autotest, part of the ZenTest suite of testing tools. I’m finding more and more very nice bits of code written in Ruby of late, but that’s probably a whole different blog post.

Speaking at Bamboo Juice

It’s the start of another year so that means it’s time to start thinking about conferences.

I just found out at the end of last week (via Twitter no less) that I’ve been confirmed as one of the speakers for Bamboo Juice. Thanks to Jon and Rich for having me. I’m really looking forward to that as it’s a new event outside London (at the rather fabulous looking Eden project in Cornwall), which should hopefully mean a chance for a new set of people to go along to what looks like a decent event. I’ll have more information about what I’m going to be talking about some time between then and now I would imagine, but the emphasis is on practical learnings for working developers and designers.

Apart from that the only other conference I’ve confirmed I’l be attending so far is Thinking Digital in May. It was great last year even if I spent the majority of the time running round organising barcamp, moving house or leaving Newcastle. This time I’m not involved in any organising (sorry Andy) so I plan on making the most of the talks and general experience. Rumour has it that their will be another barcamp as well so I’ll look forward to that too. It will be good to get back up to Newcastle and see people again.

I’m also looking forward to what Patrick is up to for @media. It was the conference that started this whole thing rolling in the UK back in the day and now it appears it might be set for a change.

Time to finalise the line-up and web site for a very different (and lower price!) @media ‘09

As well as these three I’ll be keeping my eye out for any news about Xtech, The Highland Fling and any other new upcoming conferences. If you know of anything else leave a comment. I didn’t find out about Think Visibility until it had sold out but I like the idea of smaller, specialised events.

Overall it already looks like being a good year for conference goers in the UK. I’ll be on the lookout for barcamp style events as well but I do like a good formal conference as well. Hopefully see a few people at some of these events?

APIs in 2009 - XMPP and WebHooks

Everyone has to have a post with a year in the title at the start of the year so I thought I better write something. Rather than one of those personal retrospective emails I thought I’d go for something different - a look at what I’d like to see in APIs in the coming year.

Webhooks

I’m pretty interested in the idea of applications exposing Webhooks at the moment. It’s a pretty simple idea. As a user of a service you can register your own HTTP end points to receive information whenever events occur in the service. Both Shopify and GiHub have pretty nifty hooks for extending their capabilities for instance. When someone pushes code to a git repository you could send a ping to trigger a process to update the documentation for example.

With the rise of hosted application development environments like AppEngine writing small, nearly throwaway, apps to subscribe to these hooks could become incredibly powerful. It’s a lot like how unix programmers think, by piping lots of small applications together to get to the expected end result. It’s not a replacement for the more standard read/write API, but it’s a potential solution to the need to constantly poll that API for some types of applications.

The idea of public webhooks would be hugely powerful, but would also likely be a scalability nightmare. Imagine if Flickr exposed a hook that you could subscribe to whenever anyone added a public photo. Or Twitter added a hook for when anyone tweeted. These sorts of hooks would quickly be swamped with subscribers. The number of HTTP requests being sent by a service like Flickr under these circumstances would be rather large to say the least. Which is where another technology that’s designed for this sort of problem becomes useful.

XMPP Interfaces

XMPP or the Extensible Messaging and Presence Protocol has been around for a while, although originally under the name Jabber. It’s predominantly being used at the moment as a instant messaging protocol, but in reality it’s far more general purpose than that. Or rather, IM is generally considered to be between two or more people. But their is no reason that either or all the participants in an XMPP session can’t be programs.

On my local machine I’m a big fan of the command line for all sorts of simple, and sometimes complex, tasks. If applications expose their APIs via an XMPP bot then you basically have a ready made command line interface to online services via your IM client of choice. Combined with a solution to the public webhook problem mentioned above and you can hopefully see why I find XMPP pretty interesting.

I’ve been playing with writing XMPP bots recently, both at our internal hackday and at the recent Last.fm event. Their are various libraries and code examples lying around the internet that make getting started easy enough. With services like Imified getting setup last year I’d imagine it will get easier still this year.

As for what I’m going to be up to along these lines, we’ll have to wait and see. I have a few pet projects that I’d like to get off the ground which might be good test beds for this sort of thing. Apart from that I’d like to write some getting started style tutorials on some of the technologies involved or maybe a full blown article on the theory if I can find somewhere to publish it. A barcamp presentation or two might also fall off the back if I do get the time to play around a little more. But that’s just me. If you’re building, or planning on developing, an API for a product at the moment I’d suggest having a look at these two areas. They might turn out to just be potential extensions to what you had planned. Or they might turn out to be just the right approach to the problems you’ll face getting people to use your API.

Local Continuous Integration with Integrity

Integrity is ace. I’m a huge fan of working under the ever watchful eye of a Continuous Integration server. I’m also becoming more and more of a fan of Git, and GitHub, for my personal projects. At work we run CruiseControl and it does it’s job well, but locally I only use it for larger projects. It comes with a little overhead and if I’m just hacking on the train I rarely check on it’s status.

Integrity is an unashamedly lightweight and straightforward continuous integration server written in Ruby. It comes with Git integration as well as a nifty notifications framework. There are already notification plugins available for jabber, IRC and email. If I get time and inclination I’d love to hack together a webhooks plugin too. It’s a simple app to get up and running with and you can always browse the code if something isn’t clear. All in all it’s perfect for my type of smaller project.

So, with a local CI server up and running you’re left with one problem; having to click the _Manual Build” button whenever you want a build. Now Integrity comes with a mechanism to allow pushes to GitHub to trigger a build. But this only works when you have an internet connection and are using GitHub and are pushing frequently. Personally I often make lots of local commits and then push at a later date. Also not all of my projects are on GitHub for various reasons.

Well it turns out that all the build button does is sent an empty HTTPpost request to a URL of the following format:

http://{integrity-url}:8910/{project-name}/builds

That means with a little bit of Git magic we can have our integration server run a new build whenever we commit our code. All we need to do is write a very simple post-commit git hook script. I’ve written the script in Python but you could write it in anything. This script is from a real project so adjust the server address and path as needed.

#! /usr/bin/env python
import httplib
conn = httplib.HTTPConnection("localhost:8910")
conn.request("POST", "/appengine-books/builds")

All you need to do is drop this script in your .git/hooks folder as post-commit. Make sure you set the executable bit with chmod +x as well, otherwise you’ll be wondering why it’s not working and probably blaming me.

Web developers and Tools Programmers

I’m pretty interested in computer games. Building them represented a big technical challenge and with that comes interesting parallels with larger web projects. Andy Budd has talked about User Experience learnings from games previously, and I’ve heard Aleks Krotoski talk about similar themes, in particular the design of social systems and user generated content. What I’m interested in however is tools programmers.

Computer game development teams generally have a decent number of people solely dedicated to building and maintaining tools. They aren’t working on a specific title, or just on maintaining existing systems, but on programming tools for other developers in the company. Now I’m not sure how much sharing of these tools goes on between the different computer games developers, but from a cursory look around I couldn’t find any examples.

Tools programming in web development teams seems to be a different kettle of fish. I’ve never seen a job position advertised specifically for tools work, nor do I know anyone who would describe themselves as such. I am however seeing more and more of these sorts of tools make their way out into the world recently thanks to GitHub. Build systems and scripts, documentation systems, testing harnesses, linting tools, etc. All can be found if you look closely. So it’s definitely not that we don’t build these tools, though it might be that we don’t do it as part of the 9 to 5.

One of the differences that might cause this difference between games developers and web developers is scale. For the most part games development teams are bigger than your average web team or agency development team. But not always and as I said I’ve never, not once seen a web tools job going. The other reason might be Open Source. Speaking from the web side of the fence their are lots of great open source tools for building web sites and applications. If your needs are pretty basic you should be able to get away with stringing some of these together, but you’ll probably still need to do the scripting to do so. Maybe this is the same for games development; I’m afraid I don’t know.

Maybe the other reason is that as web developers we all also do a little bit of tools programming? From personal experience this is certainly true, but then I like and think about this sort of stuff more than most anyway.

So a few questions for anyone also interested in this sort of thing:

  • Do you or your colleagues do any tools development at work?
  • Is that work just done as part of other work or is it defined separately?
  • If you don’t do any tools work per se do you string available tools together using basic scripting?
  • Or do you just stick with as few tools as possible in order to make live easier?

I also have a feeling some communities are happier knocking out little tools than others. The Ruby community in particular seems busy at the moment and their is a great deal of good stuff in Java.

One last thought. How do you go about finding new tools that might work nicely in a web development world? I read far too much of the internet on a daily basis and I still miss interesting stuff more often than not. I only came across Integrity yesterday for instance.

Search for Genres on Spotify

Spotify is great. It seems stable, the desktop interface is simple and straight forward, and it has an entire album of emo bands doing covers of other emo bands. One thing that’s not quite clear just yet in the interface however is how to search for all the Rock tracks, or more specifically how to search for all the tracks in a given genre.

Turns out it’s quite simple. Just do a search for genre:Rock. My guess is there are various other textual shortcuts hidden in that search box. If anyone knows of any others post a comment.

Gmemsess

I’ve been looking at different ways of using simple sessions on App Engine, in particular for one shot flash messaging after redirects and the like. Their are some issues with Cookie support at the moment but Gmemsess solved my problem perfectly.

gmemsess is a secure lightweight memcache-backed session class for Google appengine.