APIs in 2009 - XMPP and WebHooks

Everyone has to have a post with a year in the title at the start of the year so I thought I better write something. Rather than one of those personal retrospective emails I thought I’d go for something different - a look at what I’d like to see in APIs in the coming year.

Webhooks

I’m pretty interested in the idea of applications exposing Webhooks at the moment. It’s a pretty simple idea. As a user of a service you can register your own HTTP end points to receive information whenever events occur in the service. Both Shopify and GiHub have pretty nifty hooks for extending their capabilities for instance. When someone pushes code to a git repository you could send a ping to trigger a process to update the documentation for example.

With the rise of hosted application development environments like AppEngine writing small, nearly throwaway, apps to subscribe to these hooks could become incredibly powerful. It’s a lot like how unix programmers think, by piping lots of small applications together to get to the expected end result. It’s not a replacement for the more standard read/write API, but it’s a potential solution to the need to constantly poll that API for some types of applications.

The idea of public webhooks would be hugely powerful, but would also likely be a scalability nightmare. Imagine if Flickr exposed a hook that you could subscribe to whenever anyone added a public photo. Or Twitter added a hook for when anyone tweeted. These sorts of hooks would quickly be swamped with subscribers. The number of HTTP requests being sent by a service like Flickr under these circumstances would be rather large to say the least. Which is where another technology that’s designed for this sort of problem becomes useful.

XMPP Interfaces

XMPP or the Extensible Messaging and Presence Protocol has been around for a while, although originally under the name Jabber. It’s predominantly being used at the moment as a instant messaging protocol, but in reality it’s far more general purpose than that. Or rather, IM is generally considered to be between two or more people. But their is no reason that either or all the participants in an XMPP session can’t be programs.

On my local machine I’m a big fan of the command line for all sorts of simple, and sometimes complex, tasks. If applications expose their APIs via an XMPP bot then you basically have a ready made command line interface to online services via your IM client of choice. Combined with a solution to the public webhook problem mentioned above and you can hopefully see why I find XMPP pretty interesting.

I’ve been playing with writing XMPP bots recently, both at our internal hackday and at the recent Last.fm event. Their are various libraries and code examples lying around the internet that make getting started easy enough. With services like Imified getting setup last year I’d imagine it will get easier still this year.

As for what I’m going to be up to along these lines, we’ll have to wait and see. I have a few pet projects that I’d like to get off the ground which might be good test beds for this sort of thing. Apart from that I’d like to write some getting started style tutorials on some of the technologies involved or maybe a full blown article on the theory if I can find somewhere to publish it. A barcamp presentation or two might also fall off the back if I do get the time to play around a little more. But that’s just me. If you’re building, or planning on developing, an API for a product at the moment I’d suggest having a look at these two areas. They might turn out to just be potential extensions to what you had planned. Or they might turn out to be just the right approach to the problems you’ll face getting people to use your API.

Local Continuous Integration with Integrity

Integrity is ace. I’m a huge fan of working under the ever watchful eye of a Continuous Integration server. I’m also becoming more and more of a fan of Git, and GitHub, for my personal projects. At work we run CruiseControl and it does it’s job well, but locally I only use it for larger projects. It comes with a little overhead and if I’m just hacking on the train I rarely check on it’s status.

Integrity is an unashamedly lightweight and straightforward continuous integration server written in Ruby. It comes with Git integration as well as a nifty notifications framework. There are already notification plugins available for jabber, IRC and email. If I get time and inclination I’d love to hack together a webhooks plugin too. It’s a simple app to get up and running with and you can always browse the code if something isn’t clear. All in all it’s perfect for my type of smaller project.

So, with a local CI server up and running you’re left with one problem; having to click the _Manual Build” button whenever you want a build. Now Integrity comes with a mechanism to allow pushes to GitHub to trigger a build. But this only works when you have an internet connection and are using GitHub and are pushing frequently. Personally I often make lots of local commits and then push at a later date. Also not all of my projects are on GitHub for various reasons.

Well it turns out that all the build button does is sent an empty HTTPpost request to a URL of the following format:

http://{integrity-url}:8910/{project-name}/builds

That means with a little bit of Git magic we can have our integration server run a new build whenever we commit our code. All we need to do is write a very simple post-commit git hook script. I’ve written the script in Python but you could write it in anything. This script is from a real project so adjust the server address and path as needed.

#! /usr/bin/env python
import httplib
conn = httplib.HTTPConnection("localhost:8910")
conn.request("POST", "/appengine-books/builds")

All you need to do is drop this script in your .git/hooks folder as post-commit. Make sure you set the executable bit with chmod +x as well, otherwise you’ll be wondering why it’s not working and probably blaming me.

Web developers and Tools Programmers

I’m pretty interested in computer games. Building them represented a big technical challenge and with that comes interesting parallels with larger web projects. Andy Budd has talked about User Experience learnings from games previously, and I’ve heard Aleks Krotoski talk about similar themes, in particular the design of social systems and user generated content. What I’m interested in however is tools programmers.

Computer game development teams generally have a decent number of people solely dedicated to building and maintaining tools. They aren’t working on a specific title, or just on maintaining existing systems, but on programming tools for other developers in the company. Now I’m not sure how much sharing of these tools goes on between the different computer games developers, but from a cursory look around I couldn’t find any examples.

Tools programming in web development teams seems to be a different kettle of fish. I’ve never seen a job position advertised specifically for tools work, nor do I know anyone who would describe themselves as such. I am however seeing more and more of these sorts of tools make their way out into the world recently thanks to GitHub. Build systems and scripts, documentation systems, testing harnesses, linting tools, etc. All can be found if you look closely. So it’s definitely not that we don’t build these tools, though it might be that we don’t do it as part of the 9 to 5.

One of the differences that might cause this difference between games developers and web developers is scale. For the most part games development teams are bigger than your average web team or agency development team. But not always and as I said I’ve never, not once seen a web tools job going. The other reason might be Open Source. Speaking from the web side of the fence their are lots of great open source tools for building web sites and applications. If your needs are pretty basic you should be able to get away with stringing some of these together, but you’ll probably still need to do the scripting to do so. Maybe this is the same for games development; I’m afraid I don’t know.

Maybe the other reason is that as web developers we all also do a little bit of tools programming? From personal experience this is certainly true, but then I like and think about this sort of stuff more than most anyway.

So a few questions for anyone also interested in this sort of thing:

  • Do you or your colleagues do any tools development at work?
  • Is that work just done as part of other work or is it defined separately?
  • If you don’t do any tools work per se do you string available tools together using basic scripting?
  • Or do you just stick with as few tools as possible in order to make live easier?

I also have a feeling some communities are happier knocking out little tools than others. The Ruby community in particular seems busy at the moment and their is a great deal of good stuff in Java.

One last thought. How do you go about finding new tools that might work nicely in a web development world? I read far too much of the internet on a daily basis and I still miss interesting stuff more often than not. I only came across Integrity yesterday for instance.

Search for Genres on Spotify

Spotify is great. It seems stable, the desktop interface is simple and straight forward, and it has an entire album of emo bands doing covers of other emo bands. One thing that’s not quite clear just yet in the interface however is how to search for all the Rock tracks, or more specifically how to search for all the tracks in a given genre.

Turns out it’s quite simple. Just do a search for genre:Rock. My guess is there are various other textual shortcuts hidden in that search box. If anyone knows of any others post a comment.

Gmemsess

I’ve been looking at different ways of using simple sessions on App Engine, in particular for one shot flash messaging after redirects and the like. Their are some issues with Cookie support at the moment but Gmemsess solved my problem perfectly.

gmemsess is a secure lightweight memcache-backed session class for Google appengine.

Where will all the web developers go?

I’ve been thinking recently about what happens when we all get older. Now, I’m not actually referring to everyone here but more specifically what I’m going to call second generation web developers. I don’t mean Tim and friends here, or the early entrepreneurs of Yahoo! and Netscape. I mean the people who came along when the commercial web design and development industry had settled down a little bit, lets say 10 years ago. People like me.

Work seems to exist in lots of places; big in-house teams, small in-house teams, agencies, startups and freelancers. Where the jobs are at any given point seems to be tied to economic conditions and location. Most people I seem to speak to have tended to work in one of these areas, but I’ve not really got anything but vague memories to back that accusation up. I’m a little odd here in having previously worked mainly for agencies followed by a stint of freelancing, and now work for a decent sized in-house team at Global Radio.

As I see it the industry rules are predominantly made up by people involved as they go along. Their is no real impetus behind any trade body that I’m aware of, no real understanding within education and no consensus on organisational structure and jobs within it. That means, unlike accountants or people in many other professions, we’ve very little idea about what will happen in the long term.

So. What happens in 10-20 years time to the now quite large number of professional web developers.

Do we all just do the same thing we’re doing now. Just with higher version numbers? The problem with this is maintaining the challenge. If the core problems remain the same will it be much fun? I fear their are only so many times you can learn a new programming language and then solve the same problem you worked on at your last job before the world catches on.

Do we all become managers? The problem then is who do we manage? If the industry just gets bigger and bigger this works. But that sounds unreasonable. So if their are a limited number of managerial positions who gets them? The people their first?

Is their another industry that will have us? Computer games are starting to look to the web for ideas about community and collaboration, pretty much at the same time as the web is looking at games for thoughts on experience and engagement. But long term I can’t see a mass exodus or a huge cross over of people, just a huge overlap of ideas.

Do some of the areas like agencies, in-house teams, etc. disappear, or at least employ less skilled people? Experience costs money, and not always because it’s worth the extra expense.

Or is it simply that the world in twnety years time will be so different to now that we don’t really have a clue. And that a large group of computer savvy problem solvers will find something to do?

The real question I guess is what does the demand for skilled professional web developers look like in ten to twenty years? In reality most of the people currently doing the job won’t be getting to retirement age any time soon. That means every newly trained graduate or kid with a computer getting their first job adds to the size of the workforce. And how big do we think that workforce can get before it reaches a nice equilibrium? Certainly in London at the moment their are jobs aplenty. Many good friends have just left Yahoo! rather promptly and I’d be worried for them if I didn’t know how many people will be knocking down their doors.

I’d love it if their were numbers somewhere on this that you could graph. Maybe spot a plateau coming. It’s the sort of thing the Institute of Physics or the Chartered Institute of Marketing do. But as I said, we don’t have anything similar.

So maybe their is another option when the workforce has expanded as much as it’s going to do. We could get all work for The Institute of Web Development.

App Engine Samples

It’s probably old hat to those who have been using Google Appengine for a while but I just found some sample apps on Google Code. I prefer learning from actual code like this, at least until the App Engine books make it out next year.

Spyder

Stuart at work has been playing with a nice Python web crawler recently. I’ve used Harvestman before but it’s not the most straightforward thing to work with. Spyder has a really nifty callback based approach and a couple of hooks which allows you to write code like:

pre. crawler = Spyder() crawler.register(my_custom_method, ‘post_fetch_html’) crawler.run(URL)

On a side note I wish Launchpad was as clean and tidy as GitHub though. I can see GitHub adding some of the features that Launchpad has eventually, but I hope they fit them in around the existing features.

CSS Test on GitHub

I upload the work I’ve done so far on testing CSS. It’s still work in progress obviously but if you’re interested do let me know. I’ve stared with the image approach but will hopefully have something up demonstrating the selenium/rendered DOM position approach as well.

I Love GitHub Two

I’d been meaning to write a quick article about GitHub, but then Mike and Neil beat me to it and stole most of the best bits. Read both of those articles then come back if you want. I agree whole heartedly.

Now I’ve used public hosted source control before. But Google Code, Sourceforge or Launchpad never felt like this. Their were always their for people to download your code if they wanted. Maybe you collaborated with a few people on a specific project. But GitHub, through a combination of neat visualisations and social features, is encouraging people to just upload all their code. All those lines of throw away scripts. All that code written on the train learning a new language. All that configuration stuff that normally gets missed.

Talk of emerging programming languages often shifts to the need for a killer application. I want to learn more about Git so I can do more with GitHub. For me, that makes GitHub a likely driving force behind a sea of Git adoption in the coming year or so. I’m also interested to see what comes out of a reasonable sized network of like-minded people absentmindedly hacking away. Already I’m seeing people branch other peoples code within my network of contacts. Often just small tweaks and changes - like a code fairy cleaning up a few rough edges or adding an extra line here.

So, if you write code head over to GitHub and signup. Create a few repositories of the things you’re hacking on at the moment and jump through the pre-portable-social-network dance just one more time. I promise it’s worth it this time.