Imified

Imified looks like an interesting way of getting started with using instant messaging bots in your applications. (Via)

Using Python and Stompserver to Get Started With Message Queues

Message Queues are cool. It’s official. Now, banks and financial institutions have been using big Enterprise Java message systems for years. But it’s only really over the last year or two that the web community at large have got interested. Wonder what all the interest is in Erlang, Scala or Haskell? Distributed systems and a lack of shared state - hopefully leading to some sort of scalability nirvana - that’s what.

Matt Biddulph of Dopplr has spoken at varying levels of technical detail on the subject over the last year or so. At barcamps and more recently at dconstruct. But you still don’t find that many people actually starting to use any of this stuff. Looking around the internet I couldn’t find that many examples of how to get started. Their are some pretty mature standards, good libraries, server interoperability, but few tutorials aimed at people who don’t know all about it.

The first problem is looking for a simple use case that most developers will have experienced problems with. The example I like to give is sending email. If you have a simple form on your site that sends email you probably just submit the request to the backend, it sends the email and then renders the success page back to the user. The problem here comes with scale. How many connections can your mailserver sustain? How many emails can you send from it before you start looking like you’ve been turned into a spam factory? At what point does the time taken for the mail server to respond to the web server cause the web server to time out or respond so slowly the user left or pressed refresh? If you’re sending lots of emails you need to think about this sort of stuff. For your average site this might not be a problem, but for the newer breed of applications or social networks this might bite you sooner than you think. You can gain more control over this process by introducing a message queue. Submitting the form simply adds a work task to the queue. A listener reads from the queue and sends the email. The advantage comes when you realise by removing the rendering of the page form the same process as sending the email you can throttle the system without affecting page rendering time.

So onto a simple working example. I’ve decided to use Python as that’s my language of choice at the moment. It’s also easy to read in a sudopseudo code sort of way. Writing these examples using equivalent libraries in Ruby or PHP should be straightforward enough. As for the message queue itself I’ve opted for stompserver which is available as a Ruby gem. So assuming you have Ruby and gem installed (good instructions for this on the Rails wiki) you can just run:

<code>sudo gem install stompserver</code>

Starting the queue is as simple as running:

<code>stompserver</code>

This will get you up and running quickly. Stompserver has a number of arguments you can pass in to use different ports or backends but for the purposes of getting started it’s enough to just run it. This ease of use is the thing I love about stompserver. ApacheMQ is something of a tricky beast to setup, though you might want to use that in a production environment.

So now we have the server up and running we can get on with talking to it. I used the Python stomp.py library to deal with the heavy lifting. All the other modules are in the standard library. Their are equivalents for PHP and Ruby available as well.

The first script is a listener. Its job is to listen for activity on the queue and then act upon it. You pass the script an argument of the name of the queue to listen to.

<code>./stomp_listen.py /queue/test</code>

This example simply prints the messages from the queue to the console, but in reality the on_message handler would be were you act upon the message received. In our email example above it would be were you parse out the email address, subject line and message and actually send the email.

Stompserver currently exposes a queue for monitoring the queue server at /queue/monitor. You can use this script to subscribe to that queue and get information about the current state of the server. It will tell you which queues currently have items in them and if these are currently being processed.

You can run multiple instances of this script subscribing to a single queue. This is one of the real advantage of message based systems, two listeners should clear a queue in half the time. This sort of horizontal scaling is hugely useful as you grow a site or application.

pre. #!/usr/bin/python import time import sys import logging import socket import stomp

  1. the stomp module uses logging so to stop it complaining
  2. we initialise the logger to log to the console logging.basicConfig()
  3. first argument is the que path queue = sys.argv[1]
  4. defaults for local stompserver instance hosts=[(‘localhost’, 61613)]
  5. we want the script to keep running def run_server(): while 1: time.sleep(20) class listener(object): “‘define the event handlers”’ # if we recieve an error from the server def on_error(self, headers, message): print ‘received an error s’ message # if we retrieve a message from the server def on_message(self, headers, message): print ‘received a message s’ message
  6. do we have a connection to the server? connected = False while not connected: # try and connect to the stomp server # sometimes this takes a few goes so we try until we succeed try: conn = stomp.Connection(host_and_ports=hosts) # register out event hander above conn.add_listener(listener()) conn.start() conn.connect() # subscribe to the names que conn.subscribe(destination=queue, ack=‘auto’) connected = True except socket.error: pass
  7. we have a connection so keep the script running if connected: run_server()

The second script allows us to send messages to the queue:

<code>./stomp_send.py /queue/test "test message 1"</code>

The script takes a couple of arguments, the first one is the name of the queue, the second is the message you want to send.

pre. #!/usr/bin/python import time import sys import logging import socket import stomp

  1. the stomp module uses logging so to stop it complaining
  2. we initialise the logger to log to the console logging.basicConfig()
  3. first argument is the queue queue = sys.argv[1]
  4. second argument is the message to send message = sys.argv[2]
  5. defaults for local stompserver instance hosts=[(‘localhost’, 61613)]
  6. do we have a connection to the server? connected = False while not connected: try: # connect to the stompserver conn = stomp.Connection(host_and_ports=hosts) conn.start() conn.connect() # send the message conn.send(message,destination=queue) # disconnect from the stomp server conn.disconnect() connected = True except socket.error: pass

Both these scripts are pretty simple examples. In the real world you would probably want to make them a little more robust and user friendly. Both could probably do with checking they have the relevant arguments and providing help information if you run them without. I’d also probably move the hosts into a config file as it’s currently hardcoded into the scripts. I’ve also not tested them with other stomp compatible servers like ApacheMQ. In theory they should work fine assuming stomp.py works as advertised.

Overall, it’s surprisingly easy to get started with message queues. If you’ve been hearing about the advantages of distributed message based architectures but assumed you had to be Matt Biddulph to use them, think again.

Django Powered

A short break from blogging ends with a new site design. As with all these things their will no doubt be a few kinks still to work out and I’ll be adding to the design a little over the coming months. The main reason for all this change? A move to a custom CMS build using Django. This was something of an excuse to play around with Django outside work and I’m pretty happy with the results. I have all the bits of wordpress I actually used, plus a few bits I didn’t have before. More importantly I have something I want to hack on. Wordpress is a kick ass blogging tool, but keeping it updated or adding new features never seemed much like fun.

It seems like it’s the week for Django related site launches in the UK. We released the new Capital Radio site on the world last week and Nat beat me by a day or so with her new Django powered site. Look for more in the future too I would wager.

I have a whole range of posts brewing about what I found out along the way, both building a personal blog and building a large site with a big team. Keeping up with the bleeding edge of Django ahead of the 1.0 release took some doing (I’m running the latest Trunk release here at the moment). Deciding to use a combination of Spawning and Nginx for serving is a nice break from Apache as well. But that is all for later when I have a little more time.

Of Hacking, Continuous Integration and Django

I’ve not written anything here for a good few weeks, my tweeting has slowed down some and I’m behind on my feed reading. I’m going to blame the new job and the daily commute from Cambridge to London I think. I’ve definitely not been any less busy that usual:

Hacking

We had an internal hackday on Thursday and Friday last week where lots of us over at GCap/Global downed tools and build cools stuff for a couple of days. This is exactly the sort of reason I took the job in London - for the opportunity to build interesting things quickly with other smart people. I got to play around with an event driven, music orientated, API hack and a more useful but less sexy documentation hack. Yes, I said documentation hack. I might be able to release the latter all being well but I need to finish it off first and kick the tyres on some internal projects

Django

As a development team we’re using Django for everything which is proving to be huge fun. It’s a decent size project and we’re pushing Django (and in particular new-forms admin) in interesting ways. Having worked previously with PHP, ASP.NET and Rails I’m loving lots of bits of Django. I mentioned the template system before but their are lots of other things to appreciate. Some of this just comes from working with people like Simon and Rob who know Django pretty well. Some of it just from being able to write Python every day.

Continuous Integration

When not working on Django, or writing HTML, CSS or Javascipt (now mainly using JQuery), I’ve been busy pushing the benefits of Continuous Integration. As someone who is actually pretty bad at writing unit tests I like the process of working with Cruise Control as a gatekeeper. I also like automation in general so have been busy with Ant scripts and some Twill scripts as well for more functional testing. Django’s test suite is pretty nifty and easy to use. The official documentation is a pretty good starting point but I’m still on the lookout for some more in depth best practices articles.

So, between writing lots of code I’ve had less time to write words other than internal documentation. I’m finding the change pretty refreshing at the moment but want to keep up with writing every now and again. Who knows how that will work out? I have a feeling that I might blog more geeky code related stuff but time will see how that plays out.

Where are the Rock Star Web Project Managers?

In maybe a more constructive manner than yesterday I started wondering where the rock star web project managers hang out? I think we’re all aware of something of a celebrity culture within web circles. Their are a hardcore of people who’s blogs, books and conference appearances we’ve all seen several times over. And in the main I think this has had a positive effect on everyone involved. People like Jeremy, Molly and Simon have at different times acted as pretty useful barometers and yard sticks for lots of people. But these people are invariably designers and developers - not product managers or project managers.

The only person I can think of who has talked a little about the topic of project management is Meri with her new book Principles of Project Management. The topic occasionally comes up in conversation, or is mentioned by the designers and developers noted above. And their are lots of blogs (more often by developers it seems) about Agile, XP and Scrumm. But what about the practice of web product management? You can point to countless blogs written by designers and developers at the likes of last.fm, flickr and Yahoo. But where are the managers?

So, my question to you is: do you know of any great web product or project managers that blog about the discipline?

@media: Designers Vs Developers

Another successful @media conference comes to a close and as usual interesting things were said and hopefully everyone learned something. As usual I have a few more things on my must find time to play with list. More on those if they happen.

But one part of the conference I felt needed addressing straight away was the first days panel. The Hot Topics panels bring together a few of the days speakers to answer questions posed by the audience. This year each day had it’s own panel discussion, with the first days session having a design theme. So far so good, and before I getting going I have to say I think Jeffrey Veen did a sterling job of prodding and prompting the session along. The rest of the panel composed of Andy Clarke, Dan Rubin, Bronwyn Jones and Indi Young.

The panel fielded a few interesting questions (and The Beatles in-joke was highly entertaining) but the whole thing took an odd turn part way through - odd in the sense that everything suddenly became very anti-developer.

Andy Clarke seemed to be the ring leader here saying things like (I’m paraphrasing here) “I hate daily stand-ups, we should do Agile”. Dan Rubin was involved too, seemingly saying all developers prefer the rigid nine to five because they’re logical people, while designers want to be able to work whenever they choose. This and other comments caused a few good friends and web developers on the front row to literally shake their fists in anger. The audience got involved too, cheering the anti-developer or anti-engineer statements on. In my mind damage had just been done to out our industry.

Buidling web sites or applications is a pretty multidisciplinary exercise, and in any team environment good communication is often the difference between executing well and failing badly. Conferences like @media are a great place to come together with people from different disciplines and similar jobs in different organisations and share stories. I’m not saying either Dan or Andy haven’t had bad experiences with developers or stand-ups. I’m not saying bad individuals or bad process don’t exist (I have scars from both). I’m saying that generalising these experiences and then promoting them to impressionable designers is dangerous.

I have the good fortune of working with a great designer, Alex Lee, at GCap at the moment. He’s involved in our stand-up meetings every morning and without him there we would waste untold amounts of time and effort. If he turned up on monday and said “I’m not going to come to stand-ups anymore, Andy Clarke says they aren’t cool”. We have a massive problem. My fear is this is exactly what’s going to happen somewhere tomorrow.

I’m something of a hybrid; I’ve run the whole gamut of design and development activities during my career to-date (I’ve even touched on project management) and have worked in small and medium sized agencies, as an independent freelancer and now in a decent sized in-house team. All of these roles pose different communication challenges which require different solutions. What works for three people in a distributed team doesn’t work for fifteen people working in-house. Some types and sizes of team work best with stand-ups, others work best being in the same small room together. Sometimes you need a centralised audit trail of everything that has happened on a project, other times it’s overkill. Somethings everyone has to be working on the project at the same time, at other times as long as tasks get done it really doesn’t matter.

I’m not saying I like all of those working conditions - I’m saying that if you understand where they work best you can work in the environment that suits you. Andy Clarke runs a boutique agency doing high quality work, working with a very small team where individual flexibility and minimal process is definitely the best approach. I now work in a cross disciplinary team of maybe sixty people and it’s a different ball game completely.

This isn’t an attack on Andy, or Dan, or even an anti-designer rant. The web is a young industry filled with young people. We’re learning as we go and stealing what we can from other disciplines. Also not that many of us really like the idea of project management, never mind the reality. Agile methods like Scrumm are hot at the moment but that’s not to say we won’t find something that better suites our discipline in the future. Iterative development, while often annoying to designers, appears to be producing better results. If designers, and developers, want to change how we work together for the better, then get interested in project management and lets have a discussion in public. If you want to complain about individual experiences then that’s why you have a blog.

Newcastle Web Week

The North East is stealing a march on London this week with all sorts going on for those who like a real world get together. Although not quite as busy as London Web Week next week we have the Thinking Digital conference kicking off tomorrow, followed by a Geek Dinner on Friday night and then BarCampNorthEast over the weekend. I’m even planning on popping out tonight with a few people in town for the festivities. As if fate somehow got involved the weather is also pretty darn nice today too.

If you didn’t already know about these events and you’re in the area their just might still be the chance to come along. We have a few places left for barcamp in particular, you just need to register over on eventwax. And the Geek Dinner on friday night at The Pitcher and Piano is open to all, at least until it fills up.

It really is great to see thing happening at this end of the country, and a real shame I’m leaving the area so soon afterwards. I really hope the local community gets behind all these events and they act as a catalyst for interesting things. I’ve always been of the mind that if you can get enough smart, interested people together in one place then good things will happen. It’s why I enjoy going along to conferences, hackdays and barcamps. It’s also one of the main reasons I joined GCap.

Once I’ve recovered from the next two weeks (which may take a while as next week I’m in London for @media, BarCampLondon4 and anything else I can get along to) I’ll hopefully write more about trying to organise a barcamp and, if I really get time to think, about knowledge workers outside and inside London.

But in the meantime leave a comment if you’re going to be around in Newcastle over the next week. It should be a great opportunity to meet new and old friends alike.

Design Strategies for a Distributed Web

I just finished my presentation on the last day of the Xtech conference in Dublin. I’d chosen to ramble on about the advantages, problems and a few solutions of building applications atop of lots of APIs. The presentation is now up on slideshare at slideshare.net/garethr/design-strategies-for-a-distributed-web/.

Lots of interesting conversations have been occurring all week and a few people have mentioned a near barcamp feel at times. It’s a pretty small, clued-up, technical audience and as always some of the best bits have been conversations in the corridors. Jeremy has been live blogging many of the talks which should be a worth a read later on. Although with three tracks most of the time that’s only some of the great talks on offer.

Moving to London, Working For GCAP

On departing pubstandards in London a few weeks ago Dirk Ginader said something along the lines of:

According to Dopplr you’ve spent 40 odd days in London in the last 8 months. Don’t you think you might as well just move here?

I’m not saying that had anything to do with it, but I’m happy to announce I’m moving to London in a permanent manner very very soon. I’m taking up a job at GCAP, working with quite a few people I know from various web goings on; Hi to Simon, Ross, Brad, Ed and Stuart amongst others.

Freelancing for the last year has been great. I’d been able to work on the variety of projects I’d wanted after leaving my previous job, and I’ve had more time to myself when I wanted to get involved with other projects. WaSP, SXSW, Highland Fling, BarCamp, Thinking Digital to name a few things I’ve been up to in the past 6 months. But something about working with a super smart team of people won me over. The fact that I’ll get to focus on Python and Django is an added bonus.

Patrick apparently closed the book on this happening some time ago. I’m just hoping someone got the date about right and won the money. I’m sure I’ll write more about the whole thing as time permits and lots of specific topics spring to mind. I have a number of events in the next month to get along to before all this happens however; if you’re along at Xtech, Thinking Digital, London Web Week, BarCamp NorthEast, BarCamp London 4 or @media then remember to say hi. My addiction to attending web events can only get worse with me being nearer London I fear.

A Really Simple Capistrano Recipe

Build scripts. I think I’ve probably mentioned before; everyone knows they should but not many actually do. I’m not talking about your large in-house development teams or your sexy web startups; both probably have a good-enough build process for different reasons. But smaller teams, web design agencies or freelancers often rely on FTP and a prayer. One of the problems is definitely finding suitable documentation. It’s not that their isn’t a lot of good quality documentation - it’s the suitability of it for those with only a passing interest and a limited systems administration experience.

Capistrano is a tool for automating tasks on remote servers. It can be used for all sorts of useful things but we’ll concentrate on it’s utility for web site deployment. It’s been popular with the Rails crowd for a while and is written in Ruby, but it can be used for deploying sites in Python, PHP, Django, whatever you happen to be using at the time.

You’ll first need to install Ruby and Capistrano. The Capistrano site has installation instructions which should get you up and running. You’ll need to be able to use a console on your machine and have a passing acquaintance with your webserver. I’ve also assumed that you’re using source control, mainly because if you’re not you should start their. Automation becomes much more useful and practical on the back of a good source control system.

Right, lets get started. Create a file called capfile somewhere on your local machine. The following two code snippets should go in that file.

<code>set :username, "<username>"
set :host, "<host>"
set :path, "<path>"
set :restart, "/etc/init.d/apache2 restart"
set :checkout, "svn export --force"
set :repo, "<repository address>"</code>

Just replace the variables (marked with angle brackets) with your own values. The path is the full path on the remote server where you want to deploy your files to. Note that I’ve abstracted out the webserver restart command and the source control checkout command. The above example values assume you’re using apache and subversion (and want to do a subversion export). It should be easy enough to change these for your preferred combination of source control system and webserver.

<code>desc "Remote checkout and restart of webserver"
task :deploy, :hosts => "#{username}@#{host}" do
  run "#{checkout} #{repo} #{path}; #{restart}"
end</code>

If your platform of choice doesn’t require a server restart (say PHP) you can always remove the ; #{restart} portion of the command.

Now we’re all set up just bring up a console in the folder containing the capfile and run the following command:

<code>cap deploy</code>

After prompting for a password to access the remote machine this should display the output from the commands being run. Depending on the size of the repository you’re checking out

Of course for most projects you would want to expand this basic recipe. Maybe you have database updates to run or need to deploy to multiple machines at once? You’ll might have machine specific configuration files which aren’t in the source control system. You will also probably want to be able to revert back if something goes wrong and you might want to only change files that have been altered. Capistrano has lots of powerful features built in that will let you do just that, and once you get started with this sort of build scripting you’ll find lots of areas to improve. And you’ll find lots of good tutorials online that will take you further.

The difficult part with project automation and build scripts is often the getting started in the first place; especially if you’re not the server admin type. Unfortunately the documentation and articles written on the subject tend to be somewhat arcane and aimed at the more hardcore developer. Hopefully this simple, usable, example convinces someone else to give it a try.