I’ve just found Dreque from Samuel Stauffer on GitHub. It’s yet another take on the whole messaging things which is definitely seeing a lot of activity at the back end of this year. It’s using Redis on the backend and looks really rather nice:

Submitting jobs:

<code>from dreque import Dreque
def some_job(argument):
dreque = Dreque("")
dreque.enqueue("queue", some_job, argument="foo")</code>


<code>from dreque import DrequeWorker
worker = DrequeWorker(["queue"], "")

DJUGL December

As mentioned at the last event I’ve taken over organising the Django User Group London event from Rob. Tickets are now available for the next event which is going to be on the 3rd of December at The Guardian offices in Kings Cross.

You can sign up on eventwax

Erlang Screencasts

I’ve been trying to learn Erlang for a while. What I actually mean is it’s been on my list of things to learn for months, along with all sorts of other incredibly interesting bits and pieces. I spend a little bit of time at home but the majority of my learning time is now spent commuting to London and back most days. Sometimes I’m even going all the way to Swindon which gives me even longer to not learn Erlang.

The main problem with learning something new on the train is space. Reading a book (or my new Kindle) or just using my laptop is fine. Trying to do both at once is nearly impossible (I’ve tried). So I’ve decided to give another approach a try, namely screencasts.

I’ve only done the first Erlang in Practice episode so far but I was hugely impressed with the content and general presentation. $5 as well doesn’t seem bad at all I don’t think. The episode was half an hour long, but took me a little longer, probably closer to 45 minutes, as I was playing along at home and typing the code examples as I went. I also got sidetracked with messing with my vim configuration at the same time but hey. This makes them perfect for my hour long commute. The full series is 8 episodes long and with luck I’ll be able to work through them this week.

So, good job Kevin Smith and Pragmatic for a nice, accessible start to Erlang. All I need to do now is find something interesting to hack on in Erlang.

Django Committers

I’ve been lurking on the django-developers mailing list for the last couple of weeks and that provided an excuse to play with the new Twitter Lists feature. So here’s a list of djangocommitters on twitter. If I missed someone do let me know. Their is a chance you won’t be able to see this if you’re not on the beta yet I think, sorry!

Problems Installing Hadoop 0.20 and Dumbo 0.21 on Ubuntu

The Hadoop wiki has a great introduction to installing this piece of software, which I wanted to do to have a play with Dumbo. The Dumbo docs also have a good getting started section which includes a few patches than need to be applied.

Dumbo can be considered to be a convenient Python API for writing MapReduce programs

Unfortunately it’s not quite that simple, at least on Ubuntu Jaunty. Hadoop now uses Java6, but if you just follow the instructions on the wikis you’ll hit a problem when you run “ant package”, namely that a third party application (Apache Forrest) requires Java 1.5. Once you fix that, the build script will complain again that you need to install Forrest. Here’s what I did to get everything working:

pre. sudo apt-get install ant sun-java5-jdk

pre. su - hadoop wget http://mirrors.dedipower.com/ftp.apache.org/forrest/apache-forrest-0.8.tar.gz tar xzf apache-forrest-0.8.tar.gz cd /usr/local/hadoop patch -p0 < /path/to/HADOOP-1722.patch patch -p0 < /path/to/HADOOP-5450.patch patch -p0 < /path/to/MAPREDUCE-764.patch ant package -Djava5.home=/usr/lib/jvm/java-1.5.0-sun -Dforrest.home=/home/hadoop/apache-forrest-0.8/

With all that out of the way you should be able to run the simple examples found on the rather excellent dumbotics blog. If you’re using the Cloudera distribution, or when the Hadoop 0.21 gets a release, these problems will disappear but in the meantime hopefully this saves someone else a bit of head scratching.

Learnings from September

I’m keep meaning to get around to writing about why I think the future of web developers is operations but in lieu of a proper post here’s a list of things I’ve been spending my work life getting to know this month:

  • Puppet - It’s brilliant. Define (with a Ruby DSL of course) what software and services you want running on all your machines, install a daemon on each of them, and hey presto central configuration management.
  • VMWare vsphere - puppet makes more sense the more boxes you have. With vsphere I can have as many boxes as I want (nearly). Command line scripts and an actually very nice windows gui for settings up virtual machines is all pretty nice, especially running on some meaty hardware.
  • Nagios - With lots of boxes comes lots of responsibility (or something). Nagios might look a bit ugly, and bug me with it’s needless frames based admin, but I can see what people see in it. Which frankly is the ability to monitor everything everywhere for any change what-so-ever.
  • Solr - I’m now also pretty well versed in using Solr. I’ve used it in the past, but always behind a Ruby or Python library. Now I know my way around all the XML based configuration inards. Heck, I’m even running a nighly release from a couple of days ago in a production environment because I wanted a cool new feature. A special mention to the Solr community on the mailing list, twitter and irc for being great when I had questions.
  • Solaris - I nearly forgot, I spend more time than I care to remember working out how to use Open Solaris (conclusion: OK, but not Debian) and eventually Solaris 10 (conclusion: hope I don’t have to do that again). My installation notes read like some hideous hack but everything works fine in production and it’s scarily repeatable so I’ll live with it for now.

I do wonder if it’s just me that’s drawn to knowing how everything in the full web stack works. But personally I can’t just write code if I don’t understand how to deploy it or what it’s running on. Front end types know this all too well. Being a master of CSS, HTML and Javascript simply isn’t enough. You need to understand the browser to get anything done. I’m not sure it’s the same for all backend inclined folk; how many PHP programmers really understand Apache and a few other useful bits of web tech?

No database test runner added to test extensions

Thanks to Brad I’ve just released a new version of Django Test Extensions (also on GitHub with support for running tests without the overhead of setting up and tearing down the database. Django still has a few places were it assumes you’ll have a database somewhere in your project - and the default test runner is one of them.

Automating web site deployment at Barcamp Brighton

On the first day at Barcamp Brighton this year I did a brief talk about getting started with automating deployment. I kept it nice and simple and didn’t focus on any specific technology or tool - just the general principles and pitfalls of doing anything manually. You can see the “slides on Slideshare”:

As part of the presentation I even did a live demo and promised I’d upload the code I used. The following is an incredibly simple fabric file that has most the basic set of tasks. Fabric is a python tool similar to capistrano in Ruby. I don’t really care whether you’re using make, ant, rake, capistrano or just plain shell scripts. Getting from not automating things to automating deployments is the important part - and it’s easier that you think.

The other part of the code example was a very basic apache virtualhost so just in case anyone needed that as well here it is:

&lt;VirtualHost *:80&gt;
    ServerName sample.local
    DocumentRoot /srv/sample/releases/current
    &lt;Directory /srv/sample/releases/current&gt;
        Order deny,allow
        Allow from all
    ErrorLog /var/log/apache2/sample/error.log
    LogLevel warn
    CustomLog /var/log/apache2/sample/access.log combined

Another chance to DJUGL

DJUGL is back, the monthly Django meetup in London. I think the last few times have been as much about useful Python stuff as just using Django, and this time it’s officially a bit more broad ranging. If you’re in or around London on the 24th September then come along.

You can get more information on Twitter or by following Rob. But expect a few short talks, some interesting conversations and maybe some beer with other like minded developers.

I’m going to be talking about automating deployment of Python web applications. If you follow me on Twitter you’ll have heard me rambling a little about some of what I’ve been up to, and some of the posts here give an insight into what I’ve been working on. But the short version is that several friends mentioned how difficult it could be to get a working Django application from a local machine to a production web server. And I though I better get down in script form my experiences of Django, WSGI applications and web server setup to make things easier.

I think this situation is partially caused by the success of the Django development server, and partially by people coming from a PHP background. In my PHP days I think I always wanted to know how Apache did it’s think, so long ago jumped into anything and everything in httpd.conf from loading modules to virtual hosts. But not everyone does the same, and PHP does make simple deployments easy enough that you might get away without doing so. Rails went through the same problems and seems to be coming out the other side. I’m hoping that Django and Python is soon to be in the same position, where basic deployment is just a given.

Django and WSGI deployment on Solaris

Now I’m generally an Ubuntu guy, but I’ve just had the need to setup some boxes running Solaris for Django and a handful of WSGI applications. I know my way around Ubuntu pretty well. I know all the packages I need to install and in what order. Hell, I even have all that scripted so I can just run a command and it works by magic. I’ll script the following steps if I can do when I get round to it but here, in one list, are the installation instructions for Apache, mod_wsgi, Mysql, MySQLdb, setuptools and memcached that worked for me on the latest version of Open Solaris (2009.06 at the time of writing).

First up I needed to install Apache and start the service running.

pfexec pkg install SUNWapch22
svcadm enable http:apache22

You should be able to test that’s running by hitting localhost on a browser running on the same box. Now for MySQL.

pfexec pkg install SUNWmysql5
svcadm enable mysql:version_50

This installs the mysql binary into /usr/mysql/5.0/bin/mysql on the system I’m working on. As I want to talk to the MySQL database server using Python I need to install MySQLdb.

pfexec pkg install SUNWmysql-python
ln -s /usr/mysql/5.0/lib/mysql/libmysqlclient_r.so.15 /usr/lib/libmysqlclient_r.so.15

This installs the library files into /usr/mysql/5.0/lib and Python doesn’t know were to find them. The above command links them into the more standard /usr/lib folder were Python will pick it up nicely.

I tend to use mod_wsgi for serving Python apps behind Apache, however a mod_wsgi package isn’t part of the default package list. It is however available in the pending list so first you need to add that list of packages.

pfexec pkg set-authority -O http://pkg.opensolaris.org/pending pending
pfexec pkg refresh
pfexec pkg install mod-wsgi

This installs the module but you then need to tell Apache to load it. Add the following line to /etc/apache2/2.2/conf.d/modules-32.load or /etc/apache2/2.2/conf.d/modules-64.load depending on your architecture.

LoadModule wsgi_module    libexec/mod_wsgi.so

To get Apache to load that module you need to restart it like so:

svcadm restart http:apache22

I use Pip for installing Python code, but tend to install setuptools to make installing Pip easier. I don’t know if an up to date Pip package exists.

pfexec pkg install python-setuptools

This should leave you with easy_install on your path so installing Pip, then virtualenv should be a breeze.

As an added bonus I also installed memcached for some snappy caching.

pfexec pkg install SUNWmemcached

This won’t start up by default and needs a little configuration. The first command will launch you into a prompt where you can type the rest of the commands.

svc:> select memcached
svc:/application/database/memcached> setprop memcached/options=("-u" "nobody")
svc:/application/database/memcached> quit

Once you’d done that you should be able to start memcache on the standard port.

svcadm refresh memcached
svcadm enable memcached

Et voila. The internet helped massively on my quest to track down this information. Not all of the following links turned out to work for me but all of them led me in the right direction. Thanks everyone.

I’m not a Solaris admin. I’m not really a sysadmin at all, I just end up pretending to be one of late. Any experienced Solaris people with experience of these tools reading this I’d be grateful for any hints and tips. Hopefully this saves a few people from the head scratching I’ve been doing for the last few days.