What's new in Django 1.1

With the release candidate for Django 1.1 out the door I decided to have a quick look at what’s new. This isn’t a complete list, rather the bits I found most interesting.

Conditional Views

Django now has much better support for conditional view processing using the standard ETag and Last-Modified HTTP headers. This means you can now easily short-circuit view processing by testing less-expensive conditions. For many views this can lead to a serious improvement in speed and reduction in bandwidth.

A nice set of decorators for dealing with ETags and Last-Modified headers. Again very simple to use and set up, and a simple way of squeezing a little more performance out of you application.

Admin Actions

The basic workflow of Django’s admin is, in a nutshell, “select an object, then change it.” This works well for a majority of use cases. However, if you need to make the same change to many objects at once, this workflow can be quite tedious. In these cases, Django’s admin lets you write and register “actions” – simple functions that get called with a list of objects selected on the change list page.

Anything that makes the admin a little more powerful and a little more flexible is a good idea in my book. Admin actions allow you to run code over multiple objects at once, simple select them with a checkbox then select an action to run. This is worth it for the delete action alone, but you can write your own actions simply enough as well (for instance for approving a batch of comments, or archiving a set or articles.)

Editable Admin List Items

You can now make fields editable on the admin list views via the new list_editable admin option. These fields will show up as form widgets on the list pages, and can be edited and saved in bulk.

Another time saving admin addition, this time for making some fields editable from the change list rather than the object view. For quick changes, especially to boolean fields, I think this again is a nice addition.

Unmanaged Models

You can now control whether or not Django creates database tables for a model using the managed model option. This defaults to True, meaning that Django will create the appropriate database tables in syncdb and remove them as part of reset command. That is, Django manages the database table’s lifecycle. If you set this to False, however, no database table creating or deletion will be automatically performed for this model. This is useful if the model represents an existing table or a database view that has been created by some other means.

I particularly like this addition. One of the issues I had with Django was some of the built in assumptions, in particular that you’d be using a SQL database backend. Using unmanaged models looks like a great approach to using an alternative database like couchdb, tokyotyrant or mongodb or representing a webservice interface in your application.

I’m sure I’ll have missed a few other interesting changes or additions. Anyone else have a favourite?

Asteroid - simple app for running scripts and recording the results

Asteroid is a simple web interface for running scripts and recording the results. It’s like a much simpler and more general purpose version of something like Cruise Control. You can get the code on Github.

Asteroid Dashboard

I built it to solve two main problems:

  • It’s sometimes useful to have a historical record of a scripts execution, in particular whether it passed or failed and what the output was. Just running a command line script probably doesn’t give you that. It’s also useful to have a more graphical interface for those members of the team who don’t use the command line.
  • When working in a team you often want to run scripts against shared infrastructure, for instance deploying a testing release or running a test suite. Seeing what is running at present helps with that.

So it should be useful for running deployments, running test suites, running backups, etc. It currently doesn’t have scheduling or similar build in, but as everything is triggered by hitting a URL it would be simple enough to use cron for something like that. It should also be useful whatever language you write your scripts in; rake, ant, shell scripts, etc. At the end of the day it just executes a command at the console.

Requirements

Asteroid uses the Django Python framework under the hood.

You’ll also need a database. The default in the shipped settings is to use sqlite but this should work with any database supported by Django.

You’ll also need a decent web browser. I’ve gone and used HTML5 as an experiment and with this being a developer tool I’m hoping to stick with it. It would be easy enough to convert the templates if this is a problem however.

The application has an optional message queue backend which can be enabled in the settings file. This is used to improve the responsiveness of the application as well as allow commands to be executed on a remote machine, rather than on the box Asteroid is running.

Other AMQP compliant message queues should work but it’s currently only tested with Rabbit.

If you are intending to do any development on Asteroid, or just want to look more closely at the code, I’d recommend installing

Usage Instructions

You should be able to just download asteroid and run it from wherever you put it, once you setup the database.

cd asteroid/configs/common
manage.py syncdb
manage.py runserver

This should bring the local web server up on port 8000 so visit http://localhost:8000 and see.

If you’re using the message queue backend you’ll need to run the listener script in order to get your commands executed. At the moment that means modifying a constant in the listener script to point at a running message queue instance at asteroid/bin/asteroid_listen.py.

cd asteroid/bin
./asteroid_listen.py

Once you’re up and running you should be able to add commands via the admin interface at http://localhost:8000/admin/. The username and password should be those you added when creating the database via the syncdb command above.

The development configs include a few additional applications (mentioned above) which I use for testing and debugging. You can run the test suite like so:

cd asteroid/configs/development
manage.py test --coverage

Todo

This is an early release that just about works for me. I can already see a number of areas I’d like to clean up a little or extend. For instance:

  • Other deployment options, including a WSGI file and a spawning startup script.
  • Use a database migration system to make upgrades easier.
  • Make the message queue listener script more robust.
  • Make the command entry more robust, it sometimes takes a bit of fiddling with to get something to run correctly.
  • Formalise running scripts on remote machines, including support for running on multiple machines.
  • Paging for long lists of commands or runs.

Notes

I’m pretty happy with how it’s shaping up so far. Under the hood it works by having the web app put a JSON document on the message queue. The JSON contains the command to be run and a callback URL. The script listening to the message queue picks up the message, runs the command, and posts a JSON document back to the webhook url. It keeps the web interface snappy, as well as meaning it can show which commands are currently in progress at any given time. It also has the side benefit of meaning you can execute commands on a remote machine, as the listener doesn’t care where it’s running.

As noted above I have a few ideas of where I want to take it, but I’m going to try using it for a bit and see how that goes. If anyone else finds it useful then do let me know.

It's the Data we Want

A spreadsheet. A CSV file. Whatever is in use internally. Made available to people like us under a suitable license.

I feel a little self adsorbed quoting myself (from a recent Refresh Cambridge discussion) but I did like the turn of phrase. What I was rambling on about was Cambridge County mapping data, after a question from a nice chap from the council about what “new, exciting map technology” we’d like to see. But it applies to any data that you’re trying to make public what-so-ever, be it government or otherwise.

What myself and a few other people were talking about, and one of the things that has been discussed as part of the Rewired State group, is that it’s all about the data, not necessarily about a nice web based API.

Now I’ve written and spoken about the need for well designed API’s being treated as part of the user interface. But remember interface design, and by association API design, isn’t easy. API design is often about building manageable flexibility. A public API is often about managing the flow of data you control out to third parties, as well as the information itself it might include limitations on usage, or request rate, or storage. A public API codifies how that information can be accessed. APIs also have to tread a fine line between making it easy for you to solve your problem, and making it easy for everyone else to solve their completely different problems. These compromises are design.

But not everything needs an API. Sometimes it’s just about the data, and the best way of getting at that data is as raw as possible. Government data is an easy sell here, as it is (or rather should be) our data. It’s also for the most part interesting to read rather than write (historical council tax data, or population data for instance). Raw data can generally be provided quicker than via an API. It doesn’t need fragile computer systems or extensive manual labour. It doesn’t need particularly clever computing resources. Just upload a spreadsheet or a CSV file to a sensible URL on a known, regular basis and away we go.

And giving data like this away to the development community is likely to have a few additional benefits if that data is useful (it probably is to someone). We’ll happily write software libraries, or create APIs over the top of it for you. We’ll also write all sorts of useful tools using the data in ways no one else thought of. So if you’re sat on a load of data that’s not core to your business, or is meant to be public anyway, then lets start talking publicly about how to just get this out on the web quickly and cheaply, rather than spending lots of your time and money on something fancy.

Thoughts on the whole XHTML/HTML5 affair

I wasn’t going to write anything about the whole XHTML2 thing. I noted its passing, got a nice message on Twitter and thought that would be it. But no. The web standards world exploded. I honestly didn’t see that coming.

Let’s get a few things straight:

  • I use XHTML 1.0 for this site. In fact I’ve been using it for the majority of things for most of my professional life.
  • I don’t serve content with an XML mime type. Neither does anyone else. It’s a complete non issue. Ignore it.
  • At my last job we used HTML 4. It meant I had to remember not to close my image elements, which bugged me, but not too much. I still quoted everything. Closed everything I could. And only used lowercase element names.
  • My latest two pet projects are using HTML5. I’m still closing everything (including image elements, yay), quoting everything and lowercasing everything.

Web Standards are interesting, in that they are standards for both implementors (browser makers) and for authors (us). I like coding standards in programming languages too, it’s one of the things I love about Python and PEP8. But with these standards it’s not about making your code work, it’s about shared conventions and readability. So common spacing, UPPERCASE for constants and Leading caps for class names for instance. It’s also about having a tool to check everyone is adhering to standards, like pep8.py or FXCOP for .NET. If everyone writes code in the same way it’s easier to read, write and to pick up someone else’ code. You can do that with HTML, but you have to do that with XHML.

Now the whole HTML 4.0 vs XHTML 1.0 thing has come up lots of times, on mailing lists, at conferences as well as down the pub. I know on occasion me, Drew, Rachel and Jeremy side against Simon and Nat on the issue. But what’s interesting is that I think we all agree on all the typographical conventions stuff. My former colleagues with a passion for front end standards and HTML 4 did the same thing. I even remember Simon looking for ways to validate against HTML 4 but also to check for all lower case elements, closed paragraphs and the like.

Which brings me to the reason why I use XHTML: The validator enforces my preferred coding standards for HTML - lowercase elements, quoted attributes and closed elements. That’s it. Not much really. I know it’s marketing XHTML rather than technical XHTML. I don’t care. Or rather I do care, I just make a conscious pragmatic decision based on a small personal advantage. I’m both pedantic and like having a tool chain which enforces that, XHTML suits my style.

The markup language debate is being talked about in terms of pragmatists vs purists. But ignoring the people who both really understood and really wanted XHTML2, it’s mainly the pragmatists arguing amongst themselves now. Some of them are big company people, others working for themselves. Some have standards or academic leanings, others are rooted in commercial web design. Some people probably work on huge long term projects, others relatively small sites and apps. And I think it’s these cultural differences that are the root of arguments now. So blog posts coming out saying the same thing but arguing with other people give a strange impression of disagreement. Throw in that the web lends itself to popular blogs gathering a crowd of like-minded people around them and hey presto we have people feeling unfairly put upon and getting agitated.

What a storm in a teacup. Who doesn’t genuinely think the best approach is to use whatever you’re using now for most projects, investigate HTML5 as time permits, and then expect to start using HTML5 in bits and pieces in the short to medium term, when being mainly dependent on your target audience?

In my opinion the only genuine problem that this saga has highlighted is the fear, uncertainty and doubt around all flavours of HTML amongst a large number of web professionals. People don’t get this stuff at all. With the added resources soon to be put into the HTML5 working group at the W3C this outreach and education side of the project has to have just as much love and attention as the spec itself.

Pants Python Code

One of the projects that came out of the Django Dash recently was PyPants which I’m finding very cool.

Urltest on PyPants

It’s basically a quality tracking service for Python modules. For instance my recent UrlTest module has a page on PyPants, scoring a good B grade after some cleanup work earlier today.

Under the hood I think it’s probably CheeseCake which is available as a command line application, maybe with a hint of PyLint and pep8.py thrown in. But the nice interface, as well as tracking of scores over time, really add something. GitHub has been credited by some as making sharing code more fun, I’m hoping projects like PyPants can do the same for quality in Python code.

Congrats to Eric Holscher, Travis Cline, and Nathan Borror on a fantastic addition to the Python community.

Urltest on PyPi

I’ve been meaning to add some of my code to the Python Package Index for a while and have finally gotten around to it with Urltest, my simple DSL for testing WSGI apps.

You can now find it at pypi.python.org/pypi/urltest and install it using setuptools with:

pre. easy_install urltest

At the moment I’ve not added any categorisation or detailed description to the setup.py file, I’ll be doing that soon. I wanted to get it working with the absolute minimum setup.py file, which turned out to look like:

pre. #!/usr/bin/env python from setuptools import setup, find_packages setup( name = “urltest”, version = “0.1”, author = “Gareth Rushgrove”, author_email = “[email protected]”, url = “http://github.com/garethr/urltest", packages = find_packages(‘src’), package_dir = {”:‘src’}, )

Uploading it to PYPI itself was incredibly simple, partly as I was already using setup tools for local installation.

pre. python setup.py register python setup.py bdist_egg upload

Let me know if anyone uses this and gets it working. I’ll be adding more details and maybe even some more features when I get the chance. Once I do that I’ll probably work on a few more packages as well.

Ant for Web Developers II - Restart Apache

Following on from yesterdays first useful ant task, here’s another commonly used task - restarting a remote service. I’ve used apache in this example, but it could be any service running on your remote machine and it doesn’t have to be the restart command.

In order to do this we’ll use the sshexec target which has a third party library dependency. This is the same third party library needed for the scp task in yesterdays post

You need first to download JSCH and then compile the source using ant. Just run ant dist in the downloaded folder and you should get a .jar file in the /dist/lib folder. Save this .jar file as as jsch.jar to a folder in your home directory ~/.ant/lib where ant can automatically load it. Alternatively you can run ant with the -lib command to load libraries from a different location.

pre. ant -lib /tools/ant-libraries

With that out of the way lets have a look at the task.

pre.

<?xml version=“1.0” encoding=“UTF-8”?>

And running it is as simple as:

pre. ant restart-apache

One potential issue with tasks like this is storing the password in the build file in plain text. The target we’re using can also use key authentication is you’re happy using ssh keys. Alternatively you can set properties on the command line each time you run ant like so.

pre. ant restart-apache -Dpassword={password}

Ant for Web Developers I - Backup Config File

I occasionally get carried away with Apache Ant. For those that haven’t come across it, Ant is a build tool written in Java, using an XML syntax to describe a series of repeatable tasks. In your typical web standards savvy, dynamic language favouring, web developer types that description is probably all they (think they) need to know. It’s Java. It’s XML. It’s only really useful in the context of building software (dull).

But I think Ant is a particularly handy tool to have around for anyone working on even simple websites. A couple of strong use cases come to mind:

  • If you’re working in a team environment then build files are hugely useful when introducing new people to the team, or when moving people around. Getting code up and running at the start of a project, or if you join the team part way through, can be tricky. A well written build file can automate this.
  • Even when working on projects on my own I tend to write simple build files. The main reason is so I don’t forget how to do something. How do you deploy this particular site? How do you run the test suite or generate the documentation? Build files can encapsulate this, and rather than documentation that might be out of date the build file will be executed to do that job in question.

So with all that in mind I’m going to try and do a series of posts each covering a single task, aiming to cover things that your regular web developer will find useful. With that in mind if anyone has any requests or questions let me know either by email or in the comments.

Out first task lets us backup a file from our remote web server, in this case it’s the apache2.conf file used to setup apache. Obviously it could be any file you want to get hold of. The example below has a couple of properties for the username and hostname of the remote machine. Save the following snippet into a file called build.xml and place it anywhere you like on your machine.

pre.

<?xml version=“1.0” encoding=“UTF-8”?>

Running the task, once you have ant installed (it comes already installed on OS X and is generally available in whatever linux package management system you prefer), is as simple as typing the following into a console.

pre. ant backup-apacheconf

This should download the apache2.conf file to you local machine, into the same directory as your build file.

The above task requires that you have scp installed on your machine, which is pretty likely if you’re using OS X or Linux. Ant comes with an inbuilt scp task, but it requires you to install a separate java library. If you’re happy doing that then you can write tasks like:

pre.

<?xml version=“1.0” encoding=“UTF-8”?>

Less CSS

Ruby people really don’t like CSS do they? But Less is actually pretty cool. It’s basically an attempt to bootstrap features, specifically Variables, Mixins, Operations and Nested Rules, into CSS. The best part about this is it uses CSS syntax and a simple one step compiler. I’d be interested to know what the folks at the W3C think about this.

So for instance you can do:

pre. /* LESS */ brand_color: #4D926F; #header { color:brand_color; } h2 { color: @brand_color; }

and compile it down to:

pre. /* CSS */ #header { color: #4D926F; } h2 { color: #4D926F; }

Message Queues at Cambridge Geek Night

Last night was the first Cambridge Geek Night and saw 35 people or so fill a room above a pub to listen to a few short talks and converse with fellow geeks. I had the pleasure of giving the first talk, a short introduction to using message queues for web developers.

I got lots of good questions from interested people and by the sounds of things it had the desired effect - for people unfamiliar with using a message queue to go out and have a play with some of the cool software available to solve your problems.

Overall the night was definitely a success. Suitably geeky conversations. A chance to meet new people as well as old friends. Good job Vero and David for organising the event and here’s to the next one.