CSSDOC looks like a good idea. I’m sure a few people mentioned this last year at @media after the talk by the people from The Guardian but nothing came from it. Hopefully tools will start to come about soon.

CSSDOC is a convention to comment Cascading Style Sheets (CSS) to help individuals and teams to improve writing/coding/styling/managing CSS files. It is an adoption of the well known JavaDoc / DocBlock based way of commenting source-code. That’s putting style, docblocks and tags together.

Django Performance Tip - Profile Your Filters

I’ve been doing some performance profiling of this here blog. Not because I really need to due to huge amounts of traffic unfortunately, more because I’m planning on releasing the code and wanted to give it a good once over before I do.

pre. 69396 function calls (67324 primitive calls) in 0.479 CPU seconds Ordered by: internal time, call count ncalls tottime percall cumtime percall filename:lineno(function) 123 0.138 0.001 0.154 0.001 build/bdist.macosx-10.5-i386/egg/textile.py:2478(qtags) 6300 0.058 0.000 0.079 0.000 build/bdist.macosx-10.5-i386/egg/textile.py:453(preg_replace) 15705 0.045 0.000 0.059 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:219(_compile) 20 0.025 0.001 0.026 0.001 /Users/gareth.rushgrove/Documents/www/django/django/db/backends/sqlite3/base.py:165(execute) 1087 0.021 0.000 0.026 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:136(sub) 123 0.014 0.000 0.017 0.000 build/bdist.macosx-10.5-i386/egg/textile.py:2701(links) 18 0.012 0.001 0.030 0.002 build/bdist.macosx-10.5-i386/egg/textile.py:892(split_text) 12719 0.011 0.000 0.063 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:178(compile) 123 0.011 0.000 0.104 0.001 build/bdist.macosx-10.5-i386/egg/textile.py:2372(glyphs) 123 0.009 0.000 0.009 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:159(findall) 191 0.008 0.000 0.008 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:154(split) 18 0.005 0.000 0.005 0.000 build/bdist.macosx-10.5-i386/egg/textile.py:803(grab_links) 11 0.005 0.000 0.005 0.000 /Users/gareth.rushgrove/Documents/www/django/django/utils/text.py:52(truncate_html_words)

The above profile is from the home page which seemed a good starting point. The majority of the other pages are more straightforward, generally displaying just one article with a few related bits and pieces.

What jumped out was that the majority of the slow traces (the list is ordered with the slowest calls first) were to do with the use of the textile filter packaged with Django. Textile is a simple markup language which converts to HTML, I’ve used it for all my writing activities since I used Textpattern on this blog many moons ago. It’s not just the traces that explicitly mentioned textile either - the regular expression calls are also to do with template filters.

I quickly added a hidden html field to my model and a custom save method. On saving an article I now automatically transform the textile version of the content and save it in the html field. In the template where I used to have article.content|textile I now have article.html. A quick check of the profile on the same page showed a dramatic increase in performance:

pre. 15054 function calls (13248 primitive calls) in 0.081 CPU seconds Ordered by: internal time, call count ncalls tottime percall cumtime percall filename:lineno(function) 20 0.017 0.001 0.018 0.001 /Users/gareth.rushgrove/Documents/www/django/django/db/backends/sqlite3/base.py:165(execute) 97854 0.003 0.000 0.006 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py:144(deepcopy) 821 0.003 0.000 0.069 0.069

From nearly half a second (0.479 CPU seconds) to 0.081 CPU seconds - roughly a 500% improvement in performance!

If you are using text transforming filters in Django that make use of regular expressions I would look to try and get rid of them and move the conversion into a save method. You might not see quite as much of a performance gain as I did here but I’d be surprised if it’s not taking an awful lot longer than you might thing.

Their are a few tools that are handy to have around if you’re wanting to profile your own applications. Django Snippets has a handy profiling middleware and the really rather nice Command Extensions provides a few more tools including profile graph generation.

Simulating Rails like Environments in Django

I was always a fan of Rail environments and as part of some work upgrading this site to the latest version of Django I decided to clean up my whole deployment process. Part of that involved replacing everything in settings.py with the following snippet of code:

pre. ENV = “development” try: exec “from environments.” + ENV + “ import *“ except ImportError: print “You must specify a valid environment” exit()

I now have two settings files stored in an environments module containing all the usual django settings; one for my development environment and one suitable for live. The settings.py above is for my local development environment, with only one small change for live (this file doesn’t get deployed along with the source code for the site, so doesn’t get overwritten).

pre. ENV = ‘live’

This isn’t quite the same as the Rails implementation obviously. I run completely different server setups so I’m not too bothered about a flag on the runserver command like the -e flag for mongel. I could also probably do something to autodetect the environment but this works fine for me.

Unit Testing CSS - Looking for a Solution

I think it’s an epic failure of web standards that CSS is the only essentially untestable technology invented in last decade - Tomasz

Talking today on Twitter with Tomasz got me thinking again about one of those problems that I come back to once in a while. Unit testing CSS. CSS development is a pain, even with some sort of system. Admit it. I actually like CSS most of the time but it’s still painful at times. Hopefully with that out of the way you feel better.

unit testing is a method of testing that verifies the individual units of source code are working properly. A unit is the smallest testable part of an application.

All testing past simple validation of CSS seems to be done visually at present. Thinking about this from the point of view of CSS seems straight forward, but turns out not to be so for a variety of reasons. The problem lies in the cascading and compounding nature of the beast. Each individual CSS rule might do something which is self contained, but the chances on a real site are probably slim. For instance:

pre. body { font-size: 100%; } p { font-size: 2em; }

What is the size of the font size of a paragraph? It turns out it depends. Not just on more than one unit of source code (we have two rules here) but also on things like the browser. And how do we get this font size from a browser in the first place? I generally dislike Selenium but does it provide a mechanism for getting at the calculated DOM attribute values? Do we have to interface directly with a browser at a lower level?

wxMozilla, wxWebKit or maybe pywebkitgtz might prove useful, but I’m not sure at what level they operate. What I’m imagining here is maybe something like (excuse the Python, hopefully you get the idea):

pre. def test_text_size_is_12px(self): response = fetch_with_browser(‘http://www.google.com') self.assertEquals(12, response.search(“p”).fontSize

So we could use CSS selectors (ie. p) to find elements and then assert various DOM properties (ie. fontSize) are equal to values we specify. The magic is in getting access to those calculated DOM attribute values from an actual browser engine.

Another approach would seem to be looking at visual rendering and comparing against a known good version. This seems to be something that the Mozilla folks got up to a while back to test different browser versions. Their are a few tools that might help us out here too; BrowserCam provides a paid for service, Webkit2png is a handy command line script I’ve had fun with in the past and IECapt appears to be a similar beast for Internet Explorer. CutyCapt is another cross platform webkit based utility. I can see a few gotchas lurking here. Animations or slow loading javascript would obviously throw this into disarray. But disabling these in the browser might get up somewhere. How to compare images produced I’m not yet sure, but I reasons someone reading this might have a good idea?

As the title would suggest this post does not contain the answer, only a few useful links and two possible approaches to the problem. The questions at this stage are:

  • Does any of this exist already? If so who do we need to cuddle up to to get access to it?
  • Are any of the technical hurdles to either of the approaches mentioned above insurmountable? If not what is the best solution?
  • Does anyone except me and Tomasz even want this?

I reason their are a fair few things that would be needed to make this first practical and later standard; nice APIs, run times in various languages, and working out whether or not it actually helps CSS development to name but a few. But right now I’d go for a limited proof of concept that works on my machine. If anyone has any links to thinks that might be good starting points please let me know. Other ideas welcome as well.

One last thing; Mozilla’s latest employees are looking at the whole spectrum of developer tools. I’d love for them to start with something like this.

Google App Engine PyUnit Test Runner

I’m starting to play around with using App Engine again for small projects. It’s great for simple, somewhat throwaway apps as long as you don’t need anything too fancy. Actually all I want really is long running processes but I digress.

I’m increasingly writing test suites as well for even small projects and was missing the convenience of the Django test runner for running them against App Engine code. So I’ve spent a little time writing a simple test running script to use for non-Django Python projects. I’ve posted it over on the App Engine Cookbook for anyone else who might want to do the same.

I ended up writing something myself as I couldn’t find anything else which quite met my needs, so it’s the typical programmers itch code and as such is provided as works for me software. The other approaches out their didn’t quite meet my needs but might be useful to know about if you want to start testing your apps.

  • Nose GAE uses the nose testing framework which I’m less familiar with.
  • GAE Unit uses PyUnit but presents it’s results in a browser rather than on the command line.

Jabber, Erlang, Debugging. Things I'm playing with at the moment

I’m busy experimenting with various blogging approaches at the moment, hence the short links I’ve been posting recently. Another type of post I thought I’d give a try to was the list of interesting things. I find this sort of thing strangely cathartic - if nothing else by writing down the things I’m thinking about I won’t forget to spend time playing with them.

  • I still need to play around some more with Jabber/XMPP. I tried a little time ago to install a server locally on my Mac with only some success. I’ve now got a load of linux virtual images handy which might make that easier. What I would really like to see is a dirt simple XMPP server that you can use for local development. Something like Morbid. I’m just not sure I have the time to build one.
  • Which brings me on to virtualisation. I’m more convinced than ever that sand boxing different local environments is a good idea. I now have a stack of VMWare images set up for configuring.
  • It turns out Yahoo used Erlang under the hood for some part of the new Delicious I’m still pretty interested in actually doing something with Erlang, though what I’m not yet sure. I’ve been kicking round an idea for ages involving logging which might fit the bill.
  • I talked at barcampbrighton recently about debugging tools for django (and more broadly any development environment) that you need once you have a reasonable sized team. I’ve been busy packaging some of the tools I mentioned. I’ll either turn it into an article somewhere, multiple blog posts and/or hopefully get some code released on google code some time soon.
  • The Sphinx documentation system used by Django for the 1.0 documentation is pretty nifty. I’m investigating it for use at work at the moment.
  • I’ve been a huge fan of Textile since I used Textpattern years ago for an earlier incarnation of this site. But recently I’ve had a few niggling issues around extensibility and slight differences in implementation. So I’m pondering using ReStructuredText for my writing duties. It appears to be more powerful, more flexible and inherently extensible.

Google on Testing

Interesting testing And coverage reporting write up by the Google Engineers behind Update Engine. This is the sort of thing we keep discussing in the office.

Headless VMWare Fusion

The latest version of VMWare Fusion lets you run virtual machines in headless mode. Which is pretty handy if you’re using a Linux VM to mirror your live environment. The strange thing is that it’s not enabled by default. To enable it you need to run the following on your console: defaults write com.vmware.fusion fluxCapacitor -bool YES

Django Admin Options

Working on a decent sized Django project at work means I’ve found myself delving into Django’s admin interface more than a few times. Although it’s always possible to just use a custom template and do everything yourself it’s nearly always easier and often quicker to use the generated admin views. One of the problems with that is, even with therecent 1.0 release, some of the options are not that well documented outside the source code or in posts buried on mailing lists.

I’ll assume a little bit of familiarity with the new-forms-admin way of doing things which is now the default in Django 1.0. If you’re just getting started with building Django sites then you might want to first have a look at a tutorial or two. It’s quite different to the examples found in the original Django book or older online tutorials but it’s also much more powerful and flexible with a better separation of concerns.

We’ll start off with a very simple model in models.py which defines a simple Article class with a couple of fields.

pre. from django.db import models class Article(models.Model): title = models.CharField(max_length=200) content = models.TextField() publish_date = models.DateTimeField(default=datetime.today)

Django 1.0 introduced the concept of admin autodiscovery. By playing your admin declarations in admin.py in an application (most likely next to models.py and views.py) you can tell django to find these automatically. To enable auto loading of admin modules you can add the following to your urls.py.

.pre from django.contrib import admin admin.autodiscover()

This will load the module admin.py for each of the apps in the installed apps list. Now Lets add an admin class in your admin.py to go with the above models.py. We’ll call it ArticleAdmin:

pre. from django.contrib import admin from models import Article class ArticleAdmin(admin.ModelAdmin): pass admin.site.register(Article, ArticleAdmin)

The important line is the last one, in which we register the admin for the Article class. This will display the relevant admin views in the Django admin for that model - allowing us to add new articles, list existing ones and delete old ones. But by default the admin is quite sparse.

Once we have a few articles in the system we’ll find it hard to find them again. Lets add a few more lines to our admin.py file:

pre. class ArticleAdmin(admin.ModelAdmin): list_display=(‘title’, ‘publish_date’) ordering = [’-publish_date’] list_per_page = 25 search_fields = [‘title’,‘content’] date_hierarchy = ‘publish_date’

Lets step though each of these statements and see what we’ve done:

  • setting list_display for the title and publish_date means these two fields will appear in the changelist. This is the view you get when you hit Articles in the admin and allows you to find the article you are looking for.
  • ordering is self explanatory, in that we choose to order the items in the changelist by the publish_date rather than the auto generated numeric id.
  • list_per_page is another straightforward option, setting the maximum number of articles to show in the changelist before the list starts paging over multiple pages.
  • search_fields adds a simple search to the changelist, the fields specified set which fields to search; title and content in this case.
  • date_hierarchy is great when you have a date associated with an object. This outputs a separate filter list which displays the years by which to filter. The option you pass to this setting is the field name which stores the date.

The simple example above hopefully demonstrates the ease of which the admin can be configured. Knowing about these capabilities already built into Django can save you quite a bit of time when it comes to producing production ready admin interfaces. Except for more complex systems this should suffice. Below is a table of the Django admin options I’ve been using. If anyone has any more let me know and I’ll add them here, along with a brief description.

Option Description
model Set the model for which this is the admin
form Set the form class if one has been created
list_display Set which fields should appear in the changelist view
list_filter Se which fields should be used to provide a filter in the changelist view
raw_id_field Useful when you have a Foreign Key on another model with lots of records. This changes the default interface from the a select box to a custom widget
ordering Specify the order of the objects in the changelist
fieldset Fieldsets allow for control over the changeform view, setting which fields to display and whether to separate them out into individual fieldsets. Worth investigating
save_on_top If you have a long form it’s useful to be able to display the save buttons at the top as well as the bottom
date_hierachy Add date based filtering to the chaneglist view
radio_fields Another alternative widget for Foreign key fields, this time using radio buttons. Useful for fixed small lists of objects
list_per_page How may objects to list per page on the changelist view
search_fields Enable search for the model and specify which fields to search
prepopulated_fields Some fields might be prepopulated based on the user entering text into another field. This is often used to prepopulate slugs based on the title of an object
filter_horizontal The default widget for many to many fields is the rather shoddy multiple select box. Filter horizontal enhances this with some super javascript, making it much more usable. Never use many to many fields without this or filter_vertical
filter_vertical Does exactly the same as filter_horizontal, except the filter lists appear one above the other rather than side to side. Useful for thinner admin views

As you can see you can customise the default admin views a great deal even without creating your own templates and defining custom admin views. The best part is still that as well as being useful for demonstrations and prototypes these interfaces are useful on a live production site. Quite an achievement I think.


Open Microblogging looks pretty interesting. An open standard built upon other open standards for the purpose of passing information between micro blogging services like Twitter or Facebook.