Why the webstandards world appears to be choosing Django

I’ve been noticing an interesting trend recently, not one I have any empirical evidence for mind, but one I though interesting non-the-less. Parts of the webstandards world appear to all be playing with Django. Part of this has been the odd mention down the pub, at barcamps or at SXSW this year. But the main source of information on the topic has been twitter. To name but a few I’ve seen tweets from Steve, Ross and Aral recently and Stuart and Cyril literally wont shut up about it.

What’s interesting is that this didn’t happen with Rails, not in the corner of the pub that generally talks more about markup, javascript and CSS anyway. I’ve worked on a couple of Rails projects both personally and commercially, and I’ve just launched a little pet project build with django called doesyourapi. What follows is, to my mind, a few reasons why I think this trend exists and also why I think it will continue, at least for the time being.

People

You can’t ignore the personal touch, and in Simon Willison and Stuart Langridge we already have two people who bridge the Python/Django community and the web standards crowd, at least in the UK. Personal technology choices at least are often driven by personal correspondence.

Templating

Django’s templating introduces a very simple syntax and nothing else. Rails lets you have the full power of Ruby to do with as you will within your views. Rails also makes heavy use of helpers, further adding to the complexity of views. Now I have mixed views here, based on my own skills more than anything. I know I’d feel much more comfortable throwing someone with good markup skills at a project using Django than Rails. For the most part with Django you use the html you’re used to, Rails often wants you to change this to helpers - in much the same way as ASP.NET does in fact. I think some of this comes from the Rails don’t repeat yourself philosophy obsession. Sometimes this leads to programmatic complexity which makes working with templates more akin to programming, even if it means less duplication. I’ve yet to work on a particularly complex Django project so maybe this simplicity might become a limitation to work around? Always a possibility.

Default craziness

Some of the bits and pieces that come bundled with Rails are just plain wrong, the Javascript helpers being one example. The abuse of HTTP by default in some of the scaffolding code being another. Oh, and the markup coming out of various helpers as well. In trying to help the application developer Rails gets in the way of the professional webstandards types. Django does next to none of this for you. Programmers coming from Rails might see this as missing features. Frontend types prefer this clean slate approach because it means you don’t have to fight the backend (sometimes including people) for control of the output. Note that you can work around much of this (in the same way as you can work around ASP.NET if you have to), it’s just nicer to not have to.

Other craziness

Rails people love Ruby. After all it’s better than Java (it’s also a pretty loveable programming language too). But like computer science departments everywhere many Rails people also dislike or simply put up with HTML, CSS and Javascript. If they can find a way of not having to write these and write something else (Rails people are also obsessed with domain specific languages) instead. Hence we have the likes of HAML and SASS. The problem is that us frontend loving folk quite like writing CSS (well, sometimes) and absolutely love writing HTML. Most of the time for good reasons too - just look at microformats for an example. Frontend developers tend to like using a mix of tools, predominantly backend developers not so much it seems.

Personally I find it interesting. You could quite easily flip many of these arguments around to support why so many people are using Rails. For two frameworks with similar goals and uses it’s interesting to see the early philosophical differences playing out in the real world. It might be interesting to see what happens with frameworks like Merb as well which seems to be set out to avoid many of these perceived issues with Rails. So, have anyone else noticed anything similar? Or even the complete opposite?

BarCamp NorthEast

BarCampNorthEast is go. A few people have finally got together and sorted out the long promised barcamp in Newcastle upon Tyne.

We’re going to be holding the event in the middle of Newcastle, at The Art Works on the weekend of the 24th/25th of May. That’s a whole two months away, ample time for everyone to make arrangements hopefully. From the early discussions we were always set upon going the whole hog and having a two day event. The venue is big enough for people to sleep over as well if they want which is great. It keeps the cost down for anyone visiting as well as meaning we can play Werewolf all night. We’re piggy backing on the Thinking Digital conference as well so hopefully some of the people from that will stick around for the barcamp.

We have the requisite Barcamp wiki page as well as a listing on Upcoming. Feel free to indicate your interest on either of these. Registration isn’t open just yet, but we should be opening that up next week on Tuesday 1st April at 11:00am. Look for the link on here, on twitter, on the upcoming and barcamp wiki pages and anywhere else I can think of to shout about it.

Feel free to contact me with any questions. If you’re not sure what all the fuss is about then have a look at the barcamp site. And if you’ve never been outside London before then maybe this is your chance. Newcastle is only two and a half hours away by a train with free wifi. We’re also on the lookout for a few sponsors; if your interested drop me a line.

Testing Websites with Twill

I’ve been playing with Twill a little recently. It’s a Python based DSL used for functional testing of websites. From the official website:

Twill is a simple language that allows users to browse the Web from a command-line interface. With twill, you can navigate through Web sites that use forms, cookies, and most standard Web features.

A simple example might make things clearer. You’ll need to install twill first - the instructions are available on the site. We can write tests directly into the shell so we’ll start their. For our first test we’ll write one that will hopefully fail - a test to check whether this website is down.

<code>twill-sh
>> go http://morethanseven.net
>> code 404</code>

First we fire up the twill shell then enter two simple commands. The first command, go, sends the browser to the specified URL. The second command is an assertion, in this case a check on the HTTP status code. If this website is available then it should return the HTTP code 200, if it’s unavailable then it will probably return a 404 Not Found. This test will hopefully fail, indicating that this website is up and running. In reality you’re more likely to test for the 200 status code and fail on anything else but for this example it’s useful to see what a failing test looks like.

Although pretty powerful the twill scripting language is nice and small. I’ve listed most of the commands below just to give you an idea of the sort of things that you can get up to. You can type help at the twill shell to get more information on the available commands and the individual commands themselves.

  • twill-sh
  • twill-sh {filename}
  • twill-sh {filename1} {filename2}
  • twill-sh -u {url} {filename}
  • twill-sh -n {filename}
  • go {url}
  • show
  • save_html {file-name}
  • showlinks
  • showforms
  • follow {url|url-name}
  • back
  • reload
  • showhistory
  • echo {variable}
  • formvalue {form} {input} {value}
  • submit {input}
  • show_cookies
  • save_cookies {file-name}
  • load_cookies {file-name}
  • clear_cookies
  • code {code}
  • url {text}
  • title {text}
  • find {text}
  • setglobal {name} {value}
  • setlocal {name} {value}
  • debug http 1
  • redirect_output {file}
  • redirect_error {file}
  • agent {ie5|ie55|ie6|opera7|konq32|saf11|aol9}
  • add_auth {realm} {url} {user} {password}
  • runfile {file-name}
  • extend_with check_links
  • check_links
  • check_links {regex}
  • twill-fork -n {number} -p {processes} {script}

You can also store tests in individual files as well as run a batch of tests at once. I have a couple of tests that I can run against any URL which might be a useful starting point for anyone else starting to look at testing their sites or applications. You can download these tests here.

If you unpack the zip archive and then open the folder in a terminal or console. You can then run all of the tests like so, note we’re passing the starting URL into the scripts which makes using the same scripts for multiple sites easier.

<code>twill-sh * -u http://morethanseven.net</code>

The tests included do a few things; from checking for the presence of several required markup elements and checking for an XHTML doctype to checking that all the links on the page are working.

All these examples are pretty simple and non-site-specific. For more complex form based applications you can write application browsers which fill out forms, create user sessions and do everything a user might do. Twill is also particularly useful when it comes to testing RESTful webservices with all the URLs and HTTP status codes floating about.

Own your endpoints

If URLs are people too then you better make sure you control your URLs.

Although some people use blog hosting services like blogger the majority of serious bloggers, web designers or companies generally use their own domain name to host their site. Controlling your own domain name is increasingly important when that URL is a representation of you on the internet.

With all these social networks we’re starting to have pieces of us scattered all over the place. I’ve joked previously about the utility of domain names over real names for interpersonal communication but this breaks down a little when not all the urls that represent me are owned by me. I control morethanseven.net/photos but I only have some influence over flickr.com/photos/garethr. With external hosts you also need to be aware of cybersquatting. I own morethanseven.net so no one else can use it. However with services that give you your own URL as part of registration everywhere people can cybersquat in hundreds of new domains that you might not even know about.

It’s not just web pages you have to worry about. Feeds are another example of URLs you just might want to keep control over. This last one is also something I see lots of people handing off to others - specifically Feedburner. Now I’m a big fan of feedburner and use it for the feeds on this site. But I don’t use the feedburner URL, anyone subscribing to the feeds here does so using morethanseven.net/feed/. If I decide to stop using feedburner I can, without having to upset the few people who subscribe to it by moving my feed address. It’s the same with email addresses; I love gmail but rarely actually give my gmail email address out.

So, start looking after your domain names a little more carefully. They are pieces of you scattered around the internet, and losing control of them is going to become increasingly socially painful.

Accepted for Xtech

So, while sat with a few people at the WaSP panel at SXSW (of which more later when I’m fully caught up) I got a nice email from the folks at Xtech accepting my presentation idea. The abstract is below. If any of that sounds interesting or up your street I’d love to hear other peoples experiences or ideas on the subject.

Design Strategies for a Distributed Web

From language frameworks to APIs

Everyone is making use of mature and stable web application or javascript frameworks these days. That means we’ve stopped reinventing things such as routing and object relational mapping, but we’re all still building very similar components again and again. Frameworks might allow us to solve lots of fine grained problems but APIs could let us solve common course grained problems quicker.

Building blocks for your applications

Their are already a few examples in the wild of APIs designed to be used as part of your application development process. Amazon has been leading the way in providing remote services such as S3, EC2 and SimpleDB. Their are also options when it comes to hosting these services yourself, the mint analytics software and the CouchDb database service are both good examples.

Quality engineering for free

The real value of outsourcing a discreet portion of your application to a third party API lies in quality. You could always use local storage and your programming language of choice to deal with a large volume of file read and write operations. But do you really think you’ll beat Amazon for reliability, scalability and speed?

Functionality vs Data

It’s not just high quality functionality that we could leverage from other providers. We’re all fed up with entering and re-entering our personal data into each new service. With advancements like OAuth and Microformats and lots of focus on data portability at the moment we might just be able to share data too.

Change the client as well as the server

Sometimes it’s not enough to just change the server. The rise of specialised browsers such as Joost and Songbird allows for functionality that would be impossible otherwise. Site specific browser, along with advancements such as local storage, may prove Problems

It’s not all in place just yet. The reliability of your application is likely to be important, and making use of a distributed set of APIs could leave you in the unenviable position of being less stable than your least stable partner. The issue of lock-in could also raise it’s head, without a vibrant ecosystem of different providers that is.

The Future

The use of third party commercial APIs has the potential to change the development landscape – bringing high quality middleware to the web. It could be the original web services dream realised. But without critical mass and an active market it could also be a new achilles heel for successful startups.

Generating Charts using Google Charts API

I had need recently to produce some nice looking charts and immediately turned to the very nice Google Charts API. Just before Christmas Brian had written up a great introduction on 24ways and ever since I’d been looking for an excuse.

Chris wrote up a pretty nice approach to enhancing well marked up data tables using the Charts API with a dash of Javascript and I decided to start with that. I made only a couple of changes to this approach based on personal requirements and preferences.

For a table like this:

Caption

Label

Data

Label

Data

I prefer to use the table header element to mark up the table row and the original script relied on these being td elements. A couple of changes to the javascript fixed that. I also decided to display the caption as a title on the resulting graph.

<code>
<table class="tochart">
  <caption>Caption</caption>  
  <tr>
    <th scope="row">Label</th>
    <td>Data</td>
  </tr>
  <tr>
    <th scope="row">Label</th>
    <td>Data</td>
  </tr>
</table>
</code>

The original script only supported the 3D pie charts and in a couple of cases I wanted to generate bar-charts from the data or flat pie-charts. A few modifications later and you can pass a type parameter into the script via a class on the table.

The default if the parameter isn’t set is the flat pie-chart, which can also be specified via:

<code><table class="tochart typep"></code>

For the 3D pie-chart:

<code><table class="tochart typep3"></code>

The horizontal bar-chart is created with:

<code><table class="tochart typebhg"></code>

And finally the vertical bar-chart is set based on:

<code><table class="tochart typebvg"></code>

The complete modified version of Chris’ script is below:

<code>(table2graph = function(){

  /* variables */
  var triggerClass = 'tochart';
  var chartClass = 'fromtable';
  var hideClass = 'hidden';
  var chartColor = 'FFCC33';
  var chartSize = '900x300';
  var chartType = 'p';

  var toTableClass = 'totable';
  var tableClass = 'generatedfromchart';
  /* end variables */

  var tables = document.getElementsByTagName('table');
  var sizeCheck = /\s?size([^\s]+)/;
  var colCheck = /\s?color([^\s]+)/;
  var typeCheck = /\s?type([^\s]+)/;
  for(var i=0;tables[i];i++){
    var t = tables[i];
    var c = t.className;
    var data = [];
    var labels = []
    if(c.indexOf(triggerClass) !== -1){
      var size = sizeCheck.exec(c);
      size = size ? size[1] : chartSize;
      var col = colCheck.exec(c);
      col = col ? col[1] : chartColor;
      var type = typeCheck.exec(c);
      type = type ? type[1] : chartType;

      if (type == 'bhg') {
        var label = 'chxl=1:|';
      } else {
       var label = 'chl=';
      }

      var caption = t.getElementsByTagName('caption')[0].innerHTML;
      var charturl = 'http://chart.apis.google.com/chart?cht=' + type + '&chtt=' + caption + '&chxt=x,y&chco=' + col + '&chs=' + size + '&chd=t:';
      t.className += ' '+ hideClass;
      var ths = t.getElementsByTagName('tbody')[0].getElementsByTagName('th');
      var tds = t.getElementsByTagName('tbody')[0].getElementsByTagName('td');
      for(var j=0;tds[j];j+=1){
        labels.push(ths[j].getElementsByTagName('em')[0].innerHTML.toLowerCase());
        data.push(tds[j].innerHTML);
      };
      var chart = document.createElement('img');
      chart.setAttribute('src',charturl+data.join(',') + '&' + label + labels.join('|'));
      chart.setAttribute('alt',t.getAttribute('summary'));
      chart.className = chartClass;
      t.parentNode.insertBefore(chart,t);
    };
  };

  /* convert charts to tables */
  var charts = document.getElementsByTagName('img');
  for(var i=0;charts[i];i++){
    if(charts[i].className.indexOf(toTableClass) !== -1){
      var t = document.createElement('table');
      var tbody = document.createElement('tbody');
      var data = charts[i].getAttribute('src');
      var th,td,tr;
      var values = data.match(/chd=t:([^&]+)&?/)[1];
      var labels = data.match(/chl=([^&]+)&?/)[1];
      var l = labels.split('|');
      var v = values.split(',');
      for(var j=0;l[j];j++){
        tr = document.createElement('tr');
        th = document.createElement('th');
        th.appendChild(document.createTextNode(l[j]));
        th.setAttribute('scope','row');
        td = document.createElement('td');
        td.appendChild(document.createTextNode(v[j]));
        tr.appendChild(th);
        tr.appendChild(td);
        tbody.appendChild(tr);
      };
      t.appendChild(tbody);
      t.setAttribute('summary',charts[i].getAttribute('alt'));
      charts[i].parentNode.insertBefore(t,charts[i]);
      charts[i].setAttribute('alt','');
      t.className = tableClass;
    };
  };
}());</code>

Thinking Digital

I’m starting to get excited about Thinking Digital. But first a bit of back-story.

I started out making the trip to @media 2005 and since then have been a regular attendee of more conferences than I can shake a stick at. Lets say I caught the conference bug. But most of those have been within the web standards community niche and more recently I’ve been looking further afield for conference kicks, party due to the eclectic nature of BarCamp style events. Anyone in that positions inevitably takes a fancy to going along to TED. The only problem is the fact it’s already booked up until 2010, costs $6,000 just to for a standard membership and is invite only.

This year the folks over at Codeworks, which is a regional body set up to help us North East digital folk develop an industry, are organising their very own full size conference. The Think and a Drink events have been running for a number of years and I’ve spoken a few times on web related bits and pieces. But these are purposely local events mainly for members and tend to crop up at the rate of once a month. Thinking Digital is a full size, 400 person, conference being held in Newcastle between the 21st and 23rd of May this year covering a similar set of wide ranging themes to TED; technology, design, mobile plus a bit of philosophy and futurism thrown in for good measure.

Now this is good for a few reasons. Only one of which is I can see the venue out of my apartment window. Costing only £475 and being in the UK would be a couple of other reasons to get excited. As an aside; I’m helping out as a board member along with the likes of Ian Forrester but I’d gladly pay my way if I wasn’t.

The real reason for the excitement though is the calibre of some of the speakers. Ray Kurzweil, Greg Dyke, Dan Lyons (a.k.a. Fake Steve Jobs), Matt Locke, Aubrey De Grey and Tara Hunt to name a few I’m particularly interested in. And remember folks, this is in Newcastle. Which for the few Londoners reading is a couple of hours North of you. On a train with free wifi even.

Now this might not be a web conference but I’m pretty interested in the take of the likes of Ray Kurzweil on the web and what will happen in the near future. I’m also interested in the types of people who will make the trip - which is really the reason for this post. As everyone knows, one of the best parts of any conference is the chance to chat with like-minded (and not so like-minded) folks. So, who fancies coming along?

Sorry, but the Flickr API isn&#39;t REST

After SemanticCamp me and Rey popped in to see Paul and everyone at Osmosoft towers. A good few interesting conversations ensued, including one about the difference between mashups and integration. All good fun basically. Simon also had an interesting take on the topic as well.

What has all this to do with the topic of this post? Well, Simon says:

When compared to small RESTful APIs like flickr and twitter…

Now I’m not really picking on Simon here, more that I’ve been meaning to write something on this topic for a while and this proved a good catalyst for a little rant.

The flickr API is pretty darn cool. But it’s not RESTful in pretty much any way you want to shake a stick at it. It’s a well designed RPC (remote procedure call) API. Now even flickr get this wrong. They even have a page which confuses the issue and makes REST out to be a response format based on XML. Bad flickr.

flickr states that:

The REST Endpoint URL is http://api.flickr.com/services/rest/

This turns out to be completely against the whole RESTful principles. Lets try and explain. You can define most of what you want to do with an API with the use of nouns, verbs and content types. REST is based around limiting the set of verbs to those available in HTTP; for instance GET, POST, PUT and DELETE. For any given application or API you’ll also specify a set of content types (representations); for instance HTML, JSON or XML. All you get to play around with are the nouns, in the case of the web these are our URLs.

In a typical RPC style API you have one URL (like flickr does) which acts as the end point to which all calls are made. You then probably define a few functions which you can call. Lets look at a simple book example.

<code>getBook()
deleteBook()
createBook()
editBook()</code>

The RESTful way of designing this might look a little bit more like this:

<code>GET     /books/{book-id}
DELETE  /books/{book-id}
POST    /books
PUT     /books/{book-id}</code>

We mentioned content types or representations earlier. Lets say instead of the default response format we might want to get a JSON representation of a given book we might do something like the following.

<code>GET /books/{book-id}.json</code>

The advantages of using URLs as the API nouns in this way include more than just sane URLs for the site or application in question. The web is based around many of these architectural principles and that seemed to scale pretty well. The idea is that fell envisaged RESTful applications have an advantage here too. For me one of the real benefits of RESTful APIs are in the simplicity they bring to documentation. I already know the available verbs, all I need to know are the set of resource URLs and I can probably use CURL to work out the REST (sorry, bad pun).

This misunderstanding is pretty common. Even WikiPedia appreciates their is a problem:

The difference between the uses of the term “REST” causes some confusion in technical discussions.

This isn’t just pedantry on my part, well not quite. I’d recommend anyone involved in designing and architecting web sites read RESTful Web Services as well as Roy Fielding’s Architectural Styles and the Design of Network-based Software Architectures. But if you just want to get the general principles and don’t fancy wading through technical documents then you have to read How I Explained REST to My Wife hilariously written by Ryan Tomayko.

And remember, just because an API makes use of HTTP doesn’t make it RESTful. I’m looking at you twitter.

Invited to Join WaSP

I had a great time at SemanticCamp over the weekend which was not too much of a surprise. What was a surprise was getting back home to find out that I’ve been invited to join the Web Standards Project (WaSP) on the Education Task Force. I even have my own page.

Thanks for inviting me guys. Anyone who has had the misfortune of having me present during a discussion of the current state of web education knows it’s one of my favourite subjects. Hopefully I can make myself useful around the place and help with getting a few things done. More on this when I know more but in the meantime feel free to pester me endlessly if you have a particular axe to grind.

Example of the Yahoo Live Api

Yahoo! Live launched recently along with a nice RESTful API. I’ve spoken before about the beauty of REST being in lowering the barrier to hacking and when I wanted a quick feature for Live it was simplicity itself to put together.

A few friends are using it far too much it seems, Ben has 7.6 hours and Si has already clocked up 15 hours. But for the most part I keep missing their no-doubt highly entertaining antics. One thing that Live misses I feel is a favourite users or previously viewed channels list. Basically I want to see which of my friends who use the service are broadcasting right now. Something like:

Yahoo! Live Online

The API request we’re interested in is the /channel/PERMALINK method. This lets us get information about whether the user is broadcasting at the moment.

<code><?php
$api = 'http://api.live.yahoo.com/api/v1/channel';
$friends = array(
  'garethr',
  'benward',
  'sijobling'
);
$statuses = array();
foreach ($friends as $friend) { 
  $response = simplexml_load_file($api . '/' . $friend);
  $name = $response->name; 
  if ($response->broadcast) {
    $status = 'live';
  } else {
    $status = 'offline';
  }
  $statuses["$name"] = $status;
}
function displaylist($array) {
  $output = '';
  if (count($array) >= 0) {
    $output .= '<ul>';
    foreach ($array as $key => $value) {
      $output .= "<li class=\"$value\">";
      $output .= "<a href=\"http://live.yahoo.com/$key\">";
      $output .= "$key</a>";
      $output .= "<span>$value</span></li>";
    }
    $output .= '</ul>';
  }
  return $output;
}
echo displaylist($statuses);
?></code>

I’ll add a few more people to my list when I discover other people using the service. If you have an account leave a comment. I’ve added a touch of javascript as well so as to avoid having to reload the page manually. This way I can loiter on my little aggregator until someone I know starts broadcasting and head over to Live for whatever Si has been spending 15 hours doing.