10 Thinks you probably don't do

The resent barcamplondon3 was huge fun and particularly interesting. I might even go as far as saying it was the best yet in terms of the sessions I attended.

Particular highlights for me included discussions of (Yahoo! flavoured) web development practices and processes from Norm and Mike, Matt Biddulph on messaging, erlang a event driven programming techniques and Dan Dixon’s panel on What should we be teaching the next gen of web designers/devs?

I tried to kick off a debate regarding automation and software tools and practices in web projects. More from anecdotal evidence and experience rather than pure research I don’t think that many companies or individuals really set their projects up on a solid base. Source control, bug tracking, scripted builds, automated quality testing, etc. aren’t that common for smaller or medium web projects.

In the end I talked though everything in general terms and the audience nicely jumped in with various platform specific toolsets for unit testing, functional testing and performance measurement. All in all good fun and a subject I think will see some activity next year.

Overall a good smattering of my favourite topics at the moment (education, development process, distributed messaging), lots of fun people and the chance to ride round Google on a segway at about two in the morning. Roll on the next barcamp.

Archiving Twitter data with Python

Alex from Twitter just got round to adding the ability to export your entire archive of tweets via the API. A few people on the mailing list had been asking for this for a while so good to see it get released.

I couldn’t resist knocking together a very quick and simple Python script to go off and get all your tweets, presented here for anyone else to play around with. Note that simple, fast and works on my machine were watchwords here. Don’t expect fancy parameters, much error handling or artificial intelligence.

<% syntax_colorize :python, type=:coderay do %> import urllib2 username = ‘‘ password = ‘‘ format = ‘json’ # json or xml filename = ‘archive.json’ # filename of the archive tweets = 164 # number of tweets pages = (int(float(tweets)/float(80)))+1 auth = urllib2.HTTPPasswordMgrWithDefaultRealm() auth.add_password(None, ‘http://twitter.com/account/', username, password) authHandler = urllib2.HTTPBasicAuthHandler(auth) opener = urllib2.build_opener(authHandler) urllib2.install_opener(opener) i = 1 response = “ print ‘Downloading tweets. Note that this may take some time’ while i <= pages: request = urllib2.Request(‘http://twitter.com/statuses/user_timeline/account.'
+ format + ‘?page=’ + str(i)) response = response + urllib2.urlopen(request).read() i = i + 1 handle = open(filename,“w”) handle.write(response) handle.close() print ‘Archived ’ + str(tweets) + ‘ of ’ + username +
‘\’s tweets to ‘ + filename <% end %>

The main thing to note here though is the ideal of data portability. Let’s just say I wanted to move to Jaiku or Pownce instead of Twitter, but didn’t want to lose all that data. If I can knock up an export script in half an hour then all you need are import data API calls and a little more scripting.

As it is, I’m more interesting in seeing what I can do with my data now I can get at it. Brian just suggested a quick visualisation tool (which already eats JSON) and I’d already been thinking about language analysis and maybe having a look at APML some more. Open data is simply more fun.

Should we just learn Java?

As a wandering web designer cum developer and now occasional consultant I’m generally pretty technology agnostic. At some point or another I’ve written HTML, CSS, Javascript, PHP, C#, Python and Rails for a living - often on the same day. But something has me thinking and I thought I’d see what other people thought as well.

That thing is Java. I have books on Java. I’ve done the usual examples (hello world in 20 lines for instance). I didn’t really like it and left it at that. But a few presentations, both from Google, at both Future of Mobile and @media Ajax got me thinking. First you have Android in the mobile space which Dave Burke did a good job of making look pretty interesting. Then you had the guys from Ajaxian talking about using Java for everything.

A few other thinks to throw into the pot. Beyond Java is an interesting read - summing up with a call for more fragmentation and for small nimble languages to be used where it makes sense. Oh, and I think Tom is busy learning Java as well. Mmmmm.

Some good coverage around of both conferences and I know my first impression was basically this way lies madness. But it would have been more interesting had their been a few more Java people around to argue with.

Note that I’m not advocating a Google like disregard for the web here. Markup really matters, I happen to disagree with Douglas and think CSS rocks and I love writing Javascript. But should we just learn Java anyhow and try to subvert some of the existing tools to be more web like?

Joost Developer Day

I spent yesterday afternoon over with Joost in London as part of their UK developer day. The event was mainly for them to show off their new widget platform and to get some input from prospective developers about what they would like to see next. About 20 people or so came along from the BBC, Ebay and a couple of london agencies, with me and Tom loitering around the edges. The afternoon was split into a series of short talks from people on the team. What follows is some of the interesting bits.

Joost describe themselves as The Best of the Internet and best of TV, if you haven’t seen it yet it’s basically TV running on the Mozilla platform. All in all pretty impressive. Dan Brickley had a different slant, seeing Joost as Web 2.0 +/- 1 Built and extended with modern web technology, modern web attitude. The idea of taking an unashamedly desktop piece of software and learning lessons from the web is pretty interesting.

The Joost development site is now public, although not linked to from the main site as yet. Their is some documentation of the various APIs but we can expect more in the future, including some videos from the developer days in a development channel.

The main emphasis for developers at this moment was on overlays and widgets (little bundles of web technology) which integrate with the Joost client. We got a sneak peak (and a chat with some of the developers) of the first commercial Joost widget for Coca Cola - Coke Bubbles and Libby Miller and Jim Ley showed off a few of the default widgets.

The most interesting bits from my point of view were to do with the technology. Joost has an interesting advantage as a platform being a single, dedicated client. No more cross browser testing here. They also keep pretty close to trunk for Mozilla, which gives them quite a bit of CSS3 support too. You can also try out other new technologies not available in too many browsers yet: SVG, RDF, Canvas and E4X for example. This was the second time in a few days I’d seen SVG pushed as cool, the first being at Future of Mobile which I’ll jot down some thoughts on later.

What was interesting was that the people along from the development team came from a W3C background rather than say an product development or design companies. Joost makes extensive use of Semantic Web technologies under the hood and exposes quite a bit of RDFa if you go looking for it. It also follows along with various other web-like ideas, unique URLs for channels and programmes, including the ability to pass these parameters for instance to deep link into a video at a current time. The widget platform as well is starting to align with standards work going on at the W3C - we saw a quick demo of a widget build for the Opera browser which worked seamlessly with Joost too. The development team are now working on dealing with signed widgets, local storage APIs, interwidget communication, as well as usable, clear privacy models. All pretty interesting.

Some of the event was hampered by a shoddy internet connection but the general mood was relaxed and friendly and no one seemed to mind too much. The beer probably helped. I did chuckle when a senior alternative marketing guy from Coke described as Insane the typical agency setup though (rich guy in Notting Hill -> account manager -> project manager -> developers).

Their will be a Coke sponsored widget competition announced after the rest of the developer days (in Amsterdam and New York) with the title of International widgets champion of Joost up for grabs along with some cash and a t-shirt.

My only real complaint was probably the low level mutterings about Javascript! Audience members and Joost developers alike made a couple of digs during the course of the day. Javascript rocks, people. Oh well, I’m back in London for @media Ajax where I don’t think I’ll have that problem.

If any of this sounds interesting (or you just want to watch more TV) I’d recommend having a look through the development site, jumping on the joost-dev freenode IRC channel, signing up to the mailing list or reading the development teams blog.

Activity

Sorry, shameless what’s going on post for historical posterity.

Before that starts thought here are a couple of useful microformatted schedules for Future of Mobile and @media Ajax created by the really rather handy Conference Schedule Creator. Hat tip to Jeremy and Brian for the lovely styles.

I’m spending the next few weeks circling London, first up attending Future of Mobile, on to Pubstandards and then popping in to see Joost. After a short break I’ll be back for @media Ajax and BarCamp London. Throw in a few business meetings and we’re all set. I even have new Moo cards.

Web Geek Moo cards

If you’re around at any of these events, or just happen to be in London, and are one of the two people who read such self serving ramblings as this let me know and we can drink beer. Actually scratch that. Due to the wonder of Dopplr I already know where you’re going to be.

Some useful mobile web design links

Earlier in the year I set aside some time to get my head into the mobile web. Most of this was reading and tinkering. Since then I’ve pointed a number of people (including clients and people in the pub) towards some or all of the following links. A good starting point for anyone starting to get interested in the mobile web.

A good starting place is Mobile Web Design by Cameron Moll which is available as a PDF, and now as a hard copy from lulu. Definitely worth your while. If you’re not convinced then you can read the original series of articles to get started.

The Mobile Web Best Practices 1.0 from the W3C Mobile Web Initiative and Global Authoring Practices for the Mobile Web are more good starting points, this time with more of an implementation focus. Blue Flavor also have an intersting presentation you can download on the subject of Designing for Mobile

Once you have a basic understanding, or if you just want the facts then check out The Wireless FAQ. Everything from What is WAP? up to Nokia S60 WAP browser cursor skip some links on my WAP site! What is wrong? is in there somewhere.

If you’re too busy to be reading everything written about mobile, or just prefer listening to the voice of Paul Boag, then have a listen to Boagworld episode 96. Heidi Pollock at the recent Future of Web Apps was also excellent, especially if you let your numbers. You can get the presentation and audio of the talk now.

If you have still have questions after reading a few good articles or books then their are a couple of high quality mailing lists, namely wmlprogramming and mobiledesign. It’s worth being subscribed to both, thought their is quite a bit of audience cross over.

The .mobi site also has lots of useful information regarding mobile. Whether you buy into the need for a new top level domain or not it’s worth reading.

Lots of reading to get you started and not one mentioned of the iPhone…

Have any other good links for getting started in the mobile web? Leave a comment with further suggestions.

CSS Snapshots, CSS3 Modules and an Agile Way Forward

The state of CSS has been a common topic of conversation this year. Ever since Andy Budd stirred things up at The Highland Fling about The Future of CSS and followed up with a call for CSS 2.2 we’ve been wanting more.

Well, in response to all this the CSS3 working group have released CSS Snapshot 2007 as a working draft. CSS3, in theory, has a modular structure. The idea behind this was that individual pieces could be worked up, specified and released without having to work on the whole thing at once. Sounds a lot like iterative development to me. The problem is we haven’t, until now, had a stable release one of CSS3.

At the time this specification enters Candidate Recommendation, Cascading Style Sheets (CSS) is defined by the following specifications:

  • CSS Level 2 Revision 1 (including errata)
  • Selectors Level 3
  • CSS Namespaces
  • CSS Color Level 3

So (at least when this document reaches Candidate Recommendation at some unknown point in the future), we can get on and use all those selectors we keep eyeing up. Colors and Namespaces are not particularly interesting, at least to me, but useful non-the-less. CSS2.1 is important as a baseline going forward, especially for the browser makers, rather than being particular interesting for the jobbing web designer.

The problems come when we look at some of the features we want, and the modules they are part of. Andy specifically references text-shadow as something that’s already got more than two sample implementations but is part of the Text module which we won’t see for a while due to it’s complexity. (As a side note text-shadow was formerly part of CSS2.1 but got dropped due to lack of implementations at the time.) Dave Storey summed this problem up nicely over on CSS3.info

I guess this is due to not wanting to put partial parts of a module in this snapshot. I’d like to see another snapshot draft in around six months that includes full or partial modules that have progressed since the time this snapshot was published.

From a quick look at the current work of the CSS working group I count six modules at candidate recommendations, six at last call, twenty three at working draft and six not yet started. If modules really are discrete blocks then this to me looks like a recipe for disaster. If modules aren’t discrete blocks then why are they divided into modules? Rather than focus on shipping modules it appears that the effort of the working group has been spread out across nearly the whole spectrum of CSS3. This pretty much invalidates the idea of a modular approach.

CSS3 is a big project. And specification, like software, is hard. In well organised and managed agile software teams it’s common to split things down into features, or groups of features (modules). The CSS working group did this. In a well managed agile team shipping something early is hugely important. Everyone focuses on that goal, rather than on their favorite part of the project that might not be needed until later on. The CSS working group definitely didn’t do that. But it’s what I think is needed now.

The working group are now starting to talk more openly about what they are up to. And the CSS Eleven are looking to add some support from a web design industry point of view. I’m interested to see what they come up with as suggestions and support. The problem I see is that, unless the working group back down on the strict nature of the modules and define sub-modules or simply go to specifying features then we could hit an impasse. The best course of action (from a complete outsider anyhow) would be to get all the people from the working group and from the CSS Eleven working on one module at a time. Only once a little momentum has been built up maybe they could work on more than one at once, just not thirty five this time. All of that would take some hefty project management and some setting aside personal and organisational politics to get done. Fingers crossed.

I actually think things are looking up, but time is certainly a factor here. Lets assume IE.next is released late next year or the start of 2009 (wild speculation). If the Internet Explorer team implement to the current specs at that time - that could easily not include CSS Snapshot 2008, which would be reaching candidate recommendation around about the same time as they ship. Which could in theory mean it’s four years or more before we see text-shadow well supported. Never mind elements of CSS3 which currently have no reference implementations at all! Anyone else with any bright idea? Or insider info that you want to share?

BBC Innovation Labs 2008

Fab. It’s that time of year again. Time to start thinking “wouldn’t it be cool if” when it comes to the BBC (even if it’s not your day job).

I had the fortune of going along to this years BBC Innovation labs back in March and they’re back on the road again next year, with applications open from the 1st of December.

The basic concept is simple. Through a simple but competitive process of ideas submission the BBC find a handful of companies to dump in nice settings in the middle of nowhere for a week to work on their ideas. They are running four labs this year; in Wales and West Midlands, North East England, North West England and Scotland. If you live in any of these areas I’d strongly recommend finding out more at one of the local events.

Been couped up in a stately home or nice hotel with other odd nice people is great fun. It’s all pretty intense and last years event really got me thinking. It’s also a pretty good business opportunity. You own your ideas whether you get on the labs or not, you get paid for your time attending them and you get to pitch your ideas to the commisioners at the end of it.

For an idea of what goes on, you spend alot of time doing thinks like this:

Flip charts rule

And if you get really into it you might end up strutting around like this:

Getting in character helps too

It’s not all fun and games though. You might have to spend your time watching BBC employees play snooker on work time or playing the odd game of werewolf:

Matt takes a shot

The briefs are already up that will form the core of the submission process in order to get on one of the labs. I’d recommend having a good read, picking a few you think are interesting and tie in with whatever your interests are at the moment and go from their.

With all that in mind I’m looking for people who fancy partnering up to brainstorm ideas, put together a few submissions and see if we get anywhere. I’m now working full time on freelance and consultancy gigs but the nature of the labs was that you need a team of at least two people. And anyway, ideas flow more freely with a few smart people sat around. So, if anyone fancies sitting around talking through a few of the briefs (and who knows, maybe drinking a glass of wine?) at some point over the next few months before the deadline let me know.

PHP Asset Packager

Performance used to be something other people thought about. If you were working on a high traffic site for a large company, chances are they would throw inordinate amounts of expensive hardware at the problem. If you had a personal site only if you got really popular would you need more than a shared host. But the number of web applications being launched by small companies or individuals from their bedrooms is raising the awareness of the importance of performant websites.

Credit to YAHOO! here. Google and Amazon might be the first companies that spring to mine when it comes to thinking about infrastructure but YAHOO! have been publishing lots of useful, practical and interesting information in this area. Steve Souders, the Chief Performance YAHOO! has a new book going into detail about YAHOO!s performance rules. The recent release of the YSlow extension and another upcoming book from Stuart Colville and Ed Eliot cap things off nicely.

According to YAHOO!

Combined files are a way to reduce the number of HTTP requests by combining all scripts into a single script, and similarly combining all stylesheets into a single stylesheet. It’s a simple idea that hasn’t seen wide adoption. The ten top U.S. web sites average 7 scripts and 2 stylesheets per page. Combining files is more challenging when the scripts and stylesheets vary from page to page, but making this part of your release process improves response times.

While all that has been going on I’ve been working with a few people on a Rails project and come across a great plugin; asset_packager from Scott Becker. Inspired by the Vitamin article by Cal Henderson (of Flickr and another good book fame), asset_packager compresses and merges CSS and Javascript based on a prescribed configuration.

I’m also working on a few PHP projects and wanted the same experience in these as well, so I’ve gone a put together a quick release of php asset packager. At the moment this is a CSS only release, with Javascript features coming later, along with all the rest of the features of the Rails plugin and anything else I can think of or need.

First off you define your stylesheet setup. I used YAML as my goal was to stick as close as possible to the Rails plugin.

stylesheets:
- base:
  - screen
  - header
- secondary:
  - foo
  - bar

The above simply states that you want to merge screen and header CSS files together to form a base stylesheet, and merge foo and bar together to create a secondary CSS file.

All you then have to do is include the following:

// define("ENVIRONMENT", "production");

echo Asset_packager::stylesheet_link_merged();

If the ENVIRONMENT constant is set to production the line includes the merged stylesheet links, otherwise each file is included seperately. This again mirrors the behaviour of the Rails plugin. A command line script is also included in the source to actually merge (and tidy) the CSS files.

The vague plan is to add the rest of the features and then maybe package up as a plugin for whatever software or frameworks I’m working with at the time. At the moment that probably means wordpress, symfony and code igniter. Everything is open source and available from svn over on the Google Code site. Contributions, in the way of code, bugs, documentation or thoughts welcome.

My Trac Ticket setup

I’ve been busy really getting to grips with Trac recently and thought I’d post up a few details. Trac for those that haven’t come across it is a wiki, issue tracking systems and source code browser all rolled into one. It’s open source and written in Python.

I’ll start off describing my current ticket setup, along with the code I use for reports. In future posts I’ll hopefully describe thinks like setting up users and permissions in a flexible way. I’ll leave describing installation to others, mainly because it’s a pain. I’ll also assume you’ve installed the WebAdmin plugin, or are familiar with trac-admin.

Create a new Ticket

By default Trac comes set up with severity, milestones, ticket types, priority and components and a few default options in each. The problem with all these options is that it takes more time and effort to add and manage issues - so they don’t get logged. Unless you know you absolutely need them I find it easier to remove most of these options in the first instance.

Trac new ticket

I prefer using Trac soley for issue tracking. It suffers badly in my opinion if used for more project management related activity. Other tools do that job better (and also do bug tracking poorly).

I keep priority in, but by default it’s blank and only has one other options - High. I only assign High priority to issues which are causing major problems, either preventing a successful build or stopping something happening that’s meant to.

Available reports

Trac reports list

Trac reports are simple (or not so simple) SQL queries with a few bits of special syntax. I have four reports set up; Open, Closed, Important and My Issues:

Open, not too suprisingly, displays all the open issues. As mixing different components here would be confusing we divide the list with headers for each components

SELECT
  component AS &#95;&#95;group&#95;&#95;,
  (CASE priority WHEN 'High' THEN '1' ELSE '' END) AS &#95;&#95;color&#95;&#95;,
  id, summary, owner,
  time AS created,
  component AS &#95;component, 
  priority AS &#95;priority,
  changetime AS &#95;changetime,
  description AS &#95;description
FROM ticket t
LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
WHERE status IN ('new', 'assigned', 'reopened') 
ORDER BY priority DESC, p.value, t.type, time

Closed is the reverse of Open (I’m starting to state the obvious here) but with a few differences in terms of the listing. I’m more bothered here about seeing a timeline of closed issues. Good for keeping a eye (or a feed) on to see progress.

SELECT
  (CASE priority WHEN 'High' THEN '1' ELSE '' END) AS &#95;&#95;color&#95;&#95;,
  id, summary, owner, component, resolution,
  priority AS &#95;priority, 
  changetime AS modified,
  description AS &#95;description
FROM ticket t
LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
WHERE status NOT IN ('new', 'assigned', 'reopened') 
ORDER BY changetime DESC

My Issues lists only those issues which are assigned to me (or the currently logged in user) and open.

SELECT
  component AS &#95;&#95;group&#95;&#95;,
  (CASE priority WHEN 'High' THEN '1' ELSE '' END) AS &#95;&#95;color&#95;&#95;,
  id, summary,
  time AS created,
  component AS &#95;component,
  priority AS &#95;priority, 
  changetime AS &#95;changetime,
  description AS &#95;description
FROM ticket t
LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
WHERE status IN ('new', 'assigned', 'reopened') AND owner = '$USER'
ORDER BY priority DESC, p.value, t.type, time

Important is an overview of critical issues. Software can have small bugs and be perfectly usable in the vast majority of cases. But some issues either effect lots of users or prevent the system working at all. These High priority issues need addressing first, before anything else.

SELECT
  component AS &#95;group&#95;&#95;,
  (CASE priority WHEN 'High' THEN '1' ELSE '' END) AS &#95;&#95;color&#95;&#95;,
  id, summary, owner,
  time AS created,
  component AS &#95;component,
  priority AS &#95;priority, 
  changetime AS &#95;changetime,
  description AS &#95;description
FROM ticket t
LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
WHERE status IN ('new', 'assigned', 'reopened') AND priority = 'High'
ORDER BY priority DESC, p.value, t.type, time

Interesting bits here include the following line which makes all High priority tasks have a red border:

(CASE priority WHEN 'High' THEN '1' ELSE '' END) AS &#95;&#95;color&#95;&#95;,

As a side note, I hadn’t written proper SQL for a while (all that hacking on APIs). Suprisingly good fun as it turns out.

Conclusions

The main benefit of this setup is simplicity. Too many bug tracking systems are either way too complicated, or come with defaults which are on the complex side (I include Trac in this last category). Garret Dimon has been writing about the development of a new bug tracking system with a focus on simplicity. Fixx and Lighthouse are other products looking to fill this niche. That’s not to say larger teams don’t need to store more information than this, just that by starting small you can actually find out what you’re missing - rather than guessing and making the whole process of adding and fixing bugs a particularly painful one.