On Process and Design

I consider myself something of a hybrid, flirting with both the design and development side of things. I also like to think of myself as something of a student of the discipline; reading and consuming as much knowledge as possible on the process as well as the practice. Something I took to pondering while reading My Job Went to India was why the majority of theory books tend to be from the software side?

It’s appears the same when it comes to discussions of process, and in turn innovation within process. Their is lots of talk around Agile processes and practices amongst software developers, but I don’t see lots of designers discussing the how. User Centred Design is maybe a slight exception, but again it seems to be the software people and information architects, as apposed to those who consider themselves designers, who talk about this. I’m just wondering whether others have found this too?

The Design Council published a pretty good resource About Design but I haven’t seen much other real literature on a modern design process. Communicating Design is a fantastic book on design documentation, but doesn’t frame this in terms of a discussion of process. If anyone knows of any books or blogs out their on the subject I’d love to know about them. I’d love to be proved wrong and end up with a good reading list.

My Job Went to India (which is excellent by the way) contains 52 hints and tips for your career as a programmer. In reality most of these ideas apply to any creative discipline, only the examples and analogies need changing. You could probably argue the same for books like The Pragmatic Programmer, Practices of an Agile Developer and Peopleware. I’m also thinking whether or things like the mythical man month apply? Or if other truths of the design practice exist?

The importance of design in any discipline is becoming increasingly well recognised. But where will the next generation of managers learn about how to manage designers and the design process? One of the problems with front end web development and design work has been the difficulty of learning the craft. Is this the same for managing the web design process?

How to deploy PHP sites with the Pake build tool

So in case you hadn’t guessed project automation is the new black. I’ve been getting back into some development work recently with Is it Birthday? and getjobsin and trying to automate as much of the repetitive and boring work as possible.

I’m not absolutely sure that that many people outside large or particularly organised teams realise that large web sites are not generally deployed by someone with an FTP client and crossed fingers. This sort of effort, along with other repetitive tasks like running tests or generating documentation, can be automated. This is a win-win for everyone. It’s more reliable (scripts don’t forget to restart a service before they go home), quicker and removes the need for having one person in charge of deployments.

Even where people know about this build process idea they might not use it for their projects, probably for similar reasons why not everyone uses source control. I think the reason is probably that most web designers and developers (even particularly geeky ones) thing this software engineering stuff is maybe a step too far. Their is also something of a barrier to entry, knowing where to look and how to get started without reading lots of documentation (often filled with XML examples) and trial and error. Also project automation is apparently not sexy?

Anyway, if you’ve been working with Rails you will have come across Rake, which is a build tool used to automate various tasks. Well the nice symfony people have written a PHP version called Pake for use as part of the framework. It’s used for all the command line action, from running tests to clearing the cache and automating deployments. Pake is however a separate tool that you can use in your own projects, whatever framework or hand rolled codebase you are using.

Pake can be downloaded using PEAR on the command line:

<code>pear upgrade PEAR
pear channel-discover pear.symfony-project.com
pear install symfony/pake</code>

The documentation for Pake is pretty much non-existent as far as I can tell, but it is a really handy tool so worth a little effort. The best source of knowledge is to look through the default Pake tasks that are provided as part of symfony. One of my favourites, which we’ll look at now, allows for incremental deployments via Rsync over SSH. I’ve been using this with non-symfony projects too.

Rsync is a command line tool for syncronising two file structures. The Rsync command that does most of the heavy lifting for the deployment looks like the following. Note I’ve used {} to denote placeholders in the following examples.

<code>rsync --progress --azC --force \
--delete -e ssh ./ {user}@{host}:{dir}</code>

The sync task I’m using is straight from symfony, but the licence allows for distribution so here is an example zip of all the files needed to follow along. You’ll need these to follow along as I haven’t printed the full sourcecode for the pakefile here.

First a little configuration. Using YAML we define an environment, staging in this case_ and specify a host, port, user and the full path on the remote server. You can of course specify multiple environments in this file, we’ll see how to use them shortly.

<code>[staging]
  host={host}
  port=22
  user={user}
  dir={dir}</code>

You can also include an rsync_exclude.txt or an rsync_include.txt file. This gives you control over the files being synced when you run the Pake task. The following example is a good starting point, it stops you pushing those pesky .DS_Store files that OSX creates to you web server, as well as avoiding subversion metadata and the configuration files for the Pake task.

<code>.svn
.DS_Store
/config/properties.ini
/config/rsync_exclude.txt
/config/rsync_include.txt
/config/rsync.txt</code>

We can now run the following command, from the directory containing the pakefile.php script, using Pake. The first example will do a dry run, showing you what will happen. You’ll be prompted for your SSH password as part of the command unless you’re using keys for authentication.

<code>pake sync staging</code>

When you’re happy you can run the sync command with the go option which will instruct Rsync to do it’s thing.

<code>pake sync staging go</code>

Pake has a handy flag to find out what tasks are available.

<code>pake -T</code>

This should give you a list of tasks and a brief description, useful to find out what you can do if you’ve been away from a project for a while.

This is a pretty simple example but one I’m already finding useful. Rsync is but one way of deploying apps but with Pake has the advantage of being simple and in lots of situations good enough. It’s certainly better than a manual deployment process. It would be simple enough to build into the task a simple logging system so you have a log of all deployments; when they happened and who did them for instance.

If that has whet your appetite then their are other deployment tools you might want to look into; Capistrano (Ruby), Ant (Java), Maven (Java) and Phing (PHP) spring to mind. If anyone knows of a Python equivalent that would be useful too? I’m also using Phing for a few tasks on projects at the moment, mainly for some nifty Subversion tasks (and you can use Phing with Pake as symfony does), but that will have to wait until another post.

So, what are peoples experiences of build tools? Any good pointers? Or maybe reasons why you don’t use them in your projects?

Refresh Map

It turns out that their are now nearly 40 Refresh groups scattered around and about. Here’s a quick map of them all. It’s pretty interesting how all these groups have picked up on a simple word.

For anyone in the Newcastle are we’re having a meetup next week, on Thursday evening, as it happens, the details are on Upcoming if you fancy coming along.

Things to entertain us in 2008

This being a blog and it being the start of a new year (2008 for those not following closely) I feel obliged to list of few predictions of sorts. Rather than concrete this will happen things here are a few bits and pieces I think will be interesting over the next year.

Site Specific Browsers

A few people have already waxed lyrical about Site Specific Browsers and after trying out Prism for a few weeks I can say I really like the idea. Their are a couple of teething problems, both with the idea and the current implementations, but all seem solvable with a little focus. Security (sandbox everything), memory footprint (as small as possible) and integration with the desktop (local storage, Google Gears).

I think their are already a couple of extreme examples of this idea in the wild, both of which I’m already pretty interested in. Joost are doing some pretty cool things with their bespoke application and web site backend and Songbird just hit version 0.4 and is looking pretty nice.

Mobile

We’ll see more and more people playing with mobile apps next year I’m thinking. Mobile browsers (special mentions to Opera Mini and Mobile Safari here) are getting better and better and the iPhone (at least in the UK) has hopefully started something with regards fixed cost unlimited data. With some good materials already in print, next years Future of Mobile likely to be bigger and better than before and half the web developers I know pondering how they get to work with mobile I see more cool stuff happening here.

The Backend

Chris has made a call to arms (of sorts) and I’m hoping this comes to pass. It’s becoming more and more apparent that the best teams are agile and multi-disciplinary and both sides of this divide have lots to gain from each other. For starters most front end people could probably do with learning some good old fashioned software engineering stuff; build tools, source control, deployment, testing, etc. As for the backend guys; how about unobtrusive applications, accessibility and the importance of good quality markup? Apologies for the huge generalisations here, but hopefully you know what I’m getting at.

Software as Infrastructure

Simon Wardley and others have been talking about software systems as infrastructure for a good while and I think we’ve seen glimpses of this in the last year. Whether it’s people using Twitter for interesting projects (Foamee for instance) or Amazon and their web service platform (S3, EC2, SimpleDB, DevPay) we have a start. I think we’ll see lots more of this sort of drop in web based software over the next year. The key seems to be thinking about things in a RESTful manner, and throwing in as much publicly exposed JSON, RSS and Microformats as possible. It will also be interesting to see if the Atom Publishing Protocol takes off next year too.

Education

Education is coming up more and more in conversation between people involved in web design and development it seams, both off and online. I had a great conversation with Rachel and Mike after @media and then another good discussion with Dan Dixon, Andy, Tom and others at barcamp and know other smart people are thinking about this too. I’m hoping to get something off the group here in the next month or so but would love to know about any other industry driven initiatives anyone might know about?

Ok, so that’s more a list of things I think I’ll end up focusing on (as much as I ever focus on anything) over the next year. We’ll see how that goes and if other things crop up that are more interesting. What about you? Anything you think is going to be play worthy next year?

Is it Birthday?

Did you love the simplicity of Is it Christmas? but feel left out, wondering why Christmas should be more important than, say, you? Well here’s your answer; Is it Birthday?

Is it Birthday? home page:http://isitbirthday.com

Nice bits of symfony: Web Debug Toolbar

I mentioned previously that I’d been playing with the symfony framework on a couple of projects and I have to admit to being more and more impressed as I wade in looking for things. I’ve worked with enough MVC-like frameworks to know they are all quite similar in more ways than they are dramatically different. So it’s the little bits and pieces that win you over.

One of my favourite little bits in symfony has to be the Web Debug Toolbar. When running in development mode you get a little menu floating above your application like so (top right hand corner).

Web page showing the symfony web debug toolbar in the top right corner

This gives you all sorts of useful information as you’re developing your application - from the number of database queries to the rendering time, with a full logger built in. It’s also been particularly useful in learning symfony - being able to follow the entire request through the stack from the logger is really handy when you don’t know what is going on.

Web page showing the symfony web debug toolbar fully expanded

With the increasing use of object relational mapping in web frameworks it’s quite easy to end up with a working application on your development box that turns out to absolutely hammer your database when used in the real world by only a few hundred people (I’m looking at you tagging). The database utilities, showing the number of queries and the SQL for those queries, is particularly handy for finding bottlenecks here.

Web page showing the symfony web debug toolbar SQL view

I’ve not come across something similar in other frameworks, PHP or otherwise. I’d be interested to know if similar things do exist for Rails or Django in particular.

Happy with my eeepc

I like small. I have a macbook rather than a macbook pro because it’s smaller. Especially now I’m wandering about more often with work (and play) lugging a machine around is less and less appealing. So, armed with that excuse I felt justified in going out and buying a new Asus Eee PC. I managed to just walk in to Micro Anvika in Newcastle and buy one from a bemused staff member who was pretty shocked to find one in stock and not allocated (sorry Rey).

For anyone who hasn’t seen one before, the Eee PC is a tiny linux laptop a little like the One Laptop Per Child machine but aimed more at traditional computer users (at the moment read early adopting geeks). The distro is setup to be pretty user friendly for those pesky Windows users but a little hacking turns it into a pretty darn nice machine.

My new eeepc with an apple logo on the back

And yes, I have stuck an apple sticker on back. I’m sure that should get some entertained looks next time I’m at a relevant event! I think the machine itself is actually pretty good looking, but the default desktop themes are all pretty uninspiring.

I’ve stuck with the appliance like IceWM window manager and the launcher application as it makes starting things up from the keyboard with one hand pretty fast. I then jumped into various hidden away files to get the desktop setup how I like. The wiki over on eee user has a number of useful tutorials; customising easy mode and the debian repo from a ruby tutorial were particularly handy.

Minimal IceWM desktop with Asus Launcher

Still lots more I could install (this is Linux after all) but so far have installed Ruby, Rails, Python, Django, Erlang, MySQL, SQLite, Apache2, PHP5, PEAR and symfony all without a hitch. I do love apt when it just works. I’ll try and get Mono and Java on their as well just for completeness when I get a moment.

The only thing I can think of that I’ll really miss at the moment is Keynote. It has a VGA out port (which not even the macbook has) which will support a decent resolution for presentations. But Keynote is so pretty and easy to use compared to the alternatives. We’ll have to see next time I have to do a presentation.

I can see lots of places (past the obsessive tech geeks like me) where a machine like this will be really interesting. Obviously the low cost of the machine means that schools might just get interested in Linux. For people who prefer a desktop machine as their main computer and only occasionally need a laptop it’s a pretty cost effective option. As a machine to experiment with Linux it’s pretty useful too - I’d recommend being familiar with basic commands on Linux to anyone doing web development work for instance. With so many possible audiences, and what is already a pretty nice machine, Asus might be onto something here.

CSS Uprising

For those not following along closely I thought it worth highlighting some interesting goings on around the web. Andy Clarke put the CSS cat amongst the W3C pigeons with a couple of posts; CSS Unworking Group and a following up CSSWG Proposals

How I’ve had a few ideas myself previously around what I’d like to see from the CSS3 working group. I’m now avidly following along with the comments and I’d urge anyone working even on an occasional basis with CSS to do the same.

Maciej Stachowiak said a few very interesting things in the comments I thought worth commenting on:

Being an expert user is not necessarily enough to know how to extend it. A key part of extending a technology is knowing how it works on the inside.

I think their-in lies an opportunity. The weight of comments on this issue, and the heat of the argument, would suggest people are interested. But I think a-lot of those interested parties are those working day-in with CSS (rather than on it) and lack this critical information. As a web designer how do I find this information out? Who are the interesting bloggers in the area of standardisation? Is their are good reference site? Or a wiki? Or an active friendly mailing list of people willing to answer simple questions? The web (standards) design community is used to open discussion and debate - whether from blogs or projects like microformats.

You could of course also argue that being an expert implementer is not necessarily enough to know what needs extending. A key part of extending a technology is knowing what is needed first in the real world.

But the cost of a serious mistake in a standard is much higher than the cost of an implementation bug.

I think this depends on the software in question - what about critical military systems, satellites or space shuttles, power stations, etc. - but I agree in principle. Most web designers do not have the experience of working on this sort of software project. But then Andy is not putting himself up as a project management guru.

Their are obviously strong feelings on both sides of this debate. But something I would be interested to hear - from someone with a standards focus - is whether you think their is a fundamental problem at the moment with regards CSS? If not then why not? If so then what are your ideas for moving forward?

And feel free to say you don’t know yet. I think what Andy has done is put down his ideas. If you don’t like them then suggest better ones. If that’s sticking to the status quo then say why you it feel will work out.

(I’ve posted the above in the comments to the original post, but wanted to repeat it here more in the hope of getting more interested people involved in that thread.)

I don’t really expect Andy to have the answers to this issue, but by him kicking the discussion off in such a public way the least we can all do is have the debate we need and come up with the solutions for everyone. Especially on the list of getting started with standards links and reading materials alluded to above, if someone has this information but not the time to get it online in a coherent fashion, or to present it to the wider community then please get in touch.

Using Curl to play with CouchDB

As a big big fan of all things HTTP, the new Erlang based database, CouchDB, piqued my interest. With the recent release of 0.7 it’s now intended for widespread use. Now I’m a fan of databases as long as I don’t have to go too near them. SQL, triggers and stored procedures are all a little too close to magic for me.

The reason CouchDB looks particularly good fun was it’s build around a REST API using JSON for data transport. I’m generally not an acronym guy, but REST, JSON, API and HTTP in one open source piece of code? Sounds like fun to me.

The CouchDB community have built up an already excellent wiki with all the information you need to get started and get the software installed. I just used the magic of MacPorts but the page covers all the various dependencies and setups (though nothing on their yet about the N800).

<code>sudo port install couchdb</code>

You might then need to set-up a user and some permissions. Again the wiki has more detailed installation instructions. All being well you should then be able to fire up CouchDB:

<code>sudo -u couchdb couchdb</code>

So first things first. Their are getting started code examples on the wiki in all the usual languages so you can just dive in. CouchDB also comes with a few in-build tools which are both pretty attractive and useful. You have a database browser, a full test suite and a command shell.

Curl, for those not familiar with it, is a fantastic command line utility for throwing around HTTP requests. I already use it for testing and generally prodding sites but it’s absolutely perfect for demonstrating CouchDB. Let’s start with creating a database called test:

<code>curl -X PUT http://localhost:8888/test/</code>

That should have created the database. Let’s get some information back about the database:

<code>curl -X GET http://localhost:8888/test/</code>

Everything should be returning JSON strings telling us something about what it finds. Let’s add a blank document to the database:

<code>curl -X POST http://localhost:8888/test/ \
-H "Content-Type: application/json" -d {}</code>

Note the blank JSON object {} passed as Post data. What about retrieving a list of document from the database?

<code>curl -X GET http://localhost:8888/test/_all_docs</code>

We’re now done with that database so let’s delete it.

<code>curl -X DELETE http://localhost:8888/test/</code>

I’ve just done the basics here, but the APIs are simple and nicely documented enough for you easily to find out more. This is the beauty of RESTfully designed APIs, they are just about self documenting. For the most part you just have a predefined HTTP method and a defined URL.

The Erlang base of CouchDB makes for some interesting horizontal scaling possibilities (as well as a good excuse to play with Erlang.) Even if CouchDB wasn’t cool enough already, it comes with probably the best start-up message I’ve seen in software for ages:

<code>CouchDB has started. Time to relax.</code>

Insert Dojo and YUI bookmarklets

I’ve talked about having fun with jQuery using Jash and Firebug before after seeing Simon throw the Google homepage around at barcamp. I’m no more a one javascript framework person than a one programming language person and recently I’ve seen cool things I want to play with in lots of the main contenders.

I’ve put together a couple of bookmarklets which load YUI or Dojo from their respective content delivery networks and insert them into your current browser context. You can then play around with them in Jash or Firebug or any other Javascript console.

Dojo is particularly interesting here due to it’s cunning loading mechanism for additional parts of the framework. Basically you can gain access to the entire (huge) framework simply by using the bookmarklet and then dynamically loading the rest via the command line.

Insert Dojo

<code>javascript:void(function(){
 var%20s=document.createElement('script');
 s.src='http://o.aolcdn.com/dojo/1.0.0/dojo/dojo.xd.js';
 document.getElementsByTagName('head')[0].appendChild(s);
}())</code>

Insert YUI

<code>javascript:void(function(){
 var%20s=document.createElement('script');
 s.src='http://yui.yahooapis.com/2.4.0/build/yahoo-dom-event/yahoo-dom-event.js';
 document.getElementsByTagName('head')[0].appendChild(s);
}())</code>

The code simply inserts a script element into the page which loads the relevant framework. I’ve shown the code on multiple lines for the sake of the example, but your bookmarklet should be on one line. Alternatively you can drag the Insert YUI or Insert Dojo links to your bookmark bar.