Web developers and Tools Programmers

I’m pretty interested in computer games. Building them represented a big technical challenge and with that comes interesting parallels with larger web projects. Andy Budd has talked about User Experience learnings from games previously, and I’ve heard Aleks Krotoski talk about similar themes, in particular the design of social systems and user generated content. What I’m interested in however is tools programmers.

Computer game development teams generally have a decent number of people solely dedicated to building and maintaining tools. They aren’t working on a specific title, or just on maintaining existing systems, but on programming tools for other developers in the company. Now I’m not sure how much sharing of these tools goes on between the different computer games developers, but from a cursory look around I couldn’t find any examples.

Tools programming in web development teams seems to be a different kettle of fish. I’ve never seen a job position advertised specifically for tools work, nor do I know anyone who would describe themselves as such. I am however seeing more and more of these sorts of tools make their way out into the world recently thanks to GitHub. Build systems and scripts, documentation systems, testing harnesses, linting tools, etc. All can be found if you look closely. So it’s definitely not that we don’t build these tools, though it might be that we don’t do it as part of the 9 to 5.

One of the differences that might cause this difference between games developers and web developers is scale. For the most part games development teams are bigger than your average web team or agency development team. But not always and as I said I’ve never, not once seen a web tools job going. The other reason might be Open Source. Speaking from the web side of the fence their are lots of great open source tools for building web sites and applications. If your needs are pretty basic you should be able to get away with stringing some of these together, but you’ll probably still need to do the scripting to do so. Maybe this is the same for games development; I’m afraid I don’t know.

Maybe the other reason is that as web developers we all also do a little bit of tools programming? From personal experience this is certainly true, but then I like and think about this sort of stuff more than most anyway.

So a few questions for anyone also interested in this sort of thing:

  • Do you or your colleagues do any tools development at work?
  • Is that work just done as part of other work or is it defined separately?
  • If you don’t do any tools work per se do you string available tools together using basic scripting?
  • Or do you just stick with as few tools as possible in order to make live easier?

I also have a feeling some communities are happier knocking out little tools than others. The Ruby community in particular seems busy at the moment and their is a great deal of good stuff in Java.

One last thought. How do you go about finding new tools that might work nicely in a web development world? I read far too much of the internet on a daily basis and I still miss interesting stuff more often than not. I only came across Integrity yesterday for instance.

Search for Genres on Spotify

Spotify is great. It seems stable, the desktop interface is simple and straight forward, and it has an entire album of emo bands doing covers of other emo bands. One thing that’s not quite clear just yet in the interface however is how to search for all the Rock tracks, or more specifically how to search for all the tracks in a given genre.

Turns out it’s quite simple. Just do a search for genre:Rock. My guess is there are various other textual shortcuts hidden in that search box. If anyone knows of any others post a comment.

Gmemsess

I’ve been looking at different ways of using simple sessions on App Engine, in particular for one shot flash messaging after redirects and the like. Their are some issues with Cookie support at the moment but Gmemsess solved my problem perfectly.

gmemsess is a secure lightweight memcache-backed session class for Google appengine.

Where will all the web developers go?

I’ve been thinking recently about what happens when we all get older. Now, I’m not actually referring to everyone here but more specifically what I’m going to call second generation web developers. I don’t mean Tim and friends here, or the early entrepreneurs of Yahoo! and Netscape. I mean the people who came along when the commercial web design and development industry had settled down a little bit, lets say 10 years ago. People like me.

Work seems to exist in lots of places; big in-house teams, small in-house teams, agencies, startups and freelancers. Where the jobs are at any given point seems to be tied to economic conditions and location. Most people I seem to speak to have tended to work in one of these areas, but I’ve not really got anything but vague memories to back that accusation up. I’m a little odd here in having previously worked mainly for agencies followed by a stint of freelancing, and now work for a decent sized in-house team at Global Radio.

As I see it the industry rules are predominantly made up by people involved as they go along. Their is no real impetus behind any trade body that I’m aware of, no real understanding within education and no consensus on organisational structure and jobs within it. That means, unlike accountants or people in many other professions, we’ve very little idea about what will happen in the long term.

So. What happens in 10-20 years time to the now quite large number of professional web developers.

Do we all just do the same thing we’re doing now. Just with higher version numbers? The problem with this is maintaining the challenge. If the core problems remain the same will it be much fun? I fear their are only so many times you can learn a new programming language and then solve the same problem you worked on at your last job before the world catches on.

Do we all become managers? The problem then is who do we manage? If the industry just gets bigger and bigger this works. But that sounds unreasonable. So if their are a limited number of managerial positions who gets them? The people their first?

Is their another industry that will have us? Computer games are starting to look to the web for ideas about community and collaboration, pretty much at the same time as the web is looking at games for thoughts on experience and engagement. But long term I can’t see a mass exodus or a huge cross over of people, just a huge overlap of ideas.

Do some of the areas like agencies, in-house teams, etc. disappear, or at least employ less skilled people? Experience costs money, and not always because it’s worth the extra expense.

Or is it simply that the world in twnety years time will be so different to now that we don’t really have a clue. And that a large group of computer savvy problem solvers will find something to do?

The real question I guess is what does the demand for skilled professional web developers look like in ten to twenty years? In reality most of the people currently doing the job won’t be getting to retirement age any time soon. That means every newly trained graduate or kid with a computer getting their first job adds to the size of the workforce. And how big do we think that workforce can get before it reaches a nice equilibrium? Certainly in London at the moment their are jobs aplenty. Many good friends have just left Yahoo! rather promptly and I’d be worried for them if I didn’t know how many people will be knocking down their doors.

I’d love it if their were numbers somewhere on this that you could graph. Maybe spot a plateau coming. It’s the sort of thing the Institute of Physics or the Chartered Institute of Marketing do. But as I said, we don’t have anything similar.

So maybe their is another option when the workforce has expanded as much as it’s going to do. We could get all work for The Institute of Web Development.

App Engine Samples

It’s probably old hat to those who have been using Google Appengine for a while but I just found some sample apps on Google Code. I prefer learning from actual code like this, at least until the App Engine books make it out next year.

Spyder

Stuart at work has been playing with a nice Python web crawler recently. I’ve used Harvestman before but it’s not the most straightforward thing to work with. Spyder has a really nifty callback based approach and a couple of hooks which allows you to write code like:

pre. crawler = Spyder() crawler.register(my_custom_method, ‘post_fetch_html’) crawler.run(URL)

On a side note I wish Launchpad was as clean and tidy as GitHub though. I can see GitHub adding some of the features that Launchpad has eventually, but I hope they fit them in around the existing features.

CSS Test on GitHub

I upload the work I’ve done so far on testing CSS. It’s still work in progress obviously but if you’re interested do let me know. I’ve stared with the image approach but will hopefully have something up demonstrating the selenium/rendered DOM position approach as well.

I Love GitHub Two

I’d been meaning to write a quick article about GitHub, but then Mike and Neil beat me to it and stole most of the best bits. Read both of those articles then come back if you want. I agree whole heartedly.

Now I’ve used public hosted source control before. But Google Code, Sourceforge or Launchpad never felt like this. Their were always their for people to download your code if they wanted. Maybe you collaborated with a few people on a specific project. But GitHub, through a combination of neat visualisations and social features, is encouraging people to just upload all their code. All those lines of throw away scripts. All that code written on the train learning a new language. All that configuration stuff that normally gets missed.

Talk of emerging programming languages often shifts to the need for a killer application. I want to learn more about Git so I can do more with GitHub. For me, that makes GitHub a likely driving force behind a sea of Git adoption in the coming year or so. I’m also interested to see what comes out of a reasonable sized network of like-minded people absentmindedly hacking away. Already I’m seeing people branch other peoples code within my network of contacts. Often just small tweaks and changes - like a code fairy cleaning up a few rough edges or adding an extra line here.

So, if you write code head over to GitHub and signup. Create a few repositories of the things you’re hacking on at the moment and jump through the pre-portable-social-network dance just one more time. I promise it’s worth it this time.

CSSDoc

CSSDOC looks like a good idea. I’m sure a few people mentioned this last year at @media after the talk by the people from The Guardian but nothing came from it. Hopefully tools will start to come about soon.

CSSDOC is a convention to comment Cascading Style Sheets (CSS) to help individuals and teams to improve writing/coding/styling/managing CSS files. It is an adoption of the well known JavaDoc / DocBlock based way of commenting source-code. That’s putting style, docblocks and tags together.

Django Performance Tip - Profile Your Filters

I’ve been doing some performance profiling of this here blog. Not because I really need to due to huge amounts of traffic unfortunately, more because I’m planning on releasing the code and wanted to give it a good once over before I do.

pre. 69396 function calls (67324 primitive calls) in 0.479 CPU seconds Ordered by: internal time, call count ncalls tottime percall cumtime percall filename:lineno(function) 123 0.138 0.001 0.154 0.001 build/bdist.macosx-10.5-i386/egg/textile.py:2478(qtags) 6300 0.058 0.000 0.079 0.000 build/bdist.macosx-10.5-i386/egg/textile.py:453(preg_replace) 15705 0.045 0.000 0.059 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:219(_compile) 20 0.025 0.001 0.026 0.001 /Users/gareth.rushgrove/Documents/www/django/django/db/backends/sqlite3/base.py:165(execute) 1087 0.021 0.000 0.026 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:136(sub) 123 0.014 0.000 0.017 0.000 build/bdist.macosx-10.5-i386/egg/textile.py:2701(links) 18 0.012 0.001 0.030 0.002 build/bdist.macosx-10.5-i386/egg/textile.py:892(split_text) 12719 0.011 0.000 0.063 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:178(compile) 123 0.011 0.000 0.104 0.001 build/bdist.macosx-10.5-i386/egg/textile.py:2372(glyphs) 123 0.009 0.000 0.009 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:159(findall) 191 0.008 0.000 0.008 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/re.py:154(split) 18 0.005 0.000 0.005 0.000 build/bdist.macosx-10.5-i386/egg/textile.py:803(grab_links) 11 0.005 0.000 0.005 0.000 /Users/gareth.rushgrove/Documents/www/django/django/utils/text.py:52(truncate_html_words)

The above profile is from the home page which seemed a good starting point. The majority of the other pages are more straightforward, generally displaying just one article with a few related bits and pieces.

What jumped out was that the majority of the slow traces (the list is ordered with the slowest calls first) were to do with the use of the textile filter packaged with Django. Textile is a simple markup language which converts to HTML, I’ve used it for all my writing activities since I used Textpattern on this blog many moons ago. It’s not just the traces that explicitly mentioned textile either - the regular expression calls are also to do with template filters.

I quickly added a hidden html field to my model and a custom save method. On saving an article I now automatically transform the textile version of the content and save it in the html field. In the template where I used to have article.content|textile I now have article.html. A quick check of the profile on the same page showed a dramatic increase in performance:

pre. 15054 function calls (13248 primitive calls) in 0.081 CPU seconds Ordered by: internal time, call count ncalls tottime percall cumtime percall filename:lineno(function) 20 0.017 0.001 0.018 0.001 /Users/gareth.rushgrove/Documents/www/django/django/db/backends/sqlite3/base.py:165(execute) 97854 0.003 0.000 0.006 0.000 /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py:144(deepcopy) 821 0.003 0.000 0.069 0.069

From nearly half a second (0.479 CPU seconds) to 0.081 CPU seconds - roughly a 500% improvement in performance!

If you are using text transforming filters in Django that make use of regular expressions I would look to try and get rid of them and move the conversion into a save method. You might not see quite as much of a performance gain as I did here but I’d be surprised if it’s not taking an awful lot longer than you might thing.

Their are a few tools that are handy to have around if you’re wanting to profile your own applications. Django Snippets has a handy profiling middleware and the really rather nice Command Extensions provides a few more tools including profile graph generation.