Smoke Testing With Cucumber On Sysadvent

I wrote a quick article last week for the excellent sysadvent advent calendar, Smoke Testing Deployments with Cucumber talks a bit more about using a few of my favourite tools to check whether a deployment just broke anything important.

Sinatra On Glassfish Example

I magically turned into a Java developer last week for a bit when I had to do some integration with a SOAP based API that really really wanted me to write Java on client as well. That led me down the route of having a good look at Jruby (which I’ve used before, mainly for testing using celerity) and in particular how easy it was to use native Java classes in Jruby (very, very easy as it turns out).

All that meant I’ll probably end up writing a nice Jruby application in the not too distant future, and not knowing too much about running such a thing in a production environment I thought I’d take a quick look. I went with Glassfish as the application server for no other reason that it took my fancy. I’d definately be interested in hearing about any positive or negative experiences people may have with it or other similar servers. My quick look turned into running a tiny Sinatra application.

First install the required gems for our little experiment. You’ll obviously need jruby which is sort of the point, I’d recommend using RVM for that.

<code>gem install sinatra warbler

Now create a sinatra app. OK, it could be any Ruby rack based application but I like Sinatra. First we need a rackup file.

<code># config.ru
require 'init'

set :run, false
set :environment, :production

run Sinatra::Application

Now for our application itself.

<code># init.rb
require 'rubygems'
require 'sinatra'
get '/' do
  "Hello World!"
end

Warble is the gem we’re going to use to create a WAR file, which is basically an all in one bundle of our application and it’s dependencies which we can deploy to a java application server.

<code># config/warble.rb
Warbler::Config.new do |config|
  config.dirs = %w(config)
  config.includes = FileList["init.rb"]
  config.gems += ["sinatra"]
  config.gems -= ["rails"]
  config.gem_dependencies = true
end

Now we’re ready to generate our WAR file.

<code>warble

This should create a file called sample.war or similar. Then just deploy that to your application server and away you go. I got this working very easily with Glassfish which seemed to be the recommended tool for such things. Installing Glassfish was time consuming but well documented here. Uploading to Glassfish was done via the web interface for the moment. I just selected a Ruby project from the deployment drop down and uploaded the war file.

Devops Weekly

I’ve really been enjoying Ruby Weekly recently, it’s an email newsletter by Peter Cooper which brings the latest Ruby related news and articles to your inbox.

I have to admit to being sceptical at first about the format, I think I unsubscribed from most email newsletters many years ago, moving my reading to RSS and then Twitter. But I’ve actually found a regularly appearing email a great way to catch up with the goings on. A think the reasons for that change of heart are:

  • Their is simply more being written now that even a few years ago, and I’m interested in more topics.
  • For a short time Hacker News solved this problem for me, but now it’s too high volume for me to subscribe and read everything.
  • I now manage my email inbox pretty well, I’m not always at inbox zero but I can’t remember the last time it was over 20.
  • RSS is great, but it’s always a limited view unless you follow 1000s of feeds.
  • Twitter is great but unless you read every tweet you’ll miss some important bit of new.s

I think Ruby Weekly works because it’s collated by someone with taste. Peter spends a bit of time putting together a small number of high quality links so I don’t have to. With Devops Weekly I’m hoping to do the same for a different niche.

So head over to devopsweekly.com to sign up. I’m not 100% sure yet when the first issue will go out but expect it in the next few weeks, and after that I’ll try and get one issue out each week. Any ideas are more than welcome, as are any news or articles that you think should go in.

Books For People Interested In Devops

Before starting with FreeAgent I decided I should spend a bit more time with Ruby and set about building something I’d been thinking about for a while. I’ve just launched the first one of my related pet projects so thought I better link to it from here.

Devops Books is exactly what it sounds like; a list of books that people interested in the whole devops concept should read. It’s not a complete list just yet and I’ll try and keep it up to date as new interesting books get released. Any suggestions to add to the list most welcome.

It gave me a proper excuse to use Heroku which has been very pleasant. Under the hood it’s a very simple sinatra application using the Mustache template language. The majority of the code is a build script that pulls down information from the Amazon API and mixes it with my own content. It then creates a JSON document that is used as the basis for generating the pages. I love static generators and hand rolling your own for a particular domain is often worthwhile. It means I have the json already lying around if I want to make a JSONP style badge for instance.

I plan on using the code as a bit of learning tool. Try out some different testing approaches, maybe add in GEOIP detection, make some of the commands I’m running into Rake tasks, that sort of thing. Given how easy it will be to throw up little sites like this using the generator I have a few other similar things in mind too.

Why You Should Be Using Virtualisation

My main development machine for a while has been an apple laptop. From looking around at conferences, offices and usergroups I know I’m not alone. But I don’t really run code on my mac very often, certainly not for work. I might edit the code on my mac but I execute it running in a virtualised Linux environment matching (as close as possible) the production environment it’s going to end up in. This blog post is an attempt to explain why this is a good idea to the majority of people who develop on a mac (or a windows machine) and deploy to something else. This isn’t language specific either. You might be working on small PHP web sites or huge Python applications, you’ll still one day run into the same issues.

Why is virtualisation a good idea?

Bugs happen, but catching them early, way before they even hit your shared source code repository makes them much less of an issue. Catching bugs only after a live release, when they affect customers and cost someone money is bad. And if your release is a long period of time after the work was done then fixing them is harder to boot. These are just some of the reasons we’re all fond of unit testing and continuous integration.

But if you’re running those tests against code executing on different hardware, on a different operating system, with different low level libraries or a different web server version or a different database server then you are not going to catch all the problems. If you take this to an extreme then you can only get rid of all of these problems by giving each developer a full production stack of their very own. This is obviously impossibly expensive for anything past the most trivial setup. But eliminating even some of these issues makes it more likely you’ll catch bugs early and less likely you’ll have a bug on your hands that you can’t recreate locally. So we’ll aim for production like rather than a 100% copy.

Here’s a real example, a case insensitive file system. Grab a terminal prompt on your mac and type the following in an empty directory. Then do the same on a typical linux machine. All we’re doing is using touch to create a file.

touch Test
touch test
ls

On you’re mac you’ll probably see:

Test

On your linux box you’re more likely to get:

Test
test

What? The mac treated Test and test as meaning the same thing. It wont let you have a file called test and one called Test in the same place. It’s case insensitive. But the linux machine didn’t have this problem. Now imagine we’re not dealing with empty files called test but either files you’re running code is creating at run time (a file cache maybe or a user uploaded file) or even more interestingly your source code files. Lets say you have git on a linux box in the corner of the office and someone checks in two files from a linux machine called Pages_controller.rb and pages_controller.rb. What happens when you get these to you’re mac? I haven’t actually tried this but it’s not going to end well. And imagine debugging this sort of issue? If you think this is all hypothetical I know about this little quirk exactly because I saw someone trying to fix a bug related to it.

What if the bug was because you had one version of lib_xml on your local development machine and a different one on production. Up to that point you might not even know what libxml was or how it got on your shiny apple laptop.

How many people can genuinely say they have never had a bug they could recreate on live and not on their development machine? Same code, different behaviour. Load and data often play a part in bugs like this as well but these can be isolated and tests created in at least some cases. Being pragmatic what we’re aiming for isn’t to eliminate all differences, it’s to get rid of those that are easy to eliminate.

How can I do this?

Virtualisation tools used to be cumbersome and expensive and generally not aimed at consumers. I’ve used both VMWare Fusion and VirtualBox on my mac and even compared to a few years ago these tools are increasingly easy to use. And VirtualBox is free and open source too. On top of that we now have tools like vagrant which I’ll give a quick example of here.

Vagrant for those that haven’t come across it yet describes itself thus:

Vagrant uses Oracle’s VirtualBox to build configurable, lightweight, and portable virtual machines dynamically

What it really is is a tool for quickly and painlessly building virtual machines based on sensible default configurations, and then providing programatic hooks for more advanced configuration. For instance you’ll have a configuration file to describe which ports you want forwarded and you can use Chef to install packages when the VM first boots. Once you have it installed it’s as easy as this to use

vagrant box add lucid32 http://files.vagrantup.com/lucid32.box
vagrant init lucid32
vagrant up

The first line downloads a 32bit Ubuntu disk image but you’ll only need to do that once. You’ll find lots of images for different distros too. The next two lines create and then boot a new headless virtual machine. That’s it.

vagrant ssh

Will let you jump straight into a ssh session with the new machine, for an idea of what else it can do here’s the help output:

Tasks:
  vagrant box                        # Commands to manage system boxes
  vagrant destroy                    # Destroy the environment, deleting the create...
  vagrant halt                       # Halt the running VMs in the environment
  vagrant help [TASK]                # Describe available tasks or one specific task
  vagrant init [box_name] [box_url]  # Initializes the current folder for Vagrant u...
  vagrant package                    # Package a Vagrant environment for distribution
  vagrant provision                  # Rerun the provisioning scripts on a running VM
  vagrant reload                     # Reload the environment, halting it then rest...
  vagrant resume                     # Resume a suspended Vagrant environment.
  vagrant ssh                        # SSH into the currently running Vagrant envir...
  vagrant ssh_config                 # outputs .ssh/config valid syntax for connect...
  vagrant status                     # Shows the status of the current Vagrant envi...
  vagrant suspend                    # Suspend a running Vagrant environment.
  vagrant up                         # Creates the Vagrant environment
  vagrant version                    # Prints the Vagrant version information

I’ll leave it their as this post is more of a rant than a tutorial, but I might write more about using vagrant later. But in the meantime read the web site for a pretty simple walkthrough. And don’t be put off by the fact it’s written in Ruby or that the example shows a Rails app, this is a great tool whatever language you’re going to be using on the virtual machine.

Arguments against

I see too few developers doing this for it to just be about a lack of awareness. Lots of developers not doing this might be running local virtual machines for cross browser testing for instance. Here’s a few complaints I’ve heard and what I think the answer is.

Speed

If something is slow and you don’t have as much RAM as you can get into your machine then do that before complaining about anything. Running a few extra operating systems inside your main operating system is obviously going to be intensive so don’t scrimp on your tools. Also the defaults when setting up new virtual machines in VirtualBox or VMWare Fusion at least are quite minimal. Try increasing the ammount of RAM you let them use or give them access to more processes. I can genuinely say I’ve had a problem with this once and the real solution was changing the code, not throwing away all the advantages of virtualisation. If you’re doing some crazy real time video processing thing then your mileage will vary, but then you probably want a faster machine anyway.

Lower level that you’re used to

As a PHP/Ruby/Python developer why should I have to care about Apache? I just write code!

This argument just bugs me, but I do know part of that is me. I need to know how all the bits work and fit together and I accept not everyone does, or indeed needs to understand everything. But someone on your team probably wants to know this stuff and importantly be able to tell others how they should do things. It’s pretty common for developers to setup a development environment for a pure frontend developer or a designer so they can make changes and commit CSS or new templates. This is no different. Most designers don’t need to know about the software environment in detail, it’s easier for them to defer to a domain expert. If a developer just wants to write code then they too should defer to someone who does know about the lower level when it comes to their development environment too.

Something else to setup

This argument has some merit. We’re all busy and downing tools to setup something for you and your team is time consuming. And I think until pretty recently the time taken and the knowledge needed was a genuine barrier. Personally I’ve tended to have few problems, but then I’m familiar enough with Linux administration to avoid some common pitfalls. But problems with setting up port forwarding or shared folders can be pretty irritating when you want to work on a pressing bug or shiny new feature. But with tools like vagrant providing a simple interface to do this I think this is hopefully a thing of the past.

Developers workstations should be personal

I agree up to a point here. Discussions of standardising individual developer tools turn into holy flame wars over whether everyone should use some IDE, Vim or Emacs (answer: Vim). This is pointless. File managers, utilities, test editors, terminal styles, host operating system. All of these and more should be up to the individual developers. But in the same way you generally don’t allow individual developers to use a new language no one knows without at least some discussion, why would this be different for the web server or operating system on which you’ll be running that code in production. Most of the time it’s not that developers make a consious decision to use a different version either. It’s more likely that they will take the path of least resistance and follow a tutorial or just use a package manager. It’s more likely if you ask the question “what specific version of Apache are you using on your development machine” they won’t know the answer.

Conclusions

I’ve not even gone into some of the other advantages of virtualisation here. Being able to snapshot your environment at any point and roll back an entire virtual machine like you do your code is hugely handy. As is the ability to create virtual machines that you can share with other members of your team. No more do new employees have to spend the first week installing dependencies and just getting code running.

I’m certainly not the only person doing this and it’s not a new idea. But it’s never been easier or cheaper. And with an increasing move towards virtualisation or cloud computing production environments it’s even easier to share good practices with your friendly systems administrators.

I’ve renabled comments on this blog after something of a break and I’d love to hear what people think, positive and negative.

Chef Hello World

I’ve been playing with Chef recently, in particular the solo variant. The new job at FreeAgent meant setting up new development virtual machines and rather than just jot down instructions I decided to script everything. I’d been wanting an excuse to take a look at Chef for a while and it’s certainly suited to this sort of job.

Unfortunately the getting started documentation isn’t yet great. I’m pretty sure this will improve over time, I had exactly the same problem with the Puppet docs a year ago. The main problem I had was I wanted to know how to use it, not just how to download someone elses cookbook. What I wanted was the following, the absolute simplest thing that will work. A Hello World example for Chef if you will. I’ll say now that I’m not an expert, their may be ways of doing this that are even simpler, but this works for me. And before someone mentions knife or rake tasks a generator isn’t simpler. It might be better when you know what’s going on but until you do it’s a big ass abstraction that will just get in the way.

All my sample cookbook is going to do is install a single package, curl. I’m going to assume you have chef installed for this already. The documentation did an OK job of that, although I’m relativly familiar with installing gems. I did find that the default system packages on Ubuntu at least were way out of date. Either get the packages direct from opscode or use the gem.

First create a directory and file structure that looks like this:

-- config
   -- node.json
   -- solo.rb
-- cookbooks
   -- example
      -- recipes
         -- default.rb

When you run the chef-solo command you need to tell it a few bits of information. The minimum appears to be just telling it where to find the cookbook we’re going to create. I think you can call this file anything you like but in the tree above it’s called solo.rb.

cookbook_path File.expand_path(File.join(File.dirname(__FILE__), '..', "    cookbooks"))

Next up is the details of the given node. This in our very simple case is just a list of the recipes we want to run when we execure chef-solo. Put the following content in the node.json file in the config directory as indicated above:

{
  "run_list": [ "recipe[example]" ]
}

Last up we want to create a cookbook. Now you can go and download example cookbooks from all over the place. This is great for learning new tricks and commands but for me at least to begin with most of them were more complicated than I needed for my simple usecase. Lots of options. Lots of knowing the package names on different distros. I’m just calling this cookbook example. That means the folder in the cookbooks folder is called example and the run list above references example. Feel free to change this to whatever you like, or create new cookbooks with different names. Inside that folder we create a recipes folder and inside that we create a default.rb file with the following content.

package "curl"

And that’s it. A bigish directory structure, three files, each with about one line of content. Simple.

Now to run all that just issue the following command:

sudo chef-solo -c config/solo.rb -j config/node.json

This should output various messages to the console about what chef is doing and, when it’s finished, you should find curl has been installed. Try and add another line to the recipe for another package (or even a gem) and rerun the chef-solo command. Now go read the docs for all the other cool things you can do.

Working For Freeagent

So, I’ve just accepted a new job at FreeAgent, the rather snappy online small business accounting startup. I’ll be starting in about a month, when I get back from devopsdays in Hamburg.

I’m joining a pretty well knit group of designers and developers working on a pretty well loved piece of software. I’ve known Roan (one of the directors) for years, ever since the first Highland Fling conference I think. And the rest of the team seem a good mix of technical smarts and professionalism. Hopefully I can live up to their already high standards.

I’m looking forward to jumping into a decent sized Ruby project and getting properly reacquainted with Rails as well. I’ve spent most of the previous few years in the Python world so that should be a nice challenge. I’m constantly impressed with the number of high quality tools written in Ruby andRails 3 looks a genuinely nice piece of software. Hopefully with a bit more idiomatic ruby under my belt, more time on my hands and some new real world problems to solve I’ll increase my open source contributions and get back to writing again as well. I’ll likely be spending most of my time on those good developer staples new feature development, supporting existing code and scaling systems. I’ll also be involved in some operations stuff if I get chance, introducing software I’m a huge fan of and automating everything I can get my hands on.

The rest of the team are based up in Edinburgh but I’ll be working from home most of the time. I’m quite looking forward to working from Cambridge after having lived here for a few years. I’ll also get to see various friends in Edinburgh that I don’t get chance to meet up with very often and still have time to make it to London for the odd event or get together. I’ll likely be looking into coworking spaces or desk shares in and around Cambridge so if anyone has any recommendations.

Given this weeks pubstandards is apparently all about weddings, birthdays and new jobs (and I have all 3 within a month) it would be rude not to go along for at least one drink. If you’re around London I might see you there.

Script Running Web Interface With Websockets

After something of a break I found a bit of time for some hacking at the weekend and decided to scratch a personal itch. I like writing shell scripts for everything from checking on things to deploying code. For anything that is more than just executing one command, or anything with detailed output or that takes a while I like to be able to see what’s going on. I also like to (OK, only sometimes) let other people run those commands and not all those people want to check something out and run a console. The web maks a pretty good interface for this sort of situation.

Enter Bolt. It’s my first proper go at using websockets for the communication between client and server, and my first stab at EventMachine. The code at the moment needs some improvement (tests, configurability, deployment options) but it works as a proof of concept.

It’s pretty simple, designed for running a single command at the push of a button and showing the results scroll past as they happen.

Bolt script running interface

Hopefully it’s useful to a few more people, or at least it will be when I clean it up a bit. For more information head over to the GitHub page.

Git Pre Recieve Hook For Integrity

I’m getting married rather soon so time has been somewhat short (in a good way) for just hacking on stuff, but I’ve finally found a little bit of time to play with something I’ve been mulling over for a while. Namely a continuous deployment workflow using the integrity continous integration server.

I’m hoping to have an incredibly simple but fully operation example available at some point - mainly to act as a good discussion point. For now here’s my current pre-receive hook.

Python: What To Use?

My friend Jamie Rumbelow has started a new project and decided to use Python. He asked a great question over on Stack Overflow which basically came down to what should I use for my first proper Python web application project. After a quick prompting on twitter I decided to have a go. I’ve cross posted my anwser below more because it took as long as a typical blog post to write.

Frameworks

OK, so I’m a little biased here as I currently make extensive use of Django and organise the Django User Group in London so bear that in mind when reading the following.

Start with Django because it’s a great gateway drug. Lots of documentation and literature, a very active community of people to talk to and lots of example code around the web.

That’s a completely non-technical reason. Pylons is probably purer in terms of Python philosophy (being much more a collection of discrete bits and pieces) but lots of the technical stuff is personal preference, at least until you get into Python more. Compare the very active Django tag on Stack Overflow with that of pylons or turbogears though and I’d argue getting started is simply easier with Django irrespective of anything to do with code.

Personally I default to Django, but find that an increasing amount of time I actually opt for writing using simpler micro frameworks (think Sinatra rather than Rails). Lots of things to choose from (good list here). I tend to use MNML (because I wrote parts of it and it’s tiny) but others are actively developed. I tend to do this for small, stupid web services which are then strung together with a Django project in the middle serving people.

Worth noting here is appengine. You have to work within it’s limitations and it’s not designed for everything but it’s a great way to just play with Python and get something up and working quickly. It makes a great testbed for learning and experimentation.

Mongo/ORM

On the MongoDB front you’ll likely want to look at the basic python mongo library first to see if it has everything you need. If you really do want something a little more ORM like then mongoengine might be what you’re looking for. A bunch of folks are also working on making Django specifically integrate more seamlessly with nosql backends. Some of that is for future Django releases, but django-norel has code now.

For relational data SQLAlchemy is good if you want something standalone. Django’s ORM is also excellent if you’re using Django.

API

The most official Oauth library is python-oauth2, which handily has a Django example as part of it’s docs.

Piston is a Django app which provides lots of tools for building APIs. It has the advantage of being pretty active and well maintained and in production all over the place. Other projects exist too, including Dagny which is an early attempt to create something akin to RESTful resources in Rails.

In reality any Python framework (or even just raw WSGI code) should be reasonably good for this sort of task.

Testing

Python has unittest as part of it’s standard library, and unittest2 is in python 2.7 (but backported to previous versions too). Some people also like Nose, which is an alternative test runner with some additional features. Twill is also nice, it’s a “a simple scripting language for Web browsing”, so handy for some functional testing. Freshen is a port of cucumber to Python. I haven’t yet gotten round to using this in anger, but a quick look now suggests it’s much better than when I last looked.

I actually also use Ruby for high level testing of Python apps and apis because I love the combination of celerity and cucumber. But I’m weird and get funny looks from other Python people for this.

Message Queues

For a message queue, whatever language I’m using, I now always use RabbitMQ. I’ve had some success with stompserver in the past but Rabbit is awesome. Don’t worry that it’s not itself written in Python, neither is PostgresSQL, Nginx or MongoDB - all for good reason. What you care about are the libraries available. What you’re looking for here is py-amqplib which is a low level library for talking amqp (the protocol for talking to rabbit as well as other message queues). I’ve also used Carrot, which is easier to get started with and provides a nicer API. Think bunny in Ruby if you’re familiar with that.

Environment

Whatever bits and pieces you decide to use from the Python ecosystem I’d recommend getting to who pip and virtualenv - note that fabric is also cool, but not essential and these docs are out of date on that tool). Think about using Ruby without gem, bundler or rvm and you’ll be in the right direction.