A Discussion of The Operational Challenges With Unikernels

What are Unikernels

Most of this post assumes a basic understanding of what unikernels are so I’d recommend reading Unikernels – the rise of the virtual library operating system before moving on.

Why are Unikernels interesting

As a starting point: complexity. Managing infrastructure, and the software that runs on it, is too complicated. You can impose organisational rules to control this complexity (we only deploy on Debian, we only run JVM applications, the only allowed database is MySQL) but that limits you in other ways too, and in reality is nearly always broken somewhere in any non-trivial environment (this appliance uses Ubuntu, this software is only certified on Windows, PostgreSQL doesn’t run on the JVM). So you turn to software to manage that complexity; Puppet or Chef do a great job of allowing configuration complexity to be managed in code (where you can test it) and Docker allows for bundles of complexity to be isolated from other bundles of complexity. But there are still an awful lot of moving parts.

Another reason is the growing realisation that security is important. Securing systems on the internet is hard. Even though the basics are broadly understood they are often not implemented, and the people attempting to compromise systems are smart, well paid and highly incentivised (basically like you). It’s generally easier to break something than to build it. Part of this is a numbers game – to run a reasonable sixed system you might need to run 50 different services, and install 200 packages on every host. An attacker has to compromise just one of those to win.

A further reason, if one were needed, is the proliferation of many small internet connected devices, aka. The Internet of Things. Part of this relates to the above points about security concerns, but some of it is simply a matter of managing that many single purpose, low power, devices. The overhead of a typical general purpose operating system and application runtime just don’t fit this model.

Enter unikernels. Unikernels actually remove unneeded complexity. You’re running a hypervisor and the unikernel and that’s it. The unikernel contains only those libraries that you have specifically required. That drastically reduces the surface area for attack as well as meaning you’re running less software, hopefully enough less that your power needs are reduced too. By specifically requiring individual libraries you’re also making complexity visible. Rather than using a general purpose operating system with it’s 100s of packages and millions of lines of code you are at least choosing what to include.

Operational challenges

While I think some part of the future looks like unikernels their are some large operational challenges to overcome before they break out of very specific niches or research projects. Note that

there are architectural and software development challenges as well, I just happen to think they’re easier to deal with.

Development environment

There are a few properties of a development environment that I think are essential to modern development; development/production parity being one of the most important. Tools like Vagrant, and a move towards infrastructure as code, and more recently Docker have made great strides here in the past several years. The different unikernel implementations are generally based on lesser known software stacks (Haskell, OCaml, Erlang, etc) so some of this is familiarity. But what does development/production partity mean for a unikernel based system? We’re not just talking about the individual unikernel here either – how do I deploy unikernels? How do I compose several unikernels together to build an application? What does a Continuous integration or deployment pipeline look like? In my view the unikernel movement should focus some efforts here. Not only will this make it easier for people to get started, but having strong opinions early will allow the nascent community to solve the problem together, rather than everyone solving it just-in-time for themselves.

Managing the hypervisor

I’d argue today most developers don’t spent much time directly working with hypervisors. Either you’re running on an in-house VMware, KVM or Xen install with some (hopefully self-service, automated) provisioning mechanism in place or you’re using a public cloud like AWS, Azure, etc. The current generation of unikernel systems mainly target Xen. I think in the short term at least this means getting to know the hypervisor. Xen is solid software, but I don’t see a great deal of automation around it – say well maintained Puppet modules, API clients or a Terraform provider. In the long term we’ll hopefully have higher level interfaces, but in the short term efforts here would lower the barrier to entry considerably.

Double down on AWS

Given the above, and given the ubiquity of EC2 (which is based on Xen) it might be wise to build up first-class tools around using EC2 as a target environment for unikernel deployments. EC2 supports custom kernels, but these require a number of convoluted steps that could be automated away (note that I’m talking about more than just a shell script here). Also what are the best practices around autoscaling groups andunikernels? Or VPC networks and unikernels?

The network

With the explosion in containers and microservices it’s becoming clearer (if it wasn’t already) how important the network is. By removing the operating system we remove things like host firewalls and the new breed of overlay networks. At the same time if we are to tap the dynamic potential of unikernels we’ll need a similarly dynamic and automatable network. Maybe this becomes more of an application concern, with services communicating via other services which act as firewalls and intelligent proxies, but that still leaves the underlying network to be managed.

Debugging

However much testing you do beforehand you’ll still likely end up with problems in production, and as you scale up you’ll hit issues that you simply can’t recreate outside the live environment. This is were good debugging capabilities come in. While general purpose operating systems might be complex they are well know, and tools like ps, top, free, ping, telnet, netcat, dtrace, etc. are commonly used by anyone debugging systems. Note that in many cases you’re debugging a combination of systems; is the performance issue an application problem, a network problem, a storage problem or some interesting combination of several facters?

By removing the general purpose operating system, unikernel based environments remove most of the current debugging tools at the same time. Part of this Is good application development hygiene (logs, metrics and status endpoints for instance), but what about the more interactive debugging practices? What does debugging a system based on unikernels look like?

Orchestration

The word may be overloaded but the need to arrange and manage a number of components that make up a larger system is a real need. This might be something like Docker’s Compose file or Brooklyn’s Blueprints, or it could be something more akin to the APIs from Cloud Foundry, Kubernetes or Mesos. Testing some of these models with unikernel based systems will be an interesting test of how coupled to containers the existing models are. The lack of legacy again opens up the potential to come up with a truly modern alternative here too.

Conclusion

Unless you’re in an environment where security is your number 1 concern then the current state of Unikernels probably means choosing to adopt them now is a little bleeding edge. But I think that will change over time as the various projects mature and address some of the issues described above. In the meantime I’d love to see more discussion of some of the operational challenges. I think talking about the needs of operators at this early stage should make the resulting ecosystems more robust whsen it comes to future production deployments.

Update to Puppet Module Skeleton

Being on holiday last week meant I had a little time for some gardening of open source projects and I decided to update puppet-module-skeleton with some new opinions.

The skeleton is a replacement for the default module skeleton that ships with Puppet and is used by puppet module generate. Unlike the default skeleton this one is super-opinionated. It comes bundled with lots of testing tools, suggestions for documentation, integration with Travis CI, module coverage reports and more.

Updates in the latest version include:

• Support for Puppet 4 paths
• The addition of Rubocop, which enforces parts of the Ruby style guide
• Adding a number of Puppet Lint plugins
• Allow installing various Puppet versions during integration tests

I also fixed a few reported bugs and extended the test matrix to test across a range of Puppet and Ruby combinations.

The skeleton is intended to help people with a basic understanding of Puppet write better modules, without having to setup everything themselves. You don’t have to agree with all the options to make use of the skeleton as it’s simple enough to delete a few files once you generate your new module. But a working out-of-the-box beaker install, and the ability to automatically run unit tests when files change are patterns worth adopting for most module developers I think.

If anyone has any suggestions for extra tools, or changes to the skeleton itself, let me know.

Information Security Reading List

I read quite a bit (probably a book a week or so) and one of the topics I’ve been reading on for a while is information security. In a recent conversation someone asked for some book suggestions, so I thought I’d write that up in a blog post rather than an email.

Most of this list isn’t particularly technical. It’s not a developers list of software engineering tomes. If you’re a developer or operator then I’d recommend reading some of the more policy or journalistic pieces as well for context. And if you’re just interested in the topic but nor particularly technical I’d skip the security engineering suggestions.

Note that I make no claims about this being a particularly balanced list, it’s biased towards what I find interesting to read. Hopefully you’ll find it interesting too.

Journalism

Understanding why Information Security is important tends to require some context. The following books provide that, with detailed real-world stories of criminal and government activities.

• The Dark Net - Jamie Bartlett - an excellent personal tale of investigating the hidden side of the internet.
• Spam Nation - Brian Krebs - everything you wanted to know about how and why Spam works.
• Countdown to Zero Day - Kim Zetter - a detailed and fast paced description of the Stuxnet attack, and it’s implications.
• Future Crimes - Marc Goodman - a focus on the criminal possibilities of the modern internet and the internet of things.
• Worm - Mark Bowden - similar to the excellent tale of Stuxnet above, this is the story of Conficker and how it was discovered

Policy and context

These books are focused more on government policy and nation state threats, and the debate about the rules of war and the internet.

• Cyber War - Richard Clarke - probably the best description of what cyber war is and isn’t, and some of the geopolitical problems emerging.
• Cyber War Will Not Take Place - Thomas Rid - a good counter to the above book, with lots more detailed discussion of policy and definition.
• Inside Cyber Warfare - Jeffrey Carr - really just a run through of current threats, especially organised crime.

Security engineering

• Security Engineering - Ross Anderson - highly technical and quite epic, but definitely the best security engineering book around.
• Threat Modelling - Adam Shostack - details descriptions of how and why to conduct threat moddelling, with lots of examples.
• Data Driven Security - Jay Jacobs and Bob Rudis - nice examples, including code samples, of applying data and statistics tools and practices to security problems.
• Cloud Security and Privacy - Tim Mather, Subra Kumaraswamy, Shahed Latif - a good book to read for anyone working in AWS, Azure or similar. Good discussion of concerns and compliance approaches in third party environments.
• The Tangled Web - Michal Zalewski - everything you ever wanted to know about the browser security model
• Silence on the Wire - Michal Zalewski - described as a field guide to passive reconnaissance and indirect attacks. Good for starting to think about non-obvious security threats

On my reading list

I’ve not read these books yet so can’t recommend them as such, but they both look good additions to the list above.

• Data and Goliath - Bruce Schneier - a look at the large scale data collection programmes of governments and their implications for everyone.
• Black Code - Ronald J. Deibert - the story of the Citizen Lab and it’s front line cyber researchers

Acceptance testing MirageOS installs

I’m pretty interested in MirageOS at the moment. Partly because I find the idea behind unikernels interesting and partly because I keep bumping into the nice folks OCaml Labs in Cambridge.

In order to write and build your MirageOS unikernel application you need an OCaml development environment. Although this is documented I wanted something a little more repeatable. I also found and reported a few bugs in the documentation which got me thinking about acceptance testing. I’m not (yet) an OCaml programmer, but infrastructure automation and testing I can do.

Into Puppet

I started out writing a Puppet module to install and manage everything, which is now available on GitHub and on the Forge.

This lets you do something like the following, and have a fully working MirageOS setup on Ubuntu 12.04 or 14.04.

class { 'mirageos':
  user      => 'vagrant',
  opam_root => '/home/vagrant/.opam',
}

Given time, inclination or pull requests I’ll add support for other operating systems in the future.

But how do you know it works?

The module has a small unit test suite, but it’s nice to know test the actual running of Puppet and installation of the software. For this I’ve used Test Kitchen and ServerSpec. This allows for spinning up 2 virtual machines (one for each supported operating system), applying the Puppet manifest and then making some assertions:

The above is simply checking whether certain packages are installed, the PPA is setup correctly and whether mirage and opam can be executed cleanly.

Can it produce a working unikernel?

The above tells us whether the installation worked, but not whether the resulting software allows us to build MirageOS unikernels. For this I used Bats running in the same Test Kitchen setup.

The above configures and builds a simple HTTP server unikernel, and then checks that when run it returns the expected response on the correct port.

Conclusion

I like the separation of concerns above. I can use the Puppet code without the test code, or even swap the Puppet code out for a shell script if I wanted. I could also run the serverspec tests anywhere I want to check state, which is the reason for separating those tests from the one’s building and running a unikernel. Overall the tool chain for ad-hoc infrastructure testing (quick mention of Infrataster too) is really quite powerful and approachable. I’d love to see more software ship with a user-facing test suite for people to verify their installation works.

Automating windows development environments

My job at Puppet Labs has given me an excuse to take a closer look at the advancements in Windows automation, in particular Chocolatey and BoxStarter. The following is very much a work in progress but it’s hopefully useful for a few things:

• If like me you’ve mainly been doing non-Windows development for a while it’s interesting to see what is possible
• If you’re starting out with infrastructure development on Windows the following could be a good starting place
• if you’re an experienced Windows pro then you can let me know of any improvements

All that’s needed is to run the following from a CMD or Powershell prompt on a new Windows machine (you can also visit the URL in Internet Explorer if you prefer).

START http://boxstarter.org/package/nr/url?https://gist.githubusercontent.com/garethr/a1838aa68355a0766de4/raw/d92b41ee9dcad68c079d24c64bac7d1d27cf37c7/garethr.ps1

This launches BoxStarter, which executes the following code:

This takes a while as it runs Windows update and repeatedly reboots the machine. But once completed you’ll have the listed software installed and configured on a newly up-to-date Windows machine.

Docker, Puppet and shared volumes

During one of the openspace sessions at Devopsdays we talked about docker and configuration management, and one of the things we touched on was using dockers shared volumes support. This is easier to explain with an example.

First, lets create a docker image to run puppet. I’m also installing r10k for managing third party modules.

Docker

FROM ubuntu:trusty

RUN apt-get update -q
RUN apt-get install -qy wget
RUN wget http://apt.puppetlabs.com/puppetlabs-release-trusty.deb
RUN dpkg -i puppetlabs-release-trusty.deb
RUN apt-get update

RUN apt-get install -y puppet ruby1.9.3 build-essential git-core
RUN echo "gem: --no-ri --no-rdoc" > ~/.gemrc
RUN gem install r10k

Lets build that and tag it locally. Feel free to use whatever name you like here.

docker build -t garethr/puppet .

Lets now use that image as a base for another image.

FROM garethr/puppet

RUN mkdir /etc/shared
ADD Puppetfile /
RUN r10k puppetfile check
RUN r10k puppetfile install
ADD init.pp /
CMD ["puppet", "apply", "--modulepath=/modules", "/init.pp","--verbose", "--show_diff"]

This image will be used to create containers that we intend to run. Here we’re including a Puppetfile (a list of module dependencies) and then running r10k to download those dependencies. Finally we add a simple puppetfile (this would likely be an entire manifests directory in most cases). The final line means that when we run a container based on this image it will run puppet and then exit.

Again lets build the image and tag it.

docker build -t garethr/puppetshared .

Just as a demo, here’s a sample Puppetfile which includes the puppetlabs stdlib module.

Puppet

mod 'puppetlabs/stdlib'

And again as an example here’s a simple puppet init.pp file. All we’re doing is creating a file at a specific location.

file { '/etc/shared/client':
  ensure => directory,
}

file { '/etc/shared/client/apache.conf':
  ensure  => present,
  content => "not a real config file",
}

Fig

Fig is a tool to declare container types in a text file, and then run and manage them from a simple CLI. We could do all this with straigh docker calls too.

master:
  image: garethr/puppetshared
  volumes:
    - /etc/shared:/etc/shared:rw

client:
  image: ubuntu:trusty
  volumes:
    - /etc/shared/client:/etc/:ro
  command: '/bin/sh -c "while true; do echo hello world; sleep 1; done"'

The important part of the above is the volumes lines. What we’re doing here is:

• Sharing the /etc/shared directory on the host with the container called master. The container will be able to write to the host filesystem.
• Sharing a subdirectory of of /etc/shared with the client container. The client can only read this information.

Note the client container here isn’t running Puppet. Here it’s just running sleep in a loop to simulate a long running process like your custom application.

Let’s run the master. Note that this will run puppet and then exit. But with the above manifest it will create a config file on the host.

fig run master

Then run the client. This won’t exit and should just print hello world to stdout.

fig run client

Docker 1.3 adds the handy exec command, which allows for one-off commands to be executed within a running container. Lets use that to see our new config file.

docker exec puppetshared_client_run_1 cat /etc/apache.conf

This should output the contents of the file we created by running the master container.

Why?

This is obviously a very simple example but I think it’s interesting for a few reasons.

• We have completely separated our code (in the container) from the configuration
• We get to use familiar tools for managing the configuration in a familiar way

It also raises a few problems:

• The host needs to know what types of container are going to run on it, in order to have the correct configuration. If you’re using Puppet module then this is simple enough to solve.
• The host ends up with all of the configuration for all the containers in one place, you could also do things with encrypting the data and having the relevant keys in one image and not others. Given how if you’re on the host you own the container anyway this isn’t as odd as it sounds.
• We’re just demonstrating files here, but if we change our manifest and rerun the puppet container then we change the config files. But depending on the application it won’t pick that up unless we restart it or create a new container.

Given enough time I may try build a reference implementation using this approach, anyone with ideas about that let me know.

This post was inspired by a conversation with Kelsey and John, thanks guys.

Using Puppet with key/value config stores

I like the central idea behind storing configuration in something like Etcd rather than lots of files on lots of disks, but a few challenges still remain. Things that spring to mind are:

• Are all your passwords now available to all of your nodes?
• How do I know when configuration changed and who changed it?

I’ll leave the first of those for today (although have a look at Conjur as one approach to this). For the second, I’m quite fond of plain text, pull requests and a well tested deployment pipeline. Before Etcd (or Consul or similar) you would probably have values in Hiera or Data Bags or similar and inject them into files on hosts using your configuration management tool of choice. So lets just do the same with our new-fangled distributed configuration store.

key_value_config { '/foo':
  ensure   => present,
  provider => etcd,
  value    => 'bar',
}

Say you wanted to switch over to using Consul instead? Just switch the provider.

key_value_config { '/foo':
  ensure   => present,
  provider => consul,
  value    => 'bar',
}

You’d probably move all of that out into something like hiera, and then generate the above resources, but you get the idea.

etcd_values:
  foo: bar

The above is implemented in a very simple proof of concept Puppet module. Anyone with any feedback please do let me know.

Leaving GDS never easy

The following is the email I sent to lots of my colleagues at the Government Digital Service last week.

So, after 3 rather exciting years I’ve decided to leave GDS.

That’s surprisingly difficult to write if I’m honest.

I was part of the team that built and shipped the beta of GOV.UK. Since then I’ve worked across half of what has become GDS, equally helping and frustrating (then hopefully helping) lots of you. I’ve done a great deal and learnt even more, as well as collected arcane knowledge about government infosec and the dark arts of procurement along the way. I’ve done, and helped others do, work I wouldn’t even have thought possible (or maybe likely is the right word?) when I started.

So why leave? For all the other things I do I’m basically a tool builder. So I’m going off to work for Puppet Labs to build infrastructure tools that don’t suck. That sort of pretty specific work is something that should be done outside government in my opinion. I’m a pretty firm believer in “government should only do what only government can do” (design principle number 2 for those that haven’t memorised them yet). And if I’m honest, focusing on something smaller than fixing a country’s civic infrastructure is appealing for now.

I’ll let you in on a secret; I didn’t join what became GDS because of the GOV.UK project. I joined to work with friends I’d not yet had the chance to work with and to see the inside of a growing organisation from the start. I remember Tom Loosemore promising me we’d be 200 people in 3 years! As far as anyone knows we’re 650+ people. That’s about a person a day for 2 years. I’m absolutely not saying that came without a cost, but for me being part of that that was part of the point - so I can be a little accepting with hindsight.

For me, apart from all the short term things (side-note: this job now has me thinking £10million is a small amount of money and 2 years is a small amount of time) there is one big mission:

Make government a sustainable part of the UK tech community

That means in 10 years time experienced tech people from across the country, as well as people straight from university, choosing to work for government. Not just for some abstract and personal reason (though that’s fine too), but because it’s a a genuinely interesting place to work. That one’s on us.

Using OWASP ZAP from the command line

I’m a big fan of OWASP ZAP or the Zed Attack Proxy. It’s suprisingly user friendly and nicely pulls of it’s aim of being useful to developers as well as more hardcore penetration testers.

One of the features I’m particularly fond of is the aforementioned proxy. Basically it can act as a transparent HTTP proxy, recording the traffic, and then analyse that to conduct various active security tests; looking for XSS issues or directory traversal vulnerabilities for instance. The simplest way of seeding the ZAP with something to analyse is using the simple inbuilt spider.

So far, so good. Unfortunately ZAP isn’t designed to be used from the command line. It’s either a thick client, or it’s a proxy with a simple API. Enter Zapr.

Zapr is a pretty simple wrapper around the ZAP API (using the owasp_zap library under the hood). All it does is:

• Launch the proxy in headless mode
• Trigger the spider
• Launch various attacks against the collected URLs
• Print out the results

This is fairly limited, in that a spider isn’t going to work particularly well for a mor interactive application, but it’s a farily good starting point. I may add different seed methods in the future (or would happily accept pull requests). Usage wise it’s as simple as:

zapr --summary http://localhost:3000/

That will print you out something like the following, assuming it finds an issue.

+-----------------------------------+----------+----------------------------------------+
| Alert                             | Risk     | URL                                    |
+-----------------------------------+----------+----------------------------------------+
| Cross Site Scripting (Reflected)  | High     |http://localhost:3000/forgot_password   |
+-----------------------------------+----------+----------------------------------------+

The above alert is taken from a simple example, using the RailsGoat vulnerable web application as a scape goat. You can see the resulting output from Travis running the tests.

Zapr is a bit of a proof of concept so it’s not particularly robust or well tested. Depending on usage and interest I may tidy it up and extend it, or I may leave it as a useful experiment and try and finally get ZAP support into Gauntlt, only time will tell.

Consul, DNS and Dnsmasq

While at Craft I decided to have a quick look at Consul, a new service discovery framework with a few intersting features. One of the main selling points is a DNS interface with a nice API. The Introduction shows how to use this via the dig command line tool, but how do you use a custom internal DNS server without modifying all your applications? One answer to this question is Dnsmasq.

I’m not explaining Consul here, the above mentioned introduction does a good job of stepping through the setup. The following assumes you have installed and started consul.

Installation and configuration

I’m running these examples on an Ubuntu 14.04 machine, but dnsmasq should be available and packaged for lots of different operating systems.

apt-get install dnsmasq

Once installed we can create a very simple configuration.

echo "server=/consul/127.0.0.1#8600" > /etc/dnsmasq.d/10-consul

All we’re doing here is specifying that DNS requests for consul services are to be dealt with by the DNS server at 127.0.0.1 on port 8600. Unless you’ve changed the consul defaults this should work.

Just in case you prefer Puppet their is already a handy dnsmasq module. The resulting puppet code then looks like this.

include dnsmasq
dnsmasq::conf { 'consul':
  ensure  => present,
  content => 'server=/consul/127.0.0.1#8600',
}

Usage

The examples from the main documentation specify a custom DNS server for dig like so:

dig @127.0.0.1 -p 8600 web.service.consul

With Dnsmasq installed and configured as above you should just be able to do the following:

dig web.service.consul

And now any of your existing applications will be able to use your consul instance for service discovery via DNS.