Infrastructure as code

How many days have you wasted setting up development machines or VMs?  How many times have you faithfully followed a multi-page setup guide only to find you missed a step and have to start again?

Wouldn’t it be great if you could simply run a script and let your development machine build itself while you do something more interesting instead?

“Infrastructure as Code” may be the solution to this problem, and also to the much bigger and more expensive problem of how to build complete dev, test and production environments quickly and consistently (think about the hassle involved in building a test environment with four web servers, four app servers and a database cluster along with attendant network configuration).

So what exactly is “Infrastructure as Code”, you might ask?  Well, it’s essentially a philosophy which seeks to automate the creation and configuration of virtual infrastructure in the way that we already automate the build, testing and deployment of software.

We learned a long time ago that manual build, testing and deployment were time consuming, error prone, difficult to repeat and needed extensive documentation.  But we never followed that through to management of the environments we develop and run our software in, largely because we were used to working with bare metal hardware instead of virtual machines.  As a result we still tend to build these environments by hand, which takes ages and often results in rework and inconsistencies. 

But imagine using the same 100% consistent automated process to build your development, test and production environments!  Imagine joining a new project and simply getting a script from Git or SVN and using it to automatically build your dev VM.  You could even write tests and use continuous integration to verify the scripts work, and that your applications still work with the new configuration!  If that’s not enough to whet your appetite, imagine that those scripts were platform agnostic, allowing you to deploy your environments and applications to any standard virtualisation platform – Amazon, Azure, VMWare and so on.

So how do we do this?

By using loads of new and exciting tools, of course!  Actually, the problem is there really are loads of tools to choose between, some of which complement each other and some of which are direct competitors.  All are made possible by the maturing and consumerisation of virtualisation technology.  The difficulty is identifying which provide the best fit with our goals and the technologies we like to use. 

The Infrastructure as Code movement grew out of the DevOps and Ruby communities and as such tends to leverage open source tools, languages and platforms, such as Ruby itself and Linux.  However, Windows environments and applications have their own set of challenges, which are now being supported by Windows specific tools. 

The main players in this field are:

Vagrant – Manages the creation and basic configuration of virtual machines and launches Chef and Puppet scripts;

Chef – Models infrastructure and environments using a Ruby DSL, allowing build and provisioning of standard ‘Recipes’ to almost any virtualisation platform;

Puppet – Covers much the same ground as Chef, but with at least one important distinction: it uses declarative models, rather than procedure installation processes.  This allows verification of the state of existing infrastructure;

CFEngine – Similar to Chef and Puppet.  Like Puppet, it uses declarative rather than procedural models;

Chocolatey – A Windows equivalent of Linux’s apt-get, based on NuGet, which allows users to install applications from an online catalogue with a single command line;

Boxstarter – A Windows-specific tools which uses Chocolatey and PowerShell to build and configure complete bare metal machines and VMs.

Chef, Puppet and Vagrant are the most mature and fully featured of these tools.  However, they focus more on Linux and are probably most suitable for developers using Java, Scala, Ruby or Python.  Their support for a wider variety of operating systems, tools and virtualisation platforms is improving but Windows/.NET developers will find Boxstarter and Chocolatey much more accessible and relevant, as they are specifically designed to cope with the challenges of installing Windows applications such as Visual Studio and SQL Server.

There are free community editions of Chef and Puppet but they have certain limitations and the fully-featured enterprise versions are not particularly cheap.  Boxstarter and Chocolatey are, however, essentially free for .NET developers.

QCon London: Craig Larman keynote

QCon London 2011 began with a very entertaining keynote from Craig Larman, titled “Scaling Lean & Agile: Large, Multisite or Offshore Delivery”.

Larman’s most important piece of advice was perhaps a little surprising, given that the talk was about scaling up agile development processes to large distributed teams and that he has written books on the subject.  His advice was basically “Don’t do it!”  He recommended that teams should be kept small – ten people or less – if possible.  This certainly struck a chord with me, having learned long ago that small teams are far more productive and effective than large teams.

Leaving that aside though, Larman went on to make a number of points about scaling Scrum up to teams of hundreds located at multiple sites, most of which is detailed in his books on the subject.  He also gave general advice such as use configuration, not branching, to customise software and build cross-functional teams with no specialists.

Ever the showman, Larman dropped a number soundbites which were quickly seized upon by the tweeters in the audience:

  • “We should encourage a culture of master programmers not PowerPoint architects”
  • “Architecture is a bad metaphor. We don’t construct our software like a building, we grow it like a garden”

The talk ended with one final piece of advice which went down very well with the audience: avoid heavyweight process tools from IBM/Rational.  Another lesson I learned many years ago after spending man-weeks producing UML models which quickly became out of date, when I could have been creating useful software.

Larman has since elaborated on some of these points in an InfoQ article:

http://www.infoq.com/articles/large-scale-agile-design-and-architecture

QCon London 2011

I’ll get this blog started by writing a series of posts about my recent visit to QCon London (March 9th-11th).

View from the conference centreWhen I first heard about QCon, a few years ago, I thought it would be just another dull tech conference in an anonymous hotel or conference centre, but I’m very pleased to say I was completely wrong!  QCon was held in the Queen Elizabeth II conference centre right in the heart of London,  opposite Westminster Abbey, and provided a range of tracks on subjects as varied as large scale  architecture, iOS and Android development, lean and Kanban, HTML5, NoSQL and state of the art .NET.

In addition, there were daily keynotes on subjects such as entrepreneurialism, innovation and scaling agile processes.

Together, these talks provided an opportunity not just to freshen up my .NET development skills, but to find out about technologies and ideas I’d never considered before and, perhaps more importantly, to think about non-technology issues too – innovation, how to learn, continuous improvement.

I’ll write a short summary of each of the talks I attended over the next few weeks (time permitting!).