23 Aug 2014
At Bloomberg my team is tasked with supporting infrastructure
for all of our consumer and subscription web properties. We have been
using Chef to solve some of our problems around configuration
management. Most of our recipes were built using Chef Solo and
eventually deployed on our cloud infrastructure without a Chef Server.
As we began to plan to deploy software services to the teams that we
support we quickly began to realize that it would be nearly impossible
to manage multiple node deployments without Chef Server.
Our immediate need was to provision a Chef Server for our build
infrastructure that we would be distributing to about twenty
teams. But in addition to this we have brought in Chef to train
several classes of our developers. After the classes concluded there
were many requests for individual Chef Server tenancies for testing.
Using Chef Server within our infrastructure was an afterthought
because most of our virtual machines had a very short lifespan. Since
we could have several dozen development teams wanting their own Chef
Server deployments we needed a solution to not only easily deploy a
Chef Server, but to hydrate it with the data necessary for each
OpenStack tenancy. This included both user and client authentication
After a bit of back and forth I was able to isolate the requirements
for our Chef Server infrastructure:
It must use the Chef Server community cookbook and the
Omnibus packages provided by Chef;
It should be quick and easy to deploy a new instance because we
may be doing so for several dozen tenancies in OpenStack;
All user and client keys should be generated locally and the server
should be hydrated during the initial convergence run;
I was first introduced to Chef Metal at this past year’s
ChefConf where John Keiser gave an excellent talk. My team has a
lot of experience using Vagrant, but it made much more sense to me to
include a metal recipe with our Chef infrastructure cookbook. It could
serve as both a teaching aide, and would also make our internal
cookbooks much more versatile. Throughout the discovery process I
found the community Chef Server Populator cookbook, ended up
completely rewriting it to use Chef resources to modify the Chef
Server, and ultimately used Chef Zero to bootstrap everything.
Using Chef Metal I was able to easily distribute a recipe with our
Chef infrastructure cookbook that would provision a Chef Server very
easily using the machine resource. The recipe that I began with
looked very similar to what is below. It first laid down changes to
the base operating system using our base cookbook, and simply used the
community cookbook to install the Chef Server from the Omnibus
packages. It was very straight forward.
Unfortunately I still needed to hydrate the local data into the newly
minted Chef Server. Luckily for me the cats over at
Heavy Water Operations have solved this problem and provided the
Chef Server Populator cookbook. There were only a few minor
modifications that I needed to make to support client validation
keys. But other than that everything seemed to be smooth sailing.
At first, with only a few minor changes, the Chef Server Populator
cookbook seemed to be the answer to my problems. I had a few cases
that were not covered, but these were all pretty simple extensions.
It was natural and made sense to convert our Chef infrastructure
cookbook into a Chef repository and use Chef Zero as an in-memory
Chef Server during the initial provisioning.
That worked out quite well at first. I was able to cobble together a
simple Rake task to generate and write out the private and public keys
to JSON data bag items. My metal recipe started to look a lot more
complex than I had initially intended, and it didn’t seem as clean
as I thought it could be. As I was reading through the code for the
Chef Metal Fog driver a light bulb went off in my head: I could
use the Cheffish gem Chef resources and build the data on the
Chef within Chef
I ended up completely rewriting the Chef Server Populator cookbook.
My fork now used the Chef resources provided by the Cheffish gem
to read from data bag items and create the user and client records
against the Chef Server node. The beauty of this approach is that I no
longer had to execute clunky knife commands since in Chef 11 erchef
exposed an API to pass both user and client public keys. The cookbook
is now much more clean and simple to grok.
The code for creating the user and client keys was also able to use
the same Chef resources from Cheffish. Instead of operating against
our newly provisioned Chef Server it executed against the in-memory
server provided by Chef Zero.
Using keys which are generated inside of the metal recipe is not
idempotent. There were two ways that I went about solving this
problem. This first was merely reading in the private keys off disk
and delegating the generation to some external process. The second
would be to use the Chef resource to generate private keys and allow
it to manage if they needed to be updated or not. Ultimately it worked
out that if the keys were generated outside of the metal recipe it was
much more straightforward. Here is a full example where keys for
clients were generated outside, but user keys are generated on the
Tying the Bow
The final result was a complete cookbook refactor which uses Chef
resources to hydrate a newly provisioned Chef Server. The default recipe
uses node attributes and creates both client and user resources on the
new server. A data bag recipe uses a Chef data bag search to find items
and applies them to the node attributes prior to including the default
Since Chef Zero provides a fully functioning in-memory Chef Server we
are able to use Chef resources to add data bag items with locally
generated public keys. When using the Chef Metal recipe this data is
now made accessible to the data bag search, and thus it gives us an
elegant way to hydrate a Chef Server. I have provided an example of
this inside of my fork of the cookbook which illustrates all of
the steps above.
As I mentioned at the top of this post I originally started work on
this solution to allow us to quickly deploy our build infrastructure
to multiple tenancies within our private cloud. In a future post I
will explain how my team was able to create an easily deployable
continuous integration framework using the community Jenkins recipe.
If you are reading this post and are interested in joining our growing
group of Chef technologists at Bloomberg feel free to reach out to me.
We are hiring!
19 Mar 2014
This weekend one of my colleagues at Bloomberg and I spent a few
hours at my apartment this weekend to deploy OpenStack. I made
the same attempt last year by myself and was barely able to get a
OpenStack service running. It was that hard. But this past year,
with the help of a vibrant community, have made the process for
deploying a cluster a hell of a lot easier.
Using Fuel from Mirantis, and with a little configuration of my
managed switch, I had a two node OpenStack cluster built from eBay
servers that I had lying around. One controller node and one compute
node we were ready to rock and roll. Even the managed switch
configuration for VLAN support went incredibly smooth.
The ultimate goal is to have a relatively portable, simple datacenter
so that distributed systems can be easily demonstrated at meetups. Chaz
and I will be working on making this process as transparent and automatic
as we can.
Even though my cluster is still in pieces on the floor I am hoping to
use it to demonstrate some Chef development at future
Scale DC meetups. Chaz will be presenting the work that he has
done setting up Fuel and his home cloud which puts my completely
to shame. After ChefConf 2014 I am hoping to introduce some of
the work that the new Web Operations team at Bloomberg have been
hacking away at.
12 Mar 2014
For the past year I’ve been working on a grassroots effort building reusable
patterns for infrastructure at Bloomberg Government. Over the course of
a year my team was able to stabalize and automate datacenter operations.
Because of legacy infrastructure our focus was primarily on the applications
rather than the machines themselves.
Of course making application deployments deterministic and improving the
continuous delivery pipeline were likely the low hanging fruit at BGOV.
But the larger issue needed to be managed at orchestration and provisioning
of new virtual machines. The more that we dug into the problem we realized
that BGOV was not the only organization at Bloomberg that was suffering
As of two weeks ago I’ve been given a tiger team of engineers throughout
research and development to work with our cloud infrastructure and solve
these problems for all of our web verticals. We were given the charge to
automate the datacenter operations - including orchestration, provisioning and
continuous delivery - and integrate with the enterprise systems that we use
throughout our firm.
This goes hand-in-hand with implementing a culture of DevOps where the
firm promotes communication, collaboration and integration amongst all of the
stakeholders in a business unit. This includes the research and development
component of a product, but also the business and technology support teams.
The ultimate goal is to deliver a stable, higher quality product to our
27 Apr 2013
My good British friend Phil Sturgeon posted an awesome writeup
regarding Vagrant and Chef. I’ve been meaning to do a post
of my own. Over the last few months a side project of mine has got me
involved a lot with Vagrant, Chef and Veewee. But to
find resources on the Internet, especially tutorials, which combine
all that knowledge is definitely a task I would not wish on anyone
So I figured I’d put together a simple little blog post that gets you
started building out your own testing environment. Just a heads up: I
am going to assume you have the Vagrant dependencies installed. I
prefer installing these from the website and not through RubyGems, but
to each his own. The version I am using is 1.1.3. Everything else
should be picked up automatically from the appropriate files.
I am going to walk you through the steps that I used to build the
example application which is available as a Gist. Remember that
you can just run this using the
vagrant up command and it will be
available (in a few minutes) by accessing the 8080 port on your host
machine. You’re still going to have to go through the whole
Veewee build of the base boxes. But after you add the base box
that you’ve built you can just start the application with
The first order of business is to checkout some base box
definitions. I tend to use Opscode bento repository which
contains Veewee definitions that pre-builds your machines with
Chef 11 already ready to rock. This obviously alleviates the need
to spend time on the nitty-gritty and just get down to the business.
$ git clone git://github.com/opscode/bento.git bento
$ cd bento
$ bundle install
Now that you have the bento repository checked out and all of the
dependencies installed using Bundler you can begin by building
your initial base box. I’m a fan of CentOS but most of the
open source cookbooks out there will work flawlessly with
Ubuntu. So let’s start with using that.
$ bundle exec veewee vbox build 'ubuntu-12.04-i386' --nogui --force
$ bundle exec veewee vbox export 'ubuntu-12.04-i386'
$ vagrant box add 'ubuntu-12.04-i386' 'ubuntu-12.04-i386.box'
The first command that we executed there will take a little while to
finish. But it performs a headless installation of Ubuntu 12.04
32-bit and will overwrite any existing Vagrant VirtualBox
machine that previously built in this directory. They key off of the
ubuntu-12.04-i386 definition name. After that we export to the
Vagrant recognized format.
Because Veewee is using Vagrant to interact with
VirtualBox we actually have all of the flexibility with our
base box definitions. What does that mean exactly? Vagrant
has several plugins which allow us to use many different
providers for provisioning. For local testing it makes sense to use
VirtualBox but perhaps our integration machines are vanilla Linux
and we want to use KVM. Its very simple with Vagrant to
configure our project to have multiple providers for provisioning.
$ vagrant init
But once again for flexibility (and making this tutorial easy) let’s
stick with the basics. Our project is very simple and we just want to
get a virtual machine ready for testing. This means we just want a
simple Ruby and Rack application since I’m more familiar
We need to modify our Vagrantfile that was created using the
vagrant init command. This is the file where we define the
directives necessary for provisioning your virtual machine. Once
again, we’re going to use Ruby and Rack so your mileage
may vary with any other languages or frameworks.
You can clear out all of the default logic, or place this at the top
so that you have that for reference. Here’s the basic changes we’re
going to do:
- Set our base box name to the one that we exported using
- Forward HTTP traffic from the Rack application so it is
accessible to our host machine.
- Only use the amount of memory that we think we’ll need.
- Provision using Chef 11 a Ruby and Rack
Now that we have our Vagrantfile setup we just need to get all of our
cookbook dependencies. I’ve kept this a little light so that you can
easily work through everything. Using the librarian-chef gem we can
define cookbook dependencies similar to using Bundler for gems.
$ bundle exec librarian-chef install
I have defined the path the Unicorn webserver to point to the current
working directory which has been mounted in the guest virtual machine
as /vagrant. This is defined in the rack_application Chef role. I am
not going to go into the Rack application itself, you can take a look
at the whole project’s repository and check it out.
At this point assuming you’ve added the base box to Vagrant you can
vagrant up and see the application from your localhost
running on port 8080. Once the virtual machine is running you can use
vagrant reload if you are aking modifications to any recipes or
roles. I’m going to post some more in depth articles about writing
custom cookbooks in the near future, but I hope that this was
useful. If you have any additional questions feel free to hit me up on
27 Feb 2013
A few days ago I wrote a post about marrying Ruby and C and
provided an easy to follow example gist. After spending some more
days hacking on C/C++ integration with the Ruby virtual machine I
decided that finding some quality examples was surely a real big
pain. Many of the problems that we are facing are directly related to
objects being accessed from multiple threads at the same time in
the Ruby virtual machine.
While learning I found writing some example code definitely
helps. This new example is a little more complex, but pretty much
covers most of the bases in regards to basic object functionality. I
decided to leave out inheritance in this example as it was getting to
be pretty big. My goal was to be able to sit down and write the
example, plus a blog post, in a single night. If it took me longer
than a few hours its probably too complex of an example.
I am going to continue diving into embedding the virtual machine, and
with that I hope to be able to continue to document examples here and
on my github account. Feel free to drop me some patches if you
find errors in the code. Or leave a few comments.
My previous example takes some time and shows very basic Ruby
virtual machine integration with a C application. This example expands
on this and includes how you might wrap a C structure and integrate it
with normal functionality inside of Ruby. Some examples that you’ll
see here include: instance and class variables, instance and class
methods, using the initialize method and yielding to the calling
Be sure to take some time and read the basic example so that
you’re sure to understand how it all works. The script can easily be
changed in the main.c file if you want to
puts some information.