Hydration of a Chef Server

At Bloomberg my team is tasked with supporting infrastructure for all of our consumer and subscription web properties. We have been using Chef to solve some of our problems around configuration management. Most of our recipes were built using Chef Solo and eventually deployed on our cloud infrastructure without a Chef Server. As we began to plan to deploy software services to the teams that we support we quickly began to realize that it would be nearly impossible to manage multiple node deployments without Chef Server.

Our immediate need was to provision a Chef Server for our build infrastructure that we would be distributing to about twenty teams. But in addition to this we have brought in Chef to train several classes of our developers. After the classes concluded there were many requests for individual Chef Server tenancies for testing.

Using Chef Server within our infrastructure was an afterthought because most of our virtual machines had a very short lifespan. Since we could have several dozen development teams wanting their own Chef Server deployments we needed a solution to not only easily deploy a Chef Server, but to hydrate it with the data necessary for each OpenStack tenancy. This included both user and client authentication information.

After a bit of back and forth I was able to isolate the requirements for our Chef Server infrastructure:

  1. It must use the Chef Server community cookbook and the Omnibus packages provided by Chef;

  2. It should be quick and easy to deploy a new instance because we may be doing so for several dozen tenancies in OpenStack;

  3. All user and client keys should be generated locally and the server should be hydrated during the initial convergence run;

I was first introduced to Chef Metal at this past year’s ChefConf where John Keiser gave an excellent talk. My team has a lot of experience using Vagrant, but it made much more sense to me to include a metal recipe with our Chef infrastructure cookbook. It could serve as both a teaching aide, and would also make our internal cookbooks much more versatile. Throughout the discovery process I found the community Chef Server Populator cookbook, ended up completely rewriting it to use Chef resources to modify the Chef Server, and ultimately used Chef Zero to bootstrap everything.

Playing with Metal

Using Chef Metal I was able to easily distribute a recipe with our Chef infrastructure cookbook that would provision a Chef Server very easily using the machine resource. The recipe that I began with looked very similar to what is below. It first laid down changes to the base operating system using our base cookbook, and simply used the community cookbook to install the Chef Server from the Omnibus packages. It was very straight forward.

Unfortunately I still needed to hydrate the local data into the newly minted Chef Server. Luckily for me the cats over at Heavy Water Operations have solved this problem and provided the Chef Server Populator cookbook. There were only a few minor modifications that I needed to make to support client validation keys. But other than that everything seemed to be smooth sailing.

Chef Zero

At first, with only a few minor changes, the Chef Server Populator cookbook seemed to be the answer to my problems. I had a few cases that were not covered, but these were all pretty simple extensions. It was natural and made sense to convert our Chef infrastructure cookbook into a Chef repository and use Chef Zero as an in-memory Chef Server during the initial provisioning.

That worked out quite well at first. I was able to cobble together a simple Rake task to generate and write out the private and public keys to JSON data bag items. My metal recipe started to look a lot more complex than I had initially intended, and it didn’t seem as clean as I thought it could be. As I was reading through the code for the Chef Metal Fog driver a light bulb went off in my head: I could use the Cheffish gem Chef resources and build the data on the fly!

Chef within Chef

I ended up completely rewriting the Chef Server Populator cookbook. My fork now used the Chef resources provided by the Cheffish gem to read from data bag items and create the user and client records against the Chef Server node. The beauty of this approach is that I no longer had to execute clunky knife commands since in Chef 11 erchef exposed an API to pass both user and client public keys. The cookbook is now much more clean and simple to grok.

The code for creating the user and client keys was also able to use the same Chef resources from Cheffish. Instead of operating against our newly provisioned Chef Server it executed against the in-memory server provided by Chef Zero.

Using keys which are generated inside of the metal recipe is not idempotent. There were two ways that I went about solving this problem. This first was merely reading in the private keys off disk and delegating the generation to some external process. The second would be to use the Chef resource to generate private keys and allow it to manage if they needed to be updated or not. Ultimately it worked out that if the keys were generated outside of the metal recipe it was much more straightforward. Here is a full example where keys for clients were generated outside, but user keys are generated on the fly:

Tying the Bow

The final result was a complete cookbook refactor which uses Chef resources to hydrate a newly provisioned Chef Server. The default recipe uses node attributes and creates both client and user resources on the new server. A data bag recipe uses a Chef data bag search to find items and applies them to the node attributes prior to including the default recipe.

Since Chef Zero provides a fully functioning in-memory Chef Server we are able to use Chef resources to add data bag items with locally generated public keys. When using the Chef Metal recipe this data is now made accessible to the data bag search, and thus it gives us an elegant way to hydrate a Chef Server. I have provided an example of this inside of my fork of the cookbook which illustrates all of the steps above.

As I mentioned at the top of this post I originally started work on this solution to allow us to quickly deploy our build infrastructure to multiple tenancies within our private cloud. In a future post I will explain how my team was able to create an easily deployable continuous integration framework using the community Jenkins recipe.

If you are reading this post and are interested in joining our growing group of Chef technologists at Bloomberg feel free to reach out to me. We are hiring!

Private Cloud at Home

This weekend one of my colleagues at Bloomberg and I spent a few hours at my apartment this weekend to deploy OpenStack. I made the same attempt last year by myself and was barely able to get a OpenStack service running. It was that hard. But this past year, with the help of a vibrant community, have made the process for deploying a cluster a hell of a lot easier.

Using Fuel from Mirantis, and with a little configuration of my managed switch, I had a two node OpenStack cluster built from eBay servers that I had lying around. One controller node and one compute node we were ready to rock and roll. Even the managed switch configuration for VLAN support went incredibly smooth.

The ultimate goal is to have a relatively portable, simple datacenter so that distributed systems can be easily demonstrated at meetups. Chaz and I will be working on making this process as transparent and automatic as we can.

Even though my cluster is still in pieces on the floor I am hoping to use it to demonstrate some Chef development at future Scale DC meetups. Chaz will be presenting the work that he has done setting up Fuel and his home cloud which puts my completely to shame. After ChefConf 2014 I am hoping to introduce some of the work that the new Web Operations team at Bloomberg have been hacking away at.

Web Operations at Bloomberg

For the past year I’ve been working on a grassroots effort building reusable patterns for infrastructure at Bloomberg Government. Over the course of a year my team was able to stabalize and automate datacenter operations. Because of legacy infrastructure our focus was primarily on the applications rather than the machines themselves.

Of course making application deployments deterministic and improving the continuous delivery pipeline were likely the low hanging fruit at BGOV. But the larger issue needed to be managed at orchestration and provisioning of new virtual machines. The more that we dug into the problem we realized that BGOV was not the only organization at Bloomberg that was suffering these problems.

As of two weeks ago I’ve been given a tiger team of engineers throughout research and development to work with our cloud infrastructure and solve these problems for all of our web verticals. We were given the charge to automate the datacenter operations - including orchestration, provisioning and continuous delivery - and integrate with the enterprise systems that we use throughout our firm.

This goes hand-in-hand with implementing a culture of DevOps where the firm promotes communication, collaboration and integration amongst all of the stakeholders in a business unit. This includes the research and development component of a product, but also the business and technology support teams. The ultimate goal is to deliver a stable, higher quality product to our customers faster.

From Veewee, to Vagrant, then Chef

My good British friend Phil Sturgeon posted an awesome writeup regarding Vagrant and Chef. I’ve been meaning to do a post of my own. Over the last few months a side project of mine has got me involved a lot with Vagrant, Chef and Veewee. But to find resources on the Internet, especially tutorials, which combine all that knowledge is definitely a task I would not wish on anyone else.

So I figured I’d put together a simple little blog post that gets you started building out your own testing environment. Just a heads up: I am going to assume you have the Vagrant dependencies installed. I prefer installing these from the website and not through RubyGems, but to each his own. The version I am using is 1.1.3. Everything else should be picked up automatically from the appropriate files.

I am going to walk you through the steps that I used to build the example application which is available as a Gist. Remember that you can just run this using the vagrant up command and it will be available (in a few minutes) by accessing the 8080 port on your host machine. You’re still going to have to go through the whole Veewee build of the base boxes. But after you add the base box that you’ve built you can just start the application with Vagrant.

The first order of business is to checkout some base box definitions. I tend to use Opscode bento repository which contains Veewee definitions that pre-builds your machines with Chef 11 already ready to rock. This obviously alleviates the need to spend time on the nitty-gritty and just get down to the business.

    $ git clone git://github.com/opscode/bento.git bento
    $ cd bento
    $ bundle install

Now that you have the bento repository checked out and all of the dependencies installed using Bundler you can begin by building your initial base box. I’m a fan of CentOS but most of the open source cookbooks out there will work flawlessly with Ubuntu. So let’s start with using that.

    $ bundle exec veewee vbox build 'ubuntu-12.04-i386' --nogui --force
    $ bundle exec veewee vbox export 'ubuntu-12.04-i386'
    $ vagrant box add 'ubuntu-12.04-i386' 'ubuntu-12.04-i386.box'

The first command that we executed there will take a little while to finish. But it performs a headless installation of Ubuntu 12.04 32-bit and will overwrite any existing Vagrant VirtualBox machine that previously built in this directory. They key off of the ubuntu-12.04-i386 definition name. After that we export to the Vagrant recognized format.

Because Veewee is using Vagrant to interact with VirtualBox we actually have all of the flexibility with our base box definitions. What does that mean exactly? Vagrant has several plugins which allow us to use many different providers for provisioning. For local testing it makes sense to use VirtualBox but perhaps our integration machines are vanilla Linux and we want to use KVM. Its very simple with Vagrant to configure our project to have multiple providers for provisioning.

    $ vagrant init

But once again for flexibility (and making this tutorial easy) let’s stick with the basics. Our project is very simple and we just want to get a virtual machine ready for testing. This means we just want a simple Ruby and Rack application since I’m more familiar with those.

We need to modify our Vagrantfile that was created using the vagrant init command. This is the file where we define the directives necessary for provisioning your virtual machine. Once again, we’re going to use Ruby and Rack so your mileage may vary with any other languages or frameworks.

You can clear out all of the default logic, or place this at the top so that you have that for reference. Here’s the basic changes we’re going to do:

  1. Set our base box name to the one that we exported using Veewee.
  2. Forward HTTP traffic from the Rack application so it is accessible to our host machine.
  3. Only use the amount of memory that we think we’ll need.
  4. Provision using Chef 11 a Ruby and Rack application.

Now that we have our Vagrantfile setup we just need to get all of our cookbook dependencies. I’ve kept this a little light so that you can easily work through everything. Using the librarian-chef gem we can define cookbook dependencies similar to using Bundler for gems.

    $ bundle exec librarian-chef install

I have defined the path the Unicorn webserver to point to the current working directory which has been mounted in the guest virtual machine as /vagrant. This is defined in the rack_application Chef role. I am not going to go into the Rack application itself, you can take a look at the whole project’s repository and check it out.

At this point assuming you’ve added the base box to Vagrant you can just do vagrant up and see the application from your localhost running on port 8080. Once the virtual machine is running you can use vagrant reload if you are aking modifications to any recipes or roles. I’m going to post some more in depth articles about writing custom cookbooks in the near future, but I hope that this was useful. If you have any additional questions feel free to hit me up on Twitter.

Defining C Objects in Ruby

A few days ago I wrote a post about marrying Ruby and C and provided an easy to follow example gist. After spending some more days hacking on C/C++ integration with the Ruby virtual machine I decided that finding some quality examples was surely a real big pain. Many of the problems that we are facing are directly related to objects being accessed from multiple threads at the same time in the Ruby virtual machine.

While learning I found writing some example code definitely helps. This new example is a little more complex, but pretty much covers most of the bases in regards to basic object functionality. I decided to leave out inheritance in this example as it was getting to be pretty big. My goal was to be able to sit down and write the example, plus a blog post, in a single night. If it took me longer than a few hours its probably too complex of an example.

I am going to continue diving into embedding the virtual machine, and with that I hope to be able to continue to document examples here and on my github account. Feel free to drop me some patches if you find errors in the code. Or leave a few comments.

My previous example takes some time and shows very basic Ruby virtual machine integration with a C application. This example expands on this and includes how you might wrap a C structure and integrate it with normal functionality inside of Ruby. Some examples that you’ll see here include: instance and class variables, instance and class methods, using the initialize method and yielding to the calling iterator function.

Be sure to take some time and read the basic example so that you’re sure to understand how it all works. The script can easily be changed in the main.c file if you want to puts some information.