Private Cloud at Home

This weekend one of my colleagues at Bloomberg and I spent a few hours at my apartment this weekend to deploy OpenStack. I made the same attempt last year by myself and was barely able to get a OpenStack service running. It was that hard. But this past year, with the help of a vibrant community, have made the process for deploying a cluster a hell of a lot easier.

Using Fuel from Mirantis, and with a little configuration of my managed switch, I had a two node OpenStack cluster built from eBay servers that I had lying around. One controller node and one compute node we were ready to rock and roll. Even the managed switch configuration for VLAN support went incredibly smooth.

The ultimate goal is to have a relatively portable, simple datacenter so that distributed systems can be easily demonstrated at meetups. Chaz and I will be working on making this process as transparent and automatic as we can.

Even though my cluster is still in pieces on the floor I am hoping to use it to demonstrate some Chef development at future Scale DC meetups. Chaz will be presenting the work that he has done setting up Fuel and his home cloud which puts my completely to shame. After ChefConf 2014 I am hoping to introduce some of the work that the new Web Operations team at Bloomberg have been hacking away at.

Web Operations at Bloomberg

For the past year I've been working on a grassroots effort building reusable patterns for infrastructure at Bloomberg Government. Over the course of a year my team was able to stabalize and automate datacenter operations. Because of legacy infrastructure our focus was primarily on the applications rather than the machines themselves.

Of course making application deployments deterministic and improving the continuous delivery pipeline were likely the low hanging fruit at BGOV. But the larger issue needed to be managed at orchestration and provisioning of new virtual machines. The more that we dug into the problem we realized that BGOV was not the only organization at Bloomberg that was suffering these problems.

As of two weeks ago I've been given a tiger team of engineers throughout research and development to work with our cloud infrastructure and solve these problems for all of our web verticals. We were given the charge to automate the datacenter operations - including orchestration, provisioning and continuous delivery - and integrate with the enterprise systems that we use throughout our firm.

This goes hand-in-hand with implementing a culture of DevOps where the firm promotes communication, collaboration and integration amongst all of the stakeholders in a business unit. This includes the research and development component of a product, but also the business and technology support teams. The ultimate goal is to deliver a stable, higher quality product to our customers faster.

From Veewee, to Vagrant, then Chef

My good British friend Phil Sturgeon posted an awesome writeup regarding Vagrant and Chef. I've been meaning to do a post of my own. Over the last few months a side project of mine has got me involved a lot with Vagrant, Chef and Veewee. But to find resources on the Internet, especially tutorials, which combine all that knowledge is definitely a task I would not wish on anyone else.

So I figured I'd put together a simple little blog post that gets you started building out your own testing environment. Just a heads up: I am going to assume you have the Vagrant dependencies installed. I prefer installing these from the website and not through RubyGems, but to each his own. The version I am using is 1.1.3. Everything else should be picked up automatically from the appropriate files.

I am going to walk you through the steps that I used to build the example application which is available as a Gist. Remember that you can just run this using the vagrant up command and it will be available (in a few minutes) by accessing the 8080 port on your host machine. You're still going to have to go through the whole Veewee build of the base boxes. But after you add the base box that you've built you can just start the application with Vagrant.

The first order of business is to checkout some base box definitions. I tend to use Opscode bento repository which contains Veewee definitions that pre-builds your machines with Chef 11 already ready to rock. This obviously alleviates the need to spend time on the nitty-gritty and just get down to the business.

    $ git clone git:// bento
    $ cd bento
    $ bundle install

Now that you have the bento repository checked out and all of the dependencies installed using Bundler you can begin by building your initial base box. I'm a fan of CentOS but most of the open source cookbooks out there will work flawlessly with Ubuntu. So let's start with using that.

    $ bundle exec veewee vbox build 'ubuntu-12.04-i386' --nogui --force
    $ bundle exec veewee vbox export 'ubuntu-12.04-i386'
    $ vagrant box add 'ubuntu-12.04-i386' ''

The first command that we executed there will take a little while to finish. But it performs a headless installation of Ubuntu 12.04 32-bit and will overwrite any existing Vagrant VirtualBox machine that previously built in this directory. They key off of the ubuntu-12.04-i386 definition name. After that we export to the Vagrant recognized format.

Because Veewee is using Vagrant to interact with VirtualBox we actually have all of the flexibility with our base box definitions. What does that mean exactly? Vagrant has several plugins which allow us to use many different providers for provisioning. For local testing it makes sense to use VirtualBox but perhaps our integration machines are vanilla Linux and we want to use KVM. Its very simple with Vagrant to configure our project to have multiple providers for provisioning.

    $ vagrant init

But once again for flexibility (and making this tutorial easy) let's stick with the basics. Our project is very simple and we just want to get a virtual machine ready for testing. This means we just want a simple Ruby and Rack application since I'm more familiar with those.

We need to modify our Vagrantfile that was created using the vagrant init command. This is the file where we define the directives necessary for provisioning your virtual machine. Once again, we're going to use Ruby and Rack so your mileage may vary with any other languages or frameworks.

You can clear out all of the default logic, or place this at the top so that you have that for reference. Here's the basic changes we're going to do:

  1. Set our base box name to the one that we exported using Veewee.
  2. Forward HTTP traffic from the Rack application so it is accessible to our host machine.
  3. Only use the amount of memory that we think we'll need.
  4. Provision using Chef 11 a Ruby and Rack application.

Now that we have our Vagrantfile setup we just need to get all of our cookbook dependencies. I've kept this a little light so that you can easily work through everything. Using the librarian-chef gem we can define cookbook dependencies similar to using Bundler for gems.

    $ bundle exec librarian-chef install

I have defined the path the Unicorn webserver to point to the current working directory which has been mounted in the guest virtual machine as /vagrant. This is defined in the rack_application Chef role. I am not going to go into the Rack application itself, you can take a look at the whole project's repository and check it out.

At this point assuming you've added the base box to Vagrant you can just do vagrant up and see the application from your localhost running on port 8080. Once the virtual machine is running you can use vagrant reload if you are aking modifications to any recipes or roles. I'm going to post some more in depth articles about writing custom cookbooks in the near future, but I hope that this was useful. If you have any additional questions feel free to hit me up on Twitter.

Defining C Objects in Ruby

A few days ago I wrote a post about marrying Ruby and C and provided an easy to follow example gist. After spending some more days hacking on C/C++ integration with the Ruby virtual machine I decided that finding some quality examples was surely a real big pain. Many of the problems that we are facing are directly related to objects being accessed from multiple threads at the same time in the Ruby virtual machine.

While learning I found writing some example code definitely helps. This new example is a little more complex, but pretty much covers most of the bases in regards to basic object functionality. I decided to leave out inheritance in this example as it was getting to be pretty big. My goal was to be able to sit down and write the example, plus a blog post, in a single night. If it took me longer than a few hours its probably too complex of an example.

I am going to continue diving into embedding the virtual machine, and with that I hope to be able to continue to document examples here and on my github account. Feel free to drop me some patches if you find errors in the code. Or leave a few comments.

My previous example takes some time and shows very basic Ruby virtual machine integration with a C application. This example expands on this and includes how you might wrap a C structure and integrate it with normal functionality inside of Ruby. Some examples that you'll see here include: instance and class variables, instance and class methods, using the initialize method and yielding to the calling iterator function.

Be sure to take some time and read the basic example so that you're sure to understand how it all works. The script can easily be changed in the main.c file if you want to puts some information.

Marrying Ruby with C

For a few weeks now I've spent some time working on embedding the Ruby virtual machine into an application service for a Celluloid server spike. Specifically we took the trunk release of Ruby 1.9.3 and started integrating our service framework utilizing some of our tried and true libraries inside of our server code. This is a similar approach that you would take when developing a native extension - the commonly used mysql2 gem is an example of this - where an interface API is written to expose functionality into the virtual machine.

We took a similar approach. The beauty of the Ruby language is that it is very easy to digest, and even easier to interface with. I've written up a small example gist which I have included below that illustrates how easily you can embed the virtual machine into an existing C application with very little effort.