Using Test Doubles in ChefSpec

One of the improvements to ChefSpec in the 3.0 release is the ability to extend test coverage to execute blocks. Doing so requires the infrastructure developer to stub out shell commands run as part of the idempotence check. This is pretty simple as ChefSpec provides a macro to stub shell commands. However doing so where the idempotence check uses a Ruby block is slightly more involved. In this article I explain how to do both.

Quick overview of ChefSpec

ChefSpec is a powerful and flexible unit testing utility for Chef recipes. Extending the popular Ruby testing tool, Rspec, it allows the developer to make assertions about the Chef resources declared and used in recipe code. The key concept here is that of Chef resources. When we write Chef code to build infrastructure, we’re using Chef’s domain-specific language to declare resources – abstractions representing the components we need to build and configure. I’m not going to provide a from-the-basics tutorial here, but dive in with a simple example. Here’s a test that asserts that our default recipe will use the package resource to install OpenJDK.

require 'chefspec'

RSpec.configure do |config|
  config.platform = 'centos'
  config.version = '6.4'
  config.color = true

  describe 'stubs-and-doubles' do

    let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }

    it 'installs OpenJDK' do
      expect(chef_run).to install_package 'java-1.7.0-openjdk'
    end

  end
end

If we run this first, we’ll see something like this:

$ rspec -fd spec/default_spec.rb 

stubs-and-doubles

================================================================================
Recipe Compile Error
================================================================================

Chef::Exceptions::RecipeNotFound
--------------------------------
could not find recipe default for cookbook stubs-and-doubles

  installs OpenJDK (FAILED - 1)

Failures:

  1) stubs-and-doubles installs OpenJDK
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     Chef::Exceptions::RecipeNotFound:
       could not find recipe default for cookbook stubs-and-doubles
     # ./spec/default_spec.rb:10:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:13:in `block (3 levels) in <top (required)>'

Finished in 0.01163 seconds
1 example, 1 failure

Failed examples:

rspec ./spec/default_spec.rb:12 # stubs-and-doubles installs OpenJDK

This is reasonable – I’ve not even written the recipe yet. Once I add the default recipe, such as:

package 'java-1.7.0-openjdk'

Now the test passes:

$ rspec -fd spec/default_spec.rb 


stubs-and-doubles
  installs OpenJDK

Finished in 0.01215 seconds
1 example, 0 failures

Test Doubles

ChefSpec works by running a fake Chef run, and checking that the resources were called, with the correct parameters. Behind the scenes, your cookbooks are loaded, but instead of performing real actions on the system, the Chef Resource class is modified such that messages are sent to ChefSpec instead. One of the key principles of Chef is that resources should be idempotent – the action should only be taken if required, and it’s safe to rerun the resource. In most cases, the Chef provider knows how to guarantee this – it knows how to check that a package was installed, that a directory was created. However, if we use an execute resource – a resource where we’re calling directly to the underlying operating system – Chef has no way to tell if the command we called did the right thing. Unless we explicitly tell Chef how to check, it will just run the command again and again. This causes a headache for ChefSpec, because it doesn’t have a built-in mechanism for faking operating system calls – so when it comes across a guard, it requires us to help it out, by stubbing the command.

This introduces some testing vocabulary – vocabulary that is worth stating explicitly, for the avoidance of confusion. I’m a fan of the approach described in Gerard Meszaros’ 2007 book xUnit Test Patterns: Refactoring Test Code, and this is the terminology used by Rspec. Let’s itemise a quick glossary:

  • System Under Test (SUT) – this is the thing we’re actually testing. In our case, we’re testing the resources in a Chef recipe. Note we’re explictly not testing the operating system.
  • Depended-on Component (DOC) – usually our SUT has some external dependency – a database, a third-party API, or on our case, the operating system. This is an example of a DOC
  • Test Double – when unit testing, we don’t want to make real calls to the DOC. It’s slow, can introduce unwanted variables into our systems, and if it becomes unavailable our tests won’t run, or will fail. Instead we want to be able to interact with something that represents the DOC. The family of approaches to implement this abstraction is commonly referred to as Test Doubles.
  • Stubbing – when our SUT depends on some input from the DOC, we need to be able to control those. A typical approach is to stub the method that makes a call to the DOC, typically returning some canned data.

Let’s look at a real example. The community runit cookbook, when run on a RHEL derivative, will, by default, build an RPM and install it. The code to accomplish this looks like this:

bash 'rhel_build_install' do
      user 'root'
      cwd Chef::Config[:file_cache_path]
      code <<-EOH
tar xzf runit-2.1.1.tar.gz
cd runit-2.1.1
./build.sh
rpm_root_dir=`rpm --eval '%{_rpmdir}'`
rpm -ivh '/root/rpmbuild/RPMS/runit-2.1.1.rpm'
EOH
      action :run
      not_if rpm_installed
end

Observe the guard – not_if rpm_installed. Earlier in the recipe, that method is defined as:

rpm_installed = "rpm -qa | grep -q '^runit'"

ChefSpec can’t handle direct OS calls, and so if we include the runit cookbook in our recipe, we’ll get an error. Let’s start by writing a simple test that asserts that we include the runit recipe. I’m going to use Berkshelf as my dependency solver, which means I need to add a dependency in my cookbook metadata, and supply a Berksfile that tells Berkshelf to check the metadata for dependencies. I also need to add Berkshelf support to my test. My test now looks like this:

require 'chefspec'
require 'chefspec/berkshelf'

RSpec.configure do |config|
  config.platform = 'centos'
  config.version = '6.4'
  config.color = true

  describe 'stubs-and-doubles' do

    let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }

    it 'includes the runit recipe' do
      expect(chef_run).to include_recipe 'runit'
    end

    it 'installs OpenJDK' do
      expect(chef_run).to install_package 'java-1.7.0-openjdk'
    end

  end
end

And my recipe like this:

include_recipe 'runit'
package 'java-1.7.0-openjdk'

Now, when I run the test, ChefSpec complains:

1) stubs-and-doubles includes the runit recipe
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     ChefSpec::Error::CommandNotStubbed:
       Executing a real command is disabled. Unregistered command: `command("rpm -qa | grep -q '^runit'")`"

       You can stub this command with:

         stub_command("rpm -qa | grep -q '^runit'").and_return(...)
     # ./spec/default_spec.rb:11:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:14:in `block (3 levels) in <top (required)>'

ChefSpec tells us exactly what we need to do, but let’s unpack it a little, using the vocabulary from above. The SUT, our stunts-and-doubles cookbook, has a dependency on the operating system – the DOC. This means we need to be able to insert a test double of the operating system, specifically a test stub, which will provide a canned answer to our rpm command. ChefSpec makes it very easy for us by providing a macro that does exactly this. We need to run this before every example, so we can put it in a before block. The new test now looks like this:

require 'chefspec'
require 'chefspec/berkshelf'

RSpec.configure do |config|
  config.platform = 'centos'
  config.version = '6.4'
  config.color = true

  describe 'stubs-and-doubles' do

    before(:each) do
      stub_command("rpm -qa | grep -q '^runit'").and_return(true)
    end

    let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }

    it 'includes the runit recipe' do
      expect(chef_run).to include_recipe 'runit'
    end

    it 'installs OpenJDK' do
      expect(chef_run).to install_package 'java-1.7.0-openjdk'
    end

  end
end

Now when we run the test, it passes:

$ rspec -fd spec/default_spec.rb 

stubs-and-doubles
  includes the runit recipe
  installs OpenJDK

Finished in 0.57793 seconds
2 examples, 0 failures

That’s all fine and dandy, but suppose we execute some Ruby for our guard instead of a shell command. Here’s an example from one of my cookbooks, in which I set the correct Selinux policy to allow apache to proxy to a locally running Netty server:

unless (node['platform'] == 'Amazon' or node['web_proxy']['selinux'] == 'Disabled')
  execute 'Allow Apache Network Connection in SELinux' do
    command '/usr/sbin/setsebool -P httpd_can_network_connect 1'
    not_if { Mixlib::ShellOut.new('getsebool httpd_can_network_connect').run_command.stdout.match(/--> on/) }
    notifies :restart, 'service[httpd]'
  end
end

Now, OK, I could have used grep, but I prefer this approach, and it’s a good enough example to illustrate how we handle this kind of case in ChefSpec. First, let’s write a test:

it 'sets the Selinux policy to allow proxying to localhost' do
  expect(chef_run).to run_execute('Allow Apache Network Connection in SELinux')
  resource = chef_run.execute('Allow Apache Network Connection in SELinux')
  expect(resource).to notify('service[httpd]').to(:restart)
end

If we were to run this, ChefSpec would complain that we didn’t have an execute resource with a run action on our run list. So we then add the execute block from above to the default recipe. I’m going to omit the platform check for simplicity, and just include the execute resource. We’re also going to need to define an httpd service. Of course we’re never going to actually run this code, so I’m not fussed that the service exists despite us never installing Apache. My concern in this article is to teach you about the testing, not write a trivial and pointless cookbook.

Now our recipe looks like this:

include_recipe 'runit'
package 'java-1.7.0-openjdk'

service 'httpd'

execute 'Allow Apache Network Connection in SELinux' do
  command '/usr/sbin/setsebool -P httpd_can_network_connect 1'
  not_if { Mixlib::ShellOut.new('getsebool httpd_can_network_connect').run_command.stdout.match(/--> on/) }
  notifies :restart, 'service[httpd]'
end

When we run the test, we’d expect all to be fine. We’re asserting that there’s an execute resource, that runs, and that it notifies the httpd service to restart. However, this is what we see:

Failures:

  1) stubs-and-doubles includes the runit recipe
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     Errno::ENOENT:
       No such file or directory - getsebool httpd_can_network_connect
     # /tmp/d20140208-30704-g1s3d4/stubs-and-doubles/recipes/default.rb:8:in `block (2 levels) in from_file'
     # ./spec/default_spec.rb:20:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:23:in `block (3 levels) in <top (required)>'

  2) stubs-and-doubles installs OpenJDK
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     Errno::ENOENT:
       No such file or directory - getsebool httpd_can_network_connect
     # /tmp/d20140208-30704-g1s3d4/stubs-and-doubles/recipes/default.rb:8:in `block (2 levels) in from_file'
     # ./spec/default_spec.rb:20:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:27:in `block (3 levels) in <top (required)>'

  3) stubs-and-doubles sets the Selinux policy to allow proxying to localhost
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     Errno::ENOENT:
       No such file or directory - getsebool httpd_can_network_connect
     # /tmp/d20140208-30704-g1s3d4/stubs-and-doubles/recipes/default.rb:8:in `block (2 levels) in from_file'
     # ./spec/default_spec.rb:20:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:31:in `block (3 levels) in <top (required)>'

Finished in 1.11 seconds
3 examples, 3 failures

Boom! What’s wrong? Well, ChefSpec isn’t smart enough to warn us about the guard we tried to run, and actually tries to run the Ruby block. I’m (deliberately) running this on a machine without the ability to run the getsebool command to trigger this response, but on my usual workstation running Fedora, this will silently pass. This is what prompted me to write this article, because my colleague who runs these tests on his mac kept getting this No such file or directory – getsebool httpd_can_network_connect error, despite the Jenkins box (running CentOS) and my workstation working just fine. So – what’s the solution? Well, we need to do something similar to that which ChefSpec did for us earlier. We need to create a test double, only this time it’s Mixlib::ShellOut that we need to stub. There are three steps we need to follow. We need to capture the :new method that is called on Mixlib::ShellOut, and instead of returning canned data, like we did when we called stub_command, we want to return the test double, standing in for the real instance of Mixlib::Shellout, and finally we want to control the behaviour of the test double, making it return the output we want for out test. So, first we need to create the test double. We do that with the double method in Rspec:

shellout = double

This just gives us a blank test double – we can do anything we like with it. Now we need to stub the constructor, and return the double:

Mixlib::ShellOut.stub(:new).and_return(shellout)

Finally, we specify how the shellout double should respond when it receives the :run_command method.

  allow(shellout).to receive(:run_command).and_return('--> off')

We want the double to return a string that won’t cause the guard to be triggered, because we want to assert that the execute method is called. We can add these three lines to the before block:

before(:each) do
  stub_command("rpm -qa | grep -q '^runit'").and_return(true)
  shellout = double
  Mixlib::ShellOut.stub(:new).and_return(shellout)
  allow(shellout).to receive(:run_command).and_return('--> off')
end

Now when we run the test, we’d expect the Mixlib guard to be stubbed, the test double returned, and the test double to respond to having the :run_command method called be that it returns a string which doesn’t match the guard, and thus the execute should run! Let’s give it a try:

Failures:

  1) stubs-and-doubles includes the runit recipe
     Failure/Error: let(:chef_run) {  ChefSpec::Runner.new.converge(described_recipe) }
     NoMethodError:
       undefined method `stdout' for "--> off":String
     # /tmp/d20140208-30741-eynz5u/stubs-and-doubles/recipes/default.rb:8:in `block (2 levels) in from_file'
     # ./spec/default_spec.rb:20:in `block (3 levels) in <top (required)>'
     # ./spec/default_spec.rb:23:in `block (3 levels) in <top (required)>'

Alas! What have we done wrong? Look closely at the error. Ruby tried to call :stdout on a String. Why did it do that? Look at the guard again:

not_if { Mixlib::ShellOut.new('getsebool httpd_can_network_connect').run_command.stdout.match(/--> on/) }

Aha… we need another double. When the first double is called, we need to return something that can accept a stdout call, which in turn will return the string. Let’s add that in:

before(:each) do
  stub_command("rpm -qa | grep -q '^runit'").and_return(true)
  shellout = double
  getsebool = double
  Mixlib::ShellOut.stub(:new).and_return(shellout)
  allow(shellout).to receive(:run_command).and_return(getsebool)
  allow(getsebool).to receive(:stdout).and_return('--> off')
end

Once more with feeling:

$ bundle exec rspec -fd spec/default_spec.rb 

stubs-and-doubles
  includes the runit recipe
  installs OpenJDK
  sets the Selinux policy to allow proxying to localhost

Finished in 0.7313 seconds
3 examples, 0 failures

Just to illustrate how the double interacts with the test, let’s quickly change what getsebool returns:

allow(getsebool).to receive(:stdout).and_return('--> on')

Now when we rerun the test, it fails:

Failures:

  1) stubs-and-doubles sets the Selinux policy to allow proxying to localhost
     Failure/Error: expect(chef_run).to run_execute('Allow Apache Network Connection in SELinux')
       expected "execute[Allow Apache Network Connection in SELinux]" actions [] to include :run
     # ./spec/default_spec.rb:31:in `block (3 levels) in <top (required)>'

This time the guard prevented the execute from running, and as such the resource collection didn’t contain this resource, and so the test failed.

Conclusion

One of the great beauties of ChefSpec (and of course Chef) is that at its heart it’s just Ruby. This means that at almost any point you can reach into the standard Ruby development toolkit for your testing or infrastructure development needs. Hopefully this little example will be helpful to you. If it inspires you to read more about mocking, I can recommend the following resources:

Comments are closed.