An Iterative Approach to Building an API – Part 3: Writing Chef Scripts for Amazon EC2 Deployment

If you’re just joining us, over the past two days we’ve written a RESTful API in Python with Tornado and ElasticSearch as our data backend. We started by stubbing out the API data in YAML files, then switched over into a complete, scalable data store with ElasticSearch.

Today, we’ll write the Chef scripts that will build out the server our API will be hosted on. In this example, we’ll walk through creating a new Amazon Web Services account, adding our private keys to SSH, and writing and deploying a complete server environment with Chef.

Creating Your Amazon Web Services SSH Key

First, you’ll need to log into, or create an Amazon Web Services account if you don’t have one already. Then, click the My Account / Console button to open the drop down and pick AWS Management Console.

Select EC2 to get access to Amazon’s server deployment frontend. We’re going to build our own servers from the command line, but first we’ll need to get a private key from Amazon so we don’t have to use passwords on our server. Instead, we’ll have a cryptographically signed certificate that we’ll use to verify ourselves.

Once at the EC2 console, on the left column there will be Networks and Security. Underneath this is the Key Pairs url. Click this to bring up the key pair management console.

Click create new pair, and you’ll be prompted to enter a name for your key pair. In this example, we’ll use ElasticAPI. After doing so, you’ll automatically download a file named ElasticAPI.pem. We’ll now need to chmod this private key to stop other users from being able to access, then add it to ssh’s list of keys.

$ mv ~/Downloads/ElasticAPI.pem ~/.ssh/
$ cd ~/.ssh
$ chmod 600 ElasticAPI.pem
$ ssh-add ElasticAPI.pem
Identity added: ElasticAPI.pem (ElasticAPI.pem)

Great! Now when we create servers with Amazon, we’ll be able to ssh into them right away, without prompting for passwords. This means we can write scripts to automatically manage our servers without requiring us to type long passwords every time something happens.

Installing knife-solo and Adding Our Amazon Access Keys

So now we need the ability to create Amazon instances from the command line. Chef has a utility named knife which allows you to deploy servers to a few different cloud providers. However, we’re going to use an extension of knife, called knife-solo instead in this example.

Knife-solo adds some utilities to bootstrap an instance of Chef scripts, and also allow for installation of Chef to a brand new server. This means we can get by with just a bare minimum of overhead with Chef, which can honestly get pretty damn confusing.

Installing knife-solo begins with making sure you’ve got a proper installation of Ruby and Gem, and then running the following set of commands to get a new Chef repository working:

$ gem install knife-solo
$ knife-solo kitchen chefAPI
$ cd chefAPI
$ git init
$ git add *
$ git commit

Great! We’ve got an empty Chef repository now, and very soon we can begin adding packages. Now, we can open up ~/.chef/knife.rb and add our Amazon Web Services API information:

log_level          :info                                                                                                                                                                                    
log_location       STDOUT                                                                                                                                                                                   
ssl_verify_mode    :verify_none                                                                                                                                                                             
#chef_server_url    "http://y.t.b.d:4000"                                                                                                                                                                   
#file_cache_path    "/var/cache/chef"                                                                                                                                                                       
#pid_file           "/var/run/chef/client.pid"                                                                                                                                                              
cache_options({ :path => "/var/cache/chef/checksums", :skip_expires => true})                                                                                                                               
signing_ca_user "chef"                                                                                                                                                                                      
Mixlib::Log::Formatter.show_time = true                                                                                                                                                                     
validation_client_name "chef-validator"                                                                                                                                                                     
knife[:aws_ssh_key_id] = "ElasticAPI"                                                                                                                                                                
knife[:aws_access_key_id]     = 'YOURACCESSKEYIDHERE'                                                                                                                                                      
knife[:aws_secret_access_key] = 'YOURSECRETACCESSKEYHERE'

Your Amazon Access Key ID and Secret Access Keys were made when you set up your Amazon account, and can be found in Security Credentials underneath the My Account / Console tab.

Verify knife-solo Works By Creating an Ubuntu Instance

Now we can verify that knife-solo works by invoking the command to bring up an Amazon t1.micro instance.

$ knife ec2 server create -I ami-137bcf7a -x ubuntu -f t1.micro
Instance ID: i-00a8957d
Flavor: t1.micro
Image: ami-137bcf7a
Region: us-east-1
Availability Zone: us-east-1b
Security Groups: default
Tags: {"Name"=>"i-00a8957d"}
SSH Key: ElasticAPI
 
Waiting for server..................
Public DNS Name: ec2-XX-XXX-XXX-XX.compute-1.amazonaws.com
Public IP Address: XX.XXX.XXX.XXX
Private DNS Name: domU-XX-XXX-XXX-XX.compute-1.internal
Private IP Address: XX.XXX.XXX.XXX
 
Waiting for sshd.done
Bootstrapping Chef on ec2-XX-XXX-XXX.compute-1.amazonaws.com

You might get an error after the last line, and that’s perfectly alright. Verify that the instance was created, and your ssh key works by doing an ssh to the server IP address you got from above, using the ubuntu username:

$ ssh -i ~/.ssh/ElasticAPI.pem ubuntu@XX.XXX.XX.XX
Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-29-virtual x86_64)
 
 * Documentation:  https://help.ubuntu.com/
 
  System information as of Sun Oct 14 21:23:54 UTC 2012
 
  System load:  0.08              Processes:           67
  Usage of /:   21.4% of 7.97GB   Users logged in:     0
  Memory usage: 46%               IP address for eth0: 10.210.218.255
  Swap usage:   0%
 
  Graph this data and manage this system at https://landscape.canonical.com/
 
42 packages can be updated.
22 updates are security updates.
 
Get cloud support with Ubuntu Advantage Cloud Guest
  http://www.ubuntu.com/business/services/cloud
ubuntu@domU-XX-XX-XX-XX-XX-XX:~$ exit

Finally! Now we can begin coding the actual Chef code, which should go fairly quickly, now that we have a way to quickly test whether or not the scripts are written correctly.

Adding Our Dependencies to Chef

Chef scripts are all written in Ruby. You download recipes, which are simple Ruby scripts that describe how to build pieces of your server.

The great people at Opscode, who created Chef have already built the majority of scripts you’ll need. So all we end up doing is cloning their repos for what we need, then writing own own recipe which describes how to use the standard recipes.

Because we’re going the simplified route, we’re just going to clone the repos by hand. So let’s begin:

$ cd elasticChef/cookbooks
$ git clone git://github.com/opscode-cookbooks/apt.git
$ git clone git://github.com/opscode-cookbooks/build-essential.git
$ git clone git://github.com/opscode-cookbooks/java.git
$ git clone git://github.com/opscode-cookbooks/python.git
$ git clone git://github.com/opscode-cookbooks/ark.git
$ git clone git://github.com/opscode-cookbooks/sudo.git
$ git clone git://github.com/opscode-cookbooks/gunicorn.git

All those commands installed the most very basic set of utilities to get our server up and running. We can now create our own recipe to invoke the proper installation of software.

$ mkdir elasticServer
$ touch elasticServer/metadata.rb
$ mkdir elasticServer/recipes
$ touch elasticServer/recipes/default.rb

Now, open elasticServer/metadata.rb, and add our dependencies to it:

depends "apt"
depends "git"
depends "build-essential"
depends "python"
depends "gunicorn"

Pretty straightforward file, right? Now, let’s actually write the bit of code that builds our server. This should be in recipes/default.rb:

include_recipe "build-essential"
include_recipe "python::default"
include_recipe "gunicorn::default"                                                                                                                                                                          
include_recipe "elasticsearch::default"
 
%w{emacs git-core rlwrap openjdk-6-jdk tmux curl tree unzip nginx python-setuptools python-dev build-essential supervisor}.each do |pkg|
  package pkg do
    action :install
  end
end
 
service "nginx" do
  enabled true
  running true
  supports :status => true, :restart => true, :reload => true
  action [:start, :enable]
end
 
python_virtualenv "/home/ubuntu/elasticEnv" do
    interpreter "python2.7"
    owner "ubuntu"
    group "ubuntu"
    action :create
end

Save this file, and we’re completely done with all the Chef scripts we’ll need to write. We just now need to create a node file that describes our node.

Adding and Building Our Node

In our elasticChef directory, you may have noticed a nodes/ directory. This is where we describe how individual nodes get built. The final step before we can push a server configuration is creating a file in nodes with the IP address of your server and the extension of json.

So, if your public Amazon IP address from before was 11.22.33.44, your file would be in nodes/ and called 11.22.33.44.json. And that file would consist of the following:

{                                                                                                                                                                                                           
    "run_list": [ "recipe[elasticServer]" ]
}

Save it, and we can now deploy our API server!

$ knife prepare ubuntu@11.22.33.44
$ knife cook ubuntu@11.22.33.44

The first command uploads Chef to the server and installs it. The second command uploads our cookbook to the server, and then installs all the software we need.

If everything went well, we should be able ssh to your server, curl localhost (curl localhost), and see an nginx message that the web server was installed.

Opening the Firewall for the HTTP Server

Amazon manages the server’s firewall with Security Groups, found underneath the Network and Security tab on your EC2 Console. If you click the default group here, you will see details pop up in the window below.

Click inbound, and you’ll see the options to create a new rule. Under port range enter 80, which is the default HTTP port. Leave source as 0.0.0.0/0 to allow the entire internet to see your server. Then, click Add Rule below then Apply Rules. You should now be able to go to http://11.22.33.44 from your web browser and see the server you created.

Tomorrow, We Deploy!

Great! Now we’ve configured the web server, and tomorrow we’ll write the fabric scripts to automatically deploy our code to the server. If you want to review again, here are the links to what we’ve accomplished so far:

Part One – Building the API skeleton with YAML
Part Two – From YAML to ElasticSearch
Part Three – Writing Chef Scripts for Amazon Deployment
Part Four – Writing Fabric Scripts for Code Deployment

Feel free to leave any questions or comments below.

As before, the code for everything is already finished and available at github, and so are the Chef scripts.

7 thoughts on “An Iterative Approach to Building an API – Part 3: Writing Chef Scripts for Amazon EC2 Deployment

  1. Is there really supposed to be an executable called knife-solo? From what I could find, knife-solo is a plugin for knife and after a gem install knife-solo you don’t get a knife-solo executable (at least that I can find). I’m referring to the line where you do:
    knife-solo kitchen chefAPI

    I was not able to do that as directed, but running:
    knife kitchen chefAPI

    seemed to work, albeit with this warning:
    “WARNING: No knife configuration file found”

  2. knife-ec2 gem is now required to do the section where you test out your knife configuration. Running
    knife ec2 server create -I ami-13bcf7a -x ubuntu -f t1.micro

    will yield this error message:

    FATAL: Cannot find sub command for: ‘ec2 server create -I ami-13bcf7a -x ubuntu -f t1.micro’
    The ec2 commands were moved to plugins in Chef 0.10
    You can install the plugin with `(sudo) gem install knife-ec2

  3. At the beginning of this part you create a directory called chefAPI but then throughout the rest of the text you talk about an elasticChef directory- I assume these were meant to be the same directory, otherwise this doesn’t make sense.

  4. you should clarify that the code starting with the line:
    include_recipe “build-essential”

    belongs in default.rb, this is not stated and makes it unclear whether you are talking about appending that code to metadata.rb

  5. Given the directory names you chose, I believe this line:
    include_recipe “elasticsearch::default”

    is incorrect and instead should be:
    include_recipe “elasticServer::default”

    • Actually, this was an error in omission. In this line, I was calling an elasticsearch recipe I tried using earlier, but didn’t end up making it into the final code.

      Thanks for all your contributions to this post though, appreciate the comments! It’s a huge help making sure everyone can use this!

  6. On a new AWS account, you will also need to add a rule to the default security group allowing access to port 22 for the ssh you use throughout the rest of this section to work. As far as I can tell, this wasn’t mentioned here.

Leave a Reply to sebastian ortiz Cancel reply

Your email address will not be published. Required fields are marked *