Bootstrapping a chef client with roles

18-Feb, 2014

I am slowly playing around more and more with chef and the more I see of it the more I like. One of things that you can do is assign roles and recipes when a client is boostrapped. This means that its default configuration is achieved during bootstrapping. To do this run the following command:

knife bootstrap oxygen.justnudge.com -r 'role[jndefault]'

Where the host is called “oxygen.justnudge.com” and we are assigning the role “jndefault” to it.

Comments

Adding a new node in chef is easy

17-Dec, 2013

I am currently playing around with chef. As I mentioned earlier, the command line client for controlling Chef is called Knife (appropriately enough given the cooking theme). To add a new node to the configuration you run the following command:

C:\Users\mransley\workspace\jn-infrastructure>knife bootstrap cobalt.justnudge.com
Bootstrapping Chef on cobalt.justnudge.com
Failed to authenticate root - trying password auth
Enter your password:
cobalt.justnudge.com --2013-12-17 12:12:43--  https://www.opscode.com/chef/install.sh
cobalt.justnudge.com Resolving www.opscode.com (www.opscode.com)... 184.106.28.90
cobalt.justnudge.com Connecting to www.opscode.com (www.opscode.com)|184.106.28.90|:443... connected.
cobalt.justnudge.com HTTP request sent, awaiting response... 200 OK
cobalt.justnudge.com Length: 14101 (14K) [application/x-sh]
cobalt.justnudge.com Saving to: ???STDOUT???
cobalt.justnudge.com
100%[======================================>] 14,101      --.-K/s   in 0s
cobalt.justnudge.com
cobalt.justnudge.com 2013-12-17 12:12:45 (325 MB/s) - written to stdout [14101/14101]
cobalt.justnudge.com
cobalt.justnudge.com Downloading Chef 11.8.0 for el...
cobalt.justnudge.com downloading https://www.opscode.com/chef/metadata?v=11.8.0&prerelease=false&p=el&pv=6&m=x86_64
cobalt.justnudge.com   to file /tmp/install.sh.765/metadata.txt
cobalt.justnudge.com trying wget...
cobalt.justnudge.com url        https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.8.0-1.el6.x86_64.rpm
cobalt.justnudge.com md5        8f72d94dffd75c3f79dc0fcd457e7f11
cobalt.justnudge.com sha256     1a1e3df541240dcf5da1d128b1a4341224db5a2e1dee83b3f0f5d41e0aadb2a3
cobalt.justnudge.com downloaded metadata file looks valid...
cobalt.justnudge.com downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.8.0-1.el6.x86_64.rpm
cobalt.justnudge.com   to file /tmp/install.sh.765/chef-11.8.0.x86_64.rpm
cobalt.justnudge.com trying wget...
cobalt.justnudge.com Checksum compare with sha256sum succeeded.
cobalt.justnudge.com Installing Chef 11.8.0
cobalt.justnudge.com installing with rpm...
cobalt.justnudge.com warning: /tmp/install.sh.765/chef-11.8.0.x86_64.rpm: Header
 V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
cobalt.justnudge.com Preparing...
################################# [100%]
cobalt.justnudge.com Updating / installing...
cobalt.justnudge.com    1:chef-11.8.0-1.el6
################################# [100%]
cobalt.justnudge.com Thank you for installing Chef!
cobalt.justnudge.com Starting Chef Client, version 11.8.0←[0m
cobalt.justnudge.com Creating a new client identity for cobalt.justnudge.com using the validator key.←[0m
cobalt.justnudge.com resolving cookbooks for run list: []←[0m
cobalt.justnudge.com Synchronizing Cookbooks:←[0m
cobalt.justnudge.com Compiling Cookbooks...←[0m
cobalt.justnudge.com [2013-12-17T12:13:29+10:00] WARN: Node cobalt.justnudge.com  has an empty run list.
cobalt.justnudge.com Converging 0 resources←[0m
cobalt.justnudge.com Chef Client finished, 0 resources updated←[0m

That method of connectivity certainly appears simpler than IEM based on my experience.

Comments

A comparison of Chef and TEM

11-Dec, 2013

Build automation has been an interest of mine for a long time (probably about ten years, which shows my age). I have always believed that our development environments should be as close to our production environments as possible and the best way to do that is to build them from the same code base.

Initially this was done by tools such as ANT that configured the application server and then deployed the application. Overtime, this became too clumsy and new tools were developed that built our infrastructure from bare metal up. There are probably many tools that cover this, but to my research the three tools that I have found are:

  • Chef from OpsCode
  • IBM (or Tivoli) End Point Manager
  • Puppet

Now, to precursor this conversation - I didn’t want to spend a lot of cash, but I did manage to play with a trial license of TEM, but for the other two products I used the open source product rather than the enterprise product.

IBM End Point Manager

I tend to run mainly a Linux (Centos and Fedora) environment and was hoping to run IEM on a CentOS host. From what I can see (and this is by no means definitive) the code base between IEM on Windows and IEM on Linux is completely different. The windows installation was relatively simple but the Linux installation I couldn’t get to work at all. My feeling was the linux is a fairly immature build for this product.

That said, once the product is installed, it is relatively easy to connect hosts to it. IEM has a great feature in that you can automatically group hosts (either for fixlets, groups or tasks) into groups depending on its configuration. For example, a Centos host can be automatically placed into the “Centos” group by the relevance rules. This allows hosts to be autogrouped as their configuration changes.

The concept of a fixlet is also interesting, as the name would imply the fixlet allows you to fix something on the host. There is a three step process with a fixlet:

  1. Find hosts for which the fixlet is “relevant”. i.e. which hosts have the vulnerability that needs to be fixed.
  2. Execute the fix (i.e. resolve the issue).
  3. Ensure that the fixlet has resolved the issue (i.e. is the vulnerability removed).

The core concept of TEM is a push concept where the jobs are “pushed” from the server out to the clients. My issue with this in TEM is two fold:

  1. The client is absolutely enormous and chews a lot of resource on the host.
  2. The push sometimes takes a huge amount of time and it is not clear where it is stalled (or not that I can see). A job that takes a few minutes to execute can take a long time to queue and it is diffcult to work out where the bottleneck is.

Chef

I compared Chef and Puppet for a long time. The main drawback of puppet from an open source point of view is the enormous complexity with the installation. Chef seems to me to a much simplified installation. Once Chef is setup its architecture is very different to IEM. Firstly, Chef is a pull framework where the client’s pull their configuration from the server. There is no real concept of a push from the server (although there is a feature available in the closed source version). Additionally, Chef has no concept of relevance from what I can see, but it deals with that in other interesting ways. Lastly, whereas in IEM the “code” is embedded in the tool, for Chef the code is embeded in a version control system such as git.

Chef uses a kitchen analogy for a lot of its terms (which is ironic because I burn toast). The work in chef is defined in cookbooks, that contain one or more recipes. These recipes can either be assigned to a host singularly, or via a role.

As an example, I have the following recipes defined in my infrastructure:

  • Centos - For configuring Centos servers to the allowable standard.
  • Fedora - For configuring Fedora servers to the allowable standard.
  • NAS - For configuring my NAS servers (HTTP, SMB etc)
  • Jekyll - for configuring my development environment for my blog staging.
  • Django on Heroku - for configuring my local django environments for Heroku.
  • vmware - for installing and configuring vmware tools.
  • SSH - for configuring SSH daemon and some standard SSH keys in my environment.

Now, within chef, you can either assign hosts to a cookbook, or to a role. The role is literally a group of cookbooks:

For example, my Centos role looks like this:

name "centos"
description "CentOS Host"
run_list "recipe[jnrepositories]", "recipe[sshconfig]", "recipe[sshkeys]", "recipe[vmware]", "recipe[cron]"

Consequently, the process in relation to chef is assigning your hosts to the various recipes and roles to which they are associated with. This is done via the chef administration console:

Chef Server Screenshot

Chef server push

The reality is that there are sometimes when you want to push configurations from the central server. I have worked out two methods to do this with the open source Chef:

  1. To ensure your environment is up to date, I call chef client each hour by sticking a reference to the “chef-client” command in the /etc/cron.hourly directory. This not only ensures that your environment is up-to-date but it allows you to push changes to your environment within an hour of uploading them to your chef server.
  2. There is also the ability to run commands on either the whole estate or various sub-sections by sending commands through knife (the command line client for chef). The syntax for doing this is as follows:
knife ssh "role:centos" "yum update -y" --ssh-user root

Final statements

I wouldn’t call myself an expert in either Chef or IEM, but having done the review they are both fine automation toolsets with a different approach to performing environment automation. I shall continue to use Chef because it suits my environment and provides less overhead, but in another circumstance I would certainly recommend IEM as well.

Comments

More Posts..