Run commands against all of your Puppet or Chef hosts

Both Puppet and Chef are incredibly powerful tools, and they are great at allowing your servers to keep themselves up to date.

However, there comes a time in every sysadmin’s life when they find there is a need to poll a large number of servers in their infrastructure. They’re faced with two problems:

  1. How do you get a list of all of your current servers?
  2. How do you connect to them all quickly?

Both Puppet and Chef are able to solve these problems for us.

Generating your server list

Let’s look at the first problem: How do you get a list of all of your current servers?
The answer can depend on where you typically store your list of servers:

  • On a piece of paper
  • In a text file
  • In a spreadsheet
  • In a database

The configuration management systems themselves also store a list of known servers internally, and getting at this list is very easy. If you accept that all of your servers run puppet or chef, then you can use the system to get at the list.

Chef:
HOSTS=$(knife status | awk '$4 ~ /example.com/{ print $4 }')
knife returns a nice display of the list of known servers and when they last ran chef-client. The fourth column is the FQDN of the server.

Puppet:
HOSTS=$(puppetca -la | awk '{ print $2 }')
puppetca‘s output is less pretty, simply showing which servers are known. The second column is the FQDN of the server.

It happens that over time, servers will be removed, renamed or be offline when you you do this work, so you want to make a list of servers which have an SSH daemon listening. Nagios is a great monitoring system, and it comes with a plugin called check_ssh, which tests to make sure you can SSH to a server. We can use this to generate a list of servers:

HOSTS_AVAIL=$( for host in ${HOSTS}; do check_ssh -t1 ${host} > /dev/null && echo ${host}; done )

Running your distributed commands

The second problem with have is: How do you connect to them all quickly?

Here we can drop back to some fun shell scripting. The following loop runs multiple ssh connections at the same time, up to some predefined limit. For this step, it is highly advised that you have a way to log in to your servers without having to type your password in each time. SSH keys are a great way to accomplish this.

MAX=5 # Don't run more than 5 SSH sessions at once
CMD="uptime" # The command you want to run
for server in ${HOSTS_AVAIL}; do
    while :; do
        if [ $(ps ax | grep -E "ssh.*${CMD}") -ge 5 ]; then
            sleep 1
        else
             break
        fi
    done
    ssh ${server} "${CMD}" >> ${server}.log 2>&1 &
done

This might look complicated, but in reality it’s quite simple:

  • Define the maximum number of processes to run, and the command to run on the remote servers
  • For each server…
  • If we have more than 5 processes running, sleep 1 second and check again
  • Once there is a slot open for a process to run, execute it into the background, and set the output to ${server}.log and loop around to the beginning immediately

Alternative methods

Chef specific way:
Chef allows us to display hosts which are subscribed specific roles. We can use this connect to all of the chef clients quickly:

knife ssh "$( for host in
    $(knife status | awk '$4 ~ /example.com/{ print $4 }'); do
        /opt/local/libexec/nagios/check_ssh -t 1 ${host} > /dev/null && \
        echo ${host}; done)" \
    "ps ax | grep httpd" \
    -m -x<username> -P'<password>' -l debug

Creating an SVN mirror

If you haven’t had the pleasure of working with a distributed version control system such as Mercurial or Git, you may find that it necessary to take some manual steps to set up a mirror of your repository.

For SVN, this process isn’t well integrated into the system itself, but there are tools we can use to achieve this goal. An important thing to be aware of, is that unlike a distributed, you won’t be able to merge changes from your mirror back into the original repo. You should make sure the only thing writing to the mirrored copy is the svnsync process.

Tthe documentation on the internet is pretty terse for this, here’re the steps which worked for me.

I wanted to locally mirror a repo which is normally servered over https (although it’d work equally well for repos servers using svn or svn+ssh).

  • Remote repo: https://svn.example.org/repos/project
  • Local directory: /home/svn/project

Create an empty SVN repository locally:
svn create /home/svn/project

You need to create a hook script at /home/svn/project/hooks/pre-revprop-change which looks like:

#!/bin/sh

USER="$3"

exit 0

if [ "$USER" = "syncuser" ]; then exit 0; fi

echo "Only the syncuser user may change revision properties" >&2
exit 1

Make the script executable:
chmod 755 /home/svn/project/hooks/pre-revprop-change

Now initialise the repo as a mirror:
svnsync init file:///home/svn/project https://svn.example.org/repos/project

Finally, you can do the sync:
svnsync sync file:///home/svn/project

I have that last command run from cron every minute to keep things up to date:
/etc/cron.d/svnsync
# Sync the SVN mirror every minute
* * * * * root /usr/bin/svnsync sync file:///home/svn/project

Resizing a Solaris partition

This was originally worked out when I had to resize a Solaris 10 partition under VMware, but it applies equally to native Solaris installs too!
WARNING: This only really works, if you have one partition. This should be true if you’re using VMware, and for the majority of Solaris installations. Partition, here, is in the true sense of the word: partitions as seen by fdisk, the BIOS and other disk management software, not the slices you mount as /usr, /home, and so on. Those are, errr, slices.
If you’re doing this for a VMware install, power off the virtual machine, and use vmware-vdiskmanager to resize the image file: vmware-vdiskmanager -x 20Gb solaris10_vmware.vmdk.
If the partition you are resizing is the root partition, you need to boot up in “failsafe” mode, or boot from a Solaris CD/DVD and choose the “shell” option at the first menu. You’ll be asked if you want to mount the hard drive install as /a. You do not. But you do want to remember the name of the disk (eg, c1t0d0p0).

Write out the current partition table to disk and edit the file:
# fdisk -W ptbl.tmp && vi ptbl.tmp

At the top of the file, you’ll see the actual current disk geometry, which should look somethign like this:

* Dimensions:
*    512 bytes/sector
*     63 sectors/track
*    255 tracks/cylinder
*   2610 cylinders

The sectors, tracks and cylinders numbers are important – write these down somewhere, and skip to the end of the file.

You’ll see a row of numbers, with words over them like Id, Act, Bhead etc. The two numbers you need to replace, correspond with Ecyl and Numsect.

Ecyl is the ending cylinder on the disk for this partition. The value for it, is the cylinders number that you wrote down in the previous step, MINUS ONE. The minus one is very important. While the total number of cylinders in my example is 2610, the ending cylinder number is actually 2609, because the numbering of cylinders on disk starts at zero, not one. This is the only time you need to worry about this.
You also need to change Numsect, which is the number of sectors on the disk. The value for this, is calculated as: sectors/track x tracks/cylinder x cylinders.
In our example, this is: 63 x 255 x 2610 = 41929650.

Save the file, and exit.

Now update the fdisk partition table on disk:
# fdisk -S ptbl.tmp -I /dev/rdsk/c1t0d0p0
Replace the partition name, with that of your disk. You’ll be presented with a little menu. The table at the top should indicate that 100% of the disk space is now used by the partition(s). Choose option 5, to save and exit fdisk.

Finally, we get to resize the actual partitions! Reboot into multiuser mode.

Look in /etc/vfstab to find the device that is mounted, to the slice you want to grow. In my case, this was the root slice mounted from /dev/dsk/c1t0d0s0.
We need to turn this into a metadevice (think: software raid), to be able to grow it. Don’t worry, this is perfectly (mostly) safe, and won’t hurt your performance (much, as far as I know!).

metainit -f d10 1 1 c1t0d0s0. This creates a metadevice named d10, from the given slice name. The -f causes this to be forced, which is needed if the slice is currently mounted.
Run metastat. This will give you the name of the new metadevice you created. For me, it was called d0.
If you are working on resizing the root filesystem, you need to run metaroot /dev/md/dsk/d0. This will update /etc/vfstab with the correct mount information. Otherwise, you need to update /etc/vfstab manually.