Both Puppet and Chef are incredibly powerful tools, and they are great at allowing your servers to keep themselves up to date.

However, there comes a time in every sysadmin’s life when they find there is a need to poll a large number of servers in their infrastructure. They’re faced with two problems:

1. How do you get a list of all of your current servers?
2. How do you connect to them all quickly?

Both Puppet and Chef are able to solve these problems for us.

Generating your server list

Let’s look at the first problem: How do you get a list of all of your current servers?
The answer can depend on where you typically store your list of servers:

• On a piece of paper
• In a text file
• In a spreadsheet
• In a database

The configuration management systems themselves also store a list of known servers internally, and getting at this list is very easy. If you accept that all of your servers run puppet or chef, then you can use the system to get at the list.

Chef:
HOSTS=$(knife status | awk '$4 ~ /example.com/{ print $4 }')
knife returns a nice display of the list of known servers and when they last ran chef-client. The fourth column is the FQDN of the server.

Puppet:
HOSTS=$(puppetca -la | awk '{ print $2 }')
puppetca‘s output is less pretty, simply showing which servers are known. The second column is the FQDN of the server.

It happens that over time, servers will be removed, renamed or be offline when you you do this work, so you want to make a list of servers which have an SSH daemon listening. Nagios is a great monitoring system, and it comes with a plugin called check_ssh, which tests to make sure you can SSH to a server. We can use this to generate a list of servers:

HOSTS_AVAIL=$( for host in ${HOSTS}; do check_ssh -t1 ${host} > /dev/null && echo ${host}; done )

Running your distributed commands

The second problem with have is: How do you connect to them all quickly?

Here we can drop back to some fun shell scripting. The following loop runs multiple ssh connections at the same time, up to some predefined limit. For this step, it is highly advised that you have a way to log in to your servers without having to type your password in each time. SSH keys are a great way to accomplish this.

MAX=5 # Don't run more than 5 SSH sessions at once
CMD="uptime" # The command you want to run
for server in ${HOSTS_AVAIL}; do
    while :; do
        if [ $(ps ax | grep -E "ssh.*${CMD}") -ge 5 ]; then
            sleep 1
        else
             break
        fi
    done
    ssh ${server} "${CMD}" >> ${server}.log 2>&1 &
done

This might look complicated, but in reality it’s quite simple:

• Define the maximum number of processes to run, and the command to run on the remote servers
• For each server…
• If we have more than 5 processes running, sleep 1 second and check again
• Once there is a slot open for a process to run, execute it into the background, and set the output to ${server}.log and loop around to the beginning immediately

Alternative methods

Chef specific way:
Chef allows us to display hosts which are subscribed specific roles. We can use this connect to all of the chef clients quickly:

knife ssh "$( for host in
    $(knife status | awk '$4 ~ /example.com/{ print $4 }'); do
        /opt/local/libexec/nagios/check_ssh -t 1 ${host} > /dev/null && \
        echo ${host}; done)" \
    "ps ax | grep httpd" \
    -m -x<username> -P'<password>' -l debug