Puppet Infrastructure SOP

This page is intended for people who are already familiar with puppet. It is specific to our Fedora Infrastructure though the techniques are general enough to apply to other environments.

VCS/GIT

All of our configs and manifests are stored in GIT. We also have a private GIT that stores keys, certificates and other files (like sudoers) that shouldn't be messed with or altered but by a select few people.

To make the GIT logs and commit notifications a bit more informative, it is a good idea to set your name and email in your ~/.gitconfig file. The easiest way to do so is:

git config --global user.name "Your Name Here"
git config --global user.email address@example.com

The following commands will check out all GIT configs from a local shell on puppet01.phx2.fedoraproject.org:

git clone /git/puppet

Note that you should not copy or manipulate this repo except on the puppet01 machine.

There is a different procedure for cloning the private repo, as it contains private data and should not be world-readable. This repo is restricted to a small group of people.

RESTORE_UMASK=$(umask -p)
umask 0077
git clone /git/private
$RESTORE_UMASK

or if you've got a checkout already:

cd puppet
git pull

You should pull changes before starting in on any changes yourself.

Altering config files

The config files need to be edited, saved and installed. The following example assumes everything is checked out already and is to alter a proxy file on the webserver.

cd configs
vi web/fedoraproject.org/wiki.conf
git commit -a

Once you have all of your changes in and ready run:

git push

Each puppet client pulls from the puppetmaster from a cron job that runs at 23 and 53 minutes past the hour +-10min. Output from puppet for each client is logged. When it changes it's mailed out, and once a month all outstanding errors from puppet runs are mailed out. There is also a job that checks to make sure all clients have checked in in the last hour, and emails an alert if they have not.

Adding new files

Adding new files consists of two things, adding the file and adding the manifest. Adding the config file is similar to the alteration section above. Just add the file to the appropriate directory, git add filename. Then commit and run 'git push'

Once the config file is in place move to your manifests directory and add the file to whichever manifest seems appropriate. If none do, add a new manifest. Most of our non-Fedora specific manifests exist in various puppet modules. For manifests specific to our setup, there is a manifests directory with a servergroups directory grouping the services into logical machines. The node file (in manifests/nodes) then includes those servergroups.

Adding new node

Adding new hosts of an already known type is easy. The following example is for a builder. First change into the manifests/nodes directory. Then add a new node file (see other node files for an example):

For example

node "xenbuilder3.fedora.phx.redhat.com" {
  include build
}

The "include build" line refers to a 'build' group in the servergroups directory.

Node Access

If this is a new node coming from a previously un-used IP address. We need to give puppet, infrastructure and the accounts system access to the node:

git clone /git/puppet
cd puppet
vi modules/puppet/files/fileserver.conf
vi modules/infra-repo/files/allows
git commit -a
git push

On the node

Do a manual run of puppet:

puppetd -t --server=puppet

This initial run will send a certificate request over to the puppet master. once you have signed the new request (below) go back to the node and re-run the command. It may be necessary to run puppet several times on the first run to make sure that all of the dependencies get worked out.

Signing new ca request

If using one of our official kickstart scripts a server will automatically request a cert from puppet1. To view a list just run (as root)

puppetca --list

followed by a sign of whatever hostname needs to be signed:

puppetca --sign xenbuilder3.fedora.phx.redhat.com

to remove an existing key from the puppet CA (for example when rebuilding a machine), run

puppetca -c xenbuilder3.fedora.phx.redhat.com

first. Note that cleaning the key (puppetca -c) and revoking the key (puppetca -r) are very different concepts. A cleaned key can still be used to access puppet, while a revoked key cannot.

Finalizing

Sometimes a new node needs to have puppet restarted in order to connect correctly after its certificate has been approved. Simply log in and run 'service puppet restart' as root.

Regenerate cert request

To regenerate the cert request (for example when re-kicking a box) first remove the old cert

puppetca -c xenbuilderx.fedora.phx.redhat.com

then restart puppet on the remote node. Then sign the certificate as mentioned above with --sign

Debugging

To debug puppet on a node (have it print verbosely) try this:

sudo run-puppet nowait

This command will generate a lot of output, so use carefully.

Renaming a node

Sometimes you will need to rename a node due to Smooge deciding that a new naming convention is needed. The steps for doing so are the following :

1. Log onto puppet01 and oldhostname in seperate windows.
2. On puppet
   1. Edit associated DNS names in any places (.pp files etc) referring to the old host name in puppet

   2.  sudo /usr/sbin/puppetca --revoke --clean ${OLDHOSTNAME}

3. On oldhostname
   1. edit /etc/sysconfig/network,/etc/hosts and any other files not controlled by puppet.
   2. remove old cert files.

       find /var/lib/puppet/ssl -type f -print | xargs rm -v

   3. create new cert files.

       /usr/sbin/puppetd --enable ; /usr/sbin/puppetd -t

4. On puppet
   1. sign new cert files

      sudo /usr/sbin/puppetd --list; sudo /usr/sbin/puppetd --sign ${NEWHOSTNAME}

5. On oldhostname

   1. /usr/sbin/puppetd -t

      and begin looking for any problems.