Puppet Infrastructure SOP

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(add some more info)
(redirect page to new infra-docs)
 
(4 intermediate revisions by 2 users not shown)
Line 2: Line 2:
 
{{shortcut|ISOP:PUPPET}}
 
{{shortcut|ISOP:PUPPET}}
  
This page is intended for people who are already familiar with puppet.  It is specific to our Fedora Infrastructure though the techniques are general enough to apply to other environments.
 
  
== VCS/GIT ==
+
This SOP has moved to the fedora Infrastructure SOP git repo. Please see the current document at: http://infrastructure.fedoraproject.org/infra/docs/puppet.txt
All of our configs and manifests are stored in GIT. We also have a private GIT that stores keys, certificates and other files (like sudoers) that shouldn't be messed with or altered but by a select few people.
+
  
To make the GIT logs and commit notifications a bit more informative, it is a good idea to set your name and email in your <code>~/.gitconfig</code> file. The easiest way to do so is:
+
For changes, questions or comments, please contact anyone in the Fedora Infrastructure team.  
  
<pre>
 
git config --global user.name "Your Name Here"
 
git config --global user.email address@example.com
 
</pre>
 
 
The following commands will check out all GIT configs from a local shell on puppet01.phx2.fedoraproject.org:
 
 
<pre>
 
git clone /git/puppet
 
</pre>
 
 
Note that you should not copy or manipulate this repo except on the puppet01 machine.
 
 
There is a different procedure for cloning the private repo, as it
 
contains private data and should not be world-readable. This repo is restricted to a small group of people.
 
<pre>
 
RESTORE_UMASK=$(umask -p)
 
umask 0077
 
git clone /git/private
 
$RESTORE_UMASK
 
</pre>
 
 
or if you've got a checkout already:
 
<pre>
 
cd puppet
 
git pull
 
</pre>
 
 
You should pull changes before starting in on any changes yourself.
 
 
== Altering config files ==
 
The config files need to be edited, saved and installed.  The following example assumes everything is checked out already and is to alter a proxy file on the webserver.
 
 
<pre>
 
cd configs
 
vi web/fedoraproject.org/wiki.conf
 
git commit -a
 
</pre>
 
 
Once you have all of your changes in and ready run:
 
<pre>
 
git push
 
</pre>
 
 
Each puppet client pulls from the puppetmaster from a cron job that runs at 23 and 53 minutes past the hour +-10min. Output from puppet for each client is logged. When it changes it's mailed out, and once a month all outstanding errors from puppet runs are mailed out. There is also a job that checks to make sure all clients have checked in in the last hour, and emails an alert if they have not.
 
 
== Adding new files ==
 
Adding new files consists of two things, adding the file and adding the manifest.  Adding the config file is similar to the alteration section above.  Just add the file to the appropriate directory, git add filename.  Then commit and run 'git push'
 
 
Once the config file is in place move to your manifests directory and add the file to whichever manifest seems appropriate.  If none do, add a new manifest.  Most of our non-Fedora specific manifests exist in various puppet modules.  For manifests specific to our setup, there is a manifests directory with a servergroups directory grouping the services into logical machines.  The node file (in manifests/nodes) then includes those servergroups.
 
 
== Adding new node ==
 
Adding new hosts of an already known type is easy.  The following example is for a builder.  First change into the manifests/nodes directory.  Then add a new node file (see other node files for an example):
 
 
For example
 
 
<pre>
 
node "xenbuilder3.fedora.phx.redhat.com" {
 
  include build
 
}
 
</pre>
 
 
The "include build" line refers to a 'build' group in the servergroups directory.
 
 
=== Node Access ===
 
 
If this is a new node coming from a previously un-used IP address.  We need to give puppet, infrastructure and the accounts system access to the node:
 
 
<pre>
 
git clone /git/puppet
 
cd puppet
 
vi modules/puppet/files/fileserver.conf
 
vi modules/infra-repo/files/allows
 
git commit -a
 
git push
 
</pre>
 
 
=== On the node ===
 
 
Do a manual run of puppet:
 
<pre>
 
puppetd -t --server=puppet
 
</pre>
 
 
This initial run will send a certificate request over to the puppet master.  once you have signed the new request (below) go back to the node and re-run the command.  It may be necessary to run puppet several times on the first run to make sure that all of the dependencies get worked out.
 
 
=== Signing new ca request ===
 
If using one of our official kickstart scripts a server will automatically request a cert from puppet1.  To view a list just run (as root)
 
<pre>
 
puppetca --list
 
</pre>
 
 
followed by a sign of whatever hostname needs to be signed:
 
 
<pre>
 
puppetca --sign xenbuilder3.fedora.phx.redhat.com
 
</pre>
 
 
to remove an existing key from the puppet CA (for example when rebuilding a machine), run
 
<pre>
 
puppetca -c xenbuilder3.fedora.phx.redhat.com
 
</pre>
 
 
first.  Note that cleaning the key (puppetca -c) and revoking the key (puppetca -r) are very different concepts.  A cleaned key can still be used to access puppet, while a revoked key cannot.
 
 
=== Finalizing ===
 
Sometimes a new node needs to have puppet restarted in order to connect correctly after its certificate has been approved.  Simply log in and run 'service puppet restart' as root.
 
 
=== Regenerate cert request ===
 
To regenerate the cert request (for example when re-kicking a box) first remove the old cert
 
<pre>
 
puppetca -c xenbuilderx.fedora.phx.redhat.com
 
</pre>
 
then restart puppet on the remote node.  Then sign the certificate as mentioned above with --sign
 
 
== Debugging ==
 
To debug puppet on a node (have it print verbosely) try this:
 
<pre>
 
sudo run-puppet nowait
 
</pre>
 
 
This command will generate a lot of output, so use carefully.
 
 
== Renaming a node ==
 
 
Sometimes you will need to rename a node due to Smooge deciding that a new naming convention is needed. The steps for doing so are the following :
 
 
# Log onto puppet01 and oldhostname in seperate windows.
 
# On puppet
 
## Edit associated DNS names in any places (.pp files etc) referring to the old host name in puppet
 
## <pre> sudo /usr/sbin/puppetca --revoke --clean ${OLDHOSTNAME}</pre>
 
# On oldhostname
 
## edit <b>/etc/sysconfig/network</b>,<b>/etc/hosts</b> and any other files not controlled by puppet.
 
## remove old cert files. <pre> find /var/lib/puppet/ssl -type f -print | xargs rm -v</pre>
 
## create new cert files. <pre> /usr/sbin/puppetd --enable ; /usr/sbin/puppetd -t</pre>
 
# On puppet
 
## sign new cert files <pre>sudo /usr/sbin/puppetd --list; sudo /usr/sbin/puppetd --sign ${NEWHOSTNAME}</pre>
 
# On oldhostname
 
## <pre>/usr/sbin/puppetd -t</pre> and begin looking for any problems.
 
  
 
[[Category:Infrastructure SOPs]]
 
[[Category:Infrastructure SOPs]]

Latest revision as of 18:51, 19 December 2011

Infrastructure InfrastructureTeamN1.png
Shortcut:
ISOP:PUPPET


This SOP has moved to the fedora Infrastructure SOP git repo. Please see the current document at: http://infrastructure.fedoraproject.org/infra/docs/puppet.txt

For changes, questions or comments, please contact anyone in the Fedora Infrastructure team.