From Fedora Project Wiki

Fedora Classroom - Configuration Management using Puppet - Jeroen van Meeuwen - Sunday, November 9, 2008

IRC Log of the Class

kanarip so this session is about configuration management, which does *not* include the soft side of configuration management for a change 23:04
*Bugz_ is 23:04
kanarip we're *not* going to talk about procedural issues, but keep it on the technical side 23:04
kanarip i'd like to invite you all to take a look at 23:04
kanarip which is a nice reader i wrote for a one-day workshop i'm giving on this topic, and that is commercially available 23:04
kanarip community people like us however *do not have to pay* 23:04
kanarip sounds great, huh? ;-) 23:04
domg472_ yes 23:04
erinlea80 :) 23:04
delhage from Delft eh? 23:04
kanarip so, a little something on configuration management first, then a sneak peak at the installation of puppet, and a quick howto on writing your "manifest" as they call the puppet configuration syntax 23:04
kanarip delhage, Utrecht 23:04
delhage ok 23:04
kanarip so, what is it we would want to do with configuration management in the technical sense of the words 23:04
kanarip we all know that the more systems we need to maintain, the harder it becomes to keep some extend of consistency between different systems 23:04
kanarip some organizations use scripts from an NFS share, or use an SVN tree with scripts periodically updated and execute scripts 23:05
kanarip does that sound familiar to anyone? 23:05
*erinlea80 nods 23:05
kanarip so what happens in these cases is: 23:05
kanarip 1) your scripts tend to get very large 23:05
kanarip describing every unique situation, every exemption, every update, for every operating system, distribution, distribution version, makes the script less maintainable 23:05
kanarip 2) your scripts are executed at arbitrary intervals 23:06
kanarip either a cronjob, on boot, may or may not fail, and you cannot make the changes you want pronto 23:06
kanarip 3) your scripts replace configuration and reconfigure stuff without actual changes being applied 23:06
kanarip either of these 3 things cries out for a configuration management like... 23:07
kanarip Puppet (or CFEngine) 23:07
kanarip anyone heard of CFEngine before? 23:07
delhage yup 23:07
jds2001 however cfengine is fail 23:07
jds2001 we use it :) 23:07
kanarip jds2001, you're in the right spot ;-) 23:07
kanarip what CFEngine does is very accurately describe the configuration state a system has to be in 23:08
kanarip it does it so accurately, in fact, that it ends up being a very low-level tool 23:08
kanarip to give you an example; 23:08
kanarip in CFEngine, to install a package called foo, you will have to manually assign a package provider to each operating system, distribution or distribution version, and then use that provider to install the package 23:09
kanarip in addition, you would need to change the name for each OS, distro and distribution version 23:09
kanarip lots to type, lots to care about, while in fact all you care about is that the package is installed 23:10
*kanarip introduces... the high-level abstraction of Puppet 23:10
kanarip no matter what the operating system, distribution or distribution may be, Puppet speaks the local language 23:10
kanarip it'll figure out on it's own whether to use yum, up2date, dpkg, apt, PackageKit, alien, smart, rpm or what-package-manager-have-you 23:11
kanarip this way, ensuring that package foo is installed on a system becomes: 23:11
kanarip package { "foo": 23:11
kanarip ensure => installed 23:11
kanarip } 23:11
kanarip and you're done 23:11
kanarip jds2001, can you confirm that is easier then CFEngine please? ;-) 23:11
jds2001 yes :) 23:12
kanarip thanks ;-) 23:12
kanarip so, how does puppet work? 23:12
kanarip given these abstracted "system resources" such as packages, services, cronjobs, users, ..., this and that and more, puppet uses so-called "types" you can use in a configuration "manifest" to describe what state a managed system should be in 23:13
kanarip here's a list of types: 23:13
kanarip now let me describe to you what a working setup looks like; 23:14
kanarip there is a single server, called the puppetmaster 23:15
kanarip personally i think the puppeteer would have been a more appropriate name, but puppetmaster it is 23:15
kanarip the managed system is called a "puppet" - whos strings you can pull using the puppetmaster 23:15
kanarip the puppet, or managed system, polls the puppetmaster at a default interval of every 30 minutes 23:16
kanarip the puppetmaster, having *all* the configuration for *all* nodes in an environment, will parse *all* it's manifests, and send to the puppet a set of resources to be managed on that specific node 23:17
nirik Q: do they all pull at once? or is it staggered? 23:17
kanarip nirik, it depends on the time at which the puppet starts counting 23:17
kanarip it has a default 1 minute randomization i believe 23:17
kanarip but it'll poll at least once every 30.9 minutes, if that makes sense ;-) 23:18
domg472_ why does it do that? 23:18
kanarip to check if there's any changes to the configuration for the node 23:19
domg472_ if yo have changes on the master can you just tell it to push them? 23:19
domg472_ ok thanks 23:19
kanarip if you change anything, you let the master pick it up and at most 59 later the puppet will have picked up on the changes 23:20
kanarip that is, if you are using the default interval of 30 minutes 23:20
kanarip sorry, should have said: at most 30.9 minutes after you apply changes, puppet will have picked up the changes 23:21
kanarip so here's how the puppetmaster distinguishes configuration to be applied to one node:;a=blob;f=puppet/manifests/nodes/ 23:21
daMaestro is there any way to push changes? or is it all pull operations? 23:21
daMaestro (in the case of needing an isolated DMZ/VLAN) 23:21
kanarip the puppetmaster will load that file and know that which is an actual system is to be configured accordingly 23:22
neverho0d is there way to stop propagating changes? 23:22
domg472_ evry 30 minutes sounds like overhead 23:22
kanarip daMaestro, puppet provides a utility called puppetrun which tells the client or clients to do a puppet run 23:22
daMaestro kanarip, yet this is still a pull? 23:22
kanarip neverho0d, yes; 23:23
kanarip 1) stage your changes in environments 23:23
neverho0d kanarip: ok 23:23
kanarip 2) stop the puppetmaster 23:23
kanarip 3) stop the puppet client daemon 23:23
*daMaestro attempts to not interrupt anymore 23:23
kanarip 4) let the puppetmaster not pull from a SCM to load new configuration 23:23
kanarip daMaestro, the puppet client will perform a pull from the puppetmaster, yes 23:23
daMaestro k 23:24
kanarip any other questions so far? 23:24
kanarip i'd like to dive into modules, if that's ok? 23:24
erinlea80 any utilities to automate data/config collections on pre-existing servers? 23:24
erinlea80 or is it all by hand? 23:24
kanarip erinlea80, it's very complex to harvest data and then roll it out but i guess a certain scale justifies putting in such effort 23:25
kanarip the harvesting would only happen once is what i'm concerned with 23:25
erinlea80 agreed -- was wondering what the options are. 23:26
kanarip so here's a sample module for puppet:;a=tree 23:26
kanarip a module is a collection of the manifest: manifests/init.pp, the files used by that module (in files/), any templates used by the module (in templates/), and more 23:26
kanarip for a reference 23:27
kanarip and of course there's something mentioned in as well 23:27
kanarip what makes modules so great is a couple of things; 23:27
kanarip 1) i can share it with you and you can just start using it 23:28
kanarip 2) you can customize it and still pull changes from an upstream SCM like the GIT repository I just pointed you to and send me/upstream patches 23:28
kanarip 3) modules allow "staging" 23:28
kanarip i guess the next topic is "staging" ;-) 23:29
kanarip 4) modules keep your files, manifests, templates and plugins organized 23:29
kanarip believe me when I say puppet configuration trees can become a mess if you start managing more and more with puppet, and not use modules 23:29
kanarip == Staging == 23:29
kanarip What I mean by staging is simply this; stage your changes from a development environment, possibly via a testing environment onto your production environment 23:30
kanarip you can develop all you want in your development environment, make syntax errors, rm -rf / sorta speak, and never nuke that business critical application server you have running 23:31
kanarip then once you're satisfied, you may move the changes into the testing or production environment 23:32
kanarip does that make sense? 23:32
nuonguy oh yeah 23:32
neverho0d sure 23:32
jds2001 yes, but how does one do this in practice? 23:32
kanarip awesome 23:32
erinlea80 mmmm yes 23:32
erinlea80 and does it require a unique environment for each corresponding production environment? 23:32
kanarip i can recommend anyone getting his feet wet in using environments 23:32
*nirik has some questions, but can save them for the end as well. 23:32
erinlea80 unique test environ. 23:32
kanarip jds2001, here's a practical example; 23:32
kanarip all the repositories you see on have a development, a testing, and a production branch 23:33
kanarip what i do when i'm satisfied with my developments, is I change the working branch to testing, and git pull development 23:34
kanarip then when i push, the puppetmaster runs a puppet that is configured so that it will pull the changes from the GIT repo 23:34
jds2001 then you have post-receive hooks to put them someplace? 23:34
kanarip;a=blob;f=puppet/manifests/nodes/ as a reference 23:35
kanarip jds2001, no, the puppetmaster in my case runs a puppet that has ^^ configuration applied to itself 23:35
kanarip a post-hook making the puppetmaster pull in the changes is a viable solution too; the Fedora Project does such 23:36
kanarip however, in my situation, the puppetmaster and the SCM are not on the same machine 23:36
jds2001 ok, makes sense 23:37
erinlea80 kk 23:37
kanarip is a presentation slide-deck BTW, that goes along with the .pdf i've already linked 23:37
kanarip note that what you see on is heavily dependent on the other modules available from 23:37
kanarip so... what do you think about setting it up? 23:38
erinlea80 lets do it? 23:39
kanarip there you go ;-) 23:39
kanarip let me refer to the documentation i shared earlier because it's a lot to type 23:39
kanarip 23:39
kanarip it's a yum install puppet-server to install it on any Fedora system 23:40
kanarip for Red Hat and CentOS, you'll need EPEL configured 23:40
kanarip 23:40
*erinlea80 nods 23:40
kanarip for anyone who's wondering about what EPEL is... I maintain or co-maintain the entire Ruby stack to puppet in that repository ;-) 23:41
kanarip and that brings me to one more nifty feature of puppet; 23:41
kanarip but first... facts! 23:42
kanarip facts are... well... facts... 23:42
kanarip little information about the current state of a client 23:42
kanarip it includes the kernel version, number of network interfaces, the architecture, operating system name, operating system version, and lots more 23:43
kanarip yum install facter and run "facter" on your machine to check it out 23:43
kanarip these are all "variables" you can use in your puppet manifests 23:43
kanarip given a small amount of RAM for example, you may want to apply a snippet similar to: package { "firefox": ensure => absent } 23:44
kanarip anyway, what i really wanted to say rather then to give a rant on firefox's memory consumption is this; 23:44
kanarip facts, types and providers, these things that make puppet do smart things... 23:44
kanarip facts for information about the client... 23:45
erinlea80 do we have a sample facter output? 23:45
kanarip i could fpaste you some 23:45
erinlea80 that would be great :) 23:45
zcat kanarip, is facter supposed to spit out "virtual => vmware_server" if detected? 23:45
domg472_ creating new ones doesnt look that easy 23:45
kanarip 23:46
domg472_ but theres good documentation on create types fact etc 23:46
kanarip types to define resources to be managed on the puppet... 23:46
erinlea80 thanks 23:47
kanarip and providers to make puppet speak the local language... 23:47
kanarip the great thing is... you can write your own for all of these 23:47
kanarip it does require ruby knowledge, but then again... there's sample recipes on for...for example zcat, a vmware custom fact ;-) 23:47
kanarip see, i was getting there ;-) 23:47
kanarip actually i think the custom fact is called "this is a vmware machine" and "whether it has vm-tools installed" 23:48
kanarip one more thing i really, really need to stress out to you 23:48
kanarip all communication between the puppet and it's puppetmaster is *secure* 23:49
*nirik has one of his questions answered. ;) 23:49
kanarip that being said, the security is based on the puppetmaster running the Certificate Authority, the client generating a certificate and putting in a request with the puppetmaster to get a signed copy of that certificate 23:50
erinlea80 can communication be tunneled over ssh if necessary? (firewall restrictions, etc.) 23:50
domg472_ but the puppets have root access obviously? 23:50
domg472_ and the master 23:50
kanarip erinlea80, you can make the puppetmaster run on any port you want 23:50
kanarip erinlea80, may i point out "autossh" with which you can create tunnels to communicate over if you wish 23:50
erinlea80 ty :) 23:50
kanarip domg472_, the puppet client daemon runs with root privileges, yes 23:51
domg472_ has anyone tested this with selinux? 23:51
kanarip domg472_, the master runs with under the puppet user 23:51
domg472_ ok thanks 23:51
kanarip domg472_, i have, and i can tell you it requires a custom selinux policy in which you basically describe the puppet daemon can do this and that and more 23:52
kanarip in my case, the puppet daemon can do anything it likes 23:52
domg472_ youd need custom modules i bet 23:52
domg472_ interesting 23:52
sdodson There are also a few minor bugs which I think have been fixed recently where puppet creates files without the proper contexts. 23:53
kanarip selinux is being integrated better into puppet as we speak 23:53
domg472_ great 23:53
kanarip it's a big issue, i concur 23:53
nuonguy Q: if I enable, for example httpd in a manifest, and 2 hours later comment it out, does the puppet uninstall/disable httpd or ignore it from then on? 23:53
kanarip nuonguy, puppet only applies changes 23:54
kanarip nuonguy, so it only enforces what you describe needs to happen 23:54
kanarip if you say package httpd should be installed at one point 23:54
nuonguy so I'd have to explicity turn it off, if that's what I meant? 23:54
kanarip and then comment that out... no other machine is going to care whether the package should be installed, but it's not going to be removed on puppet's behalf 23:55
kanarip nuonguy, yes 23:55
nuonguy I dig, thanks 23:55
kanarip nuonguy, same goes for the service type; ensure => running, enable => true would need to become ensure => stopped, enable => false 23:56
nuonguy precisely 23:56
nuonguy I can just see myself not remembering that... 23:56
kanarip so, another thing on the topic; what do you think happens if the puppet daemon cannot contact the puppetmaster? 23:56
nuonguy and commenting it out instead 23:56
nirik does it do them in order in the manifest? ie, make sure to stop before ensure 23:56
kanarip nuonguy, there's some discussion going on about supporting an "exclude" statement like we do "include webserver" now 23:57
nuonguy nice 23:57
kanarip nuonguy, and then flipping all the false's into true's and so forth 23:57
kanarip nirik, good point! 23:57
sdodson Could create a baseline that ensures none of that is installed/running then include webserver for only your webservers which would override those baselines? 23:57
kanarip puppet allows ordering the resources; 23:57
kanarip let me look you up an example 23:57
domg472_ why was ruby chosen as the language for writing types etc? 23:58
domg472_ why not bash? 23:59
kanarip here's a yp manifest: 23:59
kanarip domg472_, i'm not sure i'm not one of the core developers, but i can try and answer it if you promise not to quote me on it ;-) 00:00
domg472_ hah ok 00:00
domg472_ i think most sysadmins know bash 00:00
kanarip nirik, you can see how "require => (...)" in that yp manifest causes one resource to be dependent on another 00:00
domg472_ and they will need to write custom types probably 00:00
kanarip domg472_, i think bash is what you use to do a quick thing, clean and simple. I think ruby allows a framework to be built, reusing snippets of code you've already written in a different place. I think ruby is more modular then is bash 00:01
domg472_ ok thanks 00:02
kanarip you could write a website in bash but you won't, right? 00:02
domg472_ i guess ill be learning ruby then 00:02
domg472_ true 00:02
domg472_ but these are simplate admin tasks 00:02
domg472_ simple 00:02
kanarip not saying you should choose php to do it BTW ;-) 00:02
*nirik notes we are over time, but this is also the last class today. 00:02
kanarip in puppet terms; distribute the bash scripts amongst the clients that need it using the file type, and exec that script 00:03
nirik I had one last question: how do you decide what should be in a module? whats the logical boundry there? 00:03
kanarip anything you; 00:03
domg472_ ah right, thanks kanarip 00:03
kanarip 1) pull from an upstream source 00:03
kanarip 2) need to share (with the general public) 00:04
kanarip 3) want to stage 00:04
nirik so it doesn't need to be a specific service or anything? just whatever you want to put in one? 00:04
kanarip 4) that is a concise set of files, manifests, etc. 00:04
sdodson They're advising people to get away from writing classes and move towards modules. 00:04
kanarip nirik, you're almost right, a module called everything-else doesn't make sense ;-) 00:05
kanarip sdodson, yes, very much so 00:05
kanarip nirik, a module webserver makes sense; keeping it close and concise within a module is way easier then dispersing it all over the place 00:05
nirik ok, makes sense. 00:06
tmz domg472_: regarding why ruby was used to write puppet, here's a short interview with Luke Kanies (primary puppet author), where he touches on that choice: 00:06
tmz 00:06
domg472_ thanks 00:06
kanarip what i tend to recommend to customers is also, to keep the systems they manage as consistent as possible 00:06
kanarip being able to determine the exceptions to the rule you've layed down saves you a lot of time in determining what the cause might be for a specific issue 00:07
nirik shall we wrap up and let kanarip get some sleep? 00:08
kanarip ;-) 00:08
daMaestro no, no sleep for kanarip ! 00:08
kanarip thank you all for your interest, and thank you for attending 00:08
*daMaestro /dcc kanarip latte 00:08
jds2001 he must stay up all night! 00:08
nirik thanks so much for the session kanarip ! 00:08
erinlea80 thanks kanarip! 00:08
domg472_ thanks again 00:08
erinlea80 :) 00:08
iondrip thanks 00:08
kanarip i hope you find a lot of use in the pdf/html/odp i've given you ;-) 00:08
neverho0d kanarip: thanks 00:08
nirik thanks everyone for coming... 00:09
Bugz_ kanarip: Thanks 00:09
erinlea80 :) 00:09
domg472_ thanks for hosting 00:09
SSlater Thanks 00:09

Generated by 2.7 by Marius Gedminas - find it at!