The 4 key causes of SELinux errors (20090503 Classroom)

From FedoraProject

Jump to: navigation, search

Given by Dan Walsh (irc: dwalsh)


OpenOffice Slides


dwalsh This is going to be the session on "What is SELinux trying to tell me?" 12:59
dwalsh My name is Dan Walsh. I am in the Boston area. 12:59
dwalsh I put together some slides for this presentation 13:00
dwalsh 13:00
dwalsh Where is everyone else? It is 9 AM here. 13:01
* ianweller waves 13:01
dgrift_ Rotterdam Netherlands 15:00 13:01
maximus007 florida 13:01
ianweller kansas, 8am 13:01
nihed Tunisia 13:01
pluvo Germany, 15:00 13:01
nihed Tunisia, 14:00 13:01
linuxguru india , 18:30 13:02
dwalsh Cool. 13:02
dwalsh I have been working on and off with the Fedora Infrastructure team on implementing SELinux and learned quite a bit. 13:03
dwalsh A few months ago I wrote a blog on it called 13:03
dwalsh Top three things to understand in fixing SELinux problems. 13:03
dwalsh 13:03
dwalsh What I found is just about every machine we went on required a little bit of massaging 13:04
dwalsh So I came up with this list. 13:04
dwalsh I plan on writing a paper on Securing Fedora Infrastructure and explaining everything we did for Red Hat Magazine. And presenting on it at Red Hat Summit in September. 13:05
dwalsh Anyways lets start. 13:05
dwalsh The #1 thing I try to explain to people about SELinux is LABELING. 13:06
dwalsh SELinux is all about labeling. Every process has a label, every file has a label. 13:06
dwalsh If the labels are wrong, SELinux breaks down. 13:06
dwalsh So when a problem happens in SELinux the first thing to check is the labeling. 13:06
dwalsh One example of labeling getting screwed up is in the vmware package. 13:07
dwalsh In the RPM spec, they decided to edit the /etc/services file. 13:07
dwalsh They did this by copying it to /tmp and adding a port number and then moving it back to /etc 13:08
dwalsh Well in SELinux if you cp /etc/services to /tmp; the tmp file gets labeled rpm_script_tmp_t 13:08
dwalsh But the mv command maintains context, so mv moves the file back to /etc and the file is still labeled rpm_script_tmp_t. 13:09
dwalsh Now every confined domain that tries to read /etc/services is going to blow up because the label is wrong. 13:09
dwalsh Fixing the problem is as simple as restorecon /etc/services. 13:09
dwalsh But SELinux generates nasty avc's saying apache process tried to read rpm_script_tmp_t. 13:10
dwalsh BTW if anyone has questions, feel free to ask at any time. 13:10
dwalsh When we did the Fedora infrastructure, the admins decided they wanted apache content served out of /srv/web. 13:11
icarus-c dwalsh: are you meaning that sometimes avc throws "wrong" message? 13:11
dwalsh No it throws the correct message, but the message is caused by bad labeling. 13:11
dwalsh The /srv/web context on a RHEL5 box was var_t 13:12
dwalsh But apache policy does not allow httpd_t to read/write var_t. so we needed to change the labeling. 13:13
dwalsh We could use chcon to change the labels. 13:13
dwalsh chcon -R -t http_sys_content_t /srv/web 13:14
dwalsh But this does not survive a relabel. 13:14
dwalsh You need to TELL selinux about the change from default labeling. 13:14
dwalsh So you use the semanage command 13:14
dwalsh semanage fcontext -a -t http_sys_content_t '/srv/web(/.*)?' 13:15
dwalsh This tells SELinux to label all files/directories under /srv/web as apache content. 13:15
dwalsh semanage fcontext -a -t httpd_sys_content_t '/srv/web(/.*)?' 13:15
dwalsh Should have been httpd_sys_content_t 13:15
dwalsh In SELinux we use regular expressions to figure out the labeling. So that is why we have the weird file expression. 13:16
dwalsh After you tell SELinux about the labeling, you need to change the actual files on disk 13:16
dwalsh restorecon -R -v /srv/web 13:16
dwalsh Now your httpd daemon is able to read the files. 13:17
dwalsh If you wanted to be able to write the files you could have set the labels to 13:17
dwalsh httpd_sys_content_rw_t. 13:17
dwalsh Questions? 13:17
moixs One :) 13:18
BounceCat dwalsh: if you want us to follow along in your slides, please say when to change slides. thanks. 13:18
dwalsh Ok, This is slide 3 13:18
dwalsh SELinux == Labeling. 13:18
linuxguru Sometimes it still doesn't work i mean setting up context to httpd_sys_content_rw_t in case of a typical dokuwiki installation in /var/www/html/ i faced this situation. 13:18
moixs When you make these changes, can you save them and replicate them easily on another system? 13:18
BounceCat ok 13:18
dwalsh The notes section has decent content also 13:18
dwalsh moixs, In Fedora infrastructure we are using puppet to basically execute the semanage commands on a large group of machines. 13:19
dwalsh We plan on using IPA also for this in RHEL6 time frame. 13:20
dwalsh linuxguru, Please explain? 13:20
moixs Ok, seems reasonable 13:20
dgrift_ one can also use a policy package to distribute file contexts across systems 13:20
dwalsh The commands are pemanent. 13:20
linuxguru dwalsh, it said me to enable a boolean , httpd_unified 13:20
dwalsh linuxguru, Yes that is a bug in httpd_unified, in F10. 13:20
linuxguru okay 13:21
dwalsh linuxguru, I will cover the boolean in the next section. 13:21
linuxguru dwalsh, okay sir 13:21
dwalsh In this case it is more secure to leave httpd_unified off and set the context to httpd_sys_content_rw_t. 13:21
dwalsh A couple more comments on labeling. 13:21
linuxguru yeah. i had to do that for every file it complained off. 13:22
linuxguru in random directories sometimes. so not sure if should apply on directory or not 13:22
dwalsh SELinux handles labels at the directory level better then individual files. 13:22
dwalsh So if you can label the directory and all its contents with a single label, SELinux becomes more managable. 13:23
dwalsh For example, If I have a confined domain that wants to manage files in /etc. 13:23
dwalsh It would be better to have the app manage /etc/myapp/* 13:24
dwalsh And label /etc/myapp as myapp_etc_rw_t 13:24
dwalsh That way by default all files created in /etc/myapp will be created as myapp_etc_rw_t 13:24
dwalsh If I just put the files in /etc. they would default to etc_t and then I would have to rely on the administrator maintaining the label. 13:25
dwalsh If an administrator decides to create his own top level directory under / 13:25
dwalsh The label will be labeled default_t. 13:25
dwalsh NO confined apps are allowed to use default_t for that reason. 13:26
dwalsh We expect the admin might be putting the secrets to the "Lost Arch" there so we don't want the untrusted domains to be able to read it. 13:26
dwalsh So if you create a directory for use by a service, you need to change the label to the correct context. 13:27
dwalsh A good command to look for context that a domain can write, is sesearch 13:27
dwalsh sesearch --allow -s httpd_t -c file -p write 13:27
dwalsh This should show you all the context that the httpd_t label can write 13:28
dwalsh That is one method to find the correct context. 13:28
dwalsh We also have man httpd_selinux and a few other man pages. 13:29
dwalsh Red Hat is developing a guide in Fedora for managing confined services. 13:29
dwalsh Which I can not find the link for now. But it is out there. 13:30
dwalsh One special context type you will see out there is "file_t" 13:30
dgrift_ 13:30
dwalsh THis type indicates the file has no label at all on it. 13:31
dwalsh It should have been called unlabeled_t if I could use a wayback machine I would go back to 2002 and change it. 13:31
dgrift_ proper url: 13:31
dwalsh If you see this label is usually means you have a very badly mislabeled machine, and you need to trigger a full relabel 13:31
dwalsh touch /.autorelabel; reboot 13:32
dwalsh Or Just run fixfiles restore. 13:32
dwalsh But you usually do not need to do this... 13:32
dwalsh One other time I see file_t showing up, might be related to usb sticks. 13:32
dwalsh Someone moving a file off of usb stick. Although that is only a theory of mine, I have not been able to prove it. 13:33
dwalsh File labeling also comes into play, when selecting the label to run a process as 13:34
dwalsh What I mean by this is SELinux defines "Transitions" 13:34
dwalsh When an SELinux system boots, it starts out running the kernel with no context, then the initrd loads the policy. 13:35
dwalsh The policy says to run the system as kernel_t 13:35
dwalsh It also defines a transition from kernel_t. 13:35
dwalsh When kernel_t runs an executable labeled init_exec_t it will transition to init_t 13:36
dwalsh So this is how upstart or init gets labeled init_t. 13:36
dwalsh Other transtions also happen 13:36
dwalsh init_t->initrc_exec_t->initrc_t 13:36
dwalsh So all init scripts are labeled initrc_exec_t by default. 13:37
dwalsh Or some varient, 13:37
dwalsh Then a transition rule is written that says 13:37
dwalsh initrc_t->httpd_exec_t->httpd_t 13:37
dwalsh And finally 13:37
dwalsh httpd_t->httpd_sys_script_exec_t->httpd_sys_script_t 13:37
dwalsh So if ANY labeling is wrong the transitions will fail and the app will end up running under the wrong context. 13:38
dwalsh So Always check you labeling. 13:38
dwalsh Slide 4 13:38
BounceCat If I want to see what domain a particular executable is labelled as, can you give a cli example how to see that? (apart from reading selinux errror message) 13:38
dwalsh Well you can check the label on disk 13:39
dwalsh ls -lZ /usr/sbin/httpd 13:39
dwalsh This will show you the current label 13:39
dwalsh You can ask the machine what the label should be 13:39
dwalsh matchpathcon /usr/sbin/httpd 13:39
dwalsh If they are different you can make it write 13:40
dwalsh restorecon /usr/sbin/httpd 13:40
BounceCat great thanks. 13:40
dwalsh But say you have another app that works just like httpd. Has the same security domain. 13:40
dwalsh /usr/sbin/myhttpd 13:40
dwalsh You would want this to be labeled like httpd 13:41
dwalsh semanage fcontext -a -t httpd_exec_t /usr/sbin/httpd 13:41
dwalsh Slide 4 shows you system-config-selinux, which allows you to set the file context 13:41
dwalsh Via a GUI, if you are so inclined. 13:41
dwalsh One interesting addition to F11 is the idea of equivalence. 13:42
dwalsh What I was looking for here was a way to say /srv/web should be labeled just like /var/www/ 13:42
dwalsh I want to copy all file context that match the directory /srv/web and substiture /var/www 13:43
dwalsh So with semanage I can execute 13:43
dwalsh semanage fcontext -a -e /var/www /srv/web 13:43
dwalsh Which is a pretty powerfull concept. 13:43
dwalsh Another example would be setting up an alternate location for home dirs. 13:44
dwalsh Ok I got to get moving... 13:44
dwalsh Slike 5 13:44
dwalsh Covers the semange equivalence and genhomedircon 13:45
dwalsh Lets move onto Slide 6 13:45
dwalsh SELinux needs to KNOW. 13:45
dwalsh The idea here is that confined apps can be configured to run in different ways. 13:45
dwalsh So you need to tell SELinux how you are going to run the apps. 13:46
dwalsh I am often asked why don't the writers of APPS write their own policy. 13:47
dwalsh I think it would be a good idea but it also could be a bad idea. 13:47
moixs (good question, who in Fedora actually does this?) 13:48
dwalsh As an example, if I asked the writer of vsftpd to define their policy 13:48
dwalsh They would say that ftp sometimes needs to read any file on disk, 13:48
dwalsh Sometimes needs to write any file on disk. 13:48
dwalsh So policy should allow ftp to read/write everything. 13:48
* maximus007 cringes 13:49
dwalsh Except that 99% of all users of ftp just use it to share a very small directori 13:49
dwalsh /var/ftp 13:49
dwalsh So A better solution would to have rules built into policy for an admin to tell SELInux how he uses frp 13:49
dwalsh ftp 13:49
dwalsh These are called booleans 13:50
dwalsh In the case of we have several booleans 13:50
dwalsh # semanage boolean -l | grep ftp 13:51
dwalsh ftp_home_dir -> off Allow ftp to read and write files in the user home directories 13:51
dwalsh allow_ftpd_full_access -> off Allow ftp servers to login to local users and read/write all files on the system, governed by DAC. 13:51
dwalsh allow_ftpd_use_nfs -> off Allow ftp servers to use nfs for public file transfer services. 13:51
dwalsh allow_ftpd_anon_write -> off Allow ftp servers to upload files, used for public file transfer services. Directories must be labeled public_content_rw_t. 13:51
dwalsh tftp_anon_write -> off Allow tftp to modify public files used for public file transfer services. 13:51
dwalsh ftpd_connect_db -> off Allow ftp servers to use connect to mysql database 13:51
dwalsh allow_ftpd_use_cifs -> off Allow ftp servers to use cifs for public file transfer services. 13:51
dwalsh httpd_enable_ftp_server -> off Allow httpd to act as a FTP server by listening on the ftp port. 13:51
dwalsh By default ftp is setup to only allow access to /var/ftp directory but you can alter these booleans to allow additional access. 13:51
dwalsh Fedora 11 has about 127 booleans now. 13:52
dwalsh Slide 7 and Slide 8 show you two different GUI tools used to manipulate booleans 13:53
dwalsh You can also use audit2allow -w (audit2why) to look at an AVC and see if a boolean would have allowed the access 13:53
dwalsh setroubleshoot uses this functionality to try to figure out if a boolean is ok 13:54
dwalsh The tool is actually attempting all possible booleans to see if any one would allow the access. 13:54
dwalsh In some cases multiple booleans are required to be turned on to allow access. 13:54
dwalsh For example allowing httpd scripts to be executed in a users home directory requires two or three booleans. 13:55
dwalsh Sadly audit2allow -w is not smart enough to figure this out. 13:55
dwalsh Again the management guide and other tools are helpful in figuring this out. 13:56
dwalsh Sline 9 13:56
dwalsh Slide 9 13:56
dwalsh Sadly SELINux Policy has bugs. 13:56
dwalsh Apps have bugs also 13:56
dwalsh Sometimes we release an app that causes new SELinux errors. 13:56
dwalsh We try not to do this but Fedora is an active community. 13:57
dwalsh I try not to add new confinement within a release. 13:57
dwalsh Once Fedora 11 is released, we tend to only loosen the policy. Add new allow rules, but not implement new confinement. 13:57
dwalsh And on RHEL releases we really frown on this. 13:58
dwalsh But What does an administrator do if SELinux or an APP are just broken. 13:58
dwalsh They labeling is correct. 13:58
dwalsh No Boolean allows the access? 13:58
dwalsh In this case the admin can fairly easily ad a custom policy module using audit2allow 13:58
dwalsh grep avc /var/log/audit/audit.log | audit2allow -M mypol 13:59
dwalsh This create a file called mypol.pp 13:59
dwalsh which can then be installed via semodule 13:59
dwalsh semodule -i mypol.pp 13:59
dwalsh mypol.te also gets created, 14:00
dwalsh You should really examine what this file is allowing, since it could be a security problem. 14:00
dwalsh For example allowing a confined app to write to etc_t is not a good idea since the app would be allowed to overwrite /etc/passwd 14:00
dwalsh But this method will allow you to customize policy to make the app just work. 14:01
dwalsh When you generate a policy using this method it creates a policy called mypol. if you execute the command later with different avcs but use the same name, you will override the first rules and replace them with new rules. 14:01
dwalsh So be carefull. 14:02
dwalsh Questions? 14:02
BounceCat "Slide 7 and Slide 8 show you two different GUI tools used to manipulate booleans" .. please name these tools 14:02
moixs Maybe a more general one 14:02
moixs How do you manage applications from external repositories, like RPM Fusion? Is the packager from RPM Fusion supposed to communicate with Fedora about SELinux policies? Does RPM Fusion maintain its own set of SELinux policies? 14:02
dwalsh BounceCat, system-config-selinux launces both. 14:03
BounceCat ok 14:03
dwalsh One is the default view, the second is created by going to the booleans screen and hitting lockdown. 14:03
BounceCat thx 14:03
dwalsh moixs, These apps may or maynot have policy defined for them. 14:04
dwalsh But they should all be able to run on a Fedora 11 box. 14:04
moixs So it's the packager's responsibility do do that? 14:04
dwalsh As I stated in an internal list, I think we can get the biggest piece of crap software to be able to run with SELinux in enforing mode. 14:05
moixs ok :p 14:05
dwalsh Of course this does not mean that the app will be secure... 14:05
dwalsh moixs, I will open a bugzilla or whatever with any app that is struggling with SELinux, or tell the bugreporter to do that 14:05
dwalsh Most problems with these software apps is badly built shared libraries. 14:06
dwalsh Which causes execmod violations. 14:06
dwalsh So we end up having to label them textrel_shlib_t w 14:06
moixs So you, in Fedora, will implement a "workaraound" even for an "external" application? 14:06
dwalsh which tells SELinux to now worry about a library being hacked, 14:06
dwalsh moixs, Yes, My goal is not to stop you from running you app. 14:07
moixs okay 14:07
dwalsh Otherwise you will dump us for some other unnamed distro. 14:07
dwalsh  :^) 14:07
moixs It's more that I package an daemonized application in RPM Fusion and never cared about SELinux :p 14:07
dwalsh So in RHEL and Fedora by default apps run as an unconfined domain, and should be able to run unconstrained by SELinux 14:07
dwalsh moixs, Fine, It should run. 14:08
dwalsh Well let me complete the talk since I got to go to a Baseball tryout in 15 minutes. Not mine  :^( 14:08
linuxguru oh. hehe 14:08
underscores lol 14:08
dwalsh When I originally wrote the blog, I talked about fixing SELinux problems. but ignored the most important thing about SELinux 14:08
dwalsh How do I tell if I my machine was cracked. 14:09
dwalsh Well Sadly we have not done a good job of that, and I hope to make strides on this in F12. 14:09
dwalsh setroubleshoot is a tool that shows you selinux reported a problem. And it tries to give you a remedy of a solution. 14:10
dwalsh We want to build some plugins that look for obvious breakins. 14:10
dwalsh For example, no confined application on an SELinux box is allowed to turn off SELinux. 14:10
dwalsh So it would be real nice to ring alarms if httpd_t tries to setenforce 14:11
dwalsh If I was breaking into an SELInux system, the first thing I would try is to turn it off. 14:11
maximus007 well if anything but uid 0 tries it i think we have a problem and maybe even then 14:11
dwalsh Similarly loading kernel modules, writeing to security directories/files 14:11
dwalsh /etc/shadow 14:12
dwalsh /etc/selinux/* 14:12
dwalsh Kernel directories ... 14:12
maximus007 my question is how to prevent even root from turning it off, will i have to directly modify the policy? 14:12
dwalsh So we need to change setroubleshoot to give a clearer indication that we believe the machine is compromised, and take action. 14:13
dwalsh But you can try to read the AVCs or look at the rules generated by audit2allow to figure out if something very strange is going on. 14:13
dwalsh That is the end of the formal presentation I can stay for about 5 more minutes. 14:14
dwalsh Any other questions? 14:14
dwalsh Or did I miss any? 14:14
dwalsh Was this useful? 14:14
dgrift_ yes 14:14
mamurdian it would be nice if you could provide more such lessons 14:14
linuxguru sure it was dwalsh , very useful. got to know about some of the new stuff in selinux planned for f11. 14:15
underscores yes :) 14:15
BounceCat It was great Dan, thanks. I have 1 more question, but others can go first. 14:15
dwalsh Ok Maybe next month I will cover svirt. 14:15
pluvo Thanks dwalsh! 14:15
maximus007 how do i keep root from turning off selinux? I can confine root but root can unconfine himself can he not? so i think i have to modify policy directly 14:15
moixs Yep, thank you :) 14:16
mamurdian maximus007> i think you can give root another role 14:16
dwalsh maximus007, Read my blog about confined users/administrators. 14:16
dwalsh THe simple answer is you define a confined admin 14:16
dwalsh Another talk. 14:16
maximus007 ok thanks for the class, well done 14:16
Bighusky we had to install a newer package (rsync) on rhel5. Rsync and Rsyncd were not able to function properly, so we ended up using audit2allow -M to write a custom policy for this package. Was that the right approach or should we have used other means? 14:16
linuxguru maximus007, i don't think that requires changing the policy directly. semanage is your friend. 14:16
dwalsh Bighusky, yes 14:16
dgrift_ Bighusky didyou read man rsync_selinux first? 14:17
dwalsh Bighusky, Although send me your changes And I will see about getting them upstream 14:17
dwalsh Bighusky, dgrift_ Is write that a boolean might have helped you 14:17
maximus007 linuxguru : yes but root is root and can generally run unconfined, so i'd have to remove the unconfined role i think but i will try it and see 14:17
DiscordianUK Thanks very much dwalsh 14:17
dwalsh rsync booleans in F11 14:18
dwalsh semanage boolean -l | grep rsync 14:18
dwalsh rsync_client -> off Allow rsync to run as a client 14:18
dwalsh rsync_export_all_ro -> off Allow rsync to export any files/directories read only. 14:18
dwalsh allow_rsync_anon_write -> off Allow rsync to modify public files used for public file transfer services. Files/Directories must be labeled public_content_rw_t. 14:18
dwalsh maximus007, You need to define an admin role and then don't give the admin the root password 14:18
dgrift_ BounceCat questions? 14:19
Bighusky dwalsh, dgrift, thanks very much, I have to admit that I didn't read man rsync_selinux. I will go back and give these suggestions a try. 14:19
BounceCat Subdirectories I've created in my home dir are either unconfined_u, system_u, or user_u, apparently at random. What has caused that and does it matter ? 14:19
dwalsh This confined admin would then log in as a confined user and use sudo to become root. 14:19
dwalsh By default all files are created and relabeled to system_u 14:19
dwalsh If you log onto a box as a particular user type 14:20
dwalsh user_u or confined_u 14:20
dwalsh user_u or uconfined_u 14:20
dwalsh Files created by you will end up using that user type 14:20
dwalsh user_u 14:20
maximus007 dwalsh, ok and thanks again excellent talk I have to get going myself but i will be sure to pester you about it in fedora-selinux :^) 14:20
dwalsh So user_u:user_r:user_t creates /home/dwalsh/foo 14:20
dwalsh will get created as user_u:object_r:user_home_t 14:21
dwalsh But the user componant of the file type is seldom used for denials. 14:21
dwalsh Bye. 14:21
dgrift_ thanks 14:21
BounceCat thank you 14:21
dgrift_ anyone have any questions feel free to join #fedora-selinux 14:22
linuxguru yes. over and out to #fedora-selinux 14:22
dgrift_ we might not be available right aways but we are glad if we can help 14:22
Bighusky than you very much and have a good sunday 14:23

Generated by 2.7 by Marius Gedminas - find it at!