From Fedora Project Wiki
(Created page with "= Feature Name <!-- The name of your feature --> = Create a tool sandboxcontainer that will allow an administrator to easily create a LXC Container to run the an application serv...")
 
 
(30 intermediate revisions by 4 users not shown)
Line 1: Line 1:
= Feature Name <!-- The name of your feature --> =
= Secure Containers <!-- The name of your feature --> =
Create a tool sandboxcontainer that will allow an administrator to easily create a LXC Container to run the an application server, with libvirt and SELinux locking it down.
Create a tool virt-sandbox-server that will allow an administrator to easily create a LXC Container to run the an application server, with libvirt and SELinux locking it down.  This tool is part of the virt-sandbox package introduced in Fedora 17.
== Summary ==
== Summary ==
<!-- A sentence or two summarizing what this feature is and what it will do.  This information is used for the overall feature summary page for each release. -->
<!-- A sentence or two summarizing what this feature is and what it will do.  This information is used for the overall feature summary page for each release. -->
This change allows an administrator to run multiple services on the same machine each service in a secure Linux Container.  sandboxcontainer takes an executable and sets up all of the mountpoints and libvirt container information to run the application within the container.
This change allows an administrator to run multiple services on the same machine each service in a secure Linux Container.  virt-sandbox-service takes an systemd unit file and sets up all of the mountpoints and libvirt container information to run the service unit within the container.
Then libvirt will be able to launch the container with a SELinux context that will prevent the containers process from interacting with other process on the system including other containers.  This could allow an administrator to run multiple web servers simultaneously each with their own data and sharing the system data, even running as root, but prevent them from breaking the host machine or other processes on the system.  
Then libvirt will be able to launch the container with a SELinux context that will prevent the containers process from interacting with other process on the system including other containers.  This could allow an administrator to run multiple web servers simultaneously each with their own data and sharing the system data, even running as root, but prevent them from breaking the host machine or other processes on the system.  
== Owner ==
== Owner ==
<!--This should link to your home wiki page so we know who you are-->
<!--This should link to your home wiki page so we know who you are-->
* Name: [[User:dwalsh| Daniel J Walsh]]
* Name: [[User:dwalsh| Daniel J Walsh]]
* Name: [[User:berrange| Daniel Berrange ]]


<!-- Include you email address that you can be reached should people want to contact you about helping with your feature, status is requested, or  technical issues need to be resolved-->
<!-- Include you email address that you can be reached should people want to contact you about helping with your feature, status is requested, or  technical issues need to be resolved-->
* Email: dwalsh@redhat.com
* Email: dwalsh@redhat.com
* Email: berrange@redhat.com


== Current status ==
== Current status ==
* Targeted release: [Fedora 17]  
* Targeted release: [Fedora 18]  
* Last updated: Mon Nov 7 2011
* Last updated: Mon Oct 15 2012
* Percentage of completion: 20%
* Percentage of completion: 100%
 
** libvirt-0.10.2.1-1.fc18.x86_64 changes in Rawhide
** selinux-policy-3.11.1-47.fc18.noarch changes in Rawhide
** libvirt-sandbox-0.1.0-1.fc18 currently in Rawhide update required.
*** Continuing to be developed and tested.
** kernel fixes for connecting applications to a container still being developed.
*** Continuing to be developed and tested.
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->


Line 28: Line 35:
== Benefit to Fedora ==
== Benefit to Fedora ==
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
Linux Containers is a low level way of doing virtualization, and allows you to run multiple copies of the same service at the same time on a system.  It has some advantages over full virtualization as it does not have to wait for an entire system to boot, it can use less memory and can use the base OS, in a read/only manner.
Linux Containers is a low level way of doing virtualization, and allows you to run multiple copies of the same service at the same time on a system.  It has some advantages over full virtualization as it does not have to wait for an entire system to boot, it can use less memory and can use the base OS, in a read-only manner.


== Scope ==
== Scope ==
Line 37: Line 44:
* Kernel and SELinux toolchain changes  
* Kernel and SELinux toolchain changes  
** One problem we have with the way the kernel currently labels files, is that MCS portion of the label is applied based off the process level not the level of the directory that the file was create in.  This means when a process running as unconfined_t:s0 creates a file in a directory labeled svirt_lxc_file_t:s0:c1,c2, the file will be created as svirt_lxc_file_t:s0 and the container will not be able to use the file, until the administrator changed the label.
** One problem we have with the way the kernel currently labels files, is that MCS portion of the label is applied based off the process level not the level of the directory that the file was create in.  This means when a process running as unconfined_t:s0 creates a file in a directory labeled svirt_lxc_file_t:s0:c1,c2, the file will be created as svirt_lxc_file_t:s0 and the container will not be able to use the file, until the administrator changed the label.
This feature requires that the kernel support new policy construct to allow the policy writer to specify whether the MCS Label is inherited from the parent directory MCS Labelor from the creating process MCS Label.
This feature requires that the kernel support new policy construct to allow the policy writer to specify whether the MCS Label is inherited from the parent directory MCS Label or from the creating process MCS Label.
* libvirt  
kernel-3.3 should have this feature.  (Completed in Current Kernel
** Has to be able to launch the container with a new SELinux context for containers,
* libvirt-sandbox package introduced in Fedora 17
** potentially libvirt will have to mount a container file image with a random MCS label
** virt-sandbox-service script used for creating/starting/stopping Secure Containers
* sandboxcontainer
*** Create
** New tool needs to be created which will look at the content related to an executable, by examining the rpm database, then optionally create an image file or a directory on disk and populate the image/directory with the default content to be used for the container. It will also create the XML database used by libvirt and actually create the virtual machine definition. Finally it will create a default "init" which libvirt will execute when it starts the linux container.
**** Create an image file or a directory on disk and populate the image/directory with the default content based on the rpm data associated with the systemd unit file.
**** It will also create the virt-sandbox database used by libvirt-sandbox to create a libvirt-lxc container.
**** Setup systemd content within the container so that systemd will start the correct services
**** Also create a systemd unit file so the administrator of the host machine can start and stop the service containers just like any other servers.
*** Start
**** Convert the Linux-sandbox configuration into libvirt-lxc configuration and tell libvirt to start container.
**** Container will execute systemd which will then execut
***** dhcpd to connect the network
***** systemd-journal to handle syslog messages
***** Unit file configured to start within the container.
**** Container will container a console that the host administrator can connect to and run a shell to debug what is going on within the container.
*** Execute
**** Allow host administrator to execute commands within the container.
*** Stop
**** Stop the container, and remove it from libvirt
*** Delete
**** Delete all content related to the container
*** Connect
**** Connect to the console and run a administrative shell with container


== How To Test ==
== How To Test ==
<!-- This does not need to be a full-fledged documentDescribe the dimensions of tests that this feature is expected to pass when it is doneIf it needs to be tested with different hardware or software configurations, indicate them. The more specific you can be, the better the community testing can be.  
Services Tested with:
 
* sshd
* httpd
* mysql
* dovecot
 
Use the virt-sandbox-service command to create a container. 
 
virt-sandbox-service create -C -l s0:c1,c2 -u httpd.service container1
Created sandbox container dir /var/lib/libvirt/filesystems/container1
Created sandbox config /etc/libvirt-sandbox/services/container1.sandbox
Created unit file /etc/systemd/system/httpd@container1.service
 
Manipulate the data within the container while running outside of the container.
 
cd /var/lib/libvirt/filesystems/container1/var/log
touch content
ls -lZ content
# Make sure the content gets created with the correct MCS label.
# Content should be labeled with s0:c1,c2 : Not s0
 
Now create a file with a bad label for the container.
cat "Secret" > badcontent
chcon -l s0:c3,c4 badcontent
 
 
Start the container
 
virt-sandbox-service start container1
 
In another window
 
Make sure the processes are running with the proper SELinux labelps -eZ | grep svirt_lxc
You should see processes like systemd, systemd-journal, dhclient and httpd running within the container with the MCS label of s0:c1,c2
 
Connect to the container
 
virt-sandbox-service connect container1
id
getenforce  # Should tell you SELinux is disabled.
  setenforce 1 # Should be denied
touch /file # Should deny you creating this file
touch /var/www/html/content  # Should be allowed
cat /var/www/html/badcontent # Should be denied
Configure the apache server any way you would like, and manipulate html pages
ifconfig eth0  # Grap IP Address for use on next test
# Use the shell running with in the container to attempt to break out of the container.
^]
 
Connect to container using Firefox
 
firefox $IP # Using IP address from container, make sure you see the content.
 
 
Shut down the container


Remember that you are writing this how to for interested testers to use to check out your feature - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your feature.
virt-sandbox-service stop container1
>


Use the sandboxcontainer command to create a container.  Manipulate the data within the container while running outside of the container. Make sure the content gets created with the correct MCS label.  Now run the container using libvirt tools virsh or virt-manager.  Make sure the processes are running with the proper SELinux labelUse the shell running with in the container to attempt to break out of the container. Use chcon to change the label of some of the data and make sure the container can no longer access the data.
 
Now lets try to do the same but starting and stoping the container using systemctl commands
 
systemctl start httpd@container1.service
  systemctl enable httpd@container1.service # Check on reboot if the container is running
 
Make sure the container is running.
 
virt-sandbox-service connect container1
ps -eZ
^]
 
Connect to container again
 
virt-sandbox-service connect container1
ps -eZ
^]
 
 
Stop container
 
systemctl stop httpd@container1.service
 
Delete the container
 
virt-sandbox-service delete container1
  # Make sure it is deleted
ls /var/lib/libvirt/filesystems/container1
ls /etc/libvirt-sandbox/services/container1.sandbox
  ls /etc/systemd/system/httpd@container1.service
 
Now lets setup hundreds of containers
cat /usr/bin/containers
#!/bin/bash
create() {
    virt-sandbox-service create -C -l s0:c$2 -u httpd.service $1
}
delete() {
    virt-sandbox-service delete $1
}
start() {
    systemctl start httpd@$1.service
}
stop() {
    systemctl stop httpd@$1.service
}
command=$1
  name=$2
repeat=$3
for i in $(seq 1 $repeat)
do
    eval $command $name$i $i
done
 
Now lets use this script to test scalability:
 
containers create containers 100
containers start containers 100
 
Make sure containers start.
 
  ps -eZ | grep systemd-journald | wc -l
 
Should be close to 101.
 
Make sure you can connect to a couple of container.
Run Top to check CPU usage and see if any processes in container are using substantial CPU.
Check for memory.
 
 
containers stop containers 100
containers delete containers 100


== User Experience ==
== User Experience ==
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
Users should be able to easily establish containers and be able to run multiple containers simultaneously.
Users should be able to easily establish containers and be able to run multiple containers simultaneously.
See Test Information for use cases.


== Documentation ==
== Documentation ==
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
The virt-sandbox-service command line tool comes with a manpage:
  # man virt-sandbox-service


This provides complete command line help and examples of usage


== Release Notes ==
== Release Notes ==
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
 
* The 'libvirt-sandbox' RPM introduces a new command 'virt-sandbox-service' for configurtin/running system service in a confined environment.
* The confined environment may have a custom view of the filesystem and custom networking configuration.


== Comments and Discussion ==
== Comments and Discussion ==
Line 69: Line 226:




[[Category:FeaturePageIncomplete]]
[[Category:FeatureAcceptedF18]]
<!-- When your feature page is completed and ready for review -->
<!-- When your feature page is completed and ready for review -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForFesco -->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Latest revision as of 10:39, 31 October 2012

Secure Containers

Create a tool virt-sandbox-server that will allow an administrator to easily create a LXC Container to run the an application server, with libvirt and SELinux locking it down. This tool is part of the virt-sandbox package introduced in Fedora 17.

Summary

This change allows an administrator to run multiple services on the same machine each service in a secure Linux Container. virt-sandbox-service takes an systemd unit file and sets up all of the mountpoints and libvirt container information to run the service unit within the container. Then libvirt will be able to launch the container with a SELinux context that will prevent the containers process from interacting with other process on the system including other containers. This could allow an administrator to run multiple web servers simultaneously each with their own data and sharing the system data, even running as root, but prevent them from breaking the host machine or other processes on the system.

Owner

  • Email: dwalsh@redhat.com
  • Email: berrange@redhat.com

Current status

  • Targeted release: [Fedora 18]
  • Last updated: Mon Oct 15 2012
  • Percentage of completion: 100%
    • libvirt-0.10.2.1-1.fc18.x86_64 changes in Rawhide
    • selinux-policy-3.11.1-47.fc18.noarch changes in Rawhide
    • libvirt-sandbox-0.1.0-1.fc18 currently in Rawhide update required.
      • Continuing to be developed and tested.
    • kernel fixes for connecting applications to a container still being developed.
      • Continuing to be developed and tested.

Detailed Description

libvirt as of Fedora 16 has the ability to run processes within a container, the problem with this is the processes can run as root and can easily break out of the container. Also it is fairly difficult to setup one of these containers to run a particular service, for example apache.

We want to make the processes of setting up these containers easier and to use SELinux and containers together to lock down the processes running within the container, meaning you could run multiple customers on a host and prevent them from attacking each other or the host.

Benefit to Fedora

Linux Containers is a low level way of doing virtualization, and allows you to run multiple copies of the same service at the same time on a system. It has some advantages over full virtualization as it does not have to wait for an entire system to boot, it can use less memory and can use the base OS, in a read-only manner.

Scope

This change effects

  • SELinux Policy writers
    • New policy has to be written to define what a confined application container is allowed to do
  • Kernel and SELinux toolchain changes
    • One problem we have with the way the kernel currently labels files, is that MCS portion of the label is applied based off the process level not the level of the directory that the file was create in. This means when a process running as unconfined_t:s0 creates a file in a directory labeled svirt_lxc_file_t:s0:c1,c2, the file will be created as svirt_lxc_file_t:s0 and the container will not be able to use the file, until the administrator changed the label.

This feature requires that the kernel support new policy construct to allow the policy writer to specify whether the MCS Label is inherited from the parent directory MCS Label or from the creating process MCS Label. kernel-3.3 should have this feature. (Completed in Current Kernel

  • libvirt-sandbox package introduced in Fedora 17
    • virt-sandbox-service script used for creating/starting/stopping Secure Containers
      • Create
        • Create an image file or a directory on disk and populate the image/directory with the default content based on the rpm data associated with the systemd unit file.
        • It will also create the virt-sandbox database used by libvirt-sandbox to create a libvirt-lxc container.
        • Setup systemd content within the container so that systemd will start the correct services
        • Also create a systemd unit file so the administrator of the host machine can start and stop the service containers just like any other servers.
      • Start
        • Convert the Linux-sandbox configuration into libvirt-lxc configuration and tell libvirt to start container.
        • Container will execute systemd which will then execut
          • dhcpd to connect the network
          • systemd-journal to handle syslog messages
          • Unit file configured to start within the container.
        • Container will container a console that the host administrator can connect to and run a shell to debug what is going on within the container.
      • Execute
        • Allow host administrator to execute commands within the container.
      • Stop
        • Stop the container, and remove it from libvirt
      • Delete
        • Delete all content related to the container
      • Connect
        • Connect to the console and run a administrative shell with container

How To Test

Services Tested with:

  • sshd
  • httpd
  • mysql
  • dovecot

Use the virt-sandbox-service command to create a container.

virt-sandbox-service create -C -l s0:c1,c2 -u httpd.service container1
Created sandbox container dir /var/lib/libvirt/filesystems/container1
Created sandbox config /etc/libvirt-sandbox/services/container1.sandbox
Created unit file /etc/systemd/system/httpd@container1.service

Manipulate the data within the container while running outside of the container.

cd /var/lib/libvirt/filesystems/container1/var/log
touch content
ls -lZ content
# Make sure the content gets created with the correct MCS label.
# Content should be labeled with s0:c1,c2 : Not s0
Now create a file with a bad label for the container.
cat "Secret" > badcontent
chcon -l s0:c3,c4 badcontent
 
 

Start the container

virt-sandbox-service start container1

In another window

Make sure the processes are running with the proper SELinux label. ps -eZ | grep svirt_lxc You should see processes like systemd, systemd-journal, dhclient and httpd running within the container with the MCS label of s0:c1,c2

Connect to the container

virt-sandbox-service connect container1
id 
getenforce   # Should tell you SELinux is disabled.
setenforce 1 # Should be denied
touch /file  # Should deny you creating this file
touch /var/www/html/content  # Should be allowed
cat /var/www/html/badcontent # Should be denied
Configure the apache server any way you would like, and manipulate html pages
ifconfig eth0  # Grap IP Address for use on next test
# Use the shell running with in the container to attempt to break out of the container. 
^] 

Connect to container using Firefox

firefox $IP # Using IP address from container, make sure you see the content.


Shut down the container

virt-sandbox-service stop container1


Now lets try to do the same but starting and stoping the container using systemctl commands

systemctl start httpd@container1.service
systemctl enable httpd@container1.service # Check on reboot if the container is running

Make sure the container is running.

virt-sandbox-service connect container1
ps -eZ
^]

Connect to container again

virt-sandbox-service connect container1
ps -eZ
^]


Stop container

systemctl stop httpd@container1.service

Delete the container

virt-sandbox-service delete container1
# Make sure it is deleted
ls /var/lib/libvirt/filesystems/container1
ls /etc/libvirt-sandbox/services/container1.sandbox
ls /etc/systemd/system/httpd@container1.service

Now lets setup hundreds of containers

cat /usr/bin/containers
#!/bin/bash
create() {
   virt-sandbox-service create -C -l s0:c$2 -u httpd.service $1
}
delete() {
    virt-sandbox-service delete $1
}
start() {
    systemctl start httpd@$1.service
}
stop() {
    systemctl stop httpd@$1.service
}
command=$1
name=$2
repeat=$3
for i in $(seq 1 $repeat)
do
    eval $command $name$i $i
done

Now lets use this script to test scalability:

containers create containers 100
containers start containers 100

Make sure containers start.

ps -eZ | grep systemd-journald | wc -l

Should be close to 101.

Make sure you can connect to a couple of container. Run Top to check CPU usage and see if any processes in container are using substantial CPU. Check for memory.


containers stop containers 100
containers delete containers 100

User Experience

Users should be able to easily establish containers and be able to run multiple containers simultaneously.

See Test Information for use cases.

Documentation

The virt-sandbox-service command line tool comes with a manpage:

 # man virt-sandbox-service

This provides complete command line help and examples of usage

Release Notes

  • The 'libvirt-sandbox' RPM introduces a new command 'virt-sandbox-service' for configurtin/running system service in a confined environment.
  • The confined environment may have a custom view of the filesystem and custom networking configuration.

Comments and Discussion