From Fedora Project Wiki

Note that this information is applicable for Fedora Core 4. For information on Fedora Core 5, see FedoraXenQuickstartFC5

About Xen

The Xen virtual machine system is now available in Fedora Core 4 and above versions .More information on Xen itself can be found at http://www.cl.cam.ac.uk/Research/SRG/netos/xen/index.html and the Fedora Xen page.

Fedora is following the -unstable (future 3.0) Xen development line at the moment, which does occasionally lead to things being broken but also lets us track a lot of the more interesting work going on there. It also prevents the current NetBSD/xen, FreeBSD 5.3 and Solaris 11beta ports from running because they are all based on Xen 2.0.x.

Quick Start

Since setting up to run Xen isn't entirely straight-forward, here's a run-through of what should work for configuring a Fedora system as a Xen host, and setting up a single Xen guest.

System Requirements

  • Your system must use GRUB, the default boot loader for Fedora [[FootNote(This is required because you actually boot the Xen hypervisor and it then starts the Linux kernel. It does this using the MultiBoot standard.)]
  • 256 MB of RAM with the default setup [[FootNote(You can conceivably get by with less, but you'll have to reduce the dom0_memory line in /boot/grub/grub.conf. The 130000 for dom0 is currently hard-coded by mkinitrd (this will get fixed before too long) and can go a little lower. The Xen hypervisor itself requires ~32 MB. Plus the memory required for your guest systems.)]
  • Sufficient storage space for the guest operating systems. A minimal command-line Fedora system requires around 600MB of storage, a standard desktop Fedora system requires around 3GB.
  • SELinux must not be enforcing it's policy, or some of the operations below will fail. Use system-config-securitylevel to set SELinux to not enforce, and reboot to let this change take effect.

Conventions

Commands which require root privileges are prefixed with the character '#'. To become root, issue the command 'su' as a normal user and supply the root password. Alternatively, use another system such as 'sudo' if you have it setup.

Installing the Xen Software

To install the main software, type the command:


This install the required packages. xen is the xen OS kernel; kernel-xen0 is the Xen-enabled host system kernel (domain 0) and kernel-xenU is the Xen guest system kernel.

The latest kernel versions may suffer from a bug which prevents them from booting. See this section below for more information on picking a good kernel / xen combination.

Once this is done, you should have an entry set up in the file /boot/grub/grub.conf for booting the xen0 kernel. The xen0 kernel is not set as the default boot option.

To set GRUB to boot with xen0 by default, edit /boot/grub/grub.conf and set the default to the xen0 kernel:

default=0
If the graphical boot screen or X-Windows fails on the next bootup, you may need to disable the graphical boot (rhgb) and switch to using runlevel 3 (command-line only mode). See the section on X-Windows on Domain 0 for another workaround for a known issue with graphics support.

This is an example /boot/grub/grub.conf, with the init 3 option to start the host system on runlevel 3:

default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Fedora Core (2.6.12-1.1369_FC4xen0)
root (hd0,0)
kernel /xen.gz
module /vmlinuz-2.6.12-1.1369_FC4xen0 ro root=/dev/VolGroup01/LogVol00 init 3
module /initrd-2.6.12-1.1369_FC4xen0.img

Enabling Xen

Once you've rebooted with the xen0 GRUB option, you should be running with a Xen-enabled kernel [[FootNote(To check that you are running a xen0 kernel use 'uname -r' - the name of the kernel should contain xen0)] .

Check that the following directories exist on your system: /var/lib/xenstored/ and /var/run/xenstored/. If they have not automatically been created, you must create them manually (as a result of bug 167121 ):


Then, start the Xen service:


To check that Xen is running successfully, enter the following command:


You should see Domain-0 (the Xen-enabled host) running.

To enable the Xen service to automatically start on bootup, enter the following:


Creating A Guest System

These instructions set up a simple base Fedora installation as a Xen guest system.


Building a Fedora Guest System

This example system uses the kernel from the kernel-xenU package, as the kernel for your guest domain must currently be kept outside of the guest itself.

First, create an image file to use as the storage for our Fedora guest system. This acts as a virtual hard drive for the guest system. This command creates an image file with a size of 1 GB.

dd if=/dev/zero of=fedora.img bs=1M count=1 seek=1024

Now, format the image file by creating an ext3 filesystem on it.

/sbin/mke2fs -F -j fedora.img

You should now be able to mount the image on a temporary mountpoint, say /mnt


Now for the ugly part. To install a guest system on to the image we need to set up some basic files on the filesystem of the image that have to be different for Xen (and for the system installation with yum).

First, create the /dev/ directory on the image with some required device nodes, since we're not using an initrd[[FootNote(Using an initrd and thus udev seems to work as well with a Fedora Core development tree as of 2005-09-16. Just run depmod and mkinitrd from xen0 for the xenU kernel; make sure to install a xenU kernel into the guest system; perform a special depmod run as this is still missing from the kernel rpm.)]  :


Second, create an /etc/ directory in the image that contains an /etc/fstab file. For the /etc/fstab file something simple like the following should work:

/dev/sda1               /                       ext3    defaults 1 1
none                    /dev/pts                devpts  gid=5,mode=620 0 0
none                    /dev/shm                tmpfs   defaults 0 0
none                    /proc                   proc    defaults 0 0
none                    /sys                    sysfs   defaults 0 0

Third, make sure /proc/ is mounted inside the image, because of a bug in the groupadd utility (this is fixed with version 4.0.7-7 of the shadow-utils package):


Now we can install whatever base operating system we want into this image file.

If you want to use a local Fedora mirror, then you'll want to download this yum.conf and modify it appropriately before using it; otherwise, it will point to the default set of mirror Websites.

Then, decide which package group(s) you want to install. Start with Base (or for the space constrained, Core, but this is more difficult).

Install the guest system with the package groups by running yum. For example, to install with Base use the command:


Now, go get some coffee and have a snack. It's going to take a little while :-)

NOTE: SELinux must be disabled to do this. SELinux does path based checks for whether or not things are allowed which do not apply when you're doing installs into a chroot environment.

Come back and if everything went okay, you'll have a minimal install in /mnt [[FootNote(current rpm seems to be broken, as pre scriptlets always fail, you may have to rerun yum inside the chroot)]

Because we are booting this Xen guest without an initrd, you should disable SELinux inside the guest, by editing /mnt/etc/selinux/config and setting SELINUX to permissive or disabled. Since we disabled it for the host, you could just run this command:


Do any other configuration you want to on the filesystem and then unmount it:


Registering a Guest System as a Domain

Now, we just have to create a configuration file in /etc/xen/. As a simple example, the following configuration file is /etc/xen/rawhide:

kernel ="/boot/vmlinuz-2.6.12-1.1454_FC4xenU"
memory = 64
name = "rawhide"
nics = 1
disk = ['file:/root/fedora.img,sda1,w'] 
root = "/dev/sda1"
extra = "ro selinux=0 3"

Substitute /boot/vmlinuz-2.6.12-1.1454_FC4xenU with the xenU kernel you installed. Additionally, the device sda1 listed here as the disk is related to the /etc/fstab you created above.

Before actually creating the guest domain you may want to decrease the amount of memory being allocated for your host system (domain0), in order to leave adequate room in RAM for your guest domains. This is especially important as the Xen system won't page your guest domains to your host domain's swap space. Constraining the RAM space used by your host domain can help assure that adequate RAM is available for your guests.

The way to do this is to run the following commands as root:

xm mem-max 0 384
xm mem-set 0 384

The mem-max command restricts the host domain (the 0 parameter signifies domain 0) from ever increasing its memory usage beyond 384 megabytes. The mem-set command forces the host domain to immediately shrink its memory allocation down to 384 megabytes.

It may take a bit of time for the domain 0 kernel to page out memory over the new limit, so you should give it a minute or so to complete.

When this is done, your system should have enough RAM free to allow you allocate one or more pretty good sized guest domains.

Now, create a new domain:

xm create -c rawhide

and off it goes. At the end, you should see the login prompt, at which point you can login as 'root' and begin playing around.

If you don't have enough free RAM available to create your guest domain, you will get a message similar to Error: Error creating domain: (2, 'No such file or directory'). This means you need to be sure to use the xm mem-max and xm mem-set commands, above.


NOTE about missing shell errors: If you get errors about a missing shell, despite /etc/passwd and /bin/bash (or whatever) being present, you need to disable SELinux for this new domain, as well as domain 0.

Note as well that, with the above /etc/xen/rawhide, our guest domain is running a Fedora Core 4 kernel built for Xen. As with normal Fedora Core kernel RPMs, a lot of functionality is kept in separate kernel modules on disk. Since we haven't installed those modules in the guest domain image, the kernel, once started, winds up being unable to find its modules.

You can correct this by copying the xenU kernel RPM into the guest domain's image by mounting it in the host domain before starting the guest. Then, once you've started the guest using xm create, you can run:

rpm -ivh kernel-xenU-2.6.12-1.1454_FC4.rpm

in order to add the needed kernel modules to the guest domain. These modules won't be used until you stop and re-start the guest domain. You can do that by connecting to the host domain with another window, and running the following:

xm shutdown rawhide
xm create -c rawhide

This is pretty early and rough, but it's enough to starting playing with. The next step for development is getting to where you can do actual installations in a Xen guest environment and then being able to boot kernels which are on the guest's filesystem.

Automatically Loading Domains On Startup

If you want your domains to automatically start when rebooting the system make sure that you enable the xendomains service, and place the Xen configuration file for the domains in the /etc/xen/auto/ directory.

To enable the xendomains service:


Note that the /etc/init.d/xendomains script, as of RPM xen-3.0-0.20050912.fc4, seems to be broken with respect to its use of the LSB init functions. This is arguably due to a bug in Fedora's LSB support, and has been filed in Fedora's bugzilla as bug 171052 and bug 171056 . The latter bug includes a modified version of the /etc/init.d/xendomains script as an attachment which you can use instead.

Troubleshooting

If you're running into issues with your Xen installation there are two log files on the host system to help you out. The file /var/log/xend.log holds the same information as you receive when running 'xm log', and contains basic information. Unfortunately these log messages are often very short and contain little useful information. The following is the output of trying to create a domain running the kernel for NetBSD/xen.

[2005-06-27 02:23:02 xend]  ERROR (SrvBase:163) op=create: Error creating domain:(0, 'Error')
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvBase.py", line 107, in _perform
val = op_method(op, req)
File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", line 71, in op_create
raise XendError("Error creating domain: " + str(ex))
XendError: Error creating domain: (0, 'Error')

The second file, /var/log/xend-debug.log usually contains much more detailed information. Trying to start the NetBSD/xen kernel will result in the following log output:

ERROR: Will only load images built for Xen v3.0
ERROR: Actually saw: 'GUEST_OS=netbsd,GUEST_VER=2.0,XEN_VER=2.0,LOADER=generic,BSD_SYMTAB'
ERROR: Error constructing guest OS

If you are reporting errors or need help with your Xen setup you should always include both the xend.log and xend-debug.log messages that you receive.

Known Issue: X-Windows in domain 0

On some systems, starting X on the host system (domain 0) results in X getting stuck in vm86 land, probably due to some memory not being mapped into domain 0. In order to get X running on those systems, you can switch off vm86 probing; X will fall back to the other method of probing:


Known Issue: Domain 0 allocation is too small for kernel

This is a problem with some fedora kernels, and has been reported to the Red Hat Bugzilla at various times for various kernels. Some combination of kernel / xen version should work. At present, kernel-xen0-2.6.13-1.1526_FC4, kernel-xen0-2.6.12-1.1454_FC4 and 2.6.12-1.1456_FC4xen0 are amongst the problematic kernels.

2.6.12-1.1456_FC4xen0 has been known to work with the xen RPM available from Riel's homepage .

Getting Help

If the Troubleshooting section above does not help you to solve your problem, check the Red Hat Bugzilla for existing bug reports on Xen in FC4. The product is "Fedora Core", and the component is "Xen". These reports contain useful advice from fellow xen testers and often describe work-arounds.

For general Xen issues and useful information check the Xen project documentation , and mailing list archives .

Finally, general discussions on Fedora development, including Xen support, occur on the Fedora development mailing list .

Footnotes