The Xen virtual machine system is now available in Fedora Core 4 and above versions .More information on Xen itself can be found at http://www.cl.cam.ac.uk/Research/SRG/netos/xen/index.html and the Fedora Xen page.
Fedora is following the -unstable (future 3.0) Xen development line at the moment, which does occasionally lead to things being broken but also lets us track a lot of the more interesting work going on there. It also prevents the current NetBSD/xen, FreeBSD 5.3 and Solaris 11beta ports from running because they are all based on Xen 2.0.x.
Since setting up to run Xen isn't entirely straight-forward, here's a run-through of what should work for configuring a Fedora system as a Xen host, and setting up a single Xen guest.
- Your system must use GRUB, the default boot loader for Fedora [[FootNote(This is required because you actually boot the Xen hypervisor and it then starts the Linux kernel. It does this using the Multi
- 256 MB of RAM with the default setup [[FootNote(You can conceivably get by with less, but you'll have to reduce the dom0_memory line in
/boot/grub/grub.conf. The 130000 for dom0 is currently hard-coded by mkinitrd (this will get fixed before too long) and can go a little lower. The Xen hypervisor itself requires ~32 MB. Plus the memory required for your guest systems.)]
- Sufficient storage space for the guest operating systems. A minimal command-line Fedora system requires around 600MB of storage, a standard desktop Fedora system requires around 3GB.
- SELinux must not be enforcing it's policy, or some of the operations below will fail. Use
system-config-securitylevelto set SELinux to not enforce, and reboot to let this change take effect.
Commands which require root privileges are prefixed with the character '#'. To become root, issue the command 'su' as a normal user and supply the root password. Alternatively, use another system such as 'sudo' if you have it setup.
Installing the Xen Software
To install the main software, type the command:
This install the required packages.
xen is the xen OS kernel;
kernel-xen0 is the Xen-enabled host system kernel (
domain 0) and
kernel-xenU is the Xen guest system kernel.
Once this is done, you should have an entry set up in the file
/boot/grub/grub.conf for booting the
xen0 kernel. The
xen0 kernel is not set as the default boot option.
To set GRUB to boot with
xen0 by default, edit
/boot/grub/grub.conf and set the default to the xen0 kernel:
This is an example
/boot/grub/grub.conf, with the
init 3 option to start the host system on runlevel 3:
default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title Fedora Core (2.6.12-1.1369_FC4xen0) root (hd0,0) kernel /xen.gz module /vmlinuz-2.6.12-1.1369_FC4xen0 ro root=/dev/VolGroup01/LogVol00 init 3 module /initrd-2.6.12-1.1369_FC4xen0.img
Once you've rebooted with the
xen0 GRUB option, you should be running with a Xen-enabled kernel [[FootNote(To check that you are running a
xen0 kernel use '
uname -r' - the name of the kernel should contain xen0)] .
Check that the following directories exist on your system:
/var/run/xenstored/. If they have not automatically been created, you must create them manually (as a result of bug 167121 ):
Then, start the Xen service:
To check that Xen is running successfully, enter the following command:
You should see Domain-0 (the Xen-enabled host) running.
To enable the Xen service to automatically start on bootup, enter the following:
Creating A Guest System
These instructions set up a simple base Fedora installation as a Xen guest system.
Building a Fedora Guest System
First, create an image file to use as the storage for our Fedora guest system. This acts as a virtual hard drive for the guest system. This command creates an image file with a size of 1 GB.
dd if=/dev/zero of=fedora.img bs=1M count=1 seek=1024
Now, format the image file by creating an ext3 filesystem on it.
/sbin/mke2fs -F -j fedora.img
You should now be able to mount the image on a temporary mountpoint, say
Now for the ugly part. To install a guest system on to the image we need to set up some basic files on the filesystem of the image that have to be different for Xen (and for the system installation with
First, create the
/dev/ directory on the image with some required device nodes, since we're not using an
initrd[[FootNote(Using an initrd and thus udev seems to work as well with a Fedora Core development tree as of 2005-09-16. Just run depmod and mkinitrd from xen0 for the xenU kernel; make sure to install a xenU kernel into the guest system; perform a special depmod run as this is still missing from the kernel rpm.)] :
Second, create an
/etc/ directory in the image that contains an
/etc/fstab file. For the
/etc/fstab file something simple like the following should work:
/dev/sda1 / ext3 defaults 1 1 none /dev/pts devpts gid=5,mode=620 0 0 none /dev/shm tmpfs defaults 0 0 none /proc proc defaults 0 0 none /sys sysfs defaults 0 0
Third, make sure
/proc/ is mounted inside the image, because of a bug in the
groupadd utility (this is fixed with version 4.0.7-7 of the
Now we can install whatever base operating system we want into this image file.
If you want to use a local Fedora mirror, then you'll want to download this yum.conf and modify it appropriately before using it; otherwise, it will point to the default set of mirror Websites.
Then, decide which package group(s) you want to install. Start with Base (or for the space constrained, Core, but this is more difficult).
Install the guest system with the package groups by running
yum. For example, to install with Base use the command:
Now, go get some coffee and have a snack. It's going to take a little while :-)
NOTE: SELinux must be disabled to do this. SELinux does path based checks for whether or not things are allowed which do not apply when you're doing installs into a chroot environment.
Come back and if everything went okay, you'll have a minimal install
/mnt [[FootNote(current rpm seems to be broken, as pre scriptlets always fail, you may have to rerun yum inside the chroot)]
Because we are booting this Xen guest without an initrd, you should disable SELinux inside the guest, by editing
/mnt/etc/selinux/config and setting SELINUX to permissive or disabled. Since we disabled it for the host, you could just run this command:
Do any other configuration you want to on the filesystem and then unmount it:
Registering a Guest System as a Domain
Now, we just have to create a configuration file in
/etc/xen/. As a simple example, the following configuration file is
kernel ="/boot/vmlinuz-2.6.12-1.1454_FC4xenU" memory = 64 name = "rawhide" nics = 1 disk = ['file:/root/fedora.img,sda1,w'] root = "/dev/sda1" extra = "ro selinux=0 3"
/boot/vmlinuz-2.6.12-1.1454_FC4xenU with the xenU kernel you installed. Additionally, the device sda1 listed here as the disk is related to the
/etc/fstab you created above.
Before actually creating the guest domain you may want to decrease the amount of memory being allocated for your host system (domain0), in order to leave adequate room in RAM for your guest domains. This is especially important as the Xen system won't page your guest domains to your host domain's swap space. Constraining the RAM space used by your host domain can help assure that adequate RAM is available for your guests.
The way to do this is to run the following commands as root:
xm mem-max 0 384 xm mem-set 0 384
The mem-max command restricts the host domain (the 0 parameter signifies domain 0) from ever increasing its memory usage beyond 384 megabytes. The mem-set command forces the host domain to immediately shrink its memory allocation down to 384 megabytes.
It may take a bit of time for the domain 0 kernel to page out memory over the new limit, so you should give it a minute or so to complete.
When this is done, your system should have enough RAM free to allow you allocate one or more pretty good sized guest domains.
Now, create a new domain:
xm create -c rawhide
and off it goes. At the end, you should see the login prompt, at which point you can login as 'root' and begin playing around.
NOTE about missing shell errors: If you get errors about a missing shell, despite
/bin/bash (or whatever) being present, you need to disable SELinux for this new domain, as well as domain 0.
Note as well that, with the above /etc/xen/rawhide, our guest domain is running a Fedora Core 4 kernel built for Xen. As with normal Fedora Core kernel RPMs, a lot of functionality is kept in separate kernel modules on disk. Since we haven't installed those modules in the guest domain image, the kernel, once started, winds up being unable to find its modules.
You can correct this by copying the xenU kernel RPM into the guest domain's image by mounting it in the host domain before starting the guest. Then, once you've started the guest using xm create, you can run:
rpm -ivh kernel-xenU-2.6.12-1.1454_FC4.rpm
in order to add the needed kernel modules to the guest domain. These modules won't be used until you stop and re-start the guest domain. You can do that by connecting to the host domain with another window, and running the following:
xm shutdown rawhide xm create -c rawhide
This is pretty early and rough, but it's enough to starting playing with. The next step for development is getting to where you can do actual installations in a Xen guest environment and then being able to boot kernels which are on the guest's filesystem.
Automatically Loading Domains On Startup
If you want your domains to automatically start when rebooting the system make sure that you enable the
xendomains service, and place the Xen configuration file for the domains in the
To enable the
Note that the /etc/init.d/xendomains script, as of RPM xen-3.0-0.20050912.fc4, seems to be broken with respect to its use of the LSB init functions. This is arguably due to a bug in Fedora's LSB support, and has been filed in Fedora's bugzilla as bug 171052 and bug 171056 . The latter bug includes a modified version of the /etc/init.d/xendomains script as an attachment which you can use instead.
If you're running into issues with your Xen installation there are two log files on the host system to help you out. The file
/var/log/xend.log holds the same information as you receive when running '
xm log', and contains basic information. Unfortunately these log messages are often very short and contain little useful information. The following is the output of trying to create a domain running the kernel for NetBSD/xen.
[2005-06-27 02:23:02 xend] ERROR (SrvBase:163) op=create: Error creating domain:(0, 'Error') Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvBase.py", line 107, in _perform val = op_method(op, req) File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", line 71, in op_create raise XendError("Error creating domain: " + str(ex)) XendError: Error creating domain: (0, 'Error')
The second file,
/var/log/xend-debug.log usually contains much more detailed information. Trying to start the NetBSD/xen kernel will result in the following log output:
ERROR: Will only load images built for Xen v3.0 ERROR: Actually saw: 'GUEST_OS=netbsd,GUEST_VER=2.0,XEN_VER=2.0,LOADER=generic,BSD_SYMTAB' ERROR: Error constructing guest OS
If you are reporting errors or need help with your Xen setup you should always include both the
xend-debug.log messages that you receive.
Known Issue: X-Windows in domain 0
On some systems, starting X on the host system (domain 0) results in X getting stuck in vm86 land, probably due to some memory not being mapped into domain 0. In order to get X running on those systems, you can switch off vm86 probing; X will fall back to the other method of probing:
Known Issue: Domain 0 allocation is too small for kernel
This is a problem with some fedora kernels, and has been reported to the Red Hat Bugzilla at various times for various kernels. Some combination of kernel / xen version should work. At present, kernel-xen0-2.6.13-1.1526_FC4, kernel-xen0-2.6.12-1.1454_FC4 and 2.6.12-1.1456_FC4xen0 are amongst the problematic kernels.
2.6.12-1.1456_FC4xen0 has been known to work with the xen RPM available from Riel's homepage .
If the Troubleshooting section above does not help you to solve your problem, check the Red Hat Bugzilla for existing bug reports on Xen in FC4. The product is "Fedora Core", and the component is "Xen". These reports contain useful advice from fellow xen testers and often describe work-arounds.
Finally, general discussions on Fedora development, including Xen support, occur on the Fedora development mailing list .