How to debug Virtualization problems
Effective bug reporting
Reporting bugs effectively is an important skill for any Fedora user or developer.
Narrowing down the possible causes of the bug and providing the right information in the bug report allows a bug to be resolved quickly. Filing a bug report with little useful information can mean that your bug lays unresolved, possibly until it is closed automatically when the distribution version reaches "end of life".
Note: if you're filing a virtualization related bug against a package which isn't on this list, then please cc the email@example.com alias in bugzilla to ensure virt developers see the bug.
Once you've ensured you have the latest updates installed for the relevant packages, gather details of the version numbers of those packages e.g.
$> rpm -q kvm python-virtinst virt-viewer virt-manager
To find out what kernel version you are currently running, and what machine architecture you're using:
$> uname -a
Of course, you should also make sure to file the bug using the appropriate version of Fedora. Rawhide users should file bugs using the "rawhide" version.
Fedora's virtualization capabilities rely heavily on hardware capabilities, so when filing bugs please include copious information on your hardware platform including:
$> cat /proc/cpuinfo $> lspci -vvv
When filing a bug related to problems seen in the guest, include full details on the guest configuration including CPU architecture, RAM size, devices etc. This is most easily done by including the output of
virsh dumpxml MyGuest or, in the case of qemu, the full qemu command line.
Virt Manager stores a logfile in
Examine the log file and include any pieces that look like they might be useful in the bug report. If in doubt, attach the whole file to the bug.
You can also run virt-manager from the command line using
virt-manager --no-fork and check whether any relevant messages were printed there.
virt-install using the
--debug option to get detailed debug spew.
In order to gain access to a serial console during the install, you can use
-x "console=ttyS0". Using a serial console combined with a VNC install can be very useful for debugging e.g.
--nographics -x "console=ttyS0 vnc"
Any program using libvirt can be debugged using the
LIBVIRT_DEBUG=1 environment variable e.g.
$> LIBVIRT_DEBUG=1 virt-manager --no-fork $> LIBVIRT_DEBUG=1 virsh list --all
If your issue looks like it might be related to
libvirtd try running
libvirtd from the command line with debugging options enabled:
$> service libvirtd stop $> LIBVIRT_DEBUG=1 libvirtd --verbose
Starting from 0.6.0 libvirt also includes support for more fined grained logging, see the libvirt logging page for more details.
$> brctl show $> iptables -L -v -n $> ps -ef | grep dnsmasq $> ifconfig -a $> cat /proc/sys/net/ipv4/ip_forward $> service libvirtd reload
If you find that
/proc/sys/net/ipv4/ip_forward is not being set to
1 at boot time, try looking at the ordering of the libvirtd and NetworkManager services:
$> find /etc/rc.d -regex '.*rc.d/S.*\(libvirtd\|NetworkManager\)' $> rm -f /etc/chkconfig.d/libvirtd /etc/chkconfig.d/NetworkManager $> chkconfig libvirtd resetpriorities $> chkconfig NetworkManager resetpriorities $> find /etc/rc.d -regex '.*rc.d/S.*\(libvirtd\|NetworkManager\)'
See also the KVM wiki page on reporting bugs.
The output of any
qemu-kvm command run by
libvirtd is stored in
If a guest is crashing you can obtain a stack trace by doing the following:
- Set "on_crash=preserve" in your domain config
- Copy the guest kernel's System.map to the host
- Once the guest has crashed, run
/usr/lib/xen/bin/xenctx -s System.map <domid>
System Log Files
Always look in
/var/log/messages etc. for any useful information.
strace can often shed light on a bug - e.g. if you run
qemu-kvm under strace you can see what files they accessed, what commands they executed, what system calls they invoked etc.:
$> strace -ttt -f libvirtd
If the program in question is already running, you can attach to it using
gdb can often be useful to trace the execution of a program. However, in order to get useable information, you will need to install "debuginfo" packages. See the StackTraces page for more information.
If you see "AVC denied" or "setroubleshoot" messages in
/var/log/messages, your bug might be caused by an SELinux policy issue. Try temporarily putting SELinux into "permissive" mode with:
$> setenforce 0
If this makes your bug go away that doesn't mean your bug is fixed, it just narrows down the cause! You should include the AVC details from
ausearch -m AVC -ts recent in the bug report, or if the message includes a
sealert -l command then include the details printed by the command.
One common cause of SELinux problems is mis-labelled files. Try:
$> restorecon /path/to/file/in/selinux/message