How to debug Virtualization problems

From FedoraProject

Revision as of 20:16, 25 June 2009 by Fcami (Talk | contribs)

Jump to: navigation, search

Effective bug reporting

Reporting bugs effectively is an important skill for any Fedora user or developer.

Narrowing down the possible causes of the bug and providing the right information in the bug report allows a bug to be resolved quickly. Filing a bug report with little useful information can mean that your bug lays unresolved, possibly until it is closed automatically when the distribution version reaches "end of life".

See BugsAndFeatureRequests and "how to file a bug report" for generic information on filing bugs. This page contains information specific to virtualization bugs.

Note: if you're filing a virtualization related bug against a package which isn't on this list, then please cc the fedora-virt-maint@redhat.com alias in bugzilla to ensure virt developers see the bug.

Contents

Version Information

Once you've ensured you have the latest updates installed for the relevant packages, gather details of the version numbers of those packages e.g.

$> rpm -q kvm python-virtinst virt-viewer virt-manager

To find out what kernel version you are currently running, and what machine architecture you're using:

$> uname -a

Of course, you should also make sure to file the bug using the appropriate version of Fedora. Rawhide users should file bugs using the "rawhide" version.

Hardware Information

Fedora's virtualization capabilities rely heavily on hardware capabilities, so when filing bugs please include copious information on your hardware platform including:

$> cat /proc/cpuinfo
$> lspci -vvv

You can also check what virtualization capabilities are available on your machine by running:

$> virsh capabilities

Guest Configuration

When filing a bug related to problems seen in the guest, include full details on the guest configuration including CPU architecture, RAM size, devices etc. This is most easily done by including the output of virsh dumpxml MyGuest or, in the case of qemu, the full qemu command line.

Virt Manager

Virt Manager stores a logfile in ~/.virt-manager/virt-manager.log.

Examine the log file and include any pieces that look like they might be useful in the bug report. If in doubt, attach the whole file to the bug.

You can also run virt-manager from the command line using virt-manager --no-fork and check whether any relevant messages were printed there.

virt-install

virt-install stores a log file in ~/.virtinst/virt-install.log.

Run virt-install using the --debug option to get detailed debug spew.

In order to gain access to a serial console during the install, you can use -x "console=ttyS0". Using a serial console combined with a VNC install can be very useful for debugging e.g. --nographics -x "console=ttyS0 vnc"

libvirt

Any program using libvirt can be debugged using the LIBVIRT_DEBUG=1 environment variable e.g.

$> LIBVIRT_DEBUG=1 virt-manager --no-fork
$> LIBVIRT_DEBUG=1 virsh list --all

If your issue looks like it might be related to libvirtd try looking in /var/log/messages for any error messages.

You can also use /etc/libvirt/libvirtd.conf logging configuration to e.g. log debug spew to a file:

log_level = 1
log_outputs = 0:file:/tmp/libvirtd.log

Alternatively, you could try running libvirtd from the command line with debugging options enabled:

$> service libvirtd stop
$> LIBVIRT_DEBUG=1 libvirtd --verbose

Networking

If you are having trouble with guests connected to a libvirt virtual network, shared physical interface or bridge, try these commands:

$> virsh net-list --all
$> brctl show
$> iptables -L -v -n
$> ps -ef | grep dnsmasq
$> ifconfig -a
$> cat /proc/sys/net/ipv4/ip_forward
$> service libvirtd reload

If you find that /proc/sys/net/ipv4/ip_forward is not being set to 1 at boot time, try looking at the ordering of the libvirtd and NetworkManager services:

$> find /etc/rc.d -regex '.*rc[35].d/S.*\(libvirtd\|NetworkManager\)'
$> rm -f /etc/chkconfig.d/libvirtd /etc/chkconfig.d/NetworkManager
$> chkconfig libvirtd resetpriorities
$> chkconfig NetworkManager resetpriorities
$> find /etc/rc.d -regex '.*rc[35].d/S.*\(libvirtd\|NetworkManager\)'

kvm

See also the KVM wiki page on reporting bugs.

The output of any qemu-kvm command run by libvirtd is stored in /var/log/libvirt/qemu/GuestName.log.

kvm-autotest is an excellent way of testing basic KVM functionality.

Is My Guest Using KVM?

Often people are unsure whether their qemu guest is actually using hardware virtualization via KVM.

Firstly, check that KVM is available:

 $> virsh capabilities  | grep kvm
     <domain type='kvm'>
       <emulator>/usr/bin/qemu-kvm</emulator>

and that the guest is configured to use KVM:

  $> virsh dumpxml ${guest} | grep kvm
  <domain type='kvm' id='18'>
      <emulator>/usr/bin/qemu-kvm</emulator>

Next, look in /var/log/libvirt/qemu/${guest}.log to check that /usr/bin/qemu-kvm is the emulator that was executed by libvirt and that there are no error messages about /dev/kvm.

If you are running a recent Fedora guest, you could try:

  $> grep CONFIG_KVM_GUEST /boot/config-$(uname -r)
  CONFIG_KVM_GUEST=y
  $> dmesg | grep KVM
  Booting paravirtualized kernel on KVM

If you want to get really funky, you can check whether qemu-kvm has /dev/kvm open:

  $> for iii in /proc/$(ps h -o tid -C qemu-kvm)/fd/*; do readlink $iii; done | grep kvm
  anon_inode:kvm-vcpu
  /dev/kvm
  anon_inode:kvm-vm

xen

If a guest is crashing you can obtain a stack trace by doing the following:

  • Set "on_crash=preserve" in your domain config
  • Copy the guest kernel's System.map to the host
  • Once the guest has crashed, run /usr/lib/xen/bin/xenctx -s System.map <domid>

General Tips

System Log Files

Always look in dmesg, /var/log/messages etc. for any useful information.

strace

strace can often shed light on a bug - e.g. if you run virt-manager, or libvirtd or qemu-kvm under strace you can see what files they accessed, what commands they executed, what system calls they invoked etc.:

$> strace -ttt -f libvirtd

If the program in question is already running, you can attach to it using strace -p.

gdb

gdb can often be useful to trace the execution of a program. However, in order to get useable information, you will need to install "debuginfo" packages. See the StackTraces page for more information.

SELinux

If you see "AVC denied" or "setroubleshoot" messages in /var/log/messages, your bug might be caused by an SELinux policy issue. Try temporarily putting SELinux into "permissive" mode with:

$> setenforce 0

If this makes your bug go away that doesn't mean your bug is fixed, it just narrows down the cause! You should include the AVC details from ausearch -m AVC -ts recent in the bug report, or if the message includes a sealert -l command then include the details printed by the command.

One common cause of SELinux problems is mis-labelled files. Try:

$> restorecon /path/to/file/in/selinux/message