How to debug Libvirt problems

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(All bug reports)
(Add virt category)
 
(25 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<!--{{header|<component>}}-->
+
__NOTOC__
 
+
This page describes how to debug <component>.  For general information on how to file a bug, should you need to do so, please see [[Bugs and feature requests]].
+
 
+
Checking the [[Bugs/Common|common bugs page]] or searching in [http://bugzilla.redhat.com bugzilla] will save you time if your problem has already been reported.
+
 
+
 
== Identifying your problem area ==
 
== Identifying your problem area ==
  
Something that ensures that the component in question is at fault not some other thing..
+
Something that ensures that the component in question is at fault not xen qemu virt-*? #TODO FIXME...
  
 
== Information to include in your report ==
 
== Information to include in your report ==
Line 17: Line 12:
 
In all cases, the following should be mentioned and attached to your bug report:  
 
In all cases, the following should be mentioned and attached to your bug report:  
  
* The exact command-line used to start libvirt if other than..  
+
* The exact command-line used to start libvirt if other than.  
 
<pre>service libvirtd start</pre>
 
<pre>service libvirtd start</pre>
* The version number of the libvirt
+
* The version number of libvirt.
 
<pre>rpm -q libvirt</pre>
 
<pre>rpm -q libvirt</pre>
* The hardware architecture being used
+
* The output url from smolt.
* The name of the hypervisor (Xen, QEMU, KVM)
+
<pre>smoltSendProfile -a | grep public</pre>
 +
* The kernel your using.
 +
<pre>uname -r </pre>
 +
* The name of the hypervisor (Xen, QEMU, KVM) your using.
 
* The XML config of the guest domain
 
* The XML config of the guest domain
* For Xen hypervisor, the XenD logfile from /var/log/xen
+
* If using Xen hypervisor, the XenD logfile.
* For QEMU/KVM, the domain logfile from /var/log/libvirt/qemu
+
<pre>/var/log/xen</pre>
 +
* If using QEMU/KVM, the domain logfile
 +
<pre>/var/log/libvirt/qemu</pre>
 +
* If altered the libvirtd.conf file
 +
<pre>/etc/libvirt/libvirtd.conf</pre>
 +
* If altered the qemu.conf file
 +
<pre>/etc/libvirt/qemu.conf</pre>
 +
* Libvirt log file with debug logging enabled.
 +
<pre>/var/log/libvirt/libvirtd.log</pre>
 +
{{admon/note|To enable libvirt debug logging to file you need to do <pre>echo -e "log_level = 1\nlog_outputs="1:file:/var/log/libvirt/libvirtd.log"" >> /etc/libvirt/libvirtd.conf && service libvirtd restart</pre>}}
  
=== <Component> related problems ===
+
{{Anchor|Debugging}}
  
Sample entry
+
== Debugging Libvirt==
  
As well as the information from the [[Template:How_to_debug#AllInfo|'All bug reports']] section, include the following information:
+
{{admon/warning|Always review and sanitize your logs before submitting them. Often when turning on debug logging or when using debuggers in application they will show/store sensitive information such as passwords etc. simply remove or replace the sensitive string with xxxx in the logs before submitting them. }}
  
{{Anchor|Debugging}}
+
As root install libvirt debug package.
 +
 
 +
<pre># debuginfo-install libvirt</pre>
 +
 
 +
Find libvirtd process ID ( PID )
 +
 
 +
<pre>Sample output...
 +
 
 +
$ ps -C libvirtd
 +
  PID TTY          TIME CMD
 +
3983 ?        00:00:00 libvirtd</pre>
 +
 
 +
Attache GNU Project Debugger to libvirt PID
 +
 
 +
<pre>$ gdb libvirtd 3983</pre>
 +
 
 +
If gdb complaints about "Missing separate debuginfos" then type "quit" in the gdb shell and do a copy paste of the debuginfo-install entry and install it the same way as above then and rerun the gdb libvirtd PID command.
 +
 
 +
<pre> Sample error entry...
 +
Missing separate debuginfos, use: debuginfo-install <component></pre>
 +
{{admon/note|If using libvirt from Fedora virt preview repository then gdb will complain about missing seperate debuginfos for libvirt-<preview-release> it's safe to ignore it.}}
 +
Enable logging to a file in gdb.
 +
 
 +
From the gdb shell type `set logging on'
 +
 
 +
<pre>Sample ouput...
 +
 
 +
(gdb) set logging on
 +
Copying output to gdb.txt.
 +
Current language:  auto; currently minimal</pre>
 +
 
 +
Then type cont and redo the steps in libvirt that caused the error your experiencing.
 +
 
 +
<pre>Sample output...
 +
 
 +
(gdb) cont
 +
Continuing.</pre>
 +
 
 +
After you have recreated the steps that causes the error your experiencing press CTRL-C in the gdb shell and type
 +
thread apply all bt.
 +
 
 +
<pre>Sample output...
 +
^C
 +
Program received signal SIGINT, Interrupt.
 +
0x0000003846ad50d3 in *__GI___poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=-1)
 +
    at ../sysdeps/unix/sysv/linux/poll.c:87
 +
87   int result = INLINE_SYSCALL (poll, 3, CHECK_N (fds, nfds), nfds, timeout);
 +
Current language:  auto; currently minimal
 +
(gdb) thread apply all bt
 +
 
 +
Thread 19 (Thread 0x7fffdacff910 (LWP 20935)):
 +
Output trunkated....
 +
 
 +
---Type <return> to continue, or q <return> to quit--
 +
</pre>
 +
 
 +
Press return until you are returned to the gdb shell then type quit and exit the debugger.
 +
 
 +
<pre>Sample output...
 +
 
 +
Current language:  auto; currently minimal
 +
(gdb) quit
 +
The program is running.  Quit anyway (and kill it)? (y or n) y
 +
</pre>
 +
 
 +
Attach gdb.txt to your bugzilla report..
 +
 
 +
If the application crashes on startup you need to start it inside the GNU Project Debugger to do so type open an terminal window and type gdb.
 +
 
 +
<pre>Sample output...
 +
 
 +
[username@example ~]$ gdb
 +
GNU gdb (GDB) Fedora (6.8.50.20090302-38.fc11)
 +
Copyright (C) 2009 Free Software Foundation, Inc.
 +
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 +
This is free software: you are free to change and redistribute it.
 +
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 +
and "show warranty" for details.
 +
This GDB was configured as "x86_64-redhat-linux-gnu".
 +
For bug reporting instructions, please see:
 +
<http://www.gnu.org/software/gdb/bugs/>.
 +
(gdb) </pre>
 +
 
 +
First we want to turn on logging then execute startup of the application and finally tell it to run at our command.
 +
 
 +
<pre>Sample output for...
 +
 
 +
(gdb) set logging on
 +
Copying output to gdb.txt.
 +
(gdb) exec-file /usr/sbin/libvirtd
 +
(gdb) run
 +
Starting program: /usr/sbin/libvirtd</pre>
 +
 
 +
 
 +
When the application has crashed type thread apply all bt and quit gdb.
 +
 
 +
<pre>Sample output...
 +
 
 +
gdb) thread apply all bt
 +
 
 +
Thread 19 (Thread 0x7fffdacff910 (LWP 20935)):
 +
Output trunkated....
  
== Debugging <component> ==
+
---Type <return> to continue, or q <return> to quit--
 +
</pre>
  
As root install <components> debug package.  
+
Press return until you are returned to the gdb shell then type quit and exit the debugger.
  
<pre># debuginfo-install <component></pre>
+
<pre>Sample output...
  
How to GDB component
+
Current language:  auto; currently minimal
 +
(gdb) quit
 +
The program is running.  Quit anyway (and kill it)? (y or n) y
 +
</pre>
  
{{admon/note| Sample Note }}
+
Attach the gdb.txt to your report
{{admon/tip| Sample Tip}}
+
{{admon/warning| Sample Warning}}
+
  
  
[[Category:Debugging]] [[Category:How to]]
+
[[Category:Debugging|L]] [[Category:How to]] [[Category:Virtualization]]

Latest revision as of 19:39, 14 March 2013

[edit] Identifying your problem area

Something that ensures that the component in question is at fault not xen qemu virt-*? #TODO FIXME...

[edit] Information to include in your report

[edit] All bug reports

In all cases, the following should be mentioned and attached to your bug report:

  • The exact command-line used to start libvirt if other than.
service libvirtd start
  • The version number of libvirt.
rpm -q libvirt
  • The output url from smolt.
smoltSendProfile -a | grep public
  • The kernel your using.
uname -r 
  • The name of the hypervisor (Xen, QEMU, KVM) your using.
  • The XML config of the guest domain
  • If using Xen hypervisor, the XenD logfile.
/var/log/xen
  • If using QEMU/KVM, the domain logfile
/var/log/libvirt/qemu
  • If altered the libvirtd.conf file
/etc/libvirt/libvirtd.conf
  • If altered the qemu.conf file
/etc/libvirt/qemu.conf
  • Libvirt log file with debug logging enabled.
/var/log/libvirt/libvirtd.log
Note.png
To enable libvirt debug logging to file you need to do
echo -e "log_level = 1\nlog_outputs="1:file:/var/log/libvirt/libvirtd.log"" >> /etc/libvirt/libvirtd.conf && service libvirtd restart

[edit] Debugging Libvirt

Warning (medium size).png
Always review and sanitize your logs before submitting them. Often when turning on debug logging or when using debuggers in application they will show/store sensitive information such as passwords etc. simply remove or replace the sensitive string with xxxx in the logs before submitting them.

As root install libvirt debug package.

# debuginfo-install libvirt

Find libvirtd process ID ( PID )

Sample output...

$ ps -C libvirtd
  PID TTY          TIME CMD
 3983 ?        00:00:00 libvirtd

Attache GNU Project Debugger to libvirt PID

$ gdb libvirtd 3983

If gdb complaints about "Missing separate debuginfos" then type "quit" in the gdb shell and do a copy paste of the debuginfo-install entry and install it the same way as above then and rerun the gdb libvirtd PID command.

 Sample error entry...
Missing separate debuginfos, use: debuginfo-install <component>
Note.png
If using libvirt from Fedora virt preview repository then gdb will complain about missing seperate debuginfos for libvirt-<preview-release> it's safe to ignore it.

Enable logging to a file in gdb.

From the gdb shell type `set logging on'

Sample ouput... 

(gdb) set logging on
Copying output to gdb.txt.
Current language:  auto; currently minimal

Then type cont and redo the steps in libvirt that caused the error your experiencing.

Sample output...

(gdb) cont
Continuing.

After you have recreated the steps that causes the error your experiencing press CTRL-C in the gdb shell and type thread apply all bt.

Sample output...
^C
Program received signal SIGINT, Interrupt.
0x0000003846ad50d3 in *__GI___poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:87
87	  int result = INLINE_SYSCALL (poll, 3, CHECK_N (fds, nfds), nfds, timeout);
Current language:  auto; currently minimal
(gdb) thread apply all bt

Thread 19 (Thread 0x7fffdacff910 (LWP 20935)):
Output trunkated.... 

---Type <return> to continue, or q <return> to quit--

Press return until you are returned to the gdb shell then type quit and exit the debugger.

Sample output...

Current language:  auto; currently minimal
(gdb) quit
The program is running.  Quit anyway (and kill it)? (y or n) y

Attach gdb.txt to your bugzilla report..

If the application crashes on startup you need to start it inside the GNU Project Debugger to do so type open an terminal window and type gdb.

Sample output...

[username@example ~]$ gdb
GNU gdb (GDB) Fedora (6.8.50.20090302-38.fc11)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) 

First we want to turn on logging then execute startup of the application and finally tell it to run at our command.

Sample output for... 

(gdb) set logging on
Copying output to gdb.txt.
(gdb) exec-file /usr/sbin/libvirtd
(gdb) run
Starting program: /usr/sbin/libvirtd


When the application has crashed type thread apply all bt and quit gdb.

Sample output...

gdb) thread apply all bt

Thread 19 (Thread 0x7fffdacff910 (LWP 20935)):
Output trunkated.... 

---Type <return> to continue, or q <return> to quit--

Press return until you are returned to the gdb shell then type quit and exit the debugger.

Sample output...

Current language:  auto; currently minimal
(gdb) quit
The program is running.  Quit anyway (and kill it)? (y or n) y

Attach the gdb.txt to your report