How to use kdump to debug kernel crashes

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
m (typo)
(Updated dump analysis instructions)
Line 16: Line 16:
 
=== Step 1: Configuring Kdump ===
 
=== Step 1: Configuring Kdump ===
  
- Install the kexec-tools, crash and kernel-debuginfo packages. Use
+
# First, install the kexec-tools, crash and kernel-debuginfo packages. Use following command line to install the packages.
following command line to install the packages.
+
#: <pre>yum install kexec-tools crash kernel-debuginfo</pre>
 
+
#:
  yum install kexec-tools crash kernel-debuginfo
+
# Next, edit /etc/grub.conf and add the "crashkernel=64M@16M" command line option. An example command line might look like this:
 
+
#: <pre>kernel /vmlinuz-2.6.29.5-191.fc11.x86_64 ro root=/dev/VolGroup00/LogVol00 rhgb console=tty0 console=ttyS0,115200 crashkernel=64M@16M"</pre>
- Boot your normal kernel with the additional command line option "crashkernel=64M@16M".
+
#:
Edit /etc/grub.conf and add the "crashkernel=64M@16M" command line option.
+
# Next, reboot your system
An example command line might look like this:
+
# Finally, active the kdump system service
 
+
#: <pre>/sbin/chkconfig kdump on</pre>
  kernel /vmlinuz-2.6.29.5-191.fc11.x86_64 ro root=/dev/VolGroup00/LogVol00 rhgb console=tty0 console=ttyS0,115200 crashkernel=64M@16M"
+
#: <pre>/sbin/service kdump start</pre>
  
  
 
Notes:
 
Notes:
  
1. Above shown parameter reserves 64MB of physical memory starting
+
# Above shown parameter reserves 64MB of physical memory starting at 16MB. This reserved memory is used to preload and run the capture kernel.
at 16MB. This reserved memory is used to preload and run the
+
# Init scripts take care of pre-loading the capture kernel at system boot time.
capture kernel.
+
# It is recommended to either set up a serial console or switch to run level 3 (init 3) for testing purposes. The reason being that kdump does not reset the console if you are in X or framebuffer mode, and no message might be visible on console after system crash.
 
+
2. Init scripts take care of pre-loading the capture kernel at
+
system boot time.
+
 
+
3. It is recommended to either set up a serial console or switch to
+
run level 3 (init 3) for testing purposes. The reason being that
+
kdump does not reset the console if you are in X or framebuffer
+
mode, and no message might be visible on console after system
+
crash.
+
  
 
=== Step 2: Capturing the Dump ===
 
=== Step 2: Capturing the Dump ===
Line 50: Line 41:
  
 
# Trigger through /proc interface  
 
# Trigger through /proc interface  
#: <pre># echo c > /proc/sysrq-trigger</pre>
+
#: <pre>echo c > /proc/sysrq-trigger</pre>
 
# Trigger by inserting a module which calls panic().
 
# Trigger by inserting a module which calls panic().
  
System will boot into capture kernel. Dump will be automatically saved in
+
The system will boot into the capture kernel. A kernel dump will be automatically saved in <code>/var/crash/<dumpdir></code> and the system will boot back into the regular kernel. The name of the dump directory will depend on date and time of crash. For example, <code>/var/crash/2006-02-17-17:02/vmcore</code>.
/var/crash/<dumpdir> and system will boot back into the regular kernel.
+
 
+
Note: <dumpdir> will be created under /var/crash depending on date and time
+
of crash. For example, /var/crash/2006-02-17-17:02/vmcore.
+
  
 
=== Step 3: Dump Analysis ===
 
=== Step 3: Dump Analysis ===
  
- Open the vmcore using crash tool.
+
Once the system has returned from recovering the crash, you may wish to analyse the kernel dump file using the <code>crash</code> tool.
  
== More Documentation ==
+
# First, locate the recent vmcore dump file:
 +
#: <pre>find /var/crash -type f -mtime -1</pre>
 +
# One you have located a vmcore dump file, call <code>crash</code>:
 +
#: <pre>crash /var/crash/2009-07-17-10\:36/vmcore /usr/lib/debug/lib/modules/`uname -r`/vmlinux</pre>
 +
 
 +
{{admon/note|Missing debuginfo?|Cannot find any files under <code>/usr/lib/debug</code>?  Make sure you have the ''kernel-debuginfo'' package installed.}}
 +
 
 +
For more information on using the <code>crash</code> tool, see [[#More Documentation]].
  
- Kernel Source (Documentation/kdump/kdump.txt).
+
== More Documentation ==
  
- http://lse.sourceforge.net/kdump/
+
* Kernel Source (Documentation/kdump/kdump.txt).
 +
* http://lse.sourceforge.net/kdump/
 +
* Using crash - http://people.redhat.com/anderson
  
 
[[Category:Debugging]]
 
[[Category:Debugging]]

Revision as of 14:49, 17 July 2009

Contents

Kernel and kdump

Kdump is a kernel crash dumping mechanism and is very reliable because the crash dump is captured from the context of a freshly booted kernel and not from the context of the crashed kernel. Kdump uses kexec to boot into a second kernel whenever system crashes. This second kernel, often called the crash kernel, boots with very little memory and captures the dump image.

The first kernel reserves a section of memory that the second kernel uses to boot. Kexec enables booting the capture kernel without going through the BIOS, so contents of the first kernel's memory are preserved, which is essentially the kernel crash dump.

How to Use Kdump

Step 1: Configuring Kdump

  1. First, install the kexec-tools, crash and kernel-debuginfo packages. Use following command line to install the packages.
    yum install kexec-tools crash kernel-debuginfo
  2. Next, edit /etc/grub.conf and add the "crashkernel=64M@16M" command line option. An example command line might look like this:
    kernel /vmlinuz-2.6.29.5-191.fc11.x86_64 ro root=/dev/VolGroup00/LogVol00 rhgb console=tty0 console=ttyS0,115200 crashkernel=64M@16M"
  3. Next, reboot your system
  4. Finally, active the kdump system service
    /sbin/chkconfig kdump on
    /sbin/service kdump start


Notes:

  1. Above shown parameter reserves 64MB of physical memory starting at 16MB. This reserved memory is used to preload and run the capture kernel.
  2. Init scripts take care of pre-loading the capture kernel at system boot time.
  3. It is recommended to either set up a serial console or switch to run level 3 (init 3) for testing purposes. The reason being that kdump does not reset the console if you are in X or framebuffer mode, and no message might be visible on console after system crash.

Step 2: Capturing the Dump

Normally kernel panic() will trigger booting into capture kernel but for testing purposes one can simulate the trigger in one of the following ways.

  1. Trigger through /proc interface
    echo c > /proc/sysrq-trigger
  2. Trigger by inserting a module which calls panic().

The system will boot into the capture kernel. A kernel dump will be automatically saved in /var/crash/<dumpdir> and the system will boot back into the regular kernel. The name of the dump directory will depend on date and time of crash. For example, /var/crash/2006-02-17-17:02/vmcore.

Step 3: Dump Analysis

Once the system has returned from recovering the crash, you may wish to analyse the kernel dump file using the crash tool.

  1. First, locate the recent vmcore dump file:
    find /var/crash -type f -mtime -1
  2. One you have located a vmcore dump file, call crash:
    crash /var/crash/2009-07-17-10\:36/vmcore /usr/lib/debug/lib/modules/`uname -r`/vmlinux
Note.png
Missing debuginfo?
Cannot find any files under /usr/lib/debug? Make sure you have the kernel-debuginfo package installed.

For more information on using the crash tool, see #More Documentation.

More Documentation