How to use kdump to debug kernel crashes

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
m (formatting section titles using wiki markup)
(Kernel and kdump)
Line 1: Line 1:
 
= Kernel and kdump =
 
= Kernel and kdump =
  
Kdump is a new kernel crash dumping mechanism and is very reliable because
+
Kdump is a kernel crash dumping mechanism and is very reliable because the
 
crash dump is captured from the context of a freshly booted kernel and not
 
crash dump is captured from the context of a freshly booted kernel and not
 
from the context of the crashed kernel. Kdump uses kexec to boot into
 
from the context of the crashed kernel. Kdump uses kexec to boot into
 
a second kernel whenever system crashes. This second kernel, often called
 
a second kernel whenever system crashes. This second kernel, often called
capture kenrel, boots with very little memory and captures the dump image.
+
the crash kernel, boots with very little memory and captures the dump image.
  
 
The first kernel reserves a section of memory that the second kernel uses
 
The first kernel reserves a section of memory that the second kernel uses
to boot. Kexec enables booting the capture kernel without going through
+
to boot. Kexec enables booting the capture kernel without going through the
BIOS hence contents of first kernel's memory are preserved, which is
+
BIOS, so contents of the first kernel's memory are preserved, which is
 
essentially the kernel crash dump.
 
essentially the kernel crash dump.
  
Line 16: Line 16:
 
=== Step 1: Configuring Kdump ===
 
=== Step 1: Configuring Kdump ===
  
- Install latest Fedora Core 5 Test 3 from http://torrent.fedoraproject.org/
+
- Install the "kexec-tools" and "kernel-debuginfo" packages. Use
or from http://download.fedora.redhat.com/pub/fedora/linux/core/test/4.92/
+
 
+
- Install "kernel-kdump", "kexec-tools" and "kernel-debuginfo" packages. Use
+
 
following command line to install the packages.
 
following command line to install the packages.
  
"yum install kernel-kdump kexec-tools kernel-debuginfo"
+
  yum install kernel-kdump kexec-tools kernel-debuginfo
  
- Boot first kernel with additional command line option "crashkernel=64M@16M".
+
- Boot your normal kernel with the additional command line option "crashkernel=64M@16M".
Edit /boot/grub/menu.lst and add "crashkernel=64M@16M" command line option.
+
Edit /etc/grub.conf and add the "crashkernel=64M@16M" command line option.
An example command line might look like as follows.
+
An example command line might look like this:
  
"kernel /vmlinuz-2.6.15-1.1955_FC5smp ro root=/dev/VolGroup00/LogVol00 rhgb console=tty0 console=ttyS0,115200 crashkernel=64M@16M"
+
  kernel /vmlinuz-2.6.29.5-191.fc11.i586 ro root=/dev/VolGroup00/LogVol00 rhgb console=tty0 console=ttyS0,115200 crashkernel=64M@16M"
  
  
Line 37: Line 34:
  
 
2. Init scripts take care of pre-loading the capture kernel at
 
2. Init scripts take care of pre-loading the capture kernel at
the system bootup time.
+
system boot time.
  
2. It is recommended to either setup a serial console or switch to
+
2. It is recommended to either set up a serial console or switch to
 
run level 3 (init 3) for testing purposes. The reason being that
 
run level 3 (init 3) for testing purposes. The reason being that
 
kdump does not reset the console if you are in X or framebuffer
 
kdump does not reset the console if you are in X or framebuffer
Line 52: Line 49:
  
 
- Trigger through /proc interface
 
- Trigger through /proc interface
 
  
 
- Trigger by inserting a module which calls panic().
 
- Trigger by inserting a module which calls panic().
  
 
System will boot into capture kernel. Dump will be automatically saved in
 
System will boot into capture kernel. Dump will be automatically saved in
/var/crash/<dumpdir> and system will boot back into regular kernel.
+
/var/crash/<dumpdir> and system will boot back into the regular kernel.
  
  
Line 63: Line 59:
  
 
- Open the vmcore using crash tool.
 
- Open the vmcore using crash tool.
 
  
 
Note: <vmcore-dir> will be created under /var/crash depending on date and time
 
Note: <vmcore-dir> will be created under /var/crash depending on date and time
Line 72: Line 67:
 
- Kernel Source (Documentation/kdump/kdump.txt).
 
- Kernel Source (Documentation/kdump/kdump.txt).
 
- http://lse.sourceforge.net/kdump/
 
- http://lse.sourceforge.net/kdump/
 
 
 
== Kdump Setup (x86_64) ==
 
 
 
=== Step 1: Configuring Kdump ===
 
 
- Install latest Fedora Core 5 Test 3 from http://torrent.fedoraproject.org/
 
or from http://download.fedora.redhat.com/pub/fedora/linux/core/test/4.92/
 
 
- Install "kernel-kdump", "kexec-tools" and "kernel-debuginfo" packages. Use
 
following command line to install the packages.
 
 
"yum install kernel-kdump kexec-tools kernel-debuginfo"
 
 
- Boot first kernel with additional command line option "crashkernel=64M@16M".
 
Edit /boot/grub/menu.lst and add "crashkernel=64M@16M" command line option.
 
An example command line might look like as follows.
 
 
"kernel /vmlinuz-2.6.15-1.1955_FC5smp ro root=/dev/VolGroup00/LogVol00 rhgb console=tty0 console=ttyS0,115200 crashkernel=64M@16M"
 
 
 
Notes:
 
1. Above shown parameter reserves 64MB of physical memory starting
 
at 16MB. This reserved memory is used to preload and run the
 
capture kernel.
 
 
2. Init scripts take care of pre-loading the capture kernel at
 
the system bootup time.
 
 
2. It is recommended to either setup a serial console or switch to
 
run level 3 (init 3) for testing purposes. The reason being that
 
kdump does not reset the console if you are in X or framebuffer
 
mode, and no message might be visible on console after system
 
crash.
 
 
=== Step 2: Capturing the Dump ===
 
 
Normally kernel panic() will trigger booting into capture kernel but for
 
testing purposes one can simulate the trigger in one of the following
 
ways.
 
 
- Trigger through /proc interface
 
 
 
- Trigger by inserting a module which calls panic().
 
 
System will boot into capture kernel. Dump will be automatically saved in
 
/var/crash/<dumpdir> and system will boot back into regular kernel.
 
  
  

Revision as of 23:24, 17 June 2009

Contents

Kernel and kdump

Kdump is a kernel crash dumping mechanism and is very reliable because the crash dump is captured from the context of a freshly booted kernel and not from the context of the crashed kernel. Kdump uses kexec to boot into a second kernel whenever system crashes. This second kernel, often called the crash kernel, boots with very little memory and captures the dump image.

The first kernel reserves a section of memory that the second kernel uses to boot. Kexec enables booting the capture kernel without going through the BIOS, so contents of the first kernel's memory are preserved, which is essentially the kernel crash dump.

How to Use Kdump (i386)

Step 1: Configuring Kdump

- Install the "kexec-tools" and "kernel-debuginfo" packages. Use following command line to install the packages.

  yum install kernel-kdump kexec-tools kernel-debuginfo

- Boot your normal kernel with the additional command line option "crashkernel=64M@16M". Edit /etc/grub.conf and add the "crashkernel=64M@16M" command line option. An example command line might look like this:

  kernel /vmlinuz-2.6.29.5-191.fc11.i586 ro root=/dev/VolGroup00/LogVol00 rhgb console=tty0 console=ttyS0,115200 crashkernel=64M@16M"


Notes: 1. Above shown parameter reserves 64MB of physical memory starting at 16MB. This reserved memory is used to preload and run the capture kernel.

2. Init scripts take care of pre-loading the capture kernel at system boot time.

2. It is recommended to either set up a serial console or switch to run level 3 (init 3) for testing purposes. The reason being that kdump does not reset the console if you are in X or framebuffer mode, and no message might be visible on console after system crash.

Step 2: Capturing the Dump

Normally kernel panic() will trigger booting into capture kernel but for testing purposes one can simulate the trigger in one of the following ways.

- Trigger through /proc interface

- Trigger by inserting a module which calls panic().

System will boot into capture kernel. Dump will be automatically saved in /var/crash/<dumpdir> and system will boot back into the regular kernel.


Step 3: Dump Analysis

- Open the vmcore using crash tool.

Note: <vmcore-dir> will be created under /var/crash depending on date and time of crash. For example, /var/crash/2006-02-17-17:02/vmcore.

More Documentation

- Kernel Source (Documentation/kdump/kdump.txt). - http://lse.sourceforge.net/kdump/


Step 3: Dump Analysis

- Existing "crash" might not be new enough to open the crash dump. Download latest crash source from http://people.redhat.com/~anderson Build and install the "crash".

- Open the vmcore using crash tool.


Note: <vmcore-dir> will be created under /var/crash depending on date and time of crash. For example, /var/crash/2006-02-17-17:02/vmcore.

More Documentation

- Kernel Source (Documentation/kdump/kdump.txt). - http://lse.sourceforge.net/kdump/