Fedora Core 6 Kdump Kexec HowTo
kdump are new features in the 2.6 mainstream kernel. Major portions of these features are now in Fedora Core 5 and later releases. The purpose of these features is to ensure faster boot up and creation of reliable kernel vmcores for diagnostic purposes. More information on the kexec/kdump project can be found at http://lse.sourceforge.net/kdump/.
Kexec is a fastboot mechanism that allows booting a Linux kernel from the context of an already running kernel without going through BIOS. BIOS can be very time consuming, especially on big servers with numerous peripherals. This can save a lot of time for developers who end up booting a machine numerous times.
Kdump is a new kernel crash dumping mechanism and is very reliable. The crash dump is captured from the context of a freshly booted kernel and not from the context of the crashed kernel. Kdump uses kexec to boot into a second kernel whenever the system crashes. This second kernel, often called a capture kernel, boots with very little memory and captures the dump image.
The first kernel reserves a section of memory that the second kernel uses to boot. Kexec enables booting the capture kernel without going through BIOS hence the contents of the first kernel's memory are preserved, which is essentially the kernel crash dump.
Currently, the standard kernel and capture kernel (kernel-kdump) are two different entities, but work is underway to make the standard kernel relocatable (within memory), and thus usable as a capture kernel, eliminating the need for a separate kdump kernel. This feature is currently targeted for delivery as part of Fedora Core 6's General Availability (GA) release.
How to configure kexec
yum install kexec-tools
1. Load a kernel with
kver=<code>uname -r</code> kexec -l /boot/vmlinuz-$kver --initrd=/boot/initrd-$kver.img \ --command-line="<code>cat /proc/cmdline</code>"
The above command boots the system back into the currently running kernel. To load a different kernel, substitute it in place of
1. Reboot the system, taking note that it should bypass the BIOS:
How to configure kdump
1. Make sure
crash are installed:
yum install kernel-kdump kexec-tools crash
1. To do a postmortem debug analysis, install the
yum --enablerepo=\*debuginfo install kernel-debuginfo
1. Modify some boot parameters to reserve a chunk of memory for the capture kernel. For i386 and x86_64, edit
/etc/grub.conf, and append
crashkernel=128M@16M to the end of the kernel line.
This is an example of
/etc/grub.conf with the kdump options added:
# #boot=/dev/hda default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title Fedora Core (2.6.17-1.2570.fc6) root (hd0,0) kernel /vmlinuz-2.6.17-1.2570.fc6 ro root=/dev/VolGroup00/root crashkernel=128M@16M initrd /initrd-2.6.17-1.2570.fc6.img
Similarly, append "crashkernel=128M@32M" to the append line in
/etc/yaboot.conf for ppc64 (the 32M offset is actually hard-coded, so the value given there doesn't actually matter). Any updates to yaboot.conf should followed by a
/sbin/ybin to load the new configuration. This is not needed for GRUB.
This is an example of
/etc/yaboot.conf (for ppc64 machines only) with the kdump options added:
boot=/dev/sda1 init-message=Welcome to Fedora Core!\nHit <TAB> for boot options partition=2 timeout=80 install=/usr/lib/yaboot/yaboot delay=5 enablecdboot enableofboot enablenetboot nonvram fstype=raw image=/vmlinuz-2.6.17-1.2570.fc6 label=linux read-only initrd=/initrd-2.6.17-1.2570.fc6.img append="root=LABEL=/ crashkernel=128M@32M"
1. After making the above changes, reboot the system. The 128M of memory (starting 16 or 32M into the memory) is left untouched by the normal system, reserved for the capture kernel. Take note that the output of
free -m shows 128M less memory than without this parameter, which is expected.
It may be possible to use less than 128M, but testing with only 64M has proven unreliable.
1. Now that the reserved memory region is set up, turn on the kdump init script:
chkconfig kdump on
1. Start up kdump:
service kdump start
1. This should load the
kernel-kdump image via
kexec, leaving the system ready to capture a vmcore upon crashing. To test this, force-crash the system using
echo "c" > /proc/sysrq-trigger
This causes the kernel to panic, followed by the system restarting into the
kdump kernel. When the boot process gets to the point where it starts the
kdump service, the vmcore should be automatically copied, from
/proc/vmcore, out to disk (by default, to
/var/crash/<YYYY-MM-DD-HH:MM>/vmcore). The system then reboots back into the normal kernel.
1. In the normal kernel, the previously installed crash kernel can be used in conjunction with the previously installed
kernel-debuginfo to perform postmortem analysis:
crash /usr/lib/debug/lib/modules/2.6.17-1.2570.fc6/vmlinux /var/crash/2006-08-23-15:34/vmcore crash> bt
Details of using crash for postmortem analysis are outside the scope of this article.
Console frame-buffers and X are not properly supported. On a system typically run with something like "vga=791" in the kernel config line or with X running, console video will be garbled when a kernel is booted via kexec. Note that the kdump kernel should still be able to create a dump, and when the system reboots, video should be restored to normal.