Common kernel problems
This page documents common problems with the Linux kernel in Fedora.
How to set kernel boot options
Kernel boot options are contained in the file
/boot/grub/grub.conf. Each installed kernel has a group of lines called a stanza describing:
- the title of the operative system to load
- where to find the boot partition (in grub named root!)
- what kernel (vmlinuz-*) to boot, with additional kernel options
- the name of the initrd to load
A typical stanza looks something like this:
title Fedora 13 (22.214.171.124-124.fc13.i686.PAE) root (hd1,7) kernel /vmlinuz-126.96.36.199-124.fc13.i686.PAE ro root=/dev/mapper/VG_f13-LV_f13_root rd_LVM_LV=VG_f13/LV_f13_root rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rhgb quiet initrd /initramfs-188.8.131.52-124.fc13.i686.PAE.img title CentOS 5 (2.6.18-194.3.1.el5) root (hd0,4) kernel /vmlinuz-2.6.18-194.3.1.el5 ro root=/dev/mapper/VG_CentOS-LV_CentOS_root rd_LVM_LV=VG_CentOS/LV_CentOS_root rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rhgb quiet initrd /initrd-2.6.18-194.3.1.el5.img title Ubuntu 10.04 LTS root (hd0,6) chainloader (hd0,6)+1 kernel /grub/core.img savedefault boot
In this example, we have three OO.SS: Fedora 13 (boot) resides on the eighth partition of the second hard disk. (Remember, that in grub the partition and disk numbers begin from 0); CentOS on the fifth partition of the first disk and Ubuntu on the seventh partition of the first disk.
Kernel options are placed at the end of the kernel line and are separated by spaces. In the example:
- ro: mounts root device read-only on boot
- root: root filesystem
- rd_LVM_LV: it activates the root filesystem in the logical volume LV_f13_root of the volume group VG_f13
- rd_NO_LUKS: disables crypto LUKS detection
- rd_NO_MD: disables MD RAID detection
- rd_NO_DM: disables DM RAID detection
- LANG: is the system language, written to /etc/sysconfig/i18n in the initramfs
- SYSFONT: is the console font, written to /etc/sysconfig/i18n in the initramfs
- KEYTABLE: is the keytable filename, written to /etc/sysconfig/keyboard in the initramfs
- rhgb: for graphical boot support
- quiet: disables most log messages
For other options view also the wiki Dracut kernel command line parameters.
When having problems, it is usually a good idea to remove the quiet option so that the full set of kernel messages is shown during boot
How to set module options for boot drivers
Module options are set in the file
/etc/modprobe.conf, or (with versions of module-init-tools in F10+) on the kernel command line. Drivers that are needed to boot the system are put into an initrd, and their options are copied from modprobe.conf by the
mkinitrd script that builds the initrd. To change module options for those drivers, you can change the
/etc/modprobe.conf file and rebuild the initrd, or alternatively (in recent releases of Fedora) you can simply append on the kernel command line.
For example, to disable adma mode on an nVidia SATA controller, add these options to the kernel command line (format is <modulename>.<option>=value):
Alternatively, add this line to
options sata_nv adma=0
To get options set in
/etc/modprobe.conf into the initrd, run the
mkinitrd program. Usually this is just the command
mkinitrd /boot/initrd-$(uname -r).new.img $(uname -r) to build a new initrd for the currently-running kernel without overwriting the exisitng one. (See
man mkinitrd for help on additional options.) To test the new initrd, reboot the system and use the command line editing facilities to change the name of the initrd. Or, create a new stanza in the
/etc/grub.conf file something like this (see above for the original):
title Fedora Core [with new initrd] (2.6.29-0.215.rc7.fc11.i586) root (hd0,1) kernel /vmlinuz-2.6.29-0.215.rc7.fc11.i586 ro root=LABEL=/ rhgb quiet initrd /initrd-2.6.29-0.215.rc7.fc11.i586.new.img
This will let you boot with either the new or the old initrd by pressing the up arrow / down arrow keys on the very first boot screen. Once everything is tested, remove the original initrd and rename the new one to the same name as the old one, then remove the "[with new initrd] " stanza from
Can't find root filesystem / error mounting /dev/root
- A lot of these bugs end up being a broken initrd due to bugs in mkinitrd. Get the user to attach their initrd for their kernel to the bz, and also their /etc/modprobe.conf, or have them examine the contents themselves if they are capable of that.
- Picking apart the initrd of a working and failing kernel and doing a diff of the init script can reveal clues.
To take apart an initrd, do the following ..
mkdir initrd cd initrd/ gzip -dc /boot/initrd-2.6.23-0.104.rc3.fc8.img | cpio -id
Another way to examine the initrd is with Midnight Commander. Add the extension .cpio.gz to the filename and then just place the cursor over the name and press Enter.
- Unsupported mount options like "relatime" in /etc/fstab can cause problems. Removing any references to the "relatime" option and rebuilding the initrd will fix this.
- Checking whether or not the CapsLock key (or NumLock or ScrollLock) causes the light on the keyboard to change state can be used as an indication of whether or not the kernel has hung completely, or if there is something else going on.
- For boot related issues we need as much info as possible, so removing
rhgbfrom the boot flags should be the first thing to ask for.
- Slowing down the speed of text output with
boot_delay=1000(the number may need to be tweaked higher/lower to suit) may allow the user to take a digital camera photo of the last thing on screen.
- Booting with
vga=791(or even just vga=1 if the video card won't support 791) will put the framebuffer into high resolution mode to get more lines of text on screen, allowing more context for bug analysis.
initcall_debugwill allow to see the last thing the kernel tried to initialise before it hung.
- There are numerous switches that change which at times have proven to be useful to diagnose failures by disabling various features.
acpi=offis a big hammer, and if that works, narrowing down by trying
pci=noacpiinstead may yield clues
noapicare sometimes useful
nolapic_timercan be useful on i386; on x86_64 this option is called
- Given it's new and still seeing quite a few changes,
highres=offmay be worth testing. (Though this is kernel 2.6.21 and above only)
- If you get no output at all from the kernel, sometimes booting with
earlyprintk=vgacan sometimes yield something of interest.
- If the kernel locks up with a 'soft lockup' report, booting with
nosoftlockupwill disable this check allowing booting to continue.
- If the kernel locks up really early, booting with
- The system can hang because the clock isn't running properly, see System clock runs too fast/slow
- Sometimes the system can hang because it is looking for nonexistent floppy drives. See Boot pauses probing floppy device
- Sometimes multiple options are needed, e.g.
clocksource=acpi_pm nohz=off highres=off
- Try to narrow down the options needed to the absolute minimum. This helps the kernel maintainers find the underlying problem.
- If it hangs after "Freeing unused kernel memory: 280k freed" you might have glibc.i686 when your processor is not capable of i686. Replace it to glibc.i386 and be sure the "i686" and "nosegneg" directories are deleted.
Boot pauses probing floppy device
On some machines (mostly laptops with removable floppy drives), boot will pause while the (non-existant) floppy device is probed. A series of the following messages will appear:
end_request: I/O error, dev fd0, sector 0 end_request: I/O error, dev fd0, sector 0 Buffer I/O error on device fd0, logical block 0
This is caused by initrd's nash searching for filesystem labels on the floppy device. This problem can be avoided by adding
floppy.allowed_drive_mask=0 to the kernel boot options.
Can't find installation CD/DVD or hard drives
pci=nomsi,nommconf. This disables PCI Message Signaled Interrupts and MMCONFIG.
- Try booting with
libata.dma=1[use DMA only for hard drives] or
libata.dma=0[do not use DMA at all] . This can at least get the system installed, then the drivers can be updated.
- Try the boot option
pci=nocrson 2.6.34 and later kernels.
- The option
pcie_aspm=offmay be needed by some SCSI and RAID drivers (and some network drivers as well.)
- Try disabling the AHCI driver by adding
rdblacklist=ahci. This forces the generic drivers to be used, which may work, but sometimes very slowly.
Install runs very slowly
If the system runs very slowly, it may have a BIOS bug that causes part of the system memory to be uncached. Playing with the
mem= parameter can work around this problem. Trying for example,
mem=1000M will limit the system to 1000 megabytes of memory and may make the install run much faster.
Sometimes, even booting with acpi=off or various other boot command line options, the kernel refuses to boot on some subsets of hardware. If none of the above tricks helps, then..
- In rawhide bugs, if the report is something that would prevent someone from installing the next release (crashes during boot, doesn't find hard disks etc), mark the bug as blocking 'F9Blocker' (bug 235706).
- if it's against the previously released version of Fedora, then it's possible that the problem was caused in a kernel bug that has since been fixed upstream. As Fedora constantly rebases to newer upstream kernels, they'll get picked up by the respins done by the folks at http://fedoraunity.org Suggest that the user tries an updated ISO if one is available.
Diagnosing "My machine locked up"
This can be a tricky one to diagnose. Most users don't have serial console capability, so we're mostly guessing in the dark.
- For possible workarounds for this problem, see Crashes/Hangs
- If it's repeatable, hooking up a serial cable to a second box can be useful for capturing kernel messages that may get printed just before the lockup. Configure the machine being debugged to boot with
console=ttyS0,115200 console=tty0and run a terminal program such as
minicomon the other end. Configure the remote end to talk at the same baud rate (115200). (In minicom
ctrl-a, p, i, enter. More info on setting up a serial terminal can be found at http://searchenterpriselinux.techtarget.com/tip/0,289483,sid39_gci1118136,00.html
- Sometimes just getting lsmod output from users can yield enough clues if there are multiple reports and common modules between both. (It also allows to filter out reports from users of nvidia,vmware etc).
- Hooking up serial console / can sometimes get debug info out of the machine.
- If the hang happened whilst in X, the machine may still respond to ssh logins from other machines. Try this to get a dmesg.
- The magic sysrq key might work. See QA/Sysrq for details.
- booting with
nmi_watchdog=2may cause a backtrace to occur when the lockup happens.
Suspend/Resume to RAM failure
The most common failure mode is 'black screen on resuming'.
- Laptops using the nv driver should be considered hibernate-only capable as per https://www.redhat.com/archives/fedora-test-list/2007-September/msg00365.html
- Find out if the system is locked up completely by hitting the caps lock key.
- If the capslock light doesn't toggle, the system is completely dead. Try again, but this time before suspending, activate the pm_trace functionality with
echo 1 > /sys/power/pm_trace. This reprograms the real time clock to contain a few bytes of information which we can use to diagnose which driver failed to resume. After the hang, reboot, boot up again. Now use the command
dmesg | grep "hash matches"and you will get a list of matches like this:
hash matches device 0000:05:06.1. The last device on the list is likely the one thats causing problems. To find out which driver is causing the problem you will have to look up the driver in
/sys/bus/pci/drivers/. This can be done using
find /sys/bus/pci/drivers/ -name "0000:05:06.1". It will return a path similar to this one:
/sys/bus/pci/drivers/firewire_ohci/0000:05:06.1which means that the firewire_ohci driver is causing troubles. Unloading the module using
modprobe -r firewire_ohcishould fix the suspend issues. Please also note that pm_trace uses the RTC for storing the data, which will result into a wrong system clock after boot. To fix it just use system-config-date to set the correct date.
- If the capslock light does toggle, then the system did come back up, and it's possible that we just failed to reinitialise the video. http://people.freedesktop.org/~hughsient/quirk may contain further useful information to diagnose this problem. It may also be useful to initiate the suspend from a tty (
ctrl-alt-f1) and run
pm-suspend ; dmesg > dmesg.out ; syncby hand. Upon resuming you'll now have some more debug info to sift through. Additionally, this way when it resumes, you already have a console logged in from which you can type commands 'blind'. Trying
vbetool postfor example may bring things back to life.
- Proprietary 3d graphics driver users should test with respective open source drivers.
rmmod'ing various modules before doing the suspend. If this makes things work again, retry with a smaller set of modules unloaded. Keep retrying until you narrow down which module is to blame.
- Another trick that sometimes works to force video to come back up is to enable the BIOS password. This makes the system resume in a VGA text mode that the kernel recovers from a lot easier. Not a real solution, but it can help to diagnose other problems.
System clock runs too fast/slow
- Try a different clock source, e.g. :
- Clock sources can be changed at runtime by writing the new clocksource name to the file
/sys/devices/system/clocksource/clocksource0/current_clocksource, but be aware that changing to an unstable/broken clock source can hang the system. Changing
acpi_pmshould be okay. (The list of available sources is in the file
available_clocksourcein the same directory.)
- The kernel's tickless mode is enabled by default in Fedora 7 and 8, but can sometimes cause incorrect timekeeping. Using
nohz=off highres=offwill disable it.
Sound card doesn't work
"High Definition Audio" devices
Many times the model can't be detected properly. Adding the correct model to the sound card driver's entry in /etc/modprobe.conf will force the driver to use that model, e.g.
options sound-card-0 model=3stack. Options for this driver are documented in the file
/usr/share/doc/kernel-doc-<version>/Documentation/sound/alsa/ALSA-Configuration.txt in the kernel-doc package.
System hangs on reboot
Changing the reboot method can work around this problem. To force a reboot method other than the default, use the
reboot= kernel option:
reboot=bforces reboot through the system BIOS.
reboot=wforces a 'warm" reboot (no memory test.)
These can be combined:
reboot=b,w forces a warm reboot using the system BIOS.
Booting is slow
The first thing to do is isolate which part of the boot process is slow to determine if the fault is the kernel, the initrd scripts, or other parts of the boot process. One way to do this is using the
bootchart application. Install this with yum, and the next time you reboot, profiling will be done during boot which can be collected by running the command
bootchart which will generate a .png file containing a graph showing where the time was spent. If the kernel appears to stall during boot, booting with the boot parameter
printk.time=1 will insert timestamps before every message the kernel prints to its ringbuffer. Retrieve these messages with dmesg, and look for large deltas between two timestamps to isolate (for eg) drivers which may be spending a long time initialising.
Creation of slab failed
In Rawhide/devel kernels (and in -debug flavors of released kernels), Fedora uses the SLUB allocator with full slab debugging enabled by default. The debugging might cause problems in some rare cases: memory allocations can fail, causing the system to panic. Slab debugging can be disabled with the option
slub_debug=- (a single minus sign.) Note, that this option will hide an actual bug that really should be reported and fixed rather than worked around.
USB devices don't work
This can be caused by USB autosuspend stopping and starting devices repeatedly. To disable autosuspend globally, use the kernel option
Problems with PCMCIA / PC Card adapters
By default, the kernel only reserves a fairly small amount of memory and I/O space for PC Card adapters. Some adapters need more space, or will not work within the default range of addresses.
- The amount of memory allocated can be set using the
cbmemsizekernel option. Default is 64 megabytes, but it can be changed to e.g. 256 megabytes using the option
cbmemsize=256M. Going over 256M is not recommended.
- The default for Cardbus IO space is 256 bytes, but it can be changed using
cbiosize, e.g. to change the size to 4096 bytes, use
cbiosize=4096. Setting this to a value larger than 4096 may cause problems.
nVidia SATA controllers don't recognize all connected drives
- Try the kernel parameter
CPU stuck at the lowest frequency
ThinkPad users who see their system throttled as soon as the processor module gets loaded and without obvious reason should check the contents of this file:
If it is set to the lowest value, you must pass
processor.ignore_ppc=1 boot parameter as a workaround.
(See kernel.org bug #16382 for details.)