Common kernel problems

This page documents common problems with the Linux kernel in Fedora.

How to set kernel boot options
Kernel boot options are contained in the file. Each installed kernel has a group of lines called a stanza describing:
 * the title of the operative system to load
 * where to find the boot partition (in grub named root!)
 * what kernel (vmlinuz-*) to boot, with additional kernel options
 * the name of the initrd to load

A typical stanza looks something like this: title Fedora 13 (2.6.33.5-124.fc13.i686.PAE) root (hd1,7) kernel /vmlinuz-2.6.33.5-124.fc13.i686.PAE ro root=/dev/mapper/VG_f13-LV_f13_root rd_LVM_LV=VG_f13/LV_f13_root rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rhgb quiet initrd /initramfs-2.6.33.5-124.fc13.i686.PAE.img

title CentOS 5 (2.6.18-194.3.1.el5) root (hd0,4) kernel /vmlinuz-2.6.18-194.3.1.el5 ro root=/dev/mapper/VG_CentOS-LV_CentOS_root rd_LVM_LV=VG_CentOS/LV_CentOS_root rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rhgb quiet initrd /initrd-2.6.18-194.3.1.el5.img

title Ubuntu 10.04 LTS root (hd0,6) chainloader (hd0,6)+1 kernel /grub/core.img savedefault boot In this example, we have three OO.SS: Fedora 13 (boot) resides on the eighth partition of the second hard disk. (Remember, that in grub the partition and disk numbers begin from 0); CentOS on the fifth partition of the first disk and Ubuntu on the seventh partition of the first disk.

Kernel options are placed at the end of the kernel line and are separated by spaces. In the example:
 * ro: mounts root device read-only on boot
 * root:	root filesystem
 * rd_LVM_LV: it activates the root filesystem in the logical volume LV_f13_root of the volume group VG_f13
 * rd_NO_LUKS: disables crypto LUKS detection
 * rd_NO_MD: disables MD RAID detection
 * rd_NO_DM: disables DM RAID detection
 * LANG: is the system language, written to /etc/sysconfig/i18n in the initramfs
 * SYSFONT: is the console font, written to /etc/sysconfig/i18n in the initramfs
 * KEYTABLE: is the keytable filename, written to /etc/sysconfig/keyboard in the initramfs
 * rhgb: for graphical boot support
 * quiet: disables most log messages

For other options view also the wiki Dracut kernel command line parameters.

When having problems, it is usually a good idea to remove the quiet option so that the full set of kernel messages is shown during boot

How to set module options for boot drivers
Module options are set in the file, or (with versions of module-init-tools in F10+) on the kernel command line. Drivers that are needed to boot the system are put into an initrd, and their options are copied from modprobe.conf by the  script that builds the initrd. To change module options for those drivers, you can change the  file and rebuild the initrd, or alternatively (in recent releases of Fedora) you can simply append on the kernel command line.

For example, to disable adma mode on an nVidia SATA controller, add these options to the kernel command line (format is &lt;modulename&gt;.&lt;option&gt;=value): sata_nv.adma=0 Alternatively, add this line to a file in : options sata_nv adma=0 To get options set in  into the initrd, run the   program. Usually this is just the command  to build a new initrd for the currently-running kernel without overwriting the exisitng one. (See  for help on additional options.) To test the new initrd, reboot the system and use the command line editing facilities to change the name of the initrd. Or, create a new stanza in the  file something like this (see above for the original): title Fedora Core [with new initrd] (2.6.29-0.215.rc7.fc11.i586) root (hd0,1) kernel /vmlinuz-2.6.29-0.215.rc7.fc11.i586 ro root=LABEL=/ rhgb quiet initrd /initrd-2.6.29-0.215.rc7.fc11.i586.new.img This will let you boot with either the new or the old initrd by pressing the up arrow / down arrow keys on the very first boot screen. Once everything is tested, remove the original initrd and rename the new one to the same name as the old one, then remove the "[with new initrd] " stanza from.

Keeping more than the default number of kernels installed
If you are having problems with some update kernels, you may want to increase the maximum number of kernels that yum will leave installed. Edit  and change the number on the   line to do that. Note that you will need enough free space in /boot to keep the extra kernels installed.

Uninstalling older non-working kernels
You may also erase kernels that you know are not working in order to keep older, working kernels from being uninstalled on the next update.

Can't find root filesystem / error mounting /dev/root
To take apart an initrd, do the following .. mkdir initrd cd initrd/ gzip -dc /boot/initrd-2.6.23-0.104.rc3.fc8.img | cpio -id Another way to examine the initrd is with Midnight Commander. Add the extension .cpio.gz to the filename and then just place the cursor over the name and press Enter.
 * A lot of these bugs end up being a broken initrd due to bugs in mkinitrd. Get the user to attach their initrd for their kernel to the bz, and also their /etc/modprobe.conf, or have them examine the contents themselves if they are capable of that.
 * Picking apart the initrd of a working and failing kernel and doing a diff of the init script can reveal clues.


 * Unsupported mount options like "relatime" in /etc/fstab can cause problems. Removing any references to the "relatime" option and rebuilding the initrd will fix this.

Crashes/Hangs

 * Checking whether or not the CapsLock key (or NumLock or ScrollLock) causes the light on the keyboard to change state can be used as an indication of whether or not the kernel has hung completely, or if there is something else going on.
 * For boot related issues we need as much info as possible, so removing    from the boot flags should be the first thing to ask for.
 * Slowing down the speed of text output with  (the number may need to be tweaked higher/lower to suit) may allow the user to take a digital camera photo of the last thing on screen.
 * Booting with  (or even just vga=1 if the video card won't support 791) will put the framebuffer into high resolution mode to get more lines of text on screen, allowing more context for bug analysis.
 * will allow to see the last thing the kernel tried to initialise before it hung.
 * There are numerous switches that change which at times have proven to be useful to diagnose failures by disabling various features.
 * is a big hammer, and if that works, narrowing down by trying  instead may yield clues
 * and  are sometimes useful
 * can be useful on i386; on x86_64 this option is called
 * Given it's new and still seeing quite a few changes,  and/or   may be worth testing. (Though this is kernel 2.6.21 and above only)
 * If you get no output at all from the kernel, sometimes booting with  can sometimes yield something of interest.
 * If the kernel locks up with a 'soft lockup' report, booting with  will disable this check allowing booting to continue.
 * If the kernel locks up really early, booting with  or   may help
 * The system can hang because the clock isn't running properly, see System clock runs too fast/slow
 * Sometimes the system can hang because it is looking for nonexistent floppy drives. See Boot pauses probing floppy device
 * Sometimes multiple options are needed, e.g.
 * Try to narrow down the options needed to the absolute minimum. This helps the kernel maintainers find the underlying problem.
 * If it hangs after "Freeing unused kernel memory: 280k freed" you might have glibc.i686 when your processor is not capable of i686. Replace it to glibc.i386 and be sure the "i686" and "nosegneg" directories are deleted.

Boot pauses probing floppy device
On some machines (mostly laptops with removable floppy drives), boot will pause while the (non-existant) floppy device is probed. A series of the following messages will appear: end_request: I/O error, dev fd0, sector 0 end_request: I/O error, dev fd0, sector 0 Buffer I/O error on device fd0, logical block 0 This is caused by initrd's nash searching for filesystem labels on the floppy device. This problem can be avoided by adding  to the kernel boot options.

Can't find installation CD/DVD or hard drives

 * Try . This disables PCI Message Signaled Interrupts and MMCONFIG.


 * Try booting with  [use DMA only for hard drives]  or   [do not use DMA at all] . This can at least get the system installed, then the drivers can be updated.


 * Try the boot option  on 2.6.34 and later kernels.


 * The option  may be needed by some SCSI and RAID drivers (and some network drivers as well.)


 * Try disabling the AHCI driver by adding . This forces the generic drivers to be used, which may work, but sometimes very slowly.

Install runs very slowly
If the system runs very slowly, it may have a BIOS bug that causes part of the system memory to be uncached. Playing with the  parameter can work around this problem. Trying for example,  will limit the system to 1000 megabytes of memory and may make the install run much faster.

Can't install
Sometimes, even booting with acpi=off or various other boot command line options, the kernel refuses to boot on some subsets of hardware. If none of the above tricks helps, then..
 * In rawhide bugs, if the report is something that would prevent someone from installing the next release (crashes during boot, doesn't find hard disks etc), mark the bug as blocking 'F9Blocker' (bug 235706).
 * if it's against the previously released version of Fedora, then it's possible that the problem was caused in a kernel bug that has since been fixed upstream. As Fedora constantly rebases to newer upstream kernels, they'll get picked up by the respins done by the folks at http://fedoraunity.org  Suggest that the user tries an updated ISO if one is available.

Diagnosing "My machine locked up"
This can be a tricky one to diagnose. Most users don't have serial console capability, so we're mostly guessing in the dark.
 * For possible workarounds for this problem, see Crashes/Hangs
 * If it's repeatable, hooking up a serial cable to a second box can be useful for capturing kernel messages that may get printed just before the lockup. Configure the machine being debugged to boot with   and run a terminal program such as   on the other end. Configure the remote end to talk at the same baud rate (115200). (In minicom  .  More info on setting up a serial terminal can be found at http://searchenterpriselinux.techtarget.com/tip/0,289483,sid39_gci1118136,00.html
 * Sometimes just getting lsmod output from users can yield enough clues if there are multiple reports and common modules between both. (It also allows to filter out reports from users of nvidia,vmware etc).
 * Hooking up serial console /   can sometimes get debug info out of the machine.
 * If the hang happened whilst in X, the machine may still respond to ssh logins from other machines. Try this to get a dmesg.
 * The magic sysrq key might work. See QA/Sysrq for details.


 * booting with  may cause a backtrace to occur when the lockup happens.

Suspend/Resume failure
The most common failure mode is 'black screen on resuming', but the system may also hang while suspending.
 * Laptops using the nv driver should be considered hibernate-only capable as per https://www.redhat.com/archives/fedora-test-list/2007-September/msg00365.html
 * If the system fails to resume, see if the system is locked up completely by hitting the caps lock key.
 * If the capslock light doesn't toggle, or the failure is during suspend, try again, but this time before suspending, activate the pm_trace functionality with . This reprograms the real time clock to contain a few bytes of information which we can use to diagnose which driver failed to suspend or resume. After the hang, reboot, boot up again. Now use the command   and you will get a list of matches like this:   . The last device on the list is likely the one thats causing problems. To find out which driver is causing the problem you will have to look up the driver in  . This can be done using  . It will return a path similar to this one:   which means that the firewire_ohci driver is causing troubles. Unloading the module using   should fix the suspend issues. Please also note that pm_trace uses the RTC for storing the data, which will result into a wrong system clock after boot. To fix it just use system-config-date to set the correct date.
 * If the capslock light does toggle when resuming, then the system did come back up, and it's possible that we just failed to reinitialise the video. It may be useful to initiate the suspend from a tty  and run   by hand.  Upon resuming you'll now have some more debug info to sift through.  Additionally, this way when it resumes, you already have a console logged in from which you can type commands 'blind'. Trying   for example may bring things back to life.
 * Proprietary 3d graphics driver users should test with respective open source drivers.
 * Try 'ing various modules before doing the suspend. If this makes things work again, retry with a smaller set of modules unloaded. Keep retrying until you narrow down which module is to blame.
 * Another trick that sometimes works to force video to come back up is to enable the BIOS password. This makes the system resume in a VGA text mode that the kernel recovers from a lot easier. Not a real solution, but it can help to diagnose other problems.

System clock runs too fast/slow

 * Try a different clock source, e.g. :


 * Clock sources can be changed at runtime by writing the new clocksource name to the file, but be aware that changing to an unstable/broken clock source can hang the system. Changing   or   to   should be okay. (The list of available sources is in the file   in the same directory.)


 * The kernel's tickless mode is enabled by default in Fedora 7 and 8, but can sometimes cause incorrect timekeeping. Using  will disable it.

"High Definition Audio" devices
Many times the model can't be detected properly. Adding the correct model to the sound card driver's entry in /etc/modprobe.d/dist.conf will force the driver to use that model, e.g. . Options for this driver are documented in the file   in the kernel-doc package.

System hangs on reboot
Changing the reboot method can work around this problem. To force a reboot method other than the default, use the  kernel option:


 * forces reboot through the system BIOS.


 * forces a 'warm" reboot (no memory test.)

These can be combined:  forces a warm reboot using the system BIOS.

Booting is slow
The first thing to do is isolate which part of the boot process is slow to determine if the fault is the kernel, the initrd scripts, or other parts of the boot process. One way to do this is using the  application. Install this with yum, and the next time you reboot, profiling will be done during boot which can be collected by running the command  which will generate a .png file containing a graph showing where the time was spent. If the kernel appears to stall during boot, booting with the boot parameter  will insert timestamps before every message the kernel prints to its ringbuffer. Retrieve these messages with dmesg, and look for large deltas between two timestamps to isolate (for eg) drivers which may be spending a long time initialising.

Creation of slab failed
In Rawhide/devel kernels (and in -debug flavors of released kernels), Fedora uses the SLUB allocator with full slab debugging enabled by default. The debugging might cause problems in some rare cases: memory allocations can fail, causing the system to panic. Slab debugging can be disabled with the option  (a single minus sign.)  Note, that this option will hide an actual bug that really should be reported and fixed rather than worked around.

USB devices don't work
This can be caused by USB autosuspend stopping and starting devices repeatedly. To disable autosuspend globally, use the kernel option.

Problems with PCMCIA / PC Card adapters
By default, the kernel only reserves a fairly small amount of memory and I/O space for PC Card adapters. Some adapters need more space, or will not work within the default range of addresses.


 * The amount of memory allocated can be set using the  kernel option. Default is 64 megabytes, but it can be changed to e.g. 256 megabytes using the option  . Going over 256M is not recommended.


 * The default for Cardbus IO space is 256 bytes, but it can be changed using, e.g. to change the size to 4096 bytes, use  . Setting this to a value larger than 4096 may cause problems.

nVidia SATA controllers don't recognize all connected drives

 * Try the kernel parameter

CPU stuck at the lowest frequency on ThinkPad machines
ThinkPad users who see their system throttled as soon as the processor module gets loaded and without obvious reason should check the contents of this file:

If it is set to the lowest value, you must pass  boot parameter as a workaround. (See kernel.org bug #16382 for details.)

Can't decrypt drive / encryption password not accepted
Try adding the boot option. You may also have to blacklist the aesni-intel driver by adding a blacklist entry in.

Elantech trackpad not recognized as a trackpad
You can add the option  to force recognition. This requires at least kernel 2.6.34 to work.

Systems with nVidia adapters using the nouveau driver lock up randomly
Try adding the boot option.

Network drives using the CIFS filesystem get inconsistent data when reading files
This is fixed in recent kernel updates, but can be worked around by adding  to the mount options.

PCI Devices Not Recognized / AHCI: "failed to stop engine"
On kernel version 2.6.34 and later, ACPI is used to determine PCI resources. Some machines have bugs in their ACPI BIOS code and fail set configure resources properly. Try using  to disable use of ACPI for resource enumeration.

Unable to Allocate Memory / page allocation failure
Heavily-loaded network servers may have trouble allocating memory even though there is no shortage. Try setting the sysctl  to 65536 in order to keep additional memory free for allocation by network drivers.

See Also...

 * Common file system problems