How to debug Dracut problems

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(Using the shell to debug)
(init.log isn't used any more)
(17 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
<!--{{header|dracut}}-->
 
<!--{{header|dracut}}-->
  
== All bug reports ==
+
'''Foreword'''
  
In all cases, the following should be mentioned and attached to your bug report:
+
If you are experiencing a problem with system initialization due to [[Dracut]], please see the [[Bugs/Common|common bugs]] document before filing a bug.  Some easy configuration tweaks that fix a wide range of issues may be listed there.  If the problem you are seeing is not listed there or none of the workarounds seem to help, please consider filing a bug to help us make Fedora run better on your hardware.
  
'''The exact kernel command line used!'''
+
Be prepared to include some information (logs) about your system as well. These should be complete (no snippets please), not in an archive, uncompressed, with MIME type set as text/plain.
  
An copy of '''/etc/fstab'''
+
= Identifying your problem area =
  
<pre>cp /etc/fstab ~USER/Desktop/fstab.txt</pre>
+
<ol>
 +
<li> Remove <code>rhgb</code> and <code>quiet</code> from the kernel command line
 +
<li> Add <code>rdshell</code> to the kernel command line. This will present a shell should dracut be unable to locate your root device
 +
<li> Add <code>rdshell rdinitdebug</code> to the kernel command line so that dracut shell commands are printed as they are executed
 +
<li> Inspect the debug output generated by the <code>rdinitdebug</code> boot argument with the following commands:
 +
<pre>
 +
# journalctl -a | less
 +
# dmesg | less
 +
</pre>
 +
</ol>
  
The output of '''dmsetup ls --tree'''.
+
= Information to include in your report =
<pre> dmsetup ls --tree > ~USER/Desktop/dmsetup.txt</pre>
+
  
If using an raid device the output of '''cat /proc/mdstat'''.
+
{{Anchor|AllInfo}}
  
<pre> cat /proc/mdstat > ~USER/Desktop/mdstat.txt</pre>
+
== All bug reports ==
  
A copy of '''/etc/dracut.conf''' if you have edited it.
+
In all cases, the following should be mentioned and attached to your bug report:
  
<pre> cp /etc/dracut.conf ~USER/home/Desktop/dracut.conf.txt</pre>
+
* The exact kernel command-line used.  Typically from the bootloader configuration file (e.g. {{filename|/etc/grub.conf}}) or from {{filename|/proc/cmdline}}
 +
* A copy of your disk partition information from {{filename|/etc/fstab}}
 +
* A device listing from device-mapper.  This can be obtained by running the command {{command|dmsetup ls --tree}}
 +
* A list of block device attributes including vol_id compatible mode.  This can be obtained by running the commands {{command|blkid}} and {{command|blkid -o udev}}
 +
* Turn on dracut debugging (see [[How_to_debug_Dracut_problems#Debugging|the 'debugging dracut' section]]), and attach all relevant information from the boot log. This can be obtained by running the command {{command|dmesg{{!}}grep dracut}}.
 +
* If you use a dracut configuration file, please include {{filename|/etc/dracut.conf}}
  
The output of on an image with debugging enabled.
+
== Logical Volume Management related problems ==
  
<pre>dmesg|grep dracut </pre>
+
As well as the information from [[How_to_debug_Dracut_problems#AllInfo|the 'all bug reports' section]], include the following information:
  
{{admon/note| You will need to have created an image with debug enabled and added the rdshell to the kernel command line see below for details on how to do that.}}
+
* Include physical volume information by running the command: {{command|lvm pvdisplay}}
 +
* Include volume group information by running the command: {{command|lvm vgdisplay}}
 +
* Include logical volume information by running the command: {{command|lvm lvdisplay}}
  
== Preparing your system for debugging ==
+
== Software RAID related problems ==
  
To be successful in debugging you will need setup and remotely connect to the serial console.
+
As well as the information from [[How_to_debug_Dracut_problems#AllInfo|the 'all bug reports' section]], include the following information:
  
Add the following lines to you '''/etc/grub.conf'''
+
* If using software RAID disk partitions, please include the output of {{filename|/proc/mdstat}}
  
{{admon/note|2='''This example uses the first serial port, giving <serial_port> the value 0, which in turn gives kernel parameter console=ttyS0'''}}
+
== Network root device related problems ==
  
Beneath '''timeout=5''' put..
+
This section details information to include when experiencing problems on a system whose root device is located on a network attached volume (e.g. iSCSI, NFS or NBD).  As well as the information from [[How_to_debug_Dracut_problems#AllInfo|the 'all bug reports' section]], include the following information:
  
<pre>serial --unit=0 --speed=9600</pre>
+
* Please include the output of {{command|ip addr show}}
<pre>terminal --timeout=5 serial console</pre>
+
  
At the end of the '''kernel''' line put..
+
{{Anchor|Debugging}}
  
<pre>console=tty0 console=ttyS0,9600</pre>
+
= Debugging dracut =
  
So '''/etc/grub.conf''' should look something like this after you've made those changes..
+
== Configure a serial console ==
  
<pre>default=0
+
Successfully debugging dracut will require some form of console logging during the system boot.  This section documents configuring a serial console connection to record boot messages.  To enable serial console output for both the kernel and the bootloader, follow the procedure below.
 +
  <ol>
 +
    <li> Open the file {{filename|/etc/grub.conf}} for editing.  Below the line ''timeout=5'', add the following:
 +
    <pre>serial --unit=0 --speed=9600
 +
terminal --timeout=5 serial console</pre>
 +
    <li> Also in {{filename|/etc/grub.conf}}, add the following boot arguemnts to the ''kernel'' line:
 +
    <pre>console=tty0 console=ttyS0,9600</pre>
 +
    <li> When finished, the {{filename|/etc/grub.conf}} file should look similar to the example below.
 +
    <pre>default=0
 
timeout=5
 
timeout=5
 
serial --unit=0 --speed=9600
 
serial --unit=0 --speed=9600
 
terminal --timeout=5 serial console
 
terminal --timeout=5 serial console
 
title Fedora (2.6.29.5-191.fc11.x86_64)
 
title Fedora (2.6.29.5-191.fc11.x86_64)
        root (hd0,0)
+
  root (hd0,0)
        kernel /vmlinuz-2.6.29.5-191.fc11.x86_64 ro root=/dev/mapper/vg_uc1-lv_root console=tty0 console=ttyS0,9600
+
  kernel /vmlinuz-2.6.29.5-191.fc11.x86_64 ro root=/dev/mapper/vg_uc1-lv_root console=tty0 console=ttyS0,9600
        initrd /dracut-2.6.29.5-191.fc11.x86_64.img</pre>
+
  initrd /dracut-2.6.29.5-191.fc11.x86_64.img</pre>
 +
</ol>
  
Further information on how to configure the kernel for console output can be found [http://www.faqs.org/docs/Linux-HOWTO/Remote-Serial-Console-HOWTO.html#CONFIGURE-KERNEL here].
+
More detailed information on how to configure the kernel for console output can be found at [http://www.faqs.org/docs/Linux-HOWTO/Remote-Serial-Console-HOWTO.html#CONFIGURE-KERNEL].
  
{{admon/tip| You can redirect all non-interactive output to /dev/kmsg and the kernel will put it out on the console when it reaches the kernel buffer by doing
+
{{admon/tip|Redirecting non-interactive output|You can redirect all non-interactive output to /dev/kmsg and the kernel will put it out on the console when it reaches the kernel buffer by doing
<pre>exec >/dev/kmsg 2>&1 </dev/console</pre> }}
+
<pre>exec >/dev/kmsg 2>&1 </dev/console</pre>}}
  
Create the image with debug enabled.
+
== Using the dracut shell ==
  
<pre>dracut -a debug /boot/debug-$(uname -r) $(uname -r) </pre>
+
Dracut offers a shell for interactive debugging in the event dracut fails to locate your root filesystem.  To enable the shell:
 +
# Add the boot parameter ''rdshell'' to your bootloader configuration file (e.g. {{filename|/etc/grub.conf}}
 +
# Remove the boot arguments ''rhgb'' and ''quiet''
  
{{admon/tip| You can overwrite an existing image by using the -f option.
+
A sample {{filename|/etc/grub.conf}} bootloader configuration file is listed below.
<pre>dracut -f -a debug <imagename> <kernel version></pre>}}
+
 
+
Boot with debug enable.
+
 
+
<pre>rdinitdebug rdnetdebug</pre>
+
 
+
== Using the shell to debug ==
+
 
+
Add "rdshell" to the kernel command line and remove "rhgb" and "quiet" and boot the image.  
+
This will drop you to the shell if the imitramfs fails.
+
 
+
{{admon/note| Sample grub entry with serial enabled and rdshell kernel parameter added }}  
+
  
 
<pre>default=0
 
<pre>default=0
Line 85: Line 98:
 
title Fedora (2.6.29.5-191.fc11.x86_64)
 
title Fedora (2.6.29.5-191.fc11.x86_64)
 
         root (hd0,0)
 
         root (hd0,0)
         kernel /vmlinuz-2.6.29.5-191.fc11.x86_64 ro root=/dev/mapper/vg_uc1-lv_root console=tty0 console=ttyS0,9600 rdshell  
+
         kernel /vmlinuz-2.6.29.5-191.fc11.x86_64 ro root=/dev/mapper/vg_uc1-lv_root console=tty0 rdshell  
 
         initrd /dracut-2.6.29.5-191.fc11.x86_64.img</pre>
 
         initrd /dracut-2.6.29.5-191.fc11.x86_64.img</pre>
  
Need commands example #FIXME
+
If system boot fails, you will be dropped into a shell as seen in the example below.
 
+
<pre>
== Dracut debugging kernel command parameters ==
+
No root device found
 +
Dropping to debug shell.
  
<pre>rdshell</pre>
+
sh: can't access tty; job control turned off
{{admon/note|Drop to a shell, if the initramfs fails.}}
+
#
 +
</pre>
  
<pre>rdinitdebug</pre>
+
Use this shell prompt to gather the information requested above (see [[How_to_debug_Dracut_problems#AllInfo|the 'all bug reports' section]]).
  
{{admon/note|set -x for the dracut shell}}
+
== Accessing the root volume from the dracut shell ==
  
<pre>rdbreak=[pre-udev|pre-mount|mount|pre-pivot|]</pre>
+
From the dracut debug shell, you can manually perform the task of locating and preparing your root volume for boot.  The required steps will depend on how your root volume is configured.  Common scenarios include:
 +
* A block device (e.g. {{filename|/dev/sda7}})
 +
* A LVM logical volume (e.g. {{filename|/dev/VolGroup00/LogVol00}})
 +
* An encrypted device (e.g. {{filename|/dev/mapper/luks-4d5972ea-901c-4584-bd75-1da802417d83}})
 +
* A network attached device (e.g. {{filename|netroot{{=}}iscsi:@192.168.0.4::3260::iqn.2009-02.org.fedoraproject:for.all}})
  
{{admon/note|drop the shell on defined breakpoint}}
+
The exact method for locating and preparing will vary.  However, to continue with a successful boot, the objective is to locate your root volume and create a symlink {{filename|/dev/root}} which points to the file system.  For example, the following example demonstrates accessing and booting a root volume that is an encrypted LVM Logical volume.
  
<pre>rdudevinfo</pre>
+
<ol>
 +
  <li> Inspect your partitions using {{command|parted}}
 +
  <pre># parted /dev/sda -s p
 +
Model: ATA HTS541060G9AT00 (scsi)
 +
Disk /dev/sda: 60.0GB
 +
Sector size (logical/physical): 512B/512B
 +
Partition Table: msdos
  
{{admon/note|set udev to loglevel info}}
+
Number  Start  End    Size    Type      File system  Flags
 +
1      32.3kB  10.8GB  107MB  primary  ext4        boot
 +
2      10.8GB  55.6GB  44.7GB  logical                lvm</pre>
 +
  <li> You recall that your root volume was a LVM logical volume.  Scan and activate any logical volumes
 +
  <pre># lvm vgscan
 +
# lvm vgchange -ay</pre>
 +
  <li> You should see any logical volumes now using the command {{command|blkid}}:
 +
  <pre># blkid
 +
/dev/sda1: UUID="3de247f3-5de4-4a44-afc5-1fe179750cf7" TYPE="ext4"
 +
/dev/sda2: UUID="Ek4dQw-cOtq-5MJu-OGRF-xz5k-O2l8-wdDj0I" TYPE="LVM2_member"
 +
/dev/mapper/linux-root: UUID="def0269e-424b-4752-acf3-1077bf96ad2c" TYPE="crypto_LUKS"
 +
/dev/mapper/linux-home: UUID="c69127c1-f153-4ea2-b58e-4cbfa9257c5e" TYPE="ext3"
 +
/dev/mapper/linux-swap: UUID="47b4d329-975c-4c08-b218-f9c9bf3635f1" TYPE="swap" </pre>
 +
  <li> From the output above, you recall that your root volume exists on an encrypted block device.  Following the guidance disk encryption guidance from the [http://docs.fedoraproject.org/install-guide/f{{FedoraVersion}}/en-US/html/apcs04s04.html {{FedoraVersion|long}} Installation Guide], you unlock your encrypted root volume.
 +
  <pre>UUID=$(cryptsetup luksUUID /dev/mapper/linux-root)
 +
cryptsetup luksOpen /dev/mapper/linux-root luks-$UUID
 +
Enter passphrase for /dev/mapper/linux-root:
 +
Key slot 0 unlocked. </pre>
 +
  <li> Next, make a symbolic link to the unlocked root volume
 +
  <pre>ln -s /dev/mapper/luks-$UUID /dev/root</pre>
 +
  <li> With the root volume available, you may continue booting the system by exiting the dracut shell
 +
  <pre>exit</pre>
 +
</ol>
  
<pre>rdudevdebug</pre>
+
== Additional dracut boot parameters ==
  
{{admon/note|set udev to loglevel debug}}
+
The following boot parameters are also available to further assist with debugging boot issues.
  
<pre>rdnetdebug</pre>
+
; rdshell : Drop to a shell, if the initramfs fails.
 +
; rdinitdebug : set -x for the dracut shell.
 +
; rdbreak=[pre-udev|pre-mount|mount|pre-pivot|] : drop the shell on defined breakpoint
 +
; rdudevinfo : set udev to loglevel info
 +
; rdudevdebug : set udev to loglevel debug
 +
; rdnetdebug : debug network scripts in dracut. Output is written to /tmp
  
{{admon/note|debug network scripts in dracut. Output is written to /tmp/}}
+
[[Category:Debugging|D]] [[Category:How to]]
[[Category:Debugging]]
+

Revision as of 02:39, 10 March 2013


Foreword

If you are experiencing a problem with system initialization due to Dracut, please see the common bugs document before filing a bug. Some easy configuration tweaks that fix a wide range of issues may be listed there. If the problem you are seeing is not listed there or none of the workarounds seem to help, please consider filing a bug to help us make Fedora run better on your hardware.

Be prepared to include some information (logs) about your system as well. These should be complete (no snippets please), not in an archive, uncompressed, with MIME type set as text/plain.

Contents

Identifying your problem area

  1. Remove rhgb and quiet from the kernel command line
  2. Add rdshell to the kernel command line. This will present a shell should dracut be unable to locate your root device
  3. Add rdshell rdinitdebug to the kernel command line so that dracut shell commands are printed as they are executed
  4. Inspect the debug output generated by the rdinitdebug boot argument with the following commands:
     # journalctl -a | less
     # dmesg | less
    

Information to include in your report

All bug reports

In all cases, the following should be mentioned and attached to your bug report:

  • The exact kernel command-line used. Typically from the bootloader configuration file (e.g. /etc/grub.conf) or from /proc/cmdline
  • A copy of your disk partition information from /etc/fstab
  • A device listing from device-mapper. This can be obtained by running the command dmsetup ls --tree
  • A list of block device attributes including vol_id compatible mode. This can be obtained by running the commands blkid and blkid -o udev
  • Turn on dracut debugging (see the 'debugging dracut' section), and attach all relevant information from the boot log. This can be obtained by running the command dmesg|grep dracut.
  • If you use a dracut configuration file, please include /etc/dracut.conf

Logical Volume Management related problems

As well as the information from the 'all bug reports' section, include the following information:

  • Include physical volume information by running the command: lvm pvdisplay
  • Include volume group information by running the command: lvm vgdisplay
  • Include logical volume information by running the command: lvm lvdisplay

Software RAID related problems

As well as the information from the 'all bug reports' section, include the following information:

  • If using software RAID disk partitions, please include the output of /proc/mdstat

Network root device related problems

This section details information to include when experiencing problems on a system whose root device is located on a network attached volume (e.g. iSCSI, NFS or NBD). As well as the information from the 'all bug reports' section, include the following information:

  • Please include the output of ip addr show

Debugging dracut

Configure a serial console

Successfully debugging dracut will require some form of console logging during the system boot. This section documents configuring a serial console connection to record boot messages. To enable serial console output for both the kernel and the bootloader, follow the procedure below.

  1. Open the file /etc/grub.conf for editing. Below the line timeout=5, add the following:
    serial --unit=0 --speed=9600
    terminal --timeout=5 serial console
  2. Also in /etc/grub.conf, add the following boot arguemnts to the kernel line:
    console=tty0 console=ttyS0,9600
  3. When finished, the /etc/grub.conf file should look similar to the example below.
    default=0
    timeout=5
    serial --unit=0 --speed=9600
    terminal --timeout=5 serial console
    title Fedora (2.6.29.5-191.fc11.x86_64)
       root (hd0,0)
       kernel /vmlinuz-2.6.29.5-191.fc11.x86_64 ro root=/dev/mapper/vg_uc1-lv_root console=tty0 console=ttyS0,9600
       initrd /dracut-2.6.29.5-191.fc11.x86_64.img

More detailed information on how to configure the kernel for console output can be found at [1].

Idea.png
Redirecting non-interactive output
You can redirect all non-interactive output to /dev/kmsg and the kernel will put it out on the console when it reaches the kernel buffer by doing
exec >/dev/kmsg 2>&1 </dev/console

Using the dracut shell

Dracut offers a shell for interactive debugging in the event dracut fails to locate your root filesystem. To enable the shell:

  1. Add the boot parameter rdshell to your bootloader configuration file (e.g. /etc/grub.conf
  2. Remove the boot arguments rhgb and quiet

A sample /etc/grub.conf bootloader configuration file is listed below.

default=0
timeout=5
serial --unit=0 --speed=9600
terminal --timeout=5 serial console
title Fedora (2.6.29.5-191.fc11.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.29.5-191.fc11.x86_64 ro root=/dev/mapper/vg_uc1-lv_root console=tty0 rdshell 
        initrd /dracut-2.6.29.5-191.fc11.x86_64.img

If system boot fails, you will be dropped into a shell as seen in the example below.

No root device found
Dropping to debug shell.

sh: can't access tty; job control turned off
# 

Use this shell prompt to gather the information requested above (see the 'all bug reports' section).

Accessing the root volume from the dracut shell

From the dracut debug shell, you can manually perform the task of locating and preparing your root volume for boot. The required steps will depend on how your root volume is configured. Common scenarios include:

  • A block device (e.g. /dev/sda7)
  • A LVM logical volume (e.g. /dev/VolGroup00/LogVol00)
  • An encrypted device (e.g. /dev/mapper/luks-4d5972ea-901c-4584-bd75-1da802417d83)
  • A network attached device (e.g. netroot=iscsi:@192.168.0.4::3260::iqn.2009-02.org.fedoraproject:for.all)

The exact method for locating and preparing will vary. However, to continue with a successful boot, the objective is to locate your root volume and create a symlink /dev/root which points to the file system. For example, the following example demonstrates accessing and booting a root volume that is an encrypted LVM Logical volume.

  1. Inspect your partitions using parted
    # parted /dev/sda -s p
    Model: ATA HTS541060G9AT00 (scsi)
    Disk /dev/sda: 60.0GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    
    Number  Start   End     Size    Type      File system  Flags
     1      32.3kB  10.8GB  107MB   primary   ext4         boot
     2      10.8GB  55.6GB  44.7GB  logical                lvm
  2. You recall that your root volume was a LVM logical volume. Scan and activate any logical volumes
    # lvm vgscan
    # lvm vgchange -ay
  3. You should see any logical volumes now using the command blkid:
    # blkid
    /dev/sda1: UUID="3de247f3-5de4-4a44-afc5-1fe179750cf7" TYPE="ext4" 
    /dev/sda2: UUID="Ek4dQw-cOtq-5MJu-OGRF-xz5k-O2l8-wdDj0I" TYPE="LVM2_member" 
    /dev/mapper/linux-root: UUID="def0269e-424b-4752-acf3-1077bf96ad2c" TYPE="crypto_LUKS" 
    /dev/mapper/linux-home: UUID="c69127c1-f153-4ea2-b58e-4cbfa9257c5e" TYPE="ext3" 
    /dev/mapper/linux-swap: UUID="47b4d329-975c-4c08-b218-f9c9bf3635f1" TYPE="swap" 
  4. From the output above, you recall that your root volume exists on an encrypted block device. Following the guidance disk encryption guidance from the Fedora 20 Installation Guide, you unlock your encrypted root volume.
    UUID=$(cryptsetup luksUUID /dev/mapper/linux-root)
    cryptsetup luksOpen /dev/mapper/linux-root luks-$UUID
    Enter passphrase for /dev/mapper/linux-root:
    Key slot 0 unlocked. 
  5. Next, make a symbolic link to the unlocked root volume
    ln -s /dev/mapper/luks-$UUID /dev/root
  6. With the root volume available, you may continue booting the system by exiting the dracut shell
    exit

Additional dracut boot parameters

The following boot parameters are also available to further assist with debugging boot issues.

rdshell 
Drop to a shell, if the initramfs fails.
rdinitdebug 
set -x for the dracut shell.
rdbreak=[pre-udev|pre-mount|mount|pre-pivot|] 
drop the shell on defined breakpoint
rdudevinfo 
set udev to loglevel info
rdudevdebug 
set udev to loglevel debug
rdnetdebug 
debug network scripts in dracut. Output is written to /tmp