How to debug Systemd problems

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(Various useful systemd related commands)
(reference the upstream wiki rather than maintaining a stale copy of it)
 
(36 intermediate revisions by 6 users not shown)
Line 5: Line 5:
 
If you are experiencing a problem with system boot up due to [[Systemd]], please see the [[Bugs/Common|common bugs]] document before filing a bug.  Some easy configuration tweaks that fix a wide range of issues may be listed there.  If the problem you are seeing is not listed there or none of the workarounds seem to help, please consider filing a bug to help us make Fedora run better on your hardware.
 
If you are experiencing a problem with system boot up due to [[Systemd]], please see the [[Bugs/Common|common bugs]] document before filing a bug.  Some easy configuration tweaks that fix a wide range of issues may be listed there.  If the problem you are seeing is not listed there or none of the workarounds seem to help, please consider filing a bug to help us make Fedora run better on your hardware.
  
Be prepared to include some information (logs) about your system as well. These should be complete (no snippets please), not in an archive, uncompressed, with MIME type set as text/plain.
 
  
= Identifying your problem area =
+
= Debugging systemd problems =
 +
{{admon/note|Follow the upstream wiki guide|To debug systemd problems please follow the helpful upstream wiki page on the topic:<br>
 +
[http://freedesktop.org/wiki/Software/systemd/Debugging http://freedesktop.org/wiki/Software/systemd/Debugging]
 +
}}
  
# Remove ''rhgb'' and ''quiet'' from the kernel command line.
+
= Various useful systemd related commands =
# Add systemd.log_level=debug to the kernel command line to log systemd with debug output enabled.
+
# Add systemd.log_target=kmsg to the kernel command line to let systemd buffer to be written to the kernel log buffer.
+
# Run /bin/systemd --test --system from command line to test run init as systemd.
+
  
= Information to include in your report =
+
* Run <code>systemctl list-jobs</code>
  
{{Anchor|AllInfo}}
+
To identify slow boot and look for the jobs that are "running" those jobs are the ones where boot waits for completion on and the ones that listed as "waiting" will be executed only after those which are "running" are completed.
  
== All bug reports ==
+
* Run <code>systemctl list-units -t service --all</code>
  
In all cases, the following should be mentioned and attached to your bug report:
+
To list all available services and their current status
  
* The exact kernel command-line used if not default.  Typically from the bootloader configuration file (e.g. {{filename|/etc/grub.conf}}) or from {{filename|/proc/cmdline}}
 
* A copy of {{filename|/var/log/messages}}
 
* The output of the dmesg command {{command|dmesg > dmesg.txt}}
 
* Systemd dump {{command|systemctl dump > systemd_dump.txt}}
 
  
 +
* Run <code>systemctl list-units -t service</code>
  
{{admon/note|If you experience bootup difficulties before systemd starts running, see [[How_to_debug_Dracut_problems|Debugging Dracut]] to diagnose that further. }}
+
To show all active services 
  
{{admon/note|If you're system hangs while starting or running udev, see [[How_to_debug_Udev_problems|Debugging Udev]] to diagnose that further. }}
 
  
== Configure a serial console ==
+
* Run <code>systemctl status sshd.service</code>
  
Successfully debugging boot up will require some form of console logging during the system boot.  This section documents configuring a serial console connection to record boot messages.
+
To examine the current runtime status of a service. (In the above example the ssh service)
  
# First, enable serial console output for both the kernel and the bootloader.
 
#* Open the file {{filename|/etc/grub.conf}} for editing.  Below the line ''timeout=5'', add the following:
 
#*:<pre>
 
#*:serial --unit=0 --speed=38400
 
#*:terminal --timeout=5 serial console
 
#*:</pre>
 
#* Also in {{filename|/etc/grub.conf}}, add the following boot arguemnts to the ''kernel'' line:
 
#*:<pre>console=tty0 console=ttyS0,38400</pre>
 
#* When finished, the {{filename|/etc/grub.conf}} file should look similar to the example below.
 
#*:<pre>
 
#*:default=0
 
#*:timeout=5
 
#*:serial --unit=0 --speed=38400
 
#*:terminal --timeout=5 serial console
 
#*:title Fedora (2.6.35.4-18.fc14.x86_64)
 
#*:        root (hd0,0)
 
#*:        kernel /vmlinuz-2.6.35.4-18.fc14.x86_64 ro root=/dev/mapper/VolGroup-lv_root rd_LVM_LV=VolGroup/lv_root #*:rd_LVM_LV=VolGroup/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc #*:KEYTABLE=is-latin1 systemd.log_level=debug systemd.log_target=kmsg console=tty0 console=ttyS0,38400
 
#*:        initrd /initramfs-2.6.35.4-18.fc14.x86_64.img </pre>
 
#* More detailed information on how to configure the kernel for console output can be found at [http://www.faqs.org/docs/Linux-HOWTO/Remote-Serial-Console-HOWTO.html#CONFIGURE-KERNEL].
 
  
{{admon/tip|Redirecting non-interactive output|You can redirect all non-interactive output to /dev/kmsg and the kernel will put it out on the console when it reaches the kernel buffer by doing
+
* Run <code>systemctl list-units -t target --all</code>
<pre>exec >/dev/kmsg 2>&1 </dev/console</pre>}}
+
  
== Enable shell access early in systemd ==
+
To show all available targets.
  
You can enable shell access early in systemd startup process to fall back on and diagnose systemd related boot up issues with various systemctl commands should systemd startup otherwise fail.
 
  
* Run {{command|echo "openvt -c9 /bin/sh" >> /etc/rc.d/rc.sysinit}}
+
* Run <code>systemctl list-units -t target</code>  
  
To enable shell on tty9.
+
To show all active targets.
  
== Various useful systemd related commands ==
 
  
* Run {{command|systemctl list-jobs}}
+
* Run <code>systemctl show -p "Wants" multi-user.target</code>
 
+
To identify slow boot and look for the jobs that are "running" those jobs are the ones where boot waits for completion on and the ones that listed as "waiting" will be executed only after those which are "running" are completed.
+
 
+
* Run {{command|systemctl status ssh.service}}
+
 
+
To examine the current runtime status of a service. ( In the above example the ssh service )
+
 
+
* Run <code>systemctl list-units --type=target</code>  
+
 
+
To show all active targets.
+
  
* Run {{command|systemctl show -p "Wants" multi-user.target}}
+
To see which services a target pulls in. ( In the above example the multi-user.target )
  
To figure out which services a target pulls in. ( In the above example the multi-user.target )
 
  
* Run {{command|systemd --test --system '--unit='multi-user.target}}
+
* Run <code>systemd --test --system --unit=multi-user.target</code>
  
 
To examine what gets started when when booted into a specific target. ( In the above example the multi-user.target )
 
To examine what gets started when when booted into a specific target. ( In the above example the multi-user.target )
  
== Additional systemd boot parameters ==
+
= Systemd boot parameters =
  
 
The following boot parameters are also available to further assist with debugging boot issues.
 
The following boot parameters are also available to further assist with debugging boot issues.
  
; systemd.unit= : Overrides the unit to activate on boot. This may be used to temporarily boot into a different boot unit, for example rescue.target or emergency.service. ( Defaults to default.target. )
+
; systemd.unit= : Overrides the unit to activate on boot. This may be used to temporarily boot into a different boot unit, for example rescue.target or emergency.target. ( Defaults to default.target. )
  
 
; systemd.dump_core= : Takes a boolean argument. If true systemd dumps core when it crashes. Otherwise no core dump is created. ( Defaults to true )
 
; systemd.dump_core= : Takes a boolean argument. If true systemd dumps core when it crashes. Otherwise no core dump is created. ( Defaults to true )

Latest revision as of 12:45, 15 June 2012


Foreword

If you are experiencing a problem with system boot up due to Systemd, please see the common bugs document before filing a bug. Some easy configuration tweaks that fix a wide range of issues may be listed there. If the problem you are seeing is not listed there or none of the workarounds seem to help, please consider filing a bug to help us make Fedora run better on your hardware.


[edit] Debugging systemd problems

Note.png
Follow the upstream wiki guide
To debug systemd problems please follow the helpful upstream wiki page on the topic:
http://freedesktop.org/wiki/Software/systemd/Debugging

[edit] Various useful systemd related commands

  • Run systemctl list-jobs

To identify slow boot and look for the jobs that are "running" those jobs are the ones where boot waits for completion on and the ones that listed as "waiting" will be executed only after those which are "running" are completed.

  • Run systemctl list-units -t service --all

To list all available services and their current status


  • Run systemctl list-units -t service

To show all active services


  • Run systemctl status sshd.service

To examine the current runtime status of a service. (In the above example the ssh service)


  • Run systemctl list-units -t target --all

To show all available targets.


  • Run systemctl list-units -t target

To show all active targets.


  • Run systemctl show -p "Wants" multi-user.target

To see which services a target pulls in. ( In the above example the multi-user.target )


  • Run systemd --test --system --unit=multi-user.target

To examine what gets started when when booted into a specific target. ( In the above example the multi-user.target )

[edit] Systemd boot parameters

The following boot parameters are also available to further assist with debugging boot issues.

systemd.unit= 
Overrides the unit to activate on boot. This may be used to temporarily boot into a different boot unit, for example rescue.target or emergency.target. ( Defaults to default.target. )
systemd.dump_core= 
Takes a boolean argument. If true systemd dumps core when it crashes. Otherwise no core dump is created. ( Defaults to true )
systemd.crash_shell= 
Takes a boolean argument. If true systemd spawns a shell when it crashes. Otherwise no core dump is created. Defaults to false, for security reasons, as the shell is not protected by any password authentication.
systemd.crash_chvt= 
Takes an integer argument. If positive systemd activates the specified virtual terminal when it crashes. ( Defaults to -1 )
systemd.confirm_spawn= 
Takes a boolean argument. If true asks for confirmation when spawning processes. ( Defaults to false )
systemd.show_status= 
Takes a boolean argument. If true shows terse service status updates on the console during bootup. ( Defaults to true )
systemd.sysv_console= 
Takes a boolean argument. If true output of SysV init scripts will be directed to the console. ( Defaults to true, unless quiet is passed as kernel command line option in which case it defaults to false. )
systemd.log_target= 
Set log target. Argument must be one of console, syslog, kmsg, syslog-or-kmsg, null.
systemd.log_level= 
Set log level. As argument this accepts a numerical log level or the well-known syslog symbolic names (lowercase): emerg, alert, crit, err, warning, notice, info, debug.
systemd.log_color= 
Highlight important log messages. Argument is a boolean value. If the argument is omitted it defaults to true.
systemd.log_location= 
Include code location in log messages. This is mostly relevant for debugging purposes. Argument is a boolean value. If the argument is omitted it defaults to true.