From Fedora Project Wiki
 
(41 intermediate revisions by 4 users not shown)
Line 16: Line 16:
* <code>/tmp/syslog</code>, messages from kernel and external programs (Network Manager)
* <code>/tmp/syslog</code>, messages from kernel and external programs (Network Manager)
* <code>/tmp/yum.log</code>, yum's internal log
* <code>/tmp/yum.log</code>, yum's internal log
* <code>/tmp/dnf.log</code>, [[dnf|DNF]]'s internal log
* <code>/tmp/dnf.hawkey.log</code>, [[dnf|DNF]]'s Hawkey internal log
* <code>/tmp/dnf.rpm.log</code>, [[dnf|DNF]]'s RPM internal log


Certain log messages are also written to the terminals.
Certain log messages are also written to the terminals:
* <code>/dev/tty3</code>, messages from <code>anaconda.log</code>, <code>storage.log</code> and <code>yum.log</code>.
* <code>/dev/tty3</code>, messages from <code>anaconda.log</code>, <code>storage.log</code> and <code>yum.log</code>.
* <code>/dev/tty4</code>, same as <code>syslog</code>
* <code>/dev/tty4</code>, same as <code>syslog</code>
Line 25: Line 28:
There are two other log files created on the target filesystem, in the <code>/root</code> directory, also accessible at <code>/mnt/sysimage/root</code> during the installation:
There are two other log files created on the target filesystem, in the <code>/root</code> directory, also accessible at <code>/mnt/sysimage/root</code> during the installation:
* <code>/mnt/sysimage/root/install.log</code>, log of the package installation process.
* <code>/mnt/sysimage/root/install.log</code>, log of the package installation process.
* <code>/mnt/sysimage/root/install.log.syslog</code>, messages from installation chroot logged through the system's syslog. Mostly information about users and groups created during yum's package installation.
* <code>/mnt/sysimage/root/install.log.syslog</code>, messages from installation chroot logged through the system's syslog. Mostly information about users and groups created during dnf|yum's package installation.


=== Log format ===
=== Log format ===
Line 38: Line 41:
*# <code>WARN</code>
*# <code>WARN</code>
*# <code>ERR</code>
*# <code>ERR</code>
*# <code>CRITICAL</code>
*# <code>CRIT</code>
* <code>facility</code> is the program or component that created the message. Could be for instance <code>kernel<code>, <code>anaconda</code>, <code>storage</code> or similar.
* <code>facility</code> is the program or component that created the message. Could be for instance <code>kernel<code>, <code>anaconda</code>, <code>storage</code> or similar.
* <code>message</code> is the log message itself.
* <code>message</code> is the log message itself.
Line 45: Line 48:
  LOGLEVEL facility:message
  LOGLEVEL facility:message


== Remote logging ==
== Remote logging via TCP ==
Anaconda supports remote logging handled through the rsyslog daemon running on the installed system. It can be configured to forward its logs through TCP to an arbitrary machine in network that is also running a syslog daemon. This is controlled with the <code>syslog</code> [[Anaconda/Options|command line option]].
Anaconda supports remote logging handled through the rsyslog daemon running on the installed system. It can be configured to forward its logs through TCP to an arbitrary machine in network that is also running a syslog daemon. This is controlled with the <code>syslog</code> [[Anaconda/Options|command line option]]. ''Do not forget to enable the port you are running your local syslog daemon on in your firewall.''


It's up to you how the remote logging daemon is configured, the following is an example that logs the messages into the same files as they are logged on the installed system:
=== What is logged remotely ===
#rsyslog v3 config file
Everything that is logged directly by anaconda should also appear in the remote logs. This includes messages emitted  by the loader and the storage subsystem. All anaconda tracebacks (/tmp/anaconda-tb-xyz) are concatenated into a single file /tmp/anaconda-tb-all.log and then transferred. Also, /tmp/x.log is transferred.
 
#### MODULES ####
The remote logging only works when the installer initializes network. Once network is up, it takes a couple of minutes for rsyslogd to realize this. Rsyslog has a queue for messages that couldn't be forwarded because of inaccessible network and it eventually forwards all of them, in the correct order.
 
$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
=== Configuration ===
$SystemLogSocketName /home/akozumpl/projects/syslog/socket
 
It's up to you how the remote logging daemon is configured, you can for instance log all incoming messages into one file or sort them into directories according to the IP address of the remote system.
# Provides UDP syslog reception
 
# $ModLoad imudp.so
The anaconda RPM provides the <code>analog</code> script, which generates a suitable rsyslogd configuration file based on a couple of install parameters. It is also able to generate a bash command to launch rsyslogd with the generated configuration. Thus you can do from a shell:
# $UDPServerRun 6080
  $ eval `scripts/analog -p 6080 -s -o ./someconf /home/akozumpl/remote_inst`
This starts an rsyslog daemon that will listen on port 6080. The logs from the remote machine with IP 10.34.33.221 will be stored under <code>/home/akozumpl/remote_inst/10.34.33.221/</code>, e.g. <code>/home/akozumpl/remote_inst/10.34.33.221/anaconda.log</code>.
# Provides TCP syslog reception
$ModLoad imtcp.so 
$InputTCPServerRun 6080
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,  
# not useful and an extreme performance hit
#$ActionFileEnableSync on
#### RULES ####
$template anaconda_tty4, "%timestamp:8:$:date-rfc3164%,%timestamp:1:3:date-subseconds% %syslogseverity-text:::uppercase% %programname%:%msg%\n"
  $template anaconda_debug, "%syslogfacility-text%|%hostname%|%syslogseverity-text%|%syslogtag%|%msg%\n"
$template anaconda_syslog, "%timestamp:8:$:date-rfc3164%,%timestamp:1:3:date-subseconds% %syslogseverity-text:::uppercase% %programname%:%msg%\n"
*.*                                                  /home/akozumpl/projects/syslog/out_all_debug.log;anaconda_debug
kern.*;\
daemon.*                                            /home/akozumpl/projects/syslog/syslog;anaconda_syslog
   
:programname, contains, "anaconda"                  /home/akozumpl/projects/syslog/anaconda.log;anaconda_syslog
:programname, contains, "program"                    /home/akozumpl/projects/syslog/program.log;anaconda_syslog
:programname, contains, "storage"                    /home/akozumpl/projects/syslog/storage.log;anaconda_syslog
:hostname, contains, "sysimage"                      /home/akozumpl/projects/syslog/install.log.syslog;anaconda_syslog
# discard those that we logged
:programname, contains, "rsyslogd"                  ~
:programname, contains, "anaconda"                  ~
:programname, contains, "program"                    ~
:programname, contains, "storage"                    ~
:hostname, contains, "sysimage"                      ~
kern.*                                              ~
daemon.*                                            ~
# dump the rest
*.*                                                  /home/akozumpl/projects/syslog/unknown_source.log;anaconda_debug


The remote syslog configuration exploits several log message characteristics to be able to sort them into the correct files:
The remote syslog configuration exploits several log message characteristics to be able to sort them into the correct files:
* the IP of the message sender to know which machine generated the message and thud what directory does the message belong to.
* <code>anaconda.log, storage.log</code> and <code>program.log</code> have the name embedded in them as <code>programname</code>.
* <code>anaconda.log, storage.log</code> and <code>program.log</code> have the name embedded in them as <code>programname</code>.
* <code>syslog/code> messages are coming in from kernel and daemon facilities, just like they do on the installed system
* <code>syslog</code> messages are coming in from kernel and daemon facilities, just like they do on the installed system
* <code>install.log.syslog</code> made during package installation is logged as a special <code>sysimage</code> hostname.
* <code>install.log.syslog</code> made during package installation is logged as a special <code>sysimage</code> hostname.


The sample configuration uses the same message format for remote logging as anaconda does, but note that the remote host can specify its own.
Run <code>analog</code> without the <code>-o</code> option to see how exactly does a fitting configuration file look like. Also notice that it uses the same message format for remote logging as anaconda does, but you can of course modify this to specify any format you want.
 
=== See also ===
* [http://www.rsyslog.com/doc Rsyslog documentation]
* <code>man tailf</code>
 
== Remote logging via virtio ==
QEMU/KVM in Fedora 13 and onwards allows one to create virtual machines with [[Features/VirtioSerial|multiple virtio char devices]] exposed to the guest machine. One such device can be used to forward anaconda logs to the host machine. In that way we can get logs forwarded in real time, as soon the anaconda logging subsystem is initialized (early) and not need to wait for the network to come up. Also, it's the only way to forward the logs in a no-network setup.
 
=== Configuration ===
Anaconda will be forwarding logs over virtio automatically if it is able to find the port <code>/dev/virtio-ports/org.fedoraproject.anaconda.log.0"</code>. This is port is created using a libvirt XML directive that wires it to a TCP socket on the host's side. It's then possible to read the logs from there directly, or make an rsyslog instance to parse them and file them into respective files. See the ascii chart below for the whole ensemble:
  Anaconda--->rsyslog(guest)--->virtio(guest char device)--->kvm hypervisor--->virtio(TCP socket)
                                                                                  |
                                                                                  v
                                                        forwarded log files<---rsyslog(host)
 
Step by step instructions to set everything up follow:
<ol>
<li> Create a testing virtual machine, e.g. using Virtual Manager </li>
<li> Add the virtio-serial port to your virtual machine, direct it to the TCP port 6080 on the host.  Start by editing the guest configuration:<pre>virsh edit <machine name></pre>
<li> In the guest editor, add following information into the <code>&lt;devices&gt;</code> section:
<pre><channel type='tcp'>
  <source mode='connect' host='127.0.0.1' service='6080'/>
  <target type='virtio' name='org.fedoraproject.anaconda.log.0'/>
</channel></pre>
<li>start the listening rsyslogd process on the host, using the <code>analog</code> script described [[#Remote_logging_via_TCP|above]]:
<pre>eval `analog -p 6080 -o rsyslogd.conf -s /home/akozumpl/remote_inst`</pre>
<li>Start the virtual machine.
<li>Continue with the installation.  Immediately after the Anaconda greeting is displayed the log messages will appear in the directory given to <code>analog</code> script, in the <code>127.0.0.1</code> subdirectory.
</ol>
 
==== virt-install ====
 
If you are using virt-install you can configure it with the --channel option:
<pre>--channel tcp,host=127.0.0.1:6080,mode=connect,target_type=virtio,name=org.fedoraproject.anaconda.log.0</pre>
 
=== Known issues and troubleshooting ===
* works in libvirt>=0.8.2
* chroot syslog messages from <code>/mnt/sysimage/root/install.log.syslog</code> are not forwarded.
* it is not possible to start the machine unless something is listening on the TCP port where virtio-serial is connected.
* if you want to test that the virtio connection is working, instead of using analog and rsyslog just let a netcat utility listen on the given port, e.g. <code>nc -l 0.0.0.0 6080</code>. You should start seeing raw logs in the terminal once the guest machine starts booting.
* if both remote TCP logging via <code>syslog=</code> and remote virtio logging via <code>virtiolog=</code> are specified on the command line, one has to setup two rsyslogd instances on the server/host to listen to both the connections otherwise the sending rsyslog's queues get full and the forwarding stops.
 
=== See also ===
* [[Features/VirtioSerial]]
* [http://wiki.libvirt.org/page/Virtio Virtio at the libvirt wiki]
* [[http://libvirt.org/formatdomain.html#elementsConsole libvirt domain XML format]]
 
== Anaconda logs on the running system ==
After every successful installation, anaconda logs are copied into <code>/var/log</code> on the system you just installed. To avoid name clashes with other log files there, the anaconda logs are renamed:
 
{|
! Name during installation !! Name on the target system
|-
| <code>/tmp/anaconda.log</code> || <code>/var/log/anaconda.log</code>
|-
| <code>/tmp/syslog</code> || <code>/var/log/anaconda.syslog</code>
|-
| <code>/tmp/X.log</code> || <code>/var/log/anaconda.xlog</code>
|-
| <code>/tmp/program.log</code> || <code>/var/log/anaconda.program.log</code>
|-
| <code>/tmp/storage.log</code> || <code>/var/log/anaconda.storage.log</code>
|-
| <code>/tmp/yum.log</code> || <code>/var/log/anaconda.yum.log</code>
|-
| <code>/tmp/ifcfg.log</code> (new in F14) || not copied
|}
 
Starting with Fedora 15 (or post F14 Rawhide), the logs go to <code>/var/log/anaconda</code> directory on the target system, including ifcfg.log inroduced in F14.
 
== Logging tips ==
 
If you are asked to provide logs for a bugzilla, your best option is switching from the anaconda GUI to tty2 and then use scp to copy the files to your computer, e.g.:
cd /tmp
scp anaconda.log aklap:/home/akozumpl/
 
It is also possible to make a complete dump of a state of running anaconda process (the same dump that is compiled automatically if an unhandled exception occurs). To do this send the main anaconda process SIGUSR2:
kill -USR2 `cat /var/run/anaconda.pid``
 
This builds a file <code>/tmp/anaconda-tb-?????</code> that also contains <code>anaconda.log</code>, <code>storage.log</code> and <code>syslog</code>.
 
If you are on a KVM virtual machine and there's no scp available (stage1), you can (after setting up the network if not up already) redirect to a special tcp file, on host:
nc -l 4444 > syslog.log
 
on guest:
ifconfig eth0 10.0.2.10/24 up
grep "" /tmp/syslog > /dev/tcp/10.0.2.2/4444


== To do ==
== To do ==
The current list of logging requirements and tasks is maintained in bugzilla [https://bugzilla.redhat.com/show_bug.cgi?id=524980 524980].
* The current list of logging requirements and tasks is maintained in bugzilla [https://bugzilla.redhat.com/show_bug.cgi?id=524980 524980].
* A support for KVM's virtio logging is coming later [https://bugzilla.redhat.com/show_bug.cgi?id=576439 576439].

Latest revision as of 11:23, 22 October 2015

Introduction

Anaconda tracks all of its activities in logs. This includes:

  • changing installation steps (that roughly correspond to different screens in the graphical installer)
  • storage devices detection and manipulation
  • installation media detection
  • network initialization
  • kernel messages
  • calls to critical methods within anaconda
  • calls to external programs

Logging on the installed system

During the installation the logs are stored in the /tmp directory:

  • /tmp/anaconda.log, the general installation information, particularly the step changes.
  • /tmp/storage.log, storage devices scan and manipulation (hard drives, partitions, LVM, RAID), partitioning
  • /tmp/program.log, calls to external programs, their output
  • /tmp/syslog, messages from kernel and external programs (Network Manager)
  • /tmp/yum.log, yum's internal log
  • /tmp/dnf.log, DNF's internal log
  • /tmp/dnf.hawkey.log, DNF's Hawkey internal log
  • /tmp/dnf.rpm.log, DNF's RPM internal log

Certain log messages are also written to the terminals:

  • /dev/tty3, messages from anaconda.log, storage.log and yum.log.
  • /dev/tty4, same as syslog
  • /dev/tty5, stdout and stderr from external programs

tty3 and tty4 reflect certain log files. Log files always contain messages from all the loglevels, including debug, but the minimal loglevel on the terminals can be controlled with the loglevel command line option.

There are two other log files created on the target filesystem, in the /root directory, also accessible at /mnt/sysimage/root during the installation:

  • /mnt/sysimage/root/install.log, log of the package installation process.
  • /mnt/sysimage/root/install.log.syslog, messages from installation chroot logged through the system's syslog. Mostly information about users and groups created during dnf|yum's package installation.

Log format

In files the format of the log messages is as follows:

H:M:S,ms LOGLEVEL facility:message

where:

  • H:M:S is the message timestamp
  • ms is the millisecond part of timestamp. Note that this will usually become zero on a remote syslog.
  • LOGLEVEL is the message loglevel. In theory, because kernel messages are part of anaconda logs, all loglevels that are defined in rsyslog can appear in the logfiles. Anaconda itself will however log only at the following loglevels:
    1. DEBUG
    2. INFO
    3. WARN
    4. ERR
    5. CRIT
  • facility is the program or component that created the message. Could be for instance kernel, anaconda, storage or similar.
  • message is the log message itself.

For the logs running in terminals, the format simply is:

LOGLEVEL facility:message

Remote logging via TCP

Anaconda supports remote logging handled through the rsyslog daemon running on the installed system. It can be configured to forward its logs through TCP to an arbitrary machine in network that is also running a syslog daemon. This is controlled with the syslog command line option. Do not forget to enable the port you are running your local syslog daemon on in your firewall.

What is logged remotely

Everything that is logged directly by anaconda should also appear in the remote logs. This includes messages emitted by the loader and the storage subsystem. All anaconda tracebacks (/tmp/anaconda-tb-xyz) are concatenated into a single file /tmp/anaconda-tb-all.log and then transferred. Also, /tmp/x.log is transferred.

The remote logging only works when the installer initializes network. Once network is up, it takes a couple of minutes for rsyslogd to realize this. Rsyslog has a queue for messages that couldn't be forwarded because of inaccessible network and it eventually forwards all of them, in the correct order.

Configuration

It's up to you how the remote logging daemon is configured, you can for instance log all incoming messages into one file or sort them into directories according to the IP address of the remote system.

The anaconda RPM provides the analog script, which generates a suitable rsyslogd configuration file based on a couple of install parameters. It is also able to generate a bash command to launch rsyslogd with the generated configuration. Thus you can do from a shell:

$ eval scripts/analog -p 6080 -s -o ./someconf /home/akozumpl/remote_inst

This starts an rsyslog daemon that will listen on port 6080. The logs from the remote machine with IP 10.34.33.221 will be stored under /home/akozumpl/remote_inst/10.34.33.221/, e.g. /home/akozumpl/remote_inst/10.34.33.221/anaconda.log.

The remote syslog configuration exploits several log message characteristics to be able to sort them into the correct files:

  • the IP of the message sender to know which machine generated the message and thud what directory does the message belong to.
  • anaconda.log, storage.log and program.log have the name embedded in them as programname.
  • syslog messages are coming in from kernel and daemon facilities, just like they do on the installed system
  • install.log.syslog made during package installation is logged as a special sysimage hostname.

Run analog without the -o option to see how exactly does a fitting configuration file look like. Also notice that it uses the same message format for remote logging as anaconda does, but you can of course modify this to specify any format you want.

See also

Remote logging via virtio

QEMU/KVM in Fedora 13 and onwards allows one to create virtual machines with multiple virtio char devices exposed to the guest machine. One such device can be used to forward anaconda logs to the host machine. In that way we can get logs forwarded in real time, as soon the anaconda logging subsystem is initialized (early) and not need to wait for the network to come up. Also, it's the only way to forward the logs in a no-network setup.

Configuration

Anaconda will be forwarding logs over virtio automatically if it is able to find the port /dev/virtio-ports/org.fedoraproject.anaconda.log.0". This is port is created using a libvirt XML directive that wires it to a TCP socket on the host's side. It's then possible to read the logs from there directly, or make an rsyslog instance to parse them and file them into respective files. See the ascii chart below for the whole ensemble:

 Anaconda--->rsyslog(guest)--->virtio(guest char device)--->kvm hypervisor--->virtio(TCP socket)
                                                                                 |
                                                                                 v
                                                       forwarded log files<---rsyslog(host)

Step by step instructions to set everything up follow:

  1. Create a testing virtual machine, e.g. using Virtual Manager
  2. Add the virtio-serial port to your virtual machine, direct it to the TCP port 6080 on the host. Start by editing the guest configuration:
    virsh edit <machine name>
  3. In the guest editor, add following information into the <devices> section:
    <channel type='tcp'>
      <source mode='connect' host='127.0.0.1' service='6080'/>
      <target type='virtio' name='org.fedoraproject.anaconda.log.0'/>
    </channel>
  4. start the listening rsyslogd process on the host, using the analog script described above:
    eval `analog -p 6080 -o rsyslogd.conf -s /home/akozumpl/remote_inst`
  5. Start the virtual machine.
  6. Continue with the installation. Immediately after the Anaconda greeting is displayed the log messages will appear in the directory given to analog script, in the 127.0.0.1 subdirectory.

virt-install

If you are using virt-install you can configure it with the --channel option:

--channel tcp,host=127.0.0.1:6080,mode=connect,target_type=virtio,name=org.fedoraproject.anaconda.log.0

Known issues and troubleshooting

  • works in libvirt>=0.8.2
  • chroot syslog messages from /mnt/sysimage/root/install.log.syslog are not forwarded.
  • it is not possible to start the machine unless something is listening on the TCP port where virtio-serial is connected.
  • if you want to test that the virtio connection is working, instead of using analog and rsyslog just let a netcat utility listen on the given port, e.g. nc -l 0.0.0.0 6080. You should start seeing raw logs in the terminal once the guest machine starts booting.
  • if both remote TCP logging via syslog= and remote virtio logging via virtiolog= are specified on the command line, one has to setup two rsyslogd instances on the server/host to listen to both the connections otherwise the sending rsyslog's queues get full and the forwarding stops.

See also

Anaconda logs on the running system

After every successful installation, anaconda logs are copied into /var/log on the system you just installed. To avoid name clashes with other log files there, the anaconda logs are renamed:

Name during installation Name on the target system
/tmp/anaconda.log /var/log/anaconda.log
/tmp/syslog /var/log/anaconda.syslog
/tmp/X.log /var/log/anaconda.xlog
/tmp/program.log /var/log/anaconda.program.log
/tmp/storage.log /var/log/anaconda.storage.log
/tmp/yum.log /var/log/anaconda.yum.log
/tmp/ifcfg.log (new in F14) not copied

Starting with Fedora 15 (or post F14 Rawhide), the logs go to /var/log/anaconda directory on the target system, including ifcfg.log inroduced in F14.

Logging tips

If you are asked to provide logs for a bugzilla, your best option is switching from the anaconda GUI to tty2 and then use scp to copy the files to your computer, e.g.:

cd /tmp
scp anaconda.log aklap:/home/akozumpl/

It is also possible to make a complete dump of a state of running anaconda process (the same dump that is compiled automatically if an unhandled exception occurs). To do this send the main anaconda process SIGUSR2:

kill -USR2 cat /var/run/anaconda.pid`

This builds a file /tmp/anaconda-tb-????? that also contains anaconda.log, storage.log and syslog.

If you are on a KVM virtual machine and there's no scp available (stage1), you can (after setting up the network if not up already) redirect to a special tcp file, on host:

nc -l 4444 > syslog.log

on guest:

ifconfig eth0 10.0.2.10/24 up
grep "" /tmp/syslog > /dev/tcp/10.0.2.2/4444

To do

  • The current list of logging requirements and tasks is maintained in bugzilla 524980.
  • A support for KVM's virtio logging is coming later 576439.