Features/RetraceServer

Retrace Server

Summary

Retrace server allows ABRT users to get better backtraces from their crashes by retracing coredumps remotely, on a server owned by Fedora Project.

Owner

• Name: Michal Toman

• Email: mtoman@redhat.com

Co-workers

• The ABRT team
• E-mail: crash-catcher@lists.fedoraproject.org
• Name: Karel Klic
• Name: Jiri Moskovcak

Current Status

• Targeted release: Fedora 15
• Last updated: 2011-02-21
• Percentage of completion: 95%

Detailed description

When ABRT generates a backtrace from a coredump, it needs debuginfo data to be available for the binary and all libraries involved in the crash. Debuginfo packages require a lot of storage space, and sometimes they are not available at all -- a package update causes the removal of older updates of the same package and their debuginfo.

Another problem is that GDB (which generates the backtrace from the coredump) needs data from the binary and libraries that were involved in the crash. If a user updated some relevant package between observing the crash and reporting it, he might be unable to generate a good quality backtrace because of the updates. This happens often, because Fedora is updated frequently.

Retrace server is one possibility to solve these issues. ABRT offers user to upload her coredump to a remote server, then the retracing step happens there. The server creates an environment identical to what was on the user's computer at the time of the crash, by installing all the required packages and their debuginfo. The retrace server is able to do that because it keeps all the older packages from updates, and relevant part of updates-testing locally on the server.

After creating the backtrace, only the submitter is allowed to download/view it.

Benefit to Fedora

Users:

1. Less disk space and processing time needed to use ABRT to report crashes.
2. Possibility to report older crashes.
3. Lower chance of failure, less time spent on crashes which cannot be retraced because debuginfo is no longer available.

Developers:

1. Higher quality of ABRT reports.
2. Possibility to quickly get a backtrace from any/random Fedora coredump.

Scope

ABRT is extended to support Retrace Server.

Retrace server is installed on https://retrace01.fedoraproject.org.

The server implementation consists of three parts:

• HTTP Interface: Receives the archive with a coredump from user, unpacks all files, puts new task into the queue.
• Analyzer: Takes a task from queue, creates a virtual root with all required packages and debuginfos installed and runs GDB to create the backtrace.
• Repository Synchronizer: Downloads packages to a local repository containing all versions of all packages (no removal of older updates).

How To Test

Testing will be possible using a new version ABRT with the retrace server support, which will land in Rawhide when stabilised.

At the moment, testing is possible using the upload.py script from the retrace branch of ABRT Git repository. The script is only designed for test purposes and does not handle all possible errors.

Usage

The script upload.py takes the following two arguments:

ABRT crash directory (mandatory)

A directory containing the ABRT crash information. By default, the directory can be found as a sub-directory of /var/spool/abrt.

Retrace server (optional)

Retrace server address. Not important at the moment since there is only one test machine running retrace server: retrace01.fedoraproject.org.

Running upload.py with the no arguments will display the following help message.

$ python ./upload.py
ABRT Retrace Uploader
For test purposes only
Usage: 'upload.py crash_directory server_address'
  Crash directory is the directory created by ABRT (default /var/spool/abrt/crash_directory/).
  Crash directory must contain analyzer, architecture, coredump, executable, package, release and uid files.
  If no server address is specified, default testing machine is used.
  Only binary crashes (caught by CCpp) need retrace.
  The script shows raw HTTP output including X-Task-Id and X-Task-Password headers.
    The tester is supposed to know what he's uploading and should handle task id and password on his own.

The retrace server at retrace01.fedoraproject.org is configured to handle crashes for the following releases:

• Fedora 13 x86_64
• Fedora 14 i686
• Fedora 14 x86_64

After successfully running the upload.py script, it will display a raw HTTP response containing X-Task-Id and X-Task-Password headers. Sample output is shown below:

[raw data]
---------
Connecting to retrace server @ ssl://retrace01.fedoraproject.org:443... OK
Sending request... OK
Receiving response... OK
----------
HTTP/1.1 201 Created
Date: Thu, 03 Mar 2011 12:32:21 GMT
Server: Apache/2.2.15 (Red Hat)
Content-Length: 0
X-Task-Id: 756381245
X-Task-Password: 72MOLQD3s3AI3EpcLibzUGsLtOuubLxf
AppTime: D=4891493
AppServer: retrace01.fedoraproject.org
Connection: close
Content-Type: text/plain

Using the information returned by upload.py, you can query the retrace server for the following crash information using the wget command.

Status

HTTP response contains X-Task-Status header with one of three values 'PENDING', 'FINISHED_SUCCESS', 'FINISHED_FAILURE'. The command wget may be used to show the output (the HTTP response body is the same as X-Task-Status header's value):

wget -S --no-check-certificate -O /dev/null --header="X-Task-Password: <Task-Password>" "https://retrace01.fedoraproject.org/<Task-Id>/"

Log

After retrace is finished (FINISHED_SUCCESS or FINISHED_FAILURE) the log is available. The wget command may be used to download the log:

 wget -S --no-check-certificate --header="X-Task-Password: <Task-Password>" "https://retrace01.fedoraproject.org/<Task-Id>/log"

Backtrace

After successful retrace (FINISHED_SUCCESS) the backtrace is available. Once again, the wget command may be used to download the backtrace:

wget -S --no-check-certificate --header="X-Task-Password: <Task-Password>" "https://retrace01.fedoraproject.org/<Task-Id>/backtrace"

User Experience

ABRT reporting wizard offers a possibility to use retrace server instead of local retracing to generate a backtrace. Local retracing is still the default action.

Dependencies

• new version of ABRT, supporting the retrace server option
• a server with the retrace server application up and running

Server code dependencies:

• mock
• xz
• elfutils
• createrepo
• mod_ssl
• mod_wsgi
• python-webob

Contingency Plan

Hide the possibility of using the retrace server from users in ABRT's graphical user interface.

Documentation

• Design document
• Retrace server wiki page on ABRT wiki

Release Notes

ABRT, a crash reporting tool in Fedora, now allows to prepare a part of crash processing remotely, on a server owned by Fedora Project. Remote coredump retracing leads to better quality of reports. Retrace server can generate good backtraces with much higher success rate than local retracing.

Comments and Discussion

• See Talk:Features/RetraceServer