KernelTriage

DRAFT
This page is a draft, don't depend on any information here yet.

Introduction

The Linux kernel is a large and complex software project, and thus gets a lot of bug reports in the Fedora bugzilla instance. This effort attempts to triage these bugs to allow Fedora kernel maintainers to focus on real bugs and issues.

The Fedora kernel team is intending to have once a month triage days. This page documents some of the things that could be done to help.

As the kernel gets more bugs than any other package in the distro, the maintainers would like to keep bug mail to a minimum, so when taking on some of the tasks below, try to do them as 'atomic operations', rather than as several separate commits.

NEW bugs

When a new bug is filed, the bugzapper kernel team should check it against the following list for actions.

1. If it seems relevant, ask for the following information (if not yet provided):
   • uname -a
   • dmesg output
   • smolt profile (optional)
   • lsmod
     DO NOT ask for all this information on every single bug. Adding unnecessary information just makes the bug harder to parse. If we have a backtrace, we usually don't need much else from this list.
2. If the reporter is not running the most current kernel update, ask them to duplicate with the newest kernel version available if possible. Note also there may be newer kernels available in koji directly.
3. If the report has a kernel OOPS that shows that their kernel is "tainted", parse the flags that follow it.
   • If there is a P flag, tell the reporter that we cannot fix problems reported with binary modules loaded, and CLOSED->CANTFIX the bug.
   • TODO: Decide what we want to do with other flags.
   • If an oops isn't tainted, but shows vboxdrv in the modules list, ask the user to try and reproduce the bug without it having been loaded. VirtualBox is opensource, but historically has caused all sorts of problems that Fedora does not have the resources to fix. If the user cannot (or will not) reproduce the problem without Vbox, then advise them to take the bug to the upstream VirtualBox developers.
4. Note to the reporter the KernelCommonProblems page.
5. If this is a feature request, point reporter upstream (also CLOSED->WONTFIX?)
6. If the bug contains a patch, add "[PATCH]" to the subject. If the patch is not a backport of upstream, ask them to submit upstream.
7. Don't bother setting the Severity field. For as long as it's a user-settable field, it's useless and is ignored.
8. If this is a regression from a previous Fedora kernel, add a 'regression' keyword. NOTE what regression means. ;)
9. If the bug is pertaining to a config option that should be enabled/disabled/made modular, add 'kernel_config' keyword.
10. The arch of the bug should be noted. If it's a secondary Fedora arch, the secondary arch team should be added to CC and/or a keyword added.
11. The High level subsystem the bug is involved in should be noted and a Keyword added for it, as well as possibly a CC to subsystem maintainers.
    • Where possible, use the module name of the driver (for example e1000e.ko bugs get tagged 'e1000e').
    • If the module name isn't obvious, do nothing.
12. Note that for the following subsystems, the bug should be switched from 'kernel' to the component within brackets:
    • AMD/ATI Radeon graphics (xorg-x11-drv-ati)
    • Intel graphics (xorg-x11-drv-intel)
    • Nvidia graphics (xorg-x11-drv-nouveau)
13. Abrt bug cleanup: Abrt has a number of problems in its automatic filing. (bugs have been filed that will hopefully get them fixed soon).
    • It always reports 'TAINTED' in the summary. If the first trace in the report has 'Not tainted', then clear the TAINTED part of the summary (and the flags or 'WARNING ISSUED' type messages that follow)
    • Remove the 'kernel: ' part of the summary while you're at it. We already know it's a kernel bug.
    • For oopses and similar traces, knowing where it happened is valuable info in the summary for looking for possible dupes. On 32 bit this is the important part of the trace:
      :IP: [<c05e48d5>] __percpu_counter_add+0xe/0x6d
      The address (<c05e48d5>) isn't important (and may change from build to build). Add the rest to the summary.
    • Using https://bugzilla.redhat.com/show_bug.cgi?id=722827 as an example. abrt filed the bug as:
      [abrt] kernel: BUG: unable to handle kernel paging request at 33e02000: TAINTED Warning Issued
      cleaned up, this would look like:
      [abrt] BUG: unable to handle kernel paging request at 33e02000 (IP: __percpu_counter_add+0xe/0x6d)
    • Finally, the "Triaged" Keyword should be added.
    • There is also a lot more information on https://fedoraproject.org/wiki/KernelBugTriage. In particular, note the 'bug assignment' table. Add Cc's where necessary.

ASSIGNED / MODIFIED bugs

• Let subsystem maintainer handle the bug.

• If reporter is impatient, ask them to report upstream in the kernel bugtracker or kernel list.

• When new kernels are released, optionally ask reporters to try them and duplicate.

• If no answer from reporter X time, close bug INSUFFICIENT_INFORMATION