QA Special Interest Group

Mission

The purpose of the Fedora QA SIG is to monitor the many aspects of the life cycle of a package in Fedora.

Packagers/Reviewers/People interested

ChristianIseli
AurelienBompard
ToshioKuratomi

Several processes which need QA

Here is an attempt to outline some QA tasks that need to be performed on a regular basis. I guess it'd be nice if some people could agree to watch after one or more on a regular basis.

Package review process

Many aspects of the package review process are now checked by the parseBZbugList Perl script. Basically, the script examines BZ tickets filled for "Package Review" and reports anomalies and stagnation.

Things that need improvement:

It would be useful to get the script to send email notices and/or add BZ comments for some of the problems it discovers. At the moment, Christian does it manually (and sporadically).
Some janitorial work is needed to remove closed blockers for NEW and REVIEW trackers.

New package inclusion process

The parseBZbugList also does some checking here, about owners.list inclusion and checking that the package actually shows up in the repo. More checks should be done, though:

check for inclusion in comps.xml
check that BZ tickets are filed for ExcludeArch packages

Package build process

This is a copy of Bill's message... I don't think we have anything in place here.

The build system should have a framework where simple post-build checks can run. Simple design would be:

tests are collected in a package of scripts (or in CVS)
packagers can add extra ones for their packages
tests have access to:
the build logs
the name of the user who submitted the build
the owner of the package (if not the same)
the previous version of the package
the repodata for the repo being built for
tests have a defined return status, such as
0 - everything's OK. Carry on, sir!
1 - some information here
7 - someone should review this and approve before pushing the package
11 - the package is bad, please throw away

These tests could then be:

e-mailed to the builder
e-mailed to the owner
posted on the web

Doing this allows for simple, automated, QA; it can check that things that are caught in the package review for initial import are then caught later if they regress, and it can catch other things.

Examples of tests:

test for files owned by the compile user
test for files with executable stack
look for specific compiler warnings in the build output, such as:
... will always overflow destination buffer ...
... warning: format argument XX unused before used argument XX ...
*** buffer overflow detected ***
... warning: ignoring return value of 'realloc' ...
check for introduced multilib conflicts
check the scriptlets for:
correct use of chkconfig
correct use of useradd or equivalents
excess complication (warning: package %post is 150 lines)
using the repo data:
check the package for broken deps
check the repository for induced breakage
using previous version of the same package:
check for added/removed files
check for added/removed/changed dependencies
check for introduction of new setuid files
check that symbols changed in libraries without changing soname
heck, just run the package through rpmlint and mail it out (or check it against a known list of exceptions for the package)

I'm sure people could come up with a lot more. By making it a simple framework, it allows for easier contribution.

Package sign and push to repo process

Michael has a nice script going (seems to live at http://home.arcor.de/ms2002sep/tmp/ under the name repoclosure...)

My understanding is that it will soon be run automatically after signing and pushing new packages.

I don't think it automatically files BZ tickets for offending packages (but I could be wrong...)

Fedora release time process

Several ideas from Thorsten:

Each time close before a new Fedora Core ships do exactly what I and some others have done in the past weeks for FE5. E.g. look at the repo for packages that might be orphaned and get them removed. Look out for packages that were not rebuild after a mass rebuild was announced and poke maintainers. Look out for packagers that vanished without telling us.
Look at the repo for packages that are useless. Example: Look at FE5 repo and sort it by date and just look at the first packages http://www.fedoraproject.org/extras/5/i386/?C=M;O=A On top is epydoc-2.1-2.i386.rpm -- seems to be obsoleted by epydoc-2.1-4.noarch.rpm , but didn't get removed. Hermes-1.3.3-6.i386.rpm -- not rebuild for 1 1/4 years? Wow. pyzor-0.4.0-9.fc4.noarch.rpm -- a package with the disttag "fc4" in the final FE5 repo? That confusing. Ohh, there are some more of these :-/

Package transfer between Core and Extras process

The parseBZbugList script does some sanity checks.

Package orphaning process

All goes rather OK when a maintainer declares a package orphaned, but something needs to be defined when a maintainer kinda disappears. The parseBZbugList script attempts to spot inactive maintainers, but I'm not sure it is very successful.

BZ ticket handling process

The parseBZbugList script tries to detect inactive tickets. Request for enhancements are ignored at this point. Security related issues should be watched more closely (no special handling ATM).

Package maintenance process

All CVS messages go through the commits list, and I know some people keep an eye on things there. No real idea if anything could be automated here...

Package retirement process

At the moment, put them on the ["Extras/PackagesNoLongerInDevel"] page. Maybe long-time orphaned packages should be migrated there with a proper notice...

Ideas/Proposals

Engage upstream developers to become involved with their package in Fedora

Search

SIGs/QA

Contents