FWN/Beats/QualityAssurance

QualityAssurance

In this section, we cover the activities of the QA team.

Contributing Writer: Adam Williamson

Test Days

There was no Test Day last week, and no Test Day is currently planned for this week. If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac.

Weekly meetings

The QA group weekly meeting was held on 2009-11-23. The full log is available. James Laska noted that a common bugs page entry had been added to cover the known issue with preupgrade and free space in the /boot partition, and Rui He had been working to update the preupgrade test cases to catch similar problems in future.

James Laska admitted that he had not yet sent out the request for feedback for the Fedora 12 QA retrospective, but promised to do it soon. John Poelstra asked whether the group would be interested in a project-wide retrospective at the upcoming FUDCon; James offered to discuss the idea with John after the meeting.

The group discussed the question of privilege escalation testing, following the PackageKit installation permission controversy. James Laska wanted to discuss the plan Tom 'spot' Callaway had proposed via a blog post and create a test plan based around it. Adam Williamson felt it was too early to begin planning testing, since Tom's blog post was only a proposal, and there was no official policy or guideline for privilege escalation issues on which a test plan could be based. Adam was also worried about defining the scope of testing, as checking every package in the distribution would be impractical given the size of the QA team. The group agreed that for any useful testing to be done, two things would be needed: a project-wide policy or set of policies and guidelines, and a tool for generating a list of packages which are capable of privilege escalation. Adam agreed to start a discussion of this on the development and security mailing lists. Will Woods offered to work on the tool for identifying escalation-capable packages.

James Laska brought up John Poelstra's plan to improve the release criteria, and asked the group to provide feedback. John noted that he was hoping people could get together to work on finalizing the new criteria at FUDCon.

Will Woods and Kamil Paral reported on the progress of the AutoQA project. Will had completed the redesign of the autoqa code to be based around a Python shared library containing functions commonly used in multiple watchers and tests. The new post-koji-build test hook is also included, and autoqa is currently running an rpmlint test on every Koji build to test the hook. He said the next objective was to solidify the post-koji-build hook, help package maintainers add post-build tests, and get the rpmguard test running. A later objective is to work on a post-bodhi-update hook and dependency check test so that all updates submitted to Bodhi will be checked for dependency consistency, to hopefully end the situation where updates are pushed which break dependency chains. Kamil had been working on the Wiki documentation, and had created a new front page which briefly explains the project and contains links to the most important relevant pages. He also pointed out that James Laska had been drafting further improvements to this page.

Jesse Keating proposed a talk during FUDCon to explain how several new ideas across the release engineering and QA groups - no frozen rawhide, autoqa, autosigning, and new milestones - would fit together in upcoming Fedora release cycles. The group thought this was a good idea, and Jesse said he would take the lead in arranging it.

The Bugzappers group weekly meeting was held on 2009-11-24. The full log is available. The group discussed housekeeping tasks, particularly updating the components and triagers page. Adam Williamson thought the list of triagers should be kept (rather than being emptied as was previously the case with each new release) but pruned, with triagers known to be inactive being removed. Edward Kirk volunteered to look into a method for updating the component list, based on the current critical path package list.

The group then discussed the topic of mentoring new members, with Edward Kirk encouraging experienced group members to help mentor new ones to make sure they got a good start on their triaging careers. He also thought it would be good for existing members to join in welcoming new members to the group when they posted their introduction emails. Adam Williamson suggested doing this via private mail to avoid cluttering up the list.

Matej Cepl brought up a problem related to the recently-implemented change in the method of marking bugs that had been triaged. He had found that the fact that this was now being done differently for different releases made it impossible to construct a Bugzilla search for all triaged or un-triaged bugs in a given component across all releases. To address this problem, he proposed adding the new Triaged keyword to all bugs in ASSIGNED state for existing supported releases (Fedora 10 through 12), which would allow searches to be performed using the keyword in all releases. The group could see no problems with this idea, as long as it was done without generating a large amount of email, and approved the plan for Matej to approach the Bugzilla maintainer for help in implementing it.

Matej Cepl pointed out that the level of duplicate bugs being filed via the abrt automated bug reporting tool was increasing the triage workload on some components significantly. After a long discussion, the group agreed a plan to try and address this. Will Woods would talk to the abrt team about the idea of reporting issues to an intermediate, abrt-specific server rather than directly to Bugzilla, based on the kerneloops.org model. Matej would talk to the abrt team about their plans to improve abrt's own automatic duplicate detection and about having abrt format its reports in ways that would aid triagers in manual duplicate detection. Adam Williamson would respond to the existing thread on the development mailing list about the problem to raise the group's concerns, and ask the abrt team whether future improvements to abrt's duplicate detection logic could be retrospectively applied to bugs already filed by older versions of abrt.

The next QA weekly meeting will be held on 2009-11-30 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-12-01 at 1500 UTC in #fedora-meeting.

Increasing the grub timeout

Scott Robbins started a long thread with the suggestion to increase the default timeout for the Fedora boot loader from its current default setting of 0 (which causes the boot loader menu never to be shown at all). There were many opinions on this idea, but the general response was positive enough for Scott to file a feature request on the idea, where some compromises were suggested. Richard Ryniker suggested having the system detect unclean shutdowns and force the boot menu to be displayed on the next boot (much as Windows does). Stewart Adam suggested having grub initially installed with a non-zero timeout, and have firstboot change it to zero on the assumption that a system that can get to firstboot must have a properly configured bootloader.

Fedora 12 QA retrospective

James Laska posted a request for feedback on the Fedora 12 QA cycle from anyone, both on things that went well and areas that could be improved. Many group members posted replies, including Adam Williamson, Jóhann Guðmundsson, and Rahul Sundaram.