FWN/Beats/QualityAssurance: Difference between revisions

Revision as of 20:59, 16 November 2009

QualityAssurance

In this section, we cover the activities of the QA team^[1].

↑ http://fedoraproject.org/wiki/QA

Test Days

There have been no Test Days for the last two weeks, due to the pressures of the Fedora 12 release.

No Test Day is currently planned for this week. If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac^[1].

↑ http://fedorahosted.org/fedora-qa/

Weekly meetings

As the QA beat was unfortunately not present for Fedora Weekly News #201, we will cover two weeks' worth of events below.

QA group weekly meetings^[1] were held on 2009-11-02 and 2009-11-09. The full logs are available^[2], ^[3].

During the meeting of 2009-11-02, Adam Williamson reported that Milos Jakubicek had not yet followed up on his idea regarding an event to work on FTBFS problems. Jóhann Guðmundsson was also not present to report on his work on creating a test case for keyboard layout issues^[4]. Will Woods was making good progress on the action items for AutoQA from the previous meeting.

The group reviewed the status of the Fedora 12 code base with regards to the then-impeding release candidate phase. It was generally agreed that the status was promising and it should be possible to make the release candidate phase on time, based on a review of the blocker bug list.

Will Woods and Kamil Paral reported on the progress of the AutoQA project. Will had been working on revising the AutoQA code to provide a Python library interface^[5]. He had been moving all shared or potentially shareable code from all current AutoQA tests into the library. He hoped to have it merged into the master branch by the end of the week. He also noted that a newer version of autotest was currently being packaged and implemented into the AutoQA system, which may cause strange results if any bugs emerged.

Adam Williamson reviewed upcoming events. The release candidate date was Wednesday 2009-11-04 and the go/no-go date Monday 2009-11-09. Jesse Keating clarified that for an RC build to be done, the blocker bug list must be clear, but new issues that emerged during RC compose and testing could be resolved up until the date of the go/no-go meeting. James Laska promised to co-ordinate with the anaconda team to ensure there were no remaining blocker issues in installation.

The meeting of 2009-11-09 was held during the final run-up to the Fedora 12 go/no-go meeting, so there was some last-minute blocker bug discussion. Jóhann Guðmundsson had not yet been able to work on creating a test case for keyboard layout issues^[6]. James Laska had followed up with the anaconda team and verified no blocker bugs remained in the installation process. Adam Williamson noted that one of the anaconda bugs that definitely wasn't left had been fixed the previous day.

Will Woods and Kamil Paral reported on the progress of the AutoQA project. Kamil had added a check to rpmguard for the case where an old version of rpmdiff is installed. Will had the new python library ready but wanted more testing before merging it into master. The new Koji watcher (for running AutoQA tests on new builds as they hit Koji) was now functional.

James Laska pointed out that Matthias Clasen had asked the group to test Fedora 12 0-day updates by enabling the Fedora 12 updates and possibly updates-testing repositories and updating their systems. James thought it would be a good idea to create a test case for testing the update repositories for a release before they were generally enabled.

A Bugzappers group weekly meeting^[7] was held on 2009-11-03. No meeting was held on 2009-11-10. The full log is available^[8]. Richard June reported that he was continuing to work on kernel triage. He had not been in touch with Jeff Hann regarding his volunteering to help out yet, but would attempt to co-ordinate via the mailing list.

Adam Williamson asked if anyone had concerns about unaddressed issues for the Fedora 12 release, and no-one did. Adam asked Matej Cepl how he was coping with X.org triage while François Cami was mostly unable to help, and he said it was difficult to stay on top of the large number of bugs. Adam promised to continue to try and manage the nouveau driver bugs, and Thomas Janssen volunteered to help out with others. Matej said he would work with Thomas to bring him up to speed on X triaging.

Edward Kirk reported that he had worked on outstanding Fedora 10 bugs, and managed to update some and close others. He also reported that the maintainer warning email for the upcoming Fedora 12 housekeeping Bugzilla changes had been sent out. John Poelstra was ready to do the Rawhide bug rebase (moving open Rawhide bugs to Fedora 12) and Fedora 10 bug end-of-life warning operations.

The next QA weekly meeting will be held on 2009-11-16 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-11-17 at 1500 UTC in #fedora-meeting.

Fedora 12 release

Much of the group activity in the last two weeks was centred around Fedora 12 release testing and validation. James Laska posted a recap^[1] of the final blocker bug review meeting for Fedora 12. James, Adam Williamson and Will Woods worked with the release engineering team throughout the final week before release to continuously monitor blocker bug status, test fixes, and monitor for newly identified blocker bugs and regressions.

Four release candidate builds were produced by the release engineering group with the help of testing and feedback from QA. Liam Li co-ordinated the planned installation testing through the test compose and release candidate process^[2] ^[3], and maintained the test results matrix^[4]. He also sent a post mortem on the testing process^[5]. Many members of the group contributed valuable test reports to the matrix. Several group members, including Gianluca Cecchi, Gene Czarcinski and others posted test installation reports which helped identify important issues that were fixed or documented during the release process.

[1] ttp://fedoraproject.org/wiki/QA

[2] ttp://fedorahosted.org/fedora-qa/

[3] ttp://fedoraproject.org/wiki/QA/Meetings

[4] ttp://meetbot.fedoraproject.org/fedora-meeting/2009-11-02/fedora-meeting.2009-11-02-16.01.log.html

[5] ttp://meetbot.fedoraproject.org/fedora-meeting/2009-11-09/fedora-meeting.2009-11-09-16.00.log.html

[6] ttp://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=530452

[7] ttp://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=lib/python;hb=wwoods-autotest

[8] ttp://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=530452

[9] ttp://fedoraproject.org/wiki/BugZappers/Meetings

[10] ttp://meetbot.fedoraproject.org/fedora-meeting/2009-11-03/fedora-meeting.2009-11-03-15.03.log.html

[11] ttp://www.redhat.com/archives/fedora-test-list/2009-November/msg00266.html

[12] ttp://www.redhat.com/archives/fedora-test-list/2009-November/msg00080.html

[13] ttp://www.redhat.com/archives/fedora-test-list/2009-November/msg00235.html

[14] ttp://fedoraproject.org/wiki/Test_Results:Fedora_12_RC4_Install

[15] ttp://www.redhat.com/archives/fedora-test-list/2009-November/msg00585.html

[1]

[1]

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[1]

[2]

[3]

[4]

[5]

@@ Line 10: / Line 10: @@
 === Test Days ===
-Last week's Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2009-10-29</ref> was on internationalization (also known as i18n)<ref>http://fedoraproject.org/wiki/I18N</ref>. We had a good turnout of testers who covered a wide variety of languages and input methods. In general many appear to be in good shape, but the testing turned up several issues in Bengali, Malayalam and a few other languages. This testing will help us to improve the implementation of these languages in future. [[User:Rhe|Rui He]] provided a summary<ref>https://www.redhat.com/archives/fedora-test-list/2009-November/msg00038.html</ref> of the event, including a list of all bugs filed.
+There have been no Test Days for the last two weeks, due to the pressures of the Fedora 12 release.
-No Test Day is planned for next week. If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.
+No Test Day is currently planned for this week. If you would like to propose a main track Test Day for the Fedora 13 cycle, please contact the QA team via email or IRC, or file a ticket in QA Trac<ref>http://fedorahosted.org/fedora-qa/</ref>.
 <references/>
@@ Line 18: / Line 18: @@
 === Weekly meetings ===
-The QA group weekly meeting<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> was held on 2009-10-26. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-10-26/fedora-meeting.2009-10-26-16.08.log.html</ref>. [[User:Jlaska|James Laska]] reported that he had renamed most of the Debugging pages<ref>http://fedoraproject.org/wiki/Category:Debugging</ref> to follow the previously agreed-upon naming scheme. The only remaining page was KernelBugTriage, and he would check with kernel maintainers before renaming this one.
+As the QA beat was unfortunately not present for Fedora Weekly News #201, we will cover two weeks' worth of events below.
-[[User:Jlaska|James Laska]] noted that Marcela Maslanova had written automated testing scripts for the previous week's Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2009-10-22</ref>, and this had produced a very positive experience. He asked the group to think about what future Test Days could potentially benefit from testing automation in this way.
+QA group weekly meetings<ref>http://fedoraproject.org/wiki/QA/Meetings</ref> were held on 2009-11-02 and 2009-11-09. The full logs are available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-02/fedora-meeting.2009-11-02-16.01.log.html</ref>, <ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-09/fedora-meeting.2009-11-09-16.00.log.html</ref>.
-[[User:Adamwill|Adam Williamson]] passed along a proposal from Milos Jakubicek that the QA and BugZappers group help with filing bugs on the remaining Fedora 12 packages with FTBFS (fails to build from scratch) issues. [[User:jkeating|Jesse Keating]] pointed out that [[User:Mdomsch|Matt Domsch]] has a script which tries to rebuild all of Rawhide and automatically files bugs on packages which fail, which he typically runs once per cycle. Jesse believed the fact that Milos is aware of several packages which fail to build but for which no bug report currently exists is a result of the fact that the list Milos is working from was generated a month after Matt's latest test run. The group agreed that Adam would ask Milos to clarify his proposal and see if it was still necessary in light of the existence of Matt's script.
+During the meeting of 2009-11-02, [[User:Adamwill|Adam Williamson]] reported that Milos Jakubicek had not yet followed up on his idea regarding an event to work on FTBFS problems. [[User:Johannbg|Jóhann Guðmundsson]] was also not present to report on his work on creating a test case for keyboard layout issues<ref>http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=530452</ref>. [[User:Wwoods|Will Woods]] was making good progress on the action items for AutoQA from the previous meeting.
-[[User:Johannbg|Jóhann Guðmundsson]] presented his proposal for an automated test of non-U.S. locale installation, prompted by the significant bugs<ref>http://bugzilla.redhat.com/show_bug.cgi?id=528317</ref> <ref>http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=530452</ref> in the Beta with installations with different locale settings which were not caught by pre-release testing. He pointed out that implementing such a test would be relatively simple and involve only defining a non-U.S. locale in a kickstart file for an installation test run. The group agreed that this would be valuable testing and asked Jóhann to write it up into a test case that could be added to the installation test matrix and also potentially automated as part of future AutoQA development.
+The group reviewed the status of the Fedora 12 code base with regards to the then-impeding release candidate phase. It was generally agreed that the status was promising and it should be possible to make the release candidate phase on time, based on a review of the blocker bug list.
-[[User:Wwoods|Will Woods]] and [[User:Kparal|Kamil Paral]] reported on the progress of the AutoQA project. Kamil had made a blog post announcing rpmguard to the world<ref>http://kparal.wordpress.com/2009/10/21/rpmguard-print-important-differences-between-rpms</ref>. He had received feedback from several people, including suggestions from [[User:Skvidal|Seth Vidal]] and [[User:atorkhov|Alexey Torkhov]] (whose feedback had prompted a ticket<ref>http://fedorahosted.org/autoqa/ticket/75</ref>). Kamil is now planning to work on integrating rpmguard into AutoQA with the help of the newly-implemented Koji watcher, which allows AutoQA to pick up - and potentially trigger tests upon - every new build which goes through Koji. Will briefly touched upon the future organization plan for all the AutoQA code, based around a library for the server-side parts such as watchers and another library for actual tests, along with separate configuration files for things like the relationships between Koji tags, so these configuration details can be separated from the main functional code. Will also noted that he had created a Python script for generating the current set of critical path packages<ref>http://wwoods.fedorapeople.org/files/critical-path/critpath.py</ref>: simply running it generates the list as critpath.txt. He plans to have this integrated into the Rawhide compose process so that a daily updated critical path package list is always available at a static URL. Finally, Will noted that a public mailing list has been created for the AutoQA project, autoqa-devel<ref>http://fedorahosted.org/mailman/listinfo/autoqa-devel</ref>. [[User:Jlaska|James Laska]] noted in passing that the hardware for the production AutoQA instance was currently likely to be delivered on 2009-11-20.
+[[User:Wwoods|Will Woods]] and [[User:Kparal|Kamil Paral]] reported on the progress of the AutoQA project. Will had been working on revising the AutoQA code to provide a Python library interface<ref>http://git.fedorahosted.org/git/?p=autoqa.git;a=tree;f=lib/python;hb=wwoods-autotest</ref>. He had been moving all shared or potentially shareable code from all current AutoQA tests into the library. He hoped to have it merged into the master branch by the end of the week. He also noted that a newer version of autotest was currently being packaged and implemented into the AutoQA system, which may cause strange results if any bugs emerged.
-[[User:Jlaska|James Laska]] reviewed upcoming events. He noted that preparation for the then-upcoming i18n Test Day<ref>http://fedoraproject.org/wiki/Test_Day:2009-10-29</ref> was well advanced, and asked for group members to help out with testing if they could. He trailed the then-upcoming second Fedora 12 blocker bug review day, which would take place on 2009-10-30, and [[User:Adamwill|Adam Williamson]] asked people to help by re-testing blocker bugs prior to the event and coming to the event to help walk the list.
+[[User:Adamwill|Adam Williamson]] reviewed upcoming events. The release candidate date was Wednesday 2009-11-04 and the go/no-go date Monday 2009-11-09. [[User:jkeating|Jesse Keating]] clarified that for an RC build to be done, the blocker bug list must be clear, but new issues that emerged during RC compose and testing could be resolved up until the date of the go/no-go meeting. [[User:Jlaska|James Laska]] promised to co-ordinate with the anaconda team to ensure there were no remaining blocker issues in installation.
-The Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-10-27. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-10-27/fedora-meeting.2009-10-27-15.09.log.html</ref>. [[User:Tk009|Edward Kirk]] asked if there was a firm date yet set for the semantics switchover (marking triaged bugs as NEW with the Triaged keyword rather than ASSIGNED). [[User:Adamwill|Adam Williamson]] looked at the schedule and noted it should be around 2009-11-12 if no further schedule changes occurred.
+The meeting of 2009-11-09 was held during the final run-up to the Fedora 12 go/no-go meeting, so there was some last-minute blocker bug discussion. [[User:Johannbg|Jóhann Guðmundsson]] had not yet been able to work on creating a test case for keyboard layout issues<ref>http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=530452</ref>. [[User:Jlaska|James Laska]] had followed up with the anaconda team and verified no blocker bugs remained in the installation process. [[User:Adamwill|Adam Williamson]] noted that one of the anaconda bugs that definitely wasn't left had been fixed the previous day.
-No-one had heard from Brennan Ashton regarding his promised summary of the status of the triage metrics project.
+[[User:Wwoods|Will Woods]] and [[User:Kparal|Kamil Paral]] reported on the progress of the AutoQA project. Kamil had added a check to rpmguard for the case where an old version of rpmdiff is installed. Will had the new python library ready but wanted more testing before merging it into master. The new Koji watcher (for running AutoQA tests on new builds as they hit Koji) was now functional.
-[[User:Tk009|Edward Kirk]] wondered if the bug workflow page and diagram<ref>http://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow</ref> would require updating when the semantics change occurred. [[User:Adamwill|Adam Williamson]] believed it would, but the necessary changes would be quite minor. Edward and Adam agreed to keep the necessary changes in mind for the meeting prior to the semantics change.
+[[User:Jlaska|James Laska]] pointed out that [[MatthiasClasen|Matthias Clasen]] had asked the group to test Fedora 12 0-day updates by enabling the Fedora 12 updates and possibly updates-testing repositories and updating their systems. James thought it would be a good idea to create a test case for testing the update repositories for a release before they were generally enabled.
-[[User:Tk009|Edward Kirk]] promised to make sure the email warning developers that the regular housekeeping changes in Bugzilla at release time would be coming soon.
+A Bugzappers group weekly meeting<ref>http://fedoraproject.org/wiki/BugZappers/Meetings</ref> was held on 2009-11-03. No meeting was held on 2009-11-10. The full log is available<ref>http://meetbot.fedoraproject.org/fedora-meeting/2009-11-03/fedora-meeting.2009-11-03-15.03.log.html</ref>. [[User:Rjune|Richard June]] reported that he was continuing to work on kernel triage. He had not been in touch with Jeff Hann regarding his volunteering to help out yet, but would attempt to co-ordinate via the mailing list.
-The next QA weekly meeting will be held on 2009-11-02 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-11-03 at 1500 UTC in #fedora-meeting. Note that the meeting times in UTC do not change even though many countries are going through daylight savings time changes around this time of year, with the result that the meetings will be one hour earlier for many people in practice.
+[[User:Adamwill|Adam Williamson]] asked if anyone had concerns about unaddressed issues for the Fedora 12 release, and no-one did. Adam asked [[User:Mcepl|Matej Cepl]] how he was coping with X.org triage while [[User:Fcami|François Cami]] was mostly unable to help, and he said it was difficult to stay on top of the large number of bugs. Adam promised to continue to try and manage the nouveau driver bugs, and [[user:thomasj|Thomas Janssen]] volunteered to help out with others. Matej said he would work with Thomas to bring him up to speed on X triaging.
-<references/>
+[[User:Tk009|Edward Kirk]] reported that he had worked on outstanding Fedora 10 bugs, and managed to update some and close others. He also reported that the maintainer warning email for the upcoming Fedora 12 housekeeping Bugzilla changes had been sent out. [[User:poelstra|John Poelstra]] was ready to do the Rawhide bug rebase (moving open Rawhide bugs to Fedora 12) and Fedora 10 bug end-of-life warning operations.
-=== Fedora 12 testing ===
+The next QA weekly meeting will be held on 2009-11-16 at 1600 UTC in #fedora-meeting, and the next Bugzappers weekly meeting on 2009-11-17 at 1500 UTC in #fedora-meeting.
-Much of the week's mailing list activity centred on testing the Fedora 12 Beta and post-beta updates, with much valuable testing being performed by many volunteers. [[User:Adamwill|Adam Williamson]] asked<ref>http://www.redhat.com/archives/fedora-test-list/2009-October/msg00650.html</ref> for group members to provide feedback on the latest accepted kernel build, which had incorporated several changes from the kernel shipped in the Beta release. Many testers replied with helpful confirmation that the new kernel worked well. [[User:Liam|Liam Li]] announced<ref>http://www.redhat.com/archives/fedora-test-list/2009-October/msg00721.html</ref> the pre-RC install testing cycle and associated test matrix<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_12_Pre-RC_Install</ref>, asking group members to try and cover as much of the install test case set as possible before the release candidate phase began on 2009-11-04; he later provided a report<ref>http://www.redhat.com/archives/fedora-test-list/2009-October/msg00816.html</ref> on this testing. Adam requested testing<ref>http://www.redhat.com/archives/fedora-test-list/2009-October/msg00805.html</ref> of an ext4 data corruption issue<ref>http://bugzilla.kernel.org/show_bug.cgi?id=14354</ref> which had surfaced in upstream kernel 2.6.32 testing to try and ensure that it was not affecting the 2.6.31 kernel included in Fedora 12.
 <references/>
-=== Blocker bug review ===
+=== Fedora 12 release ===
-The second Fedora 12 blocker bug review meeting took place on Friday 2009-11-30, and [[User:Adamwill|Adam Williamson]] posted a summary<ref>http://www.redhat.com/archives/fedora-test-list/2009-October/msg00845.html</ref>. He noted that all remaining 43 blocker bugs had been reviewed, linked to the meeting summary<ref>http://meetbot.fedoraproject.org/fedora-bugzappers/2009-10-30/fedora-bugzappers.2009-10-30-15.01.html</ref> which outlined the status for each bug, and thanked the many members of the QA, release engineering and development groups who had contributed to the meeting.
-<references/>
-=== Fedora 10 bug review event ===
+Much of the group activity in the last two weeks was centred around Fedora 12 release testing and validation. [[User:Jlaska|James Laska]] posted a recap<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00266.html</ref> of the final blocker bug review meeting for Fedora 12. James, [[User:Adamwill|Adam Williamson]] and [[User:Wwoods|Will Woods]] worked with the release engineering team throughout the final week before release to continuously monitor blocker bug status, test fixes, and monitor for newly identified blocker bugs and regressions.
-[[User:Tk009|Edward Kirk]] announced<ref>http://www.redhat.com/archives/fedora-test-list/2009-October/msg00791.html</ref> a BugZappers event on 2009-10-30 at which the group would gather to try and review remaining Fedora 10 bugs and see which could be either closed or promoted to Fedora 11 or 12, prior to the automated closing of these bugs as old when Fedora 12 is released.
+Four release candidate builds were produced by the release engineering group with the help of testing and feedback from QA. [[User:Liam|Liam Li]] co-ordinated the planned installation testing through the test compose and release candidate process<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00080.html</ref> <ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00235.html</ref>, and maintained the test results matrix<ref>http://fedoraproject.org/wiki/Test_Results:Fedora_12_RC4_Install</ref>. He also sent a post mortem on the testing process<ref>http://www.redhat.com/archives/fedora-test-list/2009-November/msg00585.html</ref>. Many members of the group contributed valuable test reports to the matrix. Several group members, including Gianluca Cecchi, Gene Czarcinski and others posted test installation reports which helped identify important issues that were fixed or documented during the release process.
 <references/>

Search