Fedora 14 QA Retrospective

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 14. The feedback will be used as a basis for identifying areas for improvement for Fedora 14 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to test@lists.fedoraproject.org.

Providing feedback

Adding feedback is fairly straight forward. If you already have a Fedora account ...

1. Login to the wiki
2. Select [Edit] for the appropriate section below.
3. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

4. When done, Submit your changes

Otherwise, if you do not have a Fedora account, follow the instructions below ...

1. Select the appropriate page for your feedback...
   • Something that worked well
   • Something that didn't work well
   • Anything on your QA wishlist
2. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

3. When done, Submit your changes

Feedback

Things that went well

1. jlaska - Beta released on time - Without a strong turn-out from the fedora-qa community, there is little chance that we would had the test results needed in order to properly assert that the beta release criteria had been met.
2. jlaska - Test Matrix - identifying concerns with the Fedora 13 test matrices, and implementing improvements for Fedora 14 worked really well. I really like to revised organization and cleaning of obsolete tests in the installation matrix (see QA:Fedora_14_Install_Results_Template).
3. jlaska - Release Criteria - For me, the release criteria continue to pay dividends. It's exciting to see bug reports, reported by testers outside the QA team regulars, escalated for blocker consideration that reference specific criteria. While we knew going into this that identifying and documenting 100% of the use cases of Fedora that we would block the release for was impossible, we seem to capture a majority of low hanging use cases as the currently documented. I also like the public discussion around proposing and debating changes to the criteria. Nice work to all involved :)
4. rhe - Group the summary of bugs in different priorities - It was confusing when one candidate met the criterion but still have a list of bugs. When group them in to different levels, it is more clear for the testers to check the bug status.
5. rhe - Delta isos - very efficient and low bandwidth for downloading delta isos.

Could have been better

Fedora 16 alpha (Gnome) fails to complete properly (and does not give compliance warnings) when an NVIDIA Quadro FX3400SLI (HP Supplied) is installed in an HP XW4600 Workstation. (Works fine with NVIDIA GeForce similarly scaled cards). Under Fedora 15 (Gnome) the Quadro card reports no support under Gnome 3. - Cheers - reynolds@mira.net

1. jlaska - Beta RC delayed - Beta RC candidates were delivered late, as there were 2 unresolved beta RC blockers (see below). Specific to the following two issues, it's not clear what can be improved/corrected next time?
   • RHBZ #629719 - parted - random failures when notifying kernel of changes to partitioned md (isw) device's partition table
   • RHBZ #633315 - anaconda - Connections not editable in nm-c-e in anaconda
2. jlaska - Fixing more than blockers - For F-14-Alpha, a new anaconda with 2 blocker bug fixes was requested, but the new version contained many more fixes. Question, did the additional bug fixes break anything?
   • Recommendation - Early creation of f14-{alpha,beta,final} git branches for anaconda to allow for more control over commits.
3. jlaska - Root cause analysis - on why the following issues were discovered late
   1. RHBZ #627789 - several users reporting problems installing from HD ISO due to this bug. However, testing RC3 showed no problems installing HD ISO ... there is a disconnect somewhere between the use case exercised and the test case? (see docs follow-up)
   2. RHBZ #638091 - Kernel panic at boot after glibc update - Updating to glibc-2.12.90-13 (see bodhi) resulted in a kernel panic upon reboot. The system would be fine until the prelink command ran, and then it would never boot and all commands would segfault on a running system. The issue was initially given positive proventester feedback, but was then reverted after the tester found the problem. The good news, the problem was discovered in updates-testing before it landed in updates. However, it would be good to have some documented test procedures for a subset of packages to guide proventesters
4. jlaska - Virt Test Day - Participation in the test day was low and not what we had hoped for. We identified this problem during the Fedora_13_QA_Retrospective, what can be done to provide meaningful public test events for Virtualization?
5. jlaska - KDE bugs - are KDE-only bugs release blockers? It's unclear how to handle this or what FESCO intends (see RHBZ #641338).
6. jlaska - Wiki Test Matrix - we've hit the maximum threshold for mediawiki parser function calls, Category:Pages_with_too_many_expensive_parser_function_calls (for specific example, see Test_Results:Fedora_14_Beta_RC3_Install). Might be able to modify Template:Result to reduce the parser function calls, but not sure what else to do. Switching to a formal test case management system for this specific issue seems like overkill, but certainly an option.
7. jlaska - Proventesters - The proventesters sign-up and instructions are awesome, but I think we have a gap around providing specific instruction for certain packages (kernel, glibc etc...). Can we outline a process and some wiki structure to encourage mailing list discussion and drafting of package-specific test procedures (e.g. [[:Category:Test_Cases] or Category:Test_Plans)?
8. mmaslano - Adding positive/negative karma - In Bodhi should be hint/help for adding karma +/-1. Imho it's not clear that I don't give -1, when I find new bug in update. Therefore, it would be nice to have it next to button 'doesnt work'.
9. mmaslano - Too few testers - I don't care when I don't have many testers. The problem is wrong testing. 0 karma - I've updated and nothing wrong happened, doesn't help me much, but it's much better than +1 without even trying to run something dependent on update. Imho tester should try reproduce bug after update. But it is even better if he run their scripts/apps dependent on update and it's still working. For example if I update threads in perl, then I would be grateful, when someone with his own threads dependent script give me feedback. I was thinking about creating list of dependent packages, so tester could choose his favourite component. It might be useful for libraries and interpreters.
10. rhe - Tests and Criteria - Criteria was kept changing during the test cycle, the tests need to sync with it, so they should be reviewed and updated or added to fit for the criterion.
11. rhe - Install Test Matrix - Every time one has to click the show/hide button to check the collapse table, better to find an alternative way to save it.
12. rhe - Reference Section Separated - Not sure if it's a good way to separate the references from the tests (the reference used to be in the same row as test case), sometimes we have to look for which test case caused the issues in the matrix.
13. dramsey - To virt or not to virt. I strongly suggest that a review of the ideas and discussion about the Fedora 13 QA Retrospective Virtualization Days be reviewed for your advantage. Too much Smorgasboard food to choose from, may be you like chicken, he likes pork, she likes fruits&vegetables and I like beef.

    "...

    * I would ask that a test week be considered in order to accomplish all the test cases,

    * scale to fit the single test day structure, or

    * cut into individual bitesize chunks...."

14. Adamwill - No Live image ticket for Final-TC - We should have live images created for the test compose milestone
15. Poelstra - ISO delivery time of day - It's never clear when QA should be prepared to announce and begin testing on ISO images. Currently, QA monitors the rel-eng ticket to determine when images are available. I wonder if it would be better to request images at, or by, a specific time of day to ensure someone in QA can begin the processing.
16. jlaska - F-14-Final-TC1 Live images - The KDE and GNOME live images had different kernel packages. It appears that during generation of the images, a different mirror was used and an older kernel package was included for the KDE TC1 live image. mkrizek - The KDE live image had different selinux-policy package which had the https://bugzilla.redhat.com/show_bug.cgi?id=590883.
17. jlaska - Test Case - For several releases now we've included the test case QA:Testcase_Boot_Methods_efidisk.img in the install matrix. However, we don't have any hardware or expertise to run this test ourselves.
18. rhe - Test case - QA:Testcase_Mediakit_ISO_Size didn't include the target sizes for different live images, need to clarify it.
19. adamwill - F-14 go/no-go - The timing of the final release go/no-go was a day earlier than expected. Unclear whether this was intentional.
20. Adamwill - Missed NTFS bug - we missed a big NTFS bug; rhbug:627153 (fix was available on f15 branch but not pulled to f14 as the issue was never escalated). We're probably missing explicit tests for NTFS partition handling (and other Windows interop stuff)
21. notting - Need more testers - as always. My experience was that most critical path things get tested quickly and reasonably, but non-critical path things linger in updates-testing, especially if they're not commonly installed.
22. notting - Testing of mash and similar items - mash is a critical path package, as it's critical to getting the distribution composed from koji. However, since it's *only* useful for getting the distribution composed from koji, it is very unlikely to get any testing outside of the developers. Not sure of a good way out here.
23. cwickert - Better testing of the spins' - as always. More testers are welcome. One tester spotted a bug but did not file it. Spin owners were not informed about composes, thus they could not test themselves.
24. rhe - live images - We are still unsure if live images should be provided or not for test composes. Such rules should be clarified somewhere on wiki or in the mail lists.
25. rhe - bug title extracted - It's better to have the bug title displayed when inputting {{bz|bug.no.}}
26. rhe - Lacking Test days Topics - There are more 'open slot's days on test days schedule than before. Should we make a overall plan in advance to avoid this?
27. rhe - More cooperation with i18n and l10n team - can work together further to increase the coverage of i18n and l10n on fedora.
28. rhe - USB installation tests - need to clarify the support for usb installation and add this kind of tests if needed since many users install using usb device.
29. adamwill - ntpd selinux alert - just about everyone who installed F14 hit an selinux alert because turning on ntpd triggered one. Our test for out-of-the-box selinux alerts specifies doing a *default* install, and ntpd isn't turned on by default. we should broaden the test out to also cover options that people commonly enable during install.

Wishlist

• I want a pony

1. jlaska - Debugging guide - Would love to have a PolicyKit debugging guide available at Category:Debugging
2. jlaska - Security related fixes - We have no criteria to appropriately prioritize security sensitive bugs
3. adamwill - Final release notes criteria - No criteria covered accepting RHBZ #643220 - Blocker bug for final version of release notes
4. jlaska - Tracking untested ON_QA - when updates are pushed, all ON_QA issues are pushed out to CLOSED ERRATA even if they haven't been tested. We should probably track these bugs.
5. jkeating - Source ISO size - The F-14 Source ISO's are too large (5Gb). QA doesn't currently check these images, should they?
6. poelstra - Test compose documentation - It would be nice to have documentation on what the test compose is for and why it's needed.
7. notting - Test plan storage - AFAIK, there's no official storage area for 'how to test' for packages, nor a way to present this to QA/proventesters. This can limit testing, as volunteer testers may not know what to test.
8. notting - Unclear list of current bugs - We have proposed blockers, accepted blockers, and nice-to-have. However, the fact that proposed blockers aren't accepted/moved-to-NTH at blocker review meetings means that in the interim between meetings, it can be difficult to get a global view of "what needs worked on now", as you have to either ignore the proposed blockers, or make some judgements yourself on them.
9. jlaska - {{result}} usage - recommend using Template:Result for all community test events. It makes it far simpler to scrub the wiki and extract metrics when using the same format. Currently, most test days don't use this format.
10. rhe - more local community attendance - will have a party with them to discuss about these.
11. rhe - TCMS? Semantic plugin? - could use another friendly system or plug in for recording test results, maybe the one that doesn't require editing the syntax.
12. Sparks - Build Docs test community - Currently in RH Docs, they QA EVERYTHING. We (fedora docs) just don't have the manpower to do that right now. I wonder if QA might have the people to help go through our guides and verify procedures and such.
13. Spot - Upgradepath - Need to schedule upgradepath test runs against the entire distro, not just for each bodhi build. Also, adjust release criteria accordingly
14. robatino - Keep xz-compat-libs package in F15 and later - Using deltaisos during F15 testing and rebuilding old deltaisos will require being able to switch between old and new xz compression in both old and new versions of Fedora, which this package makes simpler - see this thread.

Recommendations

After enough time has been given for feedback, the QA team will discuss and make recommendations on changes to prioritize for Fedora 14. This section organizes and lists the recommendations.

In order to coordinate efforts, and measure effectiveness of recommendations, please record and track any action taken in the Fedora 15 roadmap in the QA TRAC instance.

Coming soon...
Recommendations will arrive typically after problems have been diagnosed and improvements reviewed. This typically occurs after the release. Stay tuned ...

References

• See Fedora_13_QA_Retrospective
• See Fedora_12_QA_Retrospective
• See Fedora_11_Retrospective_Notes