From Fedora Project Wiki

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 15. The feedback will be used as a basis for identifying areas for improvement for Fedora 15 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to test@lists.fedoraproject.org.

Providing feedback

  • Gwjasu - I like ____ about the new ____ process

Adding feedback is fairly straight forward. If you already have a Fedora account ...

  1. Login to the wiki
  2. Select [Edit] for the appropriate section below.
  3. Add your feedback using the format: 
    * ~~~ - I like ____ about the new ____ process
  4. When done, Submit your changes

Otherwise, if you do not have a Fedora account, follow the instructions below ...

  1. Select the appropriate page for your feedback...
  2. Add your feedback using the format: 
    * ~~~ - I like ____ about the new ____ process
  3. When done, Submit your changes

Feedback

Things that went well

  1. jlaska - Mailing list - Seeing an increased level of mailing list traffic discussing test issues with systemd and gnome3. This is pretty exciting and exactly the type of communication I'd love to see on the list. Testers helping testers identify and triage issues.
  2. jlaska - Beta - Because of the FESCO approval for the late arriving NM-0.9 changes (refer to [https://fedorahosted.org/fesco/ticket/572 FESCO#572), QA held firm that the schedule must slip one week. At the time of the decision, 2 test milestones were missed (pre-beta acceptance, TC1). Attempting to absorb the late changes, and the bugs introduced by those changes, would have negatively impacted the release and definitely resulted in a slip regardless. It was the right decision to slip the release after accepting the late NM-0.9 change.
  3. rhe - boot.iso for anaconda updates - it's a great idea to post boot.iso with new anaconda version for testing before composing it to the next TC or RC versions.

Could have been better

  1. jlaska - Alpha-TC wasn't branched - F15 Alpha test compose was created using pre-branched content. This was unexpected, but it appears that there is no clear guidance/documentation on what is expected (see https://fedorahosted.org/rel-eng/ticket/4399)
  2. jsmith - Mass rebuild not included in schedule - The mass rebuild timing wasn't ideal, it landed at the same time of the branch. While release engineering did an outstanding job resolving rebuild issues in a timely manner, the schedule didn't account for the mass rebuild, and test composes were delayed.
  3. jlaska - insufficient live testing - Because of RHBZ #672265 and RHBZ #676904 we didn't get a lot of testing on the live images prior to, and during the Alpha test composes. As a result, once those issues were fixed, we found 3 additional Alpha blockers
    • RHBZ #679107 - TypeError: argument 2 to map() must support iteration
    • RHBZ #663294 - RuntimeError: XOpenDisplay failed }} - TypeError: argument 2 to map() must support iteration
    • RHBZ #672030 - AttributeError: 'NoneType' object has no attribute 'format' }} - TypeError: argument 2 to map() must support iteration
  4. jlaska - missed first 2 blocker review meetings - Not sure what else to say ... we forgot about these meetings. With FUDCon and PTO, the first two completely fell off my radar.
  5. jlaska - i18n - i18n/l10n continues to highlight problems early on in the release. F-15-Alpha slipped one week due to RHBZ #676827 (keyboard with german layout doesn't work in gdm). We need release criteria for LANG/keymap issues.
  6. jlaska - Missing criteria - When proposing and publishing release criteria, use caution when making changes. Particularly when release criteria pages are already available for the next release. You must be sure that any changes made to 40 are copied forward to 41. This was problem for Fedora 15 Alpha with the desktop artwork criteria. The artwork criteria was added to the Fedora 14 Alpha criteria page, but the Fedora 15 Alpha criteria page was already created. The new criteria were not carried forward, and Fedora 15 Alpha released with the incorrect artwork. As a result, a bug was filed, but not escalated.
    • RHBZ #677080 - 'F14' artwork is shown during F-15 installation
    • Also, the f15 design schedule does not have a task to update fedora-logos with Alpha installer artwork.
  7. Rhe 06:50, 14 March 2011 (UTC) - VNC test - Add password test for VNC to current test run since the bug was found:
    • RHBZ #678150 - VNC install w/ password, fails to establish password for VNC session
  8. vhumpa - GNOME 3 Test Day attendance - for some reason, for this #2 testing event, significantly less testers showed up: 19 compared to over 50 for #1. Still - a nice number of bugs have been encountered, 1.58 bugs per tester at #2 compared to 1.28 at #1, which is due to a larger number of tests this time (25 vs. 16). Although a test day was finally a success, it might be useful to reason about the lower turnout. Here are some ideas, but please note that they are just speculations:
    • Generally, there might have been less simply curious people as they'd already get the curiosity satisfied by #1
    • At the last moment, we wanted to add some of the latest updates that were not yet available in repositories, which caused:
      • We weren't sure if we wouldn't actually want to move the event, so perhaps notifications were sent a little late
        • I have no data on this - but perhaps also less announce channels were used this time
      • We couldn't use the nightly composes as previously planned and had to make custom images - which (as Adamw's sleepless nights confirm) brings trouble and takes time. Particularly the i386 image wasn't available until the late Test Day, which could have driven away some of the testers
    • A significant number of test cases could also have discouraged some testers with less time. We had to divide the tests in two sections/tables and majority of the testers did go through just the first part - which can be seen as a little proof of this.
  9. jlaska - Beta - Missed the 2011-03-17 - F-15 pre-beta acceptance milestone (https://fedorahosted.org/rel-eng/ticket/4530)
  10. jlaska - Beta - Slipped release 1 week due to decision by FESCO (refer to FESCO#572) to accept late NetworkManager-0.9 changes which impacted *all* desktop environments. QA felt strongly that the right decision was to slip the release one week and not attempt to absorb the changes given that we already missed 2 test milestones (pre-beta acceptance and beta TC1). Could the late arriving changes have been seen/predicted earlier? Ticket filed on 03/10/11
  11. Rhe - btrfs - Btrfs file system was added for partitioning, need make sure if it's supported and then add test case for it.
  12. We had a complaint that "Release Candidate" was confusing when applied to Alpha and Beta releases. A suggestion was made to start calling these "Alpha Candidate" and "Beta Candidate".
  13. AdamW - the introduction of the /run directory was done very late in Beta timeframe, and this caused most of the trouble we had getting the Beta out on time. The impact of the change was somewhat misrepresented (at first devel list was told that /var/run would not be made a symlink to /run in F15 timeframe, but in fact it *is* being made a symlink on new installations).
  14. jlaska - 'Install ISOs for updates-testing' - building custom boot.iso images to supply anaconda karma seems broken. This should be enabled by default. Why aren't we also providing nightly updates-testing live images for consumption?
  15. Rhe - nfsiso repo - Clumens recommended that the nfsiso repository should only have the correct ISOs the anaconda points to instead of a set of different isos. If so, the test case/documentation need be modified to reflect it. See RHBZ #695280.
  16. Adamw - BetaNag - https://bugzilla.redhat.com/show_bug.cgi?id=703815#c8, we should have criteria that include removal of the betanag for Final release.
  17. Rhe - test case update - initrd.img is not in gzip format from F-15-Final-RC1, thus the QA/TestCases/KickstartKsFilePathKsCfg is not applicable to include ks.cfg in it. Test case need be updated.
  18. jlaska - F-15-Final-RC1 blockers -
    • RHBZ #704030 - package conflicts in 15 Final RC1 - the generic* packages made it into RC1, resulting in broken dependencies. How'd this get into RC1, and no other interim milestone tree? adamw: releng inadvertently left the necessary exclude statements out of the repo definition when transferring it from tc1 to rc1 build
    • RHBZ #704042 - X fails to start after firstboot in 15 Final RC1 - dconf-0.7.5-1 from updates-testing appears to fix it - root cause? How did dconf-0.7.4-1 get into RC1, how did dconf-0.7.5-1 not get into RC1? adamw: 0.7.4 landed before freeze, and got positive karma: the bug is only apparent when tested with a new user account, so it's not surprising that no-one hit it in normal 'update and see if everything works' testing. A bug was filed but not proposed as a blocker, and the 0.7.5 update was not marked as fixing any particular bug, which made it a bit harder to see how important it was at a casual glance. Probably reasonable to think desktop team might have marked this as blocker: we could talk to them about flagging up significant fixes like this
    • RHBZ #699113 - During boot, message 'mkdir: cannot create directory /run, file exists' appears on virt. console 1 - root cause? Could this have been discovered (or resolved) earlier? adamw: the actual blocker here was caused by the *fix* for this bug: the fix to make the /run error go away was incomplete and caused live boots to fail. Could probably have been discovered earlier as nightlies were failing this way for a while. May be one where we should be stricter on NTH; the error was entirely cosmetic (though likely to confuse users, which is why we accepted it as NTH)
    • RHBZ #704415 - abrt-dump-oops breaks live media creation - root cause? Could this have been discovered (or resolved) sooner? adamw: root cause - badly-written rpm scriptlet. like most of the others, this was not noticeable in typical update-and-use testing, only noticeable in composing live images. i think we actually tracked this one down reasonably quickly given the subtlety of the bug
  19. jdulaney - Glibc update dropping support for dep that is needed by multiple packages early in the Final process, causing all manner of trouble. This should not happen. Maybe setup a process that must be completed before some feature that may be a dependency is removed. Does such a thing exist?
  20. jdulaney - In Bugzilla I haven't got permissions to set blocker status. This came up as a hindrance during the F15 process.
  21. jlaska - Bugs missed ...
    • RHBZ #706122 - USB overlay (persistent storage) not being mounted - Use of overlay not explicitly called out in any test matrix
    • RHBZ #706542 - AttributeError: 'NoneType' object has no attribute 'rfind' - FIXME
  22. pbrobinson - NetworkManager 0.9 was a major change to multiple features/desktops and should have been feature, and landed prior to the Feature Freeze and advertised appropriately. Instead it landed just before the beta and was only discussed on a non advertised bug in a fedora hosted trac instance (sorry can't remember which one). It also affected anaconda (points 9/10 above). Changes like this need to be landed as per policies or pushed to the next release, or what is the point of having policies if core components are allowed to completely ignore them!
  23. Bruno Wolff - There were a few 3+ hour blocker review meetings. It's hard to get people to volunteer to participate in those.

Wishlist

  1. AdamW - More package specific test cases - We need more package specific test cases (NetworkManager and kernel).
  2. robatino - Proposed regular naming scheme for blocker and NTH tracker bugs (https://fedoraproject.org/wiki/Talk:BugZappers/HouseKeeping/Trackers)
  3. robatino - Possibility of using "Install" instead of "Fedora" as the directory name for the install images (http://lists.fedoraproject.org/pipermail/devel/2011-April/150370.html and https://fedorahosted.org/rel-eng/ticket/4654)
  4. robatino - I'd like to implement https://fedorahosted.org/fedora-infrastructure/ticket/2241 by keeping the disos for TCs/RCs at http://alt.fedoraproject.org/pub/alt/stage/deltaisos/ indefinitely instead of deleting them after each development cycle. I have disos for all the F14 TCs and RCs which in total take about 7 GB. The filesystem where the disos live is the same one used by the stable release archive, and currently has about 6 TB available out of 9 TB total. The elimination of CDs in F15 frees up roughly the same amount of space as needed by the disos. So it seems that space should not be a problem. The disos between Alpha/Beta/Final can be deleted eventually since they are not necessary for recovering the TCs/RCs. As long as there is one diso for each TC/RC from an earlier one, all of them are retrievable. Update: Implemented with archive/ subdirectory (not mirrored so visible only on alt). Is it possible that having a large set of install images with known problems (from the test matrices) could help in developing automated tools for testing these images?
  5. jlaska - Add artwork test case to the desktop and installation matrix?
  6. jlaska - fedora-release - We always seem to forget to request an updated fedora-release package near the end of the release. It's in the SOP for rel-eng to update fedora-release, but that doesn't always remove the betanag. We should add a test to confirm the betanag is removed for final, and another test to confirm the betanag is present anywhere else.

References

Retrieved from "https://fedoraproject.org/w/index.php?title=Fedora_15_QA_Retrospective&oldid=238178"