Fedora 11 Test Day Survey
From FedoraProject
One Wed, Jun 03 2009, a survey was sent to fedora-test-list@redhat.com and participants of Fedora 11 Test Days. The survey was intended to gather feedback on several aspects regarding Test Days. The initial survey is available here and included below.
Survey
The Fedora QA team would like your feedback on Fedora 11 Test Days. You may have seen Adam Williamson's planet post [1] kicking off Fedora 12 Test Day planning. We're interested in identifying areas for improvement to increase participation and improve effectiveness.
Please take 10-15 minutes to answer any/all of the questions below. You may reply to the mailing list, or send feedback directly to me. Your responses to this survey are instrumental in making Fedora 12 Test Days successful.
Many thanks to User:cward for his help in getting things moving with the survey questions!
1. How did you find out about Fedora Test Days?
2. Was sufficient documentation available to help you participate in a Fedora Test Day? If not, what did you find missing or in need of improvement?
3. Did you encounter any obstacles preventing participation in Fedora test Days? How might they have been avoided? Did you discover any workaround?
4. Were you able to locate and download installation media for testing? Did it function as expected?
5. What follow-up actions do you expect after the Test Day? Are your expectations currently being met?
6. Would you participate again in future Fedora Test Days?
7. Do you have any more general comments or any suggestions for improving future test days?
[1] http://www.happyassassin.net/2009/06/02/whats-goin-on-f12-test-days/
Feedback
How did you find out about Fedora Test Days?
- devel mailing list
- Fedora test mail list
- Fedora project WiKi page of test day
- Search for the nouveau feature in f11 and find out that a test day is planned
- Through Planet Fedora
- IRC
- fedora-devel-announce
- saw reminders in FWN
- notified by a colleague. Got used to it taking place on Thursdays and I participated more
Was sufficient documentation available to help you participate in a Fedora Test Day? If not, what did you find missing or in need of improvement?
- Good example ... Test Day:2009-03-26 Nouveau
- Bad example ... Test Day:2009-04-09 UEFI - how to identify if your system supports UEFI?
- In all test cases I tried, concise but complete instructions were listed to help me test specific features or capabilities.
- For the ones I participated in (storage and radeon) things seemed fine.
- The only Test Day I recall participating was the EXT4. I was having some issues with F11 at the time, however my tests with EXT4 where all successful.
- Never did any test day, but I looked at some pages on the wiki, and everything seemed clear, no question arose that could have prevented me from running the tests.
- Sure. But I'm running rawhide anyway on several machines and am probably not really representing the average user.
- For me, yes [...] But I guess I'm not an average Joe Tester.
- it depends, sometimes yes, sometimes no ... I'd appreciate exact, step by step, setup and results reporting instructions (something like "fool around and share your observations and ideas" is not what would motivate me, I have enough other - more interesting - things to just play with)
- I found docs sufficient
- some testday wasnt so much good documented. For example I have problems with testday focused on virtualization. It was nice testday, but I think only for people who know enough about virt, because It was focused on bug hunting, not testing features or exact scenarios. I Think, there should be example of scenarios (how install virt machine. how setup proper features for testing (for example storage testing. how setup virtio, ide, scsi)
Did you encounter any obstacles preventing participation in Fedora test Days? How might they have been avoided?
- HW, Time.
- Failure while creating live images using Fedora 10 USB creator tools
- Participants needed FAS account for wiki editing, but I know this was resolved.
- For the tester,maybe downloading test media and burning them will save some time
- Not directly to me, but I have read some complains about to represent the result in the wiki, because you need a FAS account and have the CLA done, with seams to be problematic for some users.
- Access to hardware, but nothing you can do about that. :-) Some more indications about whether specific tests can be accurately performed in a KVM virtual guest would be appreciated, though.
- I found out about the Intel graphics test day too late to be able to participate on the right day. I first had to create a FAS account as I hadn't yet taken all the steps to become a packager. I got side-tracked by an incorrect error message in FAS and had some problems before I could report that. Then I had to create a Smolt profile. SmoltGUI crashed but I could work around the crash by changing the locale. Smolt using two kinds of UUIDs caused some confusion. I could eventually go through the test cases for Intel graphics a week after the actual test day. #* Some related to being pretty busy on Thursdays. For one off tests I don't see Fedora doing anything about that, but if you do repeats on the same test later it is probably worth thinking about if it would be better to keep the followup on the same day of the week to make it more likely that people who participated in the earlier session can participate in the followup or if making it a different day of the week to make it more likely that some people who couldn't participate in the earlier session because of a conflict would be able to participate in the followup.
- My only obstacle was that Fedora 11 itself was not booting corectly (did not get to Gnome), that issue was independent from the one test day I participated in though.
- Time. I never had the time to actually commit myself to a whole feature and pass the test cases.
- Also, I had a lot of troubles with Anaconda that prevented me from installing most of the pre-releases. When I finally managed to do it, I simply used Rawhide on a day to day basis and thus reported the bugs I could spot.
- Time is really the problem here in my opinion. I really hope I can participate in some test days for Fedora 12 as I really like the concept. And yes, I know I could have run the tests after the test day, but I wouldn't have benefited from all the available people as on the actual day and that still requires quite some time. So I justsettled for my own little day to day tests...
- Also, the test days are hold while I'm at work, which make it harder to participate (but I guess that's the same for everybody).
- The live CD might help in some cases e.g. a nouveau test but would be mostly useless for installer-related tests. I guess more care should be taken into identifying "known-good" snapshots and testing with these where we made certain that there are no other, unrelated bugs interfering with the aim of the test day.
- My biggest obstacle with test days was that they were not planned early enough. Most of the test days seemed to be planned less than 48 hours ahead of time. If test days were planned better, I could actually participate more.
- missing test case management system
- uploading data to wiki instead of some FTP is a PITA
- lack of time :-(
- crash-catcher testday was waste of time for all tester - crash catcher doesn't work at all - in the beginning. it would be good if devel|someone runs|tests their package on CLEAN freshly install system before.
- Big problem was, that sometimes was very hard to install fedora rawhide. Because not all ways can I use. For example I needed upgrade from few days late rawhide or from F10, and sometimes happened that upgrade broke system at all. LiveCDs was ideal, but wasnt able to use for every testing. Good way was to install system from nightly built live CD (but this way should be enought tested, because I then found the problem with installation on single partition (ext3 setuped for / but system give there ext4) and then installed system wasnt bootable. So, I suppose to have liveCD with proper packages for each of testdays
Were you able to locate and download installation media for testing? Did it function as expected?
- expected testing instructions on the live media to perform the tests without the need to setup an internet connection on the testing machine.
- some bug report info gathering scripts would have been nice, that e.g. collect all the data required by the packages maintainers for a bug report in some file on the usb medium. E.g. the Xorg people wanted a lot of information that I did not have anymore after I rebooted to my stable F10 system and reported the bug.
- Yes, now I know how to get installation media in the least amount of time. (download speed an issue)
- Live CD images were linked from the wiki pages. I had no problems downloading them. I eventually managed to make a working live USB stick from the one for the Intel test day, after I transferred it to my work computer where I had Fedora 10 and could install the latest Syslinux from Rawhide. It wasn't possible to do this on Fedora 9.
- The live USB stick I made from the CD image for the Nvidia test day was more dead than live. It wouldn't boot, so I couldn't participate in that test day.
- Could be easier to find
- Yes, but I couldn't write a LiveUSB from Fedora 10 for Fedora 11 (I think the bug's been fixed), so I had to burn a CD instead.
- [...] Anaconda gave me a lot of trouble, and in the end I had to workaround the issues in order to get F11 installed.
- Never tried livecds but yeah, my PXE server is doing great for installs.
- I was able to download them. No, the media didn't work. It generally hung the computer, but that's not the fault of the test days.
- I always have rawhide on one of my machines, so I didn't need media.
- NP for me. Well, to be precise, I did my installation just once, then I was yum-upgrading from rawhide before any testing.
- no - just query bugzilla for anaconda problems ... I met a number of them, mostly reported already
- Several times F11 wasn't available on our local mirror
- your live cd was pretty good idea - next time they could be available on more mirrors
What follow-up actions do you expect after the Test Day? Are your expectations currently being met?
- The reported bugs should be fixed soon.
- Some article with results on some Linux servers. I liked what jscotka written (in czech) about PowerManagement Testday: http://www.root.cz/clanky/jak-probihal-fedora-powermanagement-testday/
- yes and no, the reported bugs are in touch with the developer, that's really good, but the issue is still there. But it's all right, because it seems hard to fix and effect no many people.
- Typically I expected to be able to file my information through some input system (in this case, the wiki), and I assumed I would hear back if there were questions about the information I left or bugs I filed. The bugs I generated seemed to get the notice that was merited, so I feel my expectations were met.
- If I were a new contributor to Fedora, an "attaboy" email might not be out of the question, though.
- Bugs fixed. Possibly a summary of how the previous test day helped before talking about the next one?
- I expected that someone would attempt to fix the bugs I reported. No such attempts have been mentioned in Bugzilla so far. I suppose I'll se whether they've been fixed when I upgrade to Fedora 11.
- Perhaps there would have been more interest in my reports if I had submitted them during the actual test day.
- The storage test day ended up with some bugs getting filed and fixed. The radeon day, not so much.
- I didn't expect a followup, and IIRC, none where made for the EXT4 test I did participate in.
- IMO, the most important thing the recent QA efforts including test days having brought in is a change in general mentality. Earlier when users would hit some regressions or new bugs, they would come up to the forum and get told by others to expect breakage since Fedora is meant to be that way. I found that a very dangerous stand point since it effectively means that over time, we will get lower quality with not enough people raising a concern since they have been told to live with it. Now with the QA efforts, even if users do hit bugs which I am sure they will, many would atleast be aware that a community is involved in helping to avoid them and fix them otherwise. We are more visibly concerned aboutquality now than ever before.
- I expect reports on the bugs that were found, and of course, that they are actually fixed after that
- For what is worth, I loved Adam's posts about the test days on graphical drivers where he did just that (http://www.happyassassin.net/2009/04/15/intel-test-day-statistics/). I'd like to see more of that.
- I'd like to see bugs fixed as the follow-up actions
- Some have been verified and put up as F11Blocker or F11Target with not much more happening. In some cases the Needinfo requests for rechecking or further info came up only a few days earlier and I haven't gotten around to them yet as the test systems are not always available to me. Faster turnaround times would be great there as I have easier access to the machines in question.
- Some of my bugs have not been fixed yet, but I have to admit that these are unusually difficult to track down or fix.
- I expected an analysis of the data received either by a mailing list post or an update on the wiki page. I saw neither and thought my data was just thrown into the wind. My expectations were not met.
- I would expect a recap of the testing efforts so that Fedora people could analyze what the issues were, track them, and fix them. I suppose they are. The mailing list enabled them to do this, but there was no formal method of doing it.
- Some blogging to show what cool things we worked on is nice (not only on FP, but spread it to different media).
- Some stats on the reported bugs, issues found and number of active people are a plus :)
- it depends on the nature of the tested feature ... mostly yes, I see guys focusing on the testday findings
- Some public report about testday and sme results (to see my effort wasnt useless) It was sometimes good, sometimes bad. But I understand that it is problem to say some results
Would you participate again in future Fedora Test Days?
- Would like to do so. Currently HW is one of the biggest blockers.
- Yes, dependent on barrier to entry.
- Yes. Encryption over raid get's messed up fairly often and actively participating seems to help in getting the issues fixed. Even though the radeon stuff didn't work out that great this time, I'd participate again. I think the stuff I am having issues with is low priority right now, but might get more attention in the future.
- Yes! I love beta testing broken products and helping in fixing them.
- Sure, time permitting.
- If they were planned better, then maybe I would be able to set aside time to do them. I would like to participate in future Test Days.
- Definitely! A lot of cool, friendly people around. Time is my only enemy.
- Yep, and also I plan to organize some testday
Do you have any more general comments or any suggestions for improving future test days?
- start them earlier in rawhide development phase. If tens of bugs are found three weeks before freeze, we can hardly expect they are fixed in time.
- If possible, can you announce the events (such as test day) in home page as news. When we open the home page we will know what is coming and what I can prepare to do, not like now, find them in thousands of mails
- Set up a reporting center just for Test Day feedback. Using the wiki is definitely not good enough. Additionally, Do not limit the test days to people subscribed to the mailing list. Take a page from Mozilla's books and announce those test days to the world. Unfortunately, to do that, test days need to be planned better.
- Better announcement of test days, also with more than 2 days ahead of time. Maybe some RSS feed only for test days or a calendar (iCal format maybe)
- is there a way to document HOWTO run a test day, including how to make the required media and a template for the test day wiki page?
- Just that I'm very glad they're happening. Well done.
- If they cover some functionality that's particularly important to me or some less than common hardware that I have.
- Coffee coffee coffee :)
- Start them earlier in the process. In particular I think this would have been advantageous for the anaconda storage rewrite.
- Advertise them a bit more. I think LiveUSB is the most enviromental friendly way to testdrive a new release. It doesn't require burning new media for a bugfix, and doesn't require setting up a VM or a new partition on your harddrive to test a specific feature.
- With concerns over Ext4, I did my own testing and didn't find any issues and now run with Ext4 on this laptop except for /boot. However that is a key piece and it would have raised the confidence level to have a dedicated test day. Having test days built into the release schedule would help everyone provide more feedback and I (and likely others) would have raised this earlier than I did in that case.
- Maybe having twice each test day would help people who couldn't attend the first time. But that would be twice as work for the developers who participate to help testers on their features.
- An obvious suggestion would be to have automated tests (scripts ?), but that might be a lot of work for only one day if a particular testday is not held again (an dI'm sure you already thought about it).
- I strongly support the new found focus on QA. I think I mentioned that to Adam on IRC repeatedly and wrote something about it
- Please get the Fedora calendar server going. I'd love to subscribe Thunderbird/Lightning to the QA calendar. People would be able to know about and participate in test days (or any QA event) without a mailing list subscription or a 24/7 IRC connection as it seems some things are discussed solely on IRC.
- We need to spread the word though different web sites, especially where Linux geeks hang out. I always try to write a short invitation on a couple of the main czech sites (at least root.cz and abclinuxu.cz), though there's a bunch of weeks where I forgot to.
- When organizing our own test day, the tasks weren't assigned very well. There wasn't good coordination with the participants of the test day. Most people did their own thing instead of someone coordinating with others through a list of tasks that need to be accomplished.
- Testday should be more announced among fedora people - I mean normal fedora people not geeks; I guess they (normal users of fedora ... ) don't know about this testday action.
- Thanks for your care about fedora testdays.
Recommendations
Advertising
- While all announcement channels used were recognized by respondents, several suggestions indicated we need to advertise test days in more circles. Where else can we solicit participants?
Scheduling
- Many suggestions indicating we should start test days earlier. I'm not sure what we can do here, test events scheduled well before Alpha suffer from rawhide-sickness. Ideas?
Documentation
- Many critical comments all center around the importance of well documented test cases or test instructions. As one respondent indicated, [events where you] fool around and share your observations and ideas" is not what would motivate me. We must continue to build on the test documentation provided by our more successful test days.
- Interesting idea to add test instructions to the live media for testers without networking. I think we've partially accomplished this by building test day live images that include a desktop icon and default firefox homepage that links testers directly to the current test day. Can we do more?
- A general theme from fellow test day coordinators that the steps/best practices were not well established or documented. I think it would make sense for Fedora QA to improve documentation to help teams prepare for a test day. There is often a lot of prep work involved with documenting, announcing and hosting a test day. Spelling this out will make for a consistent Test Day experience and allow other teams to roll their own test days.
Distribution
- Several suggestions to provide test day live media through fedora mirrors
Reporting
- Producing a more formal test day summary must be standard practice for all test events
- Improving on the current wiki test results was identified by several respondents. Some suggestions include a complete test case management framework. There weren't indications that people refused to participate due to the lack of a formal test case management system. However, it's likely that in order to scale test results reporting, something more robust than the wiki may be needed. While I don't think this is something we can accomplish in the short term, setting some short term objectives that help move things in the direction of formalized test case management is a good idea. For Fedora 12, I feel it's worthwhile to continue the Mediawiki + semantic proof-of-concept to determine if this can be helpful for improving the test results workflow for the next release.
Your suggestion here
Have an idea to improve Fedora 12 test days? Please feel free to update the wiki directly with your suggestion. Alternatively, send your suggestion to our mailing list.
Have an idea to improve Fedora 12 test days? Please feel free to update the wiki directly with your suggestion. Alternatively, send your suggestion to our mailing list.
