QA/Meetings/20100215

Attendees

People present (lines said)

• jlaska (135)
• Oxf13 (34)
• adamw (32)
• kparal (16)
• jskladan (11)
• wwoods (11)
• zodbot (3)
• pjones (2)
• Viking-Ice (1)
• jeff_hann (1)

Regrets:

• User:liam
• User:rhe

Agenda

• Proposed meeting agenda
• meetbot summary

Previous meeting follow-up

1. jlaska will follow-up w/ nirik after the meeting to setup zodbot feed watchers
2. jlaska to work w/ poelstra to give 'release validation' spin to quality milestones

Rawhide Acceptance Testing

Test Summary (see recap)

• Automated rawhide acceptance tests (RATS) PASS
• The 'last known good' install images can be changed to 2010-02-04 (see rel-eng ticket#3373).
• Test results are available at Test_Results:Fedora_13_Rawhide_Acceptance_Test_3

Next Steps:

• Fix AutoQA post-tree-compose hook to monitor for RAT droppings (see ticket#115)
• Commit reviewed patches and rework one proposed change.

Alpha TC test status

Alpha TC#2 is available for testing. Instructions for downloading and providing feedback available at:

• QA:Fedora_13_Alpha_TC_Install_Test_Results
• QA:Fedora_13_Alpha_TC_Desktop_Test_Results

IDEA - Live images for test compose milestone

Adamw proposed including live images with future 'test compose' milestones. adamw would like a single set of nightlies to be associated with each TC build and kept available in that TC directory during TC testing

After some debate, no conclusions were reached. Jkeating felt providing nightly live images was not needed.

IDEA - Unique ISO image names for each drop

kparal proposed using unique ISO file names for each official test drop. Kparal was concerned that since each the ISO file names from each test drop are the same, it might not be clear to testers what the latest images to use are. After some discussion, Jkeating noted that the ISO file name produced comes from specific arguments used during the compose process and the current values are tightly integrated with the build process.

Jkeating noted that eventually, this information would be present in a future release engineering SOP (see Composing_Test_Composes).

AutoQA project update

Deps/conflicts prevention

• Milestone - https://fedorahosted.org/autoqa/milestone/autoqa%20depcheck
• Owner - User:wwoods

Last week

• Not much progress, wwoods was out sick for most of last week

This week

• Build a testsuite to validate depchecks functionality (possibly using rpmfluff)

Results collection

• Milestone - https://fedorahosted.org/autoqa/milestone/Package%20update%20tests
• Owner - User:kparal

Last week

• Held design session last week with wwoods, kparal and jskladan (see recap)

This week

• Continue research on other test frameworks handling of results

Beakerlib integration

• Owner - User:skladan

Last week

• kamil and I created a simple test, which demonstrates usage of beakerlib-based tests
• With help from Kamil, created a hello world beakerlib test and began the work of wrapping AutoQA/Autotest around it (see http://jskladan.fedorapeople.org/beakerlib_helloworld.tar)

This week

• we'll be trying to assimilate other tests this week - probably init script testing stuff - and look into some more complex uses

install automation

• Milestone - https://fedorahosted.org/autoqa/milestone/Automate%20installation%20test%20plan
• Owner - User:lili, User:rhe

Last week

• Reviewing dvd install options
• Sent summary of dvd install options for review

This week

• Chinese new year, so no action expected.

packaging/deployment

• Milestone - https://fedorahosted.org/autoqa/milestone/Packaging%20and%20Review
• Owner - User:jlaska

Last week

• Akurtakov's help has been tremendous ... more deps keep popping up and he has helped play wack-a-mole

This week

• I still don't have a final picture for what needs packaging. I'd like to schedule some sit-down time with akurtakov (others welcome) to finalize the packaging wish-list.
• Adamw suggested to start packaging a leaf-node dependency just to get the foot in the door

Upcoming QA events

• 2010-01-21 - Pre-Alpha Rawhide Acceptance Test Plan #1
• 2010-01-28 - Pre-Alpha Rawhide Acceptance Test Plan #2
• 2010-02-04 - Pre-Alpha Rawhide Acceptance Test Plan #3
• 2010-02-04 - NFSv4 Test Day
• 2010-02-05 - Alpha Blocker Meeting (F13Alpha) #1 (recap)
• 2010-02-11 - Test Alpha 'Test Compose' (boot media testing)
• 2010-02-12 - Alpha Blocker Meeting (F13Alpha) #2 (recap)
• 2010-02-12 - Alpha Test Candidate verification (announcement)
• WE ARE HERE
• 2010-02-18 - Alpha Release Candidate verification
• 2010-02-19 - Alpha Blocker Meeting (F13Alpha) #3
• 2010-02-24 - Alpha Go/No-Go Meeting (20:00 EST)

Action items

• No action items

IRC transcript

jlaska	#startmeeting Fedora QA Meeting	16:00
zodbot	Meeting started Mon Feb 15 16:00:23 2010 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot.	16:00
zodbot	Useful Commands: #action #agreed #halp #info #idea #link #topic.	16:00
jlaska	#meetingname fedora-qa	16:00
zodbot	The meeting name has been set to 'fedora-qa'	16:00
jlaska	#topic Gathering	16:00
* kparal raises up his hand		16:00
adamw	morning	16:00
jskladan	hello	16:00
jlaska	greetings folks	16:01
adamw	Software Engineer Barbie also says hi	16:01
adamw	http://www.engadget.com/2010/02/14/barbies-slides-into-the-cubical-becomes-a-computer-software-en/	16:01
jlaska	I believe maxamillion has a conflict today	16:01
jlaska	finally, some mainstream recognition! :)	16:02
adamw	looks like me on a good day!	16:02
jlaska	adamw: does it come with more accessories (external monitors, bluetooth keyboard(s) etc...)	16:02
adamw	this being barbie, all signs point to 'yes'	16:03
jlaska	alright ... let's get started. hopefully wwoods, Viking-Ice and tk009 can join in as we go	16:04
jlaska	#topic Previous Meeting Review	16:04
jlaska	I've only captured 2 items from last week ...	16:04
jlaska	#info jlaska will follow-up w/ nirik after the meeting to setup zodbot feed watchers	16:04
jlaska	thanks nirik, we can check this off	16:05
jlaska	we have zodbot notification for any additions to the F13Alpha, F13Beta and F13Blocker bugs	16:05
jlaska	we might want to consider fedora-qa TRAC ticket additions as well	16:05
jlaska	#info completed ... IRC zodbot notification for any additions to the F13Alpha, F13Beta and F13Blocker bugs (thanks nirik)	16:05
jlaska	next up ...	16:05
jlaska	#info jlaska to work w/ poelstra to give 'release validation' spin to quality milestones	16:06
jlaska	This is completed as well ... I suggested a minor wording change (nothing crazy)	16:06
jlaska	the test milestones are no longer specific to install	16:06
jlaska	(e.g. Test Alpha 'Test Compose' and Test Alpha Candidate)	16:06
* Oxf13 is here, sorry		16:06
jlaska	Oxf13: welcome!	16:07
jlaska	#info completed, the wording around scheduled test milestones are no longer specific to install	16:07
jlaska	that's all I had to track from last week ... anything I missed?	16:07
jlaska	alright ... diving into the proposed agenda	16:08
jlaska	#topic Rawhide Acceptance Test #3	16:08
jlaska	Just a quick update on the last of 3 rawhide acceptance test (RAT) drops	16:08
jlaska	you probably saw the announcement ... so I'm just summarizing here	16:09
jlaska	#info RATS#3 test recap at http://lists.fedoraproject.org/pipermail/test/2010-February/088407.html	16:09
jlaska	This milestone helped flesh out a few patches against rats_install	16:09
jlaska	thanks wwoods for review feedback, I'll follow-up on those shortly	16:10
jlaska	I need to update the RATS watcher script to watch the new URL (see ticket#115) ... I don't have immediate plans to look into that	16:10
jlaska	so anyone with interest and time is welcome to pitch in	16:10
jlaska	as always, Oxf13 thanks for coordinating the drops with dlehman	16:11
jlaska	that's all I had on that ... let's dive into the Alpha ...	16:11
jlaska	#topic Alpha TC test status	16:11
jlaska	just a quick summary ... then onto some ideas raised by kparal and adamw	16:11
jlaska	#info F-13-Alpha-TC2 is available for testing	16:12
jlaska	#link QA:Fedora_13_Alpha_TC_Install_Test_Results	16:12
jlaska	#link QA:Fedora_13_Alpha_TC_Desktop_Test_Results	16:12
adamw	yay desktop testing	16:12
adamw	wait, now I actually have to do some	16:13
jlaska	woot!	16:13
adamw	whose stupid idea was this?	16:13
jlaska	adamw: deatils deatils	16:13
jlaska	details too :)	16:13
jlaska	so first note I have here was from adamw ...	16:13
jlaska	#topic IDEA - publish live images along with 'test composes'	16:14
jlaska	adamw: you raised this last week, I put this on the agenda so we wouldn't forget	16:14
* Viking-Ice sneaks in..		16:15
jlaska	Oxf13 I believe live images are provided for the release candidate drops (alpha, beta and final) ... with the new focus around desktop validation, are we able to provide them for 'test compose' as well?	16:15
jlaska	Viking-Ice: welcome!	16:15
Oxf13	jlaska: well, we already make live images nightly	16:16
jlaska	we've historically relied on the nightly images ... but depending on the state of the repo ... we can go days without any live images	16:16
Oxf13	duplicating that work would be, well, duplicated work	16:16
Oxf13	if the repo is busted for making the nightly images, how am I supposed to make them?	16:16
* jeff_hann here, sorry I'm late		16:16
* wwoods appears		16:16
adamw	as I said in IRC, there's no need for a separate image generation process	16:16
jlaska	wwoods: jeff_hann hey gang	16:16
adamw	just take the nightlies from the appropriate day and stick them in the TC directory	16:16
adamw	just want to have a single 'approved' live image that everyone's working from	16:17
adamw	again, if it's too much work, it's not vital.	16:17
Oxf13	...	16:17
jlaska	yeah, doesn't matter to me which images we use ... just that we have images to test with	16:17
Oxf13	pointing to the live isos at the same time you point to the TC doesn't work?	16:17
adamw	Oxf13: no, because anyone who reads it the next day can't use the same image	16:17
adamw	but they can still get the right TC images	16:17
Oxf13	I suppose I could just make hardlinks	16:17
adamw	Oxf13: the nightlies don't stick around, they get wiped and regenerated every day	16:18
jlaska	would kparal's suggestion of always keeping around the previous successful live image build solve this?	16:18
Oxf13	hardlinks would solve this	16:18
adamw	no, because you still have the problem if the next build is successful	16:18
adamw	it's not that i'm worried a later build won't be 'successful' and no one will be able to test	16:18
adamw	just worried that people would be testing with different stuff, leading to possibly confusing results...	16:19
adamw	(tester A tests with 2010-02-15 and feature B is broken but feature C works, tester D tests with 2010-02-16 and gets the opposite result, world goes 'huh'?)	16:19
jlaska	seems like a reasonable request	16:20
kparal	nicely said :)	16:20
adamw	Oxf13: yeah, a hardlink should do it I guess.	16:20
jlaska	so the issue is making it clearer what people should be testing with ... instead of providing a URL to a directory that may or may not contain images?	16:20
Oxf13	well	16:20
adamw	not just that. as I said, at present, it's not just an information issue	16:21
Oxf13	there is the problem of testing the static image instead of the next nightly	16:21
Oxf13	because the next nightly is what's going to be in the release	16:21
Oxf13	so testing what's older isn't all that useful, if the newer is broken	16:21
adamw	Oxf13: but that applies equally to the non-live bits	16:21
Oxf13	adamw: except we don't generate the non-live bits every day	16:21
adamw	the test compose isn't what's going to be in the release either	16:21
Oxf13	so we don't have the opportunity to test the newer bits	16:21
adamw	still, it's difficult to do consistent testing and be sure about the results if different testers are testing different bits.	16:22
Oxf13	adamw: but we're still in "things change daily"	16:22
Oxf13	mode	16:22
Oxf13	we haven't stopped yet	16:22
Oxf13	these TC images aren't us stopping	16:22
Oxf13	they're just a snapshot of something we haven't yet produced this cycle to see what might be broken	16:23
Oxf13	in those specific things we don't always produce	16:23
jlaska	so adamw, want to summarize what QA is requesting?	16:23
adamw	i think we've wasted enough time already and should just move on	16:23
jlaska	yup ... summarize, next steps and we'll move on	16:23
* jlaska suggests excessive use of #info		16:23
adamw	if it takes this much debate to get it done it's probably not worth doing	16:23
adamw	#info adamw would like a single set of nightlies to be associated with each TC build and kept available in that TC directory during TC testing	16:24
adamw	#info oxf13 doesn't believe it's necessary	16:24
adamw	next topic.	16:24
jlaska	alrighty ... next up, I noted a topic from kparal. But kparal and Oxf13 may have discussed this already	16:25
Oxf13	more like counterproductive to testing the bits that will be in the release.	16:25
kparal	I had almost the same question, this time about Alpha TC naming	16:25
jlaska	Oxf13: it's something QA is asking for	16:25
jlaska	I don't think it's counterproductive	16:25
jlaska	we'll follow this up in a ticket after the meeting	16:25
jlaska	#topic IDEA - Unique ISO image names for each drop	16:25
jlaska	kparal: what's the latest	16:26
kparal	well currently all TC iso's are named the same as the final Alpha iso	16:26
kparal	Oxf13 replied: by not naming the isos etc the same for TC and the alpha release, we risk missing some bug in the behavior of Anaconda and the release name/number	16:26
jlaska	you may have already discussed, but I wasn't clear on how the ISO file names are related to testing itself, was that cleared up?	16:27
kparal	I had similar concerns as adamw, people would be testing something other than we expect. or they might suppose they already have the F13 Alpha final iso (it's called like that, isn't it?), but they won't	16:27
pjones	kparal: Is there some reason to believe we're likely to have that kind of bug?	16:27
pjones	considering that we change the names for every release and haven't been having issues with it...	16:28
jlaska	we have people downloading the old ISO's a lot	16:28
kparal	well, I wasn't sure why the names are not simply unique	16:28
jlaska	normally, we have them validate the CHECKSUMS	16:28
Oxf13	not only that, but the TC stuff is on a completely different mirror system	16:29
kparal	so the Oxf13's explanation is above. I didn't know about anaconda issues, then I suppose it makes more sense	16:29
Oxf13	we can't really protect against stupid people.	16:29
Oxf13	buildinstall does do things with version and product which does change anaconda behaviour	16:29
wwoods	is there a TC# in the metadata anywhere? It'd be simplest to say "current test candidate is TC7" and then if .treeinfo says test-candidate = 6..	16:30
Oxf13	of course, this is one of the dangers of opening pre-release test isos up to more people	16:30
wwoods	but I don't really want to add new data if we don't have to. timestamps work, but people always get confused	16:30
Oxf13	you risk more people getting them who won't use them responsibly	16:30
Oxf13	wwoods: no	16:30
jlaska	Oxf13: I don' think there is bad intent for the images	16:30
Oxf13	wwoods: it's at the top of the URL, but from that point down, by design, it looks exactly like the Alpha would	16:31
Oxf13	jlaska: I don't mean malintent, I mean ignorance	16:31
jlaska	just looking for ways to make sure we are doing what's reasonable to make sure people know they are using the latest test images	16:31
jlaska	okay	16:31
kparal	I think we can move on	16:31
jlaska	kparal: Oxf13: so in summary the filename used for the ISO image files is tightly coupled to the whole build process?	16:31
Oxf13	jlaska: yes	16:31
jlaska	ah okay, not something I knew about	16:32
jlaska	probably one of the new SOPs has this listed	16:32
jlaska	Composing_test_images ?	16:32
Oxf13	the input to the compose process drives the name of the iso, I don't manually make the isos and pick a new name each time	16:32
Oxf13	jlaska: probably Composing_Test_Composes once we make that	16:33
jlaska	Oxf13: ah sweet, thanks	16:33
jlaska	alrighty, kparal anything else to discuss here, otherwise we'll dive into AutoQA	16:33
Oxf13	test_images doesn't have any isos produced (beyond boot.iso)	16:33
kparal	we can go on	16:34
jlaska	alrighty	16:34
jlaska	#topic AutoQA - depcheck update	16:34
jlaska	wwoods: you have the mic ... any updates or milestones to share?	16:34
jlaska	wwoods: we can come back to this later if you prefer?	16:36
jlaska	alright ... moving on to resultsdb ...	16:37
jlaska	#topic AutoQA - resultsdb discussion	16:37
jlaska	kparal, wwoods or jskladan, anything to add here?	16:38
kparal	well I would just mention that we had a small QA team discussion over phone	16:38
kparal	and the results are available in autoqa-devel ML	16:38
* wwoods was afk a sec, deferring to kparal for the moment		16:38
jlaska	#link https://fedorahosted.org/pipermail/autoqa-devel/2010-February/000201.html	16:38
jlaska	wwoods: okay	16:38
kparal	we are currently studying other projects, how they define their results database, etc	16:39
kparal	so any input can be sent to autoqa-devel	16:39
kparal	that's all I suppose	16:40
jlaska	kparal: okay, thanks for the update	16:40
jlaska	#topic AutoQA - beakerlib	16:40
wwoods	the current notion - if memory serves - is that we want to provide one unified resultsdb that al the autoqa tests can use to report (somewhat simplified) results	16:40
wwoods	which will give us one place to view all test results - and ideally a few nice views of that data.	16:41
jlaska	#topic AutoQA - resultsdb discussion	16:41
jlaska	#info we want to provide one unified resultsdb that all the autoqa tests can use to report (somewhat simplified) results	16:42
jlaska	one sec ... meeting tags	16:42
jlaska	#info had a small QA team discussion over phone last week (results on autoqa-devel ML)	16:42
wwoods	I think we had some example views we wanted to see - test results for a given package, test results for a given installable tree, all test results for package builds / updates for a given day, etc.	16:42
wwoods	if you can think of a good use case you'd like - a specific view of the autoqa results data - please let us know	16:42
jlaska	nice, thanks wwoods and kparal	16:43
wwoods	further discussion will be on autoqa-devel.	16:43
wwoods	oh, and as for depcheck: no progress, owing to the fact that I was out of the office and then sick all last week	16:43
jlaska	status - plague :(	16:44
jlaska	#topic AutoQA - beakerlib	16:44
jskladan	ok, time for me to speak, i suppose :)	16:44
jlaska	Adding a new topic here for the work jskladan and kparal have been looking into around beakerlib	16:44
jlaska	any updates or problems you are experiencing?	16:44
jskladan	well, idk if all af you are familiar with beakerlib	16:45
jlaska	#link https://fedorahosted.org/beakerlib/	16:45
jskladan	its a set of libraries which RHEL guys use for writing automated tests	16:46
jskladan	so we'd like to reuse those in out testing system (autoqa)	16:46
jskladan	today kamil and me created a simple test, which demonstrates usage of beakerlib-based tests	16:47
jskladan	#link http://jskladan.fedorapeople.org/beakerlib_helloworld.tar	16:47
jskladan	it's nothing fancy - in fact, it only makes sure, that you have beakerlib installed, and then runs the test itself	16:47
jlaska	nice, using the provided beakerlib vim syntax highlight! :)	16:47
jlaska	rlAssertRpm "setup"	16:48
jlaska	rlAssertExists "/etc/passwd"	16:48
jlaska	rlAssertGrep "root" "/etc/passwd"	16:48
jlaska	cool	16:48
jskladan	and of course stores the report produced by beakerlib in the directory for autotest-lib result files	16:48
jskladan	we'll be trying to assimilate other tests this week - probably init script testing stuff - and look into some more complex uses	16:49
jlaska	this is great news, it's moving along faster than I had expected	16:50
jlaska	nice work :)	16:50
jskladan	so that's about it for now - if kamil does not have anything to add	16:50
jlaska	#info today kamil and me created a simple test, which demonstrates usage of beakerlib-based tests	16:50
jlaska	#info we'll be trying to assimilate other tests this week - probably init script testing stuff - and look into some more complex uses	16:50
kparal	jskladan described it all	16:51
jlaska	alright thanks for the update gang	16:51
jlaska	I'll do a quick run through on the 2 remaining AutoQA topics	16:51
jlaska	#topic AutoQA - install automation	16:51
jlaska	#info Last week, Liam summarized current DVD automation efforts in an email to autoqa-devel	16:52
jlaska	#link https://fedorahosted.org/pipermail/autoqa-devel/2010-February/000199.html	16:52
jlaska	Automating media (CD/DVD) installs using virt is funky b/c you have to do some strange stuff to provide your own boot arguments	16:52
jlaska	so Liam outlined several techniques he's tried	16:53
jlaska	including using dogtail to add custom boot args using virt-viewer	16:53
jlaska	any feedback folks have on those proposed solutions would be good to have	16:53
jlaska	once back from Chinese new year, he'll be working to integrate the best method into the current test	16:53
* Oxf13 has to bail, need to prepare for next appointment		16:53
jlaska	and last up ...	16:54
jlaska	#topic AutoQA - packaging	16:54
jlaska	more small progress last week on identifying new build dependencies	16:54
jlaska	it exploded to 50+ missing deps at one point, but akurtakov helped identify how to address that	16:54
jlaska	This seems like the list of deps isn't every finalized :(	16:55
jlaska	I'd really like to squash this so I can start making packaging progress	16:55
jlaska	any ideas would be appreciated	16:55
jlaska	My current plan, is to see if akurtakov (and others) can dedicate some time on IRC To help walk the list of missing deps once and for all	16:55
wwoods	where's the work-in-progress stuff for people who want to help out?	16:56
jlaska	#link User:Jlaska/gwt	16:56
adamw	i would suggest you make progress by starting packaging	16:56
adamw	the only way you can ever be sure about the deps list is when you actually start packaging stuff and find out if it works	16:56
jlaska	adamw: I'd like to as well, but I feel like I still don't know how long the packaging road is	16:57
jlaska	I'm trying to find the leaf nodes to package ... and every time I look, I find another deps branch	16:57
jlaska	so I'll revisit this week, and work towards adamw's suggestion ... just start packaging :)	16:58
jlaska	#info current plan, is to see if akurtakov (and others) can dedicate some time on IRC To help walk the list of missing deps once and for all	16:58
jlaska	#info I'll revisit this week, and work towards adamw's suggestion ... just start packaging	16:58
jlaska	alright ... open discussion time	16:59
jlaska	#topic Open discussion - <Your topic here>	16:59
jlaska	anything not previously mentioned that we need to discuss today?	16:59
jlaska	If no suggestions, I'll close out the meeting in 1 minute	17:00
adamw	nup	17:01
jlaska	alrighty ... thanks for the updates everyone	17:01
jlaska	as usual, I'll send minutes to the list for review	17:01
jlaska	#endmeeting	17:01

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!