QA/Meetings/20081008

From FedoraProject

Jump to: navigation, search
wwoods okay! let's talk about the Beta for a bit. 08:06
wwoods It's out. Yay. 08:06
wwoods Some good testing has been going on, and I've been trying to keep track of stuff reported against the Beta 08:06
* f13 08:06
wwoods but a question occurred to me - do we care about keeping track of the major Beta issues as a group 08:07
* jds2001 hasn't had a ton of free time, but my last few rawhide attempts have gone up in flames. 08:07
jlaska wwoods: how do you mean? 08:07
wwoods like, should we continue to maintain the F10Beta tracker to keep track of Common Bugs reported against the beta 08:08
jds2001 interesting idea. 08:08
wwoods (or some other way of keeping track of F10Beta common bugs, separate from the big F10Target/F10Blocker lists) 08:08
jds2001 kind of 'go here for common issues we already know about' type of thing? 08:08
wwoods right. For the final releases we typically have a CommonBugs page 08:08
f13 don't we have a common bugs page for beta already, linked from the release notes? 08:09
jlaska http://fedoraproject.org/wiki/Releases/10/Beta/ReleaseNotes#Known_bugs_and_issues 08:09
jlaska only 2 issues :( 08:09
f13 there ya go, a place to track the common bugs we know are in the shipped Beta 08:10
mcepl jlaska: I would venture to say that we have more problems than that ;-) 08:11
wwoods wonderful 08:11
f13 the bugs themselves should be moved over to the appropriate tracker. 08:11
wwoods yeah I've been putting them on F10Blocker/F10Target as appropriate 08:11
wwoods but wanted a summary page somewhere. the release notes seems as good a place as any 08:12
f13 worksforme 08:12
stickster https://fedoraproject.org/wiki/Common_F10_bugs 08:12
wwoods We could use that page too, so long as we maintain it carefully 08:12
jds2001 stickster: is that the common bugs page for final? 08:14
stickster jds2001: It will be, yes 08:14
stickster So the Intel entry is deprecated, I suppose 08:14
wwoods so should we use that to track F10 bugs from Beta on 08:15
wwoods or just leave it for a roundup at the end? 08:15
poelcat how about the Common Bugs page points to the blocker/trackers ? 08:16
jlaska wwoods: it'd be great if we could maintain a wiki page with the bugs ... but I fear that might be too much mgmt of wiki ... 08:16
jlaska yeah ... I was just going to suggest putting a link in that wiki page to the blocker dependency bugzilla page? 08:16
wwoods but *which*? 08:17
wwoods F10Blocker? F10Preview? F10Beta? 08:17
wwoods how do we track the stuff that's open in F10Beta but closed in Rawhide since then? 08:17
f13 in my opinion, we need different wiki pages, or sections for each milestone 08:18
poelcat list them all... removing them as they are no longer relevant 08:18
f13 we definitely need an ongoing record of known bugs in Beta (until the next milestone is out) so that people who install beta, see that their bug is known. 08:18
jlaska yeah, I like the break it out for milestones approach 08:18
f13 but for rawhide users, or the next milestone user, the bug may not be there, we don't want it listed 08:18
wwoods poelcat: I don't think that scales 08:18
poelcat no, list the links 08:19
f13 but if we remove it, then people installing beta will hit it, not see it listed, etc... 08:19
wwoods poelcat: that doesn't help 08:19
jlaska we can just drop it from the blocks:F10Beta 08:19
jlaska err nm 08:19
stickster The purpose of the "Common F10 bugs" page needs to be a readable summary for non-experts. That may not fit what you guys need. 08:19
jds2001 why not, link to the bug, and they can see the status from there. 08:19
wwoods giving people a link to all the trackers does not tell them "yes, we know that f10beta will wait for 30sec if your CD drive is empty" 08:19
jds2001 there's even a rhbug:123456 notation in the wiki 08:20
jlaska if we have a manageable (smallish) list of bugs ...I suspect we could engage folks with doc skills to help word smith the Common_BUgs entries 08:20
stickster jlaska: darn tootin' 08:20
jds2001 stickster: you're sounding like you're from the midwest, not DC :) 08:21
stickster oh yah 08:21
wwoods maybe we should just set the fedora_requires_release_note flag on bugs that should go into the common_bugs page? 08:22
jlaska I think that's a great idea 08:22
wwoods I mean, that's kind of why we have it.. right? 08:22
jlaska wwoods: yeah, you got it 08:22
jlaska do we have folks in thne fedora space that monitor that queue of fedora_requires_release_note? 08:23
jds2001 there's even some list that gets email notification of that flag being requested :) 08:23
poelcat why not just call the whole thing "known rawhide bugs?" 08:23
wwoods stickster might know more about that..? 08:23
jds2001 i think that quaid does that, not sure. 08:23
poelcat people keep telling me that 3 or 4 days after "the beta" is released it is no longer relevant... "use rawhide" 08:23
stickster wwoods: Yes, relnotes@fp.o writes to the fedora-relnotes-content list when that happens. 08:23
stickster http://www.redhat.com/mailman/listinfo/fedora-relnotes-content 08:24
wwoods poelcat: it's relevant as a way to get rawhide on your system. we've had this conversation a dozen times. 08:24
poelcat wwoods: i guess we keep having it because it has never been answered in a way most people can understand :) 08:25
f13 poelcat: here is the work flow. 08:25
f13 poelcat: people install beta, they run into bugs. Often times they file those bugs or bother people on forums/IRC about them. 08:25
f13 poelcat: they get told to update, since it's fixed in rawhide. 08:25
wwoods maybe we need to rename the post-Beta snapshots 08:26
wwoods this week's snapshot should be Beta 2 08:26
f13 poelcat: if we had a place and led people to it that would list the known bugs they're going to encounter with a fresh beta install, that also shows them that it has been fixed since beta, they can update (when possible due to broken deps), see their bug fixed, and move on. 08:26
wwoods increment the number 'til we hit Preview 1 08:26
f13 wwoods: erm... I didn't really want to use that naming scheme since it's a much much different distribution method for snapshots 08:27
f13 torrent only, potentially smaller payload 08:27
f13 and it's a trial (twice, thursday/friday) and if it still doesn't work, punt until next week 08:27
f13 not guarenteed to actually show up 08:27
wwoods right 08:27
wwoods so here's the thing, poelcat 08:27
wwoods you're not being told that Beta is irrelevant 08:28
viking_ice why not just drop the whole alpha beta etc and have just rawhide-snapshot-$date ? 08:28
wwoods it's actually the case that the bug is fixed and it'll be in the Preview 08:28
wwoods but if you *want* you can use rawhide to get early access 08:28
wwoods now. since rawhide is public and just as easy to install.. everyone uses that instead 08:28
f13 viking_ice: people and press don't look at those. 08:28
f13 the Beta iso set is not irrelevant. It provides a relatively known stable install point 08:29
f13 just installing and immediately testing the Beta is not very relevant, particularly the older it gets 08:29
wwoods that doesn't negate the relevance of the Beta, it just provides early access to fixes destined for Preview 08:29
f13 installing, /updating/, and then testing is absolutely relevant. 08:30
wwoods right. but reporting bugs about the Beta itself is not real useful. but apparently people aren't getting this message. 08:30
jlaska wwoods: I think that's just it 08:31
wwoods so how do we make that plainer? I think the obvious first step is: keep a list of known bugs 08:31
viking_ice like always better documentation would fix that 08:31
wwoods bugs *in* beta, and their current status 08:31
jlaska I don't think this is the case ... but if we don't want people ti file bugs against the beta, they need ot rawhide update first ... which opens up the whole rawhide failure loop 08:31
f13 right 08:32
wwoods so. F10Beta should continue to be used for tracking Beta bugs post-release 08:32
f13 for the love of god, we need a document on people's desktops that explains this. 08:32
poelcat jlaska: +1 08:32
jlaska  ;) 08:32
wwoods f13: seriously 08:32
poelcat f13: or we need a better/different process 08:32
jlaska f13: wwoods: I understand the model 08:32
f13 poelcat: hey I'm all ears. 08:32
wwoods poelcat: write up a proposal for a better model and we'll talk about it 08:32
jlaska I'm just saying that I can see how for a newbie the interaction between the rawhide and a milestone can be hazy 08:33
wwoods right, so let's unhaze it 08:33
viking_ice why not just have the default web pages in the browser common_f10_bugs ? instead of search ? 08:33
wwoods desktop document / panel applet that says "here's the current list of known bugs, don't bother reporting these, just upgrade your system already" 08:34
viking_ice what the first thing that people do after install go to the internet.. 08:34
viking_ice easy-er to maintain.. 08:34
viking_ice then just change it for final 08:34
viking_ice back to search 08:34
wwoods What controls the default page? 08:34
jlaska it is an interesting model ... here install the beta which ideally we put a lot of effort into, then update your system to rawhide which has no guarruntee of success. 08:35
f13 wwoods: start.fedoraproject.org 08:35
f13 wwoods: we could possibly trigger on client headers and direct them to a different page 08:35
jlaska we definitely want folks to be running the latest-greatest ... it just often comes with warts 08:35
wwoods jlaska: post-beta that's not supposed to be the case 08:36
viking_ice f13: was going to suggest that.. 08:36
wwoods post-beta people aren't supposed to be putting destabilizing changes into rawhide 08:36
f13 man, if only 08:36
poelcat but everyone knows that isn't the case 08:36
wwoods so "no guarantee of success" != "guaranteed unstable as all hell" 08:36
* f13 cries a little 08:36
jlaska f13: cry for baby jesus :) 08:36
wwoods okay, fine, rawhide sucks. what do you want to do about it? 08:36
viking_ice complain... 08:37
* viking_ice ducks...+ 08:37
jlaska just thinking out loud ... I'm not providing any alternatives or trying to break things down 08:37
wwoods we're 30 minutes into the meeting and I'm hearing pushback on the very idea that people should *update their systems* after Beta 08:38
wwoods so, okay, fine. What would be better? 08:38
viking_ice to get rawhide more stable we need more testing on rawhide.. simple 08:38
wwoods and how do we accomplish that? by *not* having people update to rawhide? 08:38
wwoods automated testing? great. who's writing those tools? 08:38
viking_ice more reports == more bugs == same amount of people behind the project == we need more developers.. 08:38
wwoods let's take it as read that the biggest QA problem we have right now is that rawhide isn't stable enough 08:39
wwoods because that's what I'm hearing 08:39
wwoods put everything else aside and give me suggestions on making rawhide more stable 08:39
viking_ice developers put their own baby's to the test on rawhide 08:39
wwoods except developers are all savvy enough to deal with instability 08:40
viking_ice should filter out simple mistakes typing errors etc.. 08:40
wwoods a lot of our devs *do* run rawhide pretty much all the time 08:40
f13 this is the age old argument. 08:40
jlaska true true 08:40
f13 We ask developers to produce better builds for rawhide 08:40
wwoods obviously we can't just say "hey. make it more stable KTHX" 08:41
f13 they ask us to develop pre-testing environments to prevent bad builds from going into rawhide 08:41
f13 we stare at eachother from across a table. 08:41
wwoods pre-testing environments. okay. so we want automated testing, then? 08:41
wwoods rawhide builds that don't pass the test don't get synced out? 08:41
wwoods cool. who's writing that? 08:41
f13 right, like I said, it's the age old argument that isn't going to resolve itself here in this meeting. 08:42
wwoods or are there other things we should be doing? 08:42
* viking_ice is taking early steps in python slowly getting there.. 08:42
f13 and I don't even think that's the right solution (preventing the rawhide push) 08:42
wwoods that's not a rhetorical question - seriously, if that's the biggest problem we have, then that's what we're going to work on as soon as F10 is out 08:42
jlaska based on our FUDCon talk ... I guess that depends on what aspect of rawhide fails (the install source vs yum repo issue) 08:42
wwoods until F11a or F11b 08:43
wwoods at which point we are invariably overwhelmed by actual testing and bug tracking and, y'know, this conversation happens again 08:43
wwoods f13: so what's the solution? 08:43
f13 wwoods: I don't necessarily have one. 08:44
wwoods I want at least two automated systems: 1) rpmdiff for each build, and 2) a set of automated tests for nightly anaconda image builds 08:44
f13 wwoods: however, are there not people within Red Hat working on automated testing systems? 08:44
wwoods that's what I was told 08:44
wwoods haven't heard a peep since FUDCon 08:44
pjones rpmdiff is mostly hype, though. 08:44
f13 "rpmdiff like" 08:45
f13 perhaps not 'rpmdiff' as you know it. 08:45
wwoods I'm not suggesting literal rpmdiff 08:45
f13 to state it better, we want 1) post-package-build testing of package, and 2) automated tests for nightly rawhide trees. 08:45
wwoods but I want to know when files are dropped/added to packages, or change perms, or sonames 08:45
pjones f13: yeah, that does sound better. 08:46
wwoods especially the latter, which should trigger a mass of emails and/or automated rebuilds 08:46
stickster wwoods: What about the stuff dmalcolm put out recently? 08:46
wwoods stickster: what stuff? 08:46
f13 once we have this magical world where developers can get automated feedback on the build they just did, we can go into a mode where builds don't automatically get tagged. 08:47
f13 the build goes through, gets tested, and IF the maintainer is happy with it, the maintainer (or somebody else) can fiddle a bit and that build is now tagged. 08:47
stickster I'm pretty sure he recently released some sort of initial code for doing a bunch of in-depth RPM analysis, at the symbol level 08:47
jlaska https://fedorahosted.org/rpmfluff/ 08:47
f13 wow what an unfortunate name. 08:47
jlaska  :) 08:48
wwoods stickster: no, that's a tool for creating sets of RPMs with weird dependencies or conflicts or whatever 08:49
wwoods it's for testing tools that handle RPMs - yum, rpm, packagekit, etc. 08:49
f13 going to a mode where builds don't automatically get tagged, we'd be closer to a sourcecontrol world, where changes committed don't automatically get tagged for "stable" 08:49
wwoods stickster: it doesn't actually analyze RPMs at all, which is what we're talking about 08:49
stickster https://fedorahosted.org/rpmgrok/ 08:50
stickster There you go ^^ 08:50
f13 In my opinion, aside from doing the actual testing, the most important things the Fedora project needs is a tool to manage quality tests cases/results, and an automated system to run said tests and report said results. 08:51
stickster "rpmgrok is a web application for browsing the payloads of a large collection of RPM software packages. It digests a set of input RPMs, analysing the metadata and payload, and storing the results in a database. There's a web UI for viewing the data, an XML-RPC interface for querying it, and a command-line tool for using the XML-RPC interface." 08:51
wwoods interesting. that's more of a distro-wide info-viewer, although you could probably plug that into bodhi and hit it with a wrench until rpmdiff fell out 08:51
wwoods f13: right. and that ain't gonna be Testopia, apparently :/ 08:51
f13 secondary to that we need to get more frequent and better information to our developers as they complete builds 08:52
stickster wwoods: I think rpmgrok will do what you asked for though -- filename changes, sonames, etc. 08:52
f13 throwing it over the rawhide wall and waiting for the users to report bugs has "worked" for a number of years, but could be so much better. 08:52
pjones f13: really the test infrastructure needs to be part of the build system, at least from the developer's point of view 08:53
f13 pjones: correct. 08:53
f13 pjones: my vision is koji reports a build complete, the test infrastructure gets this notice and runs the available/configured tests on the build and reports the results. 08:54
f13 and we can play games with when/if a tag is applied to the build somewhere in that timeline. 08:54
wwoods pjones: do you have a suggested structure for the dev view of the testing stuff? 08:54
wwoods mostly I'm wondering about where devs put their tests 08:55
wwoods you'd think we'd want a %test section in the spec, or a test dir in package CVS, or similar 08:55
pjones well, there are three aspects: scratch builds, non-scratch builds, and trees 08:55
f13 pjones: 4 really. 08:55
pjones the middle is the easiest one -- best case, if a problem is detected, it results in a build failure. 08:55
f13 pjones: scratch builds, tagless builds, tagged builds, freeze tags. 08:55
pjones f13: man, we really have too much jargon. which of those is which? 08:56
f13 pjones: I really don't want to have koji have to know specifics about a test infrastructure that would come between a build "complete" and a build "failure" 08:56
f13 pjones: scratch-build is an anonymous build in koji, can come from srpm, no restrictions on what the package is, where it comes from, and what it's built against. 08:57
f13 scratch-build isn't recorded in the database, the output isn't kept. 08:57
pjones f13: without that, I fear we'll have a situation like we have with rpmdiff now, where it's not really part of the process, and developers get to it as an afterthought 08:57
f13 tagless build is in all ways a "normal" build, comes from package scm, recorded in the database, must be a known package in a known tag. 08:57
pjones oh, by tagged and untagged you meant WRT CVS 08:57
pjones or not. 08:57
f13 the kicker with a tagless build, is that the final step of the build process, applying a tag to the build (eg dist-f10), isn't done. 08:58
pjones ok 08:58
pjones so it's basically a second form of scratch build 08:58
f13 the build completes, the rpms are stored, but the tag itself isn't applied 08:58
pjones except with some more rigorous restrictions 08:58
f13 pjones: with a scratch-build, you can never tag it 08:58
f13 pjones: with a tagless build, you can go back and apply a tag 08:58
pjones ok. 08:58
f13 so you tagless build, you run said build through testing environment 08:59
f13 if it passes, or if the developer forces, the tag gets applied 08:59
pjones right. 08:59
f13 so in essense, you have your "failure" if it doesn't pass qa 08:59
pjones right. 08:59
f13 but we don't have to teach koji about a testing environment 08:59
f13 all of that logic can live outside the koji codebase. 08:59
wwoods hey, that's pretty clever. 09:00
f13 wwoods: it was a design goal when koji was written. 09:01
pjones f13: I'm still worried about the rpmdiff issue, where you can basically ignore it if you choose to. 09:01
f13 pjones: that comes down to policy 09:01
f13 pjones: but if we go this route, if you ignore rpmdiff, your build doesn't get tagged, doesn't show up in rawhide, doesn't show up in the release 09:01
pjones right. 09:02
wwoods okay, so. some assumptions for the test system. will it be in the same lab as the buildsys? should we be able to assume filesystem (NFS, etc) access to the builds? 09:02
f13 we can send email until we're blue in the fingers if you've got a pending build, but unless the maintainer does something about it.... 09:02
pjones that can work. I just don't want a process like the RHEL errata process, where I have to go ask somebody to unlock something so that I can go fix something I accidentally skipped. 09:02
f13 pjones: no, as long as I'm at the helm, the ability to do cvs and do builds of some kind will never be locked down. 09:03
f13 pjones: I've chosen to draw my line in the sand at the tagging of the completed build level, no lower. 09:03
pjones ok. 09:03
pjones good :) 09:03
f13 I've listened to enough people rail on the RHEL process to follow that disaster. 09:03
f13 wwoods: good quesitons for mmcgrath 09:04
f13 wwoods: ideally it would be in the PHX infrastructure, where the koji filestore is, and that you'd have either nfs, or http access to the builds. 09:04
f13 nfs-readonly is probably doable. 09:05
wwoods will that be access to the entire koji db? ideally we want to be able to query, you know, what tag this build wants, so we can find the previous package with that tag 09:05
wwoods so we can do diffs 09:05
f13 the koji db is public 09:06
f13 you can do anonymous queries like that via the API 09:06
f13 so you'd have both access to the API to get information, and access to the filesystem where the packages are stored to poke at the rpms themselves. 09:06
wwoods okay, but you need to have the newly-built package, and some idea of what tag to look at 09:08
wwoods anyway, implementation detail 09:08
wwoods in the long run this should help make rawhide more stable 09:08
f13 wwoods: you'd have that. 09:09
f13 via the api 09:09
wwoods f13: no, I mean, the notification to the test system needs those two things 09:09
wwoods the way you described it, new builds would be tagless 09:09
wwoods so something has to pass that buildID and, I guess, the target tag 09:10
f13 wwoods: they would be tagless, however you'd have information in the builddata about what they were targetting 09:10
wwoods ah - the builddata would still contain it, even if it's not applied? 09:10
f13 you /target/ dist-f10, you just don't apply the final tag. 09:10
f13 a build target has 3 peices of data. 09:10
f13 the target name, the buildroot tag to use, and the tag to apply to builds that complete. 09:11
wwoods "target name" is.. the package name? 09:11
f13 $ koji list-targets --name dist-f10 09:11
f13 Name Buildroot Destination 09:11
f13 --------------------------------------------------------------------------------------------- 09:11
f13 dist-f10 dist-f10-build dist-f10 09:11
wwoods ahhh 09:11
wwoods okay, I get it now. 09:12
f13 when you do a build from the devel/ branch, it executes: koji build --target dist-f10 cvs://path 09:12
wwoods so the other thing we talked about is automated testing of installer images 09:12
f13 yep. 09:13
f13 <future world> 09:13
f13 compose process goes through the rawhide compose, builds images, and stages them on the kojifilestore. 09:13
f13 compose process drops a message on our communication bus that says "New tree, path here" 09:13
f13 testing system hears the notification on teh bus and starts it's pre-configured 'rawhide' tests on the location 09:14
f13 results from testing are dropped on the bus 09:14
f13 sync tool gets results from bus, makes decision about what parts of the tree gets synced to the master mirror 09:14
wwoods I'd think we'd want a simpler test input, though - it'd be better if anaconda devs could kick off automated tests without needing to wait for the rawhide compose 09:15
f13 information about the whole compose, including test results and whether or not images were synced dropped on the bus. 09:15
f13 notification bot gets this info from the bus, sends various mails, rss notices, etc.. 09:15
f13 </future world> 09:15
f13 wwoods: there is no reason why we couldn't manually schedule a run of tests on a tree location 09:16
wwoods but that requires composing a tree 09:16
f13 or even upload a boot.iso via a webform to run a manual set of tests on 09:16
wwoods especially with the way that stage2 works these days, you'd think you could take, as inputs, kernel/initrd + tree URL 09:17
wwoods or boot.iso + tree URL, sure 09:17
f13 the trick is coming up with the boot.iso 09:17
f13 IE stage1/2 made from the changes they're attempting 09:17
f13 given the way that buildinstall works, that's not an easy task, since the goddamn thing pulls a anaconda package from a repo, exploads it, and re-executes itself from the exploaded package. 09:18
wwoods right, but we can work on that within anaconda 09:19
wwoods there's supposedly a makefile target to build a boot.iso 09:19
wwoods but it's unreliable 09:19
f13 right 09:19
f13 my above scenario was focused on the nightly compose 09:19
wwoods right, which is also an important thing 09:19
f13 since we could make a decision there to sync the new repos and packages, but hold back bad images. 09:19
wwoods but ideally this test service should be able to handle either one 09:20
f13 yep 09:20
f13 I wouldn't think of it any other way 09:20
wwoods so clumens and dcantrell and friends can throw bits at it until something sticks, and then *that* goes through the compose 09:20
f13 realistically, we're likely going to need to provide them shell access on a machine close to the test system, because asking them to upload stage2 images from their house (dcantrell), or even the bos office (clumens) is going to introduce a lot of lag. 09:21
f13 or we make the test system capable of having nodes in their house, and in the bos office 09:22
f13 so that they can run the tests locally with local input 09:22
wwoods right, I want this system to be something you can instantiate lolcally 09:23
wwoods err, locally. lol-cat-ly 09:23
wwoods running on bare metal will require things like a PXE/tftp server for recovery 09:24
wwoods so the initial test run might be virt stuff 09:24
wwoods anyway, we're waaay over our normal meeting time 09:24
wwoods and I need food before my brain leaks out my ears 09:24
wwoods so let's summarize 09:25
wwoods 1) we need to track Beta common bugs and put them somewhere, like the Beta release notes page 09:25
wwoods 2) people need to be informed about that page, and about the fact that many bugs will be fixed by updating 09:26
wwoods 2a) we need to figure out a way to get that info onto people's desktops for pre-releases 09:26
wwoods 3) if we're going to continue encouraging people to update their Beta/etc. systems to Rawhide, we need to make Rawhide more stable. 09:27
wwoods 3a) we should have a per-build test system attached to koji that runs tests on each package built - before the package is tagged in koji - and can refuse to tag the package if the test fails. 09:28
wwoods 3ai) Devs can still force-tag stuff that fails tests, but they do so at their own peril. 09:29
jwb erm.. 09:30
wwoods 3b) we should have an installer test system that takes the nightly compose and runs some "rawhide" tests on it - if the tests fail, we don't sync out the broken images/data 09:30
wwoods Does that cover it? 09:30
jwb shouldn't 3a be done by the developer before they submit the build to koji? 09:31
wwoods jwb: we're talking about koji tags here, not CVS tags, and only the koji tags that normal devs can apply, like 'dist-f10' for rawhide builds 09:31
jwb i think (maybe) my comment still applies 09:32
wwoods jwb: so you run 'make build', koji builds your new package, but doesn't tag it 'dist-f10' until the tests pass 09:32
f13 jwb: please read some of the backscroll. 09:32
jwb yeah ok 09:32
jwb ignore me for now 09:32
f13 jwb: essentially it's a "scratch build" until the tests pass, or the developer forces the tag application. 09:33
f13 with the bonus that the developer wouldn't have to re-submit the build "officially" if the tests passed. 09:33
jwb this is for post-Beta, or all the time? 09:33
f13 up for discussion 09:33
f13 I'd push hard for at /least/ post-beta 09:33
f13 but there could also be different "pass" criteria for different periods of the development cycle. 09:34
f13 more lenient earlier, more strict later. 09:34
wwoods it depends on us writing some infrastructure bits first 09:35
wwoods but, basically, ASAP 09:35
wwoods stuff like: hey your library changed soname 09:36
wwoods should probably require some intervention / notifications to maintainers of dependent packages 09:36
f13 yeah, this brings in a wrinkle. 09:36
wwoods *immediately* on package build, not during the nightly compose 09:36
f13 chain-builds 09:36
wwoods I think it's obvious that we move too fast to wait for nightly builds for all our tests 09:36
f13 chain-builds allow you to set up a chain of N+ packages, say 'build foo, once done and in buildroots, build bar, baz, and bangle' 09:37
jwb where are these tests coming from? 09:37
f13 since the building of foo will break bar, baz, bangle. We'd probably want to be able to allow those builds, and do the testing on the whole set. 09:37
f13 jwb: combination of common tests for all packages, and then developer/QA generated tests that hopefully live near hte package SCM 09:38
f13 or in the package SCM 09:38
f13 like a tests/ subdir for each branch dir 09:38
jwb ok... so for the kernel, you'd run LTP? 09:38
f13 *shrug* up to the kernel people 09:38
wwoods that's a little ambitious 09:38
jwb so is this whole idea 09:39
wwoods not really 09:39
wwoods the current test suite consists of nothing, so everything passes 09:39
wwoods so don't freak out just yet 09:39
f13 jwb: sadly it's something that Red Hat has had in one form or fashion for a very long time, and Fedora lost it when we moved Fedora out of Red Hat infrastructure. 09:39
jwb i'm not freaking out 09:39
jwb i think the idea is great. but i also think it's ambitious 09:40
wwoods the initial test suite would probably be something like: did we lose any files? did anything change perms? 09:40
f13 and Fedora has suffered because of this 09:40
wwoods it's really not that ambitious. we're just adding an "are you sure?" step for builds where weird things are happening 09:40
f13 "did this build break deps?" 09:40
wwoods everything beyond that is up to the package maintainers discretion 09:40
f13 I think we're finally to the point in Fedora that focusing on this is our #1 priority 09:42
f13 we've got the opensource buildsystem thing down, we've got good infrastructure now, we've solved some of our storage issues 09:42
wwoods stickster specifically called out QA as a priority for improvement 09:43
f13 we've got compose tools pretty well done 09:43
stickster f13: agreed 09:43
wwoods building good QA depends on everything else being fairly stable already 09:43
f13 it's time to focus on getting the QA superstructure built up. 09:43
wwoods and I think we're just now finally reaching that point 09:43
wwoods luckily we've got a *lot* of really good ideas 09:43
wwoods and a lot of really talented people 09:43
f13 it's taken a while, but goddamn does 6 month release cycles drain a lot of energy :/ 09:43
wwoods but, y'know, no matter how many women you hire, it takes 9 months to make a baby 09:43
wwoods f13: for serious 09:44
f13 wwoods: not according to Fringe! 09:44
wwoods my wife advises people not to talk to me during October and April 09:44
wwoods heh 09:44
wwoods okay, so, aside from tracking bugs for PR and doing Rawhide testing 09:45
wwoods which should be first priority - per-build tests or per-compose installer tests? 09:46
f13 I feel like per-compose installer tests because it seems like those could more easily be done hodge-podge 09:48
wwoods also I'd like to ensure - and have proof - that rawhide is installable more of the time, so people stop using anaconda as a scapegoat 09:49
wwoods unfortunately I think this one is the tougher problem.. but hey, I've got two machines with virt stuff handy 09:49
wwoods so I'll see what I can get cooking 09:50
wwoods okay. anything else we need to discuss? 09:50
wwoods oh! right! Features. 09:50
-!- wwoods changed the topic of #fedora-meeting to: Fedora QA | Feature testing 09:50
wwoods bpepple (on behalf of FESCo) has asked about testing accepted features so FESCo can review ones that need review 09:51
wwoods https://fedoraproject.org/wiki/Category:FeatureAcceptedF10 has the feature list 09:51
wwoods https://fedoraproject.org/wiki/QA/TestResults/Fedora10Features/Beta has the ongoing results (which needs updating) 09:52
wwoods if you can provide a quick status on any of those things 09:52
wwoods or want to nominate something for a Test Day 09:52
wwoods *please* let me know.. or just go ahead and do it 09:52
-!- wwoods changed the topic of #fedora-meeting to: Fedora QA | misc 09:55
wwoods anything else? 09:55
* wwoods takes that as a no 09:56
wwoods okay - thanks for the time and discussion, folks 09:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!