Testing/Logs/20061109

Note: this log may go away if I find a good place to host the (much more readable) html-ized log.

--- Log opened Thu Nov 09 12:06:13 2006 12:06 plus some convert to HTML script? 12:06 can someone point me to the agenda on the wiki ? 12:06 dmalcolm: http://mg.pov.lt/irclog2html 12:06 <@wwoods> craigt: it's in the topic 12:07 <@wwoods> http://fedoraproject.org/wiki/Testing/Meetings/20061109 12:07 doh! thanks 12:07 dmalcolm: i need your python skills to make it read xchat better... right now it does irssi 12:07 [Users #fedora-testing] 12:07 [@ChanServ] [ craigt  ]  [ ixs    ]  [ rdieter] 12:07 [@wwoods ]  [ dmalcolm]  [ poelcat] 12:07 -!- Irssi: #fedora-testing: Total of 7 nicks [2 ops, 0 halfops, 0 voices, 5 normal] 12:07 * dmalcolm hides 12:08 <@wwoods> okay, I'm going to get started. 12:08 <@wwoods> So, hi and welcome to the first-ever Fedora Testing meeting, yay! 12:08 <@wwoods> The agenda, as mentioned, is at: http://fedoraproject.org/wiki/Testing/Meetings/20061109 12:09 <@wwoods> First thing: a quick FC6 review 12:09 <@wwoods> We delayed.. 3 times, if memory serves 12:10 <@wwoods> Since there's just the half-dozen of us I'll ask: who did FC6 testing prior to release? 12:11 * dmalcolm was running rawhide up to the release 12:11 <@wwoods> dmalcolm: cool, that definitely counts. 12:11 but that covers daily changes, rather than fresh installation/upgrade from FC5 12:12 BTW do you regard FE6 as on-topic for the discussion? 12:12 <@wwoods> right, I suspect that most testers are doing that 12:12 <@wwoods> dmalcolm: sure 12:12 I only did minimal (< 2 hrs) playing with fc6t3 clean install one afternoon. 12:12 I hope that we're going to converge Core and Extras, so anything we say will hopefully apply to the Brave New Amalgam thing 12:13 * craigt ran rawhide aslo 12:13 <@wwoods> dmalcolm: right. was there something in particular about FE6 before release? 12:14 IIRC there were plenty of autogenerated emails going to fedora-extras-list during the FE6 timeframe from Fedora Extras repoclosur e  12:14 <@wwoods> Quite a few of the bugs that we held up FC6 for were PPC-specific 12:15 <@wwoods> and that's just because.. we have very, very few ppc users 12:15 <@wwoods> and even fewer testers 12:16 <@wwoods> is it worth the effort to track down some ppc users and try to make them Official Testers in some capacity? 12:16 re ppc: of those users, is there a specific goal? I'm thinking "Make Fedora run well on my Powerbook" might be a specific thing to test for 12:16 <@wwoods> dmalcolm: indeed. The specific goal we had at the time was "Make PPC (both 32- and 64-bit) actually work right" 12:17 <@wwoods> we could target the powerbook and g5 desktop specifically 12:17 <@wwoods> since those are probably reprasentative of the most common ppc hardware we'll see 12:18 <@wwoods> looking at the announce messages for the release slips: 12:18 <@wwoods> https://www.redhat.com/archives/fedora-announce-list/2006-October/msg00000.html 12:18 <@wwoods> https://www.redhat.com/archives/fedora-announce-list/2006-October/msg00005.html 12:18 <@wwoods> https://www.redhat.com/archives/fedora-announce-list/2006-October/msg00006.html 12:19 -!- thl [n=thl@fedora/thl] has joined #fedora-testing 12:19 <@wwoods> The main problems we encountered were: PPC didn't get enough testing, iscsi (and other new features) don't get enough testing 12:19 <@wwoods> and a bunch of multiarch/multilib problems that were due to buildsystem issues and a really late change to yum behaviour 12:19 <@wwoods> (that's what's causing the i586-kernel-on-i686 problem) 12:19 I started writing a yum testsuite BTW 12:20 <@wwoods> dmalcolm: seriously? fantastic 12:20 only the beginnings of it 12:20 * dmalcolm looks up URL 12:21 rpmfluff: https://testing.108.redhat.com/source/browse/testing/trunk/incubator/rpmfluff/ is a tool for easily generating RPMs wi th certain properties e.g. specific Requires/Provides 12:22 and also to make yum repos of these 12:22 and then here's the test so far: https://testing.108.redhat.com/source/browse/testing/trunk/rhts/tests/sandbox/yum/smoketest/ 12:22 <@wwoods> yum is something we really need to test hard, so that's excellent 12:23 I started this but don't have time ATM to continue it, so if someone's looking for an interesting thing to hack on... 12:24 <@wwoods> Well, for testing FC7 I really want to have automated regression tests for the 'critical path' of the system 12:24 <@wwoods> i.e. everything you need to boot the system and run 'yum' 12:24 <@wwoods> pretty much anything else that's broken you can fix with an update, but if yum doesn't work out-of-the-box 12:25 <@wwoods> then we're dead in the water. 12:25 <@wwoods> but let's get back to FC6 for a minute. 12:26 <@wwoods> So, the Big Problems I've identified: not enough testing on PPC, not enough testing of hard-to-test new features (e.g. iscsi), late yum change 12:26 <@wwoods> are there any other Big Problems from the FC6 test cycle that people can think of? 12:27 <@wwoods> I'm looking for things that are likely to come up again during FC7 12:27 <@wwoods> and, hopefully, ways to keep them from happening. 12:28 <@wwoods> Let me rephrase the 'late yum change' as 'big change after freeze with lots of side effects' 12:28 <@wwoods> it's *still* causing problems for us. 12:28 wwoods: mhm. the first two issues are basically no brainers and apart from either handing out the necessary hardware, I do not really see any way to fix that easily. There are ppc emulators, but they might have quirks real hardware does not have or the other way around and iSCSI i s some protocoll where dozends of clients are available but no real free (not even as in beer) servers. 12:29 <@wwoods> ixs: really? no free servers? I thought we had an OSS initiator (or whatever the terminology is) in FC somewhere? 12:29 * dmalcolm has a spare Powerbook BTW (with a broken screen) 12:29 <@wwoods> I'll admit to being totally ignorant of iSCSI 12:29 <@wwoods> it would be nice if big new features came with a 'how to test' document of some kind 12:31 http://linux-iscsi.sourceforge.net/ 12:31 http://www.cuddletech.com/articles/iscsi/index.html 12:31 you can create an iSCSI target fairly easily 12:32 okay. Then I take that back. 12:32 THe last time I looked at iSCSI I only found clients. 12:32 thx, I'll take a look at it. 12:32 <@wwoods> poelcat: interesting. So yeah, it's really not that we lack hardware for iSCSI, but that nobody actually knew how to test it 12:33 maybe such "how to test" documents ought to come-from/contributed-to by maintainers of relavent packages (in Fedora). 12:34 <@wwoods> rdieter: yes! that would be an excellent idea. 12:34 i.e. for each feature proposed for Core, we need a "how do I get to play with it"? document? 12:34 ixs: google "iscsi linux target" 12:34 <@wwoods> dmalcolm: yes, definitely 12:34 rdieter: great idea 12:34 <@wwoods> it's not good for devel to add features but not tell anyone how to test them. 12:34 bingo, agreed. 12:35 wwoods: what about a central repository of sorts for test cases? 12:35 should we ask the FAB to include this proposal in the Fedora Summit? 12:35 <@wwoods> it's good that we're having this meeting now - with the Fedora Board having their summit soon, I can pass along some recommend... y es 12:35 <@wwoods> dmalcolm, you type faster than I do 12:35 <@wwoods> heh 12:36 <@wwoods> poelcat: yes, absolutely! we have the 'testify' tool inside Red Hat for keeping track of these things, but it wouldn't be good for Fedora as-is 12:37 <@wwoods> all we really need is a test-plan template in the wiki 12:37 we could track test cases "e.g. Try installing on a Powerbook G4" on the wiki... 12:37 I'd say documenting new features is a no-brainer. just do it. may be a bit off-topic for the smerge summit thingy anyway. 12:37 <@wwoods> and we need to get people to write test plans for their packages 12:37 ...but that doesn't automatically track the "who tested this last, and on what hardware" issue 12:38 <@wwoods> rdieter: well, they're talking about stuff for the FC7 roadmap, so I want to be sure they know they'll have to document whatever th ey plan to add 12:38 fair enough. 12:38 <@wwoods> and also they need to buy us all old powerbooks. *cough* 12:38 wwoods: good idea! :D 12:39 <@wwoods> dmalcolm: good point. the wiki is good for holding the test plans, but we need something to keep track of who executed the test pla ns 12:39 <@wwoods> and when, and on which packages and what hardware, etc. 12:39 and success/failure, with failures mapped to bugzilla 12:40 wwoods: care to define a bit what you understand as "test plans". Something like test cases to be shipped with the package as make che ck or just a checklist of things to do? 12:40 <@wwoods> jumping ahead a bit in the agenda, I want to have something like this set up, so that packages can't get out of updates-testing unt il they pass some tests 12:40 wwoods: automated, manual, or both? 12:41 <@wwoods> ixs: ah, sorry. When I say "test plan" I'm thinking about a "how-to-test" document, probably stored in the wiki 12:41 <@wwoods> dmalcolm: both, ideally 12:41 wwoods: also, would this apply to both Core and Extras? 12:41 <@wwoods> dmalcolm: definitely - especially with them merging 12:41 I think it would be very easy to increase the amount of test coverage if we have clear instructions (tests cases) on what to test 12:41 <@wwoods> I don't think we should treat them differently unless necessary 12:42 I'd suggest considerring instead of "cannot leave updates-testing if tests reported as failed" 12:42 * craigt agrees with poelcat 12:42 s/instead of/instead/ (makes more sense) 12:42 <@wwoods> rdieter: that sounds reasonable to me. 12:43 rdieter: sounds good, though I think if we have automated tests I'd want a way of waiving false positives 12:43 <@wwoods> getting back to the agenda a bit, the second thing I wanted to talk about was automated testing 12:44 <@wwoods> luckily we're already talking about it 12:44 <@wwoods> have any of your non-redhatters looked at the RHTS stuff? 12:44 I made my original case here: https://www.redhat.com/archives/fedora-test-list/2006-August/msg00517.html 12:44 * rdieter hasn't (yet) 12:44 * craigt has not 12:44 <@wwoods> https://wiki.108.redhat.com/wiki/index.php/Testing/Rhts is a good place to start, I think 12:45 <@wwoods> although dmalcolm might have a better suggestion 12:45 * dmalcolm looks for URLs 12:45 I've got a yum repo with some test-writing/execution tools here: http://people.redhat.com/dmalcolm/tablecloth/tools/ 12:45 <@wwoods> anyway, the quick summary for "What's RHTS?" is that.. well it's a bunch of things, but mostly it's a framework for automated testi ng stuff 12:46 the idea is to have a common packaging model for tests 12:46 so that for each test, you can make an RPM of it 12:46 I've used this to make yum repos of tests 12:46 for example: http://people.redhat.com/dmalcolm/tablecloth/tests/sandbox/ 12:47 and http://people.redhat.com/dmalcolm/tablecloth/tests/promoted/ 12:47 so you can install an RPM of a test and run it 12:47 or simply run the test from a checkout of the test source code 12:48 <@wwoods> right. so, ideally, we should have a set of automated tests (similar to the 'make check' stuff ixs mentioned) for each package in C ore/Extras 12:48 so it's easy to automate the tests: install FC7 test1 (or whatever), and enable the repos, and then run the tests 12:48 <@wwoods> so when there's a package update for, say, ImageMagick in updates-testing 12:48 tests identify what packages they test you can do things like "yum install tests-of(ImageMagick)-\*" 12:49 <@wwoods> whoa 12:49 (I was typing another package name, but edited it quickly) 12:49 <@wwoods> (heh, well done) 12:49 <@wwoods> so yes, you could run that yum command, and run all the tests that get downloaded 12:50 <@wwoods> we want to have that be automatic but.. it's not there yet. also we don't have any ImageMagick tests yet 12:50 note that tests can apply to more than one package 12:50 so, 2 challenges: write (more) tests, actually *do/use* the tests. (: 12:51 who writes the tests? 12:51 <@wwoods> rdieter: exactly. writing tests is actually pretty easy, as it turns out 12:51 craigt: pretty much anyone willing to contribute, I'd imagine. (: 12:51 <@wwoods> and Red Hat has bought hardware to run the tests on 12:52 <@wwoods> https://wiki.108.redhat.com/wiki/index.php/Testing/Rhts/Docs/TestWriting is a fine guide 12:53 mhm. but most tests should be simple shellscripts. install fedora-logos, do a bit of convert and compare with the reference image or s o. That should be doable for imagemagick e.g. 12:53 <@wwoods> dmalcolm and poelcat have worked out a whole process for writing tests, reviewing them, getting them into the test system, etc. 12:53 nifty. 12:53 <@wwoods> ixs: right, exactly. most tests are pretty dang simple 12:54 I even wrote a tool "rhts-create-new-test" which generates all the boilerplate for you 12:54 <@wwoods> so for most cases, you just need to add some extra metadata to turn a simple script into a test 12:54 https://testing.108.redhat.com/index.html 12:54 if you want to wrap up an existing shell script 12:54 <@wwoods> and, as always, dmalcolm is one step ahead of me! heh 12:54 (it's is the rhts-devel package IIRC) 12:55 s/is/in/ 12:55 so, start pinging package maintainers to start writing tests? 12:55 for Core, many within Red Hat already are. I'd love to see some tests for Extras 12:55 (and of course, hopefully that distinction can go away) 12:56 also: if you run into a bug, write a test for it 12:56 you don't have to be the package maintainer 12:56 <@wwoods> rdieter: yes, absolutely 12:56 test writers (if not the dev) would need the "how to test" doc from the devs to add the meta data 12:56 I did this for the rpmbuild/tar bug 12:56 <@wwoods> this isn't something that's required but it will really, really help a lot 12:57 * rdieter will have to go FESCo in a few... (will try to keep an eye here too) 12:57 <@wwoods> craigt: right, especially for new stuff, test-writers will need help from the package dev 12:57 https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=206841#c5 12:58 I think an important step will be to try to rig things up so that tests get run on the nightly rawhide 12:59 <@wwoods> dmalcolm: definitely. I also need to talk to jkeating et. al. about hooking the tests into the updates-testing process. 12:59 so, in the same way fedora-devel-list gets emailed about changes to packages, we should run tests on the "before" and "after" tre es, and report the results 12:59 <@wwoods> While I'm at it: rdieter, do you think extras could have an extras-testing repo 12:59 <@wwoods> rdieter: so we can apply these same ideas for extras (at least until The Great Merge is finished) 12:59 wwoods: yeah, I would advocate that. 13:00 (and have been) 13:00 <@wwoods> rdieter: fantastic, thank you 13:00 all hail the "Great Merge" 13:00 <@wwoods> The People's Glorious Package Merging 13:00 * dmalcolm blames Zod 13:01 <@wwoods> do you think we have enough testers to *require* that someone tests each package before it can go from -testing to updates? 13:02 <@wwoods> and, as a side note - do we need to have Official Testers? 13:02 <@wwoods> I'd be much happier with updates-testing and updates if someone tested every package that went into updates 13:02 fedora-test-list often has flurries of updates e.g. on the desktop 13:02 <@wwoods> even if it's just a "yep, these packages all install and nothing blew up" 13:03 if we require testing, do we require testing each package individually, or just that each package was used as part of a test run? 13:03 the latter is far easier 13:03 <@wwoods> dmalcolm: well, it's hard with big GNOME updates to test each package individually 13:04 <@wwoods> I don't want to hamper updates with meticulous testing 13:04 <@wwoods> carried to its extreme you end up with something like the RHEL errata process 13:04 could the update process kick off test runs, and have the Test Update emails contain results from the run? 13:04 <@wwoods> which has driven lesser men insane 13:04 (gibber) 13:05 <@wwoods> dmalcolm: in theory, Jesse tells me that's possible, but I'm completely ignorant of the mechanics of it 13:05 <@wwoods> I will talk to him about how that would work 13:06 great 13:06 <@wwoods> since the updates process is supposed to be moved outside the company, we should end up with an opportunity to write these parts in 13:07 <@wwoods> but anyway - if we require signoff on all updates before they can leave -testing, that means we'll need how-to-test documents. 13:07 perhaps the Fedora Updates Orbital Laser could make XML-RPC calls to our lab, to say stuff like "please test FC6 with all updates as of 20061109, plus this candidate package" 13:07 <@wwoods> but we don't have those yet. 13:08 <@wwoods> dmalcolm: well, I'm not sure our lab is going to do a fresh install for all testing 13:09 <@wwoods> dmalcolm: I think most Fedora users keep their systems pretty well up-to-date, so probably the default state of the machine would b e "core + all updates" 13:10 that's a good test case then - though I'd like it if we also tested a fresh "install Core then apply all non-test Updates" 13:10 on an ongoing basis 13:10 <@wwoods> err, sorry, got sidetracked. yes, the Fedora Update-o-ma-tron should totally be able to send a message to the test lab to say "test package X" 13:10 <@wwoods> dmalcolm: oh definitely 13:10 <@wwoods> In theory you might be able to add 'updates' at installtime now 13:11 stand well back 13:11 <@wwoods> so you don't have to do a big 'yum update' at first boot 13:11 we definitely need to autotest that, then 13:11 <@wwoods> it just picks up the most recent version of all the packages that have been updated, and installs the rest from DVD (or whatever) 13:11 I can see that being fragile 13:11 <@wwoods> dmalcolm: yeah, that's part of the installer testing stuff I'm working on (snake) 13:11 <@wwoods> installing from extra repos is always a bit fragile 13:12 is "snake" downloadable somewhere? 13:12 <@wwoods> dmalcolm: not public yet, but I'll give you a pointer to the internal CVS later 13:13 not much use to everyone else here :-( 13:13 <@wwoods> well, I'm still working on the uninteresting parts right now - the parts that would be interesting here are future work 13:14 ok 13:14 <@wwoods> but yeah, I need to get someone to assure me that I can release the interesting bits 13:14 how's the agenda looking? 13:14 <@wwoods> stay tuned on that stuff. 13:14 <@wwoods> well, I'm still thinking about 3b - policy for moving packages from updates-testing to updates 13:15 <@wwoods> do we think we can *require* signoff on updates? or is that something we need to wait on 13:15 (I need to head off for 10 mins shortly) 13:15 <@wwoods> e.g. wait until we have more automated tests and how-to-test docs 13:16 there's also the issue of how invasive an update should be... 13:16 ...I'm thinking of that X update controversy 13:16 <@wwoods> which X update? 13:17 * dmalcolm is frantically looking 13:18 as to rquireing testing singoff, that just doesn't seem possible to me 13:18 s/rquireing /requireing 13:18 <@wwoods> craigt: not enough testers? or too hard to get stuff tested quickly? 13:19 not enough testers (which seems the == too hard to get stuff tested quickly) 13:19 <@wwoods> dmalcolm: sorry, I'm not sure what you mean by 'invasive'.. was it just a package that radically changed system behavior? 13:20 I'm thinking of the change that broke proprietary drivers 13:20 <@wwoods> craigt: gotcha. do you think it would be possible if we recruited a lot of volunteers to be Official Fedora Testers? 13:20 <@wwoods> craigt: or is it just not possible because of the sheer number of updates? 13:21 yeah, but still....exactly 13:21 seems to me that around FCx-testx releases we have more, but during 13:22 wwoods: http://lwn.net/Articles/195351/ FWIW 13:22 but prior to those releases and after, the number of 'testers' shrinks 13:22 (at least based on the test-list traffic) 13:22 <@wwoods> craigt: yeah, it definitely seems that way 13:23 (how many testers are there anyway? ) 13:23 based on the .5 dozen here, doesn't seem that many are interested 13:23 until their systems break :( 13:23 <@wwoods> heh, yeah. 13:23 <@wwoods> I wonder how many people actually use updates-testing 13:24 <@wwoods> they're 'testers', in a way 13:24 <@wwoods> maybe we ought to have an official Fedora Testers group though 13:24 <@wwoods> it'd help determine what's possible if we knew how many people we had available 13:25 <@wwoods> also it would help motivate people to actually do testing if there was some recognition (and perhaps reward) in it 13:26 BugZappers == Fedora Testers group ? 13:26 <@wwoods> not officially, but yeah 13:26 <@wwoods> we could make that official 13:26 and some docs / help on what needed testing 13:27 <@wwoods> 1) test things in updates-testing 13:27 <@wwoods> 2) bug triage 13:27 <@wwoods> 3) write test plans and automated tests 13:29 <@wwoods> this is a list of What Testers Do. Anything else that should go on there? 13:30 <@wwoods> I guess I should have mentioned it before, but I'm thinking 90 minutes is about the max for a meeting like this 13:30 <@wwoods> so before I declare this meeting over - is there anything else we should discuss? 13:30 bzbot? 13:30 <@wwoods> oh, bzbot, right. Would it be useful to have a bugzilla bot on freenode? 13:31 <@wwoods> I'm not sure if we want it sending messages for all new bugs 13:31 <@wwoods> or just have it give you summary info etc. for bug numbers 13:31 youch 13:32 what's the rate at which Fedora bugs get filed? 13:32 BPM? 13:32 <@wwoods> also things like 'whoowns' for product/package names would be cool, I guess 13:32 <@wwoods> dmalcolm: I can give you BPW 13:32 <@wwoods> 374 FC6 bugs this week 13:32 is there an RSS feed or the like for this now ? 13:33 <@wwoods> craigt: hrm. I think I have a link, but it's very long 13:33 tinyurl ? 13:33 <@wwoods> http://rdr.to/RI 13:33 <@wwoods> that should be an RSS feed for 'FC6 bugs in the past 7 days' 13:34 <@wwoods> you can get an RSS feed for any given bugzilla query 13:34 <@wwoods> maybe RSS is sufficient for people interested in that sort of thing 13:35 <@wwoods> but I should see who's still in the bugzapper group 13:35 <@wwoods> and talk to them about bzbot 13:35 <@wwoods> and see if there are others clamoring for it 13:37 <@wwoods> unless someone wants to demand it now? or take responsibility for setting it up? 13:38 * wwoods takes that as a no 13:38 <@wwoods> okay, bzbot is back-burner'd for now 13:39 <@wwoods> And that's all I got. Anyone else? 13:41 <@wwoods> Okay then. Meeting over! Notes will be posted to the wiki and fedora-test-list after I get them together 13:41 <@wwoods> I'll put the IRC log up somewhere too. 13:41 <@wwoods> Thanks, everyone! --- Log closed Thu Nov 09 13:41:41 2006