Testing/Logs/20061109

Note: this log may go away if I find a good place to host the (much more readable) html-ized log.

--- Log opened Thu Nov 09 12:06:13 2006
12:06 < dmalcolm> plus some convert to HTML script?
12:06 < craigt> can someone point me to the agenda on the wiki ?
12:06 < poelcat> dmalcolm: http://mg.pov.lt/irclog2html
12:06 <@wwoods> craigt: it's in the topic
12:07 <@wwoods> http://fedoraproject.org/wiki/Testing/Meetings/20061109
12:07 < craigt> doh! thanks
12:07 < poelcat> dmalcolm: i need your python skills to make it read xchat better... right now it does irssi
12:07 [Users #fedora-testing] 
12:07 [@ChanServ]  [ craigt  ]  [ ixs    ]  [ rdieter] 
12:07 [@wwoods  ]  [ dmalcolm]  [ poelcat] 
12:07 -!- Irssi: #fedora-testing: Total of 7 nicks [2 ops, 0 halfops, 0 voices, 5 normal] 
12:07  * dmalcolm hides
12:08 <@wwoods> okay, I'm going to get started.
12:08 <@wwoods> So, hi and welcome to the first-ever Fedora Testing meeting, yay!
12:08 <@wwoods> The agenda, as mentioned, is at: http://fedoraproject.org/wiki/Testing/Meetings/20061109
12:09 <@wwoods> First thing: a quick FC6 review
12:09 <@wwoods> We delayed.. 3 times, if memory serves
12:10 <@wwoods> Since there's just the half-dozen of us I'll ask: who did FC6 testing prior to release?
12:11  * dmalcolm was running rawhide up to the release
12:11 <@wwoods> dmalcolm: cool, that definitely counts.
12:11 < dmalcolm> but that covers daily changes, rather than fresh installation/upgrade from FC5
12:12 < dmalcolm> BTW do you regard FE6 as on-topic for the discussion?
12:12 <@wwoods> right, I suspect that most testers are doing that
12:12 <@wwoods> dmalcolm: sure
12:12 < rdieter> I only did minimal (< 2 hrs) playing with fc6t3 clean install one afternoon.
12:12 < dmalcolm> I hope that we're going to converge Core and Extras, so anything we say will hopefully apply to the Brave New Amalgam thing
12:13  * craigt ran rawhide aslo
12:13 <@wwoods> dmalcolm: right. was there something in particular about FE6 before release?
12:14 < dmalcolm> IIRC there were plenty of autogenerated emails going to fedora-extras-list during the FE6 timeframe from Fedora Extras repoclosur
e <buildsys@fedoraproject.org>
12:14 <@wwoods> Quite a few of the bugs that we held up FC6 for were PPC-specific
12:15 <@wwoods> and that's just because.. we have very, very few ppc users
12:15 <@wwoods> and even fewer testers
12:16 <@wwoods> is it worth the effort to track down some ppc users and try to make them Official Testers in some capacity?
12:16 < dmalcolm> re ppc: of those users, is there a specific goal?  I'm thinking "Make Fedora run well on my Powerbook" might be a specific thing
to test for
12:16 <@wwoods> dmalcolm: indeed. The specific goal we had at the time was "Make PPC (both 32- and 64-bit) actually work right"
12:17 <@wwoods> we could target the powerbook and g5 desktop specifically
12:17 <@wwoods> since those are probably reprasentative of the most common ppc hardware we'll see
12:18 <@wwoods> looking at the announce messages for the release slips:
12:18 <@wwoods> https://www.redhat.com/archives/fedora-announce-list/2006-October/msg00000.html
12:18 <@wwoods> https://www.redhat.com/archives/fedora-announce-list/2006-October/msg00005.html
12:18 <@wwoods> https://www.redhat.com/archives/fedora-announce-list/2006-October/msg00006.html
12:19 -!- thl [n=thl@fedora/thl]  has joined #fedora-testing
12:19 <@wwoods> The main problems we encountered were: PPC didn't get enough testing, iscsi (and other new features) don't get enough testing
12:19 <@wwoods> and a bunch of multiarch/multilib problems that were due to buildsystem issues and a really late change to yum behaviour
12:19 <@wwoods> (that's what's causing the i586-kernel-on-i686 problem)
12:19 < dmalcolm> I started writing a yum testsuite BTW
12:20 <@wwoods> dmalcolm: seriously? fantastic
12:20 < dmalcolm> only the beginnings of it
12:20  * dmalcolm looks up URL
12:21 < dmalcolm> rpmfluff: https://testing.108.redhat.com/source/browse/testing/trunk/incubator/rpmfluff/  is a tool for easily generating RPMs wi
th certain properties e.g. specific Requires/Provides
12:22 < dmalcolm> and also to make yum repos of these
12:22 < dmalcolm> and then here's the test so far: https://testing.108.redhat.com/source/browse/testing/trunk/rhts/tests/sandbox/yum/smoketest/
12:22 <@wwoods> yum is something we really need to test hard, so that's excellent
12:23 < dmalcolm> I started this but don't have time ATM to continue it, so if someone's looking for an interesting thing to hack on...
12:24 <@wwoods> Well, for testing FC7 I really want to have automated regression tests for the 'critical path' of the system
12:24 <@wwoods> i.e. everything you need to boot the system and run 'yum'
12:24 <@wwoods> pretty much anything else that's broken you can fix with an update, but if yum doesn't work out-of-the-box
12:25 <@wwoods> then we're dead in the water.
12:25 <@wwoods> but let's get back to FC6 for a minute.
12:26 <@wwoods> So, the Big Problems I've identified: not enough testing on PPC, not enough testing of hard-to-test new features (e.g. iscsi), late
yum change
12:26 <@wwoods> are there any other Big Problems from the FC6 test cycle that people can think of?
12:27 <@wwoods> I'm looking for things that are likely to come up again during FC7
12:27 <@wwoods> and, hopefully, ways to keep them from happening.
12:28 <@wwoods> Let me rephrase the 'late yum change' as 'big change after freeze with lots of side effects'
12:28 <@wwoods> it's *still* causing problems for us.
12:28 < ixs> wwoods: mhm. the first two issues are basically no brainers and apart from either handing out the necessary hardware, I do not really
see any way to fix that easily. There are ppc emulators, but they might have quirks real hardware does not have or the other way around and iSCSI i
s some protocoll where dozends of clients are available but no real free (not even as in beer) servers.
12:29 <@wwoods> ixs: really? no free servers? I thought we had an OSS initiator (or whatever the terminology is) in FC somewhere?
12:29  * dmalcolm has a spare Powerbook BTW (with a broken screen)
12:29 <@wwoods> I'll admit to being totally ignorant of iSCSI
12:29 <@wwoods> it would be nice if big new features came with a 'how to test' document of some kind
12:31 < poelcat> http://linux-iscsi.sourceforge.net/
12:31 < poelcat> http://www.cuddletech.com/articles/iscsi/index.html
12:31 < poelcat> you can create an iSCSI target fairly easily
12:32 < ixs> okay. Then  I take that back.
12:32 < ixs> THe last time I looked at iSCSI I only found clients.
12:32 < ixs> thx, I'll take a look at it.
12:32 <@wwoods> poelcat: interesting. So yeah, it's really not that we lack hardware for iSCSI, but that nobody actually knew how to test it
12:33 < rdieter> maybe such "how to test" documents ought to come-from/contributed-to by maintainers of relavent packages (in Fedora).
12:34 <@wwoods> rdieter: yes! that would be an excellent idea.
12:34 < dmalcolm> i.e. for each feature proposed for Core, we need a "how do I get to play with it"? document?
12:34 < poelcat> ixs: google "iscsi linux target"
12:34 <@wwoods> dmalcolm: yes, definitely
12:34 < dmalcolm> rdieter: great idea
12:34 <@wwoods> it's not good for devel to add features but not tell anyone how to test them.
12:34 < rdieter> bingo, agreed.
12:35 < poelcat> wwoods: what about a central repository of sorts for test cases?
12:35 < dmalcolm> should we ask the FAB to include this proposal in the Fedora Summit?
12:35 <@wwoods> it's good that we're having this meeting now - with the Fedora Board having their summit soon, I can pass along some recommend... y
es
12:35 <@wwoods> dmalcolm, you type faster than I do
12:35 <@wwoods> heh
12:36 <@wwoods> poelcat: yes, absolutely! we have the 'testify' tool inside Red Hat for keeping track of these things, but it wouldn't be good for
Fedora as-is
12:37 <@wwoods> all we really need is a test-plan template in the wiki
12:37 < dmalcolm> we could track test cases "e.g. Try installing on a Powerbook G4" on the wiki...
12:37 < rdieter> I'd say documenting new features is a no-brainer.  just do it.  may be a bit off-topic for the smerge summit thingy anyway.
12:37 <@wwoods> and we need to get people to write test plans for their packages
12:37 < dmalcolm> ...but that doesn't automatically track the "who tested this last, and on what hardware" issue
12:38 <@wwoods> rdieter: well, they're talking about stuff for the FC7 roadmap, so I want to be sure they know they'll have to document whatever th
ey plan to add
12:38 < rdieter> fair enough.
12:38 <@wwoods> and also they need to buy us all old powerbooks. *cough*
12:38 < ixs> wwoods: good idea! :D
12:39 <@wwoods> dmalcolm: good point. the wiki is good for holding the test plans, but we need something to keep track of who executed the test pla
ns
12:39 <@wwoods> and when, and on which packages and what hardware, etc.
12:39 < dmalcolm> and success/failure, with failures mapped to bugzilla
12:40 < ixs> wwoods: care to define a bit what you understand as "test plans". Something like test cases to be shipped with the package as make che
ck or just a checklist of things to do?
12:40 <@wwoods> jumping ahead a bit in the agenda, I want to have something like this set up, so that packages can't get out of updates-testing unt
il they pass some tests
12:40 < dmalcolm> wwoods: automated, manual, or both?
12:41 <@wwoods> ixs: ah, sorry. When I say "test plan" I'm thinking about a "how-to-test" document, probably stored in the wiki
12:41 <@wwoods> dmalcolm: both, ideally
12:41 < dmalcolm> wwoods: also, would this apply to both Core and Extras?
12:41 <@wwoods> dmalcolm: definitely - especially with them merging
12:41 < poelcat> I think it would be very easy to increase the amount of test coverage if we have clear instructions (tests cases) on what to test
12:41 <@wwoods> I don't think we should treat them differently unless necessary
12:42 < rdieter> I'd suggest considerring instead of "cannot leave updates-testing if tests reported as failed"
12:42  * craigt agrees with poelcat
12:42 < rdieter> s/instead of/instead/  (makes more sense)
12:42 <@wwoods> rdieter: that sounds reasonable to me.
12:43 < dmalcolm> rdieter: sounds good, though I think if we have automated tests I'd want a way of waiving false positives
12:43 <@wwoods> getting back to the agenda a bit, the second thing I wanted to talk about was automated testing
12:44 <@wwoods> luckily we're already talking about it
12:44 <@wwoods> have any of your non-redhatters looked at the RHTS stuff?
12:44 < poelcat> I made my original case here: https://www.redhat.com/archives/fedora-test-list/2006-August/msg00517.html
12:44  * rdieter hasn't (yet)
12:44  * craigt has not
12:44 <@wwoods> https://wiki.108.redhat.com/wiki/index.php/Testing/Rhts is a good place to start, I think
12:45 <@wwoods> although dmalcolm might have a better suggestion
12:45  * dmalcolm looks for URLs
12:45 < dmalcolm> I've got a yum repo with some test-writing/execution tools here: http://people.redhat.com/dmalcolm/tablecloth/tools/
12:45 <@wwoods> anyway, the quick summary for "What's RHTS?" is that.. well it's a bunch of things, but mostly it's a framework for automated testi
ng stuff
12:46 < dmalcolm> the idea is to have a common packaging model for tests
12:46 < dmalcolm> so that for each test, you can make an RPM of it
12:46 < dmalcolm> I've used this to make yum repos of tests
12:46 < dmalcolm> for example: http://people.redhat.com/dmalcolm/tablecloth/tests/sandbox/
12:47 < dmalcolm> and http://people.redhat.com/dmalcolm/tablecloth/tests/promoted/
12:47 < dmalcolm> so you can install an RPM of a test and run it
12:47 < dmalcolm> or simply run the test from a checkout of the test source code
12:48 <@wwoods> right. so, ideally, we should have a set of automated tests (similar to the 'make check' stuff ixs mentioned) for each package in C
ore/Extras
12:48 < dmalcolm> so it's easy to automate the tests: install FC7 test1 (or whatever), and enable the repos, and then run the tests
12:48 <@wwoods> so when there's a package update for, say, ImageMagick in updates-testing
12:48 < dmalcolm> tests identify what packages they test you can do things like "yum install tests-of(ImageMagick)-\*"
12:49 <@wwoods> whoa
12:49 < dmalcolm> (I was typing another package name, but edited it quickly)
12:49 <@wwoods> (heh, well done)
12:49 <@wwoods> so yes, you could run that yum command, and run all the tests that get downloaded
12:50 <@wwoods> we want to have that be automatic but.. it's not there yet. also we don't have any ImageMagick tests yet
12:50 < dmalcolm> note that tests can apply to more than one package
12:50 < rdieter> so, 2 challenges: write (more) tests, actually *do/use* the tests. (:
12:51 < craigt> who writes the tests?
12:51 <@wwoods> rdieter: exactly. writing tests is actually pretty easy, as it turns out
12:51 < rdieter> craigt: pretty much anyone willing to contribute, I'd imagine. (:
12:51 <@wwoods> and Red Hat has bought hardware to run the tests on
12:52 <@wwoods> https://wiki.108.redhat.com/wiki/index.php/Testing/Rhts/Docs/TestWriting is a fine guide
12:53 < ixs> mhm. but most tests should be simple shellscripts. install fedora-logos, do a bit of convert and compare with the reference image or s
o. That should be doable for imagemagick e.g.
12:53 <@wwoods> dmalcolm and poelcat have worked out a whole process for writing tests, reviewing them, getting them into the test system, etc.
12:53 < rdieter> nifty.
12:53 <@wwoods> ixs: right, exactly. most tests are pretty dang simple
12:54 < dmalcolm> I even wrote a tool "rhts-create-new-test" which generates all the boilerplate for you
12:54 <@wwoods> so for most cases, you just need to add some extra metadata to turn a simple script into a test
12:54 < poelcat> https://testing.108.redhat.com/index.html
12:54 < dmalcolm> if you want to wrap up an existing shell script
12:54 <@wwoods> and, as always, dmalcolm is one step ahead of me! heh
12:54 < dmalcolm> (it's is the rhts-devel package IIRC)
12:55 < dmalcolm> s/is/in/
12:55 < rdieter> so, start pinging package maintainers to start writing tests?
12:55 < dmalcolm> for Core, many within Red Hat already are.  I'd love to see some tests for Extras
12:55 < dmalcolm> (and of course, hopefully that distinction can go away)
12:56 < dmalcolm> also: if you run into a bug, write a test for it
12:56 < dmalcolm> you don't have to be the package maintainer
12:56 <@wwoods> rdieter: yes, absolutely
12:56 < craigt> test writers (if not the dev) would need the "how to test" doc from the devs to add the meta data
12:56 < dmalcolm> I did this for the rpmbuild/tar bug
12:56 <@wwoods> this isn't something that's required but it will really, really help a lot
12:57  * rdieter will have to go FESCo in a few... (will try to keep an eye here too)
12:57 <@wwoods> craigt: right, especially for new stuff, test-writers will need help from the package dev
12:57 < dmalcolm> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=206841#c5
12:58 < dmalcolm> I think an important step will be to try to rig things up so that tests get run on the nightly rawhide
12:59 <@wwoods> dmalcolm: definitely. I also need to talk to jkeating et. al. about hooking the tests into the updates-testing process.
12:59 < dmalcolm> so, in the same way fedora-devel-list gets emailed about changes to packages, we should run tests on the "before" and "after" tre
es, and report the results
12:59 <@wwoods> While I'm at it: rdieter, do you think extras could have an extras-testing repo
12:59 <@wwoods> rdieter: so we can apply these same ideas for extras (at least until The Great Merge is finished)
12:59 < rdieter> wwoods: yeah, I would advocate that.
13:00 < rdieter> (and have been)
13:00 <@wwoods> rdieter: fantastic, thank you
13:00 < rdieter> all hail the "Great Merge"
13:00 <@wwoods> The People's Glorious Package Merging
13:00  * dmalcolm blames Zod
13:01 <@wwoods> do you think we have enough testers to *require* that someone tests each package before it can go from -testing to updates?
13:02 <@wwoods> and, as a side note - do we need to have Official Testers?
13:02 <@wwoods> I'd be much happier with updates-testing and updates if someone tested every package that went into updates
13:02 < dmalcolm> fedora-test-list often has flurries of updates e.g. on the desktop
13:02 <@wwoods> even if it's just a "yep, these packages all install and nothing blew up"
13:03 < dmalcolm> if we require testing, do we require testing each package individually, or just that each package was used as part of a test run?
13:03 < dmalcolm> the latter is far easier
13:03 <@wwoods> dmalcolm: well, it's hard with big GNOME updates to test each package individually
13:04 <@wwoods> I don't want to hamper updates with meticulous testing
13:04 <@wwoods> carried to its extreme you end up with something like the RHEL errata process
13:04 < dmalcolm> could the update process kick off test runs, and have the Test Update emails contain results from the run?
13:04 <@wwoods> which has driven lesser men insane
13:04 < dmalcolm> (gibber)
13:05 <@wwoods> dmalcolm: in theory, Jesse tells me that's possible, but I'm completely ignorant of the mechanics of it
13:05 <@wwoods> I will talk to him about how that would work
13:06 < dmalcolm> great
13:06 <@wwoods> since the updates process is supposed to be moved outside the company, we should end up with an opportunity to write these parts in
13:07 <@wwoods> but anyway - if we require signoff on all updates before they can leave -testing, that means we'll need how-to-test documents.
13:07 < dmalcolm> perhaps the Fedora Updates Orbital Laser could make XML-RPC calls to our lab, to say stuff like "please test FC6 with all updates
as of 20061109, plus this candidate package"
13:07 <@wwoods> but we don't have those yet.
13:08 <@wwoods> dmalcolm: well, I'm not sure our lab is going to do a fresh install for all testing
13:09 <@wwoods> dmalcolm: I think most Fedora users keep their systems pretty well up-to-date, so probably the default state of the machine would b
e "core + all updates"
13:10 < dmalcolm> that's a good test case then - though I'd like it if we also tested  a fresh "install Core then apply all non-test Updates"
13:10 < dmalcolm> on an ongoing basis
13:10 <@wwoods> err, sorry, got sidetracked. yes, the Fedora Update-o-ma-tron should totally be able to send a message to the test lab to say "test
package X"
13:10 <@wwoods> dmalcolm: oh definitely
13:10 <@wwoods> In theory you might be able to add 'updates' at installtime now
13:11 < dmalcolm> stand well back
13:11 <@wwoods> so you don't have to do a big 'yum update' at first boot
13:11 < dmalcolm> we definitely need to autotest that, then
13:11 <@wwoods> it just picks up the most recent version of all the packages that have been updated, and installs the rest from DVD (or whatever)
13:11 < dmalcolm> I can see that being fragile
13:11 <@wwoods> dmalcolm: yeah, that's part of the installer testing stuff I'm working on (snake)
13:11 <@wwoods> installing from extra repos is always a bit fragile
13:12 < dmalcolm> is "snake" downloadable somewhere?
13:12 <@wwoods> dmalcolm: not public yet, but I'll give you a pointer to the internal CVS later
13:13 < dmalcolm> not much use to everyone else here :-(
13:13 <@wwoods> well, I'm still working on the uninteresting parts right now - the parts that would be interesting here are future work
13:14 < dmalcolm> ok
13:14 <@wwoods> but yeah, I need to get someone to assure me that I can release the interesting bits
13:14 < dmalcolm> how's the agenda looking?
13:14 <@wwoods> stay tuned on that stuff.
13:14 <@wwoods> well, I'm still thinking about 3b - policy for moving packages from updates-testing to updates
13:15 <@wwoods> do we think we can *require* signoff on updates? or is that something we need to wait on
13:15 < dmalcolm> (I need to head off for 10 mins shortly)
13:15 <@wwoods> e.g. wait until we have more automated tests and how-to-test docs
13:16 < dmalcolm> there's also the issue of how invasive an update should be...
13:16 < dmalcolm> ...I'm thinking of that X update controversy
13:16 <@wwoods> which X update?
13:17  * dmalcolm is frantically looking
13:18 < craigt> as to rquireing testing singoff, that just doesn't seem possible to me
13:18 < craigt> s/rquireing /requireing
13:18 <@wwoods> craigt: not enough testers? or too hard to get stuff tested quickly?
13:19 < craigt> not enough testers (which seems the == too hard to get stuff tested quickly)
13:19 <@wwoods> dmalcolm: sorry, I'm not sure what you mean by 'invasive'.. was it just a package that radically changed system behavior?
13:20 < dmalcolm> I'm thinking of the change that broke proprietary drivers
13:20 <@wwoods> craigt: gotcha. do you think it would be possible if we recruited a lot of volunteers to be Official Fedora Testers?
13:20 <@wwoods> craigt: or is it just not possible because of the sheer number of updates?
13:21 < craigt> yeah, but still....exactly
13:21 < craigt> seems to me that around FCx-testx releases we have more, but during
13:22 < dmalcolm> wwoods: http://lwn.net/Articles/195351/  FWIW
13:22 < craigt> but prior to those releases and after, the number of 'testers' shrinks
13:22 < craigt> (at least based on the test-list traffic)
13:22 <@wwoods> craigt: yeah, it definitely seems that way
13:23 < craigt> (how many testers are there anyway?  )
13:23 < craigt> based on the .5 dozen here, doesn't seem that many are interested
13:23 < craigt> until their systems break :(
13:23 <@wwoods> heh, yeah.
13:23 <@wwoods> I wonder how many people actually use updates-testing
13:24 <@wwoods> they're 'testers', in a way
13:24 <@wwoods> maybe we ought to have an official Fedora Testers group though
13:24 <@wwoods> it'd help determine what's possible if we knew how many people we had available
13:25 <@wwoods> also it would help motivate people to actually do testing if there was some recognition (and perhaps reward) in it
13:26 < craigt>  BugZappers == Fedora Testers group ?
13:26 <@wwoods> not officially, but yeah
13:26 <@wwoods> we could make that official
13:26 < craigt> and some docs / help on what needed testing
13:27 <@wwoods> 1) test things in updates-testing
13:27 <@wwoods> 2) bug triage
13:27 <@wwoods> 3) write test plans and automated tests
13:29 <@wwoods> this is a list of What Testers Do. Anything else that should go on there?
13:30 <@wwoods> I guess I should have mentioned it before, but I'm thinking 90 minutes is about the max for a meeting like this
13:30 <@wwoods> so before I declare this meeting over - is there anything else we should discuss?
13:30 < dmalcolm> bzbot?
13:30 <@wwoods> oh, bzbot, right. Would it be useful to have a bugzilla bot on freenode?
13:31 <@wwoods> I'm not sure if we want it sending messages for all new bugs
13:31 <@wwoods> or just have it give you summary info etc. for bug numbers
13:31 < dmalcolm> youch
13:32 < dmalcolm> what's the rate at which Fedora bugs get filed?
13:32 < dmalcolm> BPM?
13:32 <@wwoods> also things like 'whoowns' for product/package names would be cool, I guess
13:32 <@wwoods> dmalcolm: I can give you BPW
13:32 <@wwoods> 374 FC6 bugs this week
13:32 < craigt> is there an RSS feed or the like for this now ?
13:33 <@wwoods> craigt: hrm. I think I have a link, but it's very long
13:33 < dmalcolm> tinyurl ?
13:33 <@wwoods> http://rdr.to/RI
13:33 <@wwoods> that should be an RSS feed for 'FC6 bugs in the past 7 days'
13:34 <@wwoods> you can get an RSS feed for any given bugzilla query
13:34 <@wwoods> maybe RSS is sufficient for people interested in that sort of thing
13:35 <@wwoods> but I should see who's still in the bugzapper group
13:35 <@wwoods> and talk to them about bzbot
13:35 <@wwoods> and see if there are others clamoring for it
13:37 <@wwoods> unless someone wants to demand it now? or take responsibility for setting it up?
13:38  * wwoods takes that as a no
13:38 <@wwoods> okay, bzbot is back-burner'd for now
13:39 <@wwoods> And that's all I got. Anyone else?
13:41 <@wwoods> Okay then. Meeting over! Notes will be posted to the wiki and fedora-test-list after I get them together
13:41 <@wwoods> I'll put the IRC log up somewhere too.
13:41 <@wwoods> Thanks, everyone!
--- Log closed Thu Nov 09 13:41:41 2006