ReleaseEngineering/Meetings/2008-nov-24

Fedora Release Engineering Meeting :: Monday 2008-11-24

Fedora 10

• In good shape for release of Fedora 10 tomorrow
• Fedora 10 zero day updates are ready to go
• f13: one thing I was toying with is when we branch and freeze, making the development/ path continue on with say F12 content, and start publishing the F11 content to releases/11/Everything/

Fedora 11 Schedule

• need to make sure F12 branching is visible
• finding ways to reduce package churn
• QA team monitoring status of features during release
• proposal at http://poelstra.fedorapeople.org/schedules/f-11/f-11-standard.html
• poelcat to make adjustments to proposed schedule and republish
  1. move alpha freeze forward one week
  2. move alpha release forward one week
  3. remove critical package freeze from schedule
  4. offset feature freeze and beta freeze by one week
• vote on schedule to present to FESCo next week

IRC Transcript

f13	ping: notting jeremy rdieter wwoods lmacken warren spot	10:00
jeremy	hi	10:00
f13	poelcat: ping	10:00
* lmacken		10:00
* poelcat here		10:01
notting	sorry, here now	10:01
* spot is here		10:01
wwoods	zoom! ding ding.	10:02
f13	whee.	10:04
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - Fedora 10		10:04
f13	so there is this thing we gotta do tomorrow.	10:04
f13	no big thing, just take few moments of our time	10:05
f13	releasing Fedora 10	10:05
wwoods	drive to alabama for thanksgiving with our wife's family?	10:05
wwoods	oh, maybe that's just me	10:05
f13	wwoods: yes, that's it	10:05
f13	We're in really good shape	10:05
wwoods	awesome. hope you guys like deep-fried cajun turkey	10:05
f13	all our bits are very well synced to a number of mirrors, we've got 0-day F10 updates in place and about to be public via mirrormanager, skvidal got the bits synced over to the torrent box and torrents made up, and we've got some pre-seeding of torrents setup too	10:06
f13	the key peices left are a tested release webpage set, a release announcement, and me pulling the trigger early tomorrow	10:06
f13	and then it's on to turkey and drinking	10:07
f13	oh and F11	10:08
notting	who is doing the release announcement?	10:08
f13	notting: the docs team usually has a release announcement made, and I send it out	10:08
wwoods	f13: don't forget flipping the bits in releases.txt for preupgraders	10:08
f13	the web team preps the pages for release	10:08
f13	wwoods: yep, that's a current filed ticket in the Fedora 10 Final tracker in trac	10:08
wwoods	(I know, there's already a ticket for that, just mentioning it with the other bits)	10:08
f13	er milestone	10:08
f13	I was going to flip the rawhide bit yesterday but wound up taking a much much needed night away from the kid on a date with my wife	10:09
f13	and spending part of that on the computer would have been not good for my health	10:09
wwoods	yeah, we all run pretty close to computer-burnout-redline around release time	10:09
f13	and since we don't want rawhide sync getting in the way of our permission bit flip, I'm not going to reset rawhide today either, I'll do it tomorrow, for Wed's rawhide.	10:10
f13	This is one aspect of the transition we should think about for F11	10:10
nirik	is there a new fedora-release waiting in rawhide ?	10:11
jeremy	f13: that sounds reasonable to me	10:11
f13	nirik: yeah, there has been one in dist-f11 for quite a while.	10:11
notting	Update 29 Package(s)	10:11
notting	Total download size: 59 M	10:11
notting	not bad for day-0	10:11
f13	nirik: a new fedora-release is needed whenever we create a new build target so it has been there.	10:11
f13	notting: I think there were close to 300 updates for F10	10:11
* nirik nods. Sounds good.		10:11
notting	f13: right, but not all of them applicable to everyone	10:12
f13	right	10:12
f13	327 now to be exact.	10:12
spot	since most of them were mine, they're applicable to almost no one. ;)	10:12
notting	f13: what's your thoughts on the rawhide transition? earlier?	10:12
f13	notting: yeah.	10:12
notting	(since there is no later)	10:12
f13	notting: one thing I was toying with is when we branch and freeze, making the development/ path continue on with say F12 content, and start publishing the F11 content to releases/11/Everything/	10:13
f13	we'd be doing 2 composes a day	10:13
f13	but we let the next rawhide move on, and we start getting things synced up in the releases/ tree	10:13
notting	hm. could work, but could also confuse people about the final release	10:13
wwoods	If I remember right, we were talking about doing the rawhide transition *at* Preview	10:13
f13	notting: yeah, there is certainly going to be confusion	10:13
f13	notting: but there is also confusion now where there is "rawhide" which is the current release, and a hidden rawerhide that is the next release	10:14
f13	one of hte issues I ran into this release was that if we were to let F11 be rawhide earlier, we have no where to redirect mirrormanager requests for fedora '10'	10:14
wwoods	making the final freeze a serious you-better-have-your-bits-in-damn-it Final Friggin' Freeze	10:14
f13	we'd effectively cut off anything but pushed updates for our 10 users.	10:14
notting	i would think (confused users who want rawerhide) << (confused users who think what's there in Everything is final)	10:16
notting	unless we push it to test/	10:16
wwoods	ah, so we solve that by populating the Everything repo before switching the fedora-release file	10:16
notting	maybe push to a RC/ tree under test	10:16
f13	notting: could do that too	10:16
f13	that would probably be less confusion	10:17
f13	confusing.	10:17
f13	We would have to time the composes differently too	10:17
f13	decide which tree starts at 0600 UTC and which one starts either at 0000 UTC or 1200 UTC	10:17
f13	It'll also fracture the testing community, which is a pretty important thing to consider	10:19
f13	anyway, don't need to decide that now.	10:21
f13	anything else on the release of Fedora 10?	10:21
notting	it's been awfully quiet on the mirror front (w.r.t, "can't sync", or "look, a leak!")	10:22
notting	f13: all the torrent stuff is ready for the flip? do we have pre-seeders?	10:23
wwoods	I think fracturing is not a huge worry, since a) moving to rawhide is opt-in, and b) early rawhide is widely known to be wildly unstable	10:23
wwoods	(or at least perceived as such)	10:23
f13	notting: skvidal was taking care of the torrents, they should be good to go, and we have a few pre-seeders who were syncing content over the weekend, I'm going to ping them and make sure they got content.	10:23
f13	notting: it has been quiet, but I think that's mostly because we haven't tried to do a 3 day full sync up like we usually do. We actually gave the mirrors enough time to sync without hysterics	10:24
notting	f13: #1093 can be closed?	10:25
f13	.rel 1093	10:25
zodbot	f13: #1093 (Prepare 0-day updates repos for F10 updates.) - Fedora Release Engineering - Trac - https://fedorahosted.org/rel-eng/ticket/1093	10:25
f13	ah yeah, that can be closed as of this morning by doing the mirrormanager change.	10:25
notting	ah. dependency errors in updates-testing. *sigh*	10:26
f13	yeah, goal for F11 dev cycle is to finally address some of that.	10:26
lmacken	update IDs are still busted. I'm going to push out a fixed bodhi-server today, and reassign about 200 or so IDs.	10:29
f13	nod	10:29
lmacken	I'll send out an errata with the fixed IDs as well	10:29
notting	oh, i noticed a bunch of updates@fp.o mail directly to me with this last push - was that intentional?	10:29
lmacken	no email settings were changed in bodhi	10:30
notting	*shrug* ok. f13: any non-f10 stuff?	10:32
* notting looks at .rel 24		10:32
f13	.rel 24	10:32
zodbot	f13: #24 (spins for Fedora 9) - Fedora Release Engineering - Trac - https://fedorahosted.org/rel-eng/ticket/24	10:32
f13	(zodbot is kind of dumb)	10:32
f13	that's still jwb's item. I keep throwing him under the bus on this one	10:32
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - F11		10:32
f13	so, first order of business for F11 is to nail down a schedule	10:32
f13	we got an end point approved by FESCo, now we have to fill it out, and start discussing some intra-schedule changes	10:33
f13	I had some ideas regarding feature freeze and wwoods has some thoughts regarding the tail end of the schedule	10:33
f13	poelcat: are you around?	10:33
poelcat	yes	10:33
f13	ok	10:35
f13	poelcat: so you know my thoughts already, staggering beta and feature freeze	10:35
f13	wwoods: have you talked to poelcat about your thoughts?	10:36
poelcat	f13: yes i formulated that here: http://poelstra.fedorapeople.org/schedules/f-11/f-11-standard.html	10:36
poelcat	spot also put forth a proposal captured in the other scenario (links at the top)	10:36
f13	nod	10:38
wwoods	well, Rawhide (F12) branching isn't on there, dunno if that's something that usually goes on the schedule	10:39
wwoods	mostly my thoughts are messaging-related: the Final Freeze has been a Semifinal Slush for a couple releases now	10:39
wwoods	I want it to be clearer that all intended final builds *must* be ready by Preview	10:40
wwoods	and I believe f13 and I discussed actually renaming Preview to RC1	10:40
poelcat	wwoods: yes, branching does go on there.. i was attempting to keep it as highlevel as possible right now	10:40
wwoods	poelcat: gotcha	10:40
wwoods	anyway - we called it Preview because we had a history of having a long slush there and calling it a Release Candidate would have been disingenuous	10:41
wwoods	one option was to call it something more accurate. the other option is to actually have that be the Final Freeze so that that build could actually be considered a Release Candidate	10:42
wwoods	we chose the easy way out - the name change	10:42
f13	also, preview came one release because we had a /terrible/ test3 and wound up doing a test4, and people kept asking for that to carry forward	10:42
wwoods	that turns out to be hellish on testing	10:42
wwoods	I'm probably not the only one who's worn out from the last-minute "oh hell quick quick quick rebuild {anaconda,kernel,...}" panics	10:42
notting	so, the only difference from a milestone standpoint with spot's schedule is moving a week between alpha and beta?	10:43
wwoods	it's bad for quality, makes it way harder to get bits into the hands of testers, and hard on all of us personally	10:43
wwoods	the flipside is that it gives developers slightly less time to get in their final bits	10:44
wwoods	but then, development *always* expands to fill the time allotted	10:44
poelcat	notting: feature freeze is offset in the not-spot scenario	10:44
f13	notting: pardon?	10:44
wwoods	so I don't think this plan hurts development unless we surprise people with the change	10:44
f13	wwoods: what is the main problem you wish to solve with the proposed schedule change?	10:45
f13	wwoods: lets look at it from that point of view	10:45
notting	poelcat: oh, blah, taskjuggler silliness again	10:45
wwoods	too much churn during the "final" freeze	10:45
f13	What do we think leads to the churn?	10:46
f13	discovery of bugs due to people using Preview?	10:46
notting	poelcat: the 'duration' numbers for the alpha/beta/preview have zero bearing on the actual duration of those cycles	10:46
f13	putting off of fixing bugs until final freeze?	10:46
wwoods	well, the glibc final bits weren't delivered until.. what, a week ago?	10:46
f13	wwoods: misnomer	10:46
f13	wwoods: the actual bits were the same, they just got a soname change	10:47
f13	which yeah, that blows and we probably shouldn't have allowed that change.	10:47
wwoods	okay, but that sort of packaging change should happen at the final freeze	10:47
wwoods	but, yes, that's a bad example.	10:47
f13	the Preview release served a number of purposes	10:47
wwoods	we allow for a *lot* of churn in key packages, typically relating to features that aren't stable yet	10:48
f13	it gave folks a last snapshot before we made the release to test with	10:48
f13	it gave folks a snapshot of all the things we fixed since the last snapshot, or for many people, since Beta	10:48
* notting wonders how bad other PM software must be if we're actually using taskjuggler		10:48
wwoods	right, I'm not suggesting we drop Preview	10:48
f13	it also served as the "these are the frozen bits" snapshot	10:48
wwoods	but.. let's see. did we revert back to Pidgin post-Preview?	10:49
f13	with lots of discussion, yes	10:49
wwoods	those sorts of decisions: - "are these bits the ones we want in Final	10:49
wwoods	" - should be settled *before* Preview	10:49
wwoods	we should *actually* have all the features testable in Beta, and anything that's not present should be deferred at that point	10:50
f13	I agree, but are we going to be so inflexible so that we would absolutely refuse any change post preview, when it's obviously a good change?	10:50
wwoods	we should *actually* discuss the readiness of those features and decide which ones are going to be OK and which ones aren't *before* Preview	10:50
f13	my thoughts were that freezes make it so that the only changes are changes we expect	10:51
wwoods	I'm not suggesting that, because there's always going to be stuff that needs a second thought after Preview	10:51
f13	be that some change, little change, or no change	10:51
wwoods	it's unavoidable and we've accepted that truth	10:51
wwoods	I guess the main source of churn is new features	10:52
f13	if we're not happy with the changes that are happening at freeze time, isn't it our place to refuse those changes from happening?	10:52
wwoods	Yes. So the simplest way to shape up Final Freeze would be to get more stringent about feature requirements at Beta and Preview	10:53
f13	I guess, is the problem that we're taking the changes, or is the problem that the change was noticed as being needed and presented to us?	10:53
notting	so, one concern about the actual schedule, as presented - freezing *directly* after fudcon may not be useful	10:53
wwoods	the problem appears to be that sometimes we've committed to delivering stuff that might not be ready until well after the Final Freeze	10:54
f13	notting: I had some concern over that as well, not for the changes considered during fudcon, but for the lack of bugfixing by key people during fudcon	10:54
wwoods	some stuff has no problem getting stable by Final Freeze - LIRC and webcam stuff comes to mind	10:54
f13	wwoods: so would the solution be to be more strict about feature acceptance, backed up by a more strict schedule?	10:55
wwoods	but some things we accepted despite the fact that they probably weren't going to be 100% complete by Final Freeze. And I don't think that's a good idea.	10:55
wwoods	yeah. I don't like being more strict but I think we're leaning a little too far toward New Shiny at the cost of Stability	10:56
wwoods	which sucks because I *love* New Shiny	10:56
notting	f13: so, how would we adjust the schedule for this? where do we get that week back?	10:57
wwoods	but we release every 6 months - surely some things can wait an extra few months to be New Shiny *and* Stable	10:57
warren	Have we done a post-mortem to identify the actual "oops that was bad" bugs in F10 final?	10:57
notting	f13: given that there's two months between alpha and beta, I think we have plenty of room to push alpha bback	10:57
f13	notting: "this" ?	10:57
notting	f13: fudcon	10:57
f13	ah	10:57
warren	I'm only aware of two really bad bugs despite the churn at the end.	10:57
wwoods	part of the process is requiring more detailed feature scope/specifications by Alpha	10:58
wwoods	and using that to write a Test Plan by Beta	10:58
wwoods	which makes it much easier to decide whether to keep or defer a feature before Preview	10:58
f13	notting: I think my off the cuff suggestion would be to move the alpha freeze one week later.	10:58
f13	I think I talked with poelcat about this and he had some thoughts to the counter.	10:58
notting	why do we have a two-month alpha anyway?	11:00
warren	wwoods: what are actual bugs that are the result of the post-freeze churn?	11:00
wwoods	so, yes, more detailed Feature requirements (scope + test plan), and more strictness from FESCo/rel-eng regarding what we'll consider acceptable changes post-Final Freeze	11:00
f13	notting: because we usually have a 2 month alpha?	11:00
spot	guys, i have to go to a 2 PM meeting	11:01
f13	spot: ok, thanks.	11:01
f13	wwoods: would it help if post-freeze tag requests had more input from QA?	11:01
f13	I would really hate going down the route of dev ack, qa ack	11:01
wwoods	no, I think we have sufficient input there	11:01
wwoods	yeah I don't want that either	11:01
poelcat	f13: all I can remember re: fudcon was alpha is non-block freeze so it wasn't that critical?	11:01
wwoods	I think most of the changes need to made at the Feature Process level	11:02
f13	poelcat: it's non-blocker, but it's still somewhat critical to get it out	11:02
f13	wwoods: ok. Lucky for you both the feature person and the schedule person are the same.	11:02
wwoods	indeed!	11:02
f13	poelcat: I think it's prudent to allow some post-fudcon time to fix up any bugs that got ignored during that time, so can we push that freeze date out a week, and then use that as the basis of a schedule for releng to vote on next week?	11:04
notting	f13: we do?	11:04
wwoods	I think the only real change needed at the rel-eng level is early (and loud) announcements that Final Freeze is supposed to be FINAL and we only accept builds that fix blocker/target bugs after that	11:04
f13	notting: we do what?	11:04
poelcat	f13: of the two proposed schedules... which one is being selected to make those changes to?	11:04
notting	f13: two month alphas	11:05
f13	poelcat: both would need that treatment wouldn't they?	11:05
notting	poelcat: it doesn't matter	11:05
f13	poelcat: don't both schedules have alpha freeze at the same time?	11:05
f13	notting: I was just going by the research that spot and poelcat had done in the past	11:05
poelcat	f13: alpha freeze is the same; feature freeze is not	11:05
notting	poelcat: ... we're talking about moving alpha. feature freeze is irrelevant, ergo it gets moved in both	11:07
f13	poelcat: we're talking about alpha freeze, which has a conflict with fudcon	11:07
notting	f13: f10 was a shade under two months. f9 was 1 1/2 - 1 3/4. f8 was 1 month	11:07
f13	notting: interesting.	11:08
f13	would it make sense then	11:08
f13	n/m	11:08
poelcat	f13: okay; move alpha freeze one week later?	11:10
f13	please	11:10
* poelcat is also going to reduce down to ONE scenario to argue about :)		11:10
f13	heh	11:10
f13	moving alpha freeze a week later should also move alpha release a week later	11:11
f13	but that should be the end of that ripple effect	11:11
poelcat	correct. that is what I was going to do	11:11
poelcat	originally i was fearing more was being suggested :)	11:11
poelcat	one other question remains unanswered: three freezes up to beta	11:12
poelcat	1) feature freeze	11:12
poelcat	2) critical package freeze	11:12
poelcat	3) beta freeze	11:12
poelcat	should they all be separated by a week?	11:12
notting	poelcat: is there a way to fix it so that the durations for '... release' consistently are from public availability to public availability of next milestone?	11:12
notting	i personally think #1 and #2 should be coincidental	11:12
f13	poelcat: I think I made a mistake a bit with pushing for critical package freeze. That's something we'll want to approach each individual package and ask them for consideration	11:13
f13	poelcat: so one week between feature freeze and beta freeze.	11:13
poelcat	f13: and remove 'critical package' freeze from proposed schedule?	11:14
f13	yeah	11:14
poelcat	notting: i'm not sure how to tweak that... the reporting, as you've commented, leaves a little to be desired :)	11:15
poelcat	f13: okay I will make those changes and update the ticket	11:15
poelcat	wwoods: let me know what changes you'd like to make the feature process/policy, etc.	11:16
f13	thanks, we should send mail / poke on IRC that we'll vote on it next week	11:16
wwoods	poelcat: I think it was discussed in the FESCo meeting last week but it will probably come up again.	11:16
wwoods	I've been trying to make sure the relevant sections in the empty Feature page are clearer, but let me know if I've missed anything	11:17
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - open floor		11:18
f13	anything else before we wrap?	11:18
poelcat	wwoods: i need to know which feature pages you believe need more substance from a QA perspective... i'm doing a general review at approval time and thereafter just checking for status	11:18
wwoods	poelcat: right, I'll be on hand for FESCo meetings wherever possible to ensure that's the case	11:19
wwoods	(except probably not this week since I'll be in Alabama tomorrow through Saturday)	11:20
* mitr raises hand		11:21
mitr	First code drop for the signing server is at http://people.redhat.com/mitr/rpmsigner-0.tar.bz2 . I'd very much appreciate if someone could take a look and check the interface is usable and the functionality sufficient.	11:22
mitr	I'd also like to temporarily get a higher-privileged account for koji (allowed to store signatures and to authenticate as other users) to test the signature storing side. Who can I calk to about that?	11:23
mitr	s/calk/talk/	11:23
mbonnet	mitr: why do you need to auth as other users?	11:24
f13	and could we actually do this with a test koji instance, rather than the production one?	11:25
mitr	mbonnet: The idea is that the user's "client" does not download the signed RPM locally in order to store the signature; instead a helper server that runs near other koji servers stores the signature on behalf of the user (who is authenticated using a SSL certificate)	11:25
mitr	f13: Is there a test instance available? If not, I can of course set one up, but i was hoping to avoid that work.	11:26
mbonnet	mitr: you might want to talk to mikem23, he has an internal test instance	11:26
mitr	mbonnet: Thank you.	11:26
f13	anything else?	11:28
dgilmore	mitr: why do you have pygpgme-0.1 python-nss-0.1 in the tarball? rather than using system provided versions	11:29
mitr	dgilmore: They are slightly patched. The patches are now in bugzilla, but not accepted upstream yet.	11:29
mitr	(#472805, #470271, #470272)	11:30
f13	ok, this conversation can continue, but I'm going to close the meeting, thanks all!	11:30

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!