From Fedora Project Wiki

Fedora Release Engineering Meeting :: Monday 2008-11-24

Fedora 10

  • In good shape for release of Fedora 10 tomorrow
  • Fedora 10 zero day updates are ready to go
  • f13: one thing I was toying with is when we branch and freeze, making the development/ path continue on with say F12 content, and start publishing the F11 content to releases/11/Everything/

Fedora 11 Schedule

  • need to make sure F12 branching is visible
  • finding ways to reduce package churn
  • QA team monitoring status of features during release
  • proposal at
  • poelcat to make adjustments to proposed schedule and republish
    1. move alpha freeze forward one week
    2. move alpha release forward one week
    3. remove critical package freeze from schedule
    4. offset feature freeze and beta freeze by one week
  • vote on schedule to present to FESCo next week

IRC Transcript

f13 ping: notting jeremy rdieter wwoods lmacken warren spot 10:00
jeremy hi 10:00
f13 poelcat: ping 10:00
* lmacken 10:00
* poelcat here 10:01
notting sorry, here now 10:01
* spot is here 10:01
wwoods zoom! ding ding. 10:02
f13 whee. 10:04
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - Fedora 10 10:04
f13 so there is this thing we gotta do tomorrow. 10:04
f13 no big thing, just take few moments of our time 10:05
f13 releasing Fedora 10 10:05
wwoods drive to alabama for thanksgiving with our wife's family? 10:05
wwoods oh, maybe that's just me 10:05
f13 wwoods: yes, that's it 10:05
f13 We're in really good shape 10:05
wwoods awesome. hope you guys like deep-fried cajun turkey 10:05
f13 all our bits are very well synced to a number of mirrors, we've got 0-day F10 updates in place and about to be public via mirrormanager, skvidal got the bits synced over to the torrent box and torrents made up, and we've got some pre-seeding of torrents setup too 10:06
f13 the key peices left are a tested release webpage set, a release announcement, and me pulling the trigger early tomorrow 10:06
f13 and then it's on to turkey and drinking 10:07
f13 oh and F11 10:08
notting who is doing the release announcement? 10:08
f13 notting: the docs team usually has a release announcement made, and I send it out 10:08
wwoods f13: don't forget flipping the bits in releases.txt for preupgraders 10:08
f13 the web team preps the pages for release 10:08
f13 wwoods: yep, that's a current filed ticket in the Fedora 10 Final tracker in trac 10:08
wwoods (I know, there's already a ticket for that, just mentioning it with the other bits) 10:08
f13 er milestone 10:08
f13 I was going to flip the rawhide bit yesterday but wound up taking a much much needed night away from the kid on a date with my wife 10:09
f13 and spending part of that on the computer would have been not good for my health 10:09
wwoods yeah, we all run pretty close to computer-burnout-redline around release time 10:09
f13 and since we don't want rawhide sync getting in the way of our permission bit flip, I'm not going to reset rawhide today either, I'll do it tomorrow, for Wed's rawhide. 10:10
f13 This is one aspect of the transition we should think about for F11 10:10
nirik is there a new fedora-release waiting in rawhide ? 10:11
jeremy f13: that sounds reasonable to me 10:11
f13 nirik: yeah, there has been one in dist-f11 for quite a while. 10:11
notting Update 29 Package(s) 10:11
notting Total download size: 59 M 10:11
notting not bad for day-0 10:11
f13 nirik: a new fedora-release is needed whenever we create a new build target so it has been there. 10:11
f13 notting: I think there were close to 300 updates for F10 10:11
* nirik nods. Sounds good. 10:11
notting f13: right, but not all of them applicable to everyone 10:12
f13 right 10:12
f13 327 now to be exact. 10:12
spot since most of them were mine, they're applicable to almost no one. ;) 10:12
notting f13: what's your thoughts on the rawhide transition? earlier? 10:12
f13 notting: yeah. 10:12
notting (since there is no later) 10:12
f13 notting: one thing I was toying with is when we branch and freeze, making the development/ path continue on with say F12 content, and start publishing the F11 content to releases/11/Everything/ 10:13
f13 we'd be doing 2 composes a day 10:13
f13 but we let the next rawhide move on, and we start getting things synced up in the releases/ tree 10:13
notting hm. could work, but could also confuse people about the final release 10:13
wwoods If I remember right, we were talking about doing the rawhide transition *at* Preview 10:13
f13 notting: yeah, there is certainly going to be confusion 10:13
f13 notting: but there is also confusion now where there is "rawhide" which is the current release, and a hidden rawerhide that is the next release 10:14
f13 one of hte issues I ran into this release was that if we were to let F11 be rawhide earlier, we have no where to redirect mirrormanager requests for fedora '10' 10:14
wwoods making the final freeze a serious you-better-have-your-bits-in-damn-it Final Friggin' Freeze 10:14
f13 we'd effectively cut off anything but pushed updates for our 10 users. 10:14
notting i would think (confused users who want rawerhide) << (confused users who think what's there in Everything is final) 10:16
notting unless we push it to test/ 10:16
wwoods ah, so we solve that by populating the Everything repo before switching the fedora-release file 10:16
notting maybe push to a RC/ tree under test 10:16
f13 notting: could do that too 10:16
f13 that would probably be less confusion 10:17
f13 confusing. 10:17
f13 We would have to time the composes differently too 10:17
f13 decide which tree starts at 0600 UTC and which one starts either at 0000 UTC or 1200 UTC 10:17
f13 It'll also fracture the testing community, which is a pretty important thing to consider 10:19
f13 anyway, don't need to decide that now. 10:21
f13 anything else on the release of Fedora 10? 10:21
notting it's been awfully quiet on the mirror front (w.r.t, "can't sync", or "look, a leak!") 10:22
notting f13: all the torrent stuff is ready for the flip? do we have pre-seeders? 10:23
wwoods I think fracturing is not a huge worry, since a) moving to rawhide is opt-in, and b) early rawhide is widely known to be wildly unstable 10:23
wwoods (or at least perceived as such) 10:23
f13 notting: skvidal was taking care of the torrents, they should be good to go, and we have a few pre-seeders who were syncing content over the weekend, I'm going to ping them and make sure they got content. 10:23
f13 notting: it has been quiet, but I think that's mostly because we haven't tried to do a 3 day full sync up like we usually do. We actually gave the mirrors enough time to sync without hysterics 10:24
notting f13: #1093 can be closed? 10:25
f13 .rel 1093 10:25
zodbot f13: #1093 (Prepare 0-day updates repos for F10 updates.) - Fedora Release Engineering - Trac - 10:25
f13 ah yeah, that can be closed as of this morning by doing the mirrormanager change. 10:25
notting ah. dependency errors in updates-testing. *sigh* 10:26
f13 yeah, goal for F11 dev cycle is to finally address some of that. 10:26
lmacken update IDs are still busted. I'm going to push out a fixed bodhi-server today, and reassign about 200 or so IDs. 10:29
f13 nod 10:29
lmacken I'll send out an errata with the fixed IDs as well 10:29
notting oh, i noticed a bunch of updates@fp.o mail directly to me with this last push - was that intentional? 10:29
lmacken no email settings were changed in bodhi 10:30
notting *shrug* ok. f13: any non-f10 stuff? 10:32
* notting looks at .rel 24 10:32
f13 .rel 24 10:32
zodbot f13: #24 (spins for Fedora 9) - Fedora Release Engineering - Trac - 10:32
f13 (zodbot is kind of dumb) 10:32
f13 that's still jwb's item. I keep throwing him under the bus on this one 10:32
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - F11 10:32
f13 so, first order of business for F11 is to nail down a schedule 10:32
f13 we got an end point approved by FESCo, now we have to fill it out, and start discussing some intra-schedule changes 10:33
f13 I had some ideas regarding feature freeze and wwoods has some thoughts regarding the tail end of the schedule 10:33
f13 poelcat: are you around? 10:33
poelcat yes 10:33
f13 ok 10:35
f13 poelcat: so you know my thoughts already, staggering beta and feature freeze 10:35
f13 wwoods: have you talked to poelcat about your thoughts? 10:36
poelcat f13: yes i formulated that here: 10:36
poelcat spot also put forth a proposal captured in the other scenario (links at the top) 10:36
f13 nod 10:38
wwoods well, Rawhide (F12) branching isn't on there, dunno if that's something that usually goes on the schedule 10:39
wwoods mostly my thoughts are messaging-related: the Final Freeze has been a Semifinal Slush for a couple releases now 10:39
wwoods I want it to be clearer that all intended final builds *must* be ready by Preview 10:40
wwoods and I believe f13 and I discussed actually renaming Preview to RC1 10:40
poelcat wwoods: yes, branching does go on there.. i was attempting to keep it as highlevel as possible right now 10:40
wwoods poelcat: gotcha 10:40
wwoods anyway - we called it Preview because we had a history of having a long slush there and calling it a Release Candidate would have been disingenuous 10:41
wwoods one option was to call it something more accurate. the other option is to actually have that be the Final Freeze so that that build could actually be considered a Release Candidate 10:42
wwoods we chose the easy way out - the name change 10:42
f13 also, preview came one release because we had a /terrible/ test3 and wound up doing a test4, and people kept asking for that to carry forward 10:42
wwoods that turns out to be hellish on testing 10:42
wwoods I'm probably not the only one who's worn out from the last-minute "oh hell quick quick quick rebuild {anaconda,kernel,...}" panics 10:42
notting so, the only difference from a milestone standpoint with spot's schedule is moving a week between alpha and beta? 10:43
wwoods it's bad for quality, makes it way harder to get bits into the hands of testers, and hard on all of us personally 10:43
wwoods the flipside is that it gives developers slightly less time to get in their final bits 10:44
wwoods but then, development *always* expands to fill the time allotted 10:44
poelcat notting: feature freeze is offset in the not-spot scenario 10:44
f13 notting: pardon? 10:44
wwoods so I don't think this plan hurts development unless we surprise people with the change 10:44
f13 wwoods: what is the main problem you wish to solve with the proposed schedule change? 10:45
f13 wwoods: lets look at it from that point of view 10:45
notting poelcat: oh, blah, taskjuggler silliness again 10:45
wwoods too much churn during the "final" freeze 10:45
f13 What do we think leads to the churn? 10:46
f13 discovery of bugs due to people using Preview? 10:46
notting poelcat: the 'duration' numbers for the alpha/beta/preview have zero bearing on the actual duration of those cycles 10:46
f13 putting off of fixing bugs until final freeze? 10:46
wwoods well, the glibc final bits weren't delivered until.. what, a week ago? 10:46
f13 wwoods: misnomer 10:46
f13 wwoods: the actual bits were the same, they just got a soname change 10:47
f13 which yeah, that blows and we probably shouldn't have allowed that change. 10:47
wwoods okay, but that sort of packaging change should happen at the final freeze 10:47
wwoods but, yes, that's a bad example. 10:47
f13 the Preview release served a number of purposes 10:47
wwoods we allow for a *lot* of churn in key packages, typically relating to features that aren't stable yet 10:48
f13 it gave folks a last snapshot before we made the release to test with 10:48
f13 it gave folks a snapshot of all the things we fixed since the last snapshot, or for many people, since Beta 10:48
* notting wonders how bad other PM software must be if we're actually using taskjuggler 10:48
wwoods right, I'm not suggesting we drop Preview 10:48
f13 it also served as the "these are the frozen bits" snapshot 10:48
wwoods but.. let's see. did we revert back to Pidgin post-Preview? 10:49
f13 with lots of discussion, yes 10:49
wwoods those sorts of decisions: - "are these bits the ones we want in Final 10:49
wwoods " - should be settled *before* Preview 10:49
wwoods we should *actually* have all the features testable in Beta, and anything that's not present should be deferred at that point 10:50
f13 I agree, but are we going to be so inflexible so that we would absolutely refuse any change post preview, when it's obviously a good change? 10:50
wwoods we should *actually* discuss the readiness of those features and decide which ones are going to be OK and which ones aren't *before* Preview 10:50
f13 my thoughts were that freezes make it so that the only changes are changes we expect 10:51
wwoods I'm not suggesting that, because there's always going to be stuff that needs a second thought after Preview 10:51
f13 be that some change, little change, or no change 10:51
wwoods it's unavoidable and we've accepted that truth 10:51
wwoods I guess the main source of churn is new features 10:52
f13 if we're not happy with the changes that are happening at freeze time, isn't it our place to refuse those changes from happening? 10:52
wwoods Yes. So the simplest way to shape up Final Freeze would be to get more stringent about feature requirements at Beta and Preview 10:53
f13 I guess, is the problem that we're taking the changes, or is the problem that the change was noticed as being needed and presented to us? 10:53
notting so, one concern about the actual schedule, as presented - freezing *directly* after fudcon may not be useful 10:53
wwoods the problem appears to be that sometimes we've committed to delivering stuff that might not be ready until well after the Final Freeze 10:54
f13 notting: I had some concern over that as well, not for the changes considered during fudcon, but for the lack of bugfixing by key people during fudcon 10:54
wwoods some stuff has no problem getting stable by Final Freeze - LIRC and webcam stuff comes to mind 10:54
f13 wwoods: so would the solution be to be more strict about feature acceptance, backed up by a more strict schedule? 10:55
wwoods but some things we accepted despite the fact that they probably weren't going to be 100% complete by Final Freeze. And I don't think that's a good idea. 10:55
wwoods yeah. I don't like being more strict but I think we're leaning a little too far toward New Shiny at the cost of Stability 10:56
wwoods which sucks because I *love* New Shiny 10:56
notting f13: so, how would we adjust the schedule for this? where do we get that week back? 10:57
wwoods but we release every 6 months - surely some things can wait an extra few months to be New Shiny *and* Stable 10:57
warren Have we done a post-mortem to identify the actual "oops that was bad" bugs in F10 final? 10:57
notting f13: given that there's two months between alpha and beta, I think we have plenty of room to push alpha bback 10:57
f13 notting: "this" ? 10:57
notting f13: fudcon 10:57
f13 ah 10:57
warren I'm only aware of two really bad bugs despite the churn at the end. 10:57
wwoods part of the process is requiring more detailed feature scope/specifications by Alpha 10:58
wwoods and using that to write a Test Plan by Beta 10:58
wwoods which makes it much easier to decide whether to keep or defer a feature before Preview 10:58
f13 notting: I think my off the cuff suggestion would be to move the alpha freeze one week later. 10:58
f13 I think I talked with poelcat about this and he had some thoughts to the counter. 10:58
notting why do we have a two-month alpha anyway? 11:00
warren wwoods: what are actual bugs that are the result of the post-freeze churn? 11:00
wwoods so, yes, more detailed Feature requirements (scope + test plan), and more strictness from FESCo/rel-eng regarding what we'll consider acceptable changes post-Final Freeze 11:00
f13 notting: because we usually have a 2 month alpha? 11:00
spot guys, i have to go to a 2 PM meeting 11:01
f13 spot: ok, thanks. 11:01
f13 wwoods: would it help if post-freeze tag requests had more input from QA? 11:01
f13 I would really hate going down the route of dev ack, qa ack 11:01
wwoods no, I think we have sufficient input there 11:01
wwoods yeah I don't want that either 11:01
poelcat f13: all I can remember re: fudcon was alpha is non-block freeze so it wasn't that critical? 11:01
wwoods I think most of the changes need to made at the Feature Process level 11:02
f13 poelcat: it's non-blocker, but it's still somewhat critical to get it out 11:02
f13 wwoods: ok. Lucky for you both the feature person and the schedule person are the same. 11:02
wwoods indeed! 11:02
f13 poelcat: I think it's prudent to allow some post-fudcon time to fix up any bugs that got ignored during that time, so can we push that freeze date out a week, and then use that as the basis of a schedule for releng to vote on next week? 11:04
notting f13: we do? 11:04
wwoods I think the only real change needed at the rel-eng level is early (and loud) announcements that Final Freeze is supposed to be FINAL and we only accept builds that fix blocker/target bugs after that 11:04
f13 notting: we do what? 11:04
poelcat f13: of the two proposed schedules... which one is being selected to make those changes to? 11:04
notting f13: two month alphas 11:05
f13 poelcat: both would need that treatment wouldn't they? 11:05
notting poelcat: it doesn't matter 11:05
f13 poelcat: don't both schedules have alpha freeze at the same time? 11:05
f13 notting: I was just going by the research that spot and poelcat had done in the past 11:05
poelcat f13: alpha freeze is the same; feature freeze is not 11:05
notting poelcat: ... we're talking about moving alpha. feature freeze is irrelevant, ergo it gets moved in both 11:07
f13 poelcat: we're talking about alpha freeze, which has a conflict with fudcon 11:07
notting f13: f10 was a shade under two months. f9 was 1 1/2 - 1 3/4. f8 was 1 month 11:07
f13 notting: interesting. 11:08
f13 would it make sense then 11:08
f13 n/m 11:08
poelcat f13: okay; move alpha freeze one week later? 11:10
f13 please 11:10
* poelcat is also going to reduce down to ONE scenario to argue about :) 11:10
f13 heh 11:10
f13 moving alpha freeze a week later should also move alpha release a week later 11:11
f13 but that should be the end of that ripple effect 11:11
poelcat correct. that is what I was going to do 11:11
poelcat originally i was fearing more was being suggested :) 11:11
poelcat one other question remains unanswered: three freezes up to beta 11:12
poelcat 1) feature freeze 11:12
poelcat 2) critical package freeze 11:12
poelcat 3) beta freeze 11:12
poelcat should they all be separated by a week? 11:12
notting poelcat: is there a way to fix it so that the durations for '... release' consistently are from public availability to public availability of next milestone? 11:12
notting i personally think #1 and #2 should be coincidental 11:12
f13 poelcat: I think I made a mistake a bit with pushing for critical package freeze. That's something we'll want to approach each individual package and ask them for consideration 11:13
f13 poelcat: so one week between feature freeze and beta freeze. 11:13
poelcat f13: and remove 'critical package' freeze from proposed schedule? 11:14
f13 yeah 11:14
poelcat notting: i'm not sure how to tweak that... the reporting, as you've commented, leaves a little to be desired :) 11:15
poelcat f13: okay I will make those changes and update the ticket 11:15
poelcat wwoods: let me know what changes you'd like to make the feature process/policy, etc. 11:16
f13 thanks, we should send mail / poke on IRC that we'll vote on it next week 11:16
wwoods poelcat: I think it was discussed in the FESCo meeting last week but it will probably come up again. 11:16
wwoods I've been trying to make sure the relevant sections in the empty Feature page are clearer, but let me know if I've missed anything 11:17
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - open floor 11:18
f13 anything else before we wrap? 11:18
poelcat wwoods: i need to know which feature pages you believe need more substance from a QA perspective... i'm doing a general review at approval time and thereafter just checking for status 11:18
wwoods poelcat: right, I'll be on hand for FESCo meetings wherever possible to ensure that's the case 11:19
wwoods (except probably not this week since I'll be in Alabama tomorrow through Saturday) 11:20
* mitr raises hand 11:21
mitr First code drop for the signing server is at . I'd very much appreciate if someone could take a look and check the interface is usable and the functionality sufficient. 11:22
mitr I'd also like to temporarily get a higher-privileged account for koji (allowed to store signatures and to authenticate as other users) to test the signature storing side. Who can I calk to about that? 11:23
mitr s/calk/talk/ 11:23
mbonnet mitr: why do you need to auth as other users? 11:24
f13 and could we actually do this with a test koji instance, rather than the production one? 11:25
mitr mbonnet: The idea is that the user's "client" does not download the signed RPM locally in order to store the signature; instead a helper server that runs near other koji servers stores the signature on behalf of the user (who is authenticated using a SSL certificate) 11:25
mitr f13: Is there a test instance available? If not, I can of course set one up, but i was hoping to avoid that work. 11:26
mbonnet mitr: you might want to talk to mikem23, he has an internal test instance 11:26
mitr mbonnet: Thank you. 11:26
f13 anything else? 11:28
dgilmore mitr: why do you have pygpgme-0.1 python-nss-0.1 in the tarball? rather than using system provided versions 11:29
mitr dgilmore: They are slightly patched. The patches are now in bugzilla, but not accepted upstream yet. 11:29
mitr (#472805, #470271, #470272) 11:30
f13 ok, this conversation can continue, but I'm going to close the meeting, thanks all! 11:30

Generated by 2.7 by Marius Gedminas - find it at!