From Fedora Project Wiki

Fedora Release Engineering Meeting :: Monday 2008-04-14

Preview Release (aka PR) Status

  • Not ready--scheduled for availability on 2008-04-11
  • Problems with yum and other compose tools
  • Snafu with package signing which caused rawhide to revert back to all unsigned copies of most packages--100G worth of churn
  • Spend rest of today untangling rawhide situation
  • Attempt Preview Release on 2008-04-15 with no change to GA date of 2008-04-29

Mass Branching

  • Toshio says that the tools (pkgdb, etc..) are ready for the mass branching.
  • Should probably have a CVS outage to get the branching done--usually takes a full work day
  • Discusison left unfinished

What is the Preview Release?

  • Just another snapshot or equivalent to Alpha and Beta which requires more formal testing before release?
  • Discussion about confusion over what preview release is
  • Need clearer definition for what Preview Release and Release Candidate mean

1. Poelcat to take a shot at documenting purpose and criteria for snapshots and test releases 1. Preview Release will be as installable and functional as other test releases (Alpha and Beta)

IRC Transcript

f13ping: notting jeremy spot lmacken warren rdieter wwoods poelcat jwb13:00
* notting is here13:00
nottingalthough still drowning in mail13:00
skvidalnotting: I've got a filter that helps with that - search for @ in the sender's address and delete the mail if it is found13:01
wwoodsooh, was about to reboot for preupgrade testing13:03
* wwoods stays his hand for a bit13:03
f13alright, lets kick this pig.13:05
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Preview Release13:05
f13I"d really like to get PR out the door, but the universe hates me.13:05
f13I'm convinced.13:05
f13After all the battling with yum and other compose tool fun, I was all set to get PR composed and out the door today.13:05
* jeremy is sort of here13:05
f13Unfortunately, that's not to be.13:05
f13Due to a snafu with package signing, we caused rawhide to revert back to all unsigned copies of most packages, some 100G worth of churn13:06
f13probably not the best to compose PR against.13:06
* poelcat here13:06
jeremyf13: doh.  I was wondering why I saw so much churn in my sync but thought it was going to signed not the unsigned packages :(13:06
f13I'm in the middle of trying to recover the situation, and we've turned off snapmirror tos that PHX/TPA don't get more of the bad bits13:06
f13jeremy: sadly, mash has a "if I don't find the preferred signed copy, I fall back to unsigned" method that I wasn't fully aware of.13:07
* rdieter was wondering why his rawhide mirror sync was taking so long to finish today, rats.13:07
nottingf13: well, the alternative is to not fall back and fail13:07
f13so I'll likely spend the entire day cleaning up from that, and investigating why sign_unsigned suddenly is getting md5 mismatches on the things it signs.13:08
nottingwhich also won't help you13:08
f13notting: or stop switching gpg sigs needlessly13:08
f13at any rate, I may end up composing a PR against a recovered snapshot from /yesterday's/ rawhide13:09
f13if it looks like recovering todays rawhide bits is going to take too long13:09
f13but I haven't looked at what all got signed last night that would be good to have in the PR13:10
f13Looking beyond PR...13:11
f13I think we'll still be taking in tag requests until early next week13:11
f13then we need to crack down and start considering release candidates, and what are actual release blockers13:11
f13anything more on Preview Release?13:13
nottingdo we know of any PR blockers aside from the 'getting it composed issues'?13:14
f13jeremy said there was one bug left on the PR blocker list13:14
jeremythe x keyboard config one13:14
f13but that's the only thing I'm aware of, I thought we were in pretty damn good shape for PR13:15
jeremyand I really really really don't know if I feel comfortable with the RC being the first thing that has a fix for that13:15
jeremyI know ajax was working on it on friday, but I didn't quite catch what he was saying the problem was on the bus (since I was in the python unicode discussion instead ;-)13:15
jeremyand I've been in class all morning13:15
nottingwhat's wrong with keyboards today?13:16
f13if you set a layout at install, it doesn't follow through to X once installed I think13:17
nottingoh, because we're only writing to sysconfig, not to X13:19
nottingunless you use s-c-keyboard, which writes to both13:19
f13I think so?  I'm not clear on the problem space13:19
f13so for the keyboard thing, we may be putning the PR to tomorrow anyway13:21
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Mass Branching13:22
rdieterpunt++ , will the package signing be resolved by then?13:22
wwoodssince we're now a week from RC I don't think we should slip the PR any further for fixes13:22
f13Toshio says that the tools (pkgdb, etc..) are ready for the mass branching.13:23
f13unfortunately unless we want to mailbomb everybody, we should probably have a CVS outage to get the branching done, and that usually takes a full work day13:23
f13and I have 0 time to focus on it right now13:23
f13lets put CVS branchingon hold for a moment.13:23
f13wwoods: You'd be OK with PR going out with the current blocker unfixed?13:24
f13(if so, mind moving that blocker off of the PR blocker list)13:24
wwoodslemme recheck the blocker list, but seriously if we're releasing another snapshot in a week *and* we have a fix approaching the proverbial runway13:25
wwoodsthen, yeah, get the dang thing into people's hands and release note it13:25
wwoods"we know, it'll be fixed next week, just ssshh"13:25
f13wwoods: I don't think there is really time in the schedule to "release" RCs13:25
f13wwoods: so we'd be relying on rawhide users to confirm if it's fixed or not13:26
wwoodswait, what? I must have misunderstood something13:26
wwoods22 April 200813:26
wwoodsRelease Candidate 113:26
f13RCs are seriously going to be compose as fast as you can, multiple times a day, and drop it on the mirrors ASAP13:26
wwoodsThere's no schedule for *subsequent* RCs - those are at our option13:26
wwoodshm, okay13:27
wwoodsso we're not planning on putting RC1 on the mirrors?13:27
f13I have to have something going to the mirrors by the 24th really for a successful 29th release13:27
f13wwoods: there is not enough time.13:27
wwoodsforgive me if I'm getting confused, I think I'm coming down with something13:27
f13mirrors == 3~ days of sync lead time13:27
f13often 3 business days13:28
f13that's for a "minor" release.  For a major release, we want a lot more time to have a lot more mirrors ready for the bit flip13:28
nottingwell, i think the issues is just that the schedule says 'Preview Release' and 'Release Candidate 1'13:28
nottingare those defined?13:28
f13notting: not well.  PR is new for this round, and "RC" is just something we used to call attempts to do the final compose13:28
f13I don't think there was any promise that "RC" would see the light of day in the public space13:29
nottingno, but we should probably add something somewhere that defines what they are, or we'll be answering this question again13:29
wwoodsYeah, my memory's hazy but I think f13's right and I've just confused myself13:30
wwoodsso yeah, that needs a more detailed explanation somewhere, I guess13:30
wwoodsand if we're not planning on doing another fullblown worldwide mirrored release until final13:30
wwoodsthen yes, I guess we'll need to continue to block on bug 43824613:31
buggbotBug low, low, ---, Kristian Høgsberg, ASSIGNED , Keyboard layout selected during install is forgotten at post-install boot13:31
f13I know we've already got a potential schedule for F10, but yeah, I think we seriously need to discuss what we're trying to accomplish with PR, and RC, etc...13:32
f13I originally thought that PR was just going to be the regularly scheduled snapshot of the week attempt13:32
f13just under a new name13:32
f13not that we'd be holding it up for bugs, or treating it all that special, or mirroring it, just bog standard snapshot on torrent13:33
f13and that we'd continue to use rawhide for testing things we want to get fixed before we say "Ok, we're good enough, start attempting the release spin" and if the spin is good, as in nothing fell over during compose, that was it, we're shipped13:33
f13trying to do public releases of candidate trees is going to put a serious hurt on things13:34
f13anyway.... back to /topic13:35
wwoodsNo, we wanted a third fullblown test release before final13:35
f13if anybody wants to tackle cvs branching, by all means, I'd be very happy13:35
f13I have some notes from last time, and toshio started an SOP page to outline calls to pkgdb and such13:36
* poelcat wonders how much we reduce our potential pool of testers when we reduce availability to rawhide13:36
nottingwwoods: f13: so, how do we resolve this dichotomy13:36
wwoodsbut RCs don't need to be fullblown mirrored releases13:36
f13by the time we get to RC state, it should seriously be an RC, as in all the bugs are fixed, we're just putting the tested builds into a tree, and making sure the tree compose itself doesn't fall over13:37
f13I'm not sure what the value is of trying to put that tree out to the public and waiting for somebody to give it thumbs up13:38
wwoodsthat's what the PR is for13:38
poelcatwwoods: +113:39
f13wwoods: so what do we do with the time between cuttin the PR, having it mirrored, getting people to download and test, and report things, and cutting RC?13:39
f13do we take in other bugfixes people find?13:39
f13do we just say "nope, sorry, th'at snot a blocker, fix it in an update?"13:39
wwoodsthe reason it's called "Preview Release" and not "Public Release Candidate" is that we *knew* there'd be unfixed blockers at this point13:40
f13and if we're going to take fixes in, what's the criteria for what we wait for to be fixed for PR, vs what we'll allow to be fixed after PR goes out?13:40
wwoodsso calling it an RC would have just been an obvious lie13:40
wwoodsthe criteria for PR are the same as for Alpha and Beta13:40
wwoodsseriously, (alpha, beta, preview_release) = (test1, test2, test3)13:41
f13if that were the case, we wouldn't be blocking on the X thing13:41
wwoodswe changed the names and shifted dates but that was the original plan13:41
wwoodsis that right? yeah I guess so13:41
f13wwoods: I think that really fell apart when we scheduled the PR for 3 days after the final freeze13:42
wwoodsI'm not sure what criteria made the X thing a blocker in the first place, actually13:42
f13and didn't give it any compose/stage/sync time13:42
f13I was under the (mistaken?) impression that PR was just going to be a weekly snapshot, tossed onto torrent13:43
wwoodsno, PR was supposed to map to test313:43
f13and would service as a post-freeze media set to be the basis of further testing13:43
wwoods*that* part is true13:43
f13ok, our signals were quite crossed then13:43
f13the schedule certainly doesn't have room for a full blown test3 like release13:44
wwoodsokay then13:44
f13I thought we obviated the need for a 3rd major developmental milestone13:44
wwoodslet's go with your plan13:44
wwoodsbecause it reduces our workload13:44
f13'cause it just make things take too long13:44
wwoodsand if anyone actually flips out13:44
f13we'll have me to blame (:13:44
wwoodsor if it actually has an adverse effect13:44
nottingso, 1) do we need to document this 2) do we need to adjust the f10 schedule to have PR be a 'release'?13:45
wwoodsthen we'll rejigger the F10 schedule around the assumption that PR == test313:45
wwoods1) yes, 2) no, unless this goes horribly awry13:45
wwoodssince we're doing (mostly) weekly snapshot releases, we're putting a lot more testable bits in people's hands13:46
wwoodsthan just test1/test2/test313:46
wwoodsobviously that's More, Fresher Bits13:47
wwoodspractially farm-fresh13:47
poelcatwwoods: except that the snaps have to install... they haven't for me13:47
wwoodsthat has nothing to do with the schedule.13:47
f13and yet they have for a very large set of users13:47
wwoodsthat's a development problem particular to this release and your setup13:47
wwoodsso: sorry13:47
poelcatwwoods: your point about people having more bits to test13:47
f13please, lets not get trapped in the "it doesn't work for me, it can't possibly work for anybody else" syndrome13:47
wwoodsI think I just showed that there's more, fresher stuff available for test13:48
nottingok, so who is going to document/announce the PR/RC cycle?13:48
nottingand we can table the 'how do we do F10' until we're less in AAAUGGH mode13:48
f13I suppose it falls to me, since I own and it's a bit vague on the definitions of these releases13:48
f13but I'll welcome anybody to edit that page in my stead.13:49
wwoodsso what's the difference between PR and previous snaps? full test-release QA effort?13:49
f13poelcat: snapshots are just that, snapshots.  If they're broken, that means more things need to be fixed.  waiting until things are perfect to snapshot for testing.... seems odd13:49
f13wwoods: From my point of view, PR has some extra attention put on it from releng and QA and developers13:50
poelcatf13: i'm not advocating that :)13:50
wwoodsI mean, it's getting the test release treatment from QA13:50
f13wwoods: the fact that we'd actually slip PR for things is one difference13:50
wwoodsright. but the release process is going to skip the normal test-release mirroring junk13:51
wwoods'cuz there's not time in the cycle13:51
wwoodsis that the plan?13:51
f13wwoods: we're /going/ to have a PR, I don't think there was any guarentee that there would be a working snapshot every week13:51
f13we'd just /try/ for weekly snapshots13:51
f13wwoods: yes, that sounds right.13:51
f13bittorrent will be our primary release path for PR, as we can get the bits up and public within a few hours (5~?)13:52
f13We've talked about adding mirrors / jigdo to this in parallel13:52
wwoodsI think the plan was for a best effort to make working weekly snapshots, but you're correct: no guarantee13:52
f13and changes to our mirror system may make that more of a reality13:52
wwoodsso the PR is the final snapshot, and it's guaranteed to be as installable/functional as any other test release13:53
f13sounds fair13:53
nottingwell, if you don't want mirrored jigdo, you can always point the jigdo files at koji *runs away*13:54
f13well, we're coming up on one hour here, and I'm no closer to fixing the rawhide snafu and the sudden inability to sign any packages.13:54
f13I'll commit to documenting this somewhere, but it won't happen today, and I'd really rather somebody else do it, I can provide input/corrections as needed13:55
poelcatf13: i'll take a shot at13:56
f13thanks a ton!13:56
nottingis there a good way to post a draft wiki edit so other people can see it?13:56
poelcatwhat is the exsiting page to update?13:56
f13poelcat: is the likely place for it, as it can be referenced by other things13:56
f13notting: other than coping the source into a new page, I don't think so13:56
poelcatf13: okay; i'll commit to something by mid-week13:56
GH-VAIOweb address will be
f13poelcat: cheers13:57
GH-VAIOhello, antbody here wanna trade shell accout?13:57
f13GH-VAIO: please go away, extremely off topic for this channel.13:57
f13ok, anything else for this week?13:59

Generated by 2.3 by Marius Gedminas - find it at!