ReleaseEngineering/Meetings/2008-apr-14

Fedora Release Engineering Meeting :: Monday 2008-04-14

Preview Release (aka PR) Status

• Not ready--scheduled for availability on 2008-04-11
• Problems with yum and other compose tools
• Snafu with package signing which caused rawhide to revert back to all unsigned copies of most packages--100G worth of churn
• Spend rest of today untangling rawhide situation
• Attempt Preview Release on 2008-04-15 with no change to GA date of 2008-04-29

Mass Branching

• Toshio says that the tools (pkgdb, etc..) are ready for the mass branching.
• Should probably have a CVS outage to get the branching done--usually takes a full work day
• Discusison left unfinished

What is the Preview Release?

• Just another snapshot or equivalent to Alpha and Beta which requires more formal testing before release?
• Discussion about confusion over what preview release is
• Need clearer definition for what Preview Release and Release Candidate mean
• DECISIONS:

1. Poelcat to take a shot at documenting purpose and criteria for snapshots and test releases 1. Preview Release will be as installable and functional as other test releases (Alpha and Beta)

• http://fedoraproject.org/wiki/QA/TestResults/Fedora9Install/PreviewRelease

IRC Transcript

f13	ping: notting jeremy spot lmacken warren rdieter wwoods poelcat jwb	<a href="#t13:00" class="time">13:00</a>
* notting is here		<a href="#t13:00" class="time">13:00</a>
rdieter	yo	<a href="#t13:00" class="time">13:00</a>
notting	although still drowning in mail	<a href="#t13:00" class="time">13:00</a>
skvidal	notting: I've got a filter that helps with that - search for @ in the sender's address and delete the mail if it is found	<a href="#t13:01" class="time">13:01</a>
wwoods	ooh, was about to reboot for preupgrade testing	<a href="#t13:03" class="time">13:03</a>
* wwoods stays his hand for a bit		<a href="#t13:03" class="time">13:03</a>
f13	alright, lets kick this pig.	<a href="#t13:05" class="time">13:05</a>
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Preview Release		<a href="#t13:05" class="time">13:05</a>
f13	I"d really like to get PR out the door, but the universe hates me.	<a href="#t13:05" class="time">13:05</a>
f13	I'm convinced.	<a href="#t13:05" class="time">13:05</a>
f13	After all the battling with yum and other compose tool fun, I was all set to get PR composed and out the door today.	<a href="#t13:05" class="time">13:05</a>
* jeremy is sort of here		<a href="#t13:05" class="time">13:05</a>
f13	Unfortunately, that's not to be.	<a href="#t13:05" class="time">13:05</a>
f13	Due to a snafu with package signing, we caused rawhide to revert back to all unsigned copies of most packages, some 100G worth of churn	<a href="#t13:06" class="time">13:06</a>
f13	probably not the best to compose PR against.	<a href="#t13:06" class="time">13:06</a>
wwoods	ack!	<a href="#t13:06" class="time">13:06</a>
* poelcat here		<a href="#t13:06" class="time">13:06</a>
jeremy	f13: doh.  I was wondering why I saw so much churn in my sync but thought it was going to signed not the unsigned packages :(	<a href="#t13:06" class="time">13:06</a>
f13	I'm in the middle of trying to recover the situation, and we've turned off snapmirror tos that PHX/TPA don't get more of the bad bits	<a href="#t13:06" class="time">13:06</a>
f13	jeremy: sadly, mash has a "if I don't find the preferred signed copy, I fall back to unsigned" method that I wasn't fully aware of.	<a href="#t13:07" class="time">13:07</a>
* rdieter was wondering why his rawhide mirror sync was taking so long to finish today, rats.		<a href="#t13:07" class="time">13:07</a>
notting	f13: well, the alternative is to not fall back and fail	<a href="#t13:07" class="time">13:07</a>
f13	so I'll likely spend the entire day cleaning up from that, and investigating why sign_unsigned suddenly is getting md5 mismatches on the things it signs.	<a href="#t13:08" class="time">13:08</a>
notting	which also won't help you	<a href="#t13:08" class="time">13:08</a>
f13	notting: or stop switching gpg sigs needlessly	<a href="#t13:08" class="time">13:08</a>
f13	at any rate, I may end up composing a PR against a recovered snapshot from /yesterday's/ rawhide	<a href="#t13:09" class="time">13:09</a>
f13	if it looks like recovering todays rawhide bits is going to take too long	<a href="#t13:09" class="time">13:09</a>
f13	but I haven't looked at what all got signed last night that would be good to have in the PR	<a href="#t13:10" class="time">13:10</a>
f13	Looking beyond PR...	<a href="#t13:11" class="time">13:11</a>
f13	I think we'll still be taking in tag requests until early next week	<a href="#t13:11" class="time">13:11</a>
f13	then we need to crack down and start considering release candidates, and what are actual release blockers	<a href="#t13:11" class="time">13:11</a>
f13	anything more on Preview Release?	<a href="#t13:13" class="time">13:13</a>
notting	do we know of any PR blockers aside from the 'getting it composed issues'?	<a href="#t13:14" class="time">13:14</a>
f13	jeremy said there was one bug left on the PR blocker list	<a href="#t13:14" class="time">13:14</a>
jeremy	the x keyboard config one	<a href="#t13:14" class="time">13:14</a>
f13	but that's the only thing I'm aware of, I thought we were in pretty damn good shape for PR	<a href="#t13:15" class="time">13:15</a>
jeremy	and I really really really don't know if I feel comfortable with the RC being the first thing that has a fix for that	<a href="#t13:15" class="time">13:15</a>
jeremy	I know ajax was working on it on friday, but I didn't quite catch what he was saying the problem was on the bus (since I was in the python unicode discussion instead ;-)	<a href="#t13:15" class="time">13:15</a>
jeremy	and I've been in class all morning	<a href="#t13:15" class="time">13:15</a>
notting	what's wrong with keyboards today?	<a href="#t13:16" class="time">13:16</a>
f13	if you set a layout at install, it doesn't follow through to X once installed I think	<a href="#t13:17" class="time">13:17</a>
notting	oh, because we're only writing to sysconfig, not to X	<a href="#t13:19" class="time">13:19</a>
notting	unless you use s-c-keyboard, which writes to both	<a href="#t13:19" class="time">13:19</a>
f13	I think so?  I'm not clear on the problem space	<a href="#t13:19" class="time">13:19</a>
f13	so for the keyboard thing, we may be putning the PR to tomorrow anyway	<a href="#t13:21" class="time">13:21</a>
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Mass Branching		<a href="#t13:22" class="time">13:22</a>
rdieter	punt++ , will the package signing be resolved by then?	<a href="#t13:22" class="time">13:22</a>
wwoods	since we're now a week from RC I don't think we should slip the PR any further for fixes	<a href="#t13:22" class="time">13:22</a>
f13	Toshio says that the tools (pkgdb, etc..) are ready for the mass branching.	<a href="#t13:23" class="time">13:23</a>
f13	unfortunately unless we want to mailbomb everybody, we should probably have a CVS outage to get the branching done, and that usually takes a full work day	<a href="#t13:23" class="time">13:23</a>
f13	and I have 0 time to focus on it right now	<a href="#t13:23" class="time">13:23</a>
f13	hrm,	<a href="#t13:23" class="time">13:23</a>
f13	lets put CVS branchingon hold for a moment.	<a href="#t13:23" class="time">13:23</a>
f13	wwoods: You'd be OK with PR going out with the current blocker unfixed?	<a href="#t13:24" class="time">13:24</a>
f13	(if so, mind moving that blocker off of the PR blocker list)	<a href="#t13:24" class="time">13:24</a>
wwoods	lemme recheck the blocker list, but seriously if we're releasing another snapshot in a week *and* we have a fix approaching the proverbial runway	<a href="#t13:25" class="time">13:25</a>
wwoods	then, yeah, get the dang thing into people's hands and release note it	<a href="#t13:25" class="time">13:25</a>
wwoods	"we know, it'll be fixed next week, just ssshh"	<a href="#t13:25" class="time">13:25</a>
f13	wwoods: I don't think there is really time in the schedule to "release" RCs	<a href="#t13:25" class="time">13:25</a>
f13	wwoods: so we'd be relying on rawhide users to confirm if it's fixed or not	<a href="#t13:26" class="time">13:26</a>
wwoods	wait, what? I must have misunderstood something	<a href="#t13:26" class="time">13:26</a>
wwoods	22 April 2008	<a href="#t13:26" class="time">13:26</a>
wwoods	Release Candidate 1	<a href="#t13:26" class="time">13:26</a>
f13	RCs are seriously going to be compose as fast as you can, multiple times a day, and drop it on the mirrors ASAP	<a href="#t13:26" class="time">13:26</a>
wwoods	There's no schedule for *subsequent* RCs - those are at our option	<a href="#t13:26" class="time">13:26</a>
wwoods	hm, okay	<a href="#t13:27" class="time">13:27</a>
wwoods	so we're not planning on putting RC1 on the mirrors?	<a href="#t13:27" class="time">13:27</a>
f13	I have to have something going to the mirrors by the 24th really for a successful 29th release	<a href="#t13:27" class="time">13:27</a>
f13	wwoods: there is not enough time.	<a href="#t13:27" class="time">13:27</a>
wwoods	forgive me if I'm getting confused, I think I'm coming down with something	<a href="#t13:27" class="time">13:27</a>
f13	mirrors == 3~ days of sync lead time	<a href="#t13:27" class="time">13:27</a>
f13	often 3 business days	<a href="#t13:28" class="time">13:28</a>
f13	that's for a "minor" release.  For a major release, we want a lot more time to have a lot more mirrors ready for the bit flip	<a href="#t13:28" class="time">13:28</a>
notting	well, i think the issues is just that the schedule says 'Preview Release' and 'Release Candidate 1'	<a href="#t13:28" class="time">13:28</a>
notting	are those defined?	<a href="#t13:28" class="time">13:28</a>
f13	notting: not well.  PR is new for this round, and "RC" is just something we used to call attempts to do the final compose	<a href="#t13:28" class="time">13:28</a>
f13	I don't think there was any promise that "RC" would see the light of day in the public space	<a href="#t13:29" class="time">13:29</a>
notting	no, but we should probably add something somewhere that defines what they are, or we'll be answering this question again	<a href="#t13:29" class="time">13:29</a>
wwoods	Yeah, my memory's hazy but I think f13's right and I've just confused myself	<a href="#t13:30" class="time">13:30</a>
wwoods	so yeah, that needs a more detailed explanation somewhere, I guess	<a href="#t13:30" class="time">13:30</a>
wwoods	and if we're not planning on doing another fullblown worldwide mirrored release until final	<a href="#t13:30" class="time">13:30</a>
wwoods	then yes, I guess we'll need to continue to block on bug 438246	<a href="#t13:31" class="time">13:31</a>
buggbot	Bug <a href="https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=438246">https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=438246</a> low, low, ---, Kristian Høgsberg, ASSIGNED , Keyboard layout selected during install is forgotten at post-install boot	<a href="#t13:31" class="time">13:31</a>
f13	I know we've already got a potential schedule for F10, but yeah, I think we seriously need to discuss what we're trying to accomplish with PR, and RC, etc...	<a href="#t13:32" class="time">13:32</a>
f13	I originally thought that PR was just going to be the regularly scheduled snapshot of the week attempt	<a href="#t13:32" class="time">13:32</a>
f13	just under a new name	<a href="#t13:32" class="time">13:32</a>
f13	not that we'd be holding it up for bugs, or treating it all that special, or mirroring it, just bog standard snapshot on torrent	<a href="#t13:33" class="time">13:33</a>
f13	and that we'd continue to use rawhide for testing things we want to get fixed before we say "Ok, we're good enough, start attempting the release spin" and if the spin is good, as in nothing fell over during compose, that was it, we're shipped	<a href="#t13:33" class="time">13:33</a>
f13	trying to do public releases of candidate trees is going to put a serious hurt on things	<a href="#t13:34" class="time">13:34</a>
f13	anyway.... back to /topic	<a href="#t13:35" class="time">13:35</a>
wwoods	No, we wanted a third fullblown test release before final	<a href="#t13:35" class="time">13:35</a>
f13	if anybody wants to tackle cvs branching, by all means, I'd be very happy	<a href="#t13:35" class="time">13:35</a>
f13	I have some notes from last time, and toshio started an SOP page to outline calls to pkgdb and such	<a href="#t13:36" class="time">13:36</a>
* poelcat wonders how much we reduce our potential pool of testers when we reduce availability to rawhide		<a href="#t13:36" class="time">13:36</a>
notting	wwoods: f13: so, how do we resolve this dichotomy	<a href="#t13:36" class="time">13:36</a>
wwoods	but RCs don't need to be fullblown mirrored releases	<a href="#t13:36" class="time">13:36</a>
f13	by the time we get to RC state, it should seriously be an RC, as in all the bugs are fixed, we're just putting the tested builds into a tree, and making sure the tree compose itself doesn't fall over	<a href="#t13:37" class="time">13:37</a>
wwoods	right	<a href="#t13:38" class="time">13:38</a>
f13	I'm not sure what the value is of trying to put that tree out to the public and waiting for somebody to give it thumbs up	<a href="#t13:38" class="time">13:38</a>
wwoods	that's what the PR is for	<a href="#t13:38" class="time">13:38</a>
poelcat	wwoods: +1	<a href="#t13:39" class="time">13:39</a>
f13	wwoods: so what do we do with the time between cuttin the PR, having it mirrored, getting people to download and test, and report things, and cutting RC?	<a href="#t13:39" class="time">13:39</a>
f13	do we take in other bugfixes people find?	<a href="#t13:39" class="time">13:39</a>
wwoods	yes?	<a href="#t13:39" class="time">13:39</a>
f13	do we just say "nope, sorry, th'at snot a blocker, fix it in an update?"	<a href="#t13:39" class="time">13:39</a>
wwoods	the reason it's called "Preview Release" and not "Public Release Candidate" is that we *knew* there'd be unfixed blockers at this point	<a href="#t13:40" class="time">13:40</a>
f13	and if we're going to take fixes in, what's the criteria for what we wait for to be fixed for PR, vs what we'll allow to be fixed after PR goes out?	<a href="#t13:40" class="time">13:40</a>
wwoods	so calling it an RC would have just been an obvious lie	<a href="#t13:40" class="time">13:40</a>
wwoods	the criteria for PR are the same as for Alpha and Beta	<a href="#t13:40" class="time">13:40</a>
f13	uh..	<a href="#t13:40" class="time">13:40</a>
wwoods	seriously, (alpha, beta, preview_release) = (test1, test2, test3)	<a href="#t13:41" class="time">13:41</a>
f13	if that were the case, we wouldn't be blocking on the X thing	<a href="#t13:41" class="time">13:41</a>
wwoods	we changed the names and shifted dates but that was the original plan	<a href="#t13:41" class="time">13:41</a>
wwoods	is that right? yeah I guess so	<a href="#t13:41" class="time">13:41</a>
f13	wwoods: I think that really fell apart when we scheduled the PR for 3 days after the final freeze	<a href="#t13:42" class="time">13:42</a>
wwoods	I'm not sure what criteria made the X thing a blocker in the first place, actually	<a href="#t13:42" class="time">13:42</a>
f13	and didn't give it any compose/stage/sync time	<a href="#t13:42" class="time">13:42</a>
f13	I was under the (mistaken?) impression that PR was just going to be a weekly snapshot, tossed onto torrent	<a href="#t13:43" class="time">13:43</a>
wwoods	no, PR was supposed to map to test3	<a href="#t13:43" class="time">13:43</a>
f13	and would service as a post-freeze media set to be the basis of further testing	<a href="#t13:43" class="time">13:43</a>
wwoods	*that* part is true	<a href="#t13:43" class="time">13:43</a>
f13	ok, our signals were quite crossed then	<a href="#t13:43" class="time">13:43</a>
f13	the schedule certainly doesn't have room for a full blown test3 like release	<a href="#t13:44" class="time">13:44</a>
wwoods	okay then	<a href="#t13:44" class="time">13:44</a>
f13	I thought we obviated the need for a 3rd major developmental milestone	<a href="#t13:44" class="time">13:44</a>
wwoods	let's go with your plan	<a href="#t13:44" class="time">13:44</a>
wwoods	because it reduces our workload	<a href="#t13:44" class="time">13:44</a>
f13	'cause it just make things take too long	<a href="#t13:44" class="time">13:44</a>
wwoods	and if anyone actually flips out	<a href="#t13:44" class="time">13:44</a>
f13	we'll have me to blame (:	<a href="#t13:44" class="time">13:44</a>
wwoods	or if it actually has an adverse effect	<a href="#t13:44" class="time">13:44</a>
notting	so, 1) do we need to document this 2) do we need to adjust the f10 schedule to have PR be a 'release'?	<a href="#t13:45" class="time">13:45</a>
wwoods	then we'll rejigger the F10 schedule around the assumption that PR == test3	<a href="#t13:45" class="time">13:45</a>
wwoods	1) yes, 2) no, unless this goes horribly awry	<a href="#t13:45" class="time">13:45</a>
wwoods	since we're doing (mostly) weekly snapshot releases, we're putting a lot more testable bits in people's hands	<a href="#t13:46" class="time">13:46</a>
wwoods	than just test1/test2/test3	<a href="#t13:46" class="time">13:46</a>
wwoods	Alpha/Beta/snapshot,snapshot,snapshot,snapshot...pr,rc,rc,rc,rc..	<a href="#t13:46" class="time">13:46</a>
wwoods	obviously that's More, Fresher Bits	<a href="#t13:47" class="time">13:47</a>
wwoods	practially farm-fresh	<a href="#t13:47" class="time">13:47</a>
poelcat	wwoods: except that the snaps have to install... they haven't for me	<a href="#t13:47" class="time">13:47</a>
wwoods	that has nothing to do with the schedule.	<a href="#t13:47" class="time">13:47</a>
f13	and yet they have for a very large set of users	<a href="#t13:47" class="time">13:47</a>
wwoods	that's a development problem particular to this release and your setup	<a href="#t13:47" class="time">13:47</a>
wwoods	so: sorry	<a href="#t13:47" class="time">13:47</a>
poelcat	wwoods: your point about people having more bits to test	<a href="#t13:47" class="time">13:47</a>
f13	please, lets not get trapped in the "it doesn't work for me, it can't possibly work for anybody else" syndrome	<a href="#t13:47" class="time">13:47</a>
wwoods	I think I just showed that there's more, fresher stuff available for test	<a href="#t13:48" class="time">13:48</a>
notting	ok, so who is going to document/announce the PR/RC cycle?	<a href="#t13:48" class="time">13:48</a>
notting	and we can table the 'how do we do F10' until we're less in AAAUGGH mode	<a href="#t13:48" class="time">13:48</a>
f13	I suppose it falls to me, since I own <a href="http://fedoraproject.org/wiki/ReleaseEngineering/Overview">http://fedoraproject.org/wiki/ReleaseEngineering/Overview</a> and it's a bit vague on the definitions of these releases	<a href="#t13:48" class="time">13:48</a>
f13	but I'll welcome anybody to edit that page in my stead.	<a href="#t13:49" class="time">13:49</a>
wwoods	so what's the difference between PR and previous snaps? full test-release QA effort?	<a href="#t13:49" class="time">13:49</a>
f13	poelcat: snapshots are just that, snapshots.  If they're broken, that means more things need to be fixed.  waiting until things are perfect to snapshot for testing.... seems odd	<a href="#t13:49" class="time">13:49</a>
f13	wwoods: From my point of view, PR has some extra attention put on it from releng and QA and developers	<a href="#t13:50" class="time">13:50</a>
wwoods	right	<a href="#t13:50" class="time">13:50</a>
poelcat	f13: i'm not advocating that :)	<a href="#t13:50" class="time">13:50</a>
wwoods	I mean, it's getting the test release treatment from QA	<a href="#t13:50" class="time">13:50</a>
f13	wwoods: the fact that we'd actually slip PR for things is one difference	<a href="#t13:50" class="time">13:50</a>
wwoods	right. but the release process is going to skip the normal test-release mirroring junk	<a href="#t13:51" class="time">13:51</a>
wwoods	'cuz there's not time in the cycle	<a href="#t13:51" class="time">13:51</a>
wwoods	is that the plan?	<a href="#t13:51" class="time">13:51</a>
f13	wwoods: we're /going/ to have a PR, I don't think there was any guarentee that there would be a working snapshot every week	<a href="#t13:51" class="time">13:51</a>
f13	we'd just /try/ for weekly snapshots	<a href="#t13:51" class="time">13:51</a>
f13	wwoods: yes, that sounds right.	<a href="#t13:51" class="time">13:51</a>
f13	bittorrent will be our primary release path for PR, as we can get the bits up and public within a few hours (5~?)	<a href="#t13:52" class="time">13:52</a>
f13	We've talked about adding mirrors / jigdo to this in parallel	<a href="#t13:52" class="time">13:52</a>
wwoods	I think the plan was for a best effort to make working weekly snapshots, but you're correct: no guarantee	<a href="#t13:52" class="time">13:52</a>
f13	and changes to our mirror system may make that more of a reality	<a href="#t13:52" class="time">13:52</a>
wwoods	so the PR is the final snapshot, and it's guaranteed to be as installable/functional as any other test release	<a href="#t13:53" class="time">13:53</a>
f13	sounds fair	<a href="#t13:53" class="time">13:53</a>
notting	well, if you don't want mirrored jigdo, you can always point the jigdo files at koji *runs away*	<a href="#t13:54" class="time">13:54</a>
f13	well, we're coming up on one hour here, and I'm no closer to fixing the rawhide snafu and the sudden inability to sign any packages.	<a href="#t13:54" class="time">13:54</a>
f13	I'll commit to documenting this somewhere, but it won't happen today, and I'd really rather somebody else do it, I can provide input/corrections as needed	<a href="#t13:55" class="time">13:55</a>
poelcat	f13: i'll take a shot at	<a href="#t13:56" class="time">13:56</a>
poelcat	it	<a href="#t13:56" class="time">13:56</a>
f13	thanks a ton!	<a href="#t13:56" class="time">13:56</a>
notting	is there a good way to post a draft wiki edit so other people can see it?	<a href="#t13:56" class="time">13:56</a>
poelcat	what is the exsiting page to update?	<a href="#t13:56" class="time">13:56</a>
f13	poelcat: <a href="http://fedoraproject.org/wiki/ReleaseEngineering/Overview">http://fedoraproject.org/wiki/ReleaseEngineering/Overview</a> is the likely place for it, as it can be referenced by other things	<a href="#t13:56" class="time">13:56</a>
f13	notting: other than coping the source into a new page, I don't think so	<a href="#t13:56" class="time">13:56</a>
poelcat	f13: okay; i'll commit to something by mid-week	<a href="#t13:56" class="time">13:56</a>
GH-VAIO	web address will be <a href="https://cripperz.org/~username/">https://cripperz.org/~username/</a>	<a href="#t13:57" class="time">13:57</a>
GH-VAIO	ops	<a href="#t13:57" class="time">13:57</a>
f13	poelcat: cheers	<a href="#t13:57" class="time">13:57</a>
GH-VAIO	hello, antbody here wanna trade shell accout?	<a href="#t13:57" class="time">13:57</a>
f13	GH-VAIO: please go away, extremely off topic for this channel.	<a href="#t13:57" class="time">13:57</a>
f13	ok, anything else for this week?	<a href="#t13:59" class="time">13:59</a>

Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!