ReleaseEngineering/Meetings/2008-apr-14

From FedoraProject

Jump to: navigation, search

Contents

Fedora Release Engineering Meeting :: Monday 2008-04-14

Preview Release (aka PR) Status

  • Not ready--scheduled for availability on 2008-04-11
  • Problems with yum and other compose tools
  • Snafu with package signing which caused rawhide to revert back to all unsigned copies of most packages--100G worth of churn
  • Spend rest of today untangling rawhide situation
  • Attempt Preview Release on 2008-04-15 with no change to GA date of 2008-04-29

Mass Branching

  • Toshio says that the tools (pkgdb, etc..) are ready for the mass branching.
  • Should probably have a CVS outage to get the branching done--usually takes a full work day
  • Discusison left unfinished

What is the Preview Release?

  • Just another snapshot or equivalent to Alpha and Beta which requires more formal testing before release?
  • Discussion about confusion over what preview release is
  • Need clearer definition for what Preview Release and Release Candidate mean
  • DECISIONS:

1. Poelcat to take a shot at documenting purpose and criteria for snapshots and test releases 1. Preview Release will be as installable and functional as other test releases (Alpha and Beta)

IRC Transcript

f13ping: notting jeremy spot lmacken warren rdieter wwoods poelcat jwb<a href="#t13:00" class="time">13:00</a>
* notting is here<a href="#t13:00" class="time">13:00</a>
rdieteryo<a href="#t13:00" class="time">13:00</a>
nottingalthough still drowning in mail<a href="#t13:00" class="time">13:00</a>
skvidalnotting: I've got a filter that helps with that - search for @ in the sender's address and delete the mail if it is found<a href="#t13:01" class="time">13:01</a>
wwoodsooh, was about to reboot for preupgrade testing<a href="#t13:03" class="time">13:03</a>
* wwoods stays his hand for a bit<a href="#t13:03" class="time">13:03</a>
f13alright, lets kick this pig.<a href="#t13:05" class="time">13:05</a>
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Preview Release<a href="#t13:05" class="time">13:05</a>
f13I"d really like to get PR out the door, but the universe hates me.<a href="#t13:05" class="time">13:05</a>
f13I'm convinced.<a href="#t13:05" class="time">13:05</a>
f13After all the battling with yum and other compose tool fun, I was all set to get PR composed and out the door today.<a href="#t13:05" class="time">13:05</a>
* jeremy is sort of here<a href="#t13:05" class="time">13:05</a>
f13Unfortunately, that's not to be.<a href="#t13:05" class="time">13:05</a>
f13Due to a snafu with package signing, we caused rawhide to revert back to all unsigned copies of most packages, some 100G worth of churn<a href="#t13:06" class="time">13:06</a>
f13probably not the best to compose PR against.<a href="#t13:06" class="time">13:06</a>
wwoodsack!<a href="#t13:06" class="time">13:06</a>
* poelcat here<a href="#t13:06" class="time">13:06</a>
jeremyf13: doh.  I was wondering why I saw so much churn in my sync but thought it was going to signed not the unsigned packages :(<a href="#t13:06" class="time">13:06</a>
f13I'm in the middle of trying to recover the situation, and we've turned off snapmirror tos that PHX/TPA don't get more of the bad bits<a href="#t13:06" class="time">13:06</a>
f13jeremy: sadly, mash has a "if I don't find the preferred signed copy, I fall back to unsigned" method that I wasn't fully aware of.<a href="#t13:07" class="time">13:07</a>
* rdieter was wondering why his rawhide mirror sync was taking so long to finish today, rats.<a href="#t13:07" class="time">13:07</a>
nottingf13: well, the alternative is to not fall back and fail<a href="#t13:07" class="time">13:07</a>
f13so I'll likely spend the entire day cleaning up from that, and investigating why sign_unsigned suddenly is getting md5 mismatches on the things it signs.<a href="#t13:08" class="time">13:08</a>
nottingwhich also won't help you<a href="#t13:08" class="time">13:08</a>
f13notting: or stop switching gpg sigs needlessly<a href="#t13:08" class="time">13:08</a>
f13at any rate, I may end up composing a PR against a recovered snapshot from /yesterday's/ rawhide<a href="#t13:09" class="time">13:09</a>
f13if it looks like recovering todays rawhide bits is going to take too long<a href="#t13:09" class="time">13:09</a>
f13but I haven't looked at what all got signed last night that would be good to have in the PR<a href="#t13:10" class="time">13:10</a>
f13Looking beyond PR...<a href="#t13:11" class="time">13:11</a>
f13I think we'll still be taking in tag requests until early next week<a href="#t13:11" class="time">13:11</a>
f13then we need to crack down and start considering release candidates, and what are actual release blockers<a href="#t13:11" class="time">13:11</a>
f13anything more on Preview Release?<a href="#t13:13" class="time">13:13</a>
nottingdo we know of any PR blockers aside from the 'getting it composed issues'?<a href="#t13:14" class="time">13:14</a>
f13jeremy said there was one bug left on the PR blocker list<a href="#t13:14" class="time">13:14</a>
jeremythe x keyboard config one<a href="#t13:14" class="time">13:14</a>
f13but that's the only thing I'm aware of, I thought we were in pretty damn good shape for PR<a href="#t13:15" class="time">13:15</a>
jeremyand I really really really don't know if I feel comfortable with the RC being the first thing that has a fix for that<a href="#t13:15" class="time">13:15</a>
jeremyI know ajax was working on it on friday, but I didn't quite catch what he was saying the problem was on the bus (since I was in the python unicode discussion instead ;-)<a href="#t13:15" class="time">13:15</a>
jeremyand I've been in class all morning<a href="#t13:15" class="time">13:15</a>
nottingwhat's wrong with keyboards today?<a href="#t13:16" class="time">13:16</a>
f13if you set a layout at install, it doesn't follow through to X once installed I think<a href="#t13:17" class="time">13:17</a>
nottingoh, because we're only writing to sysconfig, not to X<a href="#t13:19" class="time">13:19</a>
nottingunless you use s-c-keyboard, which writes to both<a href="#t13:19" class="time">13:19</a>
f13I think so?  I'm not clear on the problem space<a href="#t13:19" class="time">13:19</a>
f13so for the keyboard thing, we may be putning the PR to tomorrow anyway<a href="#t13:21" class="time">13:21</a>
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Mass Branching<a href="#t13:22" class="time">13:22</a>
rdieterpunt++ , will the package signing be resolved by then?<a href="#t13:22" class="time">13:22</a>
wwoodssince we're now a week from RC I don't think we should slip the PR any further for fixes<a href="#t13:22" class="time">13:22</a>
f13Toshio says that the tools (pkgdb, etc..) are ready for the mass branching.<a href="#t13:23" class="time">13:23</a>
f13unfortunately unless we want to mailbomb everybody, we should probably have a CVS outage to get the branching done, and that usually takes a full work day<a href="#t13:23" class="time">13:23</a>
f13and I have 0 time to focus on it right now<a href="#t13:23" class="time">13:23</a>
f13hrm,<a href="#t13:23" class="time">13:23</a>
f13lets put CVS branchingon hold for a moment.<a href="#t13:23" class="time">13:23</a>
f13wwoods: You'd be OK with PR going out with the current blocker unfixed?<a href="#t13:24" class="time">13:24</a>
f13(if so, mind moving that blocker off of the PR blocker list)<a href="#t13:24" class="time">13:24</a>
wwoodslemme recheck the blocker list, but seriously if we're releasing another snapshot in a week *and* we have a fix approaching the proverbial runway<a href="#t13:25" class="time">13:25</a>
wwoodsthen, yeah, get the dang thing into people's hands and release note it<a href="#t13:25" class="time">13:25</a>
wwoods"we know, it'll be fixed next week, just ssshh"<a href="#t13:25" class="time">13:25</a>
f13wwoods: I don't think there is really time in the schedule to "release" RCs<a href="#t13:25" class="time">13:25</a>
f13wwoods: so we'd be relying on rawhide users to confirm if it's fixed or not<a href="#t13:26" class="time">13:26</a>
wwoodswait, what? I must have misunderstood something<a href="#t13:26" class="time">13:26</a>
wwoods22 April 2008<a href="#t13:26" class="time">13:26</a>
wwoodsRelease Candidate 1<a href="#t13:26" class="time">13:26</a>
f13RCs are seriously going to be compose as fast as you can, multiple times a day, and drop it on the mirrors ASAP<a href="#t13:26" class="time">13:26</a>
wwoodsThere's no schedule for *subsequent* RCs - those are at our option<a href="#t13:26" class="time">13:26</a>
wwoodshm, okay<a href="#t13:27" class="time">13:27</a>
wwoodsso we're not planning on putting RC1 on the mirrors?<a href="#t13:27" class="time">13:27</a>
f13I have to have something going to the mirrors by the 24th really for a successful 29th release<a href="#t13:27" class="time">13:27</a>
f13wwoods: there is not enough time.<a href="#t13:27" class="time">13:27</a>
wwoodsforgive me if I'm getting confused, I think I'm coming down with something<a href="#t13:27" class="time">13:27</a>
f13mirrors == 3~ days of sync lead time<a href="#t13:27" class="time">13:27</a>
f13often 3 business days<a href="#t13:28" class="time">13:28</a>
f13that's for a "minor" release.  For a major release, we want a lot more time to have a lot more mirrors ready for the bit flip<a href="#t13:28" class="time">13:28</a>
nottingwell, i think the issues is just that the schedule says 'Preview Release' and 'Release Candidate 1'<a href="#t13:28" class="time">13:28</a>
nottingare those defined?<a href="#t13:28" class="time">13:28</a>
f13notting: not well.  PR is new for this round, and "RC" is just something we used to call attempts to do the final compose<a href="#t13:28" class="time">13:28</a>
f13I don't think there was any promise that "RC" would see the light of day in the public space<a href="#t13:29" class="time">13:29</a>
nottingno, but we should probably add something somewhere that defines what they are, or we'll be answering this question again<a href="#t13:29" class="time">13:29</a>
wwoodsYeah, my memory's hazy but I think f13's right and I've just confused myself<a href="#t13:30" class="time">13:30</a>
wwoodsso yeah, that needs a more detailed explanation somewhere, I guess<a href="#t13:30" class="time">13:30</a>
wwoodsand if we're not planning on doing another fullblown worldwide mirrored release until final<a href="#t13:30" class="time">13:30</a>
wwoodsthen yes, I guess we'll need to continue to block on bug 438246<a href="#t13:31" class="time">13:31</a>
buggbotBug <a href="https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=438246">https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=438246</a> low, low, ---, Kristian Høgsberg, ASSIGNED , Keyboard layout selected during install is forgotten at post-install boot<a href="#t13:31" class="time">13:31</a>
f13I know we've already got a potential schedule for F10, but yeah, I think we seriously need to discuss what we're trying to accomplish with PR, and RC, etc...<a href="#t13:32" class="time">13:32</a>
f13I originally thought that PR was just going to be the regularly scheduled snapshot of the week attempt<a href="#t13:32" class="time">13:32</a>
f13just under a new name<a href="#t13:32" class="time">13:32</a>
f13not that we'd be holding it up for bugs, or treating it all that special, or mirroring it, just bog standard snapshot on torrent<a href="#t13:33" class="time">13:33</a>
f13and that we'd continue to use rawhide for testing things we want to get fixed before we say "Ok, we're good enough, start attempting the release spin" and if the spin is good, as in nothing fell over during compose, that was it, we're shipped<a href="#t13:33" class="time">13:33</a>
f13trying to do public releases of candidate trees is going to put a serious hurt on things<a href="#t13:34" class="time">13:34</a>
f13anyway.... back to /topic<a href="#t13:35" class="time">13:35</a>
wwoodsNo, we wanted a third fullblown test release before final<a href="#t13:35" class="time">13:35</a>
f13if anybody wants to tackle cvs branching, by all means, I'd be very happy<a href="#t13:35" class="time">13:35</a>
f13I have some notes from last time, and toshio started an SOP page to outline calls to pkgdb and such<a href="#t13:36" class="time">13:36</a>
* poelcat wonders how much we reduce our potential pool of testers when we reduce availability to rawhide<a href="#t13:36" class="time">13:36</a>
nottingwwoods: f13: so, how do we resolve this dichotomy<a href="#t13:36" class="time">13:36</a>
wwoodsbut RCs don't need to be fullblown mirrored releases<a href="#t13:36" class="time">13:36</a>
f13by the time we get to RC state, it should seriously be an RC, as in all the bugs are fixed, we're just putting the tested builds into a tree, and making sure the tree compose itself doesn't fall over<a href="#t13:37" class="time">13:37</a>
wwoodsright<a href="#t13:38" class="time">13:38</a>
f13I'm not sure what the value is of trying to put that tree out to the public and waiting for somebody to give it thumbs up<a href="#t13:38" class="time">13:38</a>
wwoodsthat's what the PR is for<a href="#t13:38" class="time">13:38</a>
poelcatwwoods: +1<a href="#t13:39" class="time">13:39</a>
f13wwoods: so what do we do with the time between cuttin the PR, having it mirrored, getting people to download and test, and report things, and cutting RC?<a href="#t13:39" class="time">13:39</a>
f13do we take in other bugfixes people find?<a href="#t13:39" class="time">13:39</a>
wwoodsyes?<a href="#t13:39" class="time">13:39</a>
f13do we just say "nope, sorry, th'at snot a blocker, fix it in an update?"<a href="#t13:39" class="time">13:39</a>
wwoodsthe reason it's called "Preview Release" and not "Public Release Candidate" is that we *knew* there'd be unfixed blockers at this point<a href="#t13:40" class="time">13:40</a>
f13and if we're going to take fixes in, what's the criteria for what we wait for to be fixed for PR, vs what we'll allow to be fixed after PR goes out?<a href="#t13:40" class="time">13:40</a>
wwoodsso calling it an RC would have just been an obvious lie<a href="#t13:40" class="time">13:40</a>
wwoodsthe criteria for PR are the same as for Alpha and Beta<a href="#t13:40" class="time">13:40</a>
f13uh..<a href="#t13:40" class="time">13:40</a>
wwoodsseriously, (alpha, beta, preview_release) = (test1, test2, test3)<a href="#t13:41" class="time">13:41</a>
f13if that were the case, we wouldn't be blocking on the X thing<a href="#t13:41" class="time">13:41</a>
wwoodswe changed the names and shifted dates but that was the original plan<a href="#t13:41" class="time">13:41</a>
wwoodsis that right? yeah I guess so<a href="#t13:41" class="time">13:41</a>
f13wwoods: I think that really fell apart when we scheduled the PR for 3 days after the final freeze<a href="#t13:42" class="time">13:42</a>
wwoodsI'm not sure what criteria made the X thing a blocker in the first place, actually<a href="#t13:42" class="time">13:42</a>
f13and didn't give it any compose/stage/sync time<a href="#t13:42" class="time">13:42</a>
f13I was under the (mistaken?) impression that PR was just going to be a weekly snapshot, tossed onto torrent<a href="#t13:43" class="time">13:43</a>
wwoodsno, PR was supposed to map to test3<a href="#t13:43" class="time">13:43</a>
f13and would service as a post-freeze media set to be the basis of further testing<a href="#t13:43" class="time">13:43</a>
wwoods*that* part is true<a href="#t13:43" class="time">13:43</a>
f13ok, our signals were quite crossed then<a href="#t13:43" class="time">13:43</a>
f13the schedule certainly doesn't have room for a full blown test3 like release<a href="#t13:44" class="time">13:44</a>
wwoodsokay then<a href="#t13:44" class="time">13:44</a>
f13I thought we obviated the need for a 3rd major developmental milestone<a href="#t13:44" class="time">13:44</a>
wwoodslet's go with your plan<a href="#t13:44" class="time">13:44</a>
wwoodsbecause it reduces our workload<a href="#t13:44" class="time">13:44</a>
f13'cause it just make things take too long<a href="#t13:44" class="time">13:44</a>
wwoodsand if anyone actually flips out<a href="#t13:44" class="time">13:44</a>
f13we'll have me to blame (:<a href="#t13:44" class="time">13:44</a>
wwoodsor if it actually has an adverse effect<a href="#t13:44" class="time">13:44</a>
nottingso, 1) do we need to document this 2) do we need to adjust the f10 schedule to have PR be a 'release'?<a href="#t13:45" class="time">13:45</a>
wwoodsthen we'll rejigger the F10 schedule around the assumption that PR == test3<a href="#t13:45" class="time">13:45</a>
wwoods1) yes, 2) no, unless this goes horribly awry<a href="#t13:45" class="time">13:45</a>
wwoodssince we're doing (mostly) weekly snapshot releases, we're putting a lot more testable bits in people's hands<a href="#t13:46" class="time">13:46</a>
wwoodsthan just test1/test2/test3<a href="#t13:46" class="time">13:46</a>
wwoodsAlpha/Beta/snapshot,snapshot,snapshot,snapshot...pr,rc,rc,rc,rc..<a href="#t13:46" class="time">13:46</a>
wwoodsobviously that's More, Fresher Bits<a href="#t13:47" class="time">13:47</a>
wwoodspractially farm-fresh<a href="#t13:47" class="time">13:47</a>
poelcatwwoods: except that the snaps have to install... they haven't for me<a href="#t13:47" class="time">13:47</a>
wwoodsthat has nothing to do with the schedule.<a href="#t13:47" class="time">13:47</a>
f13and yet they have for a very large set of users<a href="#t13:47" class="time">13:47</a>
wwoodsthat's a development problem particular to this release and your setup<a href="#t13:47" class="time">13:47</a>
wwoodsso: sorry<a href="#t13:47" class="time">13:47</a>
poelcatwwoods: your point about people having more bits to test<a href="#t13:47" class="time">13:47</a>
f13please, lets not get trapped in the "it doesn't work for me, it can't possibly work for anybody else" syndrome<a href="#t13:47" class="time">13:47</a>
wwoodsI think I just showed that there's more, fresher stuff available for test<a href="#t13:48" class="time">13:48</a>
nottingok, so who is going to document/announce the PR/RC cycle?<a href="#t13:48" class="time">13:48</a>
nottingand we can table the 'how do we do F10' until we're less in AAAUGGH mode<a href="#t13:48" class="time">13:48</a>
f13I suppose it falls to me, since I own <a href="http://fedoraproject.org/wiki/ReleaseEngineering/Overview">http://fedoraproject.org/wiki/ReleaseEngineering/Overview</a> and it's a bit vague on the definitions of these releases<a href="#t13:48" class="time">13:48</a>
f13but I'll welcome anybody to edit that page in my stead.<a href="#t13:49" class="time">13:49</a>
wwoodsso what's the difference between PR and previous snaps? full test-release QA effort?<a href="#t13:49" class="time">13:49</a>
f13poelcat: snapshots are just that, snapshots.  If they're broken, that means more things need to be fixed.  waiting until things are perfect to snapshot for testing.... seems odd<a href="#t13:49" class="time">13:49</a>
f13wwoods: From my point of view, PR has some extra attention put on it from releng and QA and developers<a href="#t13:50" class="time">13:50</a>
wwoodsright<a href="#t13:50" class="time">13:50</a>
poelcatf13: i'm not advocating that :)<a href="#t13:50" class="time">13:50</a>
wwoodsI mean, it's getting the test release treatment from QA<a href="#t13:50" class="time">13:50</a>
f13wwoods: the fact that we'd actually slip PR for things is one difference<a href="#t13:50" class="time">13:50</a>
wwoodsright. but the release process is going to skip the normal test-release mirroring junk<a href="#t13:51" class="time">13:51</a>
wwoods'cuz there's not time in the cycle<a href="#t13:51" class="time">13:51</a>
wwoodsis that the plan?<a href="#t13:51" class="time">13:51</a>
f13wwoods: we're /going/ to have a PR, I don't think there was any guarentee that there would be a working snapshot every week<a href="#t13:51" class="time">13:51</a>
f13we'd just /try/ for weekly snapshots<a href="#t13:51" class="time">13:51</a>
f13wwoods: yes, that sounds right.<a href="#t13:51" class="time">13:51</a>
f13bittorrent will be our primary release path for PR, as we can get the bits up and public within a few hours (5~?)<a href="#t13:52" class="time">13:52</a>
f13We've talked about adding mirrors / jigdo to this in parallel<a href="#t13:52" class="time">13:52</a>
wwoodsI think the plan was for a best effort to make working weekly snapshots, but you're correct: no guarantee<a href="#t13:52" class="time">13:52</a>
f13and changes to our mirror system may make that more of a reality<a href="#t13:52" class="time">13:52</a>
wwoodsso the PR is the final snapshot, and it's guaranteed to be as installable/functional as any other test release<a href="#t13:53" class="time">13:53</a>
f13sounds fair<a href="#t13:53" class="time">13:53</a>
nottingwell, if you don't want mirrored jigdo, you can always point the jigdo files at koji *runs away*<a href="#t13:54" class="time">13:54</a>
f13well, we're coming up on one hour here, and I'm no closer to fixing the rawhide snafu and the sudden inability to sign any packages.<a href="#t13:54" class="time">13:54</a>
f13I'll commit to documenting this somewhere, but it won't happen today, and I'd really rather somebody else do it, I can provide input/corrections as needed<a href="#t13:55" class="time">13:55</a>
poelcatf13: i'll take a shot at<a href="#t13:56" class="time">13:56</a>
poelcatit<a href="#t13:56" class="time">13:56</a>
f13thanks a ton!<a href="#t13:56" class="time">13:56</a>
nottingis there a good way to post a draft wiki edit so other people can see it?<a href="#t13:56" class="time">13:56</a>
poelcatwhat is the exsiting page to update?<a href="#t13:56" class="time">13:56</a>
f13poelcat: <a href="http://fedoraproject.org/wiki/ReleaseEngineering/Overview">http://fedoraproject.org/wiki/ReleaseEngineering/Overview</a> is the likely place for it, as it can be referenced by other things<a href="#t13:56" class="time">13:56</a>
f13notting: other than coping the source into a new page, I don't think so<a href="#t13:56" class="time">13:56</a>
poelcatf13: okay; i'll commit to something by mid-week<a href="#t13:56" class="time">13:56</a>
GH-VAIOweb address will be <a href="https://cripperz.org/~username/">https://cripperz.org/~username/</a><a href="#t13:57" class="time">13:57</a>
GH-VAIOops<a href="#t13:57" class="time">13:57</a>
f13poelcat: cheers<a href="#t13:57" class="time">13:57</a>
GH-VAIOhello, antbody here wanna trade shell accout?<a href="#t13:57" class="time">13:57</a>
f13GH-VAIO: please go away, extremely off topic for this channel.<a href="#t13:57" class="time">13:57</a>
f13ok, anything else for this week?<a href="#t13:59" class="time">13:59</a>

Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!