From Fedora Project Wiki

< ReleaseEngineering‎ | Meetings

Revision as of 16:28, 18 February 2014 by Holmja (talk | contribs) (IRC Transcript: Removed some HTML markup.)

Fedora Release Engineering Meeting :: Monday 2008-02-04


Reversing the netapp streams

  • scheduled for today--we're working on pushing it back after the alpha
  • change setup so that the content we generate in PHX can be put on a netapp in PHX
  • instead of rsynced to RDU to be put on a netapp in RDU, and then mirrored by netapp back out to PHX
  • this will result in shorter times between compose and visible to the world for things like rawhide and updates
  • may delay the rawhide/updates for a bit

gcc 4.3

  • gcc4.3 is in the buildroots for dist-f9 now, and some fallout has happened, namely kernel.
  • striving to complete a mass rebuild before beta release


  • Feature page:
  • 300 perl dependent packages as a sample set
  • these packages were the base for which virtually all of the other perl dependent packages depend on
  • out of those 300 about 60 more to build
  • once complete the plan is to merge the new perl and the rebuilt bits into rawhide
  • Fedora is the place where perl-5.10 was really first tested/used and the development ground for getting all the upstreams ported to the new perl

IRC Transcript

-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting<a href="#t13:05" class="time">13:05</a>
f13ping: notting jeremy jwb spot warren rdieter wwoods poelcat lmacken<a href="#t13:06" class="time">13:06</a>
* notting is here<a href="#t13:06" class="time">13:06</a>
* lmacken <a href="#t13:06" class="time">13:06</a>
* jeremy is here<a href="#t13:06" class="time">13:06</a>
* spot is here<a href="#t13:07" class="time">13:07</a>
* poelcat here<a href="#t13:07" class="time">13:07</a>
f13aight.<a href="#t13:08" class="time">13:08</a>
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - F9 Alpha<a href="#t13:08" class="time">13:08</a>
f13so the Alpha is being staged by mirrors.  They've had access to it since Friday<a href="#t13:08" class="time">13:08</a>
f13so hopefully we'll have enough ready to go tomorrow<a href="#t13:08" class="time">13:08</a>
f13I'm in teh process of uploading the isos to the torrent server and creating the torrents.  That should be done this afternoon<a href="#t13:08" class="time">13:08</a>
lmackenf13: F9-Alpha-i686-20080129.0.iso ?<a href="#t13:09" class="time">13:09</a>
f13lmacken: yeah, but renamed.<a href="#t13:09" class="time">13:09</a>
lmackencool.<a href="#t13:09" class="time">13:09</a>
f13we'll hope the bit gets flipped and all will be happy tomorrow.<a href="#t13:10" class="time">13:10</a>
f13anybody have anything else to bring up wrt Alpha?<a href="#t13:10" class="time">13:10</a>
wwoods<a href=""></a> is the test status page<a href="#t13:11" class="time">13:11</a>
wwoodswe may need an updates.img for upgrades<a href="#t13:11" class="time">13:11</a>
f13yeah, it's a bit rough around the edges, but it is an Alpha<a href="#t13:11" class="time">13:11</a>
wwoodsyup<a href="#t13:11" class="time">13:11</a>
wwoodsjust trying to keep ahead of bug reporters to make life easier for triagers/developers<a href="#t13:12" class="time">13:12</a>
f13nod<a href="#t13:12" class="time">13:12</a>
wwoodsoh - I want to pick a week sometime this month<a href="#t13:12" class="time">13:12</a>
wwoodsa couple weeks before the string freeze<a href="#t13:12" class="time">13:12</a>
wwoodsto make double-damn sure we have working boot/install images<a href="#t13:12" class="time">13:12</a>
wwoodsso that we can put out a call for translators to grab the rescuecd (or minimal-installer.iso or whatever we call it)<a href="#t13:13" class="time">13:13</a>
jeremywwoods: *nod*<a href="#t13:13" class="time">13:13</a>
jeremyand I'm probably going to write a script in the next few days which will help make that a bit easier<a href="#t13:13" class="time">13:13</a>
wwoodsprobably that'd be a good time to also make sure we have good Live images so we can have a big internal Laptop Test-fest<a href="#t13:14" class="time">13:14</a>
wwoodsjeremy: sweet<a href="#t13:14" class="time">13:14</a>
wwoodsdo we have ppc images for alpha?<a href="#t13:15" class="time">13:15</a>
f13wwoods: yes<a href="#t13:15" class="time">13:15</a>
f13not live ones though<a href="#t13:15" class="time">13:15</a>
f13sorry<a href="#t13:15" class="time">13:15</a>
wwoods's cool<a href="#t13:16" class="time">13:16</a>
wwoodsI should probably do some ppc testing.. or ask jwb to update the test status page<a href="#t13:17" class="time">13:17</a>
wwoodsf13: if you can give me a pointer to an internal copy of the ppc bits I'll follow up on that after this meeting<a href="#t13:17" class="time">13:17</a>
f13wwoods: it's on reducto<a href="#t13:17" class="time">13:17</a>
f13moving on...<a href="#t13:19" class="time">13:19</a>
-!- f13 changed the topic of #fedora-meeting to: Fedora RelEng: Reversing the netapp streams<a href="#t13:19" class="time">13:19</a>
nottingthis was scheduled for today. we're working on pushing it back after the alpha<a href="#t13:19" class="time">13:19</a>
f13later this week we would like to reverse the netapp streams so that the content we generate in PHX can be put on a netapp in PHX, rather than rsynced to RDU to be put on a netapp in RDU, and then mirrored by netapp back out to PHX<a href="#t13:19" class="time">13:19</a>
f13this will result in shorter times between compose and visible to the world for things like rawhide and updates<a href="#t13:20" class="time">13:20</a>
f13a number of our processes will have to change though<a href="#t13:20" class="time">13:20</a>
f13and ti's possible that rawhide/updates may be delayed for a bit<a href="#t13:21" class="time">13:21</a>
f13as we trial and error our way through it<a href="#t13:21" class="time">13:21</a>
f13notting, lmacken, mmcgrath, and I are the primary folks on this task, so if you have any thoughts/concerns/suggestions hit up one of us.<a href="#t13:22" class="time">13:22</a>
f13anything further on this subject?<a href="#t13:23" class="time">13:23</a>
f13aight.<a href="#t13:23" class="time">13:23</a>
-!- f13 changed the topic of #fedora-meeting to: Fedora RelEng: Gcc43<a href="#t13:23" class="time">13:23</a>
f13gcc4.3 is in the buildroots for dist-f9 now, and some fallout has happened, namely kernel.<a href="#t13:24" class="time">13:24</a>
f13but more importantly is that jakub has indicated that he'd like to see a massrebuild of our packages against the new gcc at some point, after a week or so if it being in the buildroots<a href="#t13:24" class="time">13:24</a>
nottingso... next week?<a href="#t13:25" class="time">13:25</a>
f13March 4th is the Beta freeze<a href="#t13:25" class="time">13:25</a>
f13so we'd likely have to do it next week or the week after<a href="#t13:25" class="time">13:25</a>
f13and we have to decide if htis is going to be maintainer driven, or releng driven.<a href="#t13:25" class="time">13:25</a>
f13IE we ask maintainers to do it and wait, or we just plow through and script massive amounts of builds.<a href="#t13:25" class="time">13:25</a>
nottingautomated seems simpler to me<a href="#t13:26" class="time">13:26</a>
f13you'd think.<a href="#t13:26" class="time">13:26</a>
nottingesp. if there aren't ordering concerns<a href="#t13:26" class="time">13:26</a>
f13trying not to have conflicts is the problem.<a href="#t13:26" class="time">13:26</a>
f13IE automation kicks off a build, maintainer comes along after that and also kicks off a build.<a href="#t13:27" class="time">13:27</a>
f13automated was a backgroundjob, so the maintainer's job gets done first, then the automated one, automated (older) one is now the last tagged, and thus the 'newest'.<a href="#t13:27" class="time">13:27</a>
poelcatf13: wouldn't we want the rebuilt packages to be in the beta?<a href="#t13:27" class="time">13:27</a>
f13poelcat: yeah... that's why we're doing it before the beta freeze<a href="#t13:28" class="time">13:28</a>
poelcatf13: sorry misread above<a href="#t13:28" class="time">13:28</a>
nottingf13: is there a way to not do 'automated/background'? or to have koji check for backwardsness?<a href="#t13:29" class="time">13:29</a>
f13notting: if we don't do background, we essentially tie up the buildsystem for a few days.  no other buidls will get through until our entire set is done.<a href="#t13:30" class="time">13:30</a>
warren_treosorry at doctor<a href="#t13:30" class="time">13:30</a>
f13notting: as for backwardness, I don't thikn there is anything koji itself can do.  We can run a query on the tag after all is said and done to detect nvr issues though.<a href="#t13:30" class="time">13:30</a>
f13the added fun is that spot is going to want to land the new perl relatively soon as well, which is going to cause all kinds of pain<a href="#t13:31" class="time">13:31</a>
warren_treowhat ver of perl?<a href="#t13:31" class="time">13:31</a>
spot5.10.0<a href="#t13:31" class="time">13:31</a>
warren_treowow<a href="#t13:31" class="time">13:31</a>
* spot has been chainbuilding for weeks<a href="#t13:32" class="time">13:32</a>
warren_treospot: is there a URL describing the compat impact of this new perl?<a href="#t13:33" class="time">13:33</a>
* warren_treo wonders if anybody tested spamassassin on perl 5.10<a href="#t13:34" class="time">13:34</a>
nottingf13: so, do we have the scripts ready to start the automated buildatron?<a href="#t13:35" class="time">13:35</a>
f13notting: nope.  I haven't spent any time on that recently<a href="#t13:35" class="time">13:35</a>
warren_treobuild order is known?<a href="#t13:35" class="time">13:35</a>
f13warren_treo: we're ignoring buildorder<a href="#t13:36" class="time">13:36</a>
warren_treooh<a href="#t13:36" class="time">13:36</a>
f13at least for the first pass.<a href="#t13:36" class="time">13:36</a>
f13to avoid cvs conflicts, I imagine the script would go through each needed build (query to fall out of koji for things not yet built with new gcc), do a checkout, do an autobump of hte spec (*shudder*), commit/tag/build --background; and move on.<a href="#t13:37" class="time">13:37</a>
nottingwhen was the last time we did one?<a href="#t13:38" class="time">13:38</a>
f13most of that has been done before<a href="#t13:38" class="time">13:38</a>
f13the autospecbump does run into problems every now and again with malformed spec files<a href="#t13:38" class="time">13:38</a>
f13and catching either cvs errors, or tag errors can be difficult, as well as gathering in the build failures for investigation.<a href="#t13:38" class="time">13:38</a>
warren_treomalformed as in encoding.<a href="#t13:39" class="time">13:39</a>
warren_treo?<a href="#t13:39" class="time">13:39</a>
f13warren_treo: no, as in dirty games with Release: tags<a href="#t13:39" class="time">13:39</a>
f13programatically editing spec files is not easy<a href="#t13:39" class="time">13:39</a>
nottingwas the last rebuild maintainer-driven?<a href="#t13:39" class="time">13:39</a>
warren_treonod<a href="#t13:39" class="time">13:39</a>
f13notting: yeah<a href="#t13:39" class="time">13:39</a>
nottingthe other option is to punt maintainer vs. automated to fesco >:)<a href="#t13:40" class="time">13:40</a>
warren_treobtw did we get past the gcc43 kernel build issue?<a href="#t13:40" class="time">13:40</a>
f13the last one that we did any scripting of was fc6 time frame.  And then some smaller less scripted ones for the last glibc fallout where ppc was boned.<a href="#t13:40" class="time">13:40</a>
f13warren_treo: not yet afaict<a href="#t13:40" class="time">13:40</a>
f13also, last time we let maintainers opt in/out of autobuilding<a href="#t13:41" class="time">13:41</a>
f13usually with a file in their module.<a href="#t13:41" class="time">13:41</a>
f13something we should consider this time.<a href="#t13:41" class="time">13:41</a>
f13basically releng should come up with a proposal on how to accomplish the rebuilds and present it to FESCo<a href="#t13:42" class="time">13:42</a>
warren_treoI hope we can do it without cvs...<a href="#t13:42" class="time">13:42</a>
f13I suppose that will fall on me unless anybody else wants to take a stab at it.<a href="#t13:42" class="time">13:42</a>
warren_treooptin out that is<a href="#t13:42" class="time">13:42</a>
f13warren_treo: have better ideas?<a href="#t13:42" class="time">13:42</a>
f13we need to identify and mark packages that need to be rebuilt<a href="#t13:42" class="time">13:42</a>
f13and then let maintainers scrub them from the list.<a href="#t13:43" class="time">13:43</a>
f13maybe a wiki page?<a href="#t13:43" class="time">13:43</a>
warren_treocould it be added as a flag in pkgdb?  might be quick<a href="#t13:43" class="time">13:43</a>
nottingpkgdb lives forever, though<a href="#t13:43" class="time">13:43</a>
warren_treomm<a href="#t13:43" class="time">13:43</a>
f13yeah, not sure if pkgdb is setup for arbitrary flags like that<a href="#t13:43" class="time">13:43</a>
f13so I'll work up a plan, and ping ya'll as it comes along.  Not enough days in the week.<a href="#t13:45" class="time">13:45</a>
-!- f13 changed the topic of #fedora-meeting to: Fedora RelEng: perl-5.10.0<a href="#t13:45" class="time">13:45</a>
warren_treoprior to gcc43 I heard mention that mass rebuild wouldn't be necessary, what happened?<a href="#t13:45" class="time">13:45</a>
f13warren_treo: it's not /necessary/, but desired by the gcc team.<a href="#t13:46" class="time">13:46</a>
warren_treook<a href="#t13:46" class="time">13:46</a>
f13spot: can you give us a rundown on the perl status?<a href="#t13:46" class="time">13:46</a>
spotsure.<a href="#t13:46" class="time">13:46</a>
spoti identified 300 perl dependent packages as a sample set<a href="#t13:46" class="time">13:46</a>
spotthese packages were the base for which virtually all of the other perl dependent packages depend on<a href="#t13:47" class="time">13:47</a>
spotOut of those 300<a href="#t13:47" class="time">13:47</a>
spoti have about 60 more to build<a href="#t13:47" class="time">13:47</a>
spotonce that is done, the plan is to merge the new perl and the rebuilt bits into rawhide<a href="#t13:48" class="time">13:48</a>
warren_treoperl-5.10 and pkgs built against it are in a separate tag now?<a href="#t13:48" class="time">13:48</a>
spotwarren: yes, dist-f9-perl<a href="#t13:48" class="time">13:48</a>
spotonce merged, there will still be lots of things broken<a href="#t13:49" class="time">13:49</a>
f13spot: are you planning on doing a Feature for perl-5.10, which includes info about when it'll land, and what we expect other perl maintianers to do once it does land?<a href="#t13:50" class="time">13:50</a>
f13(and a rollback plan)<a href="#t13:50" class="time">13:50</a>
spotf13: Yep, just hadn't gotten a round tuit yet.<a href="#t13:50" class="time">13:50</a>
nottingspot: why will there be things broken?<a href="#t13:50" class="time">13:50</a>
warren_treorollback sounds scary when hundreds of pkgs re involved :)<a href="#t13:50" class="time">13:50</a>
f13warren_treo: untag!<a href="#t13:51" class="time">13:51</a>
f13unless you want to force epoch into all of those.... (:<a href="#t13:51" class="time">13:51</a>
spotnotting: because everything hasn't been rebuilt against the new perl yet<a href="#t13:51" class="time">13:51</a>
nottingspot: .... is dist-f9-perl not going to encompass 'everything'?<a href="#t13:51" class="time">13:51</a>
warren_treoeven things like pidgin build against perl<a href="#t13:51" class="time">13:51</a>
nottingor are you only rebuilding pure-perl<a href="#t13:52" class="time">13:52</a>
notting?<a href="#t13:52" class="time">13:52</a>
spotnotting: it would be much faster if the individual maintainers did the rebuilds<a href="#t13:52" class="time">13:52</a>
* warren_treo wonders what will utterly fail<a href="#t13:52" class="time">13:52</a>
spotwarren: so far, only one thing has failed, and I fixed it.<a href="#t13:52" class="time">13:52</a>
f13this could put Fedora in a pretty cool place<a href="#t13:52" class="time">13:52</a>
warren_treospot: what was the cause of that failure?<a href="#t13:52" class="time">13:52</a>
spotthe difficulty is in the ordering<a href="#t13:52" class="time">13:52</a>
f13Fedora was the place where perl-5.10 was really first tested/used and the development ground for getting all the upstreams ported to the new perl.<a href="#t13:53" class="time">13:53</a>
spotwarren: old crufty code making bad assumptions about array behavior<a href="#t13:53" class="time">13:53</a>
warren_treof13: yes assuming the code continues to work<a href="#t13:53" class="time">13:53</a>
warren_treooh<a href="#t13:53" class="time">13:53</a>
nottingf13: 'fedora 9 shipped a new perl. it worked. no one noticed.'?<a href="#t13:53" class="time">13:53</a>
warren_treogotta go now, appt coming up<a href="#t13:54" class="time">13:54</a>
spotnotting: upstream perl certainly is watching this.<a href="#t13:54" class="time">13:54</a>
spotwe've got their backing as we go forward<a href="#t13:54" class="time">13:54</a>
spoti did the rebuild of these 300 packages so that the maintainers could just rebuild their bits without having to hunt down the other deps in the chain<a href="#t13:55" class="time">13:55</a>
spotthis is the trunk. everything else is leaves.<a href="#t13:55" class="time">13:55</a>
f13the day we merge, is the broken dep report going to be 400 packages long?<a href="#t13:55" class="time">13:55</a>
spotyes.<a href="#t13:55" class="time">13:55</a>
f13hawt<a href="#t13:56" class="time">13:56</a>
spotbut, for those folks tracking rawhide, it should still be mostly upgradeable.<a href="#t13:56" class="time">13:56</a>
nottingspot: is it possible to spit out a compat-libperl for the interim?<a href="#t13:57" class="time">13:57</a>
spotnotting: nope.<a href="#t13:57" class="time">13:57</a>
spoti really don't want to do that.<a href="#t13:57" class="time">13:57</a>
nottingspot: how many of the broken leaves are in the build chain?<a href="#t13:57" class="time">13:57</a>
spotin the build chain for other, non-perl packages?<a href="#t13:58" class="time">13:58</a>
nottingyes. say, auto*<a href="#t13:58" class="time">13:58</a>
spotnada.<a href="#t13:58" class="time">13:58</a>
spotthe 300 rebuilds cover those cases.<a href="#t13:58" class="time">13:58</a>
nottingok then. carry on, sir!<a href="#t13:58" class="time">13:58</a>
spot(base perl pulled in a lot of external modules in 5.10.0)<a href="#t13:58" class="time">13:58</a>
spotmade things somewhat simpler.<a href="#t13:59" class="time">13:59</a>
spota few packages will have to be blocked in devel at the same time that we merge<a href="#t13:59" class="time">13:59</a>
spoti've identified those (still need to email their maintainers and let them know)<a href="#t13:59" class="time">13:59</a>
nirikcouldn't you just add that dist-f9-perl tag to the inheritance in koji, and rawhide would pick up all those builds? or were they just testing builds?<a href="#t13:59" class="time">13:59</a>
spotnirik: that's certainly one way to do it. i defer to f13 on how he wants to do it.<a href="#t14:00" class="time">14:00</a>
spotwe need to block those packages because their n-v-r might be higher than what is in base perl.<a href="#t14:00" class="time">14:00</a>
f13nirik: at the end of the day it doesn't necessarily matter.<a href="#t14:00" class="time">14:00</a>
f13actually wait<a href="#t14:00" class="time">14:00</a>
f13it does matter.<a href="#t14:01" class="time">14:01</a>
nirikwell, it seems a waste to rebuild them all... but whatever works.<a href="#t14:01" class="time">14:01</a>
f13a number of these perl packages have already been built in dist-f9<a href="#t14:01" class="time">14:01</a>
f13nirik: they won't be rebuilt.<a href="#t14:01" class="time">14:01</a>
f13we can't add dist-f9-perl to the inheritance chain, as anythign expecitly built in dist-f9 will win over anythign inherited, regardless of nvr<a href="#t14:01" class="time">14:01</a>
f13instead, we can just koji tag-pkg everything in dist-f9-perl to dist-f9.<a href="#t14:01" class="time">14:01</a>
f13no rebuilding, then everything has a dist-f9 tag.<a href="#t14:02" class="time">14:02</a>
nirikcool.<a href="#t14:02" class="time">14:02</a>
spotyeah, i wasn't planning on doing two rounds of rebuilds. :)<a href="#t14:02" class="time">14:02</a>
f13spot: you'll still have to identify the things that have been built since you did your dist-f9-perl build of them<a href="#t14:02" class="time">14:02</a>
spotyep.<a href="#t14:02" class="time">14:02</a>
spotthankfully, most of these things don't move much at all.<a href="#t14:03" class="time">14:03</a>
f13nod<a href="#t14:03" class="time">14:03</a>
f13ok, we're running out of time here.<a href="#t14:03" class="time">14:03</a>
f13spot: do you want to try and land perl before or after the mass rebuild?<a href="#t14:03" class="time">14:03</a>
spotperfect world, before.<a href="#t14:04" class="time">14:04</a>
f13(and I imagine a number of your packages will have to get built anyway, for the new gcc)<a href="#t14:04" class="time">14:04</a>
f13ok.<a href="#t14:04" class="time">14:04</a>
spotf13: some of them have already picked up the new gcc<a href="#t14:04" class="time">14:04</a>
spotit doesn't seem to break anything yet.<a href="#t14:04" class="time">14:04</a>
nottingbreak-everything-at-once?<a href="#t14:04" class="time">14:04</a>
f13notting: please no.<a href="#t14:04" class="time">14:04</a>
spoti did a rebuild of perl-5.10 itself after gcc43 hit rawhide<a href="#t14:04" class="time">14:04</a>
spotand it is fine.<a href="#t14:04" class="time">14:04</a>
f13spot: goot.<a href="#t14:04" class="time">14:04</a>
f13spot: lets see where you're at next week?  I"d like to see FESCo get some visibility on this beofre we start breaking the world.<a href="#t14:05" class="time">14:05</a>
spotsure.<a href="#t14:05" class="time">14:05</a>
spothopefully, i'll be ready before then.<a href="#t14:05" class="time">14:05</a>
nottingwell, that would push back the 'regular' mass rebuild, then<a href="#t14:06" class="time">14:06</a>
notting(which is ok, just making sure we have a plan together)<a href="#t14:06" class="time">14:06</a>
f13notting: yeah, perl next week, mass rebuild the week after, leading up to Beta<a href="#t14:06" class="time">14:06</a>
-!- f13 changed the topic of #fedora-meeting to: Fedora RelEng: Open Discussion<a href="#t14:09" class="time">14:09</a>
f13any thing else people would like to talk about this week?<a href="#t14:09" class="time">14:09</a>
f13alright, calling it a meeting.<a href="#t14:12" class="time">14:12</a>
f13cheers all!<a href="#t14:12" class="time">14:12</a>

Generated by 2.3 by Marius Gedminas - find it at!