From Fedora Project Wiki

Fedora Release Engineering Meeting :: Monday 2008-02-04


Reversing the netapp streams

  • scheduled for today--we're working on pushing it back after the alpha
  • change setup so that the content we generate in PHX can be put on a netapp in PHX
  • instead of rsynced to RDU to be put on a netapp in RDU, and then mirrored by netapp back out to PHX
  • this will result in shorter times between compose and visible to the world for things like rawhide and updates
  • may delay the rawhide/updates for a bit

gcc 4.3

  • gcc4.3 is in the buildroots for dist-f9 now, and some fallout has happened, namely kernel.
  • striving to complete a mass rebuild before beta release


  • Feature page:
  • 300 perl dependent packages as a sample set
  • these packages were the base for which virtually all of the other perl dependent packages depend on
  • out of those 300 about 60 more to build
  • once complete the plan is to merge the new perl and the rebuilt bits into rawhide
  • Fedora is the place where perl-5.10 was really first tested/used and the development ground for getting all the upstreams ported to the new perl

IRC Transcript

-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting13:05
f13ping: notting jeremy jwb spot warren rdieter wwoods poelcat lmacken13:06
* notting is here13:06
* lmacken 13:06
* jeremy is here13:06
* spot is here13:07
* poelcat here13:07
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - F9 Alpha13:08
f13so the Alpha is being staged by mirrors.  They've had access to it since Friday13:08
f13so hopefully we'll have enough ready to go tomorrow13:08
f13I'm in teh process of uploading the isos to the torrent server and creating the torrents.  That should be done this afternoon13:08
lmackenf13: F9-Alpha-i686-20080129.0.iso ?13:09
f13lmacken: yeah, but renamed.13:09
f13we'll hope the bit gets flipped and all will be happy tomorrow.13:10
f13anybody have anything else to bring up wrt Alpha?13:10
wwoods is the test status page13:11
wwoodswe may need an updates.img for upgrades13:11
f13yeah, it's a bit rough around the edges, but it is an Alpha13:11
wwoodsjust trying to keep ahead of bug reporters to make life easier for triagers/developers13:12
wwoodsoh - I want to pick a week sometime this month13:12
wwoodsa couple weeks before the string freeze13:12
wwoodsto make double-damn sure we have working boot/install images13:12
wwoodsso that we can put out a call for translators to grab the rescuecd (or minimal-installer.iso or whatever we call it)13:13
jeremywwoods: *nod*13:13
jeremyand I'm probably going to write a script in the next few days which will help make that a bit easier13:13
wwoodsprobably that'd be a good time to also make sure we have good Live images so we can have a big internal Laptop Test-fest13:14
wwoodsjeremy: sweet13:14
wwoodsdo we have ppc images for alpha?13:15
f13wwoods: yes13:15
f13not live ones though13:15
wwoods's cool13:16
wwoodsI should probably do some ppc testing.. or ask jwb to update the test status page13:17
wwoodsf13: if you can give me a pointer to an internal copy of the ppc bits I'll follow up on that after this meeting13:17
f13wwoods: it's on reducto13:17
f13moving on...13:19
-!- f13 changed the topic of #fedora-meeting to: Fedora RelEng: Reversing the netapp streams13:19
nottingthis was scheduled for today. we're working on pushing it back after the alpha13:19
f13later this week we would like to reverse the netapp streams so that the content we generate in PHX can be put on a netapp in PHX, rather than rsynced to RDU to be put on a netapp in RDU, and then mirrored by netapp back out to PHX13:19
f13this will result in shorter times between compose and visible to the world for things like rawhide and updates13:20
f13a number of our processes will have to change though13:20
f13and ti's possible that rawhide/updates may be delayed for a bit13:21
f13as we trial and error our way through it13:21
f13notting, lmacken, mmcgrath, and I are the primary folks on this task, so if you have any thoughts/concerns/suggestions hit up one of us.13:22
f13anything further on this subject?13:23
-!- f13 changed the topic of #fedora-meeting to: Fedora RelEng: Gcc4313:23
f13gcc4.3 is in the buildroots for dist-f9 now, and some fallout has happened, namely kernel.13:24
f13but more importantly is that jakub has indicated that he'd like to see a massrebuild of our packages against the new gcc at some point, after a week or so if it being in the buildroots13:24
nottingso... next week?13:25
f13March 4th is the Beta freeze13:25
f13so we'd likely have to do it next week or the week after13:25
f13and we have to decide if htis is going to be maintainer driven, or releng driven.13:25
f13IE we ask maintainers to do it and wait, or we just plow through and script massive amounts of builds.13:25
nottingautomated seems simpler to me13:26
f13you'd think.13:26
nottingesp. if there aren't ordering concerns13:26
f13trying not to have conflicts is the problem.13:26
f13IE automation kicks off a build, maintainer comes along after that and also kicks off a build.13:27
f13automated was a backgroundjob, so the maintainer's job gets done first, then the automated one, automated (older) one is now the last tagged, and thus the 'newest'.13:27
poelcatf13: wouldn't we want the rebuilt packages to be in the beta?13:27
f13poelcat: yeah... that's why we're doing it before the beta freeze13:28
poelcatf13: sorry misread above13:28
nottingf13: is there a way to not do 'automated/background'? or to have koji check for backwardsness?13:29
f13notting: if we don't do background, we essentially tie up the buildsystem for a few days.  no other buidls will get through until our entire set is done.13:30
warren_treosorry at doctor13:30
f13notting: as for backwardness, I don't thikn there is anything koji itself can do.  We can run a query on the tag after all is said and done to detect nvr issues though.13:30
f13the added fun is that spot is going to want to land the new perl relatively soon as well, which is going to cause all kinds of pain13:31
warren_treowhat ver of perl?13:31
* spot has been chainbuilding for weeks13:32
warren_treospot: is there a URL describing the compat impact of this new perl?13:33
* warren_treo wonders if anybody tested spamassassin on perl 5.1013:34
nottingf13: so, do we have the scripts ready to start the automated buildatron?13:35
f13notting: nope.  I haven't spent any time on that recently13:35
warren_treobuild order is known?13:35
f13warren_treo: we're ignoring buildorder13:36
f13at least for the first pass.13:36
f13to avoid cvs conflicts, I imagine the script would go through each needed build (query to fall out of koji for things not yet built with new gcc), do a checkout, do an autobump of hte spec (*shudder*), commit/tag/build --background; and move on.13:37
nottingwhen was the last time we did one?13:38
f13most of that has been done before13:38
f13the autospecbump does run into problems every now and again with malformed spec files13:38
f13and catching either cvs errors, or tag errors can be difficult, as well as gathering in the build failures for investigation.13:38
warren_treomalformed as in encoding.13:39
f13warren_treo: no, as in dirty games with Release: tags13:39
f13programatically editing spec files is not easy13:39
nottingwas the last rebuild maintainer-driven?13:39
f13notting: yeah13:39
nottingthe other option is to punt maintainer vs. automated to fesco >:)13:40
warren_treobtw did we get past the gcc43 kernel build issue?13:40
f13the last one that we did any scripting of was fc6 time frame.  And then some smaller less scripted ones for the last glibc fallout where ppc was boned.13:40
f13warren_treo: not yet afaict13:40
f13also, last time we let maintainers opt in/out of autobuilding13:41
f13usually with a file in their module.13:41
f13something we should consider this time.13:41
f13basically releng should come up with a proposal on how to accomplish the rebuilds and present it to FESCo13:42
warren_treoI hope we can do it without cvs...13:42
f13I suppose that will fall on me unless anybody else wants to take a stab at it.13:42
warren_treooptin out that is13:42
f13warren_treo: have better ideas?13:42
f13we need to identify and mark packages that need to be rebuilt13:42
f13and then let maintainers scrub them from the list.13:43
f13maybe a wiki page?13:43
warren_treocould it be added as a flag in pkgdb?  might be quick13:43
nottingpkgdb lives forever, though13:43
f13yeah, not sure if pkgdb is setup for arbitrary flags like that13:43
f13so I'll work up a plan, and ping ya'll as it comes along.  Not enough days in the week.13:45
-!- f13 changed the topic of #fedora-meeting to: Fedora RelEng: perl-5.10.013:45
warren_treoprior to gcc43 I heard mention that mass rebuild wouldn't be necessary, what happened?13:45
f13warren_treo: it's not /necessary/, but desired by the gcc team.13:46
f13spot: can you give us a rundown on the perl status?13:46
spoti identified 300 perl dependent packages as a sample set13:46
spotthese packages were the base for which virtually all of the other perl dependent packages depend on13:47
spotOut of those 30013:47
spoti have about 60 more to build13:47
spotonce that is done, the plan is to merge the new perl and the rebuilt bits into rawhide13:48
warren_treoperl-5.10 and pkgs built against it are in a separate tag now?13:48
spotwarren: yes, dist-f9-perl13:48
spotonce merged, there will still be lots of things broken13:49
f13spot: are you planning on doing a Feature for perl-5.10, which includes info about when it'll land, and what we expect other perl maintianers to do once it does land?13:50
f13(and a rollback plan)13:50
spotf13: Yep, just hadn't gotten a round tuit yet.13:50
nottingspot: why will there be things broken?13:50
warren_treorollback sounds scary when hundreds of pkgs re involved :)13:50
f13warren_treo: untag!13:51
f13unless you want to force epoch into all of those.... (:13:51
spotnotting: because everything hasn't been rebuilt against the new perl yet13:51
nottingspot: .... is dist-f9-perl not going to encompass 'everything'?13:51
warren_treoeven things like pidgin build against perl13:51
nottingor are you only rebuilding pure-perl13:52
spotnotting: it would be much faster if the individual maintainers did the rebuilds13:52
* warren_treo wonders what will utterly fail13:52
spotwarren: so far, only one thing has failed, and I fixed it.13:52
f13this could put Fedora in a pretty cool place13:52
warren_treospot: what was the cause of that failure?13:52
spotthe difficulty is in the ordering13:52
f13Fedora was the place where perl-5.10 was really first tested/used and the development ground for getting all the upstreams ported to the new perl.13:53
spotwarren: old crufty code making bad assumptions about array behavior13:53
warren_treof13: yes assuming the code continues to work13:53
nottingf13: 'fedora 9 shipped a new perl. it worked. no one noticed.'?13:53
warren_treogotta go now, appt coming up13:54
spotnotting: upstream perl certainly is watching this.13:54
spotwe've got their backing as we go forward13:54
spoti did the rebuild of these 300 packages so that the maintainers could just rebuild their bits without having to hunt down the other deps in the chain13:55
spotthis is the trunk. everything else is leaves.13:55
f13the day we merge, is the broken dep report going to be 400 packages long?13:55
spotbut, for those folks tracking rawhide, it should still be mostly upgradeable.13:56
nottingspot: is it possible to spit out a compat-libperl for the interim?13:57
spotnotting: nope.13:57
spoti really don't want to do that.13:57
nottingspot: how many of the broken leaves are in the build chain?13:57
spotin the build chain for other, non-perl packages?13:58
nottingyes. say, auto*13:58
spotthe 300 rebuilds cover those cases.13:58
nottingok then. carry on, sir!13:58
spot(base perl pulled in a lot of external modules in 5.10.0)13:58
spotmade things somewhat simpler.13:59
spota few packages will have to be blocked in devel at the same time that we merge13:59
spoti've identified those (still need to email their maintainers and let them know)13:59
nirikcouldn't you just add that dist-f9-perl tag to the inheritance in koji, and rawhide would pick up all those builds? or were they just testing builds?13:59
spotnirik: that's certainly one way to do it. i defer to f13 on how he wants to do it.14:00
spotwe need to block those packages because their n-v-r might be higher than what is in base perl.14:00
f13nirik: at the end of the day it doesn't necessarily matter.14:00
f13actually wait14:00
f13it does matter.14:01
nirikwell, it seems a waste to rebuild them all... but whatever works.14:01
f13a number of these perl packages have already been built in dist-f914:01
f13nirik: they won't be rebuilt.14:01
f13we can't add dist-f9-perl to the inheritance chain, as anythign expecitly built in dist-f9 will win over anythign inherited, regardless of nvr14:01
f13instead, we can just koji tag-pkg everything in dist-f9-perl to dist-f9.14:01
f13no rebuilding, then everything has a dist-f9 tag.14:02
spotyeah, i wasn't planning on doing two rounds of rebuilds. :)14:02
f13spot: you'll still have to identify the things that have been built since you did your dist-f9-perl build of them14:02
spotthankfully, most of these things don't move much at all.14:03
f13ok, we're running out of time here.14:03
f13spot: do you want to try and land perl before or after the mass rebuild?14:03
spotperfect world, before.14:04
f13(and I imagine a number of your packages will have to get built anyway, for the new gcc)14:04
spotf13: some of them have already picked up the new gcc14:04
spotit doesn't seem to break anything yet.14:04
f13notting: please no.14:04
spoti did a rebuild of perl-5.10 itself after gcc43 hit rawhide14:04
spotand it is fine.14:04
f13spot: goot.14:04
f13spot: lets see where you're at next week?  I"d like to see FESCo get some visibility on this beofre we start breaking the world.14:05
spothopefully, i'll be ready before then.14:05
nottingwell, that would push back the 'regular' mass rebuild, then14:06
notting(which is ok, just making sure we have a plan together)14:06
f13notting: yeah, perl next week, mass rebuild the week after, leading up to Beta14:06
-!- f13 changed the topic of #fedora-meeting to: Fedora RelEng: Open Discussion14:09
f13any thing else people would like to talk about this week?14:09
f13alright, calling it a meeting.14:12
f13cheers all!14:12

Generated by 2.3 by Marius Gedminas - find it at!