From Fedora Project Wiki

Fedora Release Engineering Meeting :: Monday 2008-02-25

Rebuild status

  • going fairly smoothly
  • the triage team is taking good swaths of those and isolating the build failures and sorting them into buckets (either compile failure, or other failure)

Orphan package removals

  • propose those orphans that haven't been picked up be blocked from dist-f9 today
  • gives some folks time to realize "oh that was important, I'll take it over" before we hit beta freeze
  • warren will work on analyzing the orphan list

Release Engineering Representative

Discussion of Custom Spins

  • 11 more in the pipeline
  • See IRC log

Misc

  • setting up a meeting tomorrow to talk about the fedora build system instability and get some time lines for fixing what we can, etc.

IRC Transcript

-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Roll Call12:59
f13ping notting jeremy poelcat jwb spot warren poelcat rdieter wwoods lmacken13:00
* warren here13:00
* jeremy 13:00
jwbhere13:00
* wwoods nom nom granola nom nom13:01
* poelcat here13:02
* notting is here13:02
f13ok, we've got a few things to discuss so lets get started.13:02
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering - Rebuild status13:02
jwbthat went surprisingly well13:03
f13All the first round of builds completed on Friday13:03
f13early Friday morning in fact13:03
f13and all the bugs for the failures were filed later that day13:03
f13the triage team is taking good swaths of those and isloating the build faiulre and sorting them into buckets (either compile failure, or other failure)13:04
f13lots of fixes have been happening, and most of the buildsystem generated failures have been resubmitted13:04
f13all in all this has been one of the smoother rebuilds from the submission and followup side of things.13:04
f13I want to clean up the scripts I used for more general purpose and check them into the rel-eng git tree13:04
f13GCC 4.3 rebuilds have been a "would be nice" sort of feature so I'm not going to sweat it too much if things aren't rebuilt by the beta or even the release.13:05
f13(unless of course the failure to rebuild results in a failure to fix some other bug or blocker)13:05
jwbrelated topic, did you see someone's request for a package for the bump script?13:06
f13yeah, I'm fine with it going into a package.13:06
f13probably makes sense in rpmdevtools13:06
f13although it is pretty Fedora specific13:06
wwoodsfedora-packager?13:06
jwbsomething like that, yeah13:07
f13wwoods: we have other pretty Fedora specific things in rpmdevtools so I don't think it really matters much13:07
wwoodsmakes sense13:07
jwboverall i don't think many of the other rel-eng scripts should really be "packaged"13:07
f13nod13:07
jwbok, sorry to derail13:07
f13typically they're mostly references to start from in order to accomplish your specific task13:07
f13and that doesn't make much sense as a package outside of examples/13:08
f13anywho, the fixes will continue, and folks like jeremy have offered to look at the real hard compile issues.13:08
jwbjakub seems to be poking at issues that get reported to -devel13:09
f13gcc43errors is the bug alias that the triage folks are putting the actual compile issue bugs on13:09
f13I think at this point, releng can just let this work itself out naturally13:09
f13moving on...13:10
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering - Orphan package removals13:10
f13warren: you were running with this I think, what's the current status?13:10
warrenf13, there were relatively few and many were picked up13:11
warrenf13, come back me in about 10 minutes, i'll check it now13:11
f13warren: ok.13:11
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering - Making releases a snap13:11
f13stickster has called for folks to report to him on SOPs for doing releases13:11
f13and we owe him a report13:11
jwbi'm sort of confused as to what he wants13:12
f13let me pull up the archived mail13:12
poelcatf13: he wants a point person13:12
f13https://www.redhat.com/archives/fedora-advisory-board/2008-February/msg00136.html13:12
poelcatso we can have a concall in advance of the beta to make sure all the groups are in sync13:12
poelcatand dependencies are handled in order and in time13:13
f13so this is just for beta this time, and we'll do it again for PR and Final?13:13
jwbhm13:13
jwbis he going to include arch teams?13:13
f13that's a fun question13:14
f13theoretically secondary arch teams move at their own schedule13:14
f13and aren't tied directly to our release dates13:14
poelcatin theory we do it for all of the releases... and each time it gets faster and more organized :)13:14
jwbi nominate f1313:14
f13So unless somebody else wants to volunteer, I'll handle this13:14
f13gee thanks jwb (:13:15
jwbdoes anyone else really make sense?13:15
f13probably not the first time around13:15
wwoodsyou *are* kind of the guy who turns the handcrank13:15
jwbif you'd like, i can attend as well if possible13:15
wwoodsI guess the theory is that we can take notes and then, eventually, other people will know how the sausage gets made13:15
jwbthough it would be more of a "listen in" kind of thing13:16
warrenf13, I can't login to the database to do a query13:16
f13warren: I saw.13:16
f13ok, I'll take it this time.13:16
* f13 adds another note to the todo list13:16
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Orphan package removals13:17
f13lets get back to this one for a moment.13:17
f13since we're nearing very close to the Beta freeze, I propose those orphans that haven't been picked up be blocked from dist-f9 this week, today.13:17
f13that'll give some folks time to realize "oh that was important, I'll take it over" before we hit beta freeze13:17
jwbagreed13:18
f13warren: that seem sane to you?13:18
nottingseems ok13:18
warrenumm13:19
warrenthe orphan list seems to have grown since last week13:19
warrenf13, can we warn with a fresh list later today with thursday the target removal day?13:20
warrenf13, I need a little time to analyze the new list13:20
* warren not sure if his SQL query is correct13:20
f13warren: sure, that's fine.  This week though.  I don't want to rip off the bandaid and immediately freeze the tree13:20
warrenf13, nod13:21
warrenf13, the freeze next week is a hard freeze?13:21
warrenf13,  or a slushy one like F8's?13:21
f13the freeze on the 4th is13:21
f13uh13:21
warrenhonor system or manual tagging?13:21
f13what is your definition of 'slushy' ?13:21
warrenRemember last time around we used honor system?13:21
f13we did?13:21
warrenMaybe that's the wrong word13:22
f13we branched and told people if they really want to build for F8, build from the branch13:22
warrenhuh?13:22
f13otherwise if they built from devel it would go to dist-f913:22
warrenisn't branching optional at freeze point?13:22
f13warren: I think we're talking about two different freeze points13:23
warrengrr, can't load the wiki13:23
warrenf13, I thought next week's freeze is for Beta and feature freeze, and the wiki said people can optionally branch CVS at that point.13:23
f13regardless, the freeze on the 4th is hard, you have to request tags from releng13:23
warrenThat sounds fine, but when was that decided?13:24
f13http://fedoraproject.org/wiki/ReleaseEngineering/Overview#head-7dfffd8d508d35c931d4b7cc519fce1b00019e4c13:24
warrenI thought the F8's version of this "freeze" worked out extremely well compared to previous freezes that operated in a manner similar to what you are saying happens next week.13:24
f13warren: F8's only works if we allow for branching13:25
f13and it really only works at the tail end of a development cycle, not in the middle of it13:25
warrenwhen do we allow branching this time?13:25
f13we never decided at what point between beta release and final freeze we'd start allowing for branches.13:25
warrenI thought the schedule said at beta freeze point13:26
warrenI could be wrong, but I can't load it to find out13:26
warrenoh, http://fedoraproject.org/wiki/Releases/9/Schedule  March 13th allow branching13:26
warrennm13:26
f13Overview still says:13:26
f13Early (optional) CVS branching13:26
f13Right after Beta release or sometime between Beta and Pre-release, we allow for early branching of software. This allows developers to check in new features and otherwise unstable changes that would not be suitable to introduce to the current release.13:26
f13but I guess the schedule took that as 'right after beta release'13:27
f13but in reality that's going to be very very early for a lot of software13:27
warrenOK, sounds good.13:27
* warren works on analyzing the orphan lits13:27
f13ok, moving on.13:28
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering - Jeremy's spin writeup13:28
f13shamefully, I haven't taken the time to read this fully through yet.13:28
f13jwb: have you?13:28
jwbf13, i told jeremy i would reply today.  i still intend to13:28
jeremyit'd be nice to get some feedback so that I can post it for more people to see13:28
jeremyso I can get spoleeba off my back :-P13:28
f13yeah.13:28
jwbjeremy, you'll have something this afternoon from me13:29
jeremyjwb: sounds good, thanks13:29
f13I really wanted to get to it before the meeting, finding out why rawhide fell over took too much time though :/13:29
f13I will also get you feedback today, soon even, so that you can send it out today13:29
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering - Open Discussion13:29
jeremysounds like a plan13:29
jwbspeaking of spins13:30
jwbi've done two now for mether.  xfce and hindi13:30
* f13 gives jwb the floor13:30
jwbhe has like 4 more queued up (or some number)13:30
jwball localized13:30
warrenoh damn13:30
warrenthe orphan list has grown13:30
warren125 packages now13:30
jwbi started a SOP for doing spins, but it's empty now.  i intend to fill it out with this next one13:31
nottingjwb: only 4?13:31
jwbno, more.13:31
* jwb counts13:31
warrenoh wait, some of these were already removed13:31
jwb11 more13:31
f13I got a scary message from spevack today, where he wants to do a localized spin for wherever we have an event13:31
f13that's going to lead to a /lot/ of spins, just for lang selections13:31
f13I for one, don't like this13:31
jwbwe have 12 total right now.  1 done, 11 to go13:31
jwbjust for .in13:31
jwb(i think)13:32
f13it's a lot of duplicated content, it's a lot of releases to try and keep track of the sources of for GPL reasons, it's a lot of QA to go through, and it'll start making our download pages rather cumbersome13:32
metherone of them is for Srilanka13:32
wwoodsugh. I'm gonna need a team of hax0rs to work on automating testing live images if we're going to do this much stuff13:32
jwbthis is an illustrative example of why we need to review jeremy's doc13:33
jwbbecause the people in this meeting will not scale to test all of these, nor _can_ we in some cases13:33
jwbi don't speak/read hindi, etc13:33
f13right.  I think we said we wanted to see interesting things folks have done with spins, and I'm just not sure of "only include my local language" is all that... interesting.13:34
f13or rather, we wanted to /host/ interesting things13:34
nottingf13: then we need to fix livecd-creator13:34
jwbnotting, ?13:34
wwoodsjwb: right - there needs to be docs and tools for letting the *spinners* test their spins13:34
nottingjwb: multiple langauges, selectable13:34
wwoodsany plan that involves dumping more work on rel-eng or QA = fail13:34
f13notting: wouldn't that be gdm in F9?13:35
nottingf13: ??? GDM in *F8* does that13:35
jwbpersonally, i have no issues mashing buttons to churn out isos and throw them on spins.fp.o13:35
jeremywouldn't that be that gdm has allowed it for a long long time13:35
f13ok, so I'm confused13:35
nottingthese 20 different spins a) set a default timezone and /etc/sysconfig/i18n LANG b) occasionally adjust the translation sset13:36
pingou!13:36
f13jwb: yeah, I just have issues with those carrying the Fedora name/logo without anybody doing QA on them, ensuring that they actually install, that no regressions creeped in due to using newer code sets, finding a place to stash the srpms used (!!) for GPL reasons, etc...13:36
f13pingou: that's not necessary in this meeting, just blurt it out.13:36
pingouthere is the language but also the keyboard that is interesting to changes between the different livecd13:37
nottingpingou: i stand by my statement - 10-20 livecd images that only differ by 1% is wasteful13:37
f13seems that creating another 700~ meg download for that is.... silly13:37
wwoodsunless we work out some magic jigdo way of storing them13:38
jwbf13, i'm not disagreeing.  i think that's where we need to go back to the board and say "please clarify why these are official spins"13:38
pingouwhat about using the local mirror ?13:38
f13pingou: in many cases "what local mirror?"13:38
jeremynotting: xdelta!13:38
pingouf13, something like storing the french version on the french mirror, the dutch on the dutch...13:39
warrenjeremy, will xdelta work when the changes are within the compressed squashfs image?13:39
jwbpingou, if they have a local mirror, why can't they do the spin themselves?13:39
pingoujwb, they start to do it13:39
f13pingou: we don't have a 1:1 mirror:country relationship13:39
jwbwe could add links to those13:39
jeremywarren: dunno.  someone is welcome to try :-)13:39
warrenjeremy, the pyjigdo guys were talking about making jigdo uncompress and recompress squashfs images with jigdo...13:40
jwbi don't think we're going to make a huge amount of progress on this today.  i just wanted to point at where i am with the spins stuff and raise the bigger question13:40
* jeremy sticks his fingers in his ears and goes "la la la la la la la la"13:40
nottingwarren: that sounds fraught with failure13:40
warrenI just mentioned it13:41
warrenok... so the orphan list has definitely grown since last week13:41
* f13 has yet to see pyjigdo in action, for it's original purpose, let along feature creep13:41
f13er alone.13:41
f13jwb: thanks, that does help13:42
warrenlooks like I'll have a complete list of orphans in about 30minutes hwen te script finishes13:42
warren(koji is really slow now)13:42
warrenf13, really, there is no need to be so down on them, they are free to work on whatever they want.13:42
mbonnetwarren: it's working fine for me, seems snappy, and neither the database nor app server are loaded13:42
warrenmm13:42
mbonnetwarren: sure you're not having connection issues?13:43
warrenmbonnet, abadger1999 mentioned having issues coming in through RH VP13:43
warrenmbonnet, RH VPN13:43
jwbf13, i've used pyjidgo.  it worked fairly well13:43
warrenmbonnet, I seem to as well13:43
jwbwarren, why are you talking to koji?  shouldn't orphans be done from pkgdb?13:43
f13warren: I'm not being down, it's just we're a week away from the beta freeze and I haven't seen a single patch for being able to do jigdo stuff for F9.13:44
warrenjwb, pkgdb knows about orphaned packages, but not if they were already removed13:44
jwbah13:44
jwbnor deltarpms... jeremy ?13:44
warrenabadger1999 and jdieter were working on deltarpms?13:44
warrenoh wait13:44
warrenno13:44
jeremyjwb: I haven't had anything to do with any of the work for deltarpms this time around13:44
warrenlmacken13:44
jwbk, sorry13:45
warrenjwb, most of the deltarpm work is needed in bodhi, with a little rel-eng scripting after that13:45
abadger1999warren: Signing me up for extra work again? ;-)13:45
warrenabadger1999, maybe.13:45
jeremyabadger1999: we try! :)13:45
f13sadly nothing tends ot be 'little' when it comes to releng scripting13:45
jwbwarren, so we'd only target updates with that?13:45
warrenjwb, that's where most of its usefulness is no?13:46
warrenf13, relatively little compared to the bodhi work13:46
jwbyes.  i was just checking.  if so, it's fairly disjoint from being tied to F9 (or any other release)13:47
f13except that unleashing deltas upon a stable release without any testing during rawhide sounds like a fun event.13:47
warrengrr, you're right13:48
metherFYI SUSE has no build system integration and just runs createdeltarpm or whatever post update generation13:48
f13mether: they also don't have a very public mirror system and are able to control that a lot better13:48
nottingmether: um, that would still be 'deltarpm work needed in bodhi'13:48
warrenf13, would you accept adding the deltas to rawhide after beta is released?  Only thing we're missing is a defined place to put them in the tree right?13:49
methernotting: they run a cron job or something async13:49
f13warren: honestly I don't know what all is missing, I haven't talked to anybody about it in a while.  Rather, nobody has talked to /me/ about it in a while.13:49
nottingisn't that what this meeting is for?13:49
warrenyes13:49
f13notting: you'd think that.13:49
f13who's the feature owner?13:50
warrenOK, it seems to me that the only thing missing to add deltas to rawhide is a defined place to put it in the tree?13:50
warrensort of me13:50
warrenjdieter has been taking point13:50
f13warren: so then we have patches to mash to generate deltas, and huristics on what to generate deltas against, and a way of dealing with signed packages vs unsigned and...13:50
warrenf13, is mash the right place to put it?13:51
f13warren: mash is what creates rawhide13:51
f13or a cron job does really13:51
f13so patches to the cron would also work I suppose.13:51
warrenwhat's the source repo for this?13:51
warrenI'll point this at jdieter13:52
warrenI can work on this after feature freeze myself, but not now.13:52
f13git.fedorahosted.org/git/releng13:52
f13I'm not going to be too keen on working on this after feature freeze though.  I really really want to land F9 on time and I don't want to make things more unstable in the rawhide creation land13:52
warrenThe sad thing about this is, there is no reason why deltas absolutely need to be done during tree creation.13:53
warrenthey can be async13:53
warrenit does make it crappy for mirroring though13:53
warrenI'll talk with jdieter13:53
warrenI'm afraid this will slow down rawhide creation13:54
f13yeah, our mirroring system is fragile enough as it is.13:54
warrenah, orphan list is complete13:54
warrenhttp://paste.ausil.us/28413:55
warrenI'll send a warning to the lists that these packages and anthing that depends on them will be removed Thursday afternoon if they are not owned.13:55
warrenok?13:55
jwbxmlrpc...13:55
jwbthat seems... important13:55
warrenI DID warn people last week13:56
f13warren: good.13:56
* warren sending13:56
f13anything else this week?13:57
jwbyah13:57
jwbwhat about the meeting you wanted to setup with the mikes and mark on koji?13:57
jwbor is that outside the scope of this meeting?13:58
f13kinda outside I think.13:59
jwbk13:59
f13but for those listening at home, we're setting up a meeting tomorrow to talk about the fedora buildsystem instability and get some timelines for fixing what we can, etc. etc..14:00
f13Ok, calling it, thanks folks.14:01

Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!