ReleaseEngineering/Meetings/2007-jul-09

Fedora Release Engineering Meeting :: Monday 2007-07-09

Composes in the colo

• downsides are that it will:

1. probably take longer to complete the compose 1. certainly take longer to get the compose into the hands of our testing lab in Raleigh 1. take longer to get onto the machines in Westford

• other benefits most likely outweigh these issues.
• f13 is single point of failure for doing composes--everything is dependent on him

Decision: Releng will continue investigations into composing trees within the colo; potentially composing Test1 there.

Fedora Signing Server

• f13 has not received any feedback since last meeting (some gave it before the meeting)--moving proposal to the ReleaseEngineering/ space as it's new home

Open Discussion

• jwb--have we documented the "you need to email rel-eng to get packages in the buildroot" thing?
• jeremy--sounds like something worth adding to the update doc. feel free to add :)
• notting--we were going to switch that based on bodhi changes with an alert for packages built against unpushed things
• f13--but until we switch it, we need it documented somewhere. Once bodhi can prevent an update from going out that was built against an unreleased update, we can make the buildroot selfupdate.
• jeremy--note that may need to be overridable, otherwise, I have a _great_ way to keep security updates from being able to go out :)
• f13--we do need to be able to override it.

Decision: No decision made

IRC Transcript

f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Composes in the colo - JesseKeating		<a href="#t12:58" class="time">12:58</a>
* spot warns that anyone caught near my chicago apt will be forced to help pack		<a href="#t12:58" class="time">12:58</a>
f13	rdieter: jwb: wwoods: ping	<a href="#t12:59" class="time">12:59</a>
jwb	here	<a href="#t13:00" class="time">13:00</a>
rdieter	here (for awhille anyway)	<a href="#t13:00" class="time">13:00</a>
f13	ok, so lets talk a bit about doing composes in the colo.	<a href="#t13:00" class="time">13:00</a>
f13	as opposed to my desk	<a href="#t13:00" class="time">13:00</a>
warren	f13, I like relying upon services under our desks.	<a href="#t13:01" class="time">13:01</a>
jwb	ew	<a href="#t13:01" class="time">13:01</a>
f13	this has some obvious benifits, others can get to the compose bits a lot quicker (or at all), theoretically somebody other than me can do a compose, etc...	<a href="#t13:01" class="time">13:01</a>
f13	poelcat: ping;	<a href="#t13:01" class="time">13:01</a>
f13	The downsides are that A) it will probably take longer to complete the compose, B) it will certainly take longer to get the compose into the hands of our testing lab in Raleigh, C) it'll take longer to get onto the machines in Westford	<a href="#t13:02" class="time">13:02</a>
* wwoods here		<a href="#t13:02" class="time">13:02</a>
jwb	f13, why can't composes be done on your machine and then uploaded somewhere?	<a href="#t13:02" class="time">13:02</a>
f13	but the other benifits most likely outweigh these issues.	<a href="#t13:02" class="time">13:02</a>
f13	jwb: upload slow, and I'm less likely to upload right away and I end up doing lots of testing locally before incurring the cost of an upload	<a href="#t13:03" class="time">13:03</a>
f13	jwb: I may end up doing composes in two locations though, at the same time.	<a href="#t13:03" class="time">13:03</a>
wwoods	jwb: it also means that nobody but jesse can do the composes, unless he syncs his config etc.	<a href="#t13:03" class="time">13:03</a>
f13	at least until the trees stop changing as much and then rsync will just work.	<a href="#t13:03" class="time">13:03</a>
jwb	you have a script, yes?	<a href="#t13:03" class="time">13:03</a>
f13	not exactly	<a href="#t13:03" class="time">13:03</a>
jwb	write a script and have it sync the compose and config at the end	<a href="#t13:04" class="time">13:04</a>
jwb	in the background	<a href="#t13:04" class="time">13:04</a>
f13	welll	<a href="#t13:04" class="time">13:04</a>
f13	given that then everybody is completely dependant upon me to produce things I think that's a bad route to go down.	<a href="#t13:05" class="time">13:05</a>
jwb	depends i guess	<a href="#t13:05" class="time">13:05</a>
jwb	if <upload location> were a username based location, then others could do composes and upload them as well, no?	<a href="#t13:05" class="time">13:05</a>
poelcat	f13: here	<a href="#t13:05" class="time">13:05</a>
jeremy	jwb: but why should "I'm a composer" be dependent on "I have tons of bandwidth"	<a href="#t13:06" class="time">13:06</a>
jeremy	(and tons of upload bandwidth at that)	<a href="#t13:06" class="time">13:06</a>
jwb	jeremy, shouldn't.  this can all be done _in addition to_ composing in the colo	<a href="#t13:06" class="time">13:06</a>
jeremy	jwb: plus, it places a lot of trust in the "how is the machine set up and the composes being done"	<a href="#t13:06" class="time">13:06</a>
f13	The process to compose a tree for releases is as such:  tag newly needed build -> sign unsigned packages -> mash a tag -> adjust pungi + mock configs to point to new mash output -> update mock chroot -> run pungi -> make tree accessable.	<a href="#t13:06" class="time">13:06</a>
rdieter	f13: you're right, of course, the pros outweigh the cons.  How much work/pain is involved to implement this?	<a href="#t13:06" class="time">13:06</a>
f13	rdieter: probably not a whole lot of work.  We already have one x86_64 box that we use for various compose like things that can be used to compose i386/x86_64 (once it's given more disk space/ram).	<a href="#t13:07" class="time">13:07</a>
jwb	so maybe i'm misunderstanding things	<a href="#t13:07" class="time">13:07</a>
f13	finding a ppc box may be more difficult, and in teh short term we may just have to (ab)use a koji builder to od the compose.	<a href="#t13:07" class="time">13:07</a>
jwb	we have dedicated boxen to do composes on?	<a href="#t13:07" class="time">13:07</a>
jwb	f13, that's what i dislike	<a href="#t13:08" class="time">13:08</a>
f13	jwb: right now?  no.	<a href="#t13:08" class="time">13:08</a>
jwb	ppc builds already take too long for some reason	<a href="#t13:08" class="time">13:08</a>
f13	jwb: we have machines in my desk that are used for various development stuff that I use to do the composes.	<a href="#t13:08" class="time">13:08</a>
jeremy	longer term, I think we want "compose" type tasks to be done in koji much like runroots in brew	<a href="#t13:08" class="time">13:08</a>
jwb	running composes on a koji ppc builder will just make them take longer	<a href="#t13:08" class="time">13:08</a>
f13	jwb: but we have no rackspace for a dedicated box that will just sit there for large amounts of time.	<a href="#t13:09" class="time">13:09</a>
jwb	f13, right and you're now getting to my other point	<a href="#t13:09" class="time">13:09</a>
f13	in the $FUTURE when we can ship the unused blade center to phx or elsewhere we can perhaps dedicate one of the blades to this task	<a href="#t13:09" class="time">13:09</a>
jwb	f13, secondary arches...	<a href="#t13:09" class="time">13:09</a>
f13	jwb: secondary arches are responsible for doing and hosting their own composes.	<a href="#t13:10" class="time">13:10</a>
jwb	right.  which is why composing in the colo is bad imho	<a href="#t13:10" class="time">13:10</a>
jeremy	jwb: don't think of it as "composing in the colo"	<a href="#t13:10" class="time">13:10</a>
jeremy	jwb: think of it as "composing with the same sets of machines used for building"	<a href="#t13:10" class="time">13:10</a>
notting	jeremy: if we start doing multilib differently, that becomes a whole lot easier	<a href="#t13:10" class="time">13:10</a>
jwb	jeremy, fine.  my point is, we need to be careful that doing that doesn't hard code "colo-isms"	<a href="#t13:11" class="time">13:11</a>
* f13 is a bit confused why secondary arches matter here. We're talking about primary content for our primary arches.		<a href="#t13:11" class="time">13:11</a>
warren	notting, will we do multilib differently before test1?	<a href="#t13:11" class="time">13:11</a>
jwb	f13, because it's important?	<a href="#t13:11" class="time">13:11</a>
notting	warren: i suppose it needs to be on the feature list	<a href="#t13:11" class="time">13:11</a>
jeremy	jwb: I think that you're getting hung up on the term "colo".  the bigger thing here is that doing the tree composes on the box in jesse's cube and the livecds on the box in my cube is wrong, wrong, wrong, wrong, wrong :)	<a href="#t13:12" class="time">13:12</a>
f13	jwb: wait, so you're worried about coloisms leaking into say pungi?	<a href="#t13:12" class="time">13:12</a>
jwb	f13, pungi, koji, etc.  yes.  2ndary arches have to compose with the exact same tools as primary arches, so i just want to make sure those tools continue to work outside the colo	<a href="#t13:12" class="time">13:12</a>
f13	yes, that is a continual goal, although a bit 'fun' wrt things like mash	<a href="#t13:13" class="time">13:13</a>
f13	but mostly it's about config file changes, not code level changes.	<a href="#t13:13" class="time">13:13</a>
jwb	fair enough.  not a show stopper, just something i want us to keep in mind	<a href="#t13:13" class="time">13:13</a>
jwb	sorry, likely making entirely too much noise about it	<a href="#t13:13" class="time">13:13</a>
f13	jwb: indeed.  For the first phase of this, all we're really doing is composing the same way, just on a different machine.	<a href="#t13:13" class="time">13:13</a>
notting	mash doesn't work outside of local koji access, for example	<a href="#t13:14" class="time">13:14</a>
f13	instead of a machine on my desk, i'ts a machine on phx	<a href="#t13:14" class="time">13:14</a>
jwb	yeah, and that's all fine.  though i still don't like the "steal ppc koji builder" part.  but it's minor i suppose	<a href="#t13:14" class="time">13:14</a>
f13	notting: would work with /a/ koji though right?  Not necessarily /our/ koji?	<a href="#t13:14" class="time">13:14</a>
notting	f13: right..	<a href="#t13:14" class="time">13:14</a>
notting	f13: technically, you could mount the koji storage over a wan. be a tad painful.	<a href="#t13:14" class="time">13:14</a>
f13	jwb: yes.  And we're mostly talking about the composes for test/final releases, not doing 20 composes a day every day.	<a href="#t13:15" class="time">13:15</a>
f13	jwb: I'll still be doing the random test compose stuff on my workstations	<a href="#t13:15" class="time">13:15</a>
f13	(which also ensures that it still works this way)	<a href="#t13:15" class="time">13:15</a>
f13	what's going to be difficult is defining who can do what parts of composes, and where.	<a href="#t13:16" class="time">13:16</a>
f13	especially since it's a rather disconnected process	<a href="#t13:17" class="time">13:17</a>
f13	each step is done differently potentially in different places with different tools	<a href="#t13:17" class="time">13:17</a>
warren	f13, we have the current pain due to a lack of dedicated compose hardware in PHX?	<a href="#t13:18" class="time">13:18</a>
warren	Or due to design issues?	<a href="#t13:18" class="time">13:18</a>
f13	which current pain?	<a href="#t13:18" class="time">13:18</a>
warren	err	<a href="#t13:19" class="time">13:19</a>
rdieter	pain = uneasiness of depending solely on f13 and the machine on his desk? or is there more to it than that?	<a href="#t13:20" class="time">13:20</a>
wwoods	the problem is high turnaround time (and reduced reproducability) on composes, because it's being done on a machine under f13's desk	<a href="#t13:20" class="time">13:20</a>
wwoods	yes, unease as well, I suppose	<a href="#t13:20" class="time">13:20</a>
f13	warren: we have it for multiple reasons.  Some of it is design yes, pungi was designed to be easily used on a user's workstation, and as such its a tad bit harder to work directly into koji runroot like tasks	<a href="#t13:21" class="time">13:21</a>
f13	warren: and secondly that we just don't have dedicated boxes in the colo to do things	<a href="#t13:21" class="time">13:21</a>
f13	but to have dedicated boxes, we need to know what we need so that mmcgrath can allocate it for us	<a href="#t13:22" class="time">13:22</a>
f13	which for right now, we need a place to run sign_unsigned (until we get a signing server), a place to run mash from, a place to run pungi from, and a place to drop the results on the netapp taking advantage of hardlinks.	<a href="#t13:24" class="time">13:24</a>
f13	we currently have app5 (which also runs bodhi) that we can use for signing, mashing, and pungi of i386/x86_64.  We just don't have a place to pungi ppc, nor a good method of getting the results up to the netapp	<a href="#t13:24" class="time">13:24</a>
wwoods	can we say "this is a good idea" and wait until we have hardware to actually make changes?	<a href="#t13:25" class="time">13:25</a>
wwoods	although now that I mention it, "wait until we have hardware to actually make changes" got us in an assload of trouble for F7	<a href="#t13:25" class="time">13:25</a>
f13	yeah	<a href="#t13:25" class="time">13:25</a>
f13	warren: we can work around the ppc by making temporary use of koji builders at compose time	<a href="#t13:26" class="time">13:26</a>
f13	(which really isn't all that far off from when we make koji do the pungi parts itself)	<a href="#t13:26" class="time">13:26</a>
jwb	are we starting to repeat ourselves?	<a href="#t13:27" class="time">13:27</a>
f13	I think so.	<a href="#t13:27" class="time">13:27</a>
jeremy	jwb: yes	<a href="#t13:27" class="time">13:27</a>
f13	basically we have an RFR, I'm going to work with mmcgrath to get that fine tuned, and start attempting to do some composes in the colo so we get a better idea of what's needed.	<a href="#t13:27" class="time">13:27</a>
f13	I may end up having to write some of pungi to work better when composing directly to nfs	<a href="#t13:27" class="time">13:27</a>
jeremy	f13: we also should make sure that livecd composes are happy with what we do	<a href="#t13:28" class="time">13:28</a>
f13	or to be smarter with gathering packages from a file:/// url and making hardlinks when possible.	<a href="#t13:28" class="time">13:28</a>
jeremy	because I'd like to off-load some of the livecd composing as a first step :-)	<a href="#t13:28" class="time">13:28</a>
f13	jeremy: indeed.  Does LiveCD composing work in mock yet?	<a href="#t13:28" class="time">13:28</a>
* warren sees "composting" whenever he sees "composing"		<a href="#t13:28" class="time">13:28</a>
jeremy	f13: I haven't tried in a while...  if I can actually get a livecd to work at all today, then I'll give it a try	<a href="#t13:28" class="time">13:28</a>
wwoods	so: 1. composing in colo = good.  2. start doing it now.  3. get hardware ASAP so we can continue doing so without messing up current infrastructure.	<a href="#t13:29" class="time">13:29</a>
f13	k	<a href="#t13:29" class="time">13:29</a>
f13	jeremy: that would be an important first step to being able to do it in the colo	<a href="#t13:29" class="time">13:29</a>
wwoods	4. make sure we keep tools / configs public so it's reproducable elsewhere. and of course:  5. profit	<a href="#t13:29" class="time">13:29</a>
f13	warren: heh sounds good.	<a href="#t13:29" class="time">13:29</a>
jeremy	f13: indeed.  although that might require some ... interesting questions for mock + selinux	<a href="#t13:29" class="time">13:29</a>
f13	poelcat: Decision: Releng will continue investigations into composing trees within the colo; potentially composing Test1 there.	<a href="#t13:30" class="time">13:30</a>
f13	jeremy: well, don't we already do some interesting things w/ selinux for buildinstall?	<a href="#t13:30" class="time">13:30</a>
jeremy	f13: buildinstall is an entirely different beast	<a href="#t13:30" class="time">13:30</a>
f13	yes, but there is some selinux context there	<a href="#t13:31" class="time">13:31</a>
jeremy	f13: the images aren't labeled at all, though	<a href="#t13:31" class="time">13:31</a>
jeremy	f13: we just generate a policy module in buildinstall	<a href="#t13:31" class="time">13:31</a>
f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Fedora Signing Server - JesseKeating		<a href="#t13:31" class="time">13:31</a>
jeremy	f13: which is far far far different from wanting to build an image with contexts	<a href="#t13:31" class="time">13:31</a>
f13	jeremy: nod.	<a href="#t13:31" class="time">13:31</a>
f13	I haven't really gotten any feedback since last meeting (some gave it before the meeting) so I'm going to move this into the ReleaseEngineering/ space as it's new home	<a href="#t13:32" class="time">13:32</a>
f13	I also haven't spent much time looking at code either.	<a href="#t13:32" class="time">13:32</a>
f13	but that's all I ahve to say about this subject.  Anybody else ?	<a href="#t13:32" class="time">13:32</a>
f13	Guess not.	<a href="#t13:34" class="time">13:34</a>
f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Open Discussion		<a href="#t13:34" class="time">13:34</a>
f13	anything else anybody would like to discuss?	<a href="#t13:34" class="time">13:34</a>
jwb	have we documented the "you need to email rel-eng to get packages in the buildroot" thing?	<a href="#t13:34" class="time">13:34</a>
jwb	i went looking the other day and couldn't find it	<a href="#t13:34" class="time">13:34</a>
jeremy	jwb: sounds like something worth adding to the update doc.  feel free to add :)	<a href="#t13:34" class="time">13:34</a>
notting	we were going to switch that based on bodhi changes	<a href="#t13:35" class="time">13:35</a>
jwb	notting, which bodhi changes?	<a href="#t13:35" class="time">13:35</a>
notting	some sort of alert for packages built against unpushed things	<a href="#t13:35" class="time">13:35</a>
f13	but until we switch it, we need it documented somewhere.	<a href="#t13:35" class="time">13:35</a>
f13	jwb: once bodhi can prevent an update from going out that was built against an unreleased update, we can make the buildroot selfupdate.	<a href="#t13:36" class="time">13:36</a>
jeremy	f13: note -- that may need to be overridable	<a href="#t13:36" class="time">13:36</a>
jeremy	f13: otherwise, I have a _great_ way to keep security updates from being able to go out :)	<a href="#t13:36" class="time">13:36</a>
f13	jeremy: yes, we do need to be able to override it.	<a href="#t13:37" class="time">13:37</a>
jwb	f13, jeremy: ok well until that gets done, then i'll update the update doc	<a href="#t13:37" class="time">13:37</a>
f13	jwb: muchas gracias.	<a href="#t13:37" class="time">13:37</a>
f13	anything else?	<a href="#t13:38" class="time">13:38</a>
f13	alright, calling the meeting.	<a href="#t13:39" class="time">13:39</a>
spot	hello? meeting? collect call from f13.	<a href="#t13:39" class="time">13:39</a>
f13 changed the topic of #fedora-meeting to: <a href="http://fedoraproject.org/wiki/Communicate/FedoraMeetingChannel">http://fedoraproject.org/wiki/Communicate/FedoraMeetingChannel</a> -- Meetings often get logged -- see schedule in the wiki for next meeting		<a href="#t13:40" class="time">13:40</a>

Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!