From FedoraProject

Jump to: navigation, search


Fedora Release Engineering Meeting :: Monday 2007-07-09

Composes in the colo

  • downsides are that it will:

1. probably take longer to complete the compose 1. certainly take longer to get the compose into the hands of our testing lab in Raleigh 1. take longer to get onto the machines in Westford

  • other benefits most likely outweigh these issues.
  • f13 is single point of failure for doing composes--everything is dependent on him

Decision: Releng will continue investigations into composing trees within the colo; potentially composing Test1 there.

Fedora Signing Server

  • f13 has not received any feedback since last meeting (some gave it before the meeting)--moving proposal to the ReleaseEngineering/ space as it's new home

Open Discussion

  • jwb--have we documented the "you need to email rel-eng to get packages in the buildroot" thing?
  • jeremy--sounds like something worth adding to the update doc. feel free to add :)
  • notting--we were going to switch that based on bodhi changes with an alert for packages built against unpushed things
  • f13--but until we switch it, we need it documented somewhere. Once bodhi can prevent an update from going out that was built against an unreleased update, we can make the buildroot selfupdate.
  • jeremy--note that may need to be overridable, otherwise, I have a _great_ way to keep security updates from being able to go out :)
  • f13--we do need to be able to override it.

Decision: No decision made

IRC Transcript

Idea.png f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Composes in the colo - JesseKeating<a href="#t12:58" class="time">12:58</a>
* spot warns that anyone caught near my chicago apt will be forced to help pack<a href="#t12:58" class="time">12:58</a>
f13rdieter: jwb: wwoods: ping<a href="#t12:59" class="time">12:59</a>
jwbhere<a href="#t13:00" class="time">13:00</a>
rdieterhere (for awhille anyway)<a href="#t13:00" class="time">13:00</a>
f13ok, so lets talk a bit about doing composes in the colo.<a href="#t13:00" class="time">13:00</a>
f13as opposed to my desk<a href="#t13:00" class="time">13:00</a>
warrenf13, I like relying upon services under our desks.<a href="#t13:01" class="time">13:01</a>
jwbew<a href="#t13:01" class="time">13:01</a>
f13this has some obvious benifits, others can get to the compose bits a lot quicker (or at all), theoretically somebody other than me can do a compose, etc...<a href="#t13:01" class="time">13:01</a>
f13poelcat: ping;<a href="#t13:01" class="time">13:01</a>
f13The downsides are that A) it will probably take longer to complete the compose, B) it will certainly take longer to get the compose into the hands of our testing lab in Raleigh, C) it'll take longer to get onto the machines in Westford<a href="#t13:02" class="time">13:02</a>
* wwoods here<a href="#t13:02" class="time">13:02</a>
jwbf13, why can't composes be done on your machine and then uploaded somewhere?<a href="#t13:02" class="time">13:02</a>
f13but the other benifits most likely outweigh these issues.<a href="#t13:02" class="time">13:02</a>
f13jwb: upload slow, and I'm less likely to upload right away and I end up doing lots of testing locally before incurring the cost of an upload<a href="#t13:03" class="time">13:03</a>
f13jwb: I may end up doing composes in two locations though, at the same time.<a href="#t13:03" class="time">13:03</a>
wwoodsjwb: it also means that nobody but jesse can do the composes, unless he syncs his config etc.<a href="#t13:03" class="time">13:03</a>
f13at least until the trees stop changing as much and then rsync will just work.<a href="#t13:03" class="time">13:03</a>
jwbyou have a script, yes?<a href="#t13:03" class="time">13:03</a>
f13not exactly<a href="#t13:03" class="time">13:03</a>
jwbwrite a script and have it sync the compose and config at the end<a href="#t13:04" class="time">13:04</a>
jwbin the background<a href="#t13:04" class="time">13:04</a>
f13welll<a href="#t13:04" class="time">13:04</a>
f13given that then everybody is completely dependant upon me to produce things I think that's a bad route to go down.<a href="#t13:05" class="time">13:05</a>
jwbdepends i guess<a href="#t13:05" class="time">13:05</a>
jwbif <upload location> were a username based location, then others could do composes and upload them as well, no?<a href="#t13:05" class="time">13:05</a>
poelcatf13: here<a href="#t13:05" class="time">13:05</a>
jeremyjwb: but why should "I'm a composer" be dependent on "I have tons of bandwidth"<a href="#t13:06" class="time">13:06</a>
jeremy(and tons of upload bandwidth at that)<a href="#t13:06" class="time">13:06</a>
jwbjeremy, shouldn't.  this can all be done _in addition to_ composing in the colo<a href="#t13:06" class="time">13:06</a>
jeremyjwb: plus, it places a lot of trust in the "how is the machine set up and the composes being done"<a href="#t13:06" class="time">13:06</a>
f13The process to compose a tree for releases is as such:  tag newly needed build -> sign unsigned packages -> mash a tag -> adjust pungi + mock configs to point to new mash output -> update mock chroot -> run pungi -> make tree accessable.<a href="#t13:06" class="time">13:06</a>
rdieterf13: you're right, of course, the pros outweigh the cons.  How much work/pain is involved to implement this?<a href="#t13:06" class="time">13:06</a>
f13rdieter: probably not a whole lot of work.  We already have one x86_64 box that we use for various compose like things that can be used to compose i386/x86_64 (once it's given more disk space/ram).<a href="#t13:07" class="time">13:07</a>
jwbso maybe i'm misunderstanding things<a href="#t13:07" class="time">13:07</a>
f13finding a ppc box may be more difficult, and in teh short term we may just have to (ab)use a koji builder to od the compose.<a href="#t13:07" class="time">13:07</a>
jwbwe have dedicated boxen to do composes on?<a href="#t13:07" class="time">13:07</a>
jwbf13, that's what i dislike<a href="#t13:08" class="time">13:08</a>
f13jwb: right now?  no.<a href="#t13:08" class="time">13:08</a>
jwbppc builds already take too long for some reason<a href="#t13:08" class="time">13:08</a>
f13jwb: we have machines in my desk that are used for various development stuff that I use to do the composes.<a href="#t13:08" class="time">13:08</a>
jeremylonger term, I think we want "compose" type tasks to be done in koji much like runroots in brew<a href="#t13:08" class="time">13:08</a>
jwbrunning composes on a koji ppc builder will just make them take longer<a href="#t13:08" class="time">13:08</a>
f13jwb: but we have no rackspace for a dedicated box that will just sit there for large amounts of time.<a href="#t13:09" class="time">13:09</a>
jwbf13, right and you're now getting to my other point<a href="#t13:09" class="time">13:09</a>
f13in the $FUTURE when we can ship the unused blade center to phx or elsewhere we can perhaps dedicate one of the blades to this task<a href="#t13:09" class="time">13:09</a>
jwbf13, secondary arches...<a href="#t13:09" class="time">13:09</a>
f13jwb: secondary arches are responsible for doing and hosting their own composes.<a href="#t13:10" class="time">13:10</a>
jwbright.  which is why composing in the colo is bad imho<a href="#t13:10" class="time">13:10</a>
jeremyjwb: don't think of it as "composing in the colo"<a href="#t13:10" class="time">13:10</a>
jeremyjwb: think of it as "composing with the same sets of machines used for building"<a href="#t13:10" class="time">13:10</a>
nottingjeremy: if we start doing multilib differently, that becomes a whole lot easier<a href="#t13:10" class="time">13:10</a>
jwbjeremy, fine.  my point is, we need to be careful that doing that doesn't hard code "colo-isms"<a href="#t13:11" class="time">13:11</a>
* f13 is a bit confused why secondary arches matter here. We're talking about primary content for our primary arches.<a href="#t13:11" class="time">13:11</a>
warrennotting, will we do multilib differently before test1?<a href="#t13:11" class="time">13:11</a>
jwbf13, because it's important?<a href="#t13:11" class="time">13:11</a>
nottingwarren: i suppose it needs to be on the feature list<a href="#t13:11" class="time">13:11</a>
jeremyjwb: I think that you're getting hung up on the term "colo".  the bigger thing here is that doing the tree composes on the box in jesse's cube and the livecds on the box in my cube is wrong, wrong, wrong, wrong, wrong :)<a href="#t13:12" class="time">13:12</a>
f13jwb: wait, so you're worried about coloisms leaking into say pungi?<a href="#t13:12" class="time">13:12</a>
jwbf13, pungi, koji, etc.  yes.  2ndary arches have to compose with the exact same tools as primary arches, so i just want to make sure those tools continue to work outside the colo<a href="#t13:12" class="time">13:12</a>
f13yes, that is a continual goal, although a bit 'fun' wrt things like mash<a href="#t13:13" class="time">13:13</a>
f13but mostly it's about config file changes, not code level changes.<a href="#t13:13" class="time">13:13</a>
jwbfair enough.  not a show stopper, just something i want us to keep in mind<a href="#t13:13" class="time">13:13</a>
jwbsorry, likely making entirely too much noise about it<a href="#t13:13" class="time">13:13</a>
f13jwb: indeed.  For the first phase of this, all we're really doing is composing the same way, just on a different machine.<a href="#t13:13" class="time">13:13</a>
nottingmash doesn't work outside of local koji access, for example<a href="#t13:14" class="time">13:14</a>
f13instead of a machine on my desk, i'ts a machine on phx<a href="#t13:14" class="time">13:14</a>
jwbyeah, and that's all fine.  though i still don't like the "steal ppc koji builder" part.  but it's minor i suppose<a href="#t13:14" class="time">13:14</a>
f13notting: would work with /a/ koji though right?  Not necessarily /our/ koji?<a href="#t13:14" class="time">13:14</a>
nottingf13: right..<a href="#t13:14" class="time">13:14</a>
nottingf13: technically, you could mount the koji storage over a wan. be a tad painful.<a href="#t13:14" class="time">13:14</a>
f13jwb: yes.  And we're mostly talking about the composes for test/final releases, not doing 20 composes a day every day.<a href="#t13:15" class="time">13:15</a>
f13jwb: I'll still be doing the random test compose stuff on my workstations<a href="#t13:15" class="time">13:15</a>
f13(which also ensures that it still works this way)<a href="#t13:15" class="time">13:15</a>
f13what's going to be difficult is defining who can do what parts of composes, and where.<a href="#t13:16" class="time">13:16</a>
f13especially since it's a rather disconnected process<a href="#t13:17" class="time">13:17</a>
f13each step is done differently potentially in different places with different tools<a href="#t13:17" class="time">13:17</a>
warrenf13, we have the current pain due to a lack of dedicated compose hardware in PHX?<a href="#t13:18" class="time">13:18</a>
warrenOr due to design issues?<a href="#t13:18" class="time">13:18</a>
f13which current pain?<a href="#t13:18" class="time">13:18</a>
warrenerr<a href="#t13:19" class="time">13:19</a>
rdieterpain = uneasiness of depending solely on f13 and the machine on his desk? or is there more to it than that?<a href="#t13:20" class="time">13:20</a>
wwoodsthe problem is high turnaround time (and reduced reproducability) on composes, because it's being done on a machine under f13's desk<a href="#t13:20" class="time">13:20</a>
wwoodsyes, unease as well, I suppose<a href="#t13:20" class="time">13:20</a>
f13warren: we have it for multiple reasons.  Some of it is design yes, pungi was designed to be easily used on a user's workstation, and as such its a tad bit harder to work directly into koji runroot like tasks<a href="#t13:21" class="time">13:21</a>
f13warren: and secondly that we just don't have dedicated boxes in the colo to do things<a href="#t13:21" class="time">13:21</a>
f13but to have dedicated boxes, we need to know what we need so that mmcgrath can allocate it for us<a href="#t13:22" class="time">13:22</a>
f13which for right now, we need a place to run sign_unsigned (until we get a signing server), a place to run mash from, a place to run pungi from, and a place to drop the results on the netapp taking advantage of hardlinks.<a href="#t13:24" class="time">13:24</a>
f13we currently have app5 (which also runs bodhi) that we can use for signing, mashing, and pungi of i386/x86_64.  We just don't have a place to pungi ppc, nor a good method of getting the results up to the netapp<a href="#t13:24" class="time">13:24</a>
wwoodscan we say "this is a good idea" and wait until we have hardware to actually make changes?<a href="#t13:25" class="time">13:25</a>
wwoodsalthough now that I mention it, "wait until we have hardware to actually make changes" got us in an assload of trouble for F7<a href="#t13:25" class="time">13:25</a>
f13yeah<a href="#t13:25" class="time">13:25</a>
f13warren: we can work around the ppc by making temporary use of koji builders at compose time<a href="#t13:26" class="time">13:26</a>
f13(which really isn't all that far off from when we make koji do the pungi parts itself)<a href="#t13:26" class="time">13:26</a>
jwbare we starting to repeat ourselves?<a href="#t13:27" class="time">13:27</a>
f13I think so.<a href="#t13:27" class="time">13:27</a>
jeremyjwb: yes<a href="#t13:27" class="time">13:27</a>
f13basically we have an RFR, I'm going to work with mmcgrath to get that fine tuned, and start attempting to do some composes in the colo so we get a better idea of what's needed.<a href="#t13:27" class="time">13:27</a>
f13I may end up having to write some of pungi to work better when composing directly to nfs<a href="#t13:27" class="time">13:27</a>
jeremyf13: we also should make sure that livecd composes are happy with what we do<a href="#t13:28" class="time">13:28</a>
f13or to be smarter with gathering packages from a file:/// url and making hardlinks when possible.<a href="#t13:28" class="time">13:28</a>
jeremybecause I'd like to off-load some of the livecd composing as a first step :-)<a href="#t13:28" class="time">13:28</a>
f13jeremy: indeed.  Does LiveCD composing work in mock yet?<a href="#t13:28" class="time">13:28</a>
* warren sees "composting" whenever he sees "composing"<a href="#t13:28" class="time">13:28</a>
jeremyf13: I haven't tried in a while...  if I can actually get a livecd to work at all today, then I'll give it a try<a href="#t13:28" class="time">13:28</a>
wwoodsso: 1. composing in colo = good.  2. start doing it now.  3. get hardware ASAP so we can continue doing so without messing up current infrastructure.<a href="#t13:29" class="time">13:29</a>
f13k<a href="#t13:29" class="time">13:29</a>
f13jeremy: that would be an important first step to being able to do it in the colo<a href="#t13:29" class="time">13:29</a>
wwoods4. make sure we keep tools / configs public so it's reproducable elsewhere. and of course:  5. profit<a href="#t13:29" class="time">13:29</a>
f13warren: heh sounds good.<a href="#t13:29" class="time">13:29</a>
jeremyf13: indeed.  although that might require some ... interesting questions for mock + selinux<a href="#t13:29" class="time">13:29</a>
f13poelcat: Decision: Releng will continue investigations into composing trees within the colo; potentially composing Test1 there.<a href="#t13:30" class="time">13:30</a>
f13jeremy: well, don't we already do some interesting things w/ selinux for buildinstall?<a href="#t13:30" class="time">13:30</a>
jeremyf13: buildinstall is an entirely different beast<a href="#t13:30" class="time">13:30</a>
f13yes, but there is some selinux context there<a href="#t13:31" class="time">13:31</a>
jeremyf13: the images aren't labeled at all, though<a href="#t13:31" class="time">13:31</a>
jeremyf13: we just generate a policy module in buildinstall<a href="#t13:31" class="time">13:31</a>
Idea.png f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Fedora Signing Server - JesseKeating<a href="#t13:31" class="time">13:31</a>
jeremyf13: which is far far far different from wanting to build an image with contexts<a href="#t13:31" class="time">13:31</a>
f13jeremy: nod.<a href="#t13:31" class="time">13:31</a>
f13I haven't really gotten any feedback since last meeting (some gave it before the meeting) so I'm going to move this into the ReleaseEngineering/ space as it's new home<a href="#t13:32" class="time">13:32</a>
f13I also haven't spent much time looking at code either.<a href="#t13:32" class="time">13:32</a>
f13but that's all I ahve to say about this subject.  Anybody else ?<a href="#t13:32" class="time">13:32</a>
f13Guess not.<a href="#t13:34" class="time">13:34</a>
Idea.png f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Open Discussion<a href="#t13:34" class="time">13:34</a>
f13anything else anybody would like to discuss?<a href="#t13:34" class="time">13:34</a>
jwbhave we documented the "you need to email rel-eng to get packages in the buildroot" thing?<a href="#t13:34" class="time">13:34</a>
jwbi went looking the other day and couldn't find it<a href="#t13:34" class="time">13:34</a>
jeremyjwb: sounds like something worth adding to the update doc.  feel free to add :)<a href="#t13:34" class="time">13:34</a>
nottingwe were going to switch that based on bodhi changes<a href="#t13:35" class="time">13:35</a>
jwbnotting, which bodhi changes?<a href="#t13:35" class="time">13:35</a>
nottingsome sort of alert for packages built against unpushed things<a href="#t13:35" class="time">13:35</a>
f13but until we switch it, we need it documented somewhere.<a href="#t13:35" class="time">13:35</a>
f13jwb: once bodhi can prevent an update from going out that was built against an unreleased update, we can make the buildroot selfupdate.<a href="#t13:36" class="time">13:36</a>
jeremyf13: note -- that may need to be overridable<a href="#t13:36" class="time">13:36</a>
jeremyf13: otherwise, I have a _great_ way to keep security updates from being able to go out :)<a href="#t13:36" class="time">13:36</a>
f13jeremy: yes, we do need to be able to override it.<a href="#t13:37" class="time">13:37</a>
jwbf13, jeremy: ok well until that gets done, then i'll update the update doc<a href="#t13:37" class="time">13:37</a>
f13jwb: muchas gracias.<a href="#t13:37" class="time">13:37</a>
f13anything else?<a href="#t13:38" class="time">13:38</a>
f13alright, calling the meeting.<a href="#t13:39" class="time">13:39</a>
spothello? meeting? collect call from f13.<a href="#t13:39" class="time">13:39</a>
Idea.png f13 changed the topic of #fedora-meeting to: <a href=""></a> -- Meetings often get logged -- see schedule in the wiki for next meeting<a href="#t13:40" class="time">13:40</a>

Generated by 2.3 by Marius Gedminas - find it at!