From Fedora Project Wiki

spot skvidal: yep 11:01
skvidal okay 11:01
skvidal abadger1999: you around? 11:01
abadger1999 Yep 11:01
skvidal abadger1999: you wanna start? you have the fancy diagram :) 11:01
abadger1999 11:02
abadger1999 Okay, that's an overview of what we think the whole thing ends up looking like. 11:02
abadger1999 The black path is the main path for building packages 11:03
abadger1999 The orange path is the main path for managing repositories 11:03
abadger1999 A lot of stuff has to get built to make this all work. 11:04
spot wow, chromium and firefox really render that badly 11:04
* spot opens it in inkscape 11:04
skvidal spot: yah - I had to open it in inkscape to see it at all 11:05
*** stickster is now known as stickster_mtg. 11:05
tremble Nice to know it's not just my browser mangled it. 11:05
skvidal as a summary for any folks just coming into this - we're talking about arbitrary package builds 11:06
<-- mcepl has left this server (Read error: Connection reset by peer). 11:06
skvidal from arbitrary sources, using arbitrary repos for arbitrary build dep providing 11:06
skvidal whereas koji can currently handle arbitrary pkgs being built 11:07
skvidal that's a scratch build 11:07
tremble So we could for example build a batch of builds for EPEL and then makke a single push into testing. 11:07
<-- JSchmitt has left this server (Remote host closed the connection). 11:07
tremble ? 11:07
skvidal it cannot do scratch chain build and pull in arbitrary buildeps 11:07
skvidal tremble: and no 0- the goal of this is not for official builds 11:08
skvidal official builds will ALWAYS have to go through koji 11:08
<-- jnettlet has left this server (Ping timeout: 252 seconds). 11:08
skvidal if only for the audit-trail that koji provides 11:08
spot (not withstanding that we will want some measure of auditing in this process as well, but still) 11:08
skvidal auditing of what? 11:09
spot well, just logging really 11:09
skvidal the ppas/build services are the remote equivalent of running mock on your own machine 11:09
spot history of who did what. 11:09
abadger1999 spot: as in build.log? or more than that? 11:09
skvidal sure 11:09
skvidal but that's not auditing 11:10
abadger1999 \okay. 11:10
skvidal in the sense that koji uses it 11:10
spot skvidal: agreed 11:10
skvidal logging who did what - that's a web server log 11:10
skvidal logging what pkgs were built, with what pkgs in the chroot and all the tags of those pkgs 11:10
skvidal that's a whole different ballpark 11:10
skvidal hell, it doesn't even look like the same GAME 11:10
skvidal the gist of what a ppa/buildservice does is allow a user to build a package not using their own cpu time 11:11
skvidal but using the cputime of the service 11:11
skvidal cputime (and bandwidth) 11:11
spot yep. 11:11
abadger1999 and build a repository that end users can use to easily grab the packages. 11:12
skvidal so the goal here is just to build the 'build and host this pkg for me' service 11:12
skvidal we've been looking around at other distros systems for this 11:13
skvidal ubuntu's ppa system and the opensuse build service in particular 11:13
skvidal and in both cases they build the pkgs inside a vm protecting the host system from the building process being evil or just bad 11:14
skvidal the interface criteria are more or less the same - you define your repo requirements 11:14
skvidal you toss a pkg at the system 11:14
skvidal it gives you back a url as to where your finished results are (or logs of why it failed) 11:14
skvidal wash, rinse, repeat 11:15
skvidal the pieces we do NOT have are: 11:15
skvidal 1. building inside a vm 11:15
skvidal 2. the web/xmlrpc backend for submitting/creating the repo and srpms 11:15
skvidal 3. the interface/storage location for the resulting pkgs 11:16
skvidal mmcgrath has been nice enough to loan us a big system for testing a bunch of these things out 11:16
skvidal and I've been talking with clark|w about mock and building inside a vm and what that entails 11:16
skvidal and with rwmjones about guestfs's capabilities 11:16
<-- inode0 has left this server (Quit: Leaving.). 11:16
skvidal and reading the opensuse build code 11:17
skvidal and step 1 is not insurmountable - but doing it 'pretty' would take a while 11:17
skvidal making it functional - might be doable in a much shorter time frame 11:17
skvidal for step 1 11:18
skvidal abadger1999: what're thoughts on the xmlrpc/whatever for requesting repos and submitting srpms? 11:18
abadger1999 i've talked with dmalcolm about the tak scheduler he has inside of rpmgrok. 11:19
abadger1999 Since rpmgrok is a TG1 app with an SQLAlchemy model, it should be pretty easy to adapt. 11:19
abadger1999 he says that koji has some things that are nice, though -- like dependencies among tasks. 11:20
abadger1999 I don't think it'll be too hard to pull just those features in. 11:20
abadger1999 My thought is that we'd want the web interface to add jobs to the db. 11:20
abadger1999 Then the task scheduler itself pulls the jobs out and sends them to task runners -- this is very similar to how koji works. 11:21
abadger1999 We'll use standard TG2 tech like the json API you get for free with any controller methods. 11:21
abadger1999 I'm kind of leaning towards seeing if we can use func to communicate between the task scheduler and task runners. 11:22
mbonnet abadger1999: not really, builders *claim* tasks in koji, they are never assigned tasks 11:22
abadger1999 mbonnet: That's true. push vs pull was one ofthe things I was going to ask you about. 11:22
abadger1999 mbonnet: What are the advantages of builders pulling jobs rather than koji hub pushing them? 11:22
skvidal abadger1999: you don't have to monitor which builders are up 11:23
mbonnet abadger1999: it means the server can be essentially passive 11:23
mbonnet and builders only need outbound connections, so they can be firewalled/natted 11:24
--> bpepple|lt has joined this channel (~bpepple| 11:24
abadger1999 Okay, and what are the downsides? 11:26
mbonnet It can make load-balancing tougher, since you don't necessarily have a global view of who is doing what 11:27
mbonnet It really depends on the design of the hub/server part 11:27
--> spoleeba has joined this channel (~one@fedora/Jef). 11:28
mbonnet For Koji, the hub is just Apache, so the builders *have* to be the ones initiating the connections 11:28
abadger1999 <nod> In our setup, the scheduler would be more active than that. 11:28
mbonnet abadger1999: But the passive hub approach means the hub is *only* a server and the builders are *only* clients. If the builders accept connections, they are both clients and servers, and the hub is both a server and client, and I think that complicates the code. 11:29
dgilmore abadger1999: plague supported both pushing and pulling 11:30
skvidal okay 11:30
skvidal before we get too far here 11:31
abadger1999 mbonnet: True.. although if we do reuse func then we are working within the architecture that it provides. 11:31
mbonnet sure 11:31
skvidal are there any more general concerns? 11:31
abadger1999 Should we be doing this at all? 11:31
mbonnet If I were going to redesign Koji right now, I would probably make all communication amqp-based, which allows bidirectional communication. 11:31
skvidal abadger1999: ? - the issue of the many many many competing and confusing "microrepos" 11:31
abadger1999 Yeah eaxactly. 11:32
mbonnet I certainly think it has the potential the make support dramatically more difficult 11:32
abadger1999 mbonnet: That's a good thought. I'll look into doing this via amqp as a possibility. 11:33
skvidal and potentially to make users' lives more difficult b/c they might not know what is a fedora pkg 11:33
skvidal and what is not 11:33
spot well, i think if we were to something like institute a repotag to mark these packages it will aid in that 11:33
skvidal spot: b/c the repotags have helped keep people from confusing rpmfusion w/fedora? 11:33
spot skvidal: its not so much people that we can really manage in this aspect 11:34
spot it is bug reporters when there is overlap 11:34
spot if a package is clearly marked in its n-v-r as being built in a personal repo 11:34
spot that at least makes bug resolution simpler. :) 11:34
mbonnet How would this affect efforts like abrt? 11:34
skvidal spot: marked in its nvr? 11:35
abadger1999 adamw: You around by chance? 11:35
spot skvidal: think forced disttag 11:35
skvidal spot: yum is storing where a pkg is installed FROM 11:35
skvidal ie the repo it was installed from 11:35
skvidal in the yumdb - which is available to every pkg installed 11:35
skvidal so we can get the repo 11:35
spot skvidal: that is an additional source of info, but not likely to make it to bugzilla 11:35
adamw abadger1999: yep 11:35
abadger1999 adamw: We're talking about actually implementing something like kopers (ppas, etc). 11:36
spot mbonnet: we have the problem now in abrt where it flags on third-party stuff not in fedora 11:36
adamw yeah, just reading a bit back 11:36
adamw personal repo packages should *definitely* have a different dist tag from fedora packages 11:36
adamw ideally, each repo would have its own dist tag 11:36
abadger1999 adamw: The immediate question was about abrt... but I know you had thoughts on the general support issues as well. 11:36
mbonnet spot: right, but would abrt be able to get debuginfo/generate backtraces for kopers packages? 11:36
spot mbonnet: abrt generally discards reports for non-Fedora stuff, except in cases where it gets confused and thinks it is a crash in an interpreter 11:36
spot mbonnet: i'd say no, nor do we want it to 11:37
mbonnet spot: agreed. How does abrt tell if something is a "Fedora" pkg or not? 11:37
spot mbonnet: good question, would have to look at the source 11:38
abadger1999 Getting to nitty gritty, we'd probably want to force both disttag and repotag. => .fc12.apb 11:38
spot abadger1999: agreed 11:38
tibbs|h Note that we set Packager and Vendor on our packages to "Fedora Project". 11:38
adamw yeah, that works. as long as it's identifiable in the nvr at a glance. 11:38
skvidal tibbs|h: and that would have to change 11:39
skvidal for these builds 11:39
tibbs|h Yep. 11:39
wwoods what about the gpg signature(s)? 11:39
abadger1999 wwoods: Ubuntu creates a separate key per repo and signs the packages when built 11:40
skvidal abadger1999: any pkg that is submitted 11:40
abadger1999 Yes 11:40
spot abadger1999: that seems like a reasonable approach. 11:40
wwoods I assume we wouldn't be signing this stuff with the Fedora Project keys. Right? Hopefully ABRT is checking the file checksum / package key ID to see if a given file is actually from Fedora. 11:40
skvidal wwoods: so it just ensures that is the pkg that came out of the build process - not anything about the goodness of it 11:40
spot wwoods: no. not with Fedora key. 11:40
dgilmore abadger1999: we could probably dynamically set it to .fc12.<submitter> 11:41
wwoods or rather, we wouldn't be signing with the "Yes This Is An Official Fedora 12 Package" key. Not to suggest they should be totally unsigned. but yeah. 11:41
spot dgilmore: eh, i'd prefer consistency here so that everyone knows at a glance that it is both not-Fedora and comes from the personal repo infrastructure 11:41
skvidal dgilmore: think about multiple submitters per ppa 11:41
abadger1999 dgilmore: We could... but that probably makes support harder... Instead of asking "Do you apb in the release" you have to ask do you have some odd characters after the .fc12. ? 11:41
dgilmore spot: its easier to have it static 11:42
adamw spot: viz my old blog post, i do think it would be useful to identify packages which come from *different* personal repos 11:42
adamw spot: so you can see where you may have a problem because you have an app from one person's repo, but it's using a lib from another's... 11:42
skvidal adamw: that's what we said 11:42
spot adamw: and yum will serve that purpose. 11:42
* nirik wonders about two repos each having zlib-1.0-1.fc12.ppa 11:42
skvidal getting out of the weeds a bit 11:42
skvidal do the benefits to fedora and our users, outweigh the debugging/bugreport/knowing what is what problems that this could create? 11:43
skvidal in short - what do we get out of a service like this? 11:43
abadger1999 here's a list of benefits and drawbacks I made up late last night: 11:43
spot abadger1999: sums it up pretty well. 11:44
--> bochecha has joined this channel (~bochecha@fedora/bochecha). 11:44
spot i would note that the existence of PPAs is commonly cited as a main reason that developers/packagers prefer to work in Ubuntu 11:45
skvidal spot: where? 11:45
spot well, in various emails sent to me, lemme see if i can back that statement up with public comments 11:45
abadger1999 spot: I've also heard a lot of Ubuntu users cite "dependency hell" as the reason they left Red Hat linux/Fedora Core 11:45
tibbs|h I'm having trouble seeing how the hosting of software not appropriate for Fedora by Fedora could be on the helpful side. 11:45
mbonnet Are we worried about kopers redirecting the energies of packagers? So they start maintaining their kopers while leaving their Fedora packages to languish? 11:46
skvidal spot: when I was looking for notes and anecdotes on how ppas have succeeded I found a lot of references to dpkg --force to resolve dep issues 11:46
spot skvidal: heh. 11:46
adamw skvidal: that sounds much like what I was blogging about. 11:46
skvidal spot: I'm sadly not kiding 11:46
skvidal kidding, even 11:46
abadger1999 tibbs|h: perhaps I was biased when I wrote that :-) But it is a good point ... lots of the benefits have their own subtle drawbacks. 11:46
abadger1999 mbonnet: I am. 11:46
skvidal now - for all intents and purposes this is happening now 11:46
skvidal people can upload stuff to 11:47
skvidal host a repo there 11:47
skvidal no problem 11:47
skvidal if they have their own cputime and bandwidth 11:47
* spot nods 11:47
skvidal they can build to their hearts content 11:47
--> biertie has joined this channel (~bert@fedora/biertie). 11:47
skvidal the difference is - by making it EASY does it become disruptive 11:47
skvidal s/difference/queston 11:47
skvidal sorry 11:47
spot skvidal: 11:47
pjones skvidal: this is one of the reasons I suggested a per-SIG basis - it makes a somewhat higher (but not too high) barrier to entry 11:47
adamw spot: that mail seems like a good example of bad use of PPAs, to me: 'we disagree with Ubuntu's update policy so we're going to circumvent it with a permanent, somewhat-competing repository' 11:48
spot pjones: right now, all it takes to be a SIG is to say "this SIG exists" 11:48
pjones spot: all it requires is ze vill to do eet, ya! 11:49
adamw do we really want to give third-party projects the power to say 'we don't want you to use Fedora's packages for our project, we want you to use ours from this sorta-official-but-not-really koper'? What happens if they all say that? 11:49
spot adamw: well, you say bad, others say "what my users want" 11:49
pjones adamw: I think that's one of the most compelling reasons to have such a thing - so we can have one update policy for the distro, but a user/SIG/whatever can have a different update policy for those that choose to be more risky 11:49
spot adamw: well, we can look at Ubuntu for example, there is some of this, but it is not overwhelming 11:49
adamw fair enough. just raising the question. 11:50
pjones So - yes, we really do want to give them that. that's the point. 11:50
mbonnet Do we have a set of use-cases for kopers? Are we really trying to address the update-policy issue with them? 11:50
spot my feeling from looking through the PPAs in use there is that most of them are either "stuff that isn't ready for Ubuntu" or "stuff that is much newer than Ubuntu, but is headed there" 11:50
skvidal mbonnet: the use cases described are in jesse's draft 11:50
abadger1999 pjones: It's both good and bad -- it gives more freedom to users/sigs... but it also makes whole distro integration and targetting harder.... 11:51
spot mbonnet: well, i think there are some use cases. my chromium packages, for one. kde-redhat for another. 11:51
mbonnet spot: isn't the "stuff that's too new" supposed to go into rawhide? 11:51
abadger1999 I'm a third party making some KDE software, do I target Fedora-13 or Fedora-13 + the PPA? 11:51
mbonnet spot: kde-redhat makes me sad 11:51
spot mbonnet: yeah, but what if i want it built for F-12? 11:51
adamw mbonnet: not if you don't know when it's going to be ready. rawhide isn't sid; right now, if you put something in rawhide, you're saying you expect it to be okay to go in F14 11:51
mbonnet adamw: "ready" is a vague term 11:52
mbonnet I guess I just worry about kopers fracturing the Fedora community, where people focus on their one area of interest and the main distro falls by the wayside 11:53
mbonnet but maybe that's paranoia 11:53
adamw i think it's a tough question to answer, it's a counterfactual 11:53
spot mbonnet: i'd rather worry about that if/when it happened. 11:53
adamw we can't know how people will use them until they exist 11:53
mbonnet spot: by then its too late 11:53
spot mbonnet: eh. 11:54
spot mbonnet: i disagree. 11:54
mbonnet spot: take away kopers after people have been using them and you'll have a revolt 11:54
--> JSchmitt has joined this channel ( 11:54
<-- JSchmitt has left this server (Changing host). 11:54
--> JSchmitt has joined this channel (~s4504kr@fedora/JSchmitt). 11:54
skvidal so I have a couple of suggestions 11:54
spot adamw: we do have some evidence on how this works in OpenSUSE and Ubuntu. 11:54
skvidal to step ourselves into things 11:54
adamw i do think if we do it, we should have clear use cases for how they're intended to be used, the system should be designed to encourage those use cases and discourage others, and the packages should be clearly identifiable. 11:54
adamw spot: sure, we do. 11:54
* spot has not heard anyone disagree that the packages should be clearly identifiable. 11:55
spot i also think targetting the use cases is vital. 11:55
adamw sure, sorry, was just summarizing my thoughts 11:55
spot skvidal: go ahead. sorry. :) 11:55
skvidal suggestions: while we work on the tools to make building these bits safe on our systems 11:55
skvidal I've started working on a simple-ish way 11:55
skvidal to make repo maintenance at a bit less bandwidth-heavy 11:56
--> mcepl has joined this channel ( 11:56
skvidal so - for people who have their own cputime 11:56
skvidal but want to host pkgs 11:56
skvidal this can help them start using fedorapeople that way 11:56
skvidal b/c no matter what- we can't start things up tomorrow 11:57
skvidal at the same time - I think we should consider talking to our counterparts at opensuse and ubuntu about their overall experience with their services 11:57
dgilmore spot: they should be clearly identifiable 11:57
skvidal and how it has impacted their users 11:57
skvidal adamw: does mandriva have a similar system or - not? 11:57
*** XulWork is now known as XulLunch. 11:58
adamw skvidal: no, it doesn't. the backports repo serves the 'provide bleeding edge stuff' purpose 11:58
skvidal adamw: thanks 11:58
skvidal it also seems to me 11:59
skvidal that even if we never implement a system like this 11:59
adamw there isn't really much for the 'experimentation' purpose, when packagers have to do that they usually just use an interactive iurt buildroot (much like the same thing with mock) 11:59
skvidal that some of the tools we'd need to implement the system - would benefit us to have anyway 11:59
skvidal so - devel time spent on them would not be a waste 11:59
dgilmore skvidal: i agree 11:59
abadger1999 Generic task scheduler, mock-vm... yep. 11:59
* spot agrees 12:00
skvidal and then we have the final concern - that if we do implement this - we may not have the cpu and storage resources to DO it 12:00
skvidal it sure seems like a non-trivial amount of disk and potentially of processors needed 12:01
skvidal but that's a second issue 12:01
adamw that'd be a good thing to ask opensuse, i guess 12:01
dgilmore skvidal: right. I dont think we have the resources right now 12:01
adamw obs does seem quite widely-used 12:01
skvidal adamw: agreed 12:01
dgilmore adamw: iirc they were donated a bunch of systems by AMD 12:01
adamw dgilmore: that rings bells 12:01
spot if the hardware is the only reason we're not able to move forward, i should be able to get it resolved. 12:02
dgilmore they have much greater cpu power than we do 12:02
skvidal so the steps from here - sound like: skvidal: go work on what it will take to build in VMs, abadger1999: go work on the task scheduler 12:02
skvidal spot: mmcgrath was kind enough to loan us cnode5 which was doing nothing and has a lot of mem/cores for testing the vm build stuff 12:03
adamw the autoqa guys have a system for unattended installation of a VM, i believe, so that may help in that line? 12:03
skvidal adamw: we don't need to install 12:03
spot skvidal: excellent 12:03
skvidal just recreate the same one 12:03
adamw skvidal: oh yeah just clone, doh 12:03
skvidal adamw: so it's a non-issue 12:03
tremble adamw : Or use something akin too unionfs 12:03
skvidal and honestly the install problem isn't at issues - it's getting the bits in and out of the vms and making that process not crazy and painful 12:04
spot skvidal: you might want to look at libguestfs 12:04
skvidal spot: look above - I already have 12:04
adamw skvidal: how about the use cases issue? there seems some uncertainty whether this should be targeted at the 'experimentation' uses - trying out new things, or packages that are too bleeding-edge for an official repo yet; or whether they should (also?) be available for 'backports' style use 12:04
spot nm then. :0 12:04
skvidal adamw: 12:05
adamw aha, thanks 12:05
adamw was looking for that 12:05
adamw so that focuses on the short-term, experimental uses. 12:05
spot i do think that we're fooling ourselves if we don't think that there will be people using this in a backports manner 12:06
skvidal except I've never seen anything that was intended for the short-term that didn't end up getting used in the longterm 12:06
abadger1999 adamw: That's a good point. On my chart I listed several use cases that weren't on Jesse's page. 12:06
adamw skvidal: heh. 12:06
skvidal adamw: worthy of note - yum was supposed to be temporary until Jeremy katz wrote the new thing which was to obsolete it 12:06
abadger1999 adamw: One of the usecases on my chart, I think we want to actively discourage b/c I can't figure out a good way to make it work. 12:06
skvidal adamw: that's pushing 8 yrs ago 12:06
skvidal abadger1999: ? 12:07
abadger1999 adamw: 12:07
--> mether has joined this channel (~Rahul@ 12:07
adamw skvidal: sure. i trust you guys to work it out, just wanted to make sure it's on the agenda. 12:07
abadger1999 skvidal: So The one to discourage is User wants to get a bugfix that isn't going into Fedora. 12:07
adamw abadger1999: heh, firefox doesn't render that right 12:07
skvidal adamw: inkscape 12:08
adamw yeah 12:08
abadger1999 adamw: yeah -- inkscape 12:08
skvidal abadger1999: it sounds like we've got our next set of steps 12:08
skvidal anyone want to help (and by help I mean actually work on it not just peanut-gallery) 12:09
dgilmore skvidal: i will where i can 12:09
--> thewonderer57 has joined this channel ( 12:09
skvidal dgilmore: great, thx 12:09
dgilmore but i am spread thinly 12:09
spot don't everyone volunteer at once. ;) 12:10
skvidal understood 12:10
skvidal does anyone want to volunteer to talk to ubuntu and open suse? 12:10
* skvidal wonders where spevack is 12:10
skvidal spevack: think you can exploit your jono connection? get us info on what impact ppas have had - both positive and negative? 12:11
skvidal spot: do you know sandy armstrong about the ppas? b/c he works for novell now - and it might be worth asking him about both 12:11
*** stickster_mtg is now known as stickster. 12:12
adamw i know jono a bit, i could ask him if you like. 12:13
skvidal adamw: that'd be great 12:13
<-- djf_jeff has left this server (Quit: I quit). 12:13
* adamw not sure what meeting this actually is, and if you'd like help from outside whatever group this meeting is in fact for :) 12:13
skvidal it's the kopers/ppas for fedora meeting 12:13
adamw ah. okay. 12:14
skvidal adamw: it's on the schedule 12:14
skvidal but our time is up 12:14
adamw i also know michael meeks at novell, i could ask him for personal thoughts but he's not directly involved in obs, i don't think. 12:14
skvidal unless we're going to do the discussing w/mizmo now, too 12:14
abadger1999 skvidal: An hour from now. Not sure which channel either. 12:15
abadger1999 Well, 45 minutes 12:15
skvidal abadger1999: ah - right 12:15
skvidal abadger1999: is there anything else you wanted to discuss? 12:15
abadger1999 Nope, that's it :-) 12:15
skvidal anyone else? 12:15
skvidal okie doke 12:17
skvidal thanks for the input everyone! 12:17

Generated by 2.7 by Marius Gedminas - find it at!