From Fedora Project Wiki
| <spot> | skvidal: yep | 11:01 |
| <skvidal> | okay | |
| abadger1999: you around? | ||
| <abadger1999> | Yep | |
| <skvidal> | abadger1999: you wanna start? you have the fancy diagram :) | |
| <abadger1999> | http://toshio.fedorapeople.org/apb-overv iew.svg | 11:02 |
| Okay, that's an overview of what we think the whole thing ends up looking like. | ||
| The black path is the main path for building packages | 11:03 | |
| The orange path is the main path for managing repositories | ||
| A lot of stuff has to get built to make this all work. | 11:04 | |
| <spot> | wow, chromium and firefox really render that badly | |
| * spot | opens it in inkscape | |
| <skvidal> | spot: yah - I had to open it in inkscape to see it at all | 11:05 |
| *** stickster | is now known as stickster_mtg. | |
| <tremble> | Nice to know it's not just my browser mangled it. | |
| <skvidal> | as a summary for any folks just coming into this - we're talking about arbitrary package builds | 11:06 |
| <-- mcepl has left this server (Read error: Connection reset by peer). | ||
| <skvidal> | from arbitrary sources, using arbitrary repos for arbitrary build dep providing | |
| whereas koji can currently handle arbitrary pkgs being built | 11:07 | |
| that's a scratch build | ||
| <tremble> | So we could for example build a batch of builds for EPEL and then makke a single push into testing. | |
| <-- JSchmitt has left this server (Remote host closed the connection). | ||
| <tremble> | ? | |
| <skvidal> | it cannot do scratch chain build and pull in arbitrary buildeps | |
| tremble: and no 0- the goal of this is not for official builds | 11:08 | |
| official builds will ALWAYS have to go through koji | ||
| <-- jnettlet has left this server (Ping timeout: 252 seconds). | ||
| <skvidal> | if only for the audit-trail that koji provides | |
| <spot> | (not withstanding that we will want some measure of auditing in this process as well, but still) | |
| <skvidal> | auditing of what? | 11:09 |
| <spot> | well, just logging really | |
| <skvidal> | the ppas/build services are the remote equivalent of running mock on your own machine | |
| <spot> | history of who did what. | |
| <abadger1999> | spot: as in build.log? or more than that? | |
| <skvidal> | sure | |
| but that's not auditing | 11:10 | |
| <abadger1999> | \okay. | |
| <skvidal> | in the sense that koji uses it | |
| <spot> | skvidal: agreed | |
| <skvidal> | logging who did what - that's a web server log | |
| logging what pkgs were built, with what pkgs in the chroot and all the tags of those pkgs | ||
| that's a whole different ballpark | ||
| hell, it doesn't even look like the same GAME | ||
| the gist of what a ppa/buildservice does is allow a user to build a package not using their own cpu time | 11:11 | |
| but using the cputime of the service | ||
| cputime (and bandwidth) | ||
| <spot> | yep. | |
| <abadger1999> | and build a repository that end users can use to easily grab the packages. | 11:12 |
| <skvidal> | so the goal here is just to build the 'build and host this pkg for me' service | |
| we've been looking around at other distros systems for this | 11:13 | |
| ubuntu's ppa system and the opensuse build service in particular | ||
| and in both cases they build the pkgs inside a vm protecting the host system from the building process being evil or just bad | 11:14 | |
| the interface criteria are more or less the same - you define your repo requirements | ||
| you toss a pkg at the system | ||
| it gives you back a url as to where your finished results are (or logs of why it failed) | ||
| wash, rinse, repeat | 11:15 | |
| the pieces we do NOT have are: | ||
| 1. building inside a vm | ||
| 2. the web/xmlrpc backend for submitting/creating the repo and srpms | ||
| 3. the interface/storage location for the resulting pkgs | 11:16 | |
| mmcgrath has been nice enough to loan us a big system for testing a bunch of these things out | ||
| w about mock and building inside a vm and what that entails | ||
| and with rwmjones about guestfs's capabilities | ||
| <-- inode0 has left this server (Quit: Leaving.). | ||
| <skvidal> | and reading the opensuse build code | 11:17 |
| and step 1 is not insurmountable - but doing it 'pretty' would take a while | ||
| making it functional - might be doable in a much shorter time frame | ||
| for step 1 | 11:18 | |
| abadger1999: what're thoughts on the xmlrpc/whatever for requesting repos and submitting srpms? | ||
| <abadger1999> | i've talked with dmalcolm about the tak scheduler he has inside of rpmgrok. | 11:19 |
| Since rpmgrok is a TG1 app with an SQLAlchemy model, it should be pretty easy to adapt. | ||
| he says that koji has some things that are nice, though -- like dependencies among tasks. | 11:20 | |
| I don't think it'll be too hard to pull just those features in. | ||
| My thought is that we'd want the web interface to add jobs to the db. | ||
| Then the task scheduler itself pulls the jobs out and sends them to task runners -- this is very similar to how koji works. | 11:21 | |
| We'll use standard TG2 tech like the json API you get for free with any controller methods. | ||
| I'm kind of leaning towards seeing if we can use func to communicate between the task scheduler and task runners. | 11:22 | |
| <mbonnet> | abadger1999: not really, builders *claim* tasks in koji, they are never assigned tasks | |
| <abadger1999> | mbonnet: That's true. push vs pull was one ofthe things I was going to ask you about. | |
| mbonnet: What are the advantages of builders pulling jobs rather than koji hub pushing them? | ||
| <skvidal> | abadger1999: you don't have to monitor which builders are up | 11:23 |
| <mbonnet> | abadger1999: it means the server can be essentially passive | |
| and builders only need outbound connections, so they can be firewalled/natted | 11:24 | |
| lt has joined this channel (~bpepple|l@d60-65-24-211.col.wideopenwe st.com). | ||
| <abadger1999> | Okay, and what are the downsides? | 11:26 |
| <mbonnet> | It can make load-balancing tougher, since you don't necessarily have a global view of who is doing what | 11:27 |
| It really depends on the design of the hub/server part | ||
| --> spoleeba has joined this channel (~one@fedora/Jef). | 11:28 | |
| <mbonnet> | For Koji, the hub is just Apache, so the builders *have* to be the ones initiating the connections | |
| <abadger1999> | <nod> In our setup, the scheduler would be more active than that. | |
| <mbonnet> | abadger1999: But the passive hub approach means the hub is *only* a server and the builders are *only* clients. If the builders accept connections, they are both clients and servers, and the hub is both a server and client, and I think that complicates the code. | 11:29 |
| <dgilmore> | abadger1999: plague supported both pushing and pulling | 11:30 |
| <skvidal> | okay | |
| before we get too far here | 11:31 | |
| <abadger1999> | mbonnet: True.. although if we do reuse func then we are working within the architecture that it provides. | |
| <mbonnet> | sure | |
| <skvidal> | are there any more general concerns? | |
| <abadger1999> | Should we be doing this at all? | |
| <mbonnet> | If I were going to redesign Koji right now, I would probably make all communication amqp-based, which allows bidirectional communication. | |
| <skvidal> | abadger1999: ? - the issue of the many many many competing and confusing "microrepos" | |
| <abadger1999> | Yeah eaxactly. | 11:32 |
| <mbonnet> | I certainly think it has the potential the make support dramatically more difficult | |
| <abadger1999> | mbonnet: That's a good thought. I'll look into doing this via amqp as a possibility. | 11:33 |
| <skvidal> | and potentially to make users' lives more difficult b/c they might not know what is a fedora pkg | |
| and what is not | ||
| <spot> | well, i think if we were to something like institute a repotag to mark these packages it will aid in that | |
| <skvidal> | spot: b/c the repotags have helped keep people from confusing rpmfusion w/fedora? | |
| <spot> | skvidal: its not so much people that we can really manage in this aspect | 11:34 |
| it is bug reporters when there is overlap | ||
| if a package is clearly marked in its n-v-r as being built in a personal repo | ||
| that at least makes bug resolution simpler. :) | ||
| <mbonnet> | How would this affect efforts like abrt? | |
| <skvidal> | spot: marked in its nvr? | 11:35 |
| <abadger1999> | adamw: You around by chance? | |
| <spot> | skvidal: think forced disttag | |
| <skvidal> | spot: yum is storing where a pkg is installed FROM | |
| ie the repo it was installed from | ||
| in the yumdb - which is available to every pkg installed | ||
| so we can get the repo | ||
| <spot> | skvidal: that is an additional source of info, but not likely to make it to bugzilla | |
| <adamw> | abadger1999: yep | |
| <abadger1999> | adamw: We're talking about actually implementing something like kopers (ppas, etc). | 11:36 |
| <spot> | mbonnet: we have the problem now in abrt where it flags on third-party stuff not in fedora | |
| <adamw> | yeah, just reading a bit back | |
| personal repo packages should *definitely* have a different dist tag from fedora packages | ||
| ideally, each repo would have its own dist tag | ||
| <abadger1999> | adamw: The immediate question was about abrt... but I know you had thoughts on the general support issues as well. | |
| <mbonnet> | spot: right, but would abrt be able to get debuginfo/generate backtraces for kopers packages? | |
| <spot> | mbonnet: abrt generally discards reports for non-Fedora stuff, except in cases where it gets confused and thinks it is a crash in an interpreter | |
| mbonnet: i'd say no, nor do we want it to | 11:37 | |
| <mbonnet> | spot: agreed. How does abrt tell if something is a "Fedora" pkg or not? | |
| <spot> | mbonnet: good question, would have to look at the source | 11:38 |
| <abadger1999> | Getting to nitty gritty, we'd probably want to force both disttag and repotag. => .fc12.apb | |
| <spot> | abadger1999: agreed | |
| h> | Note that we set Packager and Vendor on our packages to "Fedora Project". | |
| <adamw> | yeah, that works. as long as it's identifiable in the nvr at a glance. | |
| <skvidal> | h: and that would have to change | 11:39 |
| for these builds | ||
| h> | Yep. | |
| <wwoods> | what about the gpg signature(s)? | |
| <abadger1999> | wwoods: Ubuntu creates a separate key per repo and signs the packages when built | 11:40 |
| <skvidal> | abadger1999: any pkg that is submitted | |
| <abadger1999> | Yes | |
| <spot> | abadger1999: that seems like a reasonable approach. | |
| <wwoods> | I assume we wouldn't be signing this stuff with the Fedora Project keys. Right? Hopefully ABRT is checking the file checksum / package key ID to see if a given file is actually from Fedora. | |
| <skvidal> | wwoods: so it just ensures that is the pkg that came out of the build process - not anything about the goodness of it | |
| <spot> | wwoods: no. not with Fedora key. | |
| <dgilmore> | abadger1999: we could probably dynamically set it to .fc12.<submitter> | 11:41 |
| <wwoods> | or rather, we wouldn't be signing with the "Yes This Is An Official Fedora 12 Package" key. Not to suggest they should be totally unsigned. but yeah. | |
| <spot> | dgilmore: eh, i'd prefer consistency here so that everyone knows at a glance that it is both not-Fedora and comes from the personal repo infrastructure | |
| <skvidal> | dgilmore: think about multiple submitters per ppa | |
| <abadger1999> | dgilmore: We could... but that probably makes support harder... Instead of asking "Do you apb in the release" you have to ask do you have some odd characters after the .fc12. ? | |
| <dgilmore> | spot: its easier to have it static | 11:42 |
| <adamw> | spot: viz my old blog post, i do think it would be useful to identify packages which come from *different* personal repos | |
| spot: so you can see where you may have a problem because you have an app from one person's repo, but it's using a lib from another's... | ||
| <skvidal> | adamw: that's what we said | |
| <spot> | adamw: and yum will serve that purpose. | |
| * nirik | wonders about two repos each having zlib-1.0-1.fc12.ppa | |
| <skvidal> | getting out of the weeds a bit | |
| do the benefits to fedora and our users, outweigh the debugging/bugreport/knowing what is what problems that this could create? | 11:43 | |
| in short - what do we get out of a service like this? | ||
| <abadger1999> | here's a list of benefits and drawbacks I made up late last night: http://fpaste.org/xmEG/ | |
| <spot> | abadger1999: sums it up pretty well. | 11:44 |
| --> bochecha has joined this channel (~bochecha@fedora/bochecha). | ||
| <spot> | i would note that the existence of PPAs is commonly cited as a main reason that developers/packagers prefer to work in Ubuntu | 11:45 |
| <skvidal> | spot: where? | |
| <spot> | well, in various emails sent to me, lemme see if i can back that statement up with public comments | |
| <abadger1999> | spot: I've also heard a lot of Ubuntu users cite "dependency hell" as the reason they left Red Hat linux/Fedora Core | |
| h> | I'm having trouble seeing how the hosting of software not appropriate for Fedora by Fedora could be on the helpful side. | |
| <mbonnet> | Are we worried about kopers redirecting the energies of packagers? So they start maintaining their kopers while leaving their Fedora packages to languish? | 11:46 |
| <skvidal> | spot: when I was looking for notes and anecdotes on how ppas have succeeded I found a lot of references to dpkg --force to resolve dep issues | |
| <spot> | skvidal: heh. | |
| <adamw> | skvidal: that sounds much like what I was blogging about. | |
| <skvidal> | spot: I'm sadly not kiding | |
| kidding, even | ||
| <abadger1999> | h: perhaps I was biased when I wrote that :-) But it is a good point ... lots of the benefits have their own subtle drawbacks. | |
| mbonnet: I am. | ||
| <skvidal> | now - for all intents and purposes this is happening now | |
| people can upload stuff to fedorapeople.org | 11:47 | |
| host a repo there | ||
| no problem | ||
| if they have their own cputime and bandwidth | ||
| * spot | nods | |
| <skvidal> | they can build to their hearts content | |
| --> biertie has joined this channel (~bert@fedora/biertie). | ||
| <skvidal> | the difference is - by making it EASY does it become disruptive | |
| s/difference/queston | ||
| sorry | ||
| <spot> | skvidal: http://lists.beatniksoftware.com/piperma il/tomboy-list-beatniksoftware.com/2009- September/001352.html | |
| <pjones> | skvidal: this is one of the reasons I suggested a per-SIG basis - it makes a somewhat higher (but not too high) barrier to entry | |
| <adamw> | spot: that mail seems like a good example of bad use of PPAs, to me: 'we disagree with Ubuntu's update policy so we're going to circumvent it with a permanent, somewhat-competing repository' | 11:48 |
| <spot> | pjones: right now, all it takes to be a SIG is to say "this SIG exists" | |
| <pjones> | spot: all it requires is ze vill to do eet, ya! | 11:49 |
| <adamw> | do we really want to give third-party projects the power to say 'we don't want you to use Fedora's packages for our project, we want you to use ours from this sorta-official-but-not-really koper'? What happens if they all say that? | |
| <spot> | adamw: well, you say bad, others say "what my users want" | |
| <pjones> | adamw: I think that's one of the most compelling reasons to have such a thing - so we can have one update policy for the distro, but a user/SIG/whatever can have a different update policy for those that choose to be more risky | |
| <spot> | adamw: well, we can look at Ubuntu for example, there is some of this, but it is not overwhelming | |
| <adamw> | fair enough. just raising the question. | 11:50 |
| <pjones> | So - yes, we really do want to give them that. that's the point. | |
| <mbonnet> | Do we have a set of use-cases for kopers? Are we really trying to address the update-policy issue with them? | |
| <spot> | my feeling from looking through the PPAs in use there is that most of them are either "stuff that isn't ready for Ubuntu" or "stuff that is much newer than Ubuntu, but is headed there" | |
| <skvidal> | mbonnet: the use cases described are in jesse's draft | |
| <abadger1999> | pjones: It's both good and bad -- it gives more freedom to users/sigs... but it also makes whole distro integration and targetting harder.... | 11:51 |
| <spot> | mbonnet: well, i think there are some use cases. my chromium packages, for one. kde-redhat for another. | |
| <mbonnet> | spot: isn't the "stuff that's too new" supposed to go into rawhide? | |
| <abadger1999> | I'm a third party making some KDE software, do I target Fedora-13 or Fedora-13 + the PPA? | |
| <mbonnet> | spot: kde-redhat makes me sad | |
| <spot> | mbonnet: yeah, but what if i want it built for F-12? | |
| <adamw> | mbonnet: not if you don't know when it's going to be ready. rawhide isn't sid; right now, if you put something in rawhide, you're saying you expect it to be okay to go in F14 | |
| <mbonnet> | adamw: "ready" is a vague term | 11:52 |
| I guess I just worry about kopers fracturing the Fedora community, where people focus on their one area of interest and the main distro falls by the wayside | 11:53 | |
| but maybe that's paranoia | ||
| <adamw> | i think it's a tough question to answer, it's a counterfactual | |
| <spot> | mbonnet: i'd rather worry about that if/when it happened. | |
| <adamw> | we can't know how people will use them until they exist | |
| <mbonnet> | spot: by then its too late | |
| <spot> | mbonnet: eh. | 11:54 |
| mbonnet: i disagree. | ||
| <mbonnet> | spot: take away kopers after people have been using them and you'll have a revolt | |
| --> JSchmitt has joined this channel (~s4504kr@p4FDD01A7.dip0.t-ipconnect.de) . | ||
| <-- JSchmitt has left this server (Changing host). | ||
| --> JSchmitt has joined this channel (~s4504kr@fedora/JSchmitt). | ||
| <skvidal> | so I have a couple of suggestions | |
| <spot> | adamw: we do have some evidence on how this works in OpenSUSE and Ubuntu. | |
| <skvidal> | to step ourselves into things | |
| <adamw> | i do think if we do it, we should have clear use cases for how they're intended to be used, the system should be designed to encourage those use cases and discourage others, and the packages should be clearly identifiable. | |
| spot: sure, we do. | ||
| * spot | has not heard anyone disagree that the packages should be clearly identifiable. | 11:55 |
| <spot> | i also think targetting the use cases is vital. | |
| <adamw> | sure, sorry, was just summarizing my thoughts | |
| <spot> | skvidal: go ahead. sorry. :) | |
| <skvidal> | suggestions: while we work on the tools to make building these bits safe on our systems | |
| I've started working on a simple-ish way | ||
| to make repo maintenance at fedorapeople.org a bit less bandwidth-heavy | 11:56 | |
| --> mcepl has joined this channel (~mcepl@184.109.broadband10.iol.cz). | ||
| <skvidal> | so - for people who have their own cputime | |
| but want to host pkgs | ||
| this can help them start using fedorapeople that way | ||
| b/c no matter what- we can't start things up tomorrow | 11:57 | |
| at the same time - I think we should consider talking to our counterparts at opensuse and ubuntu about their overall experience with their services | ||
| <dgilmore> | spot: they should be clearly identifiable | |
| <skvidal> | and how it has impacted their users | |
| adamw: does mandriva have a similar system or - not? | ||
| *** XulWork | is now known as XulLunch. | 11:58 |
| <adamw> | skvidal: no, it doesn't. the backports repo serves the 'provide bleeding edge stuff' purpose | |
| <skvidal> | adamw: thanks | |
| it also seems to me | 11:59 | |
| that even if we never implement a system like this | ||
| <adamw> | there isn't really much for the 'experimentation' purpose, when packagers have to do that they usually just use an interactive iurt buildroot (much like the same thing with mock) | |
| <skvidal> | that some of the tools we'd need to implement the system - would benefit us to have anyway | |
| so - devel time spent on them would not be a waste | ||
| <dgilmore> | skvidal: i agree | |
| <abadger1999> | Generic task scheduler, mock-vm... yep. | |
| * spot | agrees | 12:00 |
| <skvidal> | and then we have the final concern - that if we do implement this - we may not have the cpu and storage resources to DO it | |
| it sure seems like a non-trivial amount of disk and potentially of processors needed | 12:01 | |
| but that's a second issue | ||
| <adamw> | that'd be a good thing to ask opensuse, i guess | |
| <dgilmore> | skvidal: right. I dont think we have the resources right now | |
| <adamw> | obs does seem quite widely-used | |
| <skvidal> | adamw: agreed | |
| <dgilmore> | adamw: iirc they were donated a bunch of systems by AMD | |
| <adamw> | dgilmore: that rings bells | |
| <spot> | if the hardware is the only reason we're not able to move forward, i should be able to get it resolved. | 12:02 |
| <dgilmore> | they have much greater cpu power than we do | |
| <skvidal> | so the steps from here - sound like: skvidal: go work on what it will take to build in VMs, abadger1999: go work on the task scheduler | |
| spot: mmcgrath was kind enough to loan us cnode5 which was doing nothing and has a lot of mem/cores for testing the vm build stuff | 12:03 | |
| <adamw> | the autoqa guys have a system for unattended installation of a VM, i believe, so that may help in that line? | |
| <skvidal> | adamw: we don't need to install | |
| <spot> | skvidal: excellent | |
| <skvidal> | just recreate the same one | |
| <adamw> | skvidal: oh yeah just clone, doh | |
| <skvidal> | adamw: so it's a non-issue | |
| <tremble> | adamw : Or use something akin too unionfs | |
| <skvidal> | and honestly the install problem isn't at issues - it's getting the bits in and out of the vms and making that process not crazy and painful | 12:04 |
| <spot> | skvidal: you might want to look at libguestfs | |
| <skvidal> | spot: look above - I already have | |
| <adamw> | skvidal: how about the use cases issue? there seems some uncertainty whether this should be targeted at the 'experimentation' uses - trying out new things, or packages that are too bleeding-edge for an official repo yet; or whether they should (also?) be available for 'backports' style use | |
| <spot> | nm then. :0 | |
| <skvidal> | adamw: http://fedoraproject.org/wiki/JesseKeati ng/KojiPersonalRepos#Usage_Case | 12:05 |
| <adamw> | aha, thanks | |
| was looking for that | ||
| so that focuses on the short-term, experimental uses. | ||
| <spot> | i do think that we're fooling ourselves if we don't think that there will be people using this in a backports manner | 12:06 |
| <skvidal> | except I've never seen anything that was intended for the short-term that didn't end up getting used in the longterm | |
| <abadger1999> | adamw: That's a good point. On my chart I listed several use cases that weren't on Jesse's page. | |
| <adamw> | skvidal: heh. | |
| <skvidal> | adamw: worthy of note - yum was supposed to be temporary until Jeremy katz wrote the new thing which was to obsolete it | |
| <abadger1999> | adamw: One of the usecases on my chart, I think we want to actively discourage b/c I can't figure out a good way to make it work. | |
| <skvidal> | adamw: that's pushing 8 yrs ago | |
| abadger1999: ? | 12:07 | |
| <abadger1999> | adamw: http://toshio.fedorapeople.org/apb-overv iew.svg | |
| --> mether has joined this channel (~Rahul@115.240.30.200). | ||
| <adamw> | skvidal: sure. i trust you guys to work it out, just wanted to make sure it's on the agenda. | |
| <abadger1999> | skvidal: So The one to discourage is User wants to get a bugfix that isn't going into Fedora. | |
| <adamw> | abadger1999: heh, firefox doesn't render that right | |
| <skvidal> | adamw: inkscape | 12:08 |
| <adamw> | yeah | |
| <abadger1999> | adamw: yeah -- inkscape | |
| <skvidal> | abadger1999: it sounds like we've got our next set of steps | |
| anyone want to help (and by help I mean actually work on it not just peanut-gallery) | 12:09 | |
| <dgilmore> | skvidal: i will where i can | |
| --> thewonderer57 has joined this channel (~roblx@cpc2-sgyl4-0-0-cust5.sgyl.cable. virginmedia.com). | ||
| <skvidal> | dgilmore: great, thx | |
| <dgilmore> | but i am spread thinly | |
| <spot> | don't everyone volunteer at once. ;) | 12:10 |
| <skvidal> | understood | |
| does anyone want to volunteer to talk to ubuntu and open suse? | ||
| * skvidal | wonders where spevack is | |
| <skvidal> | spevack: think you can exploit your jono connection? get us info on what impact ppas have had - both positive and negative? | 12:11 |
| spot: do you know sandy armstrong about the ppas? b/c he works for novell now - and it might be worth asking him about both | ||
| *** stickster_mtg | is now known as stickster. | 12:12 |
| <adamw> | i know jono a bit, i could ask him if you like. | 12:13 |
| <skvidal> | adamw: that'd be great | |
| <-- djf_jeff has left this server (Quit: I quit). | ||
| * adamw | not sure what meeting this actually is, and if you'd like help from outside whatever group this meeting is in fact for :) | |
| <skvidal> | it's the kopers/ppas for fedora meeting | |
| <adamw> | ah. okay. | 12:14 |
| <skvidal> | adamw: it's on the schedule | |
| but our time is up | ||
| <adamw> | i also know michael meeks at novell, i could ask him for personal thoughts but he's not directly involved in obs, i don't think. | |
| <skvidal> | unless we're going to do the discussing w/mizmo now, too | |
| <abadger1999> | skvidal: An hour from now. Not sure which channel either. | 12:15 |
| Well, 45 minutes | ||
| <skvidal> | abadger1999: ah - right | |
| abadger1999: is there anything else you wanted to discuss? | ||
| <abadger1999> | Nope, that's it :-) | |
| <skvidal> | anyone else? | |
| okie doke | 12:17 | |
| thanks for the input everyone! |