From Fedora Project Wiki

Fedora Release Engineering Meeting :: Monday 2008-02-11



  • Release Engineering is expected to produce custom spins for F8

1. it's a lot of bit moving 1. are we supposed to be producing them from F8 GOLD or F8 + Updates, and if from Updates, does installing from them still work?

ACTION: Jeremy to write first draft of how Spins should work--in detail--using Rahul's proposal as a starting point

Mirror Master Move

  • Targeting Wednesday
  • Rough plan:

1. turn off rawhide Tuesday night 1. turn off everything else on Wednesday

  • Continue discussion on #fedora-admin

IRC Transcript

-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Roll Call13:18
f13ping: notting jeremy spot jwb rdieter wwoods poelcat lmacken13:18
* notting is here13:18
* jeremy waves13:18
* lmacken 13:18
rdieterhere .513:19
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - gcc4.313:21
f13I'm skipping over perl5.10.0 since spot isn't here.13:21
f13FESCo approved my proposal for doing mass rebuilds, but there are still a couple things to get answered.13:21
f131) I'd like a way for folks to opt-out of the rebuild machine, and I'm looking for good ideas on how to implement this.13:21
jwbcvs it13:22
jwbadd a 'norebuild' file13:22
f13Oh FYI:
lmackenpkgdb ?13:22
jwblmacken, that seems complicated13:22
f13one thing I thought of, is that they could just mail us, and we'll add it to a 'blacklist' that if the checker hits upon said package will just skip and not add it to the need build list13:22
lmackena checkbox is complicated ?13:22
f13maybe a wiki page where they can drop the package in13:22
jwblmacken, for a script, yes13:23
lmackennot when you have a JSON api13:23
jwblmacken, particularly when that script will be going through CVS anyway...13:23
f13It already takes 40~ minutes to discover what needs to be built, I'd like to not add to it by adding more DB calls13:23
f13jwb: I'd prefer to not have to checkout from CVS to skip the package.13:23
jwbf13, ok13:24
f13that's the script I'm using13:24
f13I could probably easily slurp down some data from a wiki page or something to populate a blacklist13:25
f13it'd just be a matter of driving users to populate said wiki page or something.13:25
jwbdunno...  just keeping the blacklist inline might be easier and have them email rel-eng13:25
warreninline meaning?13:26
jwbas an array in the script13:26
jwbor a list13:26
jwbor whatever it's called in python13:26
f13the problem I have with that, is mailing rel-eng can be lossy13:26
f13whereas if they fail to update the wiki page, they only one to blame is them (:13:26
jwbwell, i don't think it's any more lossy13:27
jwbwiki pages can be screwed up.  particularly when lots of people edit the same page13:27
warrenso either rebuild it yourself or update the wiki page to tell you not to do it?13:27
f13jwb: note the missed tag requests we seem to have every freeze (and part of why I want to turn rel-eng@ into a Trac ticket input)13:27
f13jwb: good points though13:28
jwbf13, there are some misses, yes.  but if we are all diligent about it, we should be able to track this13:28
f13warren: also find the things which aren't really gcc built.13:28
jwbdo we really expect a massive list?13:28
f13warren: 'cause discovering which mono packages don't actually use gcc is somewhat troublesome13:28
f13jwb: I honestly don't know.13:28
f13jwb: I'd say I usually only get 20~ 'don't build this for me' requests when I did it for Core before13:29
f13so in the interest of getting the bikeshed in place, lets go with mail rel-eng and we'll add it to a static list in the script.  Any opposed?13:30
f13PROPOSAL ^^13:30
jwbfine with me13:31
lmackensounds good13:31
f13I'll announce that we're accepting opt-outs.13:31
f13and we'll start processing those.  Once I automate the list generation I'll make sure to add a git pull before the script run so that it's always updated13:32
f13next question13:32
f132) # What to do about non-responsive maintainers for failed builds13:32
f13this is probably just a simple 'go in and rebuild it anyway'.13:32
f13but I feel it would be due dilligence to process these nonresponsive maintainers.13:32
f13but that's a lot of... paperwork.13:33
jwbf13, we could take that to FESCo13:33
f13does anybody want to volunteer to chase up that side, or should we put out a call to arms via.. FESCo?13:34
jwbi can take it to FESCo13:34
jeremycall to arms I think13:34
f13alright, I think that does it for outstanding questions on this issue13:36
f13I have a bit of work ahead of me, particularly in the autospecbump land13:36
f13but I think it can by ready by the auto build start13:36
f13I plan on doing a small number of scripted just to make sure things are going well before turning on the firehose.13:37
jwbfeel free to use my packages :)13:37
warrenWhen is te scheduled mass again?13:37
f13warren: Feb 18th13:38
f13provided perl lands safely before then13:38
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering - Spins... WTF?13:39
jwbas of right now, i'm waiting on mmcgrath13:40
jwbbut long term.. wtf13:40
f13so we now have some spin requests coming in, and releng is expected to produce them... for F813:40
f13I've got some pretty serious issues with this13:40
jeremythat was the plan all along13:40
f131) it's a lot of bit moving13:40
nottingf13: the premise being that rel-eng created all previous ones, so....?13:40
jeremythat's why there was an attempt to limit to one a month13:40
f132) Are we supposed to be producing them from F8 GOLD or F8 + Updates, and if from Updates, does installing from them still work?13:41
jwbf13, updates13:41
jwbmether explicitly asked about that13:41
jwband the .ks files he has have the updates repo in them i believe13:41
f13so the next question is, does installing from them still work, given that we don't as of yet do anaconda updates?13:42
jwbthese are all livecds, so i think install works?13:42
jeremy+ updates I presumed.  and yes, installs could run against problems.  although given history, less problems13:42
f13jeremy: do we (the greater Fedora we) care?13:42
jeremyf13: care about which?13:42
jwbfrankly, i think that burden falls onto the spin owner13:42
f13when/if the install breaks?13:42
f13do we have a checklist of criteria that the spin owner has to run their product through?13:43
f13or hell13:43
f13does releng have a checklist of what we look for when approving a spin from a technical aspect?13:43
f13(rhetorical, of course we don't)13:43
jwbi asked mether about that a while ago13:43
jwbfedora unity has a fairly large test matrix they run though13:43
jwber, through13:43
jwbi have no idea if he looked at it or not13:44
jeremyf13: the "what we look for" mostly comes down to a) is the package list "reasonable" looking b) smoke test off of building them13:44
jeremyat least, that's what I've done13:44
f13jeremy: yeah, but what does "smoke test" entail?13:45
* nirik wonders if there shouldn't be a spin review process very much like a package review process. 13:45
f13if we're going to be on the hook for approving these things, I'd like a guideline of what to look for13:45
jeremyf13: build it, boot it, install it.  do some clicking around for sanity13:45
warrennirik, does spin include respin, or things like KDE?13:45
jwbjeremy, build it, boot it is fine.  install it is harder13:45
f13jwb: kvm!13:45
jeremylonger term, it would be great to have something that was more concrete, but we only managed to get packaging guidelines after the better part of ten years of building packages :P13:46
nirikif you are talking about the Xfce spin, I did do some installs here...13:46
jwbmeh, true13:46
jwbnirik, no, we know your spin is going to be broken and don't care :P13:46
nirikwarren: well, respins are not really controlled directly by fedora, so no idea...13:46
f13so what would be great, is at least an SOP on spin review13:46
nirikjwb: excellent. ;)13:46
warrendo spins have to worry about upgrades?13:46
f13after that, an SOP on spin production13:47
jwbwarren, huh?13:47
f13warren: as of yet, all 'spins' have been live based, which don't allow for upgrades13:47
warrenOK, was just being sure.13:47
jwbf13, i can try the SOP on spin production since i'm going to to e spinning the ones that have already been approved13:47
f13jwb: that would be awesome13:47
jwbit will start as a list of things i did, and then i'll put some words around it13:48
f13mmcgrath hooked us up with a xen guest to do i386/ x86_64 live image production, but we're kind of SOL for ppc production right now13:48
jwbum.. no13:48
jeremyjwb: sounds good.  if you do it on the wiki, I'll wiki watch it and provide quick feedback13:48
jwbhe's trying to.  still having issues with xen and f813:48
jwbjeremy, yep13:48
f13jwb: ah, fun. :/13:48
warrenjwb, did he try fullvirt instead?13:49
nottingare there specific comments on rahul's proposal we would like to provide as feedback?13:49
jwbwarren, i think he is doing fullvirt, but with xen as the host13:49
jeremyoh yeah, I need to look at rahul's actual proposal13:49
warrenjwb, oh13:49
jwbi could be wrong.  it just seems f8 and xen don't like each other when it comes to dhcp or static ip or something13:50
f13notting: not sure what you mean13:50
* nirik thinks no one likes xen. ;) 13:50
* warren wonders how functional xenner is13:50
nottingf13: spoleeba is trying to work towards hashing it out to where it can be a real process/policy. does rel-eng have issues with things that are listed there that they'd like to raise?13:51
jeremywarren: not functional enough13:51
spoleebanotting, is this the spins draft?13:51
jeremywarren: I played with it a week and a half ago13:51
jwbspoleeba, yes13:51
spoleebanotting, talking to rahul... the big problem he has is the interaction with releaseeng... we need to figure out exactly what we expect spin developers to bring to releaseeng..and give them enough community infrastructure to actually get that crap done13:52
f13notting: nothing specific comes to my mind, but that's because it's still rather wild and undefined.  I'd love to review spoleeba's proposal as part of a releng meeting13:52
spoleebanotting, in terms of pre-testing on arches and what not13:52
nottingthat' simple. no resources :P13:53
* nirik also wonders if the board should be the one approving spins, shouldn't that be a fesco thing? or is it due to them using the fedora name ?13:53
spoleebaf13, i have rahul's perspective on what he needs from a spin creator pov.. i need some assessment from releaseeng about what they need from spin creators13:53
nottingyes, it's due to the trademark13:53
jwbspoleeba, "community infrastructure" meaning what?13:53
jeremynirik: due to trademark13:53
spoleebaf13, than i can make an oreo cookie out of it13:53
spoleebajwb, there's an amorphous issue concerning the amount of pre-testing that spin creators need to do before coming to keep the releaseeng burden down to something sane13:54
spoleebajwb, but at the same time.. spin creators dont have ppc hardware13:54
jwbwe haven't made a ppc spin yet, and i don't plan to13:54
nottingspins don't have to be for every arch13:54
jwbspoleeba, i think, for all intents and purposes, ppc can be considered a secondary arch here13:54
spoleebajwb, its was just an abstract comment.. .the point is... we need to be able to have a pre-test matrix13:54
jeremyspoleeba: ppc spins would need to have the ppc team involved as they do the coordination for that arch13:55
jwbspoleeba, sure.  that's why i suggested mether look at the fedora unity test matrix13:55
spoleebajwb, that covers as much of what releaseeng needs people to do13:55
warrenthere is no demand for 64bit spins of edu, for exanple13:55
spoleebajwb, so did i13:55
spoleebawarren, i demand it!13:55
jwbpersonally, i'd be very happy if a spin owner came to me with a wiki page that had that test matrix filled out13:55
f13part of the problem here is that we haven't had a good matrix for the spins we produce as part of final releases too13:56
spoleebawarren, im pretty sure i ncssm had 64bit hardware :->13:56
f13live ones that is13:56
f13we're looking to wwoods' team for providing some of that13:56
f13perhaps we should look to them to provide this as well13:56
jwbf13, don't we derive much of that from wwoods' testing of the actual non-live spins?13:56
jeremyI can try to make something up to start from13:56
f13jwb: sortof13:57
jeremy(might be a day or three before I can get to it)13:57
spoleebaf13, so like i said.. i need releaseeng's pov on bottleneck's so far with custom spins that have come up for review13:57
wwoodsyeah we don't have a solid test plan for live media yet13:57
jeremyjwb: quite a bit, yes13:57
nottingat a minimum, they need to test before sending to rel-eng that it composes, builds, boots, and installs something that matches what it boots13:57
EvilBobjwb: the Matrix we have in Fedora Unity was based on a list of tests that wwoods had at one point, we have expanded on it for roblems we have found and want to confirm closed13:57
f13spoleeba: quite honestly the bottleneck was 0 idea as to what was expected of us.13:57
f13spoleeba: it felt like the board said "Go forth and do custom spins" without any indication of /how/ that would be done and who would be responsible for doing what.13:58
jeremyf13: also, to be fair, some of the bottleneck was that I was going to look at them and wasn't around for a month13:58
jwbi'm still concerned about the hosting aspect, but *shrug*13:58
f13jeremy: there is that.13:58
spoleebaf13, well we could cut releaseeng out completely..but im pretyt sure that's not desired13:58
wwoodsfor installation we've got the old test matrix (which I believe the Unity matrix is based on):
spoleebaf13, releaseeng does want an up/down veto opportunity in the process right?13:59
f13spoleeba: I'd like to know how you'll get the spin produced/hosted without releng (:13:59
wwoodsAnd the new TestPlan (with many descendent detailed TestCases):
wwoodsSo! Assuming we write a TestPlan for Live images, that'd be a fine base for any planned testing of new live spins13:59
spoleebaf13, there was also an issue concerning cvs access..since the ks files that are'blessed' live in the live-cd-tools tree13:59
warrenjeremy, when do things get busy for you again?13:59
f13spoleeba: releng/fesco/somebody should give a technical ACK on the proposed spin.13:59
nottingspoleeba: that seems like a long-term lose13:59
spoleebaf13, we might need to create a beta spin tree for crap still under review and being worked on14:00
jeremywarren: things are always busy for me now! :)   but not january-busy14:00
wwoodsI'd think it was reasonable to require spin owners to at least get through Tier1 (whatever we define for that) before handing it off14:00
spoleebanotting, what's a long term lose?14:00
f13spoleeba: wiki?14:00
nottingspoleeba: having all the spin ks files go into livecd-tools14:00
jwbnotting, agreed14:00
spoleebaf13, i think we want to treat them as code, not content14:00
jeremynotting: indeed14:00
spoleebaf13, so people can patch it...equitable14:00
jeremy /cvs/configs!14:01
spoleebanotting, so how do we magically fix that?14:01
nirikmake livecd-tools have a spins.d/ and make new spins submit a ks file package for their spin? :)14:01
f13spoleeba: eh, code can be attached to a wiki page.14:01
f13spoleeba: for work in progress, thats not a terrible thing.14:01
jwbnirik, no.  that's what we want to avoid14:01
f13also, how do we drive bug reports regarding the .. spin.. to the spin owners?14:01
jwbnirik, because that would still be keeping the .ks files in the livecd-tools package14:01
nirikjwb: no...14:02
spoleebaf13, perhaps... there's a place in hosted for customspin developers to work together14:02
lmackenwhy not give them their own hosted so they can manage their own wiki and bug queue ?14:02
nirikI mean 'Review request: Xfce-spin' and make a package with the ks file.14:02
nirikit gets a bugzilla component, cvs, etc14:02
jwblmacken, it's not that clear cut14:02
nottingnirik: that's a lot of packages to keep around14:02
spoleebanirik, there are sort of two issues here... blessed spins..and in progress spins14:02
jeremylmacken: because pointing people to trac bug reporting is a disaster.  because the bugs will get filed in bugzilla first and you can't transfer between14:03
niriknotting: potentially, but we have lots of packages already, whats a few dozen more. ;)14:03
nirikspoleeba: sure, the in progress ones are not yet approved.14:03
spoleebanirik, i dont think we can give fedora bugzilla components to in progress ones... but perhaps a hosted space for those14:03
jwbthat's all too complicated14:04
lmackenjeremy: indeed14:04
nirikthey get a bugzilla component when the package is approved and added to cvs.14:04
f13things were much easier when we didn't sign up for these custom things (:14:04
nirikit's just an idea.14:04
jwbjust create a hosted.fp.o/spins project, give rel-eng access to it, and we can commit the spin .ks files there when we produce them14:04
spoleebajwb, its too complicated to try to organize all the people doing this as a group with a common code control system to help each other out?14:04
jwbspoleeba, no, i just said to do that14:05
* jeremy doesn't really think we're getting anywhere here (other than going in circles)14:05
spoleebajwb, oky..violent agreement14:05
jwbspoleeba, but one per spin, or bugzilla review requests, etc are just overkill IMHO14:05
spoleebajwb, that i would agree with14:05
f13ok, I think there are still some serious issues to work out14:05
spoleebaf13, like... wtf is releaseeng actually suppose to do14:06
f13and working them out in releng space is perhaps not the best, as this is more of a FESCo/Board issue14:06
jwbif we wanted, we could create a fedora-spin-config package from that spin project14:06
wwoodsalso if I start getting spins dropped in my lap with post-its14:06
spoleebaf13, i could still use input on what releaseeng would like to do and would absolutely not like to do with regard to spins14:06
wwoodsall "so how's testing of this going?"14:06
jwbspoleeba, we actually do the final compose14:06
jeremyjwb: one package with all the configs is not scalable really14:06
wwoodsI'll cut someone14:06
jwbjeremy, why is that?14:06
f13jwb: disjoint of when configs get updated14:07
* nirik doesn't think going thru a new package process for a spin would be that much overkill, but hey, it was just an idea... 14:07
jeremyjwb: it's painful enough just with desktop + kde + fel + developer.  it'd be like one package for every spec file14:07
spoleebajwb, right... the issue is how much say does releaseeng want/need in approval process14:07
jeremyI have an idea or two, let me have a few days to write them up (so that they're coherent rather than stream of consciousness on irc)14:07
jwbwait... what?14:07
nottingjeremy: but having them all separate and managing dependencies isn't much better14:08
jwbit's not that hard... changes in hosted, new tarball, update package14:08
f13Let jeremy spend a few days working on a writeup of how Custom Spins should be handled within the Fedora project14:08
f13releng will review, give input, etc...14:08
f13as part of a larger definition of how these things should work.14:08
jeremynotting: if there's a lot of inter-depending, that means something is being done wrong14:09
nottingf13: as an adjunct/revision of rahul's spin process?14:09
nottingnot sure i want competing proposals flying around14:09
spoleebanotting, no i dont want competing proposals.. but i need releaseeng's pov14:09
spoleebanotting, as to releaseeng's burden14:09
jeremynotting: rahul's page is decidedly stark in the way of actual details14:10
spoleebanotting, i plan to scrap rahuls and jeremys ideas and write my own anyways14:10
f13alright, now I'm really confused.  Is there a proposal from rahul actually written anywhere?14:10
lmackenlink ?14:10
f13ah, but ... stark... according to jeremy14:10
spoleebaf13, the draft..which is admittedly very undetailed14:10
jwbit basically says "write a .ks, get it approved by the board, give it to rel-eng"14:11
spoleebajwb, i had a sidebar with rahul...concerning bottlenecks..which is why im looking for releaseeng's pov concerning their burden in the approval process14:11
jwbi think burden is the wrong word to use14:11
f13ok, let me revise my proposal14:11
warrenWhat happened to the previous idea of "If it is comprised entirely of Fedora packages then it may be called Fedora?"14:12
f13Let jeremy spend a few days adding actual details to rahuls proposal intos omething we can actually discuss.14:12
f13warren: Board still has to give it the go ahead, as protection of the brand.  That guideline is just somethign to keep in mind when preparing your proposal.14:12
spoleebawarren, the devil is in the terms of making sure it doesnt take 3 months to get a spin approved and built14:12
jwbspoleeba, it's not that we have a massive burden.  i think we can enumerate what that would be fairly easily (most of it falling into time).  it's what we just don't know what we're supposed to do14:13
spoleebajwb, hey if its easy to enumerate thats great...14:13
f13spoleeba: we can't tell you what we're supposed to do if we don't know... what we're supposed to do.14:14
f13spoleeba: welcome to 'makeing it up as we go'14:14
spoleebajwb, its not enumerated yet.. and i think its best if i give releaseeng a chance to self assess before I decree as to what I expect.  Just like I wanted rahul to assess things from a creator pov14:14
jwbspoleeba, yeah.  and i think that's where we need jeremy's details14:14
nottingright. but <board hat on> the board does not want to be in the business of writing the whole policy </board hat>. ergo, this discussion14:14
f13spoleeba: NB: can you call it 'releng' if you want to shorten it?  I have a hard time reading 'releaseeng'14:14
spoleebaf13, don't worry to hard.. whatever i decree will be difficult for everyone to live with14:15
f13notting: I agree, but there is a difference between the board saying "We'd like releng to approve, produce, and host these custom spins" and saying "you must check x x x, discuss y, login to bar, press button x, and upload to zing"14:16
spoleebanotting, i have no problem writing whole policies..if everyone first swears the fealty to me on pain of death14:16
f13notting: what I'm looking for here is a high level 'What does the Board expect releng to do as part of fulfilling Custom Spins" so that releng can decide /how/ to do the /what/.14:16
jwbspoleeba, be sure those come with a mission statement14:16
spoleebaf13, okay... ill work on that14:17
spoleebaf13, we'll meet somewhere in the middle on round two14:17
f13k, we're over meeting time now.14:17
nottingf13: mirror master move is still wednesday?14:18
f13can we at least agree to "table" this discussion until next week?14:18
f13notting: I don't see a reason why not.  You I and lmacken really need to create a punch list of what we're going to do.14:18
jwbf13, i should still do the spins if mmcgrath gets me spinners?14:18
nottingf13: when on wednesday is the switch?14:19
f13jwb: what spins are pending?14:19
f13notting: that's a mmcgrath question, I think it's in the ticket14:19
jwbf13, XFCE would be the one i would start with.  then there's the list of localized ones14:19
f13notting: I don't think an exact time was picked.  We should pick one14:20
nottingmmcgrath: got a time from the ticket?14:20
f13jwb: I'd say go ahead, just to "clear the decks", with the understanding that what we do this time isn't a precident for what we'll do in the future.14:20
nottingf13: well, given the length of syncs, i'm thinking a) turn off rawhide tuesday night. b) turn off everything else on wednesday14:20
nottingf13: that is: 1) updates 2) epel 3) extras14:21
jwbf13, yeah.  it'll help me write the SOP too14:21
mmcgrathnotting: ?14:21
f13notting: ok, that works for me.  We could continue this on #fedora-admin14:21
mmcgrathoh oh14:21
mmcgrathI never did, I'll follow up now.14:21
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering - Open Discussion14:21
jwbclumens wants boot isos14:22
f13is there any other pressing items to discuss today (before we adjurn to #fedora-admin to discuss reversing the netapp stream)14:22
jwbdo we know why we don't have boot isos?14:22
f13jwb: we didn't get boot.isos today due to system-config-firewall dep issues.14:22
jwbah, right14:22
f13which have already been fixed14:22
f13I could theoretically create them by hand for him later today, but he seemed ok to wait for the rawhide attempt tomorrow14:22
jwbis the theory that beta freeze will help that a bit?14:23
jwbcause i think for the anaconda team that's sort of important, no?14:23
f13help what?14:23
jeremyjwb: something that's going to help even more is I'm trying to get us a script to do "here's a tree of packages, build an netinstall.iso"  (which is what I've started calling rescuecd.iso)14:24
jwbf13, help them test...14:24
nottingjeremy: are you attacking mk-images/upd-instroot with a rusty chainsaw?14:24
jeremynotting: that's the general idea, yes14:24
f13jwb: freezes help cut down the churn yes, and the unexpected changes.14:24
jeremyI started a little on friday.  tomorrow will probably be when I get back to it14:25
jeremy(or late tonight)14:25
* jwb has to go to a real meeting14:25
f13perl + gcc in the next couple weeks isn't going to help much with that whole 'stability' thing14:25
jwbmaybe we need a --force-fscking-boot-iso switch14:26
jeremyjwb: please remove boot.iso from your vocabulary... it's a bad word ;-)14:26
jeremyjwb: and that's harder to do since we switched to using yum instead of putting together images on our own14:26
nottingjeremy: nice. as part of it, please rename instimg, img-template, etc. to actually correspond to what they are (stage1, stage2, etc.)14:26
jwbwhat would you like me to call it?14:27
jeremyjwb: netinstall.iso is the running best candidate.  because it needs the kernel bits (boot.iso) *and* the stage2.img to really be useful14:27
f13and we just ignore that you can install from media with netinstall.iso (:14:28
jwbbut you all know what i mean14:28
warrenthat's what it does?14:28
f13ok, ending the meeting in 30 seconds unless somebody barks.14:29
jwbi think we should name it anaconda.iso14:29
nottingjwb: and we can drop the scripts entirely and create it with anaconda.ks using livecd-creator14:30
jwbi'm too ignorant to get that joke, but ok14:30
nottingwho said it was a joke?14:30
jwbsee.  ignorant.14:31
f13ok, meeting over.14:31

Generated by 2.3 by Marius Gedminas - find it at!