Infrastructure/Meetings/2006-10-26

= Meeting of 2006-10-26 =


 * Time shown in EDT

16:00 Lets get started. 16:00 Who's here? 16:00 * abadger1999 1/2 here 16:00 yo 16:01 * dgilmore is here 16:01 So FC6 is out (hurray) 16:02 running it at home already 16:02 same here. 16:02 oops! 16:02 -!- nman64 [n=n-man@fedora/nman64] has joined #fedora-admin 16:02 We never did use smtp.fedora did we 16:02 warren: not yet, there's a few final touches we need. 16:02 ok 16:02 But as far as I know, its clos. 16:02 warren: mostly it needs some testing 16:02 nman64: you here? 16:03 warren: did you want to put postgrey on it? 16:03 Why would test guests require ssh access? 16:03 dgilmore, sqlgrey preferably 16:03 warren: ok is that in extras now? 16:03 let me check.. 16:03 warren: pong 16:03 lmacken, sent you mail 16:03 k 16:03 * dgilmore will upgrade smtp to fc6 16:04 mmcgrath: Yeah. 16:04 we should upgrade all the xen guests to it also 16:04 as well as the hosts 16:04 uhh, is anyone else having issues bringing up the schedule page? 16:04 I updated the InfrastructurePrivate xen page install example too 16:04 dgilmore: i can upgrade test6 16:04 anybody with root access on xen1/2 should be able to easily install a guest. 16:05 lmacken: :) all yours 16:05 mmcgrath: nope 16:06 Ok, well first off lets talk about the VCS. 16:06 dgilmore: I don't know what box you touched, but I had to fix ssh when I got back to my desk (: 16:06 abadger1999: you and f13 were talking yesterday. 16:06 how are we going to cooridnate this with the developers? 16:06 f13, where? 16:06 mmcgrath, which aspect of what? 16:06 So far we only have one developer on each VCS. 16:06 f13: its acting goofy. the console keeps locking up 16:07 But we do need to coordinate what they're doing so they can learn from each other. 16:07 warren: the new VCS, I think f13 has shown a paticular interest in mercurial. 16:07 dgilmore, what is acting goofy? 16:07 mmcgrath: so I'm setting up a trial box with mercurial 16:07 warren: console on test1 16:07 (Although there's several new people expressing interest in svn to me.) 16:07 abadger1999: :) 16:07 mmcgrath: I want to figure out some tasks I need, and then post to various lists asking for folks to help me with what I need 16:08 things like make file adjustments, or plague edits. 16:08 brb 16:08 I don't think jeremy will ever accept svn. he doesn't see it as an improvement over CVS. 16:08 (at least not enough of an improvement to switch to it) 16:08 warren: thats the other question. 16:08 who's going to decide the new vcs? 16:08 -!- abompard [n=gauret@bne75-8-88-161-125-228.fbx.proxad.net]  has joined #fedora-admin 16:08 Not us. 16:09 should it be completely community driven?  Or will we leave this to the core dev's and maybe the FESCo? 16:09 Fedora Extras? 16:09 FESCo? 16:09 It probably goes down to whoever implements a proof of concept that satisfies all goals first. 16:09 but who decides the goals?  I mean, everyone will want everything. 16:09 warren: No testing? 16:09 The VCS goals have been on that web page for a while now. 16:10 For scalabilty and user interface feel? 16:10 Essentially what package CVS currently does, except with the ability to have fine grained ACL's and distributed operation. 16:10 i think the Extras/Core developers should decide 16:10 testing would be manditory 16:10 user interface actually isn't important, abstraction layers can make it look anyway we want later 16:10 the core capabilities are what is important 16:10 but thats what the proof of ocncepts are for 16:11 import a big chunk of packages and have people use it and abuse it for a bit. 16:11 Anyway, Jeremy suggested that we (RH) finally sit down and focus on this when Max visits the Westford office on the week of November 6th. 16:11 I've never quite gotten whats required with 'distributed operation' 16:11 someone educate me :D 16:11 f13: test1 is back 16:11 dgilmore: I know, I fixed it myself. 16:11 * f13 wonders where the disconnect is. 16:11 f13: i just rebooted it 16:12 ?! 16:12 why? 16:12 a feww minutes ago 16:12 VCS is a big first step, part of many pieces necessary for the merge. Our RH meeting November 6th is really the first time RH has the room to breathe after FC5 and FC6 and focus on this problem. 16:12 Meanwhile, we can provide proof of concept examples of how mercurial or (something else) is awesome for package VCS needs. 16:13 * f13 discovers hg grep and wets himself. 16:13 Ok, should we continue this on the list or do people have more to discuss right now? 16:13 Jeremy suggested that we will fail if we attempt to implement everything at once. (Everything being VCS, package database, next gen account system, etc.) 16:13 lets discuss on the list 16:13 This is just my status update. Conclusion is infrastructure team needs to provide proof of concept implementations 16:14 November 6th is the target date 16:14 warren: Jeremy's right on that. I've been way too stretched just dealing with two of those areas. 16:14 nod 16:14 K, so the VCS is priority 1 then. 16:14 I think VCS will just happen with all packages (Core + Extras), and we'll figure out the ACL policy at that point. 16:14 ACL will be course at first, and later we can make it well controlled. 16:15 ACL stuff will depend on what VCS we use 16:15 nod 16:15 Ok, regarding the package database. Whats the word? 16:16 I didn't work on it this weekend: c4chris, did you? 16:16 lmacken: anything? 16:16 on the package database? no 16:17 heh, sorry 16:17 -!- rmm [i=keefejoh@gateway/web/cgi-irc/ircatwork.com/session] has joined #fedora-admin 16:17 -!- kimo [n=ahmed@196.202.18.251] has quit ["rebooting for FC6"] 16:17 lmacken: Did you get any sense of how hard multiple projects from one TG server would be? 16:17 -!- rmm [i=keefejoh@gateway/web/cgi-irc/ircatwork.com/x-cbf51c6b60953a39] has quit [Client Quit] 16:18 Ok, how about the new accounts system 16:18 lyz: ? 16:18 abompard: ? 16:18 abadger1999: i didn't get a chance to read that thread yet, but i'm sure we'll figure something out 16:18 -!- rmm [i=keefejoh@gateway/web/cgi-irc/ircatwork.com/session] has joined #fedora-admin 16:18 FDS is up with the schema 16:18 no news from my front..... busy time 16:18 cool 16:18 I sent a screenshot to the list 16:18 Saw the screenshot. 16:18 where do you want to proceed from here? 16:18 next task is export from the db and import to LDAP 16:19 I may need a dump of the current db 16:19 to test with 16:19 abadger1999, got pretty swamped, and my network ADSL broke down :( 16:19 Hmmm, can you create your own to test with for now? 16:20 yeah,, but it's not the best way to do it 16:20 brb 16:20 Yeah. I'll come up with something for you. 16:20 -!- rmm [i=keefejoh@gateway/web/cgi-irc/ircatwork.com/x-278c16f77914ca43]  has quit [Client Quit] 16:21 thanks 16:21 I started to grab old review data from the ml and put some data on test3 16:21 After I get an importer going, we can start testing some of our systems on it 16:22 tickets perhaps? 16:22 lmacken: We could accept that all projects would live in the same DB -- I just like paritioning it because it seems more organized, secure, and easier for new users to get a handle on. 16:22 lyz: What about porting the Accounts API? 16:23 abadger1999 what's the accounts API,  is that the web site? 16:23 lyz: nope, thats all you ;-) 16:23 step 2. 16:24 We'll need to provide everyone with an easy way to send requests to the system. Perhaps code snipits in some of the primary languages (python, perl, bash?) 16:24 Just curious, of the new guys I've only had a couple actually come to me asking for work. Of the other officers have you been contacted? 16:24 mmcgrath: no one has contacted me 16:24 isn't LDAP the API? 16:24 lyz: Sorta. The website uses it, but at least the voting app uses functions from website.py to authenticate the users. 16:25 mmcgrath: Two people contacted me. 16:25 lyz: yeah, but we should set it up so that people can use LDAP in our situation for authentication and authorization. 16:25 abadger1999 I see. I'll check that out 16:25 mmcgrath: i've been contacted a couple of times 16:25 Both interested in the VCS. 16:25 not recently though 16:25 thats something at least. 16:26 Websites always has people interested in helping. 16:26 I suggested tickets as a sample cause it just uses apache pgsql authentication. Should be easy to move to LDAP 16:26 nman64: Any of them coders? 16:26 nman64: I sent an email to DaMaestro but so far haven't heard back from him about sponsorship :( 16:26 abadger1999: Haven't seen any Python coders that I can recall. 16:27 Ok, I only have a couple of more things. 16:27 1) network stuff on the xen guests. 16:27 abadger1999: Plenty with basic knowledge of PHP, though that's not useful for our purposes. 16:27 what would we like stacy to do? 16:27 mmcgrath: What does he need to do? 16:29 mmcgrath: probably port 80 and 443. and maybe on two of them 8888 and 8889 16:30 then we could have a test on each xen host for building 16:30 he keeps pushing the plone instance so we're giving it to him, the doc guys have been wanting it too. 16:30 nman64: We might be able to have them work on front end stuff (accounts interface, packageDB interface) since I'd like to use kid templates within TurboGears. The backend isn't formalized enough to be able to get them started yet, though. 16:30 My only concern is that he doesn't seem to be coordinating with anyone 16:30 unless we wanted ports for different VCS's 16:30 mmcgrath, f13: Do we need more ips for people to connect to ssh on the xenguests? 16:30 Similar to cvs-int and cvs-ext? 16:30 -!- rannis [n=lycos@did75-5-82-224-183-241.fbx.proxad.net] has quit [Remote closed the connection] 16:31 abadger1999: I have no idea. 16:31 -!- warren [i=warren@redhat/wombat/warren] has quit [Read error: 104 (Connection reset by peer)] 16:32 abadger1999: we can always give some boxes multiple test ip's. 16:32 -!- warren [i=warren@nat/redhat/x-9dade552838d22ff] has joined #fedora-admin 16:32 we'll just have to see how it goes. 16:32 do we want to talk about hammer3? 16:32 warren: how's that sound 16:32 abadger1999: That sounds like a good idea. Whenever you see an opportunity for an ambitious volunteer to help out with simple tasks, let me know. We might start a wiki page full of tasks within Websites to point them to. 16:33 sorry, lost my connection =( 16:33 mmcgrath: That should be fine as long as we're testing internally. 16:33 Stacy said in private e-mail he'll take a look at hammer3 16:33 at this point I don't think it is important 16:33 next topic 16:33 warren: :D cool   its not really 16:33 mmcgrath: We're down to two ip addresses in the list on the wiki. 16:33 BTW, I need to leave for another meeting at 5pm sharp 16:33 wow, that went QUICK. 16:34 I want to talk about network addresses 16:34 warren: your floor 16:34 Should we request more test addresses? 16:34 k, we'll go quick. 16:34 warren: we're talking about it now, what do you have in mind :D 16:34 warren: probably yes 16:34 we need to put a limit on the number 16:34 OTOH, we will have capacity to run even more guests when the two new dell servers come in 16:34 yes 16:34 +1 16:34 or should I say +10 16:34 I'm pretty sure VCS prototypes need two IPs apiece. 16:34 why two? 16:35 warren: what do you think the hosts will support as guests ? 4 on xen1 and 8 on xen2? 16:35 internal and external ssh 16:35 Like the cvs server has now 16:35 Or do you have another plan? 16:35 I strongly believe that the future is best to run ALL services within xen guests. 16:36 at some point the mercurial box will need external people to log in to it 16:36 and be able to get http or ssh repo checkouts 16:36 warren: in our infrastructure I'm fine with that. 16:36 Before we add +10 test addresses, let us think about how test* addresses are to be used. 16:36 warren: so do I, we should test the migration of guests between the hosts 16:36 dgilmore, I think we would need SAN for that? 16:36 one option... 16:36 f13: So make the second IP external right from the start? 16:37 warren: actually it would be easier in our xen environment if we used files instead of partitions. 16:37 Small performance hit, but not much. 16:37 then we're an scp away.l 16:37 test1-20 have both an internal and external IP address, with port 80, 443, 8888 and 8889 forwarded by default. 16:37 warren: test ip's  should be throwaway systems 16:37 They are to be used for TEST purposes until we are ready to launch a production service. At that point we request a new internal and external IP assignment and new names. 16:37 dgilmore: +1 16:38 Then change the IP address of the test guest to the new address. 16:38 abadger1999: I"d thin kso yes. 16:38 warren: if we know  we are going to have a production system  why not request it from the start 16:39 like we did with smtp 16:39 *but*... is 80 and 443 sufficient to allow users to test mercurial? 16:39 we know we want db2  request it 16:40 dgilmore, does db2 require external IP and port forwards?  (i'm guessing no) 16:40 -!- tibbs [n=tibbs@fedora/tibbs]  has quit [Remote closed the connection] 16:40 warren: no it shouldn't 16:40 uhh, what happened to hammer1? 16:40 ?? 16:40 hammer1  was ok last i checked 16:40 warren: I think port 80 and 443 are good defaults.  mercurial and bazaar would need 22 as well. 16:40 ssh wtogami@hammer1.fedora.phx.redhat.com 16:40 ssh_exchange_identification: Connection closed by remote host 16:40 try it now. 16:41 hmm... what happened indeed 16:41 thats the first sign i got on hammer3 16:41 abadger1999, would that port 22 be a different sshd than the one used to login to the system itself? 16:41 Since commit access needs to work over ssh. 16:42 dgilmore, maybe the hammer's are committing suicide at the same time... 16:42 warren: Yes. 16:42 should we run ssh on a nonstandard port? 16:42 warren: i hope not 16:42 OK, I suppose this is fine for now. 16:43 maybe the warranty expired on them last week 16:43 hehe 16:43 well at least we have the xen boxes. 16:43 oh man. 16:43 hammer2 is OK so far? 16:43 Ok, so honestly I don't have anything else required for this meeting. its been a long, long week. 16:43 Anyone else have it. 16:43 err anyone else have anything? 16:43 looking at how dist-cvs is setup, since EVERYTHING is the same repo, conversion is going to be a BITCH 16:43 yeah 16:44 just a quick status update on the updates system 16:44 that is conversion with keeping any kind of history 16:44 I did a little bit of hacking on the updates system this week (not as much as I would have liked, due to exams), and I hope to have a bunch of code checked into CVS in the very near future. 16:44 I also updated the Infrastructure/UpdatesSystem page with screenshots of the current system (RH internal), an ideal workflow design (comments/suggestions are encouraged), and where to get the new code. Once I get a solid codebase and some unit tests committed, we should have plenty of tasks to go around. 16:44 f13: yep. 16:44 and I'll hopefully have our test6 xen guest up and running as our TG application server soon as well. 16:44 nagios is reporting the problem on hammer1, but not on hammer3. 16:44 -!- BobJensen is now known as BobJensen-Away 16:44 f13: I was thinking about that a bit. 16:44 nman64: we acknowledged hammer 3 last week sometime. 16:44 Does it help if we just save the tagged history? 16:44 lmacken: thats cool. 16:45 Plus whatever's in the head revision 16:45 I should request port 22 to be opened to test1 now? 16:45 abadger1999: possibly. 16:46 hammer1 looks to be ok 16:47 hrm, let me try something... 16:47 dgilmore: echo | nc hammer1 22 16:47 says different :D 16:47 warren: and test2. If you want all prototypes in a week and a half. 16:47 mmcgrath: well hardware wise 16:48 [root@hammer1 ~] # /etc/init.d/sshd restart 16:48 Stopping sshd: [ OK  ] 16:48 Starting sshd: Privilege separation user sshd does not exist 16:48 [FAILED] 16:48 Hmm... inconsistent firewall policy between test1/2 and the other test boxes. 16:48 * iWolf wanders back in 16:48 dgilmore, that's special! 16:48 snmp is responding. 16:48 warren: yeah i just saw selinux logs of it 16:48 some FC5 update broke it? 16:48 dgilmore: did a package build do that by chance? 16:48 err, check /etc/passwd /etc/shadow 16:49 i'm not able to ssh into that box at all 16:50 dgilmore: you on the console? 16:50 mmcgrath: yeah i am 16:50 grep ssh /etc/passwd 16:50 ? 16:51 no ssh in /etc/passwd 16:51 i'm on KVM now 16:51 * lmacken has to head out for a bit 16:51 * lmacken & 16:51 [root@hammer1 ~] # grep ssh /etc/passwd 16:51 [root@hammer1 ~] # 16:51 warren: Are firewall configs for xen guests checked into fedora-config? 16:52 dgilmore: less /etc/passwd find out why its not in there. 16:52 abadger1999: there are no pyroman configs for xen guest yet. they should be pretty trivial to make though 16:52 abadger1999, AFAIK no configs of xen guests are checked into fedora-config 16:52 fedora-config itself does not match our servers too well 16:52 one of the things we need to do is re-explore how we will handle config management 16:53 and talk to skvidal about his awesome system 16:53 yeah, dist-conf is pretty painful 16:53 yeah, skvidal is trying to get his system gpl'd. 16:53 warren: its not 'awesome' its simple :D 16:53 warren: I made a start on using fedora-config for the xenguests 16:53 ah 16:54 hmm.... according to yum.log the last time something changed on the buildhost of hammer1 was september 24th 16:54 it is a little worrisome that something somehow broke sshd in this way 16:54 *HOW* did this happen? 16:54 So I could send changes to all the relevant guests instead of individually configuring. 16:54 dgilmore: ls -l /etc/passwd 16:55 -rw-r--r-- 1 root root 637 Jul 6 01:58 /etc/passwd 16:55 that might be the default passwd as shipped in the setup package? 16:55 Perhaps somehow mock screwed up and modified stuff in the / ? 16:55 wow. 16:56 dgilmore: there should be a sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin 16:57 openssh-server is not even installed here! 16:57 abadger1999: ok, I'm able to convert a single module (rpms/yum) to an HG repo 16:57 abadger1999: now, how this all sorts out is goign to be... interesting. 16:57 ok ive reinstalled sshd  its up now 16:57 gotta go to my next meeting, bbl 16:58 later warren: 16:58 does anyone else have anything for the meeting or should we just get to hammer? 16:58 hammer the hammer 16:58 mmcgrath: just the hammer 16:58 mmcgrath: meeting done 16:58 < iWolf> mmcgrath: I have nothing else. I can grab the logs later and email them to the lsit. 16:59 f13: Great! Do you have that scripted? 17:00 --- MEETING END ---