Infrastructure FedoraHosted Version2 Notes

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(Here are a number of items we need in order to make this happen)
(add some more notes)
Line 22: Line 22:
  
 
== Here are a number of items we need in order to make this happen ==
 
== Here are a number of items we need in order to make this happen ==
 
 
  
 
* <strike> setup fedorahosted.org's domain zone as a template and allow it to handle a map of project name to destination</strike>
 
* <strike> setup fedorahosted.org's domain zone as a template and allow it to handle a map of project name to destination</strike>
Line 34: Line 32:
 
** last commit to dvcs  
 
** last commit to dvcs  
 
** last ticket or wiki change
 
** last ticket or wiki change
 +
** (we should have all this info in various places, just need to gather it together)
 
* backups - how should we be backing up these smaller hosted instances
 
* backups - how should we be backing up these smaller hosted instances
 +
** Data is trac, releases, git/scm.
 +
** rdiff-backup? rsync?
 +
** storage local to the instances would be much faster.
 
* git:// proxy - we need to setup a test a way of proxying the connections to git://fedorahosted.org/git/project.git to    git://project.fedorahosted.org/git/project.git  
 
* git:// proxy - we need to setup a test a way of proxying the connections to git://fedorahosted.org/git/project.git to    git://project.fedorahosted.org/git/project.git  
 
** this may be gitpod or dulwich based but we have to do SOMETHING here
 
** this may be gitpod or dulwich based but we have to do SOMETHING here
 +
** we may want a sunset on this after a year or something.
 
* trac needs to be setup to use the authopenid auth mechanism by default for new instances
 
* trac needs to be setup to use the authopenid auth mechanism by default for new instances
 
* trace authopenid needs to be tested against older trac repos to see if it breaks ticket/wiki ownership
 
* trace authopenid needs to be tested against older trac repos to see if it breaks ticket/wiki ownership
 
* the /usr/bin/run-git user script used for folks connecting needs to be cleaned up and tested against projectname.fedorahosted.org urls
 
* the /usr/bin/run-git user script used for folks connecting needs to be cleaned up and tested against projectname.fedorahosted.org urls
 +
* figure out how fasClient will need to be configured on each shard.
 
* git.fedorahosted.org/cgit/ needs to have its index page populated and pointing to the right set of urls/repos for all projects
 
* git.fedorahosted.org/cgit/ needs to have its index page populated and pointing to the right set of urls/repos for all projects
 +
** Or could we have project.fedorahosted.org/cgit just show that single projects scm?
 +
** we do likely need an index of all projects tho.
 
* when making new projects, how do we figure out where something should go?
 
* when making new projects, how do we figure out where something should go?
 
* locating a project - assumedly just ssh'ing to it's projectname.fedorahosted.org hostname
 
* locating a project - assumedly just ssh'ing to it's projectname.fedorahosted.org hostname
Line 48: Line 54:
 
** maybe searchable?
 
** maybe searchable?
 
* How do we make a new project - how 'self service' can we make it?
 
* How do we make a new project - how 'self service' can we make it?
 +
** I think we can visit that down the road some, since it's not critical right now.
 
* How to document and handle some optional web apps for projects
 
* How to document and handle some optional web apps for projects
 +
* Enable way for projects to have html/web space. (existing ticket already on this)
 
* reviewboard - what do we do with it and who maintains it? Where should it live? Which projects use it?
 
* reviewboard - what do we do with it and who maintains it? Where should it live? Which projects use it?
 +
** I think we can ask them to move to openshift instances.
 +
** sgallagh maintains it.
 
* wildcard ssl cert
 
* wildcard ssl cert
 +
** Being ordered.
 
* the ssh key(s) we'll need for things (probably the same one for all of hosted, I'd bet)
 
* the ssh key(s) we'll need for things (probably the same one for all of hosted, I'd bet)
 
* eventually test out gitolite instead of locally created shell accounts for users
 
* eventually test out gitolite instead of locally created shell accounts for users

Revision as of 16:46, 21 March 2013

FedoraHostedv2

Hosted needs a bunch of work done to it to make it all happy for everyone.

Leading problems:

  • performance
  • scaling out
  • managing older projects
  • what interfaces we want to offer/maintain
  • automation

The goal right now is to support sharding out as much of the projects into smaller batches to make them manageable on smaller instances/vms and to make them easier maintain b/c of scale, more or less.

The general idea is that every project will be listed by:

$projectname.fedorahosted.org

This should point to a cname of the host where the project's dvcs and web app live (dvcs at minimum).

This should allow us to move around projects more or less at will.

Right now our 2-instance infrastructure using gluster is not performing very well and we really need to increase our horizontal scaling.

Here are a number of items we need in order to make this happen

  • setup fedorahosted.org's domain zone as a template and allow it to handle a map of project name to destination
  • data we need from each project:
    • name
    • contact/leader
    • scm(s) used
    • using trac?
    • fas group(s) related
    • last commit to dvcs
    • last ticket or wiki change
    • (we should have all this info in various places, just need to gather it together)
  • backups - how should we be backing up these smaller hosted instances
    • Data is trac, releases, git/scm.
    • rdiff-backup? rsync?
    • storage local to the instances would be much faster.
  • git:// proxy - we need to setup a test a way of proxying the connections to git://fedorahosted.org/git/project.git to git://project.fedorahosted.org/git/project.git
    • this may be gitpod or dulwich based but we have to do SOMETHING here
    • we may want a sunset on this after a year or something.
  • trac needs to be setup to use the authopenid auth mechanism by default for new instances
  • trace authopenid needs to be tested against older trac repos to see if it breaks ticket/wiki ownership
  • the /usr/bin/run-git user script used for folks connecting needs to be cleaned up and tested against projectname.fedorahosted.org urls
  • figure out how fasClient will need to be configured on each shard.
  • git.fedorahosted.org/cgit/ needs to have its index page populated and pointing to the right set of urls/repos for all projects
    • Or could we have project.fedorahosted.org/cgit just show that single projects scm?
    • we do likely need an index of all projects tho.
  • when making new projects, how do we figure out where something should go?
  • locating a project - assumedly just ssh'ing to it's projectname.fedorahosted.org hostname
  • document most/all of the above in new fedorahosted docs - both for infradocs but also for users
  • reskinning the trac main webpage
  • making the index of projects at fedorahosted.org a little more manageable and a lot more fedora-y
    • maybe searchable?
  • How do we make a new project - how 'self service' can we make it?
    • I think we can visit that down the road some, since it's not critical right now.
  • How to document and handle some optional web apps for projects
  • Enable way for projects to have html/web space. (existing ticket already on this)
  • reviewboard - what do we do with it and who maintains it? Where should it live? Which projects use it?
    • I think we can ask them to move to openshift instances.
    • sgallagh maintains it.
  • wildcard ssl cert
    • Being ordered.
  • the ssh key(s) we'll need for things (probably the same one for all of hosted, I'd bet)
  • eventually test out gitolite instead of locally created shell accounts for users
  • Is there a way to redirect users commiting via git+ssh to fedorahosted.org to projectname.fedorahosted.org
  • How do we handle outages if these pieces are all over the place?
  • managing user expectations?
  • what fedorahosted.org should look like and have on it.
  • when we're ready we need some way of redirecting people from: fedorahosted.org/projectname/somepage to projectname.fedorahosted.org/somepage/

things to think about to make the future more manageable

  • switching to gitolite instead of restricted shell accounts?
    • dealing with /releases/ uploads if we did
  • gitlab-public - talk to osas people?
  • supporting only git cloning via the 'smart' http protocol and via ssh?
  • supporting git:// urls only for our already existing projects and decomming the rest
  • stopping taking projects which are:
    • svn?
    • bzr?
    • hg?
    • or should we continue supporting the world?