Infrastructure/MirrorManagement
From FedoraProject
Contents |
Fedora Mirror Management System
Please also see for how to become a Fedora mirror.
System Requirements
- Web-based
- Provide a mechanism for mirrors to provide their own data, including, but not limited to:
- Architectures carried
- Releases carried
- Whether or not Extras is carried. Would this be obsolete? thl: only for F7 and later; and there is also epel; Further: maybe other projects arise in the future that need to be handled...
- Whether or not development branch is carried
- Whether or not source packages and images are carried?
- Whether or not debuginfo for a branch is carried
- IPs of servers used to pull from the masters (for the rsync ACLs), allow both IPv4 and IPv6 addresses
- private (to other mirrors only) rsync IP/username/passwords for mirror<->mirror syncs
- If the mirror is company-internal private or worldwide-public (a few large companies have their own internal mirrors that pull from the masters)
- bandwidth available for downloading
- per-host geographical hints to override GeoIP-based data if needed
- method for mirror sites to announce they are synchronized prior to the public bitflip (after which automatic sync test methods will work)
- approval method for new mirror sites (to prevent leaks through bogus mirrors) - tie a site to a user in the Fedora Account System, and approve users in the FAS into a mirrors group.
- multiple FAS accounts can manage multiple mirror sites (M:N relation)
- A single mirror site may have multiple hosts (e.g. mirror1.foobar.com, mirror2.foobar.com)
- Sites can indicate if they'll automatically pick up new Releases (test, final, architectures etc.) so system can automatically add the right fields to their account when a new Release is posted.
- Sites indicate where they pull from, to let the system know about tiered mirrors.
- Sites provide an email address that, when receiving a signed message from the mirror system, are notified of changes to a Release and that they can start rsync'ing again. This allows pushes as well as polling.
- Sites can provide their rsync/ftp/http statistics to showcase the value the global mirrors bring to Fedora.
- Automatically generate
- Mirror lists for test and final releases
- Mirror lists for yum and up2date, including geographic-specific lists. Do we need up2date support anymore? Dont think we need up2date support - RahulSundaram
- IP ACL list for mirror<->mirror syncing
- mod_rewrite redirects for download.fedoraproject.org
- Mirror validation
- Automatically verify the synced state of mirrors, and remove them from lists where needed
Workflow
This is how Matt expects it to work in the end, not that it does today. :-)
Add a new mirror site
- Create a user account in the FAS:
- Log into the mirrormanager website as your FAS user.
- Create a new mirror site with the requested details. This site won't be live until approved. Some stuff asked will be:
- mirror site name
- mirror site admin(s) FAS account names
- where is mirror located network-wise?
- system emails mirror-list@redhat.com with above information for approval.
- Mirrormanager admin approves new site
- user account email automatically subscribed to mirrors-list and mirrors-list-d lists.
- as additional users are added to a particular Site, those emails are also added to these lists.
- wait 24 hours for rsync ACLs to update
- begin syncing
- site notifies system that sync is complete (using client-side tool)
- site added to proper Release lists externally visible
Add a new Release (embargoed until a certain date)
This is much the same as it works today, however the addition of mirrormanager lets us send emails to the sites and have them start syncing immediately.
Before embargo is lifted:
- Mirrormanager admin creates the Release
- rsync manager posts the release to the master rsync server
- download[123] .fedora.redhat.com automatically mirror everything from master
- sites notified by email that a new Release is available on download[123] .
- or sites begin picking up new Release on their regular sync
- site notifies system that sync is complete (using client-side tool)
When embargo is lifted, repeat from step 2 to get the bitflip.
Client-side tool
There are several instances where we'd benefit from having a client-side tool that runs on the mirror site immediately after rsync is finished. This lets us cleanly get around the fact that when embargoed content is posted, it's not world-readable, therefore no Fedora-side process can see what a given mirror has, until after the bitflip and subsequent rsync to pick up the bitflip. Having a client-side tool that can send mirror status back up to the MirrorManager database before the bitflip solves this. I envision an XMLRPC-over-HTTPS process using a mirror site's login/password (not a user's login/password) for this.
Code
https://fedorahosted.org/mirrormanager/wiki/WikiStart
If you are interested in contributing to this project, swing by #fedora-admin on irc.freenode.net and ask for more information. Farshad Khoshkhui, MattDomsch, MikeMcGrath, and LukeMacken have expressed interest in working on this.
