Infrastructure/Mirroring/Tiering

= Tiering =

Fedora mirror servers use Tiering, whereby a select few fast mirrors get read access to the master rsync servers, and all the other mirrors pull from those mirrors.

It turns out, 9 of our 10 Tier 1 mirrors are available over Internet2. And, over half of our total mirrors are reachable over Internet2. So, let's make use of that whereever we can.

For our purposes, define:
 * master: The Fedora-owned servers dl.fedoraproject.org and download-i2.fedoraproject.org
 * Tier 1: The fast mirrors which pull from a master mirror.
 * Tier 2: The mirrors that pull from the Tier 1 servers.

Properties of Tier 1 mirrors:


 * Limit the number of Tier 1 mirrors, to ensure adequate bandwidth for these. Adjust number up or down depending on capability of the masters.
 * Must carry everything under fedora-enchilada and fedora-epel. This allows Tier 2 mirrors to exclude what they wish, but get everything if they so wish.  This means at least 1TB of disk space for the Fedora portion of this server.
 * Must have a 1 Gigabit connection to the Internet, or faster.
 * Must have an active, available, responsive mirror administrator during the days content is staged.
 * Must have at least 2 Internet2-connected Tier 1 mirrors.
 * Must have at least 1 Tier 1 mirror on each continent for which we have Tier 2 mirrors
 * Must serve private rsync (see below for configuration)

Master Mirrors

 * dl0[12345].fedoraproject.org, in Phoenix, AZ, USA.
 * dl.fedoraproject.org is a DNS round-robin to dl[12345].
 * download-i2.fedoraproject.org in Raleigh, NC, USA (Internet2, NLR, and those reachable over NLR only) This is the preferred master mirror for downstreams reachable on Internet2.

Master Mirror rsync modules
The master mirrors provide two additional rsync modules which provide pre-bitflip content. Fedora tiered mirrors should use these modules to be able to get pre-bitflip content.

Tier 1 Mirrors
Tier 1 mirrors pull from one of the master mirrors.

Tier 1 Rsync configuration
Below is an example rsyncd.conf file for a Tier 1 mirror that provides private rsync access to select downstream Tier 2 mirrors. You may do this via either IP or DNS-based access control, or by a shared username/password which you give to your selected Tier 2 mirrors directly.

The key to this is that the Tier 1 mirror rsyncs content using a user account (e.g. mirror used below), and you serve content to Tier 2 mirrors using a private rsync module that runs as that same user account, while providing public non-authenticated rsync using the nobody account. In this way, Tier 2 mirrors may obtain content before the permissions are made world readable.

use chroot = yes uid = nobody gid = nobody dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.bz2 *.iso *.ogg *.ogv *.tbz exclude = .snapshot/ .~tmp~/ /.private/ /.private/** **/.nfs* ignore nonreadable = yes list = true read only = yes refuse options = checksum

[ fedora-enchilada ] comment = Fedora - The whole enchilada path = /srv/pub/fedora

[ fedora-epel ] comment = Extra Packages for Enterprise Linux path = /srv/pub/epel


 * 1) The following are not seen and are limited by IP.
 * 1) The following are not seen and are limited by IP.

[fedora-enchilada0] comment = Fedora Enchilada for Tier0|1 Mirrors path = /srv/pub/fedora/ list = no      uid = mirror gid = mirror hosts allow = (IP or DNS address) ...

[fedora-epel0] comment = Fedora EPEL for Tier0|1 Mirrors path = /srv/pub/epel/ list = no      uid = mirror gid = mirror hosts allow = (IP or DNS address) ...