Infrastructure/Mirroring/Tiering

From FedoraProject

< Infrastructure | Mirroring(Difference between revisions)
Jump to: navigation, search
(ibiblio dropped acls)
m (Master Mirrors: fix missing number)
(22 intermediate revisions by 6 users not shown)
Line 5: Line 5:
 
Fedora mirror servers use Tiering, whereby a select few fast mirrors get read access to the master rsync servers, and all the other mirrors pull from those mirrors.
 
Fedora mirror servers use Tiering, whereby a select few fast mirrors get read access to the master rsync servers, and all the other mirrors pull from those mirrors.
  
It turns out, 9 of our 10 Tier 0 or Tier 1 mirrors are available over Internet2.  And, over half of our total mirrors are reachable over Internet2.  So, let's make use of that whereever we can.
+
It turns out, 9 of our 10 Tier 1 mirrors are available over Internet2.  And, over half of our total mirrors are reachable over Internet2.  So, let's make use of that whereever we can.
  
 
For our purposes, define:
 
For our purposes, define:
* '''master''': The Red Hat servers download*.fedora.redhat.com.
+
* '''master''': The Fedora-owned servers dl.fedoraproject.org and download-i2.fedoraproject.org
* '''Tier 0''': The fast mirrors which pull from Red Hat's Internet2-connected master
+
* '''Tier 1''': The fast mirrors which pull from a master mirror.
* '''Tier 1''': The fast mirrors which pull from the Tier 0 servers (or one of the other masters).
+
 
* '''Tier 2''': The mirrors that pull from the Tier 1 servers.
 
* '''Tier 2''': The mirrors that pull from the Tier 1 servers.
  
Properties of Tier 0 and 1 mirrors:
+
Properties of Tier 1 mirrors:
  
* Limit the number of Tier 1 mirrors to 10, to ensure adequate bandwidth for these.  Adjust number up or down depending on capability of the masters.
+
* Limit the number of Tier 1 mirrors, to ensure adequate bandwidth for these.  Adjust number up or down depending on capability of the masters.
 
* Must carry everything under fedora-enchilada and fedora-epel.  This allows Tier 2 mirrors to exclude what they wish, but get everything if they so wish.  This means at least 1TB of disk space for the Fedora portion of this server.
 
* Must carry everything under fedora-enchilada and fedora-epel.  This allows Tier 2 mirrors to exclude what they wish, but get everything if they so wish.  This means at least 1TB of disk space for the Fedora portion of this server.
 
* Must have a 1 Gigabit connection to the Internet, or faster.
 
* Must have a 1 Gigabit connection to the Internet, or faster.
Line 21: Line 20:
 
* Must have at least 2 Internet2-connected Tier 1 mirrors.
 
* Must have at least 2 Internet2-connected Tier 1 mirrors.
 
* Must have at least 1 Tier 1 mirror on each continent for which we have Tier 2 mirrors
 
* Must have at least 1 Tier 1 mirror on each continent for which we have Tier 2 mirrors
* Must serve private rsync
+
* Must serve private rsync (see below for configuration)
  
== Tier 0 Mirrors ==
+
== Master Mirrors ==
 +
* dl0[12345].fedoraproject.org, in Phoenix, AZ, USA.
 +
** dl.fedoraproject.org is a DNS round-robin to dl0[12345].
 +
* download-i2.fedoraproject.org in Raleigh, NC, USA (Internet2, NLR, and those reachable over NLR only)  This is the preferred master mirror for downstreams reachable on Internet2.
  
Tier 0 mirrors can pull from Red Hat directly over the Internet2 connection.
+
== Master Mirror rsync modules ==
 +
The master mirrors provide two additional rsync modules which provide pre-bitflip content.  Fedora tiered mirrors should use these modules to be able to get pre-bitflip content.
  
 
{| border="1"
 
{| border="1"
 
|-
 
|-
| Server || Comment || Contact for ACL
+
| module name || content
 
|-
 
|-
| fedora-archives.ibiblio.org || Internet2 / National Lamba Rail (NLR) connected hosts. || Don Sizemore <dls at metalab.unc.edu>.  No ACLs - open for syncing.
+
| fedora-enchilada0 || Everything under /pub/fedora/, including pre-bitflip content
 
|-
 
|-
| archive.linux.duke.edu ||Internet2.  Uses ACL from MirrorManager database. || Drew Stinnett <drew.stinnett at duke.edu> (spacepope on IRC)
+
| fedora-epel0 || Everything under /pub/epel, including pre-bitflip content (even though EPEL doesn't do bitflips
 
|}
 
|}
  
 
== Tier 1 Mirrors ==
 
== Tier 1 Mirrors ==
  
Tier 1 mirrors pull from one of the Tier 0 mirrors.
+
Tier 1 mirrors pull from one of the master mirrors.
  
 
{| border="1"
 
{| border="1"
Line 44: Line 47:
 
| Server || Comment || Contact for ACL
 
| Server || Comment || Contact for ACL
 
|-
 
|-
| mirrors.kernel.org || USx2, SE, NL || <ftpadmin at kernel.org>
+
| fedora-archives.ibiblio.org || Internet2 / National Lamba Rail (NLR) connected hosts. || <fedora-admin@ibiblio.org>  No ACLs - open for syncing.
 +
|-
 +
| archive.linux.duke.edu ||Internet2.  Uses ACL from MirrorManager database. || Drew Stinnett <drew.stinnett at duke.edu> (spacepope on IRC)
 +
|-
 +
| kernel.org || mirrors1.kernel.org, mirrors2.kernel.org - USx2, mirrors3.kernel.org - NL, mirrors4.kernel.org - SE<br>
 +
Do not sync from mirrors.kernel.org, choose one of the ones above and use that.
 +
|| <ftpadmin at kernel.org>
 
|-
 
|-
 
| wpi.edu || IPv6-connected or Internet2-connected mirrors only || Chuck Anderson <cra at wpi.edu>
 
| wpi.edu || IPv6-connected or Internet2-connected mirrors only || Chuck Anderson <cra at wpi.edu>
Line 54: Line 63:
 
| sunsite.mff.cuni.cz ||
 
| sunsite.mff.cuni.cz ||
 
|-
 
|-
| ftp.heanet.ie || IPv6 and Internet2 connectivity. ftp.heanet.ie::fedora-enchilada, ftp.heanet.ie::fedora-epel || mirrors at heanet.ie ||
+
| ftp.heanet.ie || IPv6 and Internet2 connectivity. ftp.heanet.ie::fedora-enchilada, ftp.heanet.ie::fedora-epel || mirrors at heanet.ie
 
|-
 
|-
 
| mirror.speedpartner.de || IPv4 and IPv6 || mirror at speedpartner.de
 
| mirror.speedpartner.de || IPv4 and IPv6 || mirror at speedpartner.de
 
|-
 
|-
 
| fedora.c3sl.ufpr.br || South America || Carlos Carvalho carlos at fisica.ufpr.br
 
| fedora.c3sl.ufpr.br || South America || Carlos Carvalho carlos at fisica.ufpr.br
 +
|-
 +
| ftp.linux.cz ||Czech Republic, Europe|| ftp-admin at fi.muni.cz
 +
|-
 +
| mirror.gtlib.gatech.edu || fedora-enchilada and fedora-epel  || Neil Bright neil.bright at oit.gatech.edu
 +
|-
 +
| mirrors.rit.edu || Rochester, NY, USA || mirrors@rit.edu
 
|}
 
|}
  
== Master Mirrors ==
+
== Tier 1 Rsync configuration ==
* download-i2.fedora.redhat.com in Raleigh, NC, USA (Internet2 users only, but requires static routes to use. Contact < mirror-admin at fedoraproject dot org> before using this mirror to ensure proper routes are in place).
+
Below is an example rsyncd.conf file for a Tier 1 mirror that provides private rsync access to select downstream Tier 2 mirrors. You may do this via either IP or DNS-based access control, or by a shared username/password which you give to your selected Tier 2 mirrors directly.
* download1.fedora.redhat.com is offline, no ETA on its return. DNS for it points at download3.
+
 
* download2.fedora.redhat.com is offline, no ETA on its return. No DNS.
+
The key to this is that the Tier 1 mirror rsyncs content using a user account (e.g. ''mirror'' used below), and you serve content to Tier 2 mirrors using a private rsync module that runs as that same user account, while providing public non-authenticated rsync using the ''nobody'' account.  In this way, Tier 2 mirrors may obtain content before the permissions are made world readable.
* download3.fedora.redhat.com in Phoenix, AZ, USA
+
 
* download4.fedora.redhat.com in Phoenix, AZ, USA
+
 
* download5.fedora.redhat.com in Phoenix, AZ, USA
+
<pre>
 +
use chroot = yes
 +
uid = nobody
 +
gid = nobody
 +
dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.bz2 *.iso *.ogg *.ogv *.tbz
 +
exclude = .snapshot/ .~tmp~/ /.private/ /.private/** **/.nfs*
 +
ignore nonreadable = yes
 +
list = true
 +
read only = yes
 +
refuse options = checksum
 +
 
 +
[ fedora-enchilada ]
 +
        comment = Fedora - The whole enchilada
 +
        path = /srv/pub/fedora
 +
 
 +
[ fedora-epel ]
 +
        comment = Extra Packages for Enterprise Linux
 +
        path = /srv/pub/epel
 +
 
 +
##
 +
## The following are not seen and are limited by IP.
 +
##
 +
 
 +
[fedora-enchilada0]
 +
      comment = Fedora Enchilada for Tier0|1 Mirrors
 +
      path = /srv/pub/fedora/
 +
      list = no
 +
      uid = mirror
 +
      gid = mirror
 +
      hosts allow = (IP or DNS address) ...
 +
 
 +
[fedora-epel0]
 +
      comment = Fedora EPEL for Tier0|1 Mirrors
 +
      path = /srv/pub/epel/
 +
      list = no
 +
      uid = mirror
 +
      gid = mirror
 +
      hosts allow = (IP or DNS address) ...
 +
</pre>
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]

Revision as of 12:12, 30 August 2012

Infrastructure InfrastructureTeamN1.png


Contents

Tiering

Fedora mirror servers use Tiering, whereby a select few fast mirrors get read access to the master rsync servers, and all the other mirrors pull from those mirrors.

It turns out, 9 of our 10 Tier 1 mirrors are available over Internet2. And, over half of our total mirrors are reachable over Internet2. So, let's make use of that whereever we can.

For our purposes, define:

  • master: The Fedora-owned servers dl.fedoraproject.org and download-i2.fedoraproject.org
  • Tier 1: The fast mirrors which pull from a master mirror.
  • Tier 2: The mirrors that pull from the Tier 1 servers.

Properties of Tier 1 mirrors:

  • Limit the number of Tier 1 mirrors, to ensure adequate bandwidth for these. Adjust number up or down depending on capability of the masters.
  • Must carry everything under fedora-enchilada and fedora-epel. This allows Tier 2 mirrors to exclude what they wish, but get everything if they so wish. This means at least 1TB of disk space for the Fedora portion of this server.
  • Must have a 1 Gigabit connection to the Internet, or faster.
  • Must have an active, available, responsive mirror administrator during the days content is staged.
  • Must have at least 2 Internet2-connected Tier 1 mirrors.
  • Must have at least 1 Tier 1 mirror on each continent for which we have Tier 2 mirrors
  • Must serve private rsync (see below for configuration)

Master Mirrors

  • dl0[12345].fedoraproject.org, in Phoenix, AZ, USA.
    • dl.fedoraproject.org is a DNS round-robin to dl0[12345].
  • download-i2.fedoraproject.org in Raleigh, NC, USA (Internet2, NLR, and those reachable over NLR only) This is the preferred master mirror for downstreams reachable on Internet2.

Master Mirror rsync modules

The master mirrors provide two additional rsync modules which provide pre-bitflip content. Fedora tiered mirrors should use these modules to be able to get pre-bitflip content.

module name content
fedora-enchilada0 Everything under /pub/fedora/, including pre-bitflip content
fedora-epel0 Everything under /pub/epel, including pre-bitflip content (even though EPEL doesn't do bitflips

Tier 1 Mirrors

Tier 1 mirrors pull from one of the master mirrors.

Server Comment Contact for ACL
fedora-archives.ibiblio.org Internet2 / National Lamba Rail (NLR) connected hosts. <fedora-admin@ibiblio.org> No ACLs - open for syncing.
archive.linux.duke.edu Internet2. Uses ACL from MirrorManager database. Drew Stinnett <drew.stinnett at duke.edu> (spacepope on IRC)
kernel.org mirrors1.kernel.org, mirrors2.kernel.org - USx2, mirrors3.kernel.org - NL, mirrors4.kernel.org - SE

Do not sync from mirrors.kernel.org, choose one of the ones above and use that.

<ftpadmin at kernel.org>
wpi.edu IPv6-connected or Internet2-connected mirrors only Chuck Anderson <cra at wpi.edu>
rsync.hrz.tu-chemnitz.de rsync.hrz.tu-chemnitz.de::fedora-enchilada/. Uses ACL from MirrorManager database . guenther.fischer at hrz.tu-chemnitz.de
fedora-rsync.ftp.pub.2iij.net rsync://fedora-rsync.ftp.pub.2iij.net/fedora-enchilada mirror-contact at iij.ad.jp
sunsite.mff.cuni.cz
ftp.heanet.ie IPv6 and Internet2 connectivity. ftp.heanet.ie::fedora-enchilada, ftp.heanet.ie::fedora-epel mirrors at heanet.ie
mirror.speedpartner.de IPv4 and IPv6 mirror at speedpartner.de
fedora.c3sl.ufpr.br South America Carlos Carvalho carlos at fisica.ufpr.br
ftp.linux.cz Czech Republic, Europe ftp-admin at fi.muni.cz
mirror.gtlib.gatech.edu fedora-enchilada and fedora-epel Neil Bright neil.bright at oit.gatech.edu
mirrors.rit.edu Rochester, NY, USA mirrors@rit.edu

Tier 1 Rsync configuration

Below is an example rsyncd.conf file for a Tier 1 mirror that provides private rsync access to select downstream Tier 2 mirrors. You may do this via either IP or DNS-based access control, or by a shared username/password which you give to your selected Tier 2 mirrors directly.

The key to this is that the Tier 1 mirror rsyncs content using a user account (e.g. mirror used below), and you serve content to Tier 2 mirrors using a private rsync module that runs as that same user account, while providing public non-authenticated rsync using the nobody account. In this way, Tier 2 mirrors may obtain content before the permissions are made world readable.


use chroot = yes
uid = nobody
gid = nobody
dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.bz2 *.iso *.ogg *.ogv *.tbz
exclude = .snapshot/ .~tmp~/ /.private/ /.private/** **/.nfs*
ignore nonreadable = yes
list = true
read only = yes
refuse options = checksum

[ fedora-enchilada ]
        comment = Fedora - The whole enchilada
        path = /srv/pub/fedora

[ fedora-epel ]
        comment = Extra Packages for Enterprise Linux
        path = /srv/pub/epel

##
## The following are not seen and are limited by IP.
##

[fedora-enchilada0]
       comment = Fedora Enchilada for Tier0|1 Mirrors
       path = /srv/pub/fedora/
       list = no
       uid = mirror
       gid = mirror
       hosts allow = (IP or DNS address) ...

[fedora-epel0]
       comment = Fedora EPEL for Tier0|1 Mirrors
       path = /srv/pub/epel/
       list = no
       uid = mirror
       gid = mirror
       hosts allow = (IP or DNS address) ...