Infrastructure/Mirroring/Tiering

From FedoraProject

< Infrastructure | Mirroring(Difference between revisions)
Jump to: navigation, search
(update duke admin info)
(Tier 1 Mirrors)
 
(24 intermediate revisions by 7 users not shown)
Line 5: Line 5:
 
Fedora mirror servers use Tiering, whereby a select few fast mirrors get read access to the master rsync servers, and all the other mirrors pull from those mirrors.
 
Fedora mirror servers use Tiering, whereby a select few fast mirrors get read access to the master rsync servers, and all the other mirrors pull from those mirrors.
  
It turns out, 9 of our 10 Tier 0 or Tier 1 mirrors are available over Internet2.  And, over half of our total mirrors are reachable over Internet2.  So, let's make use of that whereever we can.
+
It turns out, 9 of our 10 Tier 1 mirrors are available over Internet2.  And, over half of our total mirrors are reachable over Internet2.  So, let's make use of that whereever we can.
  
 
For our purposes, define:
 
For our purposes, define:
* '''master''': The Red Hat servers download*.fedora.redhat.com.
+
* '''master''': The Fedora-owned servers dl.fedoraproject.org and download-i2.fedoraproject.org
* '''Tier 0''': The fast mirrors which pull from Red Hat's Internet2-connected master
+
* '''Tier 1''': The fast mirrors which pull from a master mirror.
* '''Tier 1''': The fast mirrors which pull from the Tier 0 servers (or one of the other masters).
+
 
* '''Tier 2''': The mirrors that pull from the Tier 1 servers.
 
* '''Tier 2''': The mirrors that pull from the Tier 1 servers.
  
Properties of Tier 0 and 1 mirrors:
+
Properties of Tier 1 mirrors:
  
* Limit the number of Tier 1 mirrors to 10, to ensure adequate bandwidth for these.  Adjust number up or down depending on capability of the masters.
+
* Limit the number of Tier 1 mirrors, to ensure adequate bandwidth for these.  Adjust number up or down depending on capability of the masters.
 
* Must carry everything under fedora-enchilada and fedora-epel.  This allows Tier 2 mirrors to exclude what they wish, but get everything if they so wish.  This means at least 1TB of disk space for the Fedora portion of this server.
 
* Must carry everything under fedora-enchilada and fedora-epel.  This allows Tier 2 mirrors to exclude what they wish, but get everything if they so wish.  This means at least 1TB of disk space for the Fedora portion of this server.
 
* Must have a 1 Gigabit connection to the Internet, or faster.
 
* Must have a 1 Gigabit connection to the Internet, or faster.
Line 21: Line 20:
 
* Must have at least 2 Internet2-connected Tier 1 mirrors.
 
* Must have at least 2 Internet2-connected Tier 1 mirrors.
 
* Must have at least 1 Tier 1 mirror on each continent for which we have Tier 2 mirrors
 
* Must have at least 1 Tier 1 mirror on each continent for which we have Tier 2 mirrors
* Must serve private rsync
+
* Must serve private rsync (see below for configuration)
  
== Tier 0 Mirrors ==
+
== Master Mirrors ==
 +
* dl0[12345].fedoraproject.org, in Phoenix, AZ, USA.
 +
** dl.fedoraproject.org is a DNS round-robin to dl0[12345].
 +
* download-i2.fedoraproject.org in Raleigh, NC, USA (Internet2, NLR, and those reachable over NLR only)  This is the preferred master mirror for downstreams reachable on Internet2.
  
Tier 0 mirrors can pull from Red Hat directly over the Internet2 connection.
+
== Master Mirror rsync modules ==
 +
The master mirrors provide two additional rsync modules which provide pre-bitflip content.  Fedora tiered mirrors should use these modules to be able to get pre-bitflip content.
  
 
{| border="1"
 
{| border="1"
 
|-
 
|-
| Server || Comment || Contact for ACL
+
| module name || content
 
|-
 
|-
| fedora-archives.ibiblio.org || Internet2 / National Lamba Rail (NLR) connected hosts. || Don Sizemore <dls at metalab.unc.edu>
+
| fedora-enchilada0 || Everything under /pub/fedora/, including pre-bitflip content
 
|-
 
|-
| archive.linux.duke.edu ||Internet2.  Uses ACL from MirrorManager database. || Drew Stinnett <drew.stinnett at duke.edu> (spacepope on IRC)
+
| fedora-epel0 || Everything under /pub/epel, including pre-bitflip content (even though EPEL doesn't do bitflips
 
|}
 
|}
  
 
== Tier 1 Mirrors ==
 
== Tier 1 Mirrors ==
  
Tier 1 mirrors pull from one of the Tier 0 mirrors.
+
Tier 1 mirrors pull from one of the master mirrors.
  
 
{| border="1"
 
{| border="1"
Line 44: Line 47:
 
| Server || Comment || Contact for ACL
 
| Server || Comment || Contact for ACL
 
|-
 
|-
| mirrors.kernel.org || USx2, SE, NL || <ftpadmin at kernel.org>
+
| fedora-archives.ibiblio.org || Internet2 / National Lamba Rail (NLR) connected hosts. || <fedora-admin@ibiblio.org>  No ACLs - open for syncing.
 +
|-
 +
| archive.linux.duke.edu ||Internet2.  Uses ACL from MirrorManager database. || Drew Stinnett <drew.stinnett at duke.edu> (spacepope on IRC)
 +
|-
 +
| mirrors.kernel.org || IPv4, US West Coast || <ftpadmin at kernel.org>
 
|-
 
|-
 
| wpi.edu || IPv6-connected or Internet2-connected mirrors only || Chuck Anderson <cra at wpi.edu>
 
| wpi.edu || IPv6-connected or Internet2-connected mirrors only || Chuck Anderson <cra at wpi.edu>
Line 54: Line 61:
 
| sunsite.mff.cuni.cz ||
 
| sunsite.mff.cuni.cz ||
 
|-
 
|-
| ftp.heanet.ie || IPv6 and Internet2 connectivity. ftp.heanet.ie::fedora-enchilada, ftp.heanet.ie::fedora-epel || mirrors at heanet.ie ||
+
| ftp.heanet.ie || IPv6 and Internet2 connectivity. ftp.heanet.ie::fedora-enchilada, ftp.heanet.ie::fedora-epel || mirrors at heanet.ie
 
|-
 
|-
 
| mirror.speedpartner.de || IPv4 and IPv6 || mirror at speedpartner.de
 
| mirror.speedpartner.de || IPv4 and IPv6 || mirror at speedpartner.de
 
|-
 
|-
 
| fedora.c3sl.ufpr.br || South America || Carlos Carvalho carlos at fisica.ufpr.br
 
| fedora.c3sl.ufpr.br || South America || Carlos Carvalho carlos at fisica.ufpr.br
 +
|-
 +
| ftp.linux.cz ||Czech Republic, Europe|| ftp-admin at fi.muni.cz
 +
|-
 +
| mirror.gtlib.gatech.edu || fedora-enchilada and fedora-epel  || Neil Bright neil.bright at oit.gatech.edu
 +
|-
 +
| mirrors.rit.edu || Rochester, NY, USA || mirrors@rit.edu
 
|}
 
|}
  
== Master Mirrors ==
+
== Tier 1 Rsync configuration ==
* download-i2.fedora.redhat.com in Raleigh, NC, USA (Internet2 users only, but requires static routes to use. Contact < mirror-admin at fedoraproject dot org> before using this mirror to ensure proper routes are in place).
+
Below is an example rsyncd.conf file for a Tier 1 mirror that provides private rsync access to select downstream Tier 2 mirrors. You may do this via either IP or DNS-based access control, or by a shared username/password which you give to your selected Tier 2 mirrors directly.
* download1.fedora.redhat.com is offline, no ETA on its return. DNS for it points at download3.
+
 
* download2.fedora.redhat.com is offline, no ETA on its return. No DNS.
+
The key to this is that the Tier 1 mirror rsyncs content using a user account (e.g. ''mirror'' used below), and you serve content to Tier 2 mirrors using a private rsync module that runs as that same user account, while providing public non-authenticated rsync using the ''nobody'' account.  In this way, Tier 2 mirrors may obtain content before the permissions are made world readable.
* download3.fedora.redhat.com in Phoenix, AZ, USA
+
 
* download4.fedora.redhat.com in Phoenix, AZ, USA
+
 
* download5.fedora.redhat.com in Phoenix, AZ, USA
+
<pre>
 +
use chroot = yes
 +
uid = nobody
 +
gid = nobody
 +
dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.bz2 *.iso *.ogg *.ogv *.tbz
 +
exclude = .snapshot/ .~tmp~/ /.private/ /.private/** **/.nfs*
 +
ignore nonreadable = yes
 +
list = true
 +
read only = yes
 +
refuse options = checksum
 +
 
 +
[ fedora-enchilada ]
 +
        comment = Fedora - The whole enchilada
 +
        path = /srv/pub/fedora
 +
 
 +
[ fedora-epel ]
 +
        comment = Extra Packages for Enterprise Linux
 +
        path = /srv/pub/epel
 +
 
 +
##
 +
## The following are not seen and are limited by IP.
 +
##
 +
 
 +
[fedora-enchilada0]
 +
      comment = Fedora Enchilada for Tier0|1 Mirrors
 +
      path = /srv/pub/fedora/
 +
      list = no
 +
      uid = mirror
 +
      gid = mirror
 +
      hosts allow = (IP or DNS address) ...
 +
 
 +
[fedora-epel0]
 +
      comment = Fedora EPEL for Tier0|1 Mirrors
 +
      path = /srv/pub/epel/
 +
      list = no
 +
      uid = mirror
 +
      gid = mirror
 +
      hosts allow = (IP or DNS address) ...
 +
</pre>
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]

Latest revision as of 15:38, 21 December 2013

Infrastructure InfrastructureTeamN1.png


Contents

[edit] Tiering

Fedora mirror servers use Tiering, whereby a select few fast mirrors get read access to the master rsync servers, and all the other mirrors pull from those mirrors.

It turns out, 9 of our 10 Tier 1 mirrors are available over Internet2. And, over half of our total mirrors are reachable over Internet2. So, let's make use of that whereever we can.

For our purposes, define:

  • master: The Fedora-owned servers dl.fedoraproject.org and download-i2.fedoraproject.org
  • Tier 1: The fast mirrors which pull from a master mirror.
  • Tier 2: The mirrors that pull from the Tier 1 servers.

Properties of Tier 1 mirrors:

  • Limit the number of Tier 1 mirrors, to ensure adequate bandwidth for these. Adjust number up or down depending on capability of the masters.
  • Must carry everything under fedora-enchilada and fedora-epel. This allows Tier 2 mirrors to exclude what they wish, but get everything if they so wish. This means at least 1TB of disk space for the Fedora portion of this server.
  • Must have a 1 Gigabit connection to the Internet, or faster.
  • Must have an active, available, responsive mirror administrator during the days content is staged.
  • Must have at least 2 Internet2-connected Tier 1 mirrors.
  • Must have at least 1 Tier 1 mirror on each continent for which we have Tier 2 mirrors
  • Must serve private rsync (see below for configuration)

[edit] Master Mirrors

  • dl0[12345].fedoraproject.org, in Phoenix, AZ, USA.
    • dl.fedoraproject.org is a DNS round-robin to dl0[12345].
  • download-i2.fedoraproject.org in Raleigh, NC, USA (Internet2, NLR, and those reachable over NLR only) This is the preferred master mirror for downstreams reachable on Internet2.

[edit] Master Mirror rsync modules

The master mirrors provide two additional rsync modules which provide pre-bitflip content. Fedora tiered mirrors should use these modules to be able to get pre-bitflip content.

module name content
fedora-enchilada0 Everything under /pub/fedora/, including pre-bitflip content
fedora-epel0 Everything under /pub/epel, including pre-bitflip content (even though EPEL doesn't do bitflips

[edit] Tier 1 Mirrors

Tier 1 mirrors pull from one of the master mirrors.

Server Comment Contact for ACL
fedora-archives.ibiblio.org Internet2 / National Lamba Rail (NLR) connected hosts. <fedora-admin@ibiblio.org> No ACLs - open for syncing.
archive.linux.duke.edu Internet2. Uses ACL from MirrorManager database. Drew Stinnett <drew.stinnett at duke.edu> (spacepope on IRC)
mirrors.kernel.org IPv4, US West Coast <ftpadmin at kernel.org>
wpi.edu IPv6-connected or Internet2-connected mirrors only Chuck Anderson <cra at wpi.edu>
rsync.hrz.tu-chemnitz.de rsync.hrz.tu-chemnitz.de::fedora-enchilada/. Uses ACL from MirrorManager database . guenther.fischer at hrz.tu-chemnitz.de
fedora-rsync.ftp.pub.2iij.net rsync://fedora-rsync.ftp.pub.2iij.net/fedora-enchilada mirror-contact at iij.ad.jp
sunsite.mff.cuni.cz
ftp.heanet.ie IPv6 and Internet2 connectivity. ftp.heanet.ie::fedora-enchilada, ftp.heanet.ie::fedora-epel mirrors at heanet.ie
mirror.speedpartner.de IPv4 and IPv6 mirror at speedpartner.de
fedora.c3sl.ufpr.br South America Carlos Carvalho carlos at fisica.ufpr.br
ftp.linux.cz Czech Republic, Europe ftp-admin at fi.muni.cz
mirror.gtlib.gatech.edu fedora-enchilada and fedora-epel Neil Bright neil.bright at oit.gatech.edu
mirrors.rit.edu Rochester, NY, USA mirrors@rit.edu

[edit] Tier 1 Rsync configuration

Below is an example rsyncd.conf file for a Tier 1 mirror that provides private rsync access to select downstream Tier 2 mirrors. You may do this via either IP or DNS-based access control, or by a shared username/password which you give to your selected Tier 2 mirrors directly.

The key to this is that the Tier 1 mirror rsyncs content using a user account (e.g. mirror used below), and you serve content to Tier 2 mirrors using a private rsync module that runs as that same user account, while providing public non-authenticated rsync using the nobody account. In this way, Tier 2 mirrors may obtain content before the permissions are made world readable.


use chroot = yes
uid = nobody
gid = nobody
dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.bz2 *.iso *.ogg *.ogv *.tbz
exclude = .snapshot/ .~tmp~/ /.private/ /.private/** **/.nfs*
ignore nonreadable = yes
list = true
read only = yes
refuse options = checksum

[ fedora-enchilada ]
        comment = Fedora - The whole enchilada
        path = /srv/pub/fedora

[ fedora-epel ]
        comment = Extra Packages for Enterprise Linux
        path = /srv/pub/epel

##
## The following are not seen and are limited by IP.
##

[fedora-enchilada0]
       comment = Fedora Enchilada for Tier0|1 Mirrors
       path = /srv/pub/fedora/
       list = no
       uid = mirror
       gid = mirror
       hosts allow = (IP or DNS address) ...

[fedora-epel0]
       comment = Fedora EPEL for Tier0|1 Mirrors
       path = /srv/pub/epel/
       list = no
       uid = mirror
       gid = mirror
       hosts allow = (IP or DNS address) ...