Infrastructure/Mirroring

From FedoraProject

Jump to: navigation, search


Fedora is fortunate to have over 200 volunteer mirror sites globally, which helps distribute the Fedora software to end users. We greatly appreciate our mirror sites and their system administrators.

Contents

Communicating

What are the size estimates?

How can someone become a public mirror?

Becoming an official public mirror is easy, and getting easier. All we request is that your site have sufficient bandwidth and disk space to handle the load. Each Fedora release can consume upwards of 200GB of disk space, and downloaders can consume as much bandwidth as you've got. Mirror sites have at least a 100Mbit/sec* connection to the Internet, many have Gigabit or larger pipes. As of the Fedora 8 release, the total space consumed on the master server, thus what a mirror could consume, is 1.1TB and growing. A 1-2TB volume would be most appropriate for a long-term mirror. This content is hardlinked; if you can't hardlink (e.g. you're on AFS), you'll need much more disk space. Actual space consumed is noted at [1] .

How can someone make a private mirror?

Private mirrors are mirrors that reside entirely within an organization (company, school, etc.) and can only be accessed by members of that organization. They exist to speed up distributing Fedora within an organization, where local bandwidth costs are far less than going across the Internet.

Private mirrors behave similarly to public mirrors, with a few exceptions:

You may also find it more beneficial to run an IntelligentMirror instead of a full rsync mirror. In this way, only the updates your local users actually need will be cached on your local mirror, saving you the bandwidth from downloading updates you don't actually need.


MirrorManager: the Fedora Mirror Management system

The MirrorManager software keeps track of all the mirrors without requiring a lot of manual text file editing.

Signing up

Fedora Account System

Registering in MirrorManager 

$ dig txt 1.1.166.143.asn.routeviews.org @archive.routeviews.org
;; ANSWER SECTION:
1.1.166.143.asn.routeviews.org.	86400 IN TXT	"3614" "143.166.0.0" "16"

Here, the answer is in the TXT record, the first value, 3614.

Mirroring

The only sane way to do mirroring is to use rsync. Note the options -H (hardlinks), --delay-updates, --numeric-ids and --delete-after are required to ensure your mirror content stays valid even during a new rsync run, until all the new data is available. 

rsync -vaH --exclude-from=${EXCLUDES} --numeric-ids --delete --delete-after --delay-updates \
 rsync://download.fedora.redhat.com/fedora-enchilada ${LOCAL_DIR}

Running report_mirror

MirrorManager includes a tool, report_mirror which can upload to the mirror database that you completed a run and what content you've got. This makes generating the yum mirrorlists and all other pages much much simpler. Please run report_mirror after every rsync job completes. 

yum install mirrormanager-client

or get the files directly from the git tree report_mirror files . Or it can be obtained using git: 

git clone git://git.fedorahosted.org/git/mirrormanager
  or
git clone http://git.fedorahosted.org/git/mirrormanager/


You need both report_mirror and report_mirror.conf, and must edit report_mirror.conf to include the content you're carrying and the path to that content on your disk.

Available content

The available content modules by rsync, and their point in the directory tree are:

Fedora 7 and newer
rsync moduleDescriptionpath on master serverComments
fedora-enchiladaFedora - the whole enchilada/pub/fedoraPlease use this if you can, it provides the all Fedora content
fedora-linux-releasesFedora Linux Releases/pub/fedora/linux/releases
fedora-linux-developmentFedora Linux Development (Rawhide)/pub/fedora/linux/development
fedora-linux-updatesFedora Linux Updates/pub/fedora/linux/updates
fedora-epelExtra Packages for Enterprise Linux/pub/epelPlease use this to mirror EPEL
Fedora Secondary Architectures

Secondary architectures content is not hosted on the master mirror servers, but instead on a different machine, secondary.fedoraproject.org. Should you wish to mirror that content, please do so at a directory such as /pub/fedora-secondary. Do not put this content into the same directory structure as the other Fedora content.. The MirrorManager Category for this content is 'Fedora Secondary Arches'.

rsync moduleDescriptionpath on master serverComments
fedora-secondaryFedora Secondary Arches/pub/fedora-secondaryon secondary.fedoraproject.org
Fedora Additional Content

Additional freely distributable content of a variety of types will be hosted at http://alt.fedoraproject.org/pub/alt/. Should you wish to mirror that content, please do so at a directory such as /pub/alt. Do not put this content into the same directory structure as the other Fedora content.. The MirrorManager Category for this content is 'Fedora Other'.

rsync moduleDescriptionpath on master serverComments
altFedora Other/pub/alton alt.fedoraproject.org
Fedora Core and Extras for FC6 and earlier

Historical content is now hosted on rsync://archive.fedoraproject.org/fedora-archive. Content here is updated once every 6 months, so please don't sync it multiple times a day.

Please use the above paths as subpaths on your own mirror servers, omitting 'pub' if necessary. We recommend using fedora-enchilada plus fedora-epel if you can. This ensures you have the same directory structure as the master servers, which makes it far easier for users to find the content on your mirror.

DVDs, CDs, and the exploded trees

When a new release is available, it can be bandwidth-efficient to download only the ISOs first (say, the DVD ISOs), then explode those into the directory structure, then run a full normal rsync run. This lets you avoid downloading the same RPMs twice (both on ISOs and as plain RPMs). There's a tool somewhere to help do this.

Regular hardlink runs

While the Fedora release maintainers try to keep as little redundant packaging around as possible, there are some duplicate packages in the tree. For example, when a Fedora Test release comes out, the package set included there looks remarkably like that of the development tree from a few days before. By copying the development tree over into the new Test directory before starting your rsync run, and using rsync -H, you can avoid downloading all that content a second time.

In addition, it's good practice to run a tool like hardlink++ on your tree occasionally (say, weekly), to ensure as much of your tree as possible is hardlinked.

Pre-Release: Copying Development tree to new release directory

In the days leading up to a release, either test or final, the development (aka rawhide) tree will stop taking new packages, and will closely resemble what winds up in the new release. As a mirror, you can avoid downloading content that already is in your copy of the development tree that matches what's in the release tree by copying those packages using hardlinks, such as: 

cp -lr fedora/linux/development/i386 fedora/linux/test/11-Preview/Fedora/
cp -lr fedora/linux/development/x86_64 fedora/linux/test/11-Preview/Fedora/
cp -lr fedora/linux/development/source fedora/linux/test/11-Preview/Fedora/

and then start the rsync process, which will clean up any changes and fix up the timestamps.

Rsync Configuration (sample)

Larger mirrors, like kernel.org, have slightly custom front-ends to rsync (mainly so that they can have a single rsync instance and have multiple ip based vhost configuration files) That said what follows is a sample rsync configuration file for public syncing (this is not intended for private pre-bitflip mirroring) 

[fedora]
        comment         = Fedora - RedHat community project
        path            = <path to your fedora directory>
        exclude         = lost+found/
        read only       = true
        max connections = 100
        lock file       = /var/run/rsyncd-mirrors.lock
        uid             = <user id (numeric, or textual) of an anonymous style user who should have read access>
        gid             = <group id (numeric, or textual) of an anonymous style user who should have read access>
        transfer logging = yes
        timeout         = 900
        ignore nonreadable = yes
        dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2
        refuse options = checksum

Things to explicitly note:

HTTPd Configuration

Keepalives

HTTP Keepalives should be enabled on your mirror server to speed up client downloads. By default, Fedora's Apache httpd package has keepalives disabled. They should be enabled, with a timeout of at least 2 seconds (the default of 15 seconds might be too high for a heavily loaded mirror server, but 2 seconds is sufficient and appropriate for yum). 

KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 100

Other http servers such as lighttpd have keepalives enabled by default.

Caching of metadata

We don't want caching proxy servers between our mirrors and our end user systems to cache our yum repository metadata. So, add explicit metadata handling. (Suggested by the OpenSUSE download redirector.) 

   <LocationMatch "\.(xml|xml\.gz|xml\.asc|sqlite)">
       Header set Cache-Control "must-revalidate"
       ExpiresActive On
       ExpiresDefault "now"
   </LocationMatch>
Redirecting ISO downloads to FTP

Apache 2.x and earlier can't distribute files larger than 2GB. This means DVD images won't work. (lighttpd doesn't have this limitation). Also, some people find FTP to be more efficient than HTTP for really large files like ISO images. These Rewrite lines will redirect all HTTP GET requests for *.iso files to a different FTP daemon. With this method, HEAD requests used by the MirrorManager crawler for *.iso files aren't rewritten, which gives better crawling results. 

RewriteCond     %{REQUEST_METHOD} GET
RewriteRule     ^(.*\.iso)$ ftp://myserver/$1  [L,R=301]
Content Types

ISO and RPM files should be served using MIME Content-Type: application/octet-stream. In Apache, this can be done inside a VirtualHost or similar section: 

<VirtualHost *:80>
AddType application/octet-stream .iso
AddType application/octet-stream .rpm
</VirtualHost>
Limiting Download Accelerators

Download accelerators will try to open the same file many times, and request chunks, hoping to download them in parallel. This can overload heavily loaded mirror servers, especially on release day. Here are some tricks to thwart such activities.

To limit connections to ISO dirs by some amount per IP: 

<IfModule mod_limitipconn.c>
MaxConnPerIP 6
</IfModule>

To block ranged requests as this is what download accelerators do indeed: 

RewriteEngine on
RewriteCond %{HTTP:Range} [0-9] $
RewriteRule \.iso$ / [F,L]

Similar things can be done with iptables and the recent module, which might give you a little more ability to control what is being done, either by limiting new connections or by dropping 50% of a users packets.

Logging Partial Content Downloads

Partial content can be logged correctly using apache: 

# this includes actual counts of actual bytes received (%I) and
# sent (%O); this requires the mod_logio module to be loaded.

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" %I %O \"%{User-Agent}i\"" combined

Pre-bitflip mirroring

Several days before each public release, the content will be staged to the master mirror servers, but with restricted permissions on the directories (generally mode 0750), specifically, not world readable.

Mirror servers should have several different user/group accounts on their server, for running the different public services. Typically you find:

The user account used to download content from the masters must be not be the same as the HTTP, FTP, or RSYNC server accounts. This guarantees that content downloaded with permissions 0750 will not be made available via your public servers yet.

On the morning of the public release, the permissions on the directories on the master servers will change to 0755 - world readable. This is called the bitflip.

Mirrors may either rsync one more time to pick up these new permissions (but won't have to download all the data again), or preferably, can schedule a batch job to bitflip: 

$ echo "chmod a+rx /pub/fedora/linux/releases/9" | at '14:45 UTC May 13 2008'
Serving content to other mirrors

Tier 1 mirrors will necessarily need to share content to Tier 2 mirrors before the bitflip. This is done by running another instance of the rsync daemon, on a different port (e.g. 874), with an Access Control List to prevent public downloads, running as a user in the same group as downloaded the content (e.g. group mirror). This could be user mirror, group mirror, who has group read/execute permissions on the still-private content.

Tier 1 mirrors have a tendency to use different authentication methods for granting access to these non-public downloads, they vary from maintaining IP based ACL's to assigning username/password combinations to mirrors wishing to sync from them. Each method has advantages / disadvantages, the IP list is 'simpler' from a mirrormanager perspective as mirrormanager can give you the list of IP's but from an automation standpoint can be more difficult (as rsync's configuration file does not allow that ACL list to be stored in a separate file). Username / passwords can be more versatile as sites mirroring can change IPs without notifying you, but it's easier for those credentials to leak out and get miss-used.

Map

File:Mirrormap.png
Worldwide public mirrors as of 23-May-2009.

Recognition

This product includes GeoLite data created by MaxMind, available from http://www.maxmind.com/.

Retrieved from "https://fedoraproject.org/wiki/Infrastructure/Mirroring"