Infrastructure/Mirroring/Amazon

From FedoraProject

< Infrastructure | Mirroring(Difference between revisions)
Jump to: navigation, search
Line 6: Line 6:
 
* Use bucket policies to limit access to each region
 
* Use bucket policies to limit access to each region
 
* bucket names s3-mirror.<region>.fedoraproject.org allow for CNAME s3-mirror.fedoraproject.org to s3.amazon.com in our DNS
 
* bucket names s3-mirror.<region>.fedoraproject.org allow for CNAME s3-mirror.fedoraproject.org to s3.amazon.com in our DNS
 +
* Need list of IP addresses for each region to populate MM.  Would be nice if we could get that programmatically.
  
 
Torrents:
 
Torrents:
Line 21: Line 22:
 
* 0.093/GB/month for data, 700GB = $75/month/region
 
* 0.093/GB/month for data, 700GB = $75/month/region
 
* no way guess number of GET requests.  $75 assumes 10M requests.
 
* no way guess number of GET requests.  $75 assumes 10M requests.
 +
  
  
 
Open questions:
 
Open questions:
 
* do we sync to one region, then COPY to others?  If so, what tool?  That'll cost $ for bandwidth.
 
* do we sync to one region, then COPY to others?  If so, what tool?  That'll cost $ for bandwidth.

Revision as of 05:59, 28 January 2012

Initial thoughts by Matt Domsch

  • Use Reduced Redundancy Storage. All the content will be replicated easily.
  • Use s3cmd sync to keep content in buckets in sync
    • exclude ISOs
  • Use bucket policies to limit access to each region
  • bucket names s3-mirror.<region>.fedoraproject.org allow for CNAME s3-mirror.fedoraproject.org to s3.amazon.com in our DNS
  • Need list of IP addresses for each region to populate MM. Would be nice if we could get that programmatically.

Torrents:

  • if we upload ISOs, we get .torrent links "for free".
  • no tracker stats :-(
  • Can't group multiple files together into a single torrent
  • we're paying for outbound bandwidth
  • bucket policies keeping traffic in a single region means we need separate buckets for torrent content


Costs:

  • none for all uploads
  • none for intra-region requests
  • 0.093/GB/month for data, 700GB = $75/month/region
  • no way guess number of GET requests. $75 assumes 10M requests.


Open questions:

  • do we sync to one region, then COPY to others? If so, what tool? That'll cost $ for bandwidth.