From Fedora Project Wiki
Line 26: Line 26:
== Current status ==
== Current status ==
* Targeted release: [[Releases/<number> | Fedora <number> ]]  
* Targeted release: [[Releases/25 | Fedora 25 ]]  
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  

Revision as of 20:57, 2 June 2016

Fedora Scale-Out Docker Registry


This is a proposal for a change to the Fedora Infrastructure and Fedora Release Engineering tooling to provide a scalable Docker Registry solution for Fedora that is integrated with the Fedoar Docker Layered Image Build Service.


  • Name: Adam Miller
  • Email:
  • Release notes owner:

Current status

  • Targeted release: Fedora 25
  • Last updated: 2016-06-02
  • Tracker bug: <will be assigned by the Wrangler>

Detailed Description

   +-----------+               +------------------------+
   |  koji     |<--------------+ fedpkg container build |
   +--+--------+               +------------------------+
      |   ^
      v   |
   |           |          +----------------------+
   |   OSBS    |          | docker/distribution  |
   |           +--------->| registry             +-+
   +-----------+          |                      | |
                          | (candidate builds)   | |
                          +----------------------+ +
           +-----------------+                     |
           |                 +<--------------------+
           |      Pulp       |
           |                 +------------+
           +----+------------+            |
                |                         |
                v                         |
+-------------------------+               v
| +--------+ +--------+   |      +-----------------------+
| |  Crane | |  Crane |   |      | Pulp Published Stable |
| +------+-+ +--+-----+   |      | Image Layeres and     |
|    ^   |      |   ^     |      | Metadata              |
|    |   |      |   |     |      +--------+--------------+
|    |   |      |   |     |               |
|    |   |      |   |     |               |
|    |   |      |   |     |               |
|    |   v      v   |     |               v
| +--+--------------+-+   |     +--------------------+
| | Fedora Infra      |   |     | Mirror Manager     |
| | HAProxy           |   |     | master mirror      |
| +-------------------+   |     +-----------------+--+
+---------+---------------+                       |
     ^    |                                       |
     |    |                                       |
     |    |                                       |
     |    |                                       v
     |    |                             +-----------------------------+
     |    |                             |                             |
     |    |   +------------------------>| "Mirror Network"            |
     |    |   |                         | (All our volunteer mirrors) |
     |    |   |   +---------------------+                             |
     |    |   |   |                     +-----------------------------+
     |    |   |   |
     |    |   |   |
     |    |   |   |
     |    |   |   |
     |    |   |   |
     |    v   |   v
  | Users           |
  | (docker pull)   |


registry: a collection of docker image repositories

repository: named after an image and is a collection of multiple tags of an that image

tag: an arbitrary string assigned to a specific docker image (identified by the image's sha256 checksum) NOTE: The "latest" tag is special and is assumed if no tag is provided. This is true also for a 'docker pull' operation and an image tagged "latest" will be the default image pulled by users.


Pulp[0] + Crane[1] + MirrorManager[2] + Docker Distribution[3]

  • Pulp is a platform for managing repositories of content, such as software packages, and making it available to a large numbers of consumers. It is also capable of managing docker content.
  • Crane is a stand-alone python flask wsgi application written by the Pulp team to serve as a API entry point for the docker client and will answer to an user's 'docker pull'. It does not however create content manifests or provide hosting for docker image content, instead it depends on someone creating the manifest metadata themselves or having pulp publish it and serves 302 redirects to the docker client so they can find where the docker images actually live.
  • MirrorManager is what Fedora uses to manage the public mirror network and distribute content.
  • Docker Distribution is the defacto standard open source implementation of the Docker Registry V2 API spec[5]. It provides many features but the ability to have it's back-end storage be provided by a "mirror network" much like the one Fedora has at it's disposal is not one of them. The reason we need this in place is because the mechanism by which you could push a docker image directly to Pulp in Docker Registry v1 no longer exists in v2 so we must instead perform a "sync" operation between the two. (This is a common problem for all known "third party" v2 registry implementations).


  • OSBS will perform Builds, as these builds complete they will be pushed to the docker-distribution (v2) registry, these will be considered "candidate images". Pulp will sync and publish the candidate repository.
  • Testing will occur using the "candidate images" (details of how we want to handle that are outside the scope of this proposal).
  • A "candidate image" will be marked stable once it's criteria have been satisfied to do so. (This is vague because this is a topic of ongoing discussion and work to decide what criteria an image will need to abide by before being considered "stable" and promoted as such)
  • Once stable, pulp will publish that repository's content to a directory, we will split that content and sync the image layers along with their metadata to Mirror Manager master mirror. We will also sync the repo metadata published by Pulp to somewhere Crane can pick it up. (This could and will likely be something that Bodhi triggers via the Pulp REST API)
  • Mirror Manager will distribute to the mirrors the image layers and their metadata.
  • Crane will get the new repository metadata and will serve redirects to the new content relative to which will perform another redirect (via MirrorManager) where the docker client upon a "docker pull" will find it's content.

Technical Details

Some more in depth technical items around this solution that I think the Fedora Infrastructure Team are likely interested in:

Pulp Requirements

  • An AMPQ message queue, currently qpid and rabbitmq are supported upstream. However, the requirement appears to stem from the use of Celery[5] and Celery upstream supports redis[6] as a broker backend so I have requested that it be made available as supported option Pulp[7]. This will obviously take some amount of dev time, but we can plan for that if adding a message queue to Fedora Infra is a show stopper.
  • MongoDB, this is currently a hard requirement but postgresql is planned replace MongoDB in the future[8] (probably a year-ish timeline on that). The question is, can we wait that long from a Fedora Project standpoint for the new feature before having a solution in place? I imagine some of this will need to be planned/scoped as time goes on and we learn more but it's worth keeping in mind
  • Storage. I've been told Pulp likes a lot of storage, I don't know hard numbers for what we'd need since we're getting into uncharted territory but I've heard that a few hundred GB is not uncommon in pulp deployments when combining the MongoDB storage needs with all the artifacts in the repos.

Crane Requirements

  • Crane is just a small python wsgi app written in flask

General Notes

A couple of things to note about maintenance and uptime considerations:

The Intermediate docker-distribution registry is needed for builds in koji+OSBS

Pulp will be required for "promotion" of builds from candidate to testing or stable

Crane will be required for end users out in the world to access in order to actually pull down Docker images from us.

The only service here that needs to be public end-user facing (i.e. wide open to the internet and not have access locked to a FAS group) is Crane. All other components should be able to be locked down similar to the "Fedora internal" components koji (builders, etc), bodhi (signing, etc) and similar.

Benefit to Fedora

This will allow for Fedora to provide packages, software, and other content in the form of a Docker Image as an officially released artifact from the Fedora Project that is released and hosted much in the same way RPMs are today. These images can then be included in the distribution in various ways. This could potentially be used by the Modularization effort or by any other part of the initiative that may arise.


  • Other developers: N/A (not a System Wide Change)
  • Release engineering: N/A (not a System Wide Change)

  • Policies and guidelines: N/A (not a System Wide Change)

  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

N/A (not a System Wide Change)

How To Test

Once the service is deployed, users can perform the following on their systems to test.

$ dnf -y install docker
$ systemctl start docker
$ docker pull

N/A (not a System Wide Change)

User Experience

N/A (not a System Wide Change)


N/A (not a System Wide Change)

Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
  • Contingency deadline: N/A (not a System Wide Change)
  • Blocks release? No (not a System Wide Change)
  • Blocks product? N/A



Release Notes