From Fedora Project Wiki

< Infrastructure‎ | Factory2‎ | Focus

Revision as of 18:43, 4 May 2017 by Ralph (talk | contribs) (→‎Introduction)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

What is a Focus Document?

The Factory 2.0 team produces a confusing number of documents. The first round was about the Problem Statements we were trying to solve. Let’s retroactively call them Problem Documents. The Focus Documents (like this one) focus on some system or some aspect of our solutions that cut across different problems. The content here doesn’t fit cleanly in one problem statement document, which is why we broke it out.


With all the new possible combinations of content (RPM packages, modules, Docker images, ...), it is no longer possible to require humans to for example remember to rebuild containers after they have built an RPM package intended for that container. There are just too many.

With the introduction of modules, the dependencies between artifacts will be even more complex, because of dependencies between modules, between modules and RPMs and even between the Docker images and modules, because modules can be used as a building block for Docker images.

The goal of Freshmaker is to address this issue by automatically rebuilding the artifacts when their sources or dependencies get updated. This for example ensures that RPM packages in latest Docker image are always up to date or that modules gets automatically rebuilt after the change of their modulemd specification and can be automatically tested by the CI.

Background on Freshmaker

  • Freshmaker is a service scheduling rebuilds of artifacts as new content becomes available.
    • With all the new possible combinations of content, it is no longer possible to require humans to remember to rebuild containers after they have built an rpm intended for that container.
  • It listens on fedmsg bus for messages from other builds systems like Koji or Module Build Service (MBS) and triggers the rebuild of higher-level artifact when lower-level artifact is built in.
    • For example, when new release of “foobar” RPM is built in Koji, all Freshmaker handles that event and triggers the rebuild of all modules which require this RPM. Later when those modules are rebuilt, it triggers the rebuild of all Docker images based on those modules and so on. More info is written further in this document.
  • It does not have a visible web UI or public API to manage or invoke builds, but does have an API to query the status of the service.
  • It cooperates with other services like PDC and PolicyEngine to make decisions about what to rebuild and if it even should do the rebuild.

Automatic rebuild events

This sub-chapter describes the events which trigger the automatic rebuild of other artifacts by Freshmaker. It hence also describes the use-cases for Freshmaker. The use-cases can be summarized in the following chart. Full description of each part of a chart is written later in this subchapter.


About the service sending the event

Although the chart and this subchapter mentions particular services which send and even to trigger the rebuilds, the real service sending the event to trigger the rebuild can be different. We will for example rebuild Docker image only when the updated module passes the tests. Therefore it is possible that the real service triggering the rebuild of a Docker image will be PolicyEngine or something similar handling the tests results, but the real cause of the rebuild is a module build and therefore we mention it as an service sending an event.

Just keep in mind that there might be multiple services on the road influencing whether and when Freshmaker sees the information about a particular artifact being built/updated.

RPM spec file change

  • Event sent by:
    • dist-git (.spec file change)
  • Triggers:
    • MBS (rebuilds all modules depending on this SRPM - spec file)
  • Mapping can be found in:
    • PDC - but needs more work, see below.

Although we have the list of SRPMs from which the module is built in PDC, it is currently not possible to search them according to SRPM. We can only fetch particular modules and then list their SRPMs. However, for this use-case, we would need to list all modules containing the SRPM. This would have to be implemented in the PDC.

Modulemd metadata change

  • Event sent by:
    • dist-git (modulemd .yaml file change)
  • Triggers:
    • MBS (rebuilds the module described by the modulemd .yaml file)
  • Mapping can be found in:
    • No mapping needed - we can directly rebuild the module based on the dist-git fedmsg message.

Dockerfile change

  • Event sent by:
    • Dist-git (Dockerfile change)
  • Triggers:
    • Koji (maybe also support for OSBS) (rebuilds all images based on this Dockerfile)
  • Mapping can be found in:
    • Dist-git Dockerfile maps directly to single container-build.

RPM build

  • Event sent by:
    • koji/brew (RPM is built)
  • Triggers:
    • Koji (maybe also support for OSBS) (rebuilds all images directly containing this RPM)
  • Mapping can be found in:
    • PDC - for example here.

Module build

  • Event sent by:
    • MBS (new version of module is built)
  • Triggers:
    • MBS (rebuilds all the modules depending on this module or including it as “included module”)
    • Koji (OSBS) (rebuilds all images containing this module)
  • Mapping can be found in:
    • Dependencies between modules are tracked in PDC now, but included modules are not. This is mostly the same situation as with tracking “RPM spec file change to module”.
    • For Module to container image, there is no mapping like that. Should it be in PDC?

To rebuild the container images automatically, Freshmaker needs to auto-generate the Dockerfiles based on the built module. We should probably have one Dockerfile per module profile, although the PolicyEngine can influence that.

Container image build

  • Event sent by:
    • OSBS (new container image is built)
    • Triggers:
    • OSBS (rebuilds all the layers images based on this image)

This is out of scope for Freshmaker. OSBS takes care of rebuilding layered container images when one of the layers changes.

Base Image Builds

  • Event sent by:
    • New RPM is built that goes into the Docker base image
  • Triggers:
    • A run of Pungi which produces a new base image and other “compose” artifacts.
  • Mapping can be found in:
    • PDC has a list of all rpms that went into the last compose for each release.

Ralph’s comment: We don’t want to do this *every* time. It is too much. Maybe only trigger new composes when further conditions are met. Take this one on last after we’ve gained experience with the other triggers.

Deciding whether to rebuild or not

Quality: Freshmaker should certainly not rebuild all the artifacts all the time. There might be good reasons why not to do it, for example when the underlying artifact did not pass its tests. To decide whether Freshmaker should rebuild the upstream artifacts, it must query the PolicyEngine.

Iterative Enablement: We also need a way to turn this on slowly for different content types - in particular for container rebuilds. The system should allow a whitelist of patterns for container names that should be considered for this, so we can start out with only doing automated rebuilds of the sssd container. Once we’re happy that it is working well, we can expand the whitelist, or remove it all together to handle all containers.

Detecting Cycles: We need some way to detect cycles, to make sure we’re not in an infinite loop….

Handling Failures: Does Freshmaker care whether builds that it kicks off pass or fail? If it does care, what actions are taken?