From Fedora Project Wiki

(What is Zuul)
(Attach packaging jobs for a distgit repository on src.fedoraproject.org)
(42 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
= Zuul Based CI =
 
= Zuul Based CI =
  
== What is Zuul ==
+
== Goals ==
  
Zuul [https://zuul-ci.org/] is the CI and gating system from the Open Infrastructure Project [https://www.openstack.org/news/view/426/kata-containers-and-zuul-are-first-pilot-projects-confirmed-as-toplevel-open-infrastructure-projects-by-the-openstack-foundation-board]. It is able to scale fine and handles by default features such as artifacts sharing between jobs and cross Git repositories testing. You can see Zuul in action here [https://zuul.opendev.org/t/openstack/status].
+
* Bring CI infrastructure based on Zuul for projects hosted on pagure.io and src.fedoraproject.org.
 +
* Propose jobs and workflow of jobs around Pull Requests for Fedora packages (distgits on src.fedoraproject.org).
 +
 
 +
== News ==
 +
 
 +
=== January 14, 2020 ===
 +
 
 +
* Thanks to an update of the Zuul's Pagure driver on softwarefactory-project.io, the zuul user no longer needs to be in the collaborator **admin** group but only in the **commit** group. The project config helper tool has been updated.
 +
 
 +
=== November 13, 2019 ===
 +
 
 +
* Introduction of the service on the Fedora CI mailing list https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thread/CXDD2VYR6PHTR76JCJ5H5VEBIRG7YL37/
 +
 
 +
=== October 22, 2019 - Service is ready to be beta tested ===
 +
 
 +
* This wiki page contains the process to attach a repository from src.fedoraproject.org or pagure.io to Zuul.
 +
* Current known Issue: Sometime, src.fedoraproject.org, does not call Zuul event hooks (https://pagure.io/fedora-infrastructure/issue/8320). Then Zuul is unable to react and run jobs from Pagure events. This does not happen with pagure.io. This is a blocker for CI which needs to be reliable in order to gain confidence, I've pinged infra folks in that issue to help us debug it.
 +
 
 +
=== August 28, 2019 - Created Taiga EPIC ===
 +
 
 +
* Created the Taiga EPIC with the stories to achieve in the goal to provide Pagure PR Zuul jobs for Fedora distgit: https://teams.fedoraproject.org/project/ci/epic/14
 +
 
 +
=== Flock 2019 ===
 +
 
 +
* Talk: Pagure CI based on Zuul:
 +
** slides: [[File:Slides-flock2019.pdf|thumb|Talk slides flock 2019]]
 +
** video: not published yet
 +
 
 +
== What is Zuul/Nodepool ==
 +
 
 +
Zuul [https://zuul-ci.org/] is the CI and gating system from the Open Infrastructure Project [https://www.openstack.org/news/view/426/kata-containers-and-zuul-are-first-pilot-projects-confirmed-as-toplevel-open-infrastructure-projects-by-the-openstack-foundation-board]. It is able to scale and handles by default features such as artifacts sharing between jobs and cross Git repositories testing. You can see Zuul in action here [https://zuul.opendev.org/t/openstack/status].
  
 
Below is a list of features proposed by Zuul and its companion Nodepool:
 
Below is a list of features proposed by Zuul and its companion Nodepool:
Line 12: Line 42:
 
* Speculative testing of new jobs before merging: jobs will be run as they are submitted to make sure they behave as expected.
 
* Speculative testing of new jobs before merging: jobs will be run as they are submitted to make sure they behave as expected.
 
* Cross repositories dependencies: a jobs' workspace can include unmerged patches from other projects if specified
 
* Cross repositories dependencies: a jobs' workspace can include unmerged patches from other projects if specified
 +
* Cross provider: a jobs' workspace can include unmerged patches from other projects even when hosted on different provider like Github and Pagure.
 
* Parallel job run, only capped by resources available or predefined quotas
 
* Parallel job run, only capped by resources available or predefined quotas
 
* Automated jobs resources lifecycle management: resources like VMs or containers needed by a given job can be defined in-repository, spawned on demand at a job's start, and destroyed when the job is finished, or held for debugging
 
* Automated jobs resources lifecycle management: resources like VMs or containers needed by a given job can be defined in-repository, spawned on demand at a job's start, and destroyed when the job is finished, or held for debugging
Line 20: Line 51:
 
Until now, Zuul was only able to listen to Gerrit or Github events. Recently a new driver [https://review.opendev.org/604404/] allow Zuul to interface with Pagure as well. Pagure, Zuul and Nodepool could therefore combine into a very efficient CI/CD stack.
 
Until now, Zuul was only able to listen to Gerrit or Github events. Recently a new driver [https://review.opendev.org/604404/] allow Zuul to interface with Pagure as well. Pagure, Zuul and Nodepool could therefore combine into a very efficient CI/CD stack.
  
== Pagure PR tests via Zuul ==
+
== How to Zuul attach a Pagure repository on Zuul ==
  
We have created a Zuul driver for Pagure that allow Pagure to benefit all the nice features of Zuul. It only relies on the Pagure web hook and API system.
+
=== Configure the repository for Zuul ===
  
We are able to run jobs and report results on PRs opened on Pagure instances like https://src.fedoraproject.org or https://pagure.io.
+
In the project settings:
  
To do so we have deployed a Zuul/Nodepool instance (From the Software Factory project https://www.softwarefactory-project.io/) here: https://fedora.softwarefactory-project.io/zuul/
+
* Add "zuul" as "commit" collaborator in settings/Users & Groups
 +
* In Project Options
 +
** Check "Notify on pull-request flag"
 +
** Web-hooks:
 +
*** For a repository hosted on pagure.io add: https://softwarefactory-project.io/zuul/api/connection/pagure.io/payload
 +
*** For a repository hosted on src.fedoraproject.org add: https://softwarefactory-project.io/zuul/api/connection/src.fedoraproject.org/payload
 +
** (For gating, optional)
 +
*** Minimum score to merge pull-request: 0 or -1
 +
*** Open metadata access to all: False
 +
*** Tags: Add the "gateit" tag
  
=== Some POC use cases for the PR workflow ===
+
This helper script could be used to ease the setup:
 +
https://pagure.io/fedora-project-config/blob/master/f/tools/project-settings-helper
  
* Build package on PR: https://stg.pagure.io/python-mock-distgit/pull-request/1
+
=== Add the repository into the Zuul configuration ===
* Build package on PR which depends on another PR and validate a BuildRequire deps is handled [https://fedora.softwarefactory-project.io/logs/1/1/34192085e0eeb917376e5b7c27210b9468f0f597/check/rawhide-rpm-build/36955dd/job-output.txt.gz#_2019-05-20_13_43_59_894900] (artifacts sharing): https://stg.pagure.io/python-redis-distgit/pull-request/1
 
* Build package on PR then run a child job to validate package via the package included tests (standard test interface): https://stg.pagure.io/attr/pull-request/2. Here a negative test where the source is changed to trigger a failure [https://fedora.softwarefactory-project.io/logs/2/2/be13de0bd8d43212f58db7ae3d214dc514abb64a/check/rawhide-rpm-test/2ee9b76/job-output.txt.gz#_2019-05-21_12_42_11_951856]
 
* Build package on PR then validate STI included functional tests and RPM lint via two childs job https://stg.pagure.io/python-redis-distgit/pull-request/2. Note that this RPM build is done on Koji as a scratch build [https://fedora.softwarefactory-project.io/logs/2/2/7d41b28e4e8004f5249043145851457746dc8e32/check/rawhide-rpm-koji-scratch-build/6f51d45/job-output.txt.gz#_2019-06-13_09_21_20_457455].
 
  
=== Some PR workflows for src.fedoraproject.org ===
+
Open a Pull Request on https://pagure.io/fedora-project-config
  
* When a PR is proposed or changed or at the packager request (by typing a specific PR comment in Pagure)
+
Edit resources/fedora.yaml and add the repository such as:
# Parent job to scratch build the package on Koji
 
# Child job to run in package functional tests
 
# Child job to run RPM lint
 
  
Here is an example of such workflow on a PR to rpms/python-gear [https://src.fedoraproject.org/rpms/python-gear/pull-request/4]
+
* For a repository hosted on pagure.io:
  
* When the PR is merged or at the packager request (by typing a specific PR comment in Pagure)
+
resources:
# Job to to build of Koji is performed
+
  projects:
 +
    Fedora-Zuul-CI:
 +
      ...
 +
      source-repositories:
 +
        ...
 +
        - '''repository-name'''
  
See this Zuul UI page [https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/project/rpms/python-gear] to view jobs attached to a project.
 
  
Zuul has a branch matching system that make a job behave differently according to the branch where the PR is opened.
+
* For a repository hosted on src.fedoraproject.org:
That means PR on master could build on the Koji rawhide target and validate on a rawhide node, a PR on f30 branch could build against the f30 target and validate on the f30 node.
 
  
Advanced scenarios that involve multiple packages could be validated at PR level. For instance a PR on rpms/mod_wsgi could have a dependency on a rpms/httpd PR (assuming both projects have a rpm build job attached and based on Zuul). The jobs for the PR on rpms/mod_wsgi could use the RPM artifacts built for the dependent rpm/httpd PR for build (BuildRequire) and validation (Require). The dependencies chain is not limited to one dependency.
+
resources:
 +
  projects:
 +
    Fedora-Zuul-CI:
 +
      ...
 +
      source-repositories:
 +
        ...
 +
        - '''repository-name''':
 +
            '''connection: src.fedoraproject.org'''
 +
            '''zuul/include: []'''
  
=== How Git repositories are attached to Zuul ===
+
Once the Pull Request is accepted and merged, the repository should be available in the Zuul project list: https://fedora.softwarefactory-project.io/zuul/projects
  
Zuul serves a web server with a dedicated endpoint to receive web event hook notifications sent by Pagure. Events are the source that will trigger actions Zuul side like a job execution. To report back CI status, comments, or even merge on Pagure (gating), Zuul relies on the Pagure REST API.
+
=== Attach packaging jobs for a distgit repository on src.fedoraproject.org ===
  
Zuul needs a project API token to act on the Pagure REST API and a project web hook token to validate event payloads sent from Pagure to the Zuul endpoint. Both tokens are per project on Pagure thus to scale Zuul needs a user API token set with the "Modify an existing project" right to read the web hook token and create project API tokens. The owner of this user API token must be added as project admin.
+
'''''Note that all repositories with a name that match the regexp '^rpms/.*' will get, by default, the jobs defined by the build, lint, test templates attached.''''' Thus, if the default configuration is sufficient, then no need to edit the zuul.d/projects.yaml file as explained at the end of this section.
  
For instance on https://stg.pagure.io there is already a bot account for Zuul called zuulbot [https://stg.pagure.io/user/zuulbot]. To attach a project from this staging instance of Pagure to https://fedora.softwarefactory-project.io 's Zuul here is the process:
+
We have managed to provide some standard Pull Request's jobs and workflow. This idea is to reduce the amount of manual steps to propose packaging changes to Fedora by taking advantage of CI.
  
* Add "zuulbot" as admin in settings/Users & Groups
+
We propose a set of templates (a template can be seen as a workfow of jobs reacting to Pull Request events). Templates are available here https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml.
* In Settings
 
** Notify on pull-request flag
 
** Web-hooks: https://fedora.softwarefactory-project.io/zuul/api/connection/stg.pagure.io/payload
 
** (For gating, optional): Minimum score to merge pull-request: N
 
** (For gating, optional): Always merge
 
  
Finally https://fedora.softwarefactory-project.io 's Zuul must be tell to handle the project. This is done by opening a PR here https://pagure.io/fedora-project-config/blob/master/f/resources/fedora.yaml and have it merged. Feel free to try !
+
Templates configure jobs through three types of pipeline (not the same as Jenkins/Gitlab pipeline concept):
  
=== How to attach job(s) to a Git repository ===
+
* '''check''': Jobs within that pipeline are triggered when a Pull Request is Opened or Updated
 +
* '''gate''': Jobs within that pipeline are triggered when a Pull Request is approved (prior to be merged and closed)
 +
* '''promote''': Jobs with that pipeline are triggered when a Pull Request is closed and merged
  
Let's have a look to jobs attached to rpms/python-gear [https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/project/rpms/python-gear].
+
Conditions for a Pull Request to be approved:
  
The project's pipelines definition is located in [https://pagure.io/fedora-zuul-jobs-config/blob/master/f/zuul.d/projects.yaml]. There is a use of a template called "basic-check" [https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml]. The template defines which jobs will run in check, gate or post pipelines.
+
* All check pipelines's jobs succeeded
 +
* The Pull Request received the metadata tag: 'gateit'
  
Check, Gate, Post pipelines are defined here [https://pagure.io/fedora-project-config/blob/master/f/zuul.d/_pipelines.yaml]. Basically a pipeline defines which Pagure Event trigger jobs and what action Zuul wiil take when a job is a success or a failure.
+
Available jobs are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml):
  
* rawhide-rpm-koji-scratch-build in fedora-zuul-jobs [https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml]
+
* '''rpm-scratch-build''': Runs a scratch build on Koji (Koji target based on the PR's branch) and retrieves artifacts on the test node (rpms).
** based on a parent job [https://pagure.io/fedora-zuul-jobs-config/blob/master/f/zuul.d/jobs.yaml]
+
* '''rpm-build''': Runs a regular build on Koji (Koji target based on the PR's branch)
** notice that job definition format is purely YAML following a format expected by Zuul [https://zuul-ci.org/docs/zuul/user/config.html#job].
+
* '''rpm-lint''': Runs a rpmlint on artifacts (rpms) passed from the parent job
** the base job use post-run and run playbooks from [https://pagure.io/fedora-zuul-jobs-config/blob/master/f/playbooks/koji]
+
* '''rpm-rpminspect''': Runs rpminspect on artifacts (rpms) passed from the parent job. The job also finds the latest build on the related Koji tag and passes it to the rpminspect job so you get the rpminspect diff.
** the base job define a Zuul secret to authenticate on Koji
+
* '''rpm-test''': Executes distgit embedded tests ''tests/tests.yml'' on the related Fedora node (rawhide VM for master branch, Fedora 30 VM for f30 branch, ...). It is compatible with the '''Fedora standard-test-interface''' as STI dependencies are available on the node.
** playbooks use roles from pagure.io/zuul-distro-jobs project [https://pagure.io/zuul-distro-jobs]  (see roles: [{zuul: zuul-distro-jobs}] on the base job definition).
 
* rawhide-rpm-tests in fedora-zuul-jobs [https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml]
 
* artifact-rpm-lint in zuul-distro-jobs [https://pagure.io/zuul-distro-jobs/blob/master/f/zuul.d/jobs.yaml]
 
  
=== Current architecture ===
+
Available templates are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml):
  
==== Configuration ====
+
* '''build''': In the check pipeline, runs the ''rpm-scratch-build'' job in the check pipeline.
 +
* '''lint''': In the check pipeline, runs, once the parent job ''rpm-scratch-build'' is done, the ''rpm-linter'' + ''rpm-rpminspect'' jobs against the rpms built by the parent job.
 +
* '''test''': In the check pipeline, runs, once the parent job ''rpm-scratch-build'' is done, the ''rpm-test'' jobs against the rpms built by the parent job.
 +
* '''publish''': In the gate pipeline, run a noop job (always succeed), to make the gate succeed. This makes Zuul to merge the Pull Request. Then in the promote pipeline, runs the rpm-build job (regular koji build).
  
Zuul and Nodepool are hosted on https://fedora.softwarefactory-project.io. Here is the list of Pagure repositories that contain the configuration:
+
To attach a template to a repository you need to open a Pull Request on https://pagure.io/fedora-zuul-jobs-config.
  
* pagure.io/fedora-project-config [https://pagure.io/fedora-project-config/tree/master] Contains the Software Factory configuration. This is  where Nodepool providers,  Nodepool images are defined and where Git projects are attached to Zuul. Each PR proposed on that repository is validated by Zuul and deployed once merged. For instance see the "config-check" job on https://pagure.io/fedora-project-config/pull-request/18. And the deployment job "config-update" here [https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/634d9d3eb87f48cc870c606101d713d7]
+
Edit zuul.d/projects.yaml to add:
* pagure.io/fedora-zuul-jobs-config [https://pagure.io/fedora-zuul-jobs-config/tree/master] Contains trusted Zuul jobs configuration. This a Zuul trusted repository. Config changes included on PRs on that repository won't be taken speculatively in account by Zuul. A trusted repository is best suited to host pipelines, project's pipelines, secrets.
 
* pagure.io/fedora-zuul-jobs  [https://pagure.io/fedora-zuul-jobs/tree/master] Contains Zuul jobs configuration but as untrusted repository. Any changes to the Zuul configuration (via a PR) on that repository will be taken in account speculatively by Zuul.
 
  
===== zuul-distro-jobs =====
+
- project:
 +
    name: '''repository'''
 +
    templates:
 +
      - '''build'''
 +
      - '''lint'''
 +
      - '''gate'''
 +
      - '''promote'''
  
pagure.io/zuul-distro-jobs [https://pagure.io/zuul-distro-jobs/tree/master] is a generic suite of jobs and roles for Zuul dedicated for the  build and publish RPMs (and hopefully containers in the future). We are working on it with the idea to provide a ready to use Zuul jobs/roles library for Zuul users having to handle RPM build in the CI. Currently there is support for Koji, Copr, DLRN, Mock. Feel free to add more !
+
repository should be like "rpms/python-gear".
  
==== Nodepool nodes ====
+
Once the Pull Request is merged, the jobs of the chosen templates are attached to the repository.
  
For the moment three node labels are defined in Nodepool [https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/labels]
+
Currently f29, f30, f31 and master branches are supported. Only the x86_64 arch is supported.
  
* Fedora 29 cloud image (VM) [https://pagure.io/fedora-project-config/blob/master/f/nodepool/nodepool.yaml]
+
=== Sequence diagrams of the PR workflow ===
* Fedora 30 container (runc) [https://pagure.io/fedora-project-config/blob/master/f/nodepool/runC/fedora-rootfs.yaml]
 
* Centos 7 container (runc) - This is the by default container provided by Software Factory. It is not customizable.
 
  
Additional image definitions and containers could be simply provided in fedora-project-config repository.
+
These diagrams show interactions between components involved in a workflow where '''build''', '''lint''', '''publish''' templates are used.
  
== Buildsys build validation via AMQP and Zuul ==
+
==== Pull Request created/updated/rechecked workflow ====
  
We have some services that use Zuul to run jobs based on event received on fedmsg.
+
[[File:Distgit-pr-updated-workflow.png|800px]]
  
The goal was to run a rpm-lint job when a package is built on Fedora's Koji.
+
==== Pull Request approved workflow ====
  
Unfortunately Zuul is not designed out of the box to handle AMQP messages mainly because Zuul is designed to react on events generated by a Git or Code Review system where each event belong to a Git repository.
+
[[File:Distgit-pr-approved-workflow.png|400px]]
  
Nevertheless it is possible to simulate a Git repository when an event on fedmsg arrive and send the proper and expected Git events to Zuul.
+
=== Example of PR managed with that workflow ===
  
=== Architecture ===
+
* https://src.fedoraproject.org/rpms/nodepool/pull-request/6
  
==== The Zuul Gateway ====
+
== Nodepool nodes ==
  
The Zuul gateway [https://pagure.io/software-factory/zuul-gateway] is a service that generates virtual Git references to trigger Zuul events from non git events. It simulates a Pagure instance and interact with Zuul via the Zuul's Pagure driver. Zuul gateway is controlled via a REST API.
+
Jobs are configured to execute on a Fedora VM. The default jobs we provide are configured to use the right node based on the PR target branch. For instance the rpm-test job for the master branch execute on a rawhide VM.
  
==== The Fedora messaging Consumer ====
+
* Fedora rawhide cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-rawhide-cloud.yaml
 +
* Fedora 31 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-31-cloud.yaml
 +
* Fedora 30 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-30-cloud.yaml
 +
* Fedora 29 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-29-cloud.yaml
  
The Fedora messaging Consumer [https://pagure.io/fedora-project-config/blob/master/f/fedora-messaging/script.py] (See Consumer class) is a simple fedora-messaging [https://github.com/fedora-infra/fedora-messaging] Callback that filter events of interest (buildsys.build.state.change) and write events as JSON file on the files system (new/ directory). Those event files will be consumed by another process (processor).
+
== Projects using Zuul for the PR workflow ==
  
==== The Fedora messaging Processor ====
+
* https://src.fedoraproject.org/rpms/nodepool
 +
* https://src.fedoraproject.org/rpms/python-gear
 +
* https://src.fedoraproject.org/rpms/python-zuul-sphinx
  
The Fedora messaging Processor [https://pagure.io/fedora-project-config/blob/master/f/fedora-messaging/script.py] (See the main function) manage to (in a loop):
+
== FAQ ==
  
* look for an event file on the files system (generated by the consumer in new/ directory)
+
== Contacts ==
* call Koji to fetch the build tasks data and fetch the list of built rpms
 
* call the Zuul gateway to convert each event to a fake in memory Git repository (and create a fake .zuul.yaml)
 
* call the Zuul gateway to trigger a fake Pagure style event for Zuul
 
** Zuul reads the fake Git repository and process the job
 
* look for the status of the Zuul jobs
 
* move the event from new/ to done/ when the job finished
 
* ''report the status on fedmsg'' -> We don't have an account to publish on the bus at the moment.
 
  
=== Access the jobs results ===
+
For any questions or issues you can contact ''fbo'' on freenode on #fedora-ci or #softwarefactory or you can create an issue on https://pagure.io/fedora-zuul-jobs/issues.
  
Triggered jobs are processed by Zuul so jobs results are available in the builds page of Zuul [https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/builds?project=gateway]. Here [https://fedora.softwarefactory-project.io/zuul/t/fedora-staging/build/1fd457bdbf0d4129be0e7a4871fb820a] is a the result of the event [https://apps.fedoraproject.org/datagrepper/id?id=2019-04761ba7-14b7-4af0-afa3-6779a6b16694&is_raw=true&size=extra-large].
+
Also the work in progress and next steps are tracked into that Taiga EPIC: https://teams.fedoraproject.org/project/ci/epic/14

Revision as of 09:09, 17 January 2020

Zuul Based CI

Goals

  • Bring CI infrastructure based on Zuul for projects hosted on pagure.io and src.fedoraproject.org.
  • Propose jobs and workflow of jobs around Pull Requests for Fedora packages (distgits on src.fedoraproject.org).

News

January 14, 2020

  • Thanks to an update of the Zuul's Pagure driver on softwarefactory-project.io, the zuul user no longer needs to be in the collaborator **admin** group but only in the **commit** group. The project config helper tool has been updated.

November 13, 2019

October 22, 2019 - Service is ready to be beta tested

  • This wiki page contains the process to attach a repository from src.fedoraproject.org or pagure.io to Zuul.
  • Current known Issue: Sometime, src.fedoraproject.org, does not call Zuul event hooks (https://pagure.io/fedora-infrastructure/issue/8320). Then Zuul is unable to react and run jobs from Pagure events. This does not happen with pagure.io. This is a blocker for CI which needs to be reliable in order to gain confidence, I've pinged infra folks in that issue to help us debug it.

August 28, 2019 - Created Taiga EPIC

Flock 2019

What is Zuul/Nodepool

Zuul [1] is the CI and gating system from the Open Infrastructure Project [2]. It is able to scale and handles by default features such as artifacts sharing between jobs and cross Git repositories testing. You can see Zuul in action here [3].

Below is a list of features proposed by Zuul and its companion Nodepool:

  • Event-driven pipelines based on Code-Review or Pull-Request workflow: jobs can be triggered automatically when a PR is submitted, changed, approved, merged, or when the repository is tagged.
  • CI-as-code: jobs are defined as YAML + Ansible playbooks, pipeline definitions as YAML files. Zuul reads and loads those definitions directly from Git repositories.
  • Support for jobs inheritance, jobs dependencies, jobs chaining (with artifacts sharing).
  • Speculative testing of new jobs before merging: jobs will be run as they are submitted to make sure they behave as expected.
  • Cross repositories dependencies: a jobs' workspace can include unmerged patches from other projects if specified
  • Cross provider: a jobs' workspace can include unmerged patches from other projects even when hosted on different provider like Github and Pagure.
  • Parallel job run, only capped by resources available or predefined quotas
  • Automated jobs resources lifecycle management: resources like VMs or containers needed by a given job can be defined in-repository, spawned on demand at a job's start, and destroyed when the job is finished, or held for debugging
  • Job resources support of OpenStack, OpenShift, K8S, Static nodes, AWS.
  • Well-defined, reproducible job environments to eliminate flakiness
  • Speculative testing before merging (gating): if several patches are about to land at the same time, they are tested on the repository's future state.

Until now, Zuul was only able to listen to Gerrit or Github events. Recently a new driver [4] allow Zuul to interface with Pagure as well. Pagure, Zuul and Nodepool could therefore combine into a very efficient CI/CD stack.

How to Zuul attach a Pagure repository on Zuul

Configure the repository for Zuul

In the project settings:

This helper script could be used to ease the setup: https://pagure.io/fedora-project-config/blob/master/f/tools/project-settings-helper

Add the repository into the Zuul configuration

Open a Pull Request on https://pagure.io/fedora-project-config

Edit resources/fedora.yaml and add the repository such as:

  • For a repository hosted on pagure.io:
resources:
  projects:
    Fedora-Zuul-CI:
      ...
      source-repositories:
        ...
        - repository-name


  • For a repository hosted on src.fedoraproject.org:
resources:
  projects:
    Fedora-Zuul-CI:
      ...
      source-repositories:
        ...
        - repository-name:
            connection: src.fedoraproject.org
            zuul/include: []

Once the Pull Request is accepted and merged, the repository should be available in the Zuul project list: https://fedora.softwarefactory-project.io/zuul/projects

Attach packaging jobs for a distgit repository on src.fedoraproject.org

Note that all repositories with a name that match the regexp '^rpms/.*' will get, by default, the jobs defined by the build, lint, test templates attached. Thus, if the default configuration is sufficient, then no need to edit the zuul.d/projects.yaml file as explained at the end of this section.

We have managed to provide some standard Pull Request's jobs and workflow. This idea is to reduce the amount of manual steps to propose packaging changes to Fedora by taking advantage of CI.

We propose a set of templates (a template can be seen as a workfow of jobs reacting to Pull Request events). Templates are available here https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml.

Templates configure jobs through three types of pipeline (not the same as Jenkins/Gitlab pipeline concept):

  • check: Jobs within that pipeline are triggered when a Pull Request is Opened or Updated
  • gate: Jobs within that pipeline are triggered when a Pull Request is approved (prior to be merged and closed)
  • promote: Jobs with that pipeline are triggered when a Pull Request is closed and merged

Conditions for a Pull Request to be approved:

  • All check pipelines's jobs succeeded
  • The Pull Request received the metadata tag: 'gateit'

Available jobs are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml):

  • rpm-scratch-build: Runs a scratch build on Koji (Koji target based on the PR's branch) and retrieves artifacts on the test node (rpms).
  • rpm-build: Runs a regular build on Koji (Koji target based on the PR's branch)
  • rpm-lint: Runs a rpmlint on artifacts (rpms) passed from the parent job
  • rpm-rpminspect: Runs rpminspect on artifacts (rpms) passed from the parent job. The job also finds the latest build on the related Koji tag and passes it to the rpminspect job so you get the rpminspect diff.
  • rpm-test: Executes distgit embedded tests tests/tests.yml on the related Fedora node (rawhide VM for master branch, Fedora 30 VM for f30 branch, ...). It is compatible with the Fedora standard-test-interface as STI dependencies are available on the node.

Available templates are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml):

  • build: In the check pipeline, runs the rpm-scratch-build job in the check pipeline.
  • lint: In the check pipeline, runs, once the parent job rpm-scratch-build is done, the rpm-linter + rpm-rpminspect jobs against the rpms built by the parent job.
  • test: In the check pipeline, runs, once the parent job rpm-scratch-build is done, the rpm-test jobs against the rpms built by the parent job.
  • publish: In the gate pipeline, run a noop job (always succeed), to make the gate succeed. This makes Zuul to merge the Pull Request. Then in the promote pipeline, runs the rpm-build job (regular koji build).

To attach a template to a repository you need to open a Pull Request on https://pagure.io/fedora-zuul-jobs-config.

Edit zuul.d/projects.yaml to add:

- project:
    name: repository
    templates:
      - build
      - lint
      - gate
      - promote

repository should be like "rpms/python-gear".

Once the Pull Request is merged, the jobs of the chosen templates are attached to the repository.

Currently f29, f30, f31 and master branches are supported. Only the x86_64 arch is supported.

Sequence diagrams of the PR workflow

These diagrams show interactions between components involved in a workflow where build, lint, publish templates are used.

Pull Request created/updated/rechecked workflow

Distgit-pr-updated-workflow.png

Pull Request approved workflow

Distgit-pr-approved-workflow.png

Example of PR managed with that workflow

Nodepool nodes

Jobs are configured to execute on a Fedora VM. The default jobs we provide are configured to use the right node based on the PR target branch. For instance the rpm-test job for the master branch execute on a rawhide VM.

Projects using Zuul for the PR workflow

FAQ

Contacts

For any questions or issues you can contact fbo on freenode on #fedora-ci or #softwarefactory or you can create an issue on https://pagure.io/fedora-zuul-jobs/issues.

Also the work in progress and next steps are tracked into that Taiga EPIC: https://teams.fedoraproject.org/project/ci/epic/14