Zuul-based-ci

Zuul Based CI

What is Zuul

Zuul is the CI and gating system form the OpenStack project. It is able to scale fine and handle by default features such as artifacts sharing between jobs and cross Git repositories testing. You can see Zuul in action here [1].

Below is a list of features proposed by Zuul and its companion Nodepool:

• Event-driven pipelines based on Code-Review or Pull-Request workflow: jobs can be triggered automatically when a PR is submitted, changed, approved, or when the repository is tagged.
• CI-as-code: jobs are defined as YAML + Ansible playbooks, pipeline definitions as YAML files. Zuul reads and loads those definitions directly from Git repositories.
• Support for jobs inheritance, jobs dependencies, jobs chaining (with artifacts sharing).
• Speculative testing of new jobs before merging: jobs will be run as they are submitted to make sure they behave as expected.
• Cross repositories dependencies: a jobs' workspace can include unmerged patches from other projects if specified
• Parallel job run, only capped by resources available or predefined quotas
• Automated jobs resources lifecycle management: resources like VMs or containers needed by a given job can be defined in-repository, spawned on demand at a job's start, and destroyed when the job is finished, or held for debugging
• Job resources support of OpenStack, OpenShift, K8S, Static nodes, AWS.
• Well-defined, reproducible job environments to eliminate flakiness
• Speculative testing before merging (gating): if several patches are about to land at the same time, they are tested on the repository's future state.

Until now, Zuul was only able to listen to Gerrit or Github events, a new upcoming driver [2] will allow Zuul to interface with Pagure as well. Pagure, Zuul and Nodepool could therefore combine into a very efficient CI/CD stack.

Pagure PR tests via the Zuul

Firstly we have worked on a Zuul driver for Pagure that bring all the nice features of Zuul usage with any recent Pagure instances (Zuul was designed for CI and Gating of OpenStack projects). It only relies on the Pagure web hook and API system.

We are able to run jobs and report results on PRs opened on Pagure instances like https://src.fedoraproject.org or https://pagure.io.

To do so we have deployed a Zuul/Nodepool instance (From the Software Factory project https://www.softwarefactory-project.io/) here: https://fedora.softwarefactoryproject.io/zuul/

Some use cases for the PR workflow

• Build package on PR: https://stg.pagure.io/python-mock-distgit/pull-request/1
• Build package on PR which depends on another PR and validate a BuildRequire deps is handled [3] (artifacts sharing): https://stg.pagure.io/python-redis-distgit/pull-request/1
• Build package on PR then run child jobs to validate package via the package included tests (standard test interface): https://stg.pagure.io/attr/pull-request/2. Here a negative test where the source is changed to trigger a failure [4]
• Build package on PR then validate STI included functional tests and RPM lint via two childs job https://stg.pagure.io/python-redis-distgit/pull-request/2. Note that this RPM build is done on Koji as a scratch build [5].

Some PR workflows for src.fedoraproject.org

• When a PR is proposed or changed or at the packager request (by typing a specific PR comment in Pagure)

1. Parent job to scratch build the package on Koji
2. Child job to run in package functional tests
3. Child job to run RPM lint

• When the PR is merged or at the packager request (by typing a specific PR comment in Pagure)

1. Job to to build of Koji is performed

Zuul has a branch matching system that make a job behave differently according to the branch where the PR is opened. That means PR on master could build on the Koji rawhide target and validate on a rawhide node, a PR on f30 branch could build against the f30 target and validate on the f30 node.

Advanced scenarios that involve multiple packages could be validated at PR level. For instance a PR on rpms/mod_wsgi could have a dependency on a rpms/httpd PR (assuming both projects have a rpm build job attached based on Zuul). The jobs for the PR on rpms/mod_wsgi could use the RPM artifacts built for the dependent rpm/httpd PR for build (BuildRequire) and validation (Require). The dependencies chain is not limited to one dependency.