From Fedora Project Wiki

(Attach packaging jobs for a distgit repository on src.fedoraproject.org)
 
(54 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
= Zuul Based CI =
 
= Zuul Based CI =
 +
 +
== Goals ==
 +
 +
* Bring CI infrastructure based on Zuul for projects hosted on pagure.io and src.fedoraproject.org.
 +
** Gerrit and Github are supported as well and we could for specific cases provide CI for projects hosted there.
 +
* Propose jobs and workflow of jobs around Pull Requests for Fedora packages (distgits on src.fedoraproject.org).
 +
 +
For Github projects, [https://packit.dev/testing-farm Packit] should be also considered.
  
 
== News ==
 
== News ==
 +
 +
=== January 24, 2020 ===
 +
 +
* Devconf.cz talk: CI/CD for Fedora packaging with Zuul
 +
** Slides with speaker notes: [[File:CI CD for Fedora packaging with Zuul - final - with notes.pdf|thumb]]
 +
 +
=== January 14, 2020 ===
 +
 +
* Thanks to an update of the Zuul's Pagure driver on softwarefactory-project.io, the zuul user no longer needs to be in the collaborator **admin** group but only in the **commit** group. The project config helper tool has been updated.
 +
 +
=== November 13, 2019 ===
 +
 +
* Introduction of the service on the Fedora CI mailing list https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thread/CXDD2VYR6PHTR76JCJ5H5VEBIRG7YL37/
  
 
=== October 22, 2019 - Service is ready to be beta tested ===
 
=== October 22, 2019 - Service is ready to be beta tested ===
  
* This wiki page contains the needed information to attach a src.fedoraproject.org or pagure.io to Zuul.
+
* This wiki page contains the process to attach a repository from src.fedoraproject.org or pagure.io to Zuul.
* Issue: src.fedoraproject.org, sometime, does not call event hooks (https://pagure.io/fedora-infrastructure/issue/8320). Then Zuul is unable to react and run jobs.
+
* Current known Issue: Sometime, src.fedoraproject.org, does not call Zuul event hooks (https://pagure.io/fedora-infrastructure/issue/8320). Then Zuul is unable to react and run jobs from Pagure events. This does not happen with pagure.io. This is a blocker for CI which needs to be reliable in order to gain confidence, I've pinged infra folks in that issue to help us debug it. UPDATE: "it was DNS" - resolved January 10, 2020
  
 
=== August 28, 2019 - Created Taiga EPIC ===
 
=== August 28, 2019 - Created Taiga EPIC ===
Line 20: Line 41:
 
== What is Zuul/Nodepool ==
 
== What is Zuul/Nodepool ==
  
Zuul [https://zuul-ci.org/] is the CI and gating system from the Open Infrastructure Project [https://www.openstack.org/news/view/426/kata-containers-and-zuul-are-first-pilot-projects-confirmed-as-toplevel-open-infrastructure-projects-by-the-openstack-foundation-board]. It is able to scale fine and handles by default features such as artifacts sharing between jobs and cross Git repositories testing. You can see Zuul in action here [https://zuul.opendev.org/t/openstack/status].
+
Zuul [https://zuul-ci.org/] is the CI and gating system from the Open Infrastructure Project [https://www.openstack.org/news/view/426/kata-containers-and-zuul-are-first-pilot-projects-confirmed-as-toplevel-open-infrastructure-projects-by-the-openstack-foundation-board]. It is able to scale and handles by default features such as artifacts sharing between jobs and cross Git repositories testing. You can see Zuul in action here [https://zuul.opendev.org/t/openstack/status].
  
 
Below is a list of features proposed by Zuul and its companion Nodepool:
 
Below is a list of features proposed by Zuul and its companion Nodepool:
Line 29: Line 50:
 
* Speculative testing of new jobs before merging: jobs will be run as they are submitted to make sure they behave as expected.
 
* Speculative testing of new jobs before merging: jobs will be run as they are submitted to make sure they behave as expected.
 
* Cross repositories dependencies: a jobs' workspace can include unmerged patches from other projects if specified
 
* Cross repositories dependencies: a jobs' workspace can include unmerged patches from other projects if specified
 +
* Cross provider: a jobs' workspace can include unmerged patches from other projects even when hosted on different provider like Github and Pagure.
 
* Parallel job run, only capped by resources available or predefined quotas
 
* Parallel job run, only capped by resources available or predefined quotas
 
* Automated jobs resources lifecycle management: resources like VMs or containers needed by a given job can be defined in-repository, spawned on demand at a job's start, and destroyed when the job is finished, or held for debugging
 
* Automated jobs resources lifecycle management: resources like VMs or containers needed by a given job can be defined in-repository, spawned on demand at a job's start, and destroyed when the job is finished, or held for debugging
Line 43: Line 65:
 
In the project settings:
 
In the project settings:
  
* Add "zuul" as admin in settings/Users & Groups
 
 
* In Project Options
 
* In Project Options
** Check "Notify on pull-request flag"
 
 
** Web-hooks:
 
** Web-hooks:
*** For a repository on pagure.io https://softwarefactory-project.io/zuul/api/connection/pagure.io/payload
+
*** For a repository hosted on pagure.io add: https://softwarefactory-project.io/zuul/api/connection/pagure.io/payload
*** For a repository on src.fedoraproject.org https://softwarefactory-project.io/zuul/api/connection/src.fedoraproject.org/payload
+
*** For a repository hosted on src.fedoraproject.org add: https://softwarefactory-project.io/zuul/api/connection/src.fedoraproject.org/payload
 
** (For gating, optional)
 
** (For gating, optional)
*** Check "Always merge"
+
*** Minimum score to merge pull-request: 0 or -1
*** Minimum score to merge pull-request: 0
+
*** Open metadata access to all: False
*** Tags: Add the "gateit" tag
+
*** In "Users & Groups" setting: add "zuul" as "commit" collaborator
 +
*** In "Tags" setting: add the "gateit" tag
 +
 
 +
This helper script could be used to ease the setup:
 +
https://pagure.io/fedora-project-config/blob/master/f/tools/project-settings-helper
  
 
=== Add the repository into the Zuul configuration ===
 
=== Add the repository into the Zuul configuration ===
Line 58: Line 82:
 
Open a Pull Request on https://pagure.io/fedora-project-config
 
Open a Pull Request on https://pagure.io/fedora-project-config
  
Edit resources/fedora.yaml and add the repository such as!
+
Edit resources/fedora.yaml and add the repository such as:
  
 
* For a repository hosted on pagure.io:
 
* For a repository hosted on pagure.io:
Line 81: Line 105:
 
         - '''repository-name''':
 
         - '''repository-name''':
 
             '''connection: src.fedoraproject.org'''
 
             '''connection: src.fedoraproject.org'''
 +
            '''zuul/include: []'''
  
 
Once the Pull Request is accepted and merged, the repository should be available in the Zuul project list: https://fedora.softwarefactory-project.io/zuul/projects
 
Once the Pull Request is accepted and merged, the repository should be available in the Zuul project list: https://fedora.softwarefactory-project.io/zuul/projects
  
=== How to attach packaging jobs for a distgit repository on src.fedoraproject.org ===
+
=== Attach packaging jobs for a distgit repository on src.fedoraproject.org ===
 +
 
 +
'''''Note that all repositories with a name that match the regexp '^rpms/.*' will get, by default, the jobs defined by the build, lint and test templates attached.''''' Thus, if the default configuration is sufficient, then no need to edit the zuul.d/projects.yaml file as explained at the end of this section.
  
 
We have managed to provide some standard Pull Request's jobs and workflow. This idea is to reduce the amount of manual steps to propose packaging changes to Fedora by taking advantage of CI.
 
We have managed to provide some standard Pull Request's jobs and workflow. This idea is to reduce the amount of manual steps to propose packaging changes to Fedora by taking advantage of CI.
  
We propose a set of templates (template can be seen as workfow of jobs reacting to Pull Request events). Templates are available here https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml.
+
We propose a set of templates (a template can be seen as a workfow of jobs reacting to Pull Request events). Templates are available here https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml.
  
The templates attach jobs through three type of pipeline (not the same as Jenkins/Gitlab pipeline concept):
+
Templates configure jobs through three types of pipeline (not the same as Jenkins/Gitlab pipeline concept):
  
 
* '''check''': Jobs within that pipeline are triggered when a Pull Request is Opened or Updated
 
* '''check''': Jobs within that pipeline are triggered when a Pull Request is Opened or Updated
Line 96: Line 123:
 
* '''promote''': Jobs with that pipeline are triggered when a Pull Request is closed and merged
 
* '''promote''': Jobs with that pipeline are triggered when a Pull Request is closed and merged
  
The available jobs are:
+
Conditions for a Pull Request to be approved:
  
* '''rpm-scratch-build''': Run a scratch build on Koji (target based on the PR's branch) and retrieve artifacts (rpms).
+
* All check pipelines's jobs succeeded
* '''rpm-build''': Run a regular build on Koji (target based on the PR's branch)
+
* The Pull Request received the metadata tag: 'gateit'
* '''rpm-lint''': Run a rpmlint on artifacts (rpms) passed by the parent job
 
* '''rpm-rpminspect''': Run rpminspect on artifacts (rpms) passed by the parent job. The job also find the latest build on the related koji tag and pass it the rpminspect so you get the rpminspect diff.
 
* '''rpm-test''': Execute distgit embedded test ''tests/tests.yml'' on the related Fedora node (rawhide VM for master branch, Fedora 30 VM for f30 branch, ...). It is compatible with the '''Fedora standard-test-interface''' as dependencies are available on the node.
 
  
Available templates:
+
Available jobs are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml):
  
* '''build''': run the ''rpm-scratch-build'' (scratch koji) job in the check pipeline.
+
* '''rpm-scratch-build''': Runs a scratch build on Koji (Koji target based on the PR's branch) and retrieves artifacts on the test node (rpms).
* '''build-lint''': run the ''rpm-scratch-build'' job then run the ''rpm-linter'' + ''rpm-rpminspect'' jobs against the rpms built by the first job.
+
* '''rpm-build''': Runs a regular build on Koji (Koji target based on the PR's branch)
* '''build-lint-test''': run the ''rpm-scratch-build'' job then run the ''rpm-linter'' + ''rpm-rpminspect'' + ''rpm-test'' jobs against the rpms built by the first job.
+
* '''rpm-linter''': Runs a rpmlint on artifacts (rpms) passed from the parent job
* '''build-lint-test-gate''': same as '''build-lint-test''' + the CI will merge the Pull Request if the Pull Request receive the metadata tag "gateit".
+
* '''rpm-rpminspect''': Runs rpminspect on artifacts (rpms) passed from the parent job. The job also finds the latest build on the related Koji tag and passes it to the rpminspect job so you get the rpminspect diff.
* '''build-lint-test-gate-promote''': same as '''build-lint-test-gate''' + rpm-build (regular koji build) when the Pull Request is merged and closed.
+
* '''rpm-test''': Executes distgit embedded tests ''tests/tests.yml'' on the related Fedora node (rawhide VM for master branch, Fedora 30 VM for f30 branch, ...). It is compatible with the '''Fedora standard-test-interface''' as STI dependencies are available on the node.
  
There are some other templates available but now you should have a clear understanding of them.
+
Available templates are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml):
 +
 
 +
* '''build''': In the check pipeline, runs the ''rpm-scratch-build'' job in the check pipeline.
 +
* '''lint''': In the check pipeline, runs, once the parent job ''rpm-scratch-build'' is done, the ''rpm-linter'' + ''rpm-rpminspect'' jobs against the rpms built by the parent job.
 +
* '''test''': In the check pipeline, runs, once the parent job ''rpm-scratch-build'' is done, the ''rpm-test'' jobs against the rpms built by the parent job.
 +
* '''publish''': In the gate pipeline, run a noop job (always succeed), to make the gate succeed. This makes Zuul to merge the Pull Request. Then in the promote pipeline, runs the rpm-build job (regular koji build).
  
 
To attach a template to a repository you need to open a Pull Request on https://pagure.io/fedora-zuul-jobs-config.
 
To attach a template to a repository you need to open a Pull Request on https://pagure.io/fedora-zuul-jobs-config.
Line 121: Line 150:
 
     name: '''repository'''
 
     name: '''repository'''
 
     templates:
 
     templates:
       - '''build-lint-gate-promote'''
+
       - '''build'''
 +
      - '''lint'''
 +
      - '''test'''
 +
      - '''publish'''
  
 
repository should be like "rpms/python-gear".
 
repository should be like "rpms/python-gear".
  
Once the Pull Request is merged should should the jobs of the chosen template displayed by clicking on the related project on the page https://fedora.softwarefactory-project.io/zuul/projects.
+
Once the Pull Request is merged, the jobs of the chosen templates are attached to the repository.
 +
 
 +
Currently f30, f31 and master branches are supported. Only the x86_64 arch is supported.
 +
 
 +
=== Sequence diagrams of the PR workflow ===
 +
 
 +
These diagrams show interactions between components involved in a workflow where '''build''', '''lint''', '''test''', '''publish''' templates are used.
 +
 
 +
==== Pull Request created/updated/rechecked workflow ====
 +
 
 +
[[File:Distgit-pr-updated-workflow-simple.png|800px]]
 +
 
 +
==== Pull Request approved workflow ====
 +
 
 +
[[File:Distgit-pr-approved-workflow-simple.png|800px]]
 +
 
 +
=== Example of PR managed with that workflow ===
  
Currently f29, f30, f31 and master branches are supported. Only the x86_64 arch is supported.
+
* https://src.fedoraproject.org/rpms/nodepool/pull-request/6
  
 
== Nodepool nodes ==
 
== Nodepool nodes ==
  
Jobs are configured to execute on a Fedora VM. The default jobs we provide are configured to use the right node. For instance the rpm-test job  
+
Jobs are configured to execute on a Fedora VM. The default jobs we provide are configured to use the right node based on the PR target branch. For instance the rpm-test job for the master branch execute on a rawhide VM.
 +
 
 +
* Fedora rawhide cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/virt_images/cloud-fedora-rawhide.yaml
 +
* Fedora 31 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/virt_images/cloud-fedora-31.yaml
 +
* Fedora 30 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/virt_images/cloud-fedora-30.yaml
 +
 
 +
== Fedora's projects using Zuul ==
 +
 
 +
Projects registered into the Fedora's Zuul are listed here: [https://fedora.softwarefactory-project.io/zuul/projects projects list]
 +
 
 +
== Advanced topics ==
 +
 
 +
=== In-project Zuul configuration (.zuul.yaml) ===
 +
 
 +
Zuul is very flexible about where configuration elements are defined. Each piece of a Zuul configuration is managed in a git repository and the full configuration can be spread across an unlimited amount of git repositories.
 +
 
 +
==== For distgit projects on src.fedoraproject.org ====
 +
 
 +
For every distgit projects we believe that the Zuul pipeline should be similar, that's why be default we do not allow package maintainer to add custom Zuul configurations in-distgit. But for specific cases a maintainer may need custom and non generic jobs and project's pipeline.
 +
 
 +
If there is a need for such custom in-distgit Zuul configuration then Zuul must be told to read that configuration from the distgit. To do so:
 +
 
 +
Open a Pull Request on https://pagure.io/fedora-project-config
 +
 
 +
Edit '''resources/fedora.yaml''' and add modify such as:
 +
 
 +
resources:
 +
  projects:
 +
    Fedora-Zuul-CI:
 +
      ...
 +
      source-repositories:
 +
        ...
 +
        - repository-name:
 +
            connection: src.fedoraproject.org
 +
            zuul/include:
 +
              '''- project'''
 +
              '''- job'''
 +
 
 +
This will let Zuul load jobs and project stanza from the '''.zuul.yaml''' stored in the distgit repository.
 +
 
 +
Then is the distgit repository, open a Pull Request with a '''.zuul.yaml''' such as:
 +
 
 +
  - project:
 +
      templates:
 +
        - build
 +
        - lint
 +
        - test
 +
      check:
 +
        jobs:
 +
          - noop
 +
 
 +
The '''noop''' special job should run as long as other jobs from the templates. From there you can write a custom job to replace '''noop'''. This project pipeline allows to keep the default Zuul jobs for a distgit project but also to add custom jobs to the default pipeline.
 +
 
 +
In the '''.zuul.yaml''' add:
 +
 
 +
  - job:
 +
      name: my-job
 +
      description: My custom job
 +
      run: ci/run.yaml
 +
 
 +
And in a file called '''ci/run.yaml''' write the job Ansible playbook:
 +
 
 +
  - hosts: all
 +
    tasks:
 +
      - name: List project directory on the test system
 +
        command: ls -al {{ansible_user_dir}}/{{zuul.project.src_dir}}
 +
 
 +
==== For a source project on pagure.io ====
 +
 
 +
The default configuration lets Zuul load all Zuul configuration elements.
 +
 
 +
Then is the repository, open a Pull Request with a '''.zuul.yaml''' such as:
 +
 
 +
  - project:
 +
      check:
 +
        jobs:
 +
          - noop
 +
 
 +
The '''noop''' special job should run as long as other jobs from the templates. From there you can write a custom job to replace '''noop'''.
 +
 
 +
In the '''.zuul.yaml''' add:
 +
 
 +
  - job:
 +
      name: my-job
 +
      description: My custom job
 +
      run: ci/run.yaml
 +
 
 +
And in a file called '''ci/run.yaml''' write the job Ansible playbook:
 +
 
 +
  - hosts: all
 +
    tasks:
 +
      - name: List project directory on the test system
 +
        command: ls -al {{ansible_user_dir}}/{{zuul.project.src_dir}}
 +
 
 +
== Contacts ==
 +
 
 +
For any questions or issues you can contact ''fbo'' on freenode on #fedora-ci or #softwarefactory or you can create an issue on https://pagure.io/fedora-ci/general/issues.
 +
 
 +
Also the work in progress and next steps are tracked into that Taiga EPIC: https://teams.fedoraproject.org/project/ci/epic/14
  
* Fedora rawhide cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-rawhide-cloud.yaml
+
== FAQ ==
* Fedora 31 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-31-cloud.yaml
 
* Fedora 30 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-30-cloud.yaml
 
* Fedora 29 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-29-cloud.yaml
 

Latest revision as of 14:20, 18 March 2020

Zuul Based CI

Goals

  • Bring CI infrastructure based on Zuul for projects hosted on pagure.io and src.fedoraproject.org.
    • Gerrit and Github are supported as well and we could for specific cases provide CI for projects hosted there.
  • Propose jobs and workflow of jobs around Pull Requests for Fedora packages (distgits on src.fedoraproject.org).

For Github projects, Packit should be also considered.

News

January 24, 2020

January 14, 2020

  • Thanks to an update of the Zuul's Pagure driver on softwarefactory-project.io, the zuul user no longer needs to be in the collaborator **admin** group but only in the **commit** group. The project config helper tool has been updated.

November 13, 2019

October 22, 2019 - Service is ready to be beta tested

  • This wiki page contains the process to attach a repository from src.fedoraproject.org or pagure.io to Zuul.
  • Current known Issue: Sometime, src.fedoraproject.org, does not call Zuul event hooks (https://pagure.io/fedora-infrastructure/issue/8320). Then Zuul is unable to react and run jobs from Pagure events. This does not happen with pagure.io. This is a blocker for CI which needs to be reliable in order to gain confidence, I've pinged infra folks in that issue to help us debug it. UPDATE: "it was DNS" - resolved January 10, 2020

August 28, 2019 - Created Taiga EPIC

Flock 2019

What is Zuul/Nodepool

Zuul [1] is the CI and gating system from the Open Infrastructure Project [2]. It is able to scale and handles by default features such as artifacts sharing between jobs and cross Git repositories testing. You can see Zuul in action here [3].

Below is a list of features proposed by Zuul and its companion Nodepool:

  • Event-driven pipelines based on Code-Review or Pull-Request workflow: jobs can be triggered automatically when a PR is submitted, changed, approved, merged, or when the repository is tagged.
  • CI-as-code: jobs are defined as YAML + Ansible playbooks, pipeline definitions as YAML files. Zuul reads and loads those definitions directly from Git repositories.
  • Support for jobs inheritance, jobs dependencies, jobs chaining (with artifacts sharing).
  • Speculative testing of new jobs before merging: jobs will be run as they are submitted to make sure they behave as expected.
  • Cross repositories dependencies: a jobs' workspace can include unmerged patches from other projects if specified
  • Cross provider: a jobs' workspace can include unmerged patches from other projects even when hosted on different provider like Github and Pagure.
  • Parallel job run, only capped by resources available or predefined quotas
  • Automated jobs resources lifecycle management: resources like VMs or containers needed by a given job can be defined in-repository, spawned on demand at a job's start, and destroyed when the job is finished, or held for debugging
  • Job resources support of OpenStack, OpenShift, K8S, Static nodes, AWS.
  • Well-defined, reproducible job environments to eliminate flakiness
  • Speculative testing before merging (gating): if several patches are about to land at the same time, they are tested on the repository's future state.

Until now, Zuul was only able to listen to Gerrit or Github events. Recently a new driver [4] allow Zuul to interface with Pagure as well. Pagure, Zuul and Nodepool could therefore combine into a very efficient CI/CD stack.

How to Zuul attach a Pagure repository on Zuul

Configure the repository for Zuul

In the project settings:

This helper script could be used to ease the setup: https://pagure.io/fedora-project-config/blob/master/f/tools/project-settings-helper

Add the repository into the Zuul configuration

Open a Pull Request on https://pagure.io/fedora-project-config

Edit resources/fedora.yaml and add the repository such as:

  • For a repository hosted on pagure.io:
resources:
  projects:
    Fedora-Zuul-CI:
      ...
      source-repositories:
        ...
        - repository-name


  • For a repository hosted on src.fedoraproject.org:
resources:
  projects:
    Fedora-Zuul-CI:
      ...
      source-repositories:
        ...
        - repository-name:
            connection: src.fedoraproject.org
            zuul/include: []

Once the Pull Request is accepted and merged, the repository should be available in the Zuul project list: https://fedora.softwarefactory-project.io/zuul/projects

Attach packaging jobs for a distgit repository on src.fedoraproject.org

Note that all repositories with a name that match the regexp '^rpms/.*' will get, by default, the jobs defined by the build, lint and test templates attached. Thus, if the default configuration is sufficient, then no need to edit the zuul.d/projects.yaml file as explained at the end of this section.

We have managed to provide some standard Pull Request's jobs and workflow. This idea is to reduce the amount of manual steps to propose packaging changes to Fedora by taking advantage of CI.

We propose a set of templates (a template can be seen as a workfow of jobs reacting to Pull Request events). Templates are available here https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml.

Templates configure jobs through three types of pipeline (not the same as Jenkins/Gitlab pipeline concept):

  • check: Jobs within that pipeline are triggered when a Pull Request is Opened or Updated
  • gate: Jobs within that pipeline are triggered when a Pull Request is approved (prior to be merged and closed)
  • promote: Jobs with that pipeline are triggered when a Pull Request is closed and merged

Conditions for a Pull Request to be approved:

  • All check pipelines's jobs succeeded
  • The Pull Request received the metadata tag: 'gateit'

Available jobs are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/jobs.yaml):

  • rpm-scratch-build: Runs a scratch build on Koji (Koji target based on the PR's branch) and retrieves artifacts on the test node (rpms).
  • rpm-build: Runs a regular build on Koji (Koji target based on the PR's branch)
  • rpm-linter: Runs a rpmlint on artifacts (rpms) passed from the parent job
  • rpm-rpminspect: Runs rpminspect on artifacts (rpms) passed from the parent job. The job also finds the latest build on the related Koji tag and passes it to the rpminspect job so you get the rpminspect diff.
  • rpm-test: Executes distgit embedded tests tests/tests.yml on the related Fedora node (rawhide VM for master branch, Fedora 30 VM for f30 branch, ...). It is compatible with the Fedora standard-test-interface as STI dependencies are available on the node.

Available templates are (https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml):

  • build: In the check pipeline, runs the rpm-scratch-build job in the check pipeline.
  • lint: In the check pipeline, runs, once the parent job rpm-scratch-build is done, the rpm-linter + rpm-rpminspect jobs against the rpms built by the parent job.
  • test: In the check pipeline, runs, once the parent job rpm-scratch-build is done, the rpm-test jobs against the rpms built by the parent job.
  • publish: In the gate pipeline, run a noop job (always succeed), to make the gate succeed. This makes Zuul to merge the Pull Request. Then in the promote pipeline, runs the rpm-build job (regular koji build).

To attach a template to a repository you need to open a Pull Request on https://pagure.io/fedora-zuul-jobs-config.

Edit zuul.d/projects.yaml to add:

- project:
    name: repository
    templates:
      - build
      - lint
      - test
      - publish

repository should be like "rpms/python-gear".

Once the Pull Request is merged, the jobs of the chosen templates are attached to the repository.

Currently f30, f31 and master branches are supported. Only the x86_64 arch is supported.

Sequence diagrams of the PR workflow

These diagrams show interactions between components involved in a workflow where build, lint, test, publish templates are used.

Pull Request created/updated/rechecked workflow

Distgit-pr-updated-workflow-simple.png

Pull Request approved workflow

Distgit-pr-approved-workflow-simple.png

Example of PR managed with that workflow

Nodepool nodes

Jobs are configured to execute on a Fedora VM. The default jobs we provide are configured to use the right node based on the PR target branch. For instance the rpm-test job for the master branch execute on a rawhide VM.

Fedora's projects using Zuul

Projects registered into the Fedora's Zuul are listed here: projects list

Advanced topics

In-project Zuul configuration (.zuul.yaml)

Zuul is very flexible about where configuration elements are defined. Each piece of a Zuul configuration is managed in a git repository and the full configuration can be spread across an unlimited amount of git repositories.

For distgit projects on src.fedoraproject.org

For every distgit projects we believe that the Zuul pipeline should be similar, that's why be default we do not allow package maintainer to add custom Zuul configurations in-distgit. But for specific cases a maintainer may need custom and non generic jobs and project's pipeline.

If there is a need for such custom in-distgit Zuul configuration then Zuul must be told to read that configuration from the distgit. To do so:

Open a Pull Request on https://pagure.io/fedora-project-config

Edit resources/fedora.yaml and add modify such as:

resources:
  projects:
    Fedora-Zuul-CI:
      ...
      source-repositories:
        ...
        - repository-name:
            connection: src.fedoraproject.org
            zuul/include:
              - project
              - job

This will let Zuul load jobs and project stanza from the .zuul.yaml stored in the distgit repository.

Then is the distgit repository, open a Pull Request with a .zuul.yaml such as:

 - project:
     templates:
       - build
       - lint
       - test
     check:
       jobs:
         - noop

The noop special job should run as long as other jobs from the templates. From there you can write a custom job to replace noop. This project pipeline allows to keep the default Zuul jobs for a distgit project but also to add custom jobs to the default pipeline.

In the .zuul.yaml add:

 - job:
     name: my-job
     description: My custom job
     run: ci/run.yaml

And in a file called ci/run.yaml write the job Ansible playbook:

 - hosts: all
   tasks:
     - name: List project directory on the test system
       command: ls -al {{ansible_user_dir}}/{{zuul.project.src_dir}}

For a source project on pagure.io

The default configuration lets Zuul load all Zuul configuration elements.

Then is the repository, open a Pull Request with a .zuul.yaml such as:

 - project:
     check:
       jobs:
         - noop

The noop special job should run as long as other jobs from the templates. From there you can write a custom job to replace noop.

In the .zuul.yaml add:

 - job:
     name: my-job
     description: My custom job
     run: ci/run.yaml

And in a file called ci/run.yaml write the job Ansible playbook:

 - hosts: all
   tasks:
     - name: List project directory on the test system
       command: ls -al {{ansible_user_dir}}/{{zuul.project.src_dir}}

Contacts

For any questions or issues you can contact fbo on freenode on #fedora-ci or #softwarefactory or you can create an issue on https://pagure.io/fedora-ci/general/issues.

Also the work in progress and next steps are tracked into that Taiga EPIC: https://teams.fedoraproject.org/project/ci/epic/14

FAQ