From Fedora Project Wiki
Comments and Explanations
The page source contains comments providing guidance to fill out each section. They are invisible when viewing this page. To read it, choose the "view source" link.
Copy the source to a new page before making changes! DO NOT EDIT THIS TEMPLATE FOR YOUR CHANGE PROPOSAL.


Unified Atomic Build Pipeline

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

Unify the build, test, signing, and delivery pipelines for all Fedora Atomic image variants -- Atomic Desktops, Fedora CoreOS, fedora-bootc, and Fedora IoT -- into a single manifest-driven pipeline running on Konflux. Adding a new variant becomes writing a YAML file instead of building a new pipeline from scratch. As a stretch goal, the pipeline will also support Hummingbird through an independent variant type.

Owner

Co-owners (affected SIGs):


Current status

  • Targeted release: Fedora Linux 45
  • Last updated: 2026-06-30
  • [<link to devel-announce post will be added by Wrangler> Announced]
  • [<will be assigned by the Wrangler> Discussion thread]
  • FESCo issue: <will be assigned by the Wrangler>
  • Tracker bug: <will be assigned by the Wrangler>
  • Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

Fedora currently maintains separate build systems for its bootc-based image variants. Fedora CoreOS uses Jenkins for disk images and Konflux for container builds. fedora-bootc has its own Konflux pipeline for development builds. Atomic Desktops, Fedora IoT, and Hummingbird each have their own build processes. Each team maintains its own build logic, test infrastructure, monitoring, and failure-triage workflows. Test suites are duplicated. Pipeline failures are triaged by hand. Adding a new variant means standing up a new pipeline from scratch.

This change introduces a single pipeline that builds any bootc-based variant from a declarative YAML manifest. The manifest declares the base image, repositories, output formats (OCI container, QCOW2, AMI, ISO, VMDK), and test suites. The pipeline reads the manifest and executes the same stages for every variant: ingest, build, layer, disk image generation (via bootc-image-builder), signing, publishing, and testing. No variant-specific logic is baked into the pipeline itself.

Two variant types are supported:

  • fedora-derived: Inherits Fedora repositories, builds FROM fedora-bootc. Used by Fedora CoreOS, Atomic Desktops, and Fedora IoT.
  • independent: Brings its own base image and/or repositories. Shares only pipeline infrastructure. Used by Hummingbird (stretch goal).

An example manifest for Fedora CoreOS: 

variant:
  name: fedora-coreos
  type: fedora-derived
  base_image:
    ref: quay.io/fedora/fedora-bootc:41
  repositories:
    inherit: fedora-base
    additional:
      - name: coreos-pool
  outputs:
    - type: oci-container
    - type: qcow2
    - type: ami
  testing:
    suites:
      - name: kola-base
        framework: kola-via-tmt
        applies_to: [qcow2, ami]

All stages run in Konflux as Tekton pipelines. Disk image generation runs in parallel per output format. Testing is dispatched through Testing Farm via Konflux IntegrationTestScenario, with kola-via-TMT wrapping CoreOS kola tests in TMT plans to enable multi-architecture testing (x86_64 and aarch64 today; s390x and ppc64le planned).

Three automation agents reduce operational toil:

  • Retry agent: Classifies failures against known transient patterns (network timeouts, Koji capacity, Testing Farm provisioning, known flaky tests). Retries up to 3x with backoff. Deterministic failures go directly to triage.
  • Triage agent: Collects logs and artifacts, classifies the failure, files a Pagure/Bugzilla issue with structured summary and pipeline run URL, and posts to the relevant Matrix channel.
  • Monitoring agent: Tracks daily pass/fail rates per variant, detects trend degradation, identifies flaky tests, and publishes a community-visible health dashboard.

Agents handle noise. Humans handle judgment. No auto-merge, no architecture decisions.

All variant manifests, shared pipeline definitions, and Tekton task definitions live in a single monorepo so all teams can see, contribute to, and review changes to the shared infrastructure.

Feedback

None yet.

Benefit to Fedora

Consolidating the per-variant pipelines into one shared pipeline reduces maintenance burden across all teams. Common bootc-integration tests run across all variants automatically, improving test coverage while eliminating duplication. Testing Farm integration provides consistent multi-architecture testing across all variants through a single framework.

Adding a new bootc-based variant to Fedora becomes writing a YAML manifest file instead of building a pipeline from scratch. This lowers the barrier for the community to create and maintain new Fedora deliverables.

The automation agents eliminate manual re-triggering of builds for transient failures and replace ad-hoc failure investigation with structured, automatically-filed issues. A community-visible health dashboard replaces per-team ad-hoc monitoring.

The architecture extends naturally to future variants with no pipeline changes.

Scope

  • Proposal owners:
    • Implement the unified Tekton pipeline in Konflux.
    • Define and publish the variant manifest YAML schema.
    • Create Day-1 manifests for fedora-bootc, Fedora CoreOS, and initial Atomic Desktop variants.
    • Implement kola-via-TMT wrapper for CoreOS test migration.
    • Develop the retry, triage, and monitoring agents.
    • Build the community health dashboard.
    • Stretch goal: Onboard Hummingbird as an independent variant type.
  • Other developers:
    • CoreOS SIG: Migrate FCOS build logic into a variant manifest; adapt kola tests for the TMT wrapper.
    • Atomic Desktops SIG: Create variant manifests for Silverblue, Kinoite, Sericea, and other desktop variants.
    • IoT SIG: Create a variant manifest for Fedora IoT.
    • bootc SIG: Ensure fedora-bootc base image compatibility as the foundation for all fedora-derived variants.
    • Hummingbird SIG (stretch goal): Create an independent variant manifest for Hummingbird.
  • Policies and guidelines: New guidelines for variant manifest authoring will be needed. FPC ticket: TBD.
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with the Fedora Strategy:
    • Reduces duplicated infrastructure and lowers the barrier to creating new image-based deliverables, supporting the growth of Fedora's bootc/immutable ecosystem.

Upgrade/compatibility impact

This change affects how Fedora Atomic images are built and tested, not how they are consumed by end users. There is no impact on installed systems or upgrade paths. Image references (registry URLs, download locations) remain unchanged. The ostree/bootc update mechanism is unaffected. Existing Ignition/Butane configurations for CoreOS continue to work.

The primary impact is on maintainers and contributors who currently interact with variant-specific build pipelines. They will need to transition to the monorepo workflow and manifest-driven build definitions. Migration guides will be provided.

Early Testing (Optional)

Do you require 'QA Blueprint' support? Y

How To Test

Once the unified pipeline is deployed in the Fedora Konflux tenant and producing Rawhide builds, the existing test infrastructure (OpenQA, Testing Farm) will begin testing the outputs automatically.

Users can verify the change by comparing the unified pipeline outputs against current official builds for functional equivalence: same package list, same boot behavior, same kola/TMT test results.

For the agentic automation specifically: the retry agent can be observed by checking whether transient pipeline failures are automatically retried without human intervention. The triage agent can be verified by confirming that persistent failures result in properly filed Pagure/Bugzilla issues with structured context.

User Experience

End users of Fedora Atomic variants will not notice any change. The images they download, install, and update continue to work exactly as before.

For contributors and maintainers, the experience improves: a single monorepo to contribute to, a YAML manifest to define new variants, shared monitoring, and automated triage that reduces the operational burden on SIG volunteers.

Dependencies

  • Konflux: The pipeline runs on Konflux (Tekton). Fedora's Konflux tenant must support the required pipeline complexity and resource demands.
  • bootc-image-builder: Used for disk image generation. Must support all required output formats.
  • Testing Farm / TMT: Used for multi-architecture test execution. The kola-via-TMT wrapper depends on TMT plan support.
  • fedora-bootc: The base image that all fedora-derived variants build FROM.
  • Image signing (releng #13115): Sigstore/cosign signing for Konflux-produced artifacts is a current open blocker.

No changes to the kernel or core OS packages are required. This change is purely in the build/test/delivery infrastructure.

Contingency Plan

  • Contingency mechanism: Revert to per-variant pipelines. Each variant team continues using their existing pipeline infrastructure (Jenkins for FCOS disk images, existing Konflux pipelines for bootc and Hummingbird, current compose processes for Atomic Desktops and IoT). No data migration is needed.
  • Contingency deadline: Beta Freeze
  • Blocks release? No.

Documentation

Notes will be written and kept as implementation progresses. The variant manifest schema will be published in the monorepo.

Release Notes

N/A

Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/AtomicUnifiedPipeline&oldid=777464"