Latest revision as of 20:02, 17 August 2016

Release Engineering Automation Workflow Engine

Summary

Centralized entry point, logging, and dash board for pre-defined Automated Workflow tasks used by the Release Engineering team with delegation and self-service tasks for members of various teams who normally depend on Release Engineering for various tasks.

Owner

Name: Adam Miller
Email: maxamillion@fedoraproject.org
Release notes owner:

Current status

Targeted release: Fedora 25
Last updated: 2016-08-17
Tracker bug: #1345757

Detailed Description

+--------------+                                            +----------------+
|              |                 +------------+             |                |
|  AutoCloud   |<----------------+            +------------>+   Taskotron    |
|              |                 |   fedmsg   |             |                |
|              +---------------->|            |<------------+                |
+--------------+                 |            |             +----------------+
                                 +------------+
                                      ^
                                      |
                                      |
                                      |
                                      |
                                      |
                                      |
                   +------------------+-----------------+
                   |                                    |
                   |      Release Engineering           +----------------+
                   |      Workflow Automation Engine    |                |
                   |                                    |                |
                   +------------------+-----------------+                |
                         |            |                                  |
                         |            |                                  |
       +-----------------+            |                                  |
       |                              |                                  |
       |                              |                                  |
       V                              V                                  |
+-------------+                   +--------------+                       |
|             |                   |              |                       |
|    bodhi    |                   |              |                       |
|             |                   |    pungi     |                       |
+-------------+                   |              |                       |
                                  |              |                       |
                                  +----------+---+                       |
                                       ^     |                           V
                                       |     |                +---------------+
                                       |     |                |               |
                                       |     +--------------->|     koji      |
                                       |                      |               |
                                       +----------------------+               |
                                                              +---------------+

Currently Fedora Release Engineering Automation tasks are performed by various scripts run on various machines within the Fedora Infrastructure with no real centralized logging. Some of these are automated by chron jobs and some run by hand by request of various members within the Fedora Community, normally around Fedora Test Days. Finding information about old tasks is not always the easiest of things to do and the delegation of tasks is currently not available. The goal here is to provide a solution that removes those barriers.

Workflows will be executed and potentially orchestrate actions between multiple other systems or tools such as bodhi, pungi, and koji. Fedmsgs will be emitted with information about the start and completion of workflows along with metadata about them.

In the event of a compose, certain fedmsg output will be picked up by taskotron and autocloud to perform various levels of testing.

Technical Implementation

Everything will be powered by Ansible as this is a toolchain that both Fedora Infrastucture and Fedora Release Engineering is familiar with and has been using heavily for automation tasks. We are simply aiming to solve a new automation problem space with the same tool and a different set of rules/policy.

https://www.ansible.com/

The main component that will define the workflows is going to be Ansible Playbooks.

Ansible

Tasks or sets of tasks should be in an "Include Playbook" such that they are not meant to stand on their own but should be included by other Playbooks or an Ansible Role.

Workflow Playbooks should effectively be "glue" that supply necessary variables to make the "Include Playbooks" and Roles useful for the Workflow at hand.

Ansible Execution

A goal of this proposal is to have a way to execute tasks or jobs that can be centralized, role based

UPDATE (2016-07-26): Loopabull has been chosen, eventually when Ansible Tower has been Open Sourced we would like to push features upstream to it and migrate because it also provides many features we and the Fedora Infrastructure Team would like to have.

The software that becomes the "Workflow Engine" itself is currently being evaluated, this will be the thing that actually executes the Ansible playbooks. The following options are being looked at:

Ansible Tower
- https://www.ansible.com/tower
- Currently closed source but has been announced by Red Hat that it will be open sourced since the acquisition of Ansible
Jenkins CI
- 2.0 Pipeline https://jenkins.io/2.0/
- Blue Ocean https://jenkins.io/blog/2016/05/26/introducing-blue-ocean/
Zuul v3
- http://inaugust.com/talks/zuulv3.html#/
- https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html
Outhouse
- https://pagure.io/outhouse
Taskotron (future stuff)
- https://taskotron.fedoraproject.org/
- https://bitbucket.org/tflink/taskotron-webtrigger/src/536f3c05072f8c6a167718d0aa6710eb38aecc1a/?at=feature%2Fnextgen-poc
Semaphore
- Open Source Alternative to Tower
- https://github.com/ansible-semaphore/semaphore
Loopabull
- Event loop driven Ansible playbook execution engine
- https://github.com/maxamillion/loopabull

Benefit to Fedora

The goal here is the benefit the Fedora Contributor Community at large by making certain processes within Release Engineering be able to be more rapidly iterated upon, allowing for changes to processes to become more flexible. Another goal is to make Fedora Release Engineering more approachable by making it easier to contribute to work we d and make it easier for new members of the community to join in the Fedora Release Engineering group.

Scope

Proposal owners

Proposal owners shall have to:

Determine what the "Engine" will be after evaluation and working with the Fedora RelEng and Infrastructure teams for advisement.
Deploy RelEng Automation Workflow Engine
- Fully automated deployment in Fedora Infrastructure Ansible
Document Workflow Automation
- How workflows are created
- How to run workflows
- How new contributors can get started

Task matrix

This is a RACI matrix for tasks required to implement the RelEng Automation Workflow Engine. Work is tracked in Taiga: https://taiga.fedorainfracloud.org/project/acarter-fedora-docker-atomic-tooling/wiki/home

Is this current?

It is, as of 2016-08-17

Definitions

Here, we're using what Wikipedia calls "RACI (alternative scheme)":

Responsible: The person responsible for the performance of the task. There should be exactly one person with this assignment for each task.

Assists: Those who assist completion of the task.

Consulted: Those whose opinions are sought; and with whom there is two-way communication.

Informed: Those who are kept up-to-date on progress; and with whom there is one-way communication.

Task Table

This an early cut. Please feel free to add new tasks as appropriate — just let one of the coordinators know.

Task	Responsible	Current Status
Determine what the "Engine" will be	Adam Miller	100%
Deploy Engine solution, including ansible playbooks added for Fedora Infrastructure Ansible repo	Adam Miller	0%
Document the Engine solution final design	Adam Miller	0%
Document Automation Workflows end-to-end	Adam Miller	0%

Glossary of Nicknames

maxamillion Adam Miller

Various Task Notes

Functional Requirements

The following features are functional requirements

Role Based Access Control
- Users in certain groups are allowed to execute only certain workflows)
- This will enable the self-service component
Public central logging
- Workflow tasks should be logged centrally and historic runs of workflows can be publicly viewed in a central location

Other developers

(anything here)?