Koji Operator Proposal
STATE: DRAFT
Abstract
A proposal for decoupling koji from the MBBOX operator and manage it in its own independent operator.
Authors
Motivation
Koji is currently being handled/deployed in the MBBOX operator together with other components such as MBS.
The operator needs to manage several components such as koji-hub, kojira and also koji internal data (koji tags, usernames, certificates, users, etc) which proved to be complex enough to require its own operator.
A dedicated operator for Koji means it can be used in a standalone fashion and other operators such as MBBOX can just depend on this operator instead of managing koji on its own (such dependency can be managed by operator-hub).
Koji builders will use mock in "simple mode" and will run one task at a time only, auto scaling the amount of builders can be implemented in the future.
Proposal
The proposal is about moving koji to its own operator, repository, container images, etc.
The MBBOX operator should just create Koji custom resources and check for its status readiness.
Operator
The Koji operator will be responsible for deploying, configuring and managing koji services.
There are several operator types (operator foundations if you will) at the time of this writing: go, helm and ansible - the proposal is to do it using the go operator since the operator will also schedule task pods and this action alone may require intense testing which can be limited if using the ansible one (There is also a matter of performance where GO is unbeatable if compared to other operator types).
While go operators are usually faster and more fexlible, migrating to the go operator would imply writing all koji RPC integrations from scratch.
The operator won't abstract koji tags creation from a kubernetes CR since those can exponentially increase in number but the operator will enable an "admin sidecar container" instead for sysadmins to run koji admin commands (such as tag management).
The operator also won't deploy Koji Builders (aka kojid) but will implement its own task scheduler instead - tasks will be executed in kubernetes pods.
Dependencies
Koji depends on some components/services which will not be handled by the operator but are considered operator dependencies:
- PostgreSQL
- RabbitMQ (optional, only needed if fedora-messaging is enabled)
Secrets for both components should be provided in the same namespace Koji is being deployed:
NOTE: those values were chosen for example purposes.
#psql secret
apiVersion: v1
kind: Secret
metadata:
name: postgres
labels:
app: postgres
data:
POSTGRES_HOST: postgresql
POSTGRES_DB: mboxdb
POSTGRES_USER: koji
POSTGRES_PASSWORD: mbox
#rabbit mq secret
apiVersion: v1
kind: Secret
metadata:
name: koji-fedora-messaging-cert
data:
ca_cert.pem: my_ca_cert_content
client_cert.pem: my_client_cert_content
client_key.pem: my_client_key_content
Resources
This section describes all Kubernetes resources that will be managed by the operator.
Most of those components are custom resources definitions which are implemented by the operator itself but there are a few common kubernetes resources (such as secrets and configmaps) that will also be watched by the operator.
The operator should re-run its reconcyle loop every N seconds to ensure both deployed components and koji data are in the epected state - this is automatically done by the operator reconcyle loop.
Koji Hub
Resource responsible for defining the koji-hub server config.
The operator should create an "admin" sidecar container If sidecar is set to true so sysadmins can run koji admin commands from that container.
The sidecar property should be optional and default to false.
Resource Proposal:
apiVersion: apps.fedoraproject.org/v1alpha1 kind: KojiHub metadata: name: hub spec: image: quay.io/fedora/koji-hub:latest replicas: 1 persistent: true host: koji.mbox.dev # change it to match the external web url/route of koji-hub configmap: koji-hub ca_cert_secret: koji-hub-ca-cert service_cert_secret: koji-hub-service-cert postgres_secret: postgres http_enabled: true https_enabled: true topic_prefix: mbox_dev fedora_messaging_url: amqps://koji:something@rabbitmq messaging_cert_cm: koji-hub-msg ingress_backend: nginx # default # ingress_backend: openshift httpd_pvc_name: koji-hub-httpd-pvc httpd_pvc_size: 1Gi mnt_pvc_name: koji-hub-mnt-pvc mnt_pvc_size: 10Gi sidecar: true
Koji Web
Koji web is the frontend web appliction of koji to list builds, tasks, etc.
Resource Proposal:
apiVersion: apps.fedoraproject.org/v1alpha1
kind: KojiWeb
metadata:
name: koji-web
spec:
image: quay.io/fedora/koji-hub:latest
replicas: 1
client_cert_secret: koji-hub-web-client-cert
client_username: kojiweb
hub_host: koji_hub:8443
cacert_secret: koji-hub-ca-cert
client_cert_secret: koji-web-client-cert
Kojira
Kojira is a koji daemon responsible for generating build roots, repos and etc. It is a core backend component for koji.
Only one instance of kojira should be allowed per "koji deployment"/namespace.
Resource proposal:
apiVersion: apps.fedoraproject.org/v1alpha1
kind: Kojira
metadata:
name: kojira
spec:
image: quay.io/fedora/kojira:latest
configmap: kojira-config
hub_host: koji-hub:8443
src: 'no'
authentication:
ssl:
cacert_secret: koji-hub-ca-cert
client_cert_secret: kojira-client-cert
Koji Builder
Builders will run one task at a time using mock in "simple mode" due to mock limitations in kubernetes.
A koji builder user and host will be created and the resource name (metadata.name) will be used ad username.
It won't have access to the koji volume at all.
Resource proposal:
apiVersion: apps.fedoraproject.org/v1alpha1
kind: KojiBuilder
metadata:
name: default
spec:
image: quay.io/fedora/koji-builder:latest
hub_host: koji-hub:8443
authentication:
ssl:
cacert_secret: koji-hub-ca-cert
client_cert_secret: koji-builder-default-client-cert
Koji User
Creating a KojiUser custom resource would make the operator create an user in koji and keeping it synchronized with the CR config (such as user perms) and even recreate the user if it gets deleted.
Resource Proposal:
apiVersion: apps.fedoraproject.org/v1alpha1
kind: KojiUser
metadata:
name: kojira
labels:
app: koji
spec:
permissions:
- repo
authentication:
ssl: #may support others, such as krb and even multiple authentication mechanisms
client_secret_name: kojira-cert
ca_secret_name: ca-cert
The username field will use the value of metadata.name.
KubeCtl Plugin
A koji kubectl plugin which runs koji commands in the admin sidecar pod as a koji admin.
This plugin enables syadmins to manage koji using kubectl.
The plugin should check if the sidecar container is present otherwise fail describing the steps to set a sidecar container for koji-hub.
Kubectl plugins development are very straightforward and well documented: https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/
Example:
kubectl koji grant-permission repo kojira
Copyright
This document has been placed in the public domain.