From Fedora Project Wiki
m (fixed a minor misspelling)
(Replaced content with "{{autolang|base=yes}} '''Setting Up a Koji Build System''' This document has moved to the [https://docs.pagure.org/koji/ Koji Documentation]. The new location is: ht...")
 
(156 intermediate revisions by 37 users not shown)
Line 1: Line 1:
= Setting Up a Koji Build System =
+
{{autolang|base=yes}}
The Koji components may ''live'' on separate resources as long as all resources are able to communicate. This document will cover how to setup each service individually, however, all services may ''live'' on the same resource.
+
'''Setting Up a Koji Build System'''
  
[[TableOfContents()]
+
This document has moved to the [https://docs.pagure.org/koji/ Koji Documentation]. The new location is:
  
== PreReqs of Knowledge ==
+
    https://docs.pagure.org/koji/server_howto/
  
* Basic understanding of SSL and authentication via certificates
+
You can submit changes to Koji's docs in [https://pagure.io/koji Pagure]
* Basic knowledge about creating a database in PostgreSQL and importing a schema
 
* Working with psql
 
* Basic knowledge about Apache configuration
 
* Basic knowledge about yum/createrepo/mock - else you'll not be able to debug problems!
 
* Basic knowledge about using command line
 
* Basic knowledge about RPM building
 
 
 
== PreReqs of Packages ==
 
 
 
'''On the server (koji-hub/koji-web)'''
 
* httpd
 
* mod_ssl
 
* postgresql-server
 
 
 
'''On the builder (koji-builder)'''
 
* mock
 
* setarch (for some archs you'll require a patched version)
 
* rpm-build
 
* createrepo
 
 
 
== PostgreSQL Server ==
 
The PostgreSQL server must be installed and the database primed.
 
 
 
'''Configuration Files:'''
 
 
 
* /var/lib/pgsql/data/pg_hba.conf
 
* /var/lib/pgsql/data/postgresql.conf
 
 
 
'''Install PostgreSQL:'''
 
<pre>
 
root@localhost$ yum install postgresql-server
 
</pre>
 
 
 
'''Setup User Accounts:'''
 
<pre>
 
root@localhost$ useradd koji
 
root@localhost$ passwd -d koji
 
</pre>
 
 
 
'''Setup PostgreSQL and populate schema:'''
 
 
 
<pre>
 
root@localhost$ su - postgres
 
postgres@localhost$ createuser koji
 
Shall the new role be a superuser? (y/n) n
 
Shall the new role be allowed to create databases? (y/n) n
 
Shall the new role be allowed to create more new roles? (y/n) n
 
postgres@localhost$ createdb -O koji koji
 
postgres@localhost$ logout
 
root@localhost$ su - koji
 
koji@localhost$ psql koji koji < /usr/share/doc/koji*/docs/schema.sql
 
koji@localhost$ exit
 
</pre>
 
 
 
'''Authorize Koji-web and Koji-hub resources:'''
 
''In this example, Koji-web and Koji-hub are running on localhost.''
 
 
 
/var/lib/pgsql/data/pg_hba.conf:
 
''These settings need to be valid and inline with other services configurations.''
 
''Please note, the first matching auth line is used so this line must be above any other potential matches.''
 
''Add:''
 
<pre>
 
host    koji        koji        127.0.0.1/32          trust
 
</pre>
 
 
 
'''Make auth changes live:'''
 
<pre>
 
root@localhost$ su - postgres
 
postgres@localhost$ pg_ctl reload
 
postgres@localhost$ exit
 
</pre>
 
 
 
 
 
== Koji Authentication Setup ==
 
Koji primarily supports Kerberos and SSL Certificate authentication. For basic koji-cli access, plain user/pass combinations are possible. For Kerberos authentication, a Kerberos server will need to be setup and configured. For SSL authentication, SSL certificates and the needed cert. chains will need to be setup.
 
 
 
'''Adding basic user/pass combinations for authentication:'''
 
Any password setting/changing still needs to be done manually in the database though (it is not exposed through the API).
 
 
 
''Add needed information to the koji database:''
 
 
 
<pre>
 
koji@localhost$ psql
 
koji=> insert into users (name, password, status, usertype) values ('my-user-name', 'my-password-in-plain-test', 0, 0);
 
koji=> insert into user_perms (user_id, perm_id) values (<id of user inserted above>, 1);
 
</pre>
 
 
 
Note: you can get the ID of the new user by running the query:
 
<pre>
 
koji=> select * from users;
 
</pre>
 
 
 
You can then perform Koji admin commands by calling
 
<pre>
 
kojiadmin@localhost$ koji --user <username> --password <password>"
 
</pre>
 
 
 
Both kojira and kojid accept "user" and "password" options in their config files, and will use those for authentication if present.  Note that for kojira, you will need to create a user:
 
 
 
<pre>
 
kojiadmin@localhost$ koji add-user <username>
 
</pre>
 
 
 
Set the password for it as above, as well as granting it the "repo" permission (perm_id == 3).  For kojid you will need to create a "host" entry (koji add-host), and set the password for the associated user (the user gets created at the same time as the host entry).
 
 
 
'''Setting up SSL Certificates for authentication:'''
 
 
 
'''Certificate generation'''
 
 
 
* Get the ssl.cnf (listed on this page) and *edit* it! (0.organizationName_default should be modified)
 
 
 
'''ssl.cnf'''
 
<pre>
 
HOME                    = .
 
RANDFILE                = .rand
 
 
 
[ca]
 
default_ca              = ca_default
 
 
 
[ca_default]
 
dir                    = .
 
certs                  = $dir/certs
 
crl_dir                = $dir/crl
 
database                = $dir/index.txt
 
new_certs_dir          = $dir/newcerts
 
certificate            = $dir/%s_ca_cert.pem
 
private_key            = $dir/private/%s_ca_key.pem
 
serial                  = $dir/serial
 
crl                    = $dir/crl.pem
 
x509_extensions        = usr_cert
 
name_opt                = ca_default
 
cert_opt                = ca_default
 
default_days            = 3650
 
default_crl_days        = 30
 
default_md              = md5
 
preserve                = no
 
policy                  = policy_match
 
 
 
[policy_match]
 
countryName            = match
 
stateOrProvinceName    = match
 
organizationName        = match
 
organizationalUnitName  = optional
 
commonName              = supplied
 
emailAddress            = optional
 
 
 
[req]
 
default_bits            = 1024
 
default_keyfile        = privkey.pem
 
distinguished_name      = req_distinguished_name
 
attributes              = req_attributes
 
x509_extensions        = v3_ca # The extentions to add to the self signed cert
 
string_mask            = MASK:0x2002
 
 
 
[req_distinguished_name]
 
countryName                    = Country Name (2 letter code)
 
countryName_default            = AT
 
countryName_min                = 2
 
countryName_max                = 2
 
stateOrProvinceName            = State or Province Name (full name)
 
stateOrProvinceName_default    = Vienna
 
localityName                    = Locality Name (eg, city)
 
localityName_default            = Vienna
 
0.organizationName              = Organization Name (eg, company)
 
0.organizationName_default      = My company
 
organizationalUnitName          = Organizational Unit Name (eg, section)
 
commonName                      = Common Name (eg, your name or your server\'s hostname)
 
commonName_max                  = 64
 
emailAddress                    = Email Address
 
emailAddress_max                = 64
 
 
 
[req_attributes]
 
challengePassword              = A challenge password
 
challengePassword_min          = 4
 
challengePassword_max          = 20
 
unstructuredName                = An optional company name
 
 
 
[usr_cert]
 
basicConstraints                = CA:FALSE
 
nsComment                      = "OpenSSL Generated Certificate"
 
subjectKeyIdentifier            = hash
 
authorityKeyIdentifier          = keyid,issuer:always
 
 
 
[v3_ca]
 
subjectKeyIdentifier            = hash
 
authorityKeyIdentifier          = keyid:always,issuer:always
 
basicConstraints                = CA:true
 
</pre>
 
 
 
'''Generate CA'''
 
<pre>
 
caname="koji"
 
mkdir -p /etc/kojiweb/clients
 
cd /etc/kojiweb/clients
 
mkdir {certs,private}
 
touch index.txt
 
echo 01 > serial
 
openssl genrsa -out private/${caname}_ca_cert.key 2048
 
 
 
openssl req -config ssl.cnf -new -x509 -days 3650 -key private/${caname}_ca_cert.key \
 
-out ${caname}_ca_cert.crt -extensions v3_ca
 
</pre>
 
 
 
'''Generate the kojira user certificate'''
 
<pre>
 
caname=koji
 
user="kojira"
 
 
 
openssl genrsa -out certs/${user}.key 2048
 
 
 
openssl req -config ssl.cnf -new -nodes -out certs/${user}.csr -key certs/${user}.key
 
 
 
openssl ca -config ssl.cnf -keyfile private/${caname}_ca_cert.key -cert ${caname}_ca_cert.crt \
 
-out certs/${user}.crt -outdir certs -infiles certs/${user}.csr
 
cat certs/${user}.crt certs/${user}.key > /etc/kojira/${user}.pem
 
</pre>
 
 
 
'''Generate the kojihub server certificate'''
 
 
 
<pre>
 
Do the same as for the kojira user, but with e.g. user="kojihub".
 
Set the common name to the fully-qualified domain name of the Kojihub server.
 
</pre>
 
 
 
'''Generate the kojiweb server certificate'''
 
 
 
<pre>
 
Do the same as for the kojira user, but with e.g. user="kojiweb".
 
Set the common name to the fully-qualified domain name of the Kojiweb server.
 
</pre>
 
 
 
'''Generate the kojid certificate'''
 
 
 
<pre>
 
Do the same as for kojira user, but "user" (i.e. the CN of the certificate) should be the fully-qualified
 
domain name of the machine where the builder will be running, for example: kojibuilder1.example.com
 
</pre>
 
 
 
'''Generate other client certificates'''
 
 
 
<pre>
 
Do the same as for the kojira user, but with e.g. user=<name of user>.  These certificates will be used to
 
authenticate to the Koji Hub.  The user will be identified in the Koji UI by the CN of their certificate.
 
</pre>
 
 
 
''' Generate a PKCS12 user certificate (for web browser)'''
 
This is only required for user certificates,
 
not for kojira, kojid, or the other service certificates created above.
 
<pre>
 
openssl pkcs12 -export -inkey certs/${user}.key -in certs/${user}.crt -CAfile ${caname}_ca_cert.crt \
 
-out certs/${user}_browser_cert.p12
 
</pre>
 
 
 
 
 
'''Setting up Kerberos for authentication:'''
 
 
 
----
 
==== Oh yeah, add some kerberos magic here! ====
 
----
 
 
 
 
 
Right now kojiweb only supports Kerberos or SSL authentication to the hub.  However, Koji Web is fully functional in read-only mode (without logging in), and all operations exposed via the web UI are also available via the cli.  Just comment out everything in /etc/httpd/conf.d/kojiweb.conf related to authentication, and don't click the login link. ;-)
 
 
 
 
 
== Koji Hub ==
 
 
 
Koji-hub is the center of all Koji operations. It is an XML-RPC server running under mod_python in Apache. koji-hub is passive in that it only receives XML-RPC calls and relies upon the build daemons and other components to initiate communication. Koji-hub is the only component that has direct access to the database and is one of the two components that have write access to the file system.
 
 
 
'''Configuration Files:'''
 
 
 
* /etc/httpd/conf.d/kojihub.conf
 
* /etc/httpd/conf.d/ssl.conf (when using ssl auth)
 
 
 
'''Install koji-hub:'''
 
<pre>
 
root@localhost$ yum install koji-hub httpd mod_ssl mod_python
 
</pre>
 
 
 
==== Required Configuration ====
 
 
 
/etc/httpd/conf.d/kojihub.conf:
 
''These settings need to be valid and inline with other services configurations.''
 
<pre>
 
PythonOption DBName koji
 
PythonOption DBUser koji
 
PythonOption DBHost db.example.com
 
PythonOption KojiDir /mnt/koji
 
PythonOption LoginCreatesUser On
 
PythonOption KojiWebURL http://kojiweb.example.com/koji
 
</pre>
 
 
 
==== Optional Configuration ====
 
 
 
/etc/httpd/conf.d/kojihub.conf:
 
''If using Kerberos, these settings need to be valid and inline with other services configurations.''
 
<pre>
 
PythonOption AuthPrincipal kojihub@EXAMPLE.COM
 
PythonOption AuthKeytab /etc/koji.keytab
 
PythonOption ProxyPrincipals kojihub@EXAMPLE.COM
 
PythonOption HostPrincipalFormat compile/%s@EXAMPLE.COM
 
</pre>
 
 
 
/etc/httpd/conf.d/kojihub.conf:
 
''If using SSL auth, these settings need to be valid and inline with other services configurations.
 
ProxyDNs should be set to the DN of the kojiweb certificate.''
 
<pre>
 
PythonOption DNUsernameComponent CN
 
PythonOption ProxyDNs "/C=US/ST=Massachusetts/O=Example Org/OU=Example User/CN=example/emailAddress=example@example.com"
 
</pre>
 
 
 
<pre>
 
<Location /kojihub>
 
SSLOptions +StdEnvVars
 
</Location>
 
</pre>
 
 
 
/etc/httpd/conf.d/ssl.conf:
 
''Add the needed SSL options for apache.''
 
<pre>
 
SSLCertificateFile /path/to/kojihub.crt
 
SSLCertificateKeyFile /path/to/kojihub.key
 
SSLCertificateChainFile /path/to/koji_ca_cert.crt
 
SSLCACertificateFile /path/to/koji_ca_cert.crt
 
SSLVerifyClient require
 
SSLVerifyDepth  10
 
</pre>
 
 
 
==== SELinux Configuration ====
 
If running in Enforcing mode, you will need to allow apache to connect to the postgreSQL server.
 
 
 
<pre>
 
root@localhost$ setsebool -P httpd_can_network_connect_db 1
 
</pre>
 
 
 
 
 
== Koji Web - Interface for the Masses ==
 
 
 
Koji-web is a set of scripts that run in mod_python and use the Cheetah templating engine to provide an web interface to Koji. koji-web exposes a lot of information and also provides a means for certain operations, such as cancelling builds.
 
 
 
'''Configuration Files:'''
 
 
 
* /etc/httpd/conf.d/kojiweb.conf
 
* /etc/httpd/conf.d/ssl.conf
 
 
 
'''Install Koji-Web:'''
 
<pre>
 
root@localhost$ yum install koji-web httpd mod_python mod_ssl
 
</pre>
 
 
 
==== Required Configuration ====
 
/etc/httpd/conf.d/kojiweb.conf:
 
<pre>
 
PythonOption KojiHubURL http://hub.example.com/kojihub
 
PythonOption KojiWebURL http://www.example.com/koji
 
PythonOption KojiPackagesURL http://server.example.com/mnt/koji/packages
 
 
 
 
 
PythonOption WebCert /etc/kojiweb/kojiweb.pem
 
PythonOption ClientCA /etc/kojiweb/clientca.crt
 
PythonOption KojiHubCA /etc/kojiweb/kojihubca.crt
 
 
 
PythonOption LoginTimeout 72
 
PythonOption Secret CHANGE_ME
 
</pre>
 
 
 
==== Optional Configuration ====
 
 
 
/etc/httpd/conf.d/kojiweb.conf:
 
''If using Kerberos, these settings need to be valid and inline with other services configurations.''
 
<pre>
 
<Location /koji/login>
 
AuthType Kerberos
 
AuthName "Koji Web UI"
 
KrbMethodNegotiate on
 
KrbMethodK5Passwd off
 
KrbServiceName HTTP
 
KrbAuthRealm EXAMPLE.COM
 
Krb5Keytab /etc/httpd.keytab
 
KrbSaveCredentials off
 
Require valid-user
 
ErrorDocument 401 /koji-static/errors/unauthorized.html
 
</Location>
 
</pre>
 
 
 
/etc/httpd/conf.d/kojiweb.conf:
 
''If using SSL auth, these settings need to be valid and inline with other services configurations.''
 
<pre>
 
<Location /koji/login>
 
SSLOptions +StdEnvVars
 
</Location>
 
</pre>
 
 
 
/etc/httpd/conf.d/ssl.conf:
 
''Add the needed SSL options for apache.''
 
<pre>
 
SSLVerifyClient require
 
SSLVerifyDepth  10
 
</pre>
 
 
 
 
 
== Koji Daemon - Builder ==
 
 
 
Kojid is the build daemon that runs on each of the build machines. Its primary responsibility is polling for incoming build requests and handling them accordingly. Koji also has support for tasks other than building. Creating install images is one example. kojid is responsible for handling these tasks as well. kojid uses mock for building. It also creates a fresh buildroot for every build. kojid is written in Python and communicates with koji-hub via XML-RPC.
 
 
 
'''Configuration Files:'''
 
 
 
* /etc/kojid/kojid.conf - Koji Daemon Configuration
 
* /etc/sysconfig/kojid - Koji Daemon Switches
 
 
 
'''Install kojid:'''
 
<pre>
 
root@localhost$ yum install koji-builder
 
</pre>
 
 
 
==== Required Configuration ====
 
 
 
/etc/kojid/kojid.conf:
 
''This needs to point at your koji-hub.''
 
<pre>
 
; The URL for the xmlrpc server
 
server=http://hub.example.com/kojihub
 
 
 
; the username has to be the same as what you used with add-host
 
; in this example follow as below
 
user = kojibuilder1.example.com
 
</pre>
 
 
 
==== Optional Configuration ====
 
 
 
/etc/kojid/kojid.conf:
 
''If using SSL, these settings need to be valid.''
 
<pre>
 
;client certificate
 
; This should reference the builder certificate we created above, for
 
; kojibuilder1.example.com
 
cert = /etc/kojid/kojid.pem
 
 
 
;certificate of the CA that issued the client certificate
 
ca = /etc/kojid/clientca.crt
 
 
 
;certificate of the CA that issued the HTTP server certificate
 
serverca = /etc/kojid/serverca.crt
 
</pre>
 
 
 
 
 
==== Add the host entry for the koji builder to the database ====
 
Make sure you do this before you start kojid for the first time,
 
or you'll need to manually remove entries from the sessions and users
 
table before it can be run successfully.
 
<pre>
 
kojiadmin@localhost$ koji add-host kojibuilder1.example.com i386 x86_64
 
</pre>
 
 
 
==== Add the host to the createrepo channel ====
 
Channels are a way to control which builders process which tasks.  By default
 
hosts are added to the ''default'' channel.  This host also needs to be added
 
to the ''createrepo'' channel so it will process repo creation tasks.
 
 
 
<pre>
 
kojiadmin@localhost$ koji add-host-to-channel kojibuilder1.example.com createrepo
 
</pre>
 
 
 
==== Start Kojid ====
 
<pre>
 
root@localhost$ /sbin/service kojid start
 
</pre>
 
Check /var/log/kojid.log to verify that kojid has started successfully.
 
 
 
== Kojira - Yum repository creation and maintenance ==
 
'''Configuration Files:'''
 
 
 
* /etc/kojira/kojira.conf - Kojira Daemon Configuration
 
* /etc/sysconfig/kojira - Kojira Daemon Switches
 
 
 
'''Install kojira'''
 
<pre>
 
root@localhost$ yum install koji-utils
 
</pre>
 
 
 
==== Required Configuration ====
 
 
 
/etc/kojira/kojira.conf:
 
''This needs to point at your koji-hub.''
 
<pre>
 
; The URL for the xmlrpc server
 
server=http://hub.example.com/kojihub
 
</pre>
 
 
 
==== Optional Configuration ====
 
 
 
/etc/kojira/kojira.conf:
 
''If using SSL, these settings need to be valid.''
 
<pre>
 
;client certificate
 
; This should reference the kojira certificate we created above
 
cert = /etc/kojira/kojira.pem
 
 
 
;certificate of the CA that issued the client certificate
 
ca = /etc/kojid/clientca.crt
 
 
 
;certificate of the CA that issued the HTTP server certificate
 
serverca = /etc/kojid/serverca.crt
 
</pre>
 
 
 
/etc/sysconfig/kojira:
 
''The local user kojira runs as needs to be able to read and write to /mnt/koji/repos/.
 
If the volume that directory resides on is root-squashed or otherwise unmodifiable
 
by root, you can set '''RUNAS=''' to a user that has the required privileges.''
 
 
 
==== Add the user entry for the kojira user ====
 
Create the kojira user, and grant it the '''repo''' permission.
 
<pre>
 
kojiadmin@localhost$ koji add-user kojira
 
kojiadmin@localhost$ koji grant-permission repo kojira
 
</pre>
 
 
 
=== Start Kojira ===
 
<pre>
 
root@localhost$ /sbin/service kojira start
 
</pre>
 
Check /var/log/kojira/kojira.log to verify that kojira has started successfully.
 
 
 
== Bootstrapping the Koji build environment ==
 
For instructions on importing packages and preparing Koji to run builds, see [[Koji/ServerBootstrap|  ServerBootstrap]] .
 

Latest revision as of 20:57, 23 August 2016

Setting Up a Koji Build System

This document has moved to the Koji Documentation. The new location is:

   https://docs.pagure.org/koji/server_howto/

You can submit changes to Koji's docs in Pagure