From Fedora Project Wiki
(Replaced content with "{{autolang|base=yes}} '''Setting Up a Koji Build System''' This document has moved to the [https://docs.pagure.org/koji/ Koji Documentation]. The new location is: ht...")
 
(158 intermediate revisions by 38 users not shown)
Line 1: Line 1:
= Setting Up a Koji Build System =
{{autolang|base=yes}}
The Koji components may ''live'' on seperate resources as long as all resources are able to communicate. This document will cover how to setup each service individually, however, all services may ''live'' on the same resource.
'''Setting Up a Koji Build System'''


[[TableOfContents()]
This document has moved to the [https://docs.pagure.org/koji/ Koji Documentation]. The new location is:


== PreReqs of Knowledge ==
    https://docs.pagure.org/koji/server_howto/


* Basic understanding of SSL and authentication via certificates
You can submit changes to Koji's docs in [https://pagure.io/koji Pagure]
* Basic knowledge about creating a database in PostgreSQL and importing a schema
* Working with psql
* Basic knowledge about Apache configuration
* Basic knowledge about yum/createrepo/mock - else you'll not be able to debug problems!
* Basic knowledge about using command line
* Basic knowledge about RPM building
 
== PreReqs of Packages ==
 
'''On the server (koji-hub/koji-web)'''
* httpd
* mod_ssl
* postgresql-server
 
'''On the builder (koji-builder)'''
* mock
* setarch (for some archs you'll require a patched version)
* rpm-build
* createrepo
 
== PostgreSQL Server ==
The PostgreSQL server must be installed and the database primed.
 
'''Configuration Files:'''
 
* /var/lib/pgsql/data/pg_hba.conf
* /var/lib/pgsql/data/postgresql.conf
 
'''Install PostgreSQL:'''
<pre>
</pre>
 
'''Setup PostgreSQL and populate schema:'''
 
<pre>
$ createuser koji
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
$ createdb -O koji koji
$ psql koji koji < /usr/share/doc/koji*/docs/schema.sql
$ exit
</pre>
 
'''Authorize Koji-web and Koji-hub resources:'''
''In this example, Koji-web and Koji-hub are running on localhost.''
 
/var/lib/pgsql/data/pg_hba.conf:
''These settings need to be valid and inline with other services configurations.''
''Please note, the first matching auth line is used so this line must be above any other potential matches.''
''Add:''
<pre>
host    koji        koji        127.0.0.1/32          trust
</pre>
 
'''Make auth changes live:'''
<pre>
$ pg_ctl reload
$ exit
</pre>
 
 
== Koji Authentication Setup ==
Koji primarily supports Kerberos and SSL Certificate authentication. For basic koji-cli access, plain user/pass combinations are possible. For Kerberos authentication, a Kerberos server will need to be setup and configured. For SSL authentication, SSL certificates and the needed cert. chains will need to be setup.
 
'''Adding basic user/pass combinations for authentication:'''
Any password setting/changing still needs to be done manually in the database though (it is not exposed through the API).
 
''Add needed information to the koji database:''
 
<pre>
$ psql
koji=> insert into users (name, password, status, usertype) values ('my-user-name', 'my-password-in-plain-test', 0, 0);
koji=> insert into user_perms (user_id, perm_id) values (<id of user inserted above>, 1);
</pre>
 
Note: you can get the ID of the new user by running the query:
<pre>
koji=> select * from users;
</pre>
 
You can then perform Koji admin commands by calling
<pre>
$ koji --user <username> --password <password>"
</pre>
 
Both kojira and kojid accept "user" and "password" options in their config files, and will use those for authentication if present.  Note that for kojira, you will need to create a user:
 
<pre>
$ koji add-user <username>
</pre>
 
Set the password for it as above, as well as granting it the "repo" permission (perm_id == 3).  For kojid you will need to create a "host" entry (koji add-host), and set the password for the associated user (the user gets created at the same time as the host entry).
 
'''Setting up SSL Certificates for authentication:'''
 
'''Certificate generation'''
 
* Get the ssl.cnf (listed on this page) and *edit* it! (0.organizationName_default should be modified)
 
'''ssl.cnf'''
<pre>
HOME                    = .
RANDFILE                = .rand
 
[ca]
default_ca              = ca_default
 
[ca_default]
dir                    = .
certs                  = $dir/certs
crl_dir                = $dir/crl
database                = $dir/index.txt
new_certs_dir          = $dir/newcerts
certificate            = $dir/%s_ca_cert.pem
private_key            = $dir/private/%s_ca_key.pem
serial                  = $dir/serial
crl                    = $dir/crl.pem
x509_extensions        = usr_cert
name_opt                = ca_default
cert_opt                = ca_default
default_days            = 3650
default_crl_days        = 30
default_md              = md5
preserve                = no
policy                  = policy_match
 
[policy_match]
countryName            = match
stateOrProvinceName    = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional
 
[req]
default_bits            = 1024
default_keyfile        = privkey.pem
distinguished_name      = req_distinguished_name
attributes              = req_attributes
x509_extensions        = v3_ca # The extentions to add to the self signed cert
string_mask            = MASK:0x2002
 
[req_distinguished_name]
countryName                    = Country Name (2 letter code)
countryName_default            = AT
countryName_min                = 2
countryName_max                = 2
stateOrProvinceName            = State or Province Name (full name)
stateOrProvinceName_default    = Vienna
localityName                    = Locality Name (eg, city)
localityName_default            = Vienna
0.organizationName              = Organization Name (eg, company)
0.organizationName_default      = My company
organizationalUnitName          = Organizational Unit Name (eg, section)
commonName                      = Common Name (eg, your name or your server\'s hostname)
commonName_max                  = 64
emailAddress                    = Email Address
emailAddress_max                = 64
 
[req_attributes]
challengePassword              = A challenge password
challengePassword_min          = 4
challengePassword_max          = 20
unstructuredName                = An optional company name
 
[usr_cert]
basicConstraints                = CA:FALSE
nsComment                      = "OpenSSL Generated Certificate"
subjectKeyIdentifier            = hash
authorityKeyIdentifier          = keyid,issuer:always
 
[v3_ca]
subjectKeyIdentifier            = hash
authorityKeyIdentifier          = keyid:always,issuer:always
basicConstraints                = CA:true
</pre>
 
'''Generate CA'''
<pre>
caname="koji"
mkdir -p /etc/kojiweb/clients
cd /etc/kojiweb/clients
mkdir {certs,private}
touch index.txt
echo 01 > serial
openssl genrsa -out private/${caname}_ca_cert.key 2048
 
openssl req -config ssl.cnf -new -x509 -days 3650 -key private/${caname}_ca_cert.key \
-out ${caname}_ca_cert.crt -extensions v3_ca
</pre>
 
'''Generate the kojira user certificate'''
<pre>
caname=koji
user="kojira"
 
openssl genrsa -out certs/${user}.key 2048
 
openssl req -config ssl.cnf -new -nodes -out certs/${user}.csr -key certs/${user}.key
 
openssl ca -config ssl.cnf -keyfile private/${caname}_ca_cert.key -cert ${caname}_ca_cert.crt \
-out certs/${user}.crt -outdir certs -infiles certs/${user}.csr
cat certs/${user}.crt certs/${user}.key > /etc/kojira/${user}.pem
</pre>
 
'''Generate the kojihub server certificate'''
 
<pre>
Do the same as for the kojira user, but with e.g. user="kojihub".
Set the common name to the fully-qualified domain name of the Kojihub server.
</pre>
 
'''Generate the kojiweb server certificate'''
 
<pre>
Do the same as for the kojira user, but with e.g. user="kojiweb".
Set the common name to the fully-qualified domain name of the Kojiweb server.
</pre>
 
'''Generate the kojid certificate'''
 
<pre>
Do the same as for kojira user, but "user" (i.e. the CN of the certificate) should be the fully-qualified
domain name of the machine where the builder will be running, for example: kojibuilder1.example.com
</pre>
 
'''Generate other client certificates'''
 
<pre>
Do the same as for the kojira user, but with e.g. user=<name of user>.  These certificates will be used to
authenticate to the Koji Hub.  The user will be identified in the Koji UI by the CN of their certificate.
</pre>
 
''' Generate a PKCS12 user certificate (for web browser)'''
This is only required for user certificates,
not for kojira, kojid, or the other service certificates created above.
<pre>
openssl pkcs12 -export -inkey certs/${user}.key -in certs/${user}.crt -CAfile ${caname}_ca_cert.crt \
-out certs/${user}_browser_cert.p12
</pre>
 
 
'''Setting up Kerberos for authentication:'''
 
----
==== Oh yeah, add some kerberos magic here! ====
----
 
 
Right now kojiweb only supports Kerberos or SSL authentication to the hub.  However, Koji Web is fully functional in read-only mode (without logging in), and all operations exposed via the web UI are also available via the cli.  Just comment out everything in /etc/httpd/conf.d/kojiweb.conf related to authentication, and don't click the login link. ;-)
 
 
== Koji Hub ==
 
Koji-hub is the center of all Koji operations. It is an XML-RPC server running under mod_python in Apache. koji-hub is passive in that it only receives XML-RPC calls and relies upon the build daemons and other components to initiate communication. Koji-hub is the only component that has direct access to the database and is one of the two components that have write access to the file system.
 
'''Configuration Files:'''
 
* /etc/httpd/conf.d/kojihub.conf
* /etc/httpd/conf.d/ssl.conf (when using ssl auth)
 
'''Install koji-hub:'''
<pre>
yum install koji-hub httpd mod_ssl mod_python
</pre>
 
==== Required Configuration ====
 
/etc/httpd/conf.d/kojihub.conf:
''These settings need to be valid and inline with other services configurations.''
<pre>
PythonOption DBName koji
PythonOption DBUser koji
PythonOption DBHost db.example.com
PythonOption KojiDir /mnt/koji
PythonOption LoginCreatesUser On
PythonOption KojiWebURL http://kojiweb.example.com/koji
</pre>
 
==== Optional Configuration ====
 
/etc/httpd/conf.d/kojihub.conf:
''If using Kerberos, these settings need to be valid and inline with other services configurations.''
<pre>
PythonOption AuthPrincipal kojihub@EXAMPLE.COM
PythonOption AuthKeytab /etc/koji.keytab
PythonOption ProxyPrincipals kojihub@EXAMPLE.COM
PythonOption HostPrincipalFormat compile/%s@EXAMPLE.COM
</pre>
 
/etc/httpd/conf.d/kojihub.conf:
''If using SSL auth, these settings need to be valid and inline with other services configurations.
ProxyDNs should be set to the DN of the kojiweb certificate.''
<pre>
PythonOption DNUsernameComponent CN
PythonOption ProxyDNs "/C=US/ST=Massachusetts/O=Example Org/OU=Example User/CN=example/emailAddress=example@example.com"
</pre>
 
<pre>
<Location /kojihub>
SSLOptions +StdEnvVars
</Location>
</pre>
 
/etc/httpd/conf.d/ssl.conf:
''Add the needed SSL options for apache.''
<pre>
SSLCertificateFile /path/to/kojihub.crt
SSLCertificateKeyFile /path/to/kojihub.key
SSLCertificateChainFile /path/to/koji_ca_cert.crt
SSLCACertificateFile /path/to/koji_ca_cert.crt
SSLVerifyClient require
SSLVerifyDepth  10
</pre>
 
==== SELinux Configuration ====
If running in Enforcing mode, you will need to allow apache to connect to the postgreSQL server.
 
<pre>
setsebool -P httpd_can_network_connect_db 1
</pre>
 
 
== Koji Web - Interface for the Masses ==
 
Koji-web is a set of scripts that run in mod_python and use the Cheetah templating engine to provide an web interface to Koji. koji-web exposes a lot of information and also provides a means for certain operations, such as cancelling builds.
 
'''Configuration Files:'''
 
* /etc/httpd/conf.d/kojiweb.conf
* /etc/httpd/conf.d/ssl.conf
 
'''Install Koji-Web:'''
<pre>
yum install koji-web httpd mod_python mod_ssl
</pre>
 
==== Required Configuration ====
/etc/httpd/conf.d/kojiweb.conf:
<pre>
PythonOption KojiHubURL http://hub.example.com/kojihub
PythonOption KojiWebURL http://www.example.com/koji
PythonOption KojiPackagesURL http://server.example.com/mnt/koji/packages
 
 
PythonOption WebCert /etc/kojiweb/kojiweb.pem
PythonOption ClientCA /etc/kojiweb/clientca.crt
PythonOption KojiHubCA /etc/kojiweb/kojihubca.crt
 
PythonOption LoginTimeout 72
PythonOption Secret CHANGE_ME
</pre>
 
==== Optional Configuration ====
 
/etc/httpd/conf.d/kojiweb.conf:
''If using Kerberos, these settings need to be valid and inline with other services configurations.''
<pre>
<Location /koji/login>
AuthType Kerberos
AuthName "Koji Web UI"
KrbMethodNegotiate on
KrbMethodK5Passwd off
KrbServiceName HTTP
KrbAuthRealm EXAMPLE.COM
Krb5Keytab /etc/httpd.keytab
KrbSaveCredentials off
Require valid-user
ErrorDocument 401 /koji-static/errors/unauthorized.html
</Location>
</pre>
 
/etc/httpd/conf.d/kojiweb.conf:
''If using SSL auth, these settings need to be valid and inline with other services configurations.''
<pre>
<Location /koji/login>
SSLOptions +StdEnvVars
</Location>
</pre>
 
/etc/httpd/conf.d/ssl.conf:
''Add the needed SSL options for apache.''
<pre>
SSLVerifyClient require
SSLVerifyDepth  10
</pre>
 
 
== Koji Daemon - Builder ==
 
Kojid is the build daemon that runs on each of the build machines. Its primary responsibility is polling for incoming build requests and handling them accordingly. Koji also has support for tasks other than building. Creating install images is one example. kojid is responsible for handling these tasks as well. kojid uses mock for building. It also creates a fresh buildroot for every build. kojid is written in Python and communicates with koji-hub via XML-RPC.
 
'''Configuration Files:'''
 
* /etc/kojid/kojid.conf - Koji Daemon Configuration
* /etc/sysconfig/kojid - Koji Daemon Switches
 
'''Install kojid:'''
<pre>
yum install koji-builder
</pre>
 
==== Required Configuration ====
 
/etc/kojid/kojid.conf:
''This needs to point at your koji-hub.''
<pre>
; The URL for the xmlrpc server
server=http://hub.example.com/kojihub
 
; the username has to be the same as what you used with add-host
; in this example follow as below
user = kojibuilder1.example.com
</pre>
 
==== Optional Configuration ====
 
/etc/kojid/kojid.conf:
''If using SSL, these settings need to be valid.''
<pre>
;client certificate
; This should reference the builder certificate we created above, for
; kojibuilder1.example.com
cert = /etc/kojid/kojid.pem
 
;certificate of the CA that issued the client certificate
ca = /etc/kojid/clientca.crt
 
;certificate of the CA that issued the HTTP server certificate
serverca = /etc/kojid/serverca.crt
</pre>
 
 
==== Add the host entry for the koji builder to the database ====
Make sure you do this before you start kojid for the first time,
or you'll need to manually remove entries from the sessions and users
table before it can be run successfully.
<pre>
$ koji add-host kojibuilder1.example.com i386 x86_64
</pre>
 
==== Add the host to the createrepo channel ====
Channels are a way to control which builders process which tasks.  By default
hosts are added to the ''default'' channel.  This host also needs to be added
to the ''createrepo'' channel so it will process repo creation tasks.
 
<pre>
$ koji add-host-to-channel kojibuilder1.example.com createrepo
</pre>
 
==== Start Kojid ====
<pre>
/sbin/service kojid start
</pre>
Check /var/log/kojid.log to verify that kojid has started successfully.
 
== Kojira - Yum repository creation and maintenance ==
'''Configuration Files:'''
 
* /etc/kojira/kojira.conf - Kojira Daemon Configuration
* /etc/sysconfig/kojira - Kojira Daemon Switches
 
'''Install kojira'''
<pre>
yum install koji-utils
</pre>
 
==== Required Configuration ====
 
/etc/kojira/kojira.conf:
''This needs to point at your koji-hub.''
<pre>
; The URL for the xmlrpc server
server=http://hub.example.com/kojihub
</pre>
 
==== Optional Configuration ====
 
/etc/kojira/kojira.conf:
''If using SSL, these settings need to be valid.''
<pre>
;client certificate
; This should reference the kojira certificate we created above
cert = /etc/kojira/kojira.pem
 
;certificate of the CA that issued the client certificate
ca = /etc/kojid/clientca.crt
 
;certificate of the CA that issued the HTTP server certificate
serverca = /etc/kojid/serverca.crt
</pre>
 
/etc/sysconfig/kojira:
''The local user kojira runs as needs to be able to read and write to /mnt/koji/repos/.
If the volume that directory resides on is root-squashed or otherwise unmodifiable
by root, you can set '''RUNAS=''' to a user that has the required privileges.''
 
==== Add the user entry for the kojira user ====
Create the kojira user, and grant it the '''repo''' permission.
<pre>
$ koji add-user kojira
$ koji grant-permission repo kojira
</pre>
 
=== Start Kojira ===
<pre>
/sbin/service kojira start
</pre>
Check /var/log/kojira/kojira.log to verify that kojira has started successfully.
 
== Bootstrapping the Koji build environment ==
For instructions on importing packages and preparing Koji to run builds, see [[Koji/ServerBootstrap|  ServerBootstrap]] .

Latest revision as of 20:57, 23 August 2016

Setting Up a Koji Build System

This document has moved to the Koji Documentation. The new location is:

   https://docs.pagure.org/koji/server_howto/

You can submit changes to Koji's docs in Pagure