Koji/BuildingImages

From FedoraProject

< Koji(Difference between revisions)
Jump to: navigation, search
 
(17 intermediate revisions by one user not shown)
Line 1: Line 1:
This page documents a few additional steps to enable building LiveCDs in an existing Koji installation, how to build one, and where the code will grow from here. This is a very new feature and there hasn't been much external testing yet, so please keep that mind if you run into difficulty. For additional questions and information, ask around in #koji on FreeNode IRC and sign up to the right mailing lists as the [https://fedorahosted.org/koji/wiki Koji project website] dictates.  
+
{{autolang|base=yes}}
 +
This page documents how images can be built in Koji. It covers the supported use cases, how to configure your Koji instance to build them, and how to build them once it is ready. This is a very new feature and there hasn't been much external testing yet, so please keep that mind if you run into difficulty. Be certain to read the next section to be certain your use case is supported as well. For additional questions and information, ask around in #koji on FreeNode IRC and sign up to the right mailing lists as the [https://fedorahosted.org/koji/wiki Koji project website] dictates.
 +
 
 +
== Supported Images & Use Cases ==
 +
 
 +
The way images are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies from the livecd-build or appliance-build package group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. A livecd-creator or appliance-creator command is executed using the mock('--chroot', ...) method. '''Note:''' this process runs as root. This produces the desired image which is uploaded to /mnt/koji/images/<image>/$imageID if it is not a scratch image.
 +
 
 +
==== %include macros ====
 +
 
 +
A word of caution about kickstart files and the %include macro. livecd-creator and appliance-creator are smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the spin-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user's local disk.
 +
 
 +
==== Package Groups in the Kickstart File ====
 +
 
 +
Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don't since Koji's comps.xml is based on the "groups" set up from the CLI. {livecd,appliance}-creator's behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are a couple possible workarounds:
 +
 
 +
* do not use package groups in the kickstart file and just specify a huge list of packages
 +
* use --repo and specify a repo that does have a comps.xml that defines the groups it uses
 +
 
 +
==== Only Include RPMs Built in Koji ====
 +
 
 +
The image building tasks will fail if your image tries to include a package that was not built in your build system. This is because the package does not have any origin information stored in Koji's database. The repos defined in the kickstart will automatically be overridden with the repo of the build tag for the build target, unless you use the --repo option. Since only packages you have built (or include from an external repo) should be there, you should never have this problem unless you use --repo.
 +
 
 +
==== No Signed or Debuginfo RPMs in Koji's Build Tags ====
 +
 
 +
If you need signed RPMs or debuginfo RPMs, you will run into trouble because Koji does not keep those in its build tag repos. The only work around for this is to create a repo yourself that includes these RPMs and then use --repo. This will force the image to take RPMs from that repo.
 +
 
 +
==== LiveCD-Specific Caveats ====
 +
 
 +
While livecd-tools does support building on SELinux disabled hosts, you can run into denials when booting if you create and use new files in the %post section of your kickstart file. If you do, you should either set the labels appropriately at the end of the %post section, or instigate an autorelabel at boot time.
 +
 
 +
==== Appliance-Specific Caveats ====
 +
 
 +
Koji only supports Appliances that are single disk images. If your kickstart file contains "part" commands for multiple devices, the task will fail. Patches are welcome to remove this limitation.
  
 
== Build System Preparation ==
 
== Build System Preparation ==
This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so [[koji/ServerHowTo| here]] if you need to. You should also have some familiarity with how livecd-creator from the livecd-tools package works.
 
  
Follow this procedure step by step to get things prepared they way they need to be:
+
This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so [[koji/ServerHowTo| here]] if you need to. Please ensure you are using the latest version of the software and that your database schema is updated as well. You should also have some familiarity with how [[FedoraLiveCD|livecd-creator]] and [http://fedoraproject.org/wiki/Features/ApplianceTools appliance-creator] work.
  
* ''mkdir /mnt/koji/images'': create a directory to put the finished images
+
Follow this procedure step by step to get things prepared they way they need to be. Italics indicate commands to run on the console. Any time you see <image>, substitute in the desired image type. This can be either "livecd" or "appliance". You can enable both at once as well, just run the command twice using each value for <image>.
* ''chown apache:apache /mnt/koji/images'': let apache read and write to it
+
* You need a livecd 'channel'. If you're running version 1.3.2 or later, it will already be available. If not, you'll have to add it to the database manually:
+
<pre>
+
sql stuff here
+
</pre>
+
* ''koji add-host-to-channel <your-host> livecd'': add a builder to the livecd channel
+
* You need a livecd 'permission'. If you're running version 1.3.2 or later, it will already be available. If not, you'll have to add it to the database manually:
+
<pre>
+
sql stuff here
+
</pre>
+
* ''koji grant-permission livecd <user>'': grant the permission to build livecds to a user. This step is optional since admins have all permissions, technically.
+
* ''koji add-group <build-tag> livecd-build'': add the livecd-build group
+
* ''koji add-group-pkg <build-tag> livecd-build <pkg> ...'': add packages to the livecd-build group. The packages should match what you have in the "build" group, plus: python-dbus, yum, squashfs-tools, livecd-tools, and selinux-targeted-policy as of Fedora 10.
+
* You will need a tag and target to build the LiveCDs from. The repos generated for the build tag of the target is what Koji will use to populate the LiveCDs by default. (the alternative is to use the --repo option, more on that later)
+
  
== Building a LiveCD ==
+
* ''mkdir /mnt/koji/images'': Create a directory to put the finished images. This should be done on the system running kojihub.
 +
* ''chown apache:apache /mnt/koji/images'': Let apache read and write to it.
 +
* Version 1.3.2 introduces another PythonOption in kojiweb.conf: KojiImagesURL. This option is set to the URL where completed LiveCD images can be downloaded from. Make sure that your apache server is allowing access to that location.
 +
* You need to set the topurl option in /etc/kojid/kojid.conf. Usually it looks something like <nowiki>http://your.domain.com/koji</nowiki>
 +
* ''koji add-host-to-channel <your-host> <image>'': add a builder to the <image> channel
 +
* ''koji grant-permission <image> <user>'': grant the permission to build an image type to a user. This step is optional since admins have all permissions.
 +
* You will need a tag and target to build the images from. The yum repo generated for the build tag of the target is what Koji will use to populate the LiveCDs with by default. (the alternative is to use the --repo option, more on that later)
 +
* ''koji add-group <build-tag> <image>-build'': add the <image>-build group
 +
* ''koji add-group-pkg <build-tag> <image>-build <pkg> ...'': add packages to the <image>-build group. These package lists vary has packages and dependencies change. As of October, 2010 for Fedora 13 the needed packages for each image type are:
 +
** livecd: bash, coreutils, dbus-python, dmraid, livecd-tools, selinux-policy-targeted, yum
 +
** appliance: appliance-tools, bash, coreutils, grub, parted, selinux-policy, shadow-utils, policycoreutils
  
Building a Livecd in Koji is accomplished with the spin-livecd directive. Like all other Koji directives, usage information is available with --help:
+
== Building an Image ==
 +
 
 +
Building a LiveCD in Koji is accomplished with the spin-livecd directive. Like all other Koji directives, usage information is available with --help:
  
 
<pre>
 
<pre>
Usage: kojay spin-livecd [options] <target> <arch> <kickstart-file>
+
Usage: koji spin-livecd [options] <target> <arch> <kickstart-file>
 
(Specify the --help global option for a list of other help options)
 
(Specify the --help global option for a list of other help options)
  
Line 46: Line 75:
 
</pre>
 
</pre>
  
The way LiveCDs are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies in the livecd-build group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. A livecd command is executed as such (using the mock('--chroot', ...) method): ''livecd-creator -f <filename> -c <kickstart-file> -d -v --logfile=/tmp/livecd.log --cache=/tmp/koji-livecd''. Once that completes, the image file is copied to /mnt/koji/images/livecd/$imageID if it is not a scratch image. Note that the logfile and cache directory were arbitrarily chosen and hardcoded.
+
For appliances, use the spin-appliance directive.
  
=== Caveats ===
+
<pre>
 +
Usage: kojay spin-appliance [options] <target> <arch> <kickstart-file>
 +
(Specify the --help global option for a list of other help options)
  
1. Firstly, word of caution about kickstart files and the %include macro. livecd-creator is smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the spin-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user's local disk.
+
Options:
 
+
  -h, --help            show this help message and exit
2. Another issue is if the kickstart file passed to livecd-creator defines repositories that Koji isn't aware of, and pulls packages from those repositories, the task will fail when storing RPM information to the database. This is because packages may be installed into the image that Koji cannot resolve the origin of, which causes the task to fail out. The way we sidestep this issue is by overriding the repos defined in the kickstart file with the repositories associated with the build target supplied on the command line. This is achieved using pykickstart. (kojid is now dependent on it with these changes) Note that if you use --repo though, you may run into this issue.
+
  --wait                Wait on the appliance creation, even if running in the
 
+
                        background
3. Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don't since Koji's comps.xml is based on the "groups" set up from the CLI. livecd-creator's behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are two possible workarounds: do not use package groups in the kickstart file and just specify a huge list of packages, or use --repo and specify a repo that does have a comps.xml that defines the groups it uses. Just make sure that repo is also one that Koji knows about. (see #2)
+
  --nowait              Don't wait on appliance creation
 
+
  --noprogress          Do not display progress of the upload
4. LiveCD images cannot be populated using local repos cannot use signed RPMs, because the repos Koji creates never have signed RPMs. You'll have to use external repos.
+
  --background          Run the appliance creation task at a lower priority
 
+
  --ksurl=SCMURL        The URL to the SCM containing the kickstart file
5. Debuginfo RPMs are tricky to get into LiveCDs as well, because by default the local repos generated by Koji do not have them. You can include them on a regeneration, but the next time Kojira regenerates them, they'll be gone again. The workaround is to regen the repo using ''koji call newRepo "debuginfo=True"'' just before building the LiveCD.
+
  --ksversion=VERSION  The syntax version used in the kickstart file
 +
  --scratch            Create a scratch appliance
 +
  --tag                Interpret <target> as a tag instead
 +
  --repo=REPO          Specify a comma-separated list of repos that will
 +
                        override the repo used to install RPMs in the
 +
                        appliance. The build tag repo associated with the
 +
                        target is the default.
 +
  --name=NAME          Specify the appliance file name
 +
  --version=VERSION    Set an appliance version, used in .xml metadata
 +
  --release=RELEASE    Set an appliance release number, used in .xml metadata
 +
  --vmem=VMEM          Set the amount of virtual memory in the appliance in
 +
                        MB, default is 512
 +
  --vcpu=VCPU          Set the number of virtual cpus in the appliance,
 +
                        default is 1
 +
  --format=DISK_FORMAT  Disk format, default is raw. Other options are qcow,
 +
                        qcow2, and vmx.
 +
</pre>

Latest revision as of 06:41, 24 March 2012

This page documents how images can be built in Koji. It covers the supported use cases, how to configure your Koji instance to build them, and how to build them once it is ready. This is a very new feature and there hasn't been much external testing yet, so please keep that mind if you run into difficulty. Be certain to read the next section to be certain your use case is supported as well. For additional questions and information, ask around in #koji on FreeNode IRC and sign up to the right mailing lists as the Koji project website dictates.

Contents

[edit] Supported Images & Use Cases

The way images are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies from the livecd-build or appliance-build package group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. A livecd-creator or appliance-creator command is executed using the mock('--chroot', ...) method. Note: this process runs as root. This produces the desired image which is uploaded to /mnt/koji/images/<image>/$imageID if it is not a scratch image.

[edit]  %include macros

A word of caution about kickstart files and the %include macro. livecd-creator and appliance-creator are smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the spin-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user's local disk.

[edit] Package Groups in the Kickstart File

Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don't since Koji's comps.xml is based on the "groups" set up from the CLI. {livecd,appliance}-creator's behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are a couple possible workarounds:

  • do not use package groups in the kickstart file and just specify a huge list of packages
  • use --repo and specify a repo that does have a comps.xml that defines the groups it uses

[edit] Only Include RPMs Built in Koji

The image building tasks will fail if your image tries to include a package that was not built in your build system. This is because the package does not have any origin information stored in Koji's database. The repos defined in the kickstart will automatically be overridden with the repo of the build tag for the build target, unless you use the --repo option. Since only packages you have built (or include from an external repo) should be there, you should never have this problem unless you use --repo.

[edit] No Signed or Debuginfo RPMs in Koji's Build Tags

If you need signed RPMs or debuginfo RPMs, you will run into trouble because Koji does not keep those in its build tag repos. The only work around for this is to create a repo yourself that includes these RPMs and then use --repo. This will force the image to take RPMs from that repo.

[edit] LiveCD-Specific Caveats

While livecd-tools does support building on SELinux disabled hosts, you can run into denials when booting if you create and use new files in the %post section of your kickstart file. If you do, you should either set the labels appropriately at the end of the %post section, or instigate an autorelabel at boot time.

[edit] Appliance-Specific Caveats

Koji only supports Appliances that are single disk images. If your kickstart file contains "part" commands for multiple devices, the task will fail. Patches are welcome to remove this limitation.

[edit] Build System Preparation

This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so here if you need to. Please ensure you are using the latest version of the software and that your database schema is updated as well. You should also have some familiarity with how livecd-creator and appliance-creator work.

Follow this procedure step by step to get things prepared they way they need to be. Italics indicate commands to run on the console. Any time you see <image>, substitute in the desired image type. This can be either "livecd" or "appliance". You can enable both at once as well, just run the command twice using each value for <image>.

  • mkdir /mnt/koji/images: Create a directory to put the finished images. This should be done on the system running kojihub.
  • chown apache:apache /mnt/koji/images: Let apache read and write to it.
  • Version 1.3.2 introduces another PythonOption in kojiweb.conf: KojiImagesURL. This option is set to the URL where completed LiveCD images can be downloaded from. Make sure that your apache server is allowing access to that location.
  • You need to set the topurl option in /etc/kojid/kojid.conf. Usually it looks something like http://your.domain.com/koji
  • koji add-host-to-channel <your-host> <image>: add a builder to the <image> channel
  • koji grant-permission <image> <user>: grant the permission to build an image type to a user. This step is optional since admins have all permissions.
  • You will need a tag and target to build the images from. The yum repo generated for the build tag of the target is what Koji will use to populate the LiveCDs with by default. (the alternative is to use the --repo option, more on that later)
  • koji add-group <build-tag> <image>-build: add the <image>-build group
  • koji add-group-pkg <build-tag> <image>-build <pkg> ...: add packages to the <image>-build group. These package lists vary has packages and dependencies change. As of October, 2010 for Fedora 13 the needed packages for each image type are:
    • livecd: bash, coreutils, dbus-python, dmraid, livecd-tools, selinux-policy-targeted, yum
    • appliance: appliance-tools, bash, coreutils, grub, parted, selinux-policy, shadow-utils, policycoreutils

[edit] Building an Image

Building a LiveCD in Koji is accomplished with the spin-livecd directive. Like all other Koji directives, usage information is available with --help:

Usage: koji spin-livecd [options] <target> <arch> <kickstart-file>
(Specify the --help global option for a list of other help options)

Options:
  -h, --help           show this help message and exit
  --nowait             Don't wait on livecd creation
  --noprogress         Do not display progress of the upload
  --background         Run the livecd creation task at a lower priority
  --isoname=ISONAME    Use a custom name for the iso file
  --ksurl=SCMURL       The URL to the SCM containing the kickstart file
  --ksversion=VERSION  The syntax version used in the kickstart file
  --scratch            Create a scratch LiveCD image.
  --repo=REPO          Specify a comma-separated list of repos that will
                       override the repo used to install RPMs in the LiveCD
                       image. The  repo associated with the target is the
                       default.

For appliances, use the spin-appliance directive.

Usage: kojay spin-appliance [options] <target> <arch> <kickstart-file>
(Specify the --help global option for a list of other help options)

Options:
  -h, --help            show this help message and exit
  --wait                Wait on the appliance creation, even if running in the
                        background
  --nowait              Don't wait on appliance creation
  --noprogress          Do not display progress of the upload
  --background          Run the appliance creation task at a lower priority
  --ksurl=SCMURL        The URL to the SCM containing the kickstart file
  --ksversion=VERSION   The syntax version used in the kickstart file
  --scratch             Create a scratch appliance
  --tag                 Interpret <target> as a tag instead
  --repo=REPO           Specify a comma-separated list of repos that will
                        override the repo used to install RPMs in the
                        appliance. The build tag repo associated with the
                        target is the default.
  --name=NAME           Specify the appliance file name
  --version=VERSION     Set an appliance version, used in .xml metadata
  --release=RELEASE     Set an appliance release number, used in .xml metadata
  --vmem=VMEM           Set the amount of virtual memory in the appliance in
                        MB, default is 512
  --vcpu=VCPU           Set the number of virtual cpus in the appliance,
                        default is 1
  --format=DISK_FORMAT  Disk format, default is raw. Other options are qcow,
                        qcow2, and vmx.