User:Skvidal/BuildSystem

From FedoraProject

< User:Skvidal(Difference between revisions)
Jump to: navigation, search
(Created page with "= Fedora Build System - beyond building the distro = == Summary == Right now the fedora build system is focused solely on building the packages for the fedora distribution. It ...")
 
Line 2: Line 2:
  
 
== Summary ==
 
== Summary ==
Right now the fedora build system is focused solely
+
Right now the fedora build system is focused solely on building  
on building the packages for the fedora distribution. It has
+
the packages for the fedora distribution. It has little or no support  
little or no support for building any other arbitrary package set
+
for building any other arbitrary package set nor for non-fedora (but free software things).
nor for non-fedora (but free software things).
+
  
 
== Our goals ==
 
== Our goals ==
- maintain existing build capabilities in koji (don't break koji)
+
* maintain existing build capabilities in koji (don't break koji)
- maintain compatibility of our tools (don't break fedpkg)
+
* maintain compatibility of our tools (don't break fedpkg)
- create a fedora-contributor-accessible space where arbitrary pkgs
+
* create a fedora-contributor-accessible space where arbitrary pkgs
 
   from arbitrary repos, with arbitrary dependencies can be built
 
   from arbitrary repos, with arbitrary dependencies can be built
 
   against fedora securely and consistently. (coprs)
 
   against fedora securely and consistently. (coprs)
- overhaul how we setup and maintain our buildboxes so we can  
+
* overhaul how we setup and maintain our buildboxes so we can  
 
   redeploy these systems quickly and scale up during high demand.
 
   redeploy these systems quickly and scale up during high demand.
 
   (builder git repo)
 
   (builder git repo)
Line 19: Line 18:
  
 
== How we get there ==
 
== How we get there ==
There are a lot of areas needed to make this all come together. I broke
+
There are a lot of areas needed to make this all come together. I broke the process into a series of smaller components such that we had
the process into a series of smaller components such that we had
+
something useful at each step and could assemble them to become the whole system, like voltron.
something useful at each step and could assemble them to become the
+
whole system.
+
  
 
=== Things done or mostly done ===
 
=== Things done or mostly done ===
Line 28: Line 25:
 
to achieve our goals
 
to achieve our goals
  
  - Changes to mock:
+
  ==== Changes to mock ====
   - added mockchain to mock - implements recursive chain builds and
+
   * added mockchain to mock - implements recursive chain builds and
 
     creates the mock config from base configs adding arbitrary user
 
     creates the mock config from base configs adding arbitrary user
 
     repos. http://git.fedorahosted.org/cgit/mock.git/tree/py/mockchain.py
 
     repos. http://git.fedorahosted.org/cgit/mock.git/tree/py/mockchain.py
 
             http://skvidal.wordpress.com/2012/04/20/mockchain-use-cases-and-examples/
 
             http://skvidal.wordpress.com/2012/04/20/mockchain-use-cases-and-examples/
   - added plugin to mock to output the chroot system state as well
+
   * added plugin to mock to output the chroot system state as well
 
     as set of pkgs available in repositories to yum/mock at the time
 
     as set of pkgs available in repositories to yum/mock at the time
 
     of the build. Important criteria for solving build failures due
 
     of the build. Important criteria for solving build failures due
 
     to missing build requirements.  
 
     to missing build requirements.  
  
- Changes to builders:
+
==== Changes to builders ====
   - reinstall and vm-ize all of our existing builder hw in the bladecenters
+
   * reinstall and vm-ize all of our existing builder hw in the bladecenters
   - setup newbuilder process using libvirtd and ansible to make our builder
+
   * setup newbuilder process using libvirtd and ansible to make our builder
 
     deployment be trivial and consistent.
 
     deployment be trivial and consistent.
 
   
 
   
  
 
=== Work Started ===
 
=== Work Started ===
- Changes to our infrastructure:
+
==== Changes to our infrastructure ====
 
   preparing for our private cloud deployment inside phx2. This should
 
   preparing for our private cloud deployment inside phx2. This should
 
   allow us to create arbitrary and COMPLETELY isolated instances on
 
   allow us to create arbitrary and COMPLETELY isolated instances on
Line 51: Line 48:
 
   sufficient resources (or if we have a way of paying for them :)
 
   sufficient resources (or if we have a way of paying for them :)
  
- Coprs web interface:
+
==== Coprs web interface ====
 
   For user/contributors to submit urls to srpms and build repos
 
   For user/contributors to submit urls to srpms and build repos
 
   and have the pkgs from the srpm urls built (recursively) against
 
   and have the pkgs from the srpm urls built (recursively) against
Line 59: Line 56:
 
   incomplete currently.
 
   incomplete currently.
  
- mockremote/ftbfs tools:
+
===== mockremote/ftbfs tools ====
 
   Main control job for spawning arbitrary jobs on remote systems.  
 
   Main control job for spawning arbitrary jobs on remote systems.  
   - mockremote is a single buildsystem mechanism to parcel out jobs
+
 
 +
   * mockremote is a single buildsystem mechanism to parcel out jobs
 
     one at a time and maintain a local repository for them to draw
 
     one at a time and maintain a local repository for them to draw
 
     build requirements from.
 
     build requirements from.
  
   - ftbfs - multi-builder/multijob builder to attempt to rebuild all
+
   * ftbfs - multi-builder/multijob builder to attempt to rebuild all
 
     of fedora distribution packages. This works but needs refactoring
 
     of fedora distribution packages. This works but needs refactoring
 
     to scale to hundreds of builder processes. It also has a couple of
 
     to scale to hundreds of builder processes. It also has a couple of
Line 76: Line 74:
 
=== Work Needed/Not Started ===
 
=== Work Needed/Not Started ===
  
- Changes to koji:
+
==== Changes to koji ====
  - Script to run in kojira to monitor builders and tasks
+
  Script to run in kojira to monitor builders and tasks
 
     which have not checked in at their regular schedule. This would check
 
     which have not checked in at their regular schedule. This would check
 
     for abandoned/unfinished jobs and free them back up for another koji
 
     for abandoned/unfinished jobs and free them back up for another koji
 
     builder to take up. This will allow us to allocate/deallocate  
 
     builder to take up. This will allow us to allocate/deallocate  
 
     builders without having to worry about:
 
     builders without having to worry about:
       1. disabling the builder in koji (finding a kojiadmin)
+
       #. disabling the builder in koji (finding a kojiadmin)
       2. if a build is currently running on any given host
+
       #. if a build is currently running on any given host
  
- Storage of COPRS/BUILDS:
+
==== Storage of COPRS/BUILDS ====
   Figure out a storage destination for the results of any of these  
+
   Figure out a storage destination for the results of any of these jobs. Right now we don't have a destination in mind for where these
  jobs. Right now we don't have a destination in mind for where these
+
   pkgs should go. Fedorapeople.org was suggested but it is a difficult fit to get files to that system and make sure they are accessible
   pkgs should go. Fedorapeople.org was suggested but it is a difficult
+
  fit to get files to that system and make sure they are accessible
+
 
   and owned by the right users.
 
   and owned by the right users.
  
- Maintenance and harvesting/clean up of all of the above:
+
==== Maintenance and harvesting/clean up of all of the above ====
   - builds/coprs
+
   * builds/coprs
   - repos
+
   * repos
   - builders spawned in scale-up
+
   * builders spawned in scale-up
   - ftbfs dumps.
+
   * ftbfs dumps.

Revision as of 22:31, 26 July 2012

Contents

Fedora Build System - beyond building the distro

Summary

Right now the fedora build system is focused solely on building the packages for the fedora distribution. It has little or no support for building any other arbitrary package set nor for non-fedora (but free software things).

Our goals

  • maintain existing build capabilities in koji (don't break koji)
  • maintain compatibility of our tools (don't break fedpkg)
  • create a fedora-contributor-accessible space where arbitrary pkgs
  from arbitrary repos, with arbitrary dependencies can be built
  against fedora securely and consistently. (coprs)
  • overhaul how we setup and maintain our buildboxes so we can
  redeploy these systems quickly and scale up during high demand.
  (builder git repo)


How we get there

There are a lot of areas needed to make this all come together. I broke the process into a series of smaller components such that we had something useful at each step and could assemble them to become the whole system, like voltron.

Things done or mostly done

Over the last 4 or 5 months I've been working on a series of tools

to achieve our goals

==== Changes to mock ====
  * added mockchain to mock - implements recursive chain builds and
    creates the mock config from base configs adding arbitrary user
    repos. http://git.fedorahosted.org/cgit/mock.git/tree/py/mockchain.py
           http://skvidal.wordpress.com/2012/04/20/mockchain-use-cases-and-examples/
  * added plugin to mock to output the chroot system state as well
    as set of pkgs available in repositories to yum/mock at the time
    of the build. Important criteria for solving build failures due
    to missing build requirements. 

Changes to builders

  * reinstall and vm-ize all of our existing builder hw in the bladecenters
  * setup newbuilder process using libvirtd and ansible to make our builder
    deployment be trivial and consistent.

Work Started

Changes to our infrastructure

  preparing for our private cloud deployment inside phx2. This should
  allow us to create arbitrary and COMPLETELY isolated instances on
  demand without a concern for where to deploy them and if we have
  sufficient resources (or if we have a way of paying for them :)

Coprs web interface

  For user/contributors to submit urls to srpms and build repos
  and have the pkgs from the srpm urls built (recursively) against
  each other and the packages in the build repos. I've started the coprs
  interface as a test case for trying out flask. I have a limited 
  interface and the data saving out to a db. This is insufficient and 
  incomplete currently.

= mockremote/ftbfs tools

  Main control job for spawning arbitrary jobs on remote systems. 
  * mockremote is a single buildsystem mechanism to parcel out jobs
    one at a time and maintain a local repository for them to draw
    build requirements from.
  * ftbfs - multi-builder/multijob builder to attempt to rebuild all
    of fedora distribution packages. This works but needs refactoring
    to scale to hundreds of builder processes. It also has a couple of
    race issues creating a local repo for the whole process. Tested
    on part of rawhide (2500ish pkgs) with good results.



Work Needed/Not Started

Changes to koji

  Script to run in kojira to monitor builders and tasks
   which have not checked in at their regular schedule. This would check
   for abandoned/unfinished jobs and free them back up for another koji
   builder to take up. This will allow us to allocate/deallocate 
   builders without having to worry about:
     #. disabling the builder in koji (finding a kojiadmin)
     #. if a build is currently running on any given host

Storage of COPRS/BUILDS

 Figure out a storage destination for the results of any of these jobs. Right now we don't have a destination in mind for where these
 pkgs should go. Fedorapeople.org was suggested but it is a difficult fit to get files to that system and make sure they are accessible
 and owned by the right users.

Maintenance and harvesting/clean up of all of the above

  * builds/coprs
  * repos
  * builders spawned in scale-up
  * ftbfs dumps.