Adding Build Targets SOP

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
m (1 revision(s))
(→‎Koji)
(27 intermediate revisions by 8 users not shown)
Line 1: Line 1:
−
<!-- page was renamed from ReleaseEngineering/RelEngSOPTemplate
 
−
-->
 
−
<!-- page was renamed from ReleaseEngineering/SOP/RelEngSOPTemplate
 
−
-->
 
−
= Adding Build Targets - SOP =
 
−
 
−
 
 
== Description ==
 
== Description ==
 
<!-- Put a description of the task here.
 
<!-- Put a description of the task here.
Line 14: Line 7:
 
<!-- Describe the action and provide examples
 
<!-- Describe the action and provide examples
 
-->
 
-->
−
Adding a build target is a complicated task.  It involves updating koji, CVS, fedora-release package, and a few wiki pages.
+
Adding a build target is a complicated task.  It involves updating koji, git, fedora-release package, and a few wiki pages.
  
 
=== Koji ===
 
=== Koji ===
 
In koji a couple collection tags need to be made, and a target created to tie them together.  We create a base collection tag named after the release, and a build tag to hold a few things we use in the buildroots that are not part of the distribution (glibc32/glibc64).  Inheritance to the previous release is used for ownership and package data, as well as buildroot content data.
 
In koji a couple collection tags need to be made, and a target created to tie them together.  We create a base collection tag named after the release, and a build tag to hold a few things we use in the buildroots that are not part of the distribution (glibc32/glibc64).  Inheritance to the previous release is used for ownership and package data, as well as buildroot content data.
  
−
The <code>add-tag</code>, <code>add-tag-inheritance</code>, and <code>add-target</code> commands are used.
+
The <code>add-tag</code>, <code>add-tag-inheritance</code>, <code>edit-tag</code>, and <code>add-target</code> commands are used.
 
<pre>
 
<pre>
 
$ koji add-tag --help
 
$ koji add-tag --help
Line 46: Line 39:
 
parent tag
 
parent tag
  
 +
$ koji edit-tag --help
 +
Usage: koji edit-tag [options] name
 +
(Specify the --help global option for a list of other help options)
 +
 +
Options:
 +
  -h, --help      show this help message and exit
 +
  --arches=ARCHES  Specify arches
 +
  --perm=PERM      Specify permission requirement
 +
  --no-perm        Remove permission requirement
 +
  --lock          Lock the tag
 +
  --unlock        Unlock the tag
 +
  --rename=RENAME  Rename the tag
  
 
$ koji add-target --help
 
$ koji add-target --help
Line 55: Line 60:
 
</pre>
 
</pre>
  
−
For example if we wanted to create the Fedora 9 build target, we would do the following:
+
For example if we wanted to create the Fedora 17 tags, we would do the following:
 
<pre>
 
<pre>
−
koji add-tag --parent dist-f8 --arches=i386,x86_64,ppc,ppc64 dist-f9
+
koji add-tag --parent f16-updates f17
−
koji add-tag --parent dist-f9 --arches=i386,x86_64,ppc,ppc64 dist-f9-build
+
koji add-tag --parent f17 f17-updates
−
koji add-tag-inheritance --priority 2 dist-f9-build dist-f8-build
+
koji add-tag --parent f17-updates f17-candidate
−
koji add-target dist-f9 dist-f9-build
+
koji add-tag --parent f17-updates f17-updates-testing
 +
koji add-tag --parent f17-updates-testing f17-updates-testing-pending
 +
koji add-tag --parent f17-updates f17-updates-pending
 +
koji add-tag --parent f17-updates f17-override
 +
koji add-tag --parent f17-override --arches=i686,x86_64 f17-build
 +
koji add-tag-inheritance --priority 1 f17-build f16-build
 +
koji edit-tag --perm=fedora-override f17-override
 +
koji edit-tag --lock f17-updates
 +
koji add-target f17 f17-build
 
</pre>
 
</pre>
 +
{{admon/note|-pending?|The <code>-pending</code> tags are used by [[Bodhi]] and [[AutoQA]] to track and test proposed updates. These tags are not build targets and they don't get made into repos, so proper inheritance isn't vital.}}
  
−
=== CVS ===
+
=== Git ===
−
Multiple parts of the CVS system need to be made aware of new build targets (and thus new branches).
+
The pkgdb2branch.py file which is hosted in puppet (modules/gitolite/files/distgit/pkgdb2branch.py) needs to be updated for the new target for making branches.
  
−
==== common/ ====
+
Update <code>BRANCHES</code> with the new branch information. The branch name maps to the branch that is its parent.
−
The branches and Makefile.common file need to be updated.
+
−
 
+
−
branches simply needs a new line defining the next release:
+
 
<pre>
 
<pre>
−
F-9:dist-f9:.fc9:fedora:9
+
BRANCHES = {'el4': 'master', 'el5': 'master', 'el6': 'f12',
−
</pre>
+
        'OLPC-2': 'f7',
−
The first field is the branch name as it would be seen on the file system.  The second field is the build target for that branch, the third is what dist would define to, and the last to define the family and release (which are used by other macros).  Depending on when the new target is introduced, you may need to also update the definition of the devel/ branch so that it targets the correct build target.
+
        'master': None,
 +
        'fc6': 'master',
 +
        'f7': 'master',
 +
        'f8': 'master',
 +
        'f9': 'master',
 +
        'f10': 'master',
 +
        'f11': 'master',
 +
        'f12': 'master',
 +
        'f13': 'master', 'f14': 'master'}
  
−
Makefile.common currently has some logic inside of it to determine which branches use koji to build and which use plague.  When new targets/branches are introduced, this logic needs to be updated (this should go away soon).
 
−
<pre>
 
−
ifneq (, $(filter devel F-9 F-8 F-7 OLPC-2, $(BRANCH)))
 
−
build: koji
 
−
else
 
−
build: plague
 
−
endif
 
 
</pre>
 
</pre>
−
Ensure that the new branch is listed in the set of branches for koji.
 
  
−
==== CVSROOT/admin/ ====
+
and update <code>GITBRANCHES</code> with the translation from pkgdb branch string to git branch string:
−
Within CVSROOT/admin/ there are several files that deal with making CVS branches.  They are mkbranchwrapper, pkgdb2branch.py, and tag-check.
+
−
 
+
−
mkbranchwrapper simply needs the <code>VALID_BRANCHES</code> alias updated with the new branch:
+
 
<pre>
 
<pre>
−
VALID_BRANCHES="OLPC-2 FC-3 FC-4 FC-6 F-7 F-8 EL-4 EL-5"
+
GITBRANCHES = {'EL-4': 'el4', 'EL-5': 'el5', 'EL-6': 'el6', 'OLPC-2': 'olpc2',
 +
              'FC-6': 'fc6', 'F-7': 'f7', 'F-8': 'f8', 'F-9': 'f9', 'F-10': 'f10',
 +
              'F-11': 'f11', 'F-12': 'f12', 'F-13': 'f13', 'F-14': 'f14', 'devel': 'master'}
 
</pre>
 
</pre>
  
−
pkgdb2branch.py needs something similar.  Update <code>BRANCHES</code> with the new branch information.  The branch name maps to the branch that is its parent.
+
The genacls.pkgdb file also needs to be updated for active branches to generate ACLs for.  Update the <code>ACTIVE</code> variable.  genacls.pkgdb lives in puppet (modules/gitolite/files/distgit/genacls.pkgdb). The format is pkgdb branch string to git branch string (until pkgdb uses git branch strings):
−
<pre>
+
−
BRANCHES = {'EL-4': 'FC-3', 'EL-5': 'FC-6',
+
−
'OLPC-2': 'F-7',
+
−
'devel': None,
+
−
'RHL-8': None, 'RHL-9': None,
+
−
'FC-1': None, 'FC-2': None, 'FC-3': None, 'FC-4': None, 'FC-5': None,
+
−
'FC-6': None,
+
−
'F-7': 'devel', 'F-8': 'devel', 'F-9': 'devel'}
+
  
−
</pre>
 
−
 
−
tag-check lists a set of tags that are valid and as such it needs to be updated with the tags that are created when a package is branched.  These are branch starting tags and branch splitting tags.  Each set needs to get updated with the new branch:
 
−
<pre>
 
−
RHL-8-start | RHL-9-start | \
 
−
OLPC-2-start | \
 
−
EL-4-start | EL-5-start | \
 
−
FC-1-start | FC-2-start | FC-3-start | FC-4-start | FC-5-start | \
 
−
FC-6-start | F-7-start | F-8-start )
 
−
</pre>
 
 
<pre>
 
<pre>
−
RHL-8-split | RHL-9-split | \
+
    ACTIVE = {'OLPC-2': 'olpc2/', 'EL-4': 'el4/', 'EL-5': 'el5/',
−
OLPC-2-split | \
+
              'EL-6': 'el6/', 'F-11': 'f11/', 'F-12': 'f12/', 'F-13': 'f13/',
−
EL-4-split | EL-5-split | \
+
              'F-14': 'f14/', 'devel': 'master'}
−
FC-1-split | FC-2-split | FC-3-split | FC-4-split | FC-5-split | \
+
−
FC-6-split | F-7-split | F-8-split )
+
 
</pre>
 
</pre>
  
 
=== PackageDB ===
 
=== PackageDB ===
 +
{{admon/note||For PackageDB, add the branches you want to be able to make in git when you want to be able to make them.  ie, when git early branching for Fedora 9 from rawhide is desired, create the Fedora 9 collection in pkgdb.}}
 
The Package Database has two tables that need to be updated with the new release information as well.  These are the commands that were run to update for Fedora 9:
 
The Package Database has two tables that need to be updated with the new release information as well.  These are the commands that were run to update for Fedora 9:
  
Line 129: Line 120:
 
$ sudo -u postgres psql pkgdb
 
$ sudo -u postgres psql pkgdb
 
Password:
 
Password:
−
pkgdb=# insert into collection (name, version, statuscode, owner) values \
+
pkgdb=# insert into collection (name, version, koji_name, statuscode, owner) values \
−
('Fedora', '9', 1, 100351);
+
('Fedora', '15', 'dist-f15', 1, 100351);
  
 
pkgdb=# insert into branch (collectionid, branchname, disttag) \
 
pkgdb=# insert into branch (collectionid, branchname, disttag) \
−
select id, 'F-9', '.fc9' from collection where name = 'Fedora' and version = '9';
+
select id, 'F-15', '.f15' from collection where name = 'Fedora' and version = '15';
 
</pre>
 
</pre>
  
 
* name and version define the release.
 
* name and version define the release.
 +
* koji_name is the tag used for this release in koji.
 
* statuscode for a new release will always be 1 (Active).
 
* statuscode for a new release will always be 1 (Active).
 
* 100351 is Jesse Keating's id.  Change this if someone else is the release manager for this release.
 
* 100351 is Jesse Keating's id.  Change this if someone else is the release manager for this release.
−
* branchname is the name of branch directories in cvs.
+
* branchname is the name of the old branch directories in git.
 
* disttag is the value for disttag in rpm.
 
* disttag is the value for disttag in rpm.
  
Line 147: Line 139:
 
=== Wiki ===
 
=== Wiki ===
 
Various wiki pages need to be updated with the new target information.  Below you'll find a (hopefully current) list of pages to change:
 
Various wiki pages need to be updated with the new target information.  Below you'll find a (hopefully current) list of pages to change:
−
* [[Koji/CurrentKojiTags| ]]  
+
* [[Koji/CurrentKojiTags| Current Koji Tags]]  
−
* [[PackageMaintainers/CVSAdminProcedure| ]]  
+
* [[PackageMaintainers/Git_Admin_Requests| Git Admin Requests]]
 +
 
 +
=== Comps ===
 +
* In the comps module in Fedora Hosted git (ssh://git.fedorarhosted.org/git/comps.git), create and add a new comps file based on the previous release. (Just copying it should be fine.) Add the new file to <code>po/POTFILES.in</code>.
 +
* When rawhide is retargeted in koji to point to the new release, update the <code>Makefile</code> to target comps-rawhide.xml at the new version
 +
* Don't forget to <code>git push</code> your changes after committing.
  
 
== Verification ==
 
== Verification ==
 
<!-- Provide a method to verify the action was successful
 
<!-- Provide a method to verify the action was successful
 
-->
 
-->
−
Given the complexity of all the changes necessary to create a new build target, the best way to verify is to attempt a build.  Given that fedora-release has to be built before anything else so that dist tags translate correctly it is a good test case.  For example, this was used to test the new Fedora 9 target:
+
Given the complexity of all the changes necessary to create a new build target, the best way to verify is to attempt a build.  Given that fedora-release has to be built before anything else so that dist tags translate correctly it is a good test case.  For example, this was used to test the new Fedora 15 target:
  
−
* Use pkgdb to request an F-8 branch of fedora-release
+
* Use pkgdb to request an F-15 branch of fedora-release
−
* Use pkgdb2branch to actually make the branch
+
* Use pkgdb2branch.py to actually make the branch
−
* Update fedora-release checkout (and common/ directory)
+
* Update fedora-release clone
−
* Adjust .spec file in devel/ for new dist defines
+
* Adjust .spec file in master for new dist defines
 
* commit/build
 
* commit/build
 
* Track build in koji to ensure proper tagging is used
 
* Track build in koji to ensure proper tagging is used
Line 164: Line 161:
 
What this won't test is translations of dist at tag time given that fedora-release doesn't use dist in it's Release.  Verifying with a second package that uses dist is a good idea.
 
What this won't test is translations of dist at tag time given that fedora-release doesn't use dist in it's Release.  Verifying with a second package that uses dist is a good idea.
  
−
== Considerations ==
+
== Consider Before Running ==
−
<!-- Create a list of things to keep in mind when performing action.
+
 
−
-->
+
* '''FIXME'''
−
Too many to list right now.
+
* Too many to list right now.
  
−
----
+
[[Category:Release Engineering SOPs]]
−
[[Category:Template]]
+

Revision as of 20:53, 21 September 2012

Contents

Description

Each new release we create a build target for the next release. This SOP will describe the steps necessary to prepare the new build target.

Action

Adding a build target is a complicated task. It involves updating koji, git, fedora-release package, and a few wiki pages.

Koji

In koji a couple collection tags need to be made, and a target created to tie them together. We create a base collection tag named after the release, and a build tag to hold a few things we use in the buildroots that are not part of the distribution (glibc32/glibc64). Inheritance to the previous release is used for ownership and package data, as well as buildroot content data.

The add-tag, add-tag-inheritance, edit-tag, and add-target commands are used. 

$ koji add-tag --help
Usage: koji add-tag [options]  name
(Specify the --help global option for a list of other help options)

Options:
-h, --help       show this help message and exit
--parent=PARENT  Specify parent
--arches=ARCHES  Specify arches


$ koji add-tag-inheritance --help
Usage: koji add-tag-inheritance [options]  tag parent-tag
(Specify the --help global option for a list of other help options)

Options:
-h, --help            show this help message and exit
--priority=PRIORITY   Specify priority
--maxdepth=MAXDEPTH   Specify max depth
--intransitive        Set intransitive
--noconfig            Set to packages only
--pkg-filter=PKG_FILTER
Specify the package filter
--force=FORCE         Force adding a parent to a tag that already has that
parent tag

$ koji edit-tag --help
Usage: koji edit-tag [options] name
(Specify the --help global option for a list of other help options)

Options:
  -h, --help       show this help message and exit
  --arches=ARCHES  Specify arches
  --perm=PERM      Specify permission requirement
  --no-perm        Remove permission requirement
  --lock           Lock the tag
  --unlock         Unlock the tag
  --rename=RENAME  Rename the tag

$ koji add-target --help
Usage: koji add-target name build-tag <dest-tag>
(Specify the --help global option for a list of other help options)

Options:
-h, --help  show this help message and exit

For example if we wanted to create the Fedora 17 tags, we would do the following: 

koji add-tag --parent f16-updates f17
koji add-tag --parent f17 f17-updates
koji add-tag --parent f17-updates f17-candidate
koji add-tag --parent f17-updates f17-updates-testing
koji add-tag --parent f17-updates-testing f17-updates-testing-pending
koji add-tag --parent f17-updates f17-updates-pending
koji add-tag --parent f17-updates f17-override
koji add-tag --parent f17-override --arches=i686,x86_64 f17-build
koji add-tag-inheritance --priority 1 f17-build f16-build
koji edit-tag --perm=fedora-override f17-override
koji edit-tag --lock f17-updates
koji add-target f17 f17-build
Note.png
-pending?
The -pending tags are used by Bodhi and AutoQA to track and test proposed updates. These tags are not build targets and they don't get made into repos, so proper inheritance isn't vital.

Git

The pkgdb2branch.py file which is hosted in puppet (modules/gitolite/files/distgit/pkgdb2branch.py) needs to be updated for the new target for making branches.

Update BRANCHES with the new branch information. The branch name maps to the branch that is its parent. 

BRANCHES = {'el4': 'master', 'el5': 'master', 'el6': 'f12',
        'OLPC-2': 'f7',
        'master': None,
        'fc6': 'master',
        'f7': 'master',
        'f8': 'master',
        'f9': 'master',
        'f10': 'master',
        'f11': 'master',
        'f12': 'master',
        'f13': 'master', 'f14': 'master'}

and update GITBRANCHES with the translation from pkgdb branch string to git branch string: 

GITBRANCHES = {'EL-4': 'el4', 'EL-5': 'el5', 'EL-6': 'el6', 'OLPC-2': 'olpc2',
               'FC-6': 'fc6', 'F-7': 'f7', 'F-8': 'f8', 'F-9': 'f9', 'F-10': 'f10',
               'F-11': 'f11', 'F-12': 'f12', 'F-13': 'f13', 'F-14': 'f14', 'devel': 'master'}

The genacls.pkgdb file also needs to be updated for active branches to generate ACLs for. Update the ACTIVE variable. genacls.pkgdb lives in puppet (modules/gitolite/files/distgit/genacls.pkgdb). The format is pkgdb branch string to git branch string (until pkgdb uses git branch strings): 

    ACTIVE = {'OLPC-2': 'olpc2/', 'EL-4': 'el4/', 'EL-5': 'el5/',
              'EL-6': 'el6/', 'F-11': 'f11/', 'F-12': 'f12/', 'F-13': 'f13/',
              'F-14': 'f14/', 'devel': 'master'}

PackageDB

Note.png
For PackageDB, add the branches you want to be able to make in git when you want to be able to make them. ie, when git early branching for Fedora 9 from rawhide is desired, create the Fedora 9 collection in pkgdb.

The Package Database has two tables that need to be updated with the new release information as well. These are the commands that were run to update for Fedora 9: 

$ ssh bastion
$ ssh db2
$ sudo -u postgres psql pkgdb
Password:
pkgdb=# insert into collection (name, version, koji_name, statuscode, owner) values \
('Fedora', '15', 'dist-f15', 1, 100351);

pkgdb=# insert into branch (collectionid, branchname, disttag) \
select id, 'F-15', '.f15' from collection where name = 'Fedora' and version = '15';
  • name and version define the release.
  • koji_name is the tag used for this release in koji.
  • statuscode for a new release will always be 1 (Active).
  • 100351 is Jesse Keating's id. Change this if someone else is the release manager for this release.
  • branchname is the name of the old branch directories in git.
  • disttag is the value for disttag in rpm.

fedora-release

Currently the fedora-release package provides the %{?dist} definitions used during building of packages. When a new target is created, fedora-release must be built for the collection with new dist definitions.

Wiki

Various wiki pages need to be updated with the new target information. Below you'll find a (hopefully current) list of pages to change:

Comps

  • In the comps module in Fedora Hosted git (ssh://git.fedorarhosted.org/git/comps.git), create and add a new comps file based on the previous release. (Just copying it should be fine.) Add the new file to po/POTFILES.in.
  • When rawhide is retargeted in koji to point to the new release, update the Makefile to target comps-rawhide.xml at the new version
  • Don't forget to git push your changes after committing.

Verification

Given the complexity of all the changes necessary to create a new build target, the best way to verify is to attempt a build. Given that fedora-release has to be built before anything else so that dist tags translate correctly it is a good test case. For example, this was used to test the new Fedora 15 target:

  • Use pkgdb to request an F-15 branch of fedora-release
  • Use pkgdb2branch.py to actually make the branch
  • Update fedora-release clone
  • Adjust .spec file in master for new dist defines
  • commit/build
  • Track build in koji to ensure proper tagging is used

What this won't test is translations of dist at tag time given that fedora-release doesn't use dist in it's Release. Verifying with a second package that uses dist is a good idea.

Consider Before Running

  • FIXME
  • Too many to list right now.
Retrieved from "https://fedoraproject.org/w/index.php?title=Adding_Build_Targets_SOP&oldid=302781"