Mass Branching SOP

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(add update critpath step)
m (Taskotron: fix link)
 
(27 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 +
{{admon/important|This page is deprecated| All Fedora Release Engineering Documentation has moved [https://docs.pagure.org/releng/ here] with source hosted along side the code in the [https://pagure.io/releng releng pagure repository]}}
 +
 
{{shortcut|RSOP:MASSBRANCH}}
 
{{shortcut|RSOP:MASSBRANCH}}
 
== Description ==
 
== Description ==
Line 7: Line 9:
 
== Action ==
 
== Action ==
  
=== PackageDB ===
+
=== Turn off auto-update of the git repos ===
  
Mass branching in the pkgdb is the first step.  It should be done near the time that the scm branches are created so as not to confuse packagers. However, it does not cause an outage so it should be done ahead of time.
+
The first step of the mass-branching is to turn off `fedmsg-hub` on pkgs.fedoraproject.org
  
First, someone needs to add the new release to the PackageDB.  [[ISOP:PKGDB#Add a new release| See this section]]
+
Usually, git repositories are automatically updated when something happens in pkgdb. For example, when a new branch is added in pkgdb, the git repo automatically gets a new branch created, and during the regular business hours, this new branch points to the very first commit in the repo allowing to merge any other branch into this new one.
  
Then a scmadmin needs to either start the mass branching process from the command line or by pushing a button on a web page.  The web page method is to browse to https://admin.fedoraproject.org/pkgdb/collections/  login, and then click the "Branch All" button.
+
When we do mass-branching, we want the new branch to point to the last commit in `master`, so we need to stop automatically updating the git repos.
  
The command line method is to run:
+
For this, simply log on to ``pkgs.fedoraproject.org`` and stop the fedmsg-hub service:
 
<pre>
 
<pre>
pkgdb-client mass-branch -b F-13
+
systemctl stop fedmsg-hub
 
</pre>
 
</pre>
  
Whichever method you choose, the command will start the mass branch process on the server and then return with a message that the branching is in progress.
 
  
When the branching is finished, you will receive an email that tells you of any unbranched packages as well as all the packages that were branched.
+
=== PackageDB ===
 
+
If something fails spectacularly, it is safe to try mass branching again at a later time.  If only a few cleanups a re needed it might be better to do that with the regular branch commands.
+
  
{{admon/warning|Target pkgdb-client against bapp01| At present there is a bug in the mass branching code so that the branching seems to work but the changes aren't saved in the databaseI think this occurs when the server handles other requests while the packagedb is processing a mass branch.  Performing the mass branch on bapp1 which is not handling other requests while the mass branch is occurring is one way to work around thisIt's likely a bug in how forking in the mass branch code is interacting with the database connections.
+
Mass branching in the pkgdb is the second stepIt should be done near the time that the scm branches are created so as not to confuse packagersHowever, it does not cause an outage so it could be done ahead of time.
  
Retarget pkgdb-client to use bapp1 by changing the value of pkgdb.url in either /etc/pkgdb-client.cfg or ~/.fedora/pkgdb-client.cfg.
+
The action on pkgdb is in a few steps:
  
 +
# Edit the dist-tag of [https://admin.fedoraproject.org/pkgdb/collection/master/ `master` collection] in PackageDB (to avoid having conflicting koji tags),
 +
# Create the new collection via the [https://admin.fedoraproject.org/pkgdb/admin/ Admin interface of pkgdb], for Fedora Branched the status should be set to <code>Under Development</code> until the [[Schedule|Final Freeze]] is reached. This allows to retire packages until then.
 +
# On one of the pkgdb host (ie: pkgdb01 or pkgdb02 or pkgdb01.stg if you want to try on staging first), call the script pkgdb2_branch.py:
 
<pre>
 
<pre>
[global]
+
sudo pkgdb2_branch.py f21 --user=<fas_user> --groups=<fas_group_allowed>
pkgdb.url = "http://bapp01/pkgdb/"
+
 
</pre>
 
</pre>
}}
 
  
=== Puppet ===
+
''fas_user'' corresponds to the FAS username of the admin doing the action.
 +
 
 +
''fas_group_allowed'' corresponds to a FAS group allowed to perform admin actions (ie: ADMIN_GROUP in the [https://infrastructure.fedoraproject.org/infra/ansible/roles/pkgdb2/templates/pkgdb2.cfg pkgdb configuration file])
 +
 
 +
The mass branch process starts on the server and will last for ~1h45
 +
 
 +
When the branching is finished, the email address defined at MAIL_ADMIN in the configuration file will receive an email that tells which were branched and which were unbranched.
 +
 
 +
If something fails spectacularly, it is safe to try mass branching again at a later time.  If only a few cleanups a re needed it might be better to do that with the regular branch commands.
 +
 
 +
=== Ansible ===
  
 
A couple files under puppet management need to be updated to be aware of a new branch.
 
A couple files under puppet management need to be updated to be aware of a new branch.
Line 72: Line 82:
 
The other file is ran by cron that will read data out of pkgdb and construct an ACL config file for our scm.  It has a section that lists active branches to deal with as pkgdb will provide data for all branches.
 
The other file is ran by cron that will read data out of pkgdb and construct an ACL config file for our scm.  It has a section that lists active branches to deal with as pkgdb will provide data for all branches.
  
Again on the puppet server in a clone: modules/gitolite/files/distgit/genacls.pkgdb
+
Again on the ansible server in a clone: modules/gitolite/files/distgit/genacls.pkgdb
  
 
<pre>
 
<pre>
Line 87: Line 97:
 
   
 
   
 
     # Create a "regex"ish list 0f the reserved branches
 
     # Create a "regex"ish list 0f the reserved branches
 +
</pre>
 +
 +
==== fedora-packages ====
 +
 +
There is a file in the fedora-packages webapp source that needs to be
 +
updated with new releases.  It tells fedora-packages what tags
 +
to ask koji about.  Just like before, make the following edit in puppet in a clone:
 +
 +
<pre>
 +
diff --git a/modules/packages/files/distmappings.py b/modules/packages/files/distmappings.py
 +
index c72fd4b..b1fbaa5 100644
 +
--- a/modules/packages/files/distmappings.py
 +
+++ b/modules/packages/files/distmappings.py
 +
@@ -1,5 +1,9 @@
 +
# Global list of koji tags we care about
 +
-tags = ({'name': 'Rawhide', 'tag': 'f20'},
 +
+tags = ({'name': 'Rawhide', 'tag': 'f21'},
 +
+
 +
+        {'name': 'Fedora 20', 'tag': 'f20-updates'},
 +
+        {'name': 'Fedora 20', 'tag': 'f20'},
 +
+        {'name': 'Fedora 20 Testing', 'tag': 'f20-updates-testing'},
 +
 +
        {'name': 'Fedora 19', 'tag': 'f19-updates'},
 +
        {'name': 'Fedora 19', 'tag': 'f19'},
 +
@@ -13,10 +17,6 @@ tags = ({'name': 'Rawhide', 'tag': 'f20'},
 +
        {'name': 'Fedora 17', 'tag': 'f17'},
 +
        {'name': 'Fedora 17 Testing', 'tag': 'f17-updates-testing'},
 +
 +
-        {'name': 'Fedora 16', 'tag': 'f16-updates'},
 +
-        {'name': 'Fedora 16', 'tag': 'f16'},
 +
-        {'name': 'Fedora 16 Testing', 'tag': 'f16-updates-testing'},
 +
-
 +
        {'name': 'EPEL 6', 'tag': 'dist-6E-epel'},
 +
        {'name': 'EPEL 6', 'tag': 'dist-6E-epel-testing'},
 
</pre>
 
</pre>
  
 
==== Push the changes ====
 
==== Push the changes ====
  
When done editing the files, commit and push them, then restart puppet on the scm server in order to get the new files in place.
+
When done editing the files, commit and push them, then run the `pkgs.yml` playbook in order to get the new files in place.
  
 
=== SCM ===
 
=== SCM ===
Line 97: Line 141:
 
The following work is performed on pkgs01
 
The following work is performed on pkgs01
  
==== Make git branches ====
+
==== Adjust pkgdb_sync_git_branches.py ====
Run pkgdb2branch.py to branch the repos on the scm server.  The --branch-for option was designed with this use case in mind::
+
<pre>
+
./pkgdb2branch.py --branch-for=f15
+
</pre>
+
  
If for some reason that doesn't work, you can try this alternative:
+
Edit `/usr/local/bin/pkgdb_sync_git_branches.py` and do the following change:
 +
(Around line 67)
 
<pre>
 
<pre>
cat pkglist.txt|./pkgdb2branch.py -c -
+
- MKBRANCH = '/usr/local/bin/mkbranch'
 +
+ MKBRANCH = '/usr/local/bin/mkbranch_branching'
 
</pre>
 
</pre>
where pkglist.txt is a list of all the packages to branch.
 
  
 
==== Update ACLs ====
 
==== Update ACLs ====
  
Although cron may have run, it is smart to manually run the cron job to make sure new ACLs are in place:
+
Update the git repos manually
  
 
<pre>
 
<pre>
Line 117: Line 158:
 
</pre>
 
</pre>
  
=== Update preupgrade releases.txt ===
+
=== Undo changes to pkgdb_sync_git_branches.py ===
File a [https://fedorahosted.org/fedora-infrastructure ticket with Infrastructure] to modify the mirrors/releases.txt file to include pointers to the new release, for use by preupgrade. This file is in the [http://git.fedorahosted.org/git/?p=fedora-web.git;a=summary fedora-web git repo.]
+
You will need to add something like:
+
  
 +
 +
Edit `/usr/local/bin/pkgdb_sync_git_branches.py` and undo the changes made above:
 +
(Around line 67)
 
<pre>
 
<pre>
[Fedora N ($codename)]
+
+ MKBRANCH = '/usr/local/bin/mkbranch'
stable=False
+
- MKBRANCH = '/usr/local/bin/mkbranch_branching'
preupgrade-ok=True
+
version=N
+
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-N&arch=$basearch
+
installmirrorlist=http://mirrors.fedoraproject.org/mirrorlist?path=pub/fedora/linux/development/N/Fedora/$basearch/os/
+
 
</pre>
 
</pre>
  
=== Update AutoQA repoinfo.conf ===
+
=== Turn back on the auto-update for the git repos ===
The [[AutoQA]] project maintains a config file ({{filename|repoinfo.conf}}) that describes available package repositories and their inheritance. In order to have automated testing of Branched package builds, the {{filename|repoinfo.conf}} file needs to be updated.  Please [https://fedorahosted.org/autoqa/newticket?summary=Update%20repoinfo.conf%20-%20Add%20branch&type=task&component=production&priority=critical&milestone=Hot%20issues file an autoqa ticket] to modify the {{filename|repoinfo.conf}} file to include pointers to the new release.
+
 
 +
On ``pkgs.fedoraproject.org`` restart ``fedmsg-hub`` so:
 +
<pre>
 +
systemctl start fedmsg-hub
 +
</pre>
 +
 
 +
 
 +
=== Taskotron ===
 +
[https://phab.qadevel.cloud.fedoraproject.org/maniphest/task/edit/form/default/?title=new%20Branched%20release&priority=80&tags=libtaskotron File a Taskotron ticket] and ask for the newly branched release support to be added. (Log in to Phabricator using your FAS_account@fedoraproject.org email address).
  
 
=== Koji ===
 
=== Koji ===
Line 147: Line 193:
 
=== Update kickstart used by nightly live ISOs ===
 
=== Update kickstart used by nightly live ISOs ===
  
On a nightly basis, a live ISO image is created for each [http://spins.fedoraproject.org spin] and hosted at http://alt.fedoraproject.org/pub/alt/nightly-composes.  The yum repositories used by [https://fedorahosted.org/spin-kickstarts/ spin-kickstarts] need to be updated to use the branched repository.  Please [https://fedorahosted.org/rel-eng/newticket?summary=Update%20nightly%20spin%20kickstart&type=task&component=production&priority=critical&milestone=Hot%20issues&cc=kevin file a rel-eng ticket] to request updating the kickstart file used to generate the nightly spin ISO's.
+
On a nightly basis, a live ISO image is created for each [http://spins.fedoraproject.org spin] and hosted at http://alt.fedoraproject.org/pub/alt/nightly-composes.  The [[dnf]]|[[yum]] repositories used by [https://fedorahosted.org/spin-kickstarts/ spin-kickstarts] need to be updated to use the branched repository.  Please [https://fedorahosted.org/rel-eng/newticket?summary=Update%20nightly%20spin%20kickstart&type=task&component=production&priority=critical&milestone=Hot%20issues&cc=kevin file a rel-eng ticket] to request updating the kickstart file used to generate the nightly spin ISO's.
  
 
=== Comps ===
 
=== Comps ===
Line 156: Line 202:
  
 
=== MirrorManager ===
 
=== MirrorManager ===
Mirror manager will have to be updated so that the yum repo redirections are going to the right places.  '''FIXME Link to MM SOP'''
+
Mirror manager will have to be updated so that the dnf|yum repo redirections are going to the right places.  '''FIXME Link to MM SOP'''
  
 
<!-- This is commented out until a new one is written.
 
<!-- This is commented out until a new one is written.
Line 177: Line 223:
 
=== Update critpath ===
 
=== Update critpath ===
  
Packagedb has information about which packages are critpath and which are not.  A script that reads the yum repodata (critpath group in comps, and the package dependencies) is used to generate this.  Read the [[Update Critpath SOP]] for the steps to take.
+
Packagedb has information about which packages are critpath and which are not.  A script that reads the dnf|yum repodata (critpath group in comps, and the package dependencies) is used to generate this.  Read the [[Update Critpath SOP]] for the steps to take.
  
 
== Consider Before Running ==
 
== Consider Before Running ==
 
* '''FIXME'''
 
* '''FIXME'''
 
[[Category:Release Engineering SOPs]]
 
[[Category:Release Engineering SOPs]]

Latest revision as of 14:40, 24 February 2016

Important.png
This page is deprecated
All Fedora Release Engineering Documentation has moved here with source hosted along side the code in the releng pagure repository
Shortcut:
RSOP:MASSBRANCH

Contents

[edit] Description

At each alpha freeze we branch the pending release away from devel/ which allows rawhide to move on while the pending release goes into bugfix and polish mode.

[edit] Action

[edit] Turn off auto-update of the git repos

The first step of the mass-branching is to turn off `fedmsg-hub` on pkgs.fedoraproject.org

Usually, git repositories are automatically updated when something happens in pkgdb. For example, when a new branch is added in pkgdb, the git repo automatically gets a new branch created, and during the regular business hours, this new branch points to the very first commit in the repo allowing to merge any other branch into this new one.

When we do mass-branching, we want the new branch to point to the last commit in `master`, so we need to stop automatically updating the git repos.

For this, simply log on to ``pkgs.fedoraproject.org`` and stop the fedmsg-hub service:

systemctl stop fedmsg-hub


[edit] PackageDB

Mass branching in the pkgdb is the second step. It should be done near the time that the scm branches are created so as not to confuse packagers. However, it does not cause an outage so it could be done ahead of time.

The action on pkgdb is in a few steps:

  1. Edit the dist-tag of `master` collection in PackageDB (to avoid having conflicting koji tags),
  2. Create the new collection via the Admin interface of pkgdb, for Fedora Branched the status should be set to Under Development until the Final Freeze is reached. This allows to retire packages until then.
  3. On one of the pkgdb host (ie: pkgdb01 or pkgdb02 or pkgdb01.stg if you want to try on staging first), call the script pkgdb2_branch.py:
sudo pkgdb2_branch.py f21 --user=<fas_user> --groups=<fas_group_allowed>

fas_user corresponds to the FAS username of the admin doing the action.

fas_group_allowed corresponds to a FAS group allowed to perform admin actions (ie: ADMIN_GROUP in the pkgdb configuration file)

The mass branch process starts on the server and will last for ~1h45

When the branching is finished, the email address defined at MAIL_ADMIN in the configuration file will receive an email that tells which were branched and which were unbranched.

If something fails spectacularly, it is safe to try mass branching again at a later time. If only a few cleanups a re needed it might be better to do that with the regular branch commands.

[edit] Ansible

A couple files under puppet management need to be updated to be aware of a new branch.

[edit] pkgdb2branch.py

This file is used by an scmadmin to read data from pkgdb and create branches in the source control. Two parts need to be updated, one part that defines valid branches and what existing branch to create them from, and the other part defines a mapping of branch names in pkgdb to branch names in scm.

On the puppet server in a clone edit the modules/gitolite/files/distgit/pkgdb2branch.py file:

diff --git a/modules/gitolite/files/distgit/pkgdb2branch.py b/modules/gitolite/file
index ce79467..c40e83c 100755
--- a/modules/gitolite/files/distgit/pkgdb2branch.py
+++ b/modules/gitolite/files/distgit/pkgdb2branch.py
@@ -29,14 +29,15 @@ BRANCHES = {'el4': 'master', 'el5': 'master', 'el6': 'f12',
         'f11': 'master',
         'f12': 'master',
         'f13': 'master',
-        'f14': 'master'}
+        'f14': 'master',
+        'f15': 'master'}
 
 # The branch names we get out of pkgdb have to be translated to git
 GITBRANCHES = {'EL-4': 'el4', 'EL-5': 'el5', 'EL-6': 'el6', 'OLPC-2': 'olpc2',
                'FC-6': 'fc6', 'F-7': 'f7', 'F-8': 'f8', 'F-9': 'f9',
                'F-10': 'f10', 'OLPC-3': 'olpc3',
                'F-11': 'f11', 'F-12': 'f12', 'F-13': 'f13', 'f14': 'f14',
-               'devel': 'master'}
+               'f15': 'f15', 'devel': 'master'}

[edit] genacls.pkgdb

The other file is ran by cron that will read data out of pkgdb and construct an ACL config file for our scm. It has a section that lists active branches to deal with as pkgdb will provide data for all branches.

Again on the ansible server in a clone: modules/gitolite/files/distgit/genacls.pkgdb

diff --git a/modules/gitolite/files/distgit/genacls.pkgdb b/modules/gitolite/files/
index e531dc2..07b2ba7 100755
--- a/modules/gitolite/files/distgit/genacls.pkgdb
+++ b/modules/gitolite/files/distgit/genacls.pkgdb
@@ -22,7 +22,7 @@ if __name__ == '__main__':
     ACTIVE = {'OLPC-2': 'olpc2/', 'OLPC-3': 'olpc3/', 'EL-4': 'el4/',
               'EL-5': 'el5/', 'EL-6': 'el6/', 'F-11': 'f11/',
               'F-12': 'f12/', 'F-13': 'f13/', 'f14': 'f14/',
-              'devel': 'master'}
+              'f15': 'f15/', 'devel': 'master'}
 
     # Create a "regex"ish list 0f the reserved branches

[edit] fedora-packages

There is a file in the fedora-packages webapp source that needs to be updated with new releases. It tells fedora-packages what tags to ask koji about. Just like before, make the following edit in puppet in a clone:

diff --git a/modules/packages/files/distmappings.py b/modules/packages/files/distmappings.py
index c72fd4b..b1fbaa5 100644
--- a/modules/packages/files/distmappings.py
+++ b/modules/packages/files/distmappings.py
@@ -1,5 +1,9 @@
 # Global list of koji tags we care about
-tags = ({'name': 'Rawhide', 'tag': 'f20'},
+tags = ({'name': 'Rawhide', 'tag': 'f21'},
+
+        {'name': 'Fedora 20', 'tag': 'f20-updates'},
+        {'name': 'Fedora 20', 'tag': 'f20'},
+        {'name': 'Fedora 20 Testing', 'tag': 'f20-updates-testing'},
 
         {'name': 'Fedora 19', 'tag': 'f19-updates'},
         {'name': 'Fedora 19', 'tag': 'f19'},
@@ -13,10 +17,6 @@ tags = ({'name': 'Rawhide', 'tag': 'f20'},
         {'name': 'Fedora 17', 'tag': 'f17'},
         {'name': 'Fedora 17 Testing', 'tag': 'f17-updates-testing'},
 
-        {'name': 'Fedora 16', 'tag': 'f16-updates'},
-        {'name': 'Fedora 16', 'tag': 'f16'},
-        {'name': 'Fedora 16 Testing', 'tag': 'f16-updates-testing'},
-
         {'name': 'EPEL 6', 'tag': 'dist-6E-epel'},
         {'name': 'EPEL 6', 'tag': 'dist-6E-epel-testing'},

[edit] Push the changes

When done editing the files, commit and push them, then run the `pkgs.yml` playbook in order to get the new files in place.

[edit] SCM

The following work is performed on pkgs01

[edit] Adjust pkgdb_sync_git_branches.py

Edit `/usr/local/bin/pkgdb_sync_git_branches.py` and do the following change: (Around line 67)

- MKBRANCH = '/usr/local/bin/mkbranch'
+ MKBRANCH = '/usr/local/bin/mkbranch_branching'

[edit] Update ACLs

Update the git repos manually

$ sudo -u jkeating /usr/local/bin/genacls.sh

[edit] Undo changes to pkgdb_sync_git_branches.py

Edit `/usr/local/bin/pkgdb_sync_git_branches.py` and undo the changes made above: (Around line 67)

+ MKBRANCH = '/usr/local/bin/mkbranch'
- MKBRANCH = '/usr/local/bin/mkbranch_branching'

[edit] Turn back on the auto-update for the git repos

On ``pkgs.fedoraproject.org`` restart ``fedmsg-hub`` so:

systemctl start fedmsg-hub


[edit] Taskotron

File a Taskotron ticket and ask for the newly branched release support to be added. (Log in to Phabricator using your FAS_account@fedoraproject.org email address).

[edit] Koji

The koji build system needs to have some tag/target work done to handle builds from the new branch and to update where builds from master go. See the section on Koji in the Adding Build Targets SOP for details.

[edit] Fedora Release

The Fedora release package needs to be updated in both the new branch and in master. FIXME Link to fedora release bump SOP

[edit] Bodhi

Bodhi needs to be turned on for the new branch. Instructions in the Bodhi SOP

[edit] Enable nightly branched compose

A cron job needs to be modified and turned on for the new branch. FIXME Link to nightly branched SOP

[edit] Update kickstart used by nightly live ISOs

On a nightly basis, a live ISO image is created for each spin and hosted at http://alt.fedoraproject.org/pub/alt/nightly-composes. The dnf|yum repositories used by spin-kickstarts need to be updated to use the branched repository. Please file a rel-eng ticket to request updating the kickstart file used to generate the nightly spin ISO's.

[edit] Comps

A new comps file needs to be created for the next fedora release (the one after what we just branched for). FIXME Link to comps SOP

[edit] Mock

Mock needs to be updated to have configs for the new branch. This should actually be done and pushed just before the branch event. FIXME Link to mock update SOP

[edit] MirrorManager

Mirror manager will have to be updated so that the dnf|yum repo redirections are going to the right places. FIXME Link to MM SOP


[edit] Update critpath

Packagedb has information about which packages are critpath and which are not. A script that reads the dnf|yum repodata (critpath group in comps, and the package dependencies) is used to generate this. Read the Update Critpath SOP for the steps to take.

[edit] Consider Before Running

  • FIXME