Mass Branching SOP

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(Make clear how to run against bapp1)
(add update critpath step)
(19 intermediate revisions by 11 users not shown)
Line 1: Line 1:
 +
{{shortcut|RSOP:MASSBRANCH}}
 
== Description ==
 
== Description ==
 
<!-- Put a description of the task here.
 
<!-- Put a description of the task here.
 
-->
 
-->
Each new release we branch all active packages for the new release.
+
At each alpha freeze we branch the pending release away from devel/ which allows rawhide to move on while the pending release goes into bugfix and polish mode.
  
 
== Action ==
 
== Action ==
Line 8: Line 9:
 
=== PackageDB ===
 
=== PackageDB ===
  
Mass branching in the pkgdb is the first step.  It should be done near the time that the cvs branches are created so as not to confuse packagers.  However, it does not cause an outage so it should be done ahead of time.
+
Mass branching in the pkgdb is the first step.  It should be done near the time that the scm branches are created so as not to confuse packagers.  However, it does not cause an outage so it should be done ahead of time.
  
Mass branching is performed by a cvsadmin either from the command line or by pushing a button on a web page.  The web page method is to browse to https://admin.fedoraproject.org/pkgdb/collections/  login, and then click the "Branch All" button.
+
First, someone needs to add the new release to the PackageDB.  [[ISOP:PKGDB#Add a new release| See this section]]
 +
 
 +
Then a scmadmin needs to either start the mass branching process from the command line or by pushing a button on a web page.  The web page method is to browse to https://admin.fedoraproject.org/pkgdb/collections/  login, and then click the "Branch All" button.
  
 
The command line method is to run:
 
The command line method is to run:
 
<pre>
 
<pre>
pkgdb-client mass-branch -b F-10
+
pkgdb-client mass-branch -b F-13
 
</pre>
 
</pre>
  
Line 23: Line 26:
 
If something fails spectacularly, it is safe to try mass branching again at a later time.  If only a few cleanups a re needed it might be better to do that with the regular branch commands.
 
If something fails spectacularly, it is safe to try mass branching again at a later time.  If only a few cleanups a re needed it might be better to do that with the regular branch commands.
  
{{admon/warning|Target pkgdb-client against bapp1| At present there is a bug in the mass branching code so that the branching seems to work but the changes aren't saved in the database.  I think this occurs when the server handles other requests while the packagedb is processing a mass branch.  Performing the mass branch on bapp1 which is not handling other requests while the mass branch is occurring is one way to work around this.  It's likely a bug in how forking in the mass branch code is interacting with the database connections.
+
{{admon/warning|Target pkgdb-client against bapp01| At present there is a bug in the mass branching code so that the branching seems to work but the changes aren't saved in the database.  I think this occurs when the server handles other requests while the packagedb is processing a mass branch.  Performing the mass branch on bapp1 which is not handling other requests while the mass branch is occurring is one way to work around this.  It's likely a bug in how forking in the mass branch code is interacting with the database connections.
  
 
Retarget pkgdb-client to use bapp1 by changing the value of pkgdb.url in either /etc/pkgdb-client.cfg or ~/.fedora/pkgdb-client.cfg.
 
Retarget pkgdb-client to use bapp1 by changing the value of pkgdb.url in either /etc/pkgdb-client.cfg or ~/.fedora/pkgdb-client.cfg.
Line 29: Line 32:
 
<pre>
 
<pre>
 
[global]
 
[global]
pkgdb.url = "https://bapp1/pkgdb/"
+
pkgdb.url = "http://bapp01/pkgdb/"
 
</pre>
 
</pre>
 
}}
 
}}
  
=== CVS ===
+
=== Puppet ===
  
The following work is performed on cvs1
+
A couple files under puppet management need to be updated to be aware of a new branch.
 +
 
 +
==== pkgdb2branch.py ====
 +
 
 +
This file is used by an scmadmin to read data from pkgdb and create branches in the source control.  Two parts need to be updated, one part that defines valid branches and what existing branch to create them from, and the other part defines a mapping of branch names in pkgdb to branch names in scm.
 +
 
 +
On the puppet server in a clone edit the modules/gitolite/files/distgit/pkgdb2branch.py file:
  
==== Disable logins to cvs ====
 
<ol>
 
  <li><code>sudo crontab -u toshio -e</code></li>
 
  <li>Comment out the gen-acl.sh cron job</li>
 
  <li><code>CVSROOT=/cvs/pkgs cvs co CVSROOT</code></li>
 
  <li><code>cp CVSROOT/admin/avail CVSROOT/admin/avail.bak</code></li>
 
  <li>edit CVSROOT/admin/avail and comment out everything below the cvsadmins acl:
 
 
<pre>
 
<pre>
unavail | | rpms
+
diff --git a/modules/gitolite/files/distgit/pkgdb2branch.py b/modules/gitolite/file
avail | @cvsadmin | rpms
+
index ce79467..c40e83c 100755
## Fedora SPARC team
+
--- a/modules/gitolite/files/distgit/pkgdb2branch.py
#avail | @fedora-sparc | rpms
+
+++ b/modules/gitolite/files/distgit/pkgdb2branch.py
## Fedora IA64 team
+
@@ -29,14 +29,15 @@ BRANCHES = {'el4': 'master', 'el5': 'master', 'el6': 'f12',
#avail | @fedora-ia64 | rpms
+
        'f11': 'master',
## Fedora S390 team
+
        'f12': 'master',
#avail | @fedora-s390 | rpms
+
        'f13': 'master',
#
+
-       'f14': 'master'}
#
+
+        'f14': 'master',
#avail | dwmw2,@provenpackager | rpms/0xFFFF
+
+        'f15': 'master'}
[...]
+
 +
# The branch names we get out of pkgdb have to be translated to git
 +
GITBRANCHES = {'EL-4': 'el4', 'EL-5': 'el5', 'EL-6': 'el6', 'OLPC-2': 'olpc2',
 +
                'FC-6': 'fc6', 'F-7': 'f7', 'F-8': 'f8', 'F-9': 'f9',
 +
                'F-10': 'f10', 'OLPC-3': 'olpc3',
 +
                'F-11': 'f11', 'F-12': 'f12', 'F-13': 'f13', 'f14': 'f14',
 +
-              'devel': 'master'}
 +
+              'f15': 'f15', 'devel': 'master'}
 
</pre>
 
</pre>
  </li>
 
  <li>cvs commit your changes</li>
 
</ol>
 
  
==== Disable logging ====
+
==== genacls.pkgdb ====
<ol>
+
  <li><code>CVSROOT=/cvs/pkgs cvs co CVSROOT</code></li>
+
  <li>edit CVSROOT/loginfo and comment out all of the lines</li>
+
  <li><cvs commit the change to the repository</li>
+
</ol>
+
  
==== Create the branch directory ====
+
The other file is ran by cron that will read data out of pkgdb and construct an ACL config file for our scm.  It has a section that lists active branches to deal with as pkgdb will provide data for all branches.
  
Similarly to /cvs/pkgs/devel, the branch directory /cvs/pkgs/F-10 will automatically be populated with symlinks to /cvs/pkgs/rpms/*/F-10.  This should be automatically handled by the mkbranch script in CVSROOT/admin/mkbranch.
+
Again on the puppet server in a clone: modules/gitolite/files/distgit/genacls.pkgdb
  
 
<pre>
 
<pre>
mkdir /cvs/pkgs/F-10
+
diff --git a/modules/gitolite/files/distgit/genacls.pkgdb b/modules/gitolite/files/
chmod g+w /cvs/pkgs/F-10
+
index e531dc2..07b2ba7 100755
 +
--- a/modules/gitolite/files/distgit/genacls.pkgdb
 +
+++ b/modules/gitolite/files/distgit/genacls.pkgdb
 +
@@ -22,7 +22,7 @@ if __name__ == '__main__':
 +
    ACTIVE = {'OLPC-2': 'olpc2/', 'OLPC-3': 'olpc3/', 'EL-4': 'el4/',
 +
              'EL-5': 'el5/', 'EL-6': 'el6/', 'F-11': 'f11/',
 +
              'F-12': 'f12/', 'F-13': 'f13/', 'f14': 'f14/',
 +
-              'devel': 'master'}
 +
+              'f15': 'f15/', 'devel': 'master'}
 +
 +
    # Create a "regex"ish list 0f the reserved branches
 
</pre>
 
</pre>
  
==== Make cvs branches ====
+
==== Push the changes ====
Run pkgdb2branch.py to put the files and directories on the cvs server.  The --branch-for option was designed with this use case in mind::
+
 
 +
When done editing the files, commit and push them, then restart puppet on the scm server in order to get the new files in place.
 +
 
 +
=== SCM ===
 +
 
 +
The following work is performed on pkgs01
 +
 
 +
==== Make git branches ====
 +
Run pkgdb2branch.py to branch the repos on the scm server.  The --branch-for option was designed with this use case in mind::
 
<pre>
 
<pre>
./pkgdb2branch.py --branch-for=F-10
+
./pkgdb2branch.py --branch-for=f15
 
</pre>
 
</pre>
  
Line 90: Line 109:
 
where pkglist.txt is a list of all the packages to branch.
 
where pkglist.txt is a list of all the packages to branch.
  
Now turn everything back on.
+
==== Update ACLs ====
  
==== Update branches file in common ====
+
Although cron may have run, it is smart to manually run the cron job to make sure new ACLs are in place:
The branches file in cvs/pkgs/common/branches dictates where builds go from cvs branches.  This needs to be updated once the branching is done.
+
  
==== Regen ACLs ====
 
As a final step, refresh the acls file with all the new F-10 branches:
 
 
<pre>
 
<pre>
sudo -u toshio /usr/local/bin/genacls.sh
+
$ sudo -u jkeating /usr/local/bin/genacls.sh
 
</pre>
 
</pre>
  
== Getting a List of Unbranched Packages ==
+
=== Update preupgrade releases.txt ===
'''Note: All further information on this page should be obsolete. After F10 successfully branches, remove it.'''
+
File a [https://fedorahosted.org/fedora-infrastructure ticket with Infrastructure] to modify the mirrors/releases.txt file to include pointers to the new release, for use by preupgrade. This file is in the [http://git.fedorahosted.org/git/?p=fedora-web.git;a=summary fedora-web git repo.]
 +
You will need to add something like:
  
There are three things that list packages in releases.  Each plays a part in this.
 
 
Koji keeps track of which packages should be branched and which are blocked.  To get a list for say the F9 packages which are valid to branch:
 
 
<pre>
 
<pre>
koji list-pkgs --quiet --tag dist-f9 |awk '{print $1}' |sort > f9-branchlist
+
[Fedora N ($codename)]
 +
stable=False
 +
preupgrade-ok=True
 +
version=N
 +
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-N&arch=$basearch
 +
installmirrorlist=http://mirrors.fedoraproject.org/mirrorlist?path=pub/fedora/linux/development/N/Fedora/$basearch/os/
 
</pre>
 
</pre>
  
The pkgdb keeps a list of packages in branches.  This can be found by running this sql query:
+
=== Update AutoQA repoinfo.conf ===
 +
The [[AutoQA]] project maintains a config file ({{filename|repoinfo.conf}}) that describes available package repositories and their inheritance.  In order to have automated testing of Branched package builds, the {{filename|repoinfo.conf}} file needs to be updated.  Please [https://fedorahosted.org/autoqa/newticket?summary=Update%20repoinfo.conf%20-%20Add%20branch&type=task&component=production&priority=critical&milestone=Hot%20issues file an autoqa ticket] to modify the {{filename|repoinfo.conf}} file to include pointers to the new release.
 +
 
 +
=== Koji ===
 +
The koji build system needs to have some tag/target work done to handle builds from the new branch and to update where builds from master go. See the [[Adding_Build_Targets_SOP#Koji|section on Koji in the Adding Build Targets SOP]] for details.
 +
 
 +
=== Fedora Release ===
 +
The Fedora release package needs to be updated in both the new branch and in master. '''FIXME Link to fedora release bump SOP'''
 +
 
 +
=== Bodhi ===
 +
Bodhi needs to be turned on for the new branch. Instructions in the [[Bodhi_Infrastructure_SOP|Bodhi SOP]]
 +
 
 +
=== Enable nightly branched compose ===
 +
A cron job needs to be modified and turned on for the new branch.  '''FIXME Link to nightly branched SOP'''
 +
 
 +
=== Update kickstart used by nightly live ISOs ===
 +
 
 +
On a nightly basis, a live ISO image is created for each [http://spins.fedoraproject.org spin] and hosted at http://alt.fedoraproject.org/pub/alt/nightly-composes.  The yum repositories used by [https://fedorahosted.org/spin-kickstarts/ spin-kickstarts] need to be updated to use the branched repository.  Please [https://fedorahosted.org/rel-eng/newticket?summary=Update%20nightly%20spin%20kickstart&type=task&component=production&priority=critical&milestone=Hot%20issues&cc=kevin file a rel-eng ticket] to request updating the kickstart file used to generate the nightly spin ISO's.
 +
 
 +
=== Comps ===
 +
A new comps file needs to be created for the next fedora release (the one after what we just branched for).  '''FIXME Link to comps SOP'''
 +
 
 +
=== Mock ===
 +
Mock needs to be updated to have configs for the new branch.  This should actually be done and pushed just before the branch event.  '''FIXME Link to mock update SOP'''
 +
 
 +
=== MirrorManager ===
 +
Mirror manager will have to be updated so that the yum repo redirections are going to the right places.  '''FIXME Link to MM SOP'''
 +
 
 +
<!-- This is commented out until a new one is written.
 +
== Getting a List of Unbranched Packages ==
 +
 
 +
After mass branching you may want to run sanity checks to see if there were packages that weren't successfully branched.  There's a script on the cvs server that can help you do this.  The script needs to be run first on the cvs server and then on a machine with the kojiclient libraries installed (your local workstation should be fine.).
 +
 
 
<pre>
 
<pre>
ssh db2
+
On cvs1:
sudo -u postgres psql pkgdb
+
CVSROOT=/pkgs/cvs cvs co CVSROOT
select p.name from package as p where p.id not in (select p.id from
+
CVSROOT/admin/find-unbranched cvs F-12 > unbranched
package as p , packagelisting as pl, collection as c
+
 
where c.id = pl.collectionid and p.id = pl.packageid and
+
On your workstation:
c.name = 'Fedora' and c.version = '9')
+
scp cvs1.fedoraproject.org:CVSROOT/admin/find-unbranched .
order by p.name \g /var/tmp/not-in-f9
+
scp cvs1.fedoraproject.org:unbranched .
 +
./find-unbranched compare F-12 unbranched
 
</pre>
 
</pre>
 +
-->
  
To get the final list of what needs branches, compare the koji output and if an entry from the koji output exists in the pkgdb output, it needs a branch.
+
=== Update critpath ===
  
== Notes on Optimization ==
+
Packagedb has information about which packages are critpath and which are not. A script that reads the yum repodata (critpath group in comps, and the package dependencies) is used to generate thisRead the [[Update Critpath SOP]] for the steps to take.
* If we can give a new flag to pkgdb's API to turn off mail on a per request basis we don't have to turn off pkgdb in order to branch.
+
* Try doing parallel pkgdb-client calls.
+
* If we could enhance the mail notification script for cvs to detect mass branching then we wouldn't need to close cvs during mass branching requests(Currently, the rate for this is 3 pkgs/minute purely for the cvs portion (pkgdb information is cached at this point.))
+
  
 +
== Consider Before Running ==
 +
* '''FIXME'''
 
[[Category:Release Engineering SOPs]]
 
[[Category:Release Engineering SOPs]]

Revision as of 20:06, 8 October 2012

Shortcut:
RSOP:MASSBRANCH

Contents

Description

At each alpha freeze we branch the pending release away from devel/ which allows rawhide to move on while the pending release goes into bugfix and polish mode.

Action

PackageDB

Mass branching in the pkgdb is the first step. It should be done near the time that the scm branches are created so as not to confuse packagers. However, it does not cause an outage so it should be done ahead of time.

First, someone needs to add the new release to the PackageDB. See this section

Then a scmadmin needs to either start the mass branching process from the command line or by pushing a button on a web page. The web page method is to browse to https://admin.fedoraproject.org/pkgdb/collections/ login, and then click the "Branch All" button.

The command line method is to run:

pkgdb-client mass-branch -b F-13

Whichever method you choose, the command will start the mass branch process on the server and then return with a message that the branching is in progress.

When the branching is finished, you will receive an email that tells you of any unbranched packages as well as all the packages that were branched.

If something fails spectacularly, it is safe to try mass branching again at a later time. If only a few cleanups a re needed it might be better to do that with the regular branch commands.

Warning (medium size).png
Target pkgdb-client against bapp01
At present there is a bug in the mass branching code so that the branching seems to work but the changes aren't saved in the database. I think this occurs when the server handles other requests while the packagedb is processing a mass branch. Performing the mass branch on bapp1 which is not handling other requests while the mass branch is occurring is one way to work around this. It's likely a bug in how forking in the mass branch code is interacting with the database connections.

Retarget pkgdb-client to use bapp1 by changing the value of pkgdb.url in either /etc/pkgdb-client.cfg or ~/.fedora/pkgdb-client.cfg.

[global]
pkgdb.url = "http://bapp01/pkgdb/"

Puppet

A couple files under puppet management need to be updated to be aware of a new branch.

pkgdb2branch.py

This file is used by an scmadmin to read data from pkgdb and create branches in the source control. Two parts need to be updated, one part that defines valid branches and what existing branch to create them from, and the other part defines a mapping of branch names in pkgdb to branch names in scm.

On the puppet server in a clone edit the modules/gitolite/files/distgit/pkgdb2branch.py file:

diff --git a/modules/gitolite/files/distgit/pkgdb2branch.py b/modules/gitolite/file
index ce79467..c40e83c 100755
--- a/modules/gitolite/files/distgit/pkgdb2branch.py
+++ b/modules/gitolite/files/distgit/pkgdb2branch.py
@@ -29,14 +29,15 @@ BRANCHES = {'el4': 'master', 'el5': 'master', 'el6': 'f12',
         'f11': 'master',
         'f12': 'master',
         'f13': 'master',
-        'f14': 'master'}
+        'f14': 'master',
+        'f15': 'master'}
 
 # The branch names we get out of pkgdb have to be translated to git
 GITBRANCHES = {'EL-4': 'el4', 'EL-5': 'el5', 'EL-6': 'el6', 'OLPC-2': 'olpc2',
                'FC-6': 'fc6', 'F-7': 'f7', 'F-8': 'f8', 'F-9': 'f9',
                'F-10': 'f10', 'OLPC-3': 'olpc3',
                'F-11': 'f11', 'F-12': 'f12', 'F-13': 'f13', 'f14': 'f14',
-               'devel': 'master'}
+               'f15': 'f15', 'devel': 'master'}

genacls.pkgdb

The other file is ran by cron that will read data out of pkgdb and construct an ACL config file for our scm. It has a section that lists active branches to deal with as pkgdb will provide data for all branches.

Again on the puppet server in a clone: modules/gitolite/files/distgit/genacls.pkgdb

diff --git a/modules/gitolite/files/distgit/genacls.pkgdb b/modules/gitolite/files/
index e531dc2..07b2ba7 100755
--- a/modules/gitolite/files/distgit/genacls.pkgdb
+++ b/modules/gitolite/files/distgit/genacls.pkgdb
@@ -22,7 +22,7 @@ if __name__ == '__main__':
     ACTIVE = {'OLPC-2': 'olpc2/', 'OLPC-3': 'olpc3/', 'EL-4': 'el4/',
               'EL-5': 'el5/', 'EL-6': 'el6/', 'F-11': 'f11/',
               'F-12': 'f12/', 'F-13': 'f13/', 'f14': 'f14/',
-              'devel': 'master'}
+              'f15': 'f15/', 'devel': 'master'}
 
     # Create a "regex"ish list 0f the reserved branches

Push the changes

When done editing the files, commit and push them, then restart puppet on the scm server in order to get the new files in place.

SCM

The following work is performed on pkgs01

Make git branches

Run pkgdb2branch.py to branch the repos on the scm server. The --branch-for option was designed with this use case in mind::

./pkgdb2branch.py --branch-for=f15

If for some reason that doesn't work, you can try this alternative:

cat pkglist.txt|./pkgdb2branch.py -c -

where pkglist.txt is a list of all the packages to branch.

Update ACLs

Although cron may have run, it is smart to manually run the cron job to make sure new ACLs are in place:

$ sudo -u jkeating /usr/local/bin/genacls.sh

Update preupgrade releases.txt

File a ticket with Infrastructure to modify the mirrors/releases.txt file to include pointers to the new release, for use by preupgrade. This file is in the fedora-web git repo. You will need to add something like:

[Fedora N ($codename)]
stable=False
preupgrade-ok=True
version=N
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-N&arch=$basearch
installmirrorlist=http://mirrors.fedoraproject.org/mirrorlist?path=pub/fedora/linux/development/N/Fedora/$basearch/os/

Update AutoQA repoinfo.conf

The AutoQA project maintains a config file (repoinfo.conf) that describes available package repositories and their inheritance. In order to have automated testing of Branched package builds, the repoinfo.conf file needs to be updated. Please file an autoqa ticket to modify the repoinfo.conf file to include pointers to the new release.

Koji

The koji build system needs to have some tag/target work done to handle builds from the new branch and to update where builds from master go. See the section on Koji in the Adding Build Targets SOP for details.

Fedora Release

The Fedora release package needs to be updated in both the new branch and in master. FIXME Link to fedora release bump SOP

Bodhi

Bodhi needs to be turned on for the new branch. Instructions in the Bodhi SOP

Enable nightly branched compose

A cron job needs to be modified and turned on for the new branch. FIXME Link to nightly branched SOP

Update kickstart used by nightly live ISOs

On a nightly basis, a live ISO image is created for each spin and hosted at http://alt.fedoraproject.org/pub/alt/nightly-composes. The yum repositories used by spin-kickstarts need to be updated to use the branched repository. Please file a rel-eng ticket to request updating the kickstart file used to generate the nightly spin ISO's.

Comps

A new comps file needs to be created for the next fedora release (the one after what we just branched for). FIXME Link to comps SOP

Mock

Mock needs to be updated to have configs for the new branch. This should actually be done and pushed just before the branch event. FIXME Link to mock update SOP

MirrorManager

Mirror manager will have to be updated so that the yum repo redirections are going to the right places. FIXME Link to MM SOP


Update critpath

Packagedb has information about which packages are critpath and which are not. A script that reads the yum repodata (critpath group in comps, and the package dependencies) is used to generate this. Read the Update Critpath SOP for the steps to take.

Consider Before Running

  • FIXME