This page is to track a proposal on changing how we have our branches structured currently in dist-git
- 1 Email Proposal
- 2 Impact
- 3 Testing
- 4 Timeline
- 5 Changeover Execution
- 6 Proposal Owner
I'm working on fixing a few long standing bugs in fedpkg that have to do with our branch structure (https://bugzilla.redhat.com/show_bug.cgi?id=619979 and https://bugzilla.redhat.com/show_bug.cgi?id=622592). This has me examining our branch structure again and trying to remember why exactly I chose it (obviously I did a poor job of documenting that...). The original thought was to have top level branches that are named after distribution releases, eg "f14", "f15", "el5". Then we would force branches of those branches use a naming structure of "f14/topic". The reason was so that our tooling could look at the name of the branch and easily work back to the "f14" part. This would work even if it was "f14/user/fred/topic/mybranch" or other such craziness. When I went to test this, I realized that git won't allow you to have both "f14" and "f14/topic" as branches, because of the way the git metadata works on the filesystem. When I encountered this, I made "f14/master" become the top level branch, and then "f14/somethingelse" could coexist. Unfortunately I also wanted to keep things easy for users and tried to maintain tooling that would allow you to just say "f14". This didn't get enough real world testing and in hindsight was a bad idea. Things go wrong quickly in git if your local branch name doesn't match the remote branch name. When thinking about the above, and the two bugs I'm working on, I realized that we don't have any real strong need to be using "/" as a delineator. It makes some code easier, but makes other things more complex and difficult. So what if we changed it? What I'm thinking about now is switching from / to - as a delineator. This would improve a couple things. First, we could achieve upstream top level branch names that are short and simple: "f14", "f15", "master". We can have branches that build upon those names: "f14-rebase", "f15-cve223", "f15-user-jkeating-private". We could keep the simple fedpkg tooling that allows users to just type "f14" and the like to reference a branch, and now the local branch will match the name of the remote branch. As for the first two bugs I mentioned, it doesn't directly help them. However I would feel better about telling people that their local branches must follow a naming scheme of <release>-<something> and then we could easily guess what release the local branch is for if it isn't tracking a remote branch. However the bug about what to do if there are no remote branches is really not touched by any of this, it just got me thinking about branches :) What kind of user impact would this have? My hope is that the impact would be minimal. Git allows branch renames, and can successfully rename "f14/master" to "f14". All the history is renamed. We should be able to do this without an outage of the git server. The ACL system will need a slight tweaking, and a regeneration of the ACLs but that is fairly quick and minor to accomplish. However there will be some issues client side. We will not be able to make use of git's symbolic-ref feature of "aliasing" a branch. We cannot make "f14/master" an alias for "f14", again because of the filesystem layout issues. These issues rear their head once again when a client does a pull of an already checked out repo that had branches. Because there was already a f14/master, when the client sees a new branch just named "f14" it will fail to create it. Git has a command that will fix this "git remote prune origin". That will remove the local reference to f14/master and a subsequent pull sees the creation of the "f14" branch happen successfully. However, if a user had a local branch of f14 or f14/master they will be left with mismatched .git/config entries. In this case it's easiest to delete the local branch (git branch -d f14) and check it out again. If there are changes on the branch one can fix the config to point it to the right upstream location. Also we would need to get a new fedpkg into the hands of all the developers that handles the new branchnames. We could do a build that handles both the oldnames and the new and have it out and available for a reasonable period of time before we make the switch. So, some pain, for some pretty good gain. This time around I can setup pkgs.stg with this branch configuration and builds of fedpkg to use it for a more through testing before rolling it into production. So please, tell me what you think!
This change will impact a few things.
Various changes would happen on the git server
The git server is where we would process all the rename jobs. A script needs to be written in order to discover and rename the branches appropriately.
Some care will have to be taken when there exists release branches beyond f14/master. The other branches will need to get renamed to f14-<branch> before finally renaming f14/master to f14.
The ACL system
The ACL system will need to be tweaked to remove the trailing "/" from branch path names in the ACL files.
We can also modify the ACL system to detect push attempts that include a path with "<branch>/<something>" and instead of generically erroring out to the user we can print a link to a wiki page explaining the migration and what the end user must do.
Branch Creation Scripts
There are a couple branch creation scripts that will need to be updated to reflect no longer having "/master" in branch names.
The branch naming comes up surprisingly few times in fedpkg code. It is fairly trivial to go from "<foo>/master" to just "<foo>". The harder part will be putting in code that handles both scenarios at the same time, so that we can roll the new fedpkg into place well ahead of the git server work.
Currently we are using pkgs01.stg.phx2.fedoraproject.org as a testing environment for changing the various server side elements listed above with client side changes. At this time it is not appropriate for public testing, nor do we have a fedpkg build for client testing.
The test plan would be to sync pkgs.stg.fedoraprojec.org with some of the production pkgs.fedoraproject.org and then convert all the branches and scripts on pkgs.stg. Then provide a build of fedpkg that is configured to use this host and configured to use the new branch names to interested testers. Feedback will be processed and things potentially tweaked until we feel that the setup is sufficient.
We would like to complete the migration prior to or just after FUDCon Tempe 2011, which means late January 2011. Date subject to change as this has not been proposed to FESCo yet.
Here is a quick todo list:
- Investigate changes requires a bit more using pkgs.stg -- IN PROGRESS
- Propose change to FESCo -- DONE and APPROVED
- Setup pkgs.stg for widespread testing -- DONE
- Publish a build of fedpkg appropriate for testing -- DONE
- Apply ACL patch to warn users on push attempt to old branch path -- DONE (locally, not in a package)
- Write client side script to update clone metadata -- DONE
- Solicit testing and feedback -- DONE
- Prepare production build of fedpkg -- DONE
- Submit updates of fedpkg -- DONE
- Prepare patch for genacls.sh to support new branch style -- IN PROGRESS
- Schedule time for production changeover of pkgs.fedoraproject.org -- DONE set for May 10
- Execute changeover
- Prepare and publish another build of fedpkg that removes code to deal with old branch style
On May 10 we will execute the changeover. Here are the steps involved:
- Start an outage of the git server
- Run rebranch.py
- Update gitolite to our custom build
- Update genacls.pkgdb for new branch style
- Update the reposconverted file on jkeating.fedorapeople.org
- End the outage
Start the Outage
First make sure an outage announcement has been sent!
All pushing makes use of the gitolite ACL system. To enact an outage, gitolite must be configured to deny everything. This can be accomplished by having
repo @pkgs R = @all
after the pkgs definition, and nothing else in the
gitolite.conf file passed to gl-compile-auth.
There is a cron job that will overwrite this though, thus the cron job must be disabled. As jkeating, run
crontab -e and comment out the cron job:
# Puppet Name: genacls #0,10,20,30,40,50 * * * * /usr/local/bin/genacls.sh &>/dev/null
Puppet will overwrite this though so we must also disable puppet for the duration of the outage. This is accomplished by touching
/var/lib/puppet/state/puppetdlock This will prevent any puppet runs from happening.
The outage is now in effect.
rebranch.py is a script written to walk the directory of git repositories and rename the branches as necessary. This is a multithreaded script although it will be limited by disk I/O. In testing it took roughly 35 minutes to complete. Currently rebranch.py lives on Jesse Keating's laptop, although there is a copy of it on pkgs01.stg. It will be uploaded to jkeating's homedir on pkgs01 before the outage.
Running under the "rebranch" screen of jkeating
A custom build of gitolite has been created that will watch for push attempts to the old style branch paths and direct users to Dist_Git_Branch_Redux. gitolite-1.5.7-2.1.el6.fedora.noarch.rpm currently exists in the infrastructure staging repo, and may be moved to the production repo during the outage so that it can be installed on pkgs01.
An update to genacls.pkgdb has been committed to the staging puppet repo. When the outage begins, this change can be cherry-picked to the production branch. However puppet is disabled on pkgs due to the outage. The patch can be manually applied to pkgs in order to generate a new set of acls. Once the patch has been applied to
/usr/local/bin/genacls.sh to generate a new set.
There is a file at fedorapeople that tells fedpkg whether or not the repos have been converted. Edit this file to say
True once the repos have been converted.
End the outage
The outage can now be ended. Replace gitolite.conf with a backup (or a newly generated copy) and re-run gl-compile-auth. Re-enable the cron job for this. Unlock puppet.
Once these things are done and the system is tested, an email announcing the end of the outage can be sent.