Infrastructure/VersionControl/dist-hg

Some notes regarding using mercurial as a replacement for CVS

= Testing dist-hg = We are now ready for testers to take part in dist-hg evaluation. Please let JesseKeating know if you're going to start testing.

Tools Needed
Install  and   from Fedora Extras. Next get a clone of disthg-utils from the test server hg clone http://publictest1.fedora.redhat.com/disthg-utils Put the  script somewhere in your $PATH

Download files from http://people.redhat.com/jkeating/dist-hg/ and put them in your home directory with a '.' in front of them.

Then email JesseKeating for an account on the build system.

How to test
Use  to fetch a patch checkout from the dist-hg server. We'll use mock for an example: This should populate a mock/ directory tree in your current working directory. Currently the only configured build target is fedora 6, so building from the FC-6 branch. Here you can make changes to the spec file, bump the revision, etc... when you're ready to commit and push changes to the server: Next,  and   will send the job off to the build system. should give you status information of your build.

Currently the plague instance will not email you anything. If somebody wants to configure that part of it, please let me know.

= Conversion =

Tools Used
To convert dist-cvs to HG repositories, I make use of:
 * cvsps - In Extras.
 * cvs-import - http://hg.beekhof.net/hg/cvs-import

Layout
The current layout I'm using is rougly the same as dist-cvs:. I've omitted the rpms/ dir for now, as it may become an alias to getting all rpms. With CVS, the package dir was basically the module, and each release directory was just a directory, even though we call it a 'branch'. For HG, I've made it so that each release directory is it's own repo. This allows for "branch" confined logging and history, as well as making it easier to setup ACLs. Since hg doesn't allow a repo of repos very well (forest plugin didn't pan out), each package 'module' is a repo itself holding a file named 'branches' that is a list of all the release branches that exist for that package.

How to Convert
I had /cvs/extras restored from backup to the xen host under /srv/extras.

The conversion tools need both the CVSROOT __and__ a checkout of the content to do proper conversion. I setup a directory structure as such: dist-cvs/ dist-hg/ checkout/ cvs-import/


 * dist-cvs: holds CVSROOT and the rpms/ trees from the CVS server
 * dist-hg: landing zone for converted modules
 * checkout: scratch space used for cvs checkouts and conversion work files
 * cvs-import: hg checkout of the cvs-import utility

Basic workflow: cd checkout cvs -d ~/dist-cvs co rpms/ cd rpms/ / TZ=UTC cvsps -u -A -q > ../../../cvsps.dump cd ../../../ mkdir -p ~/dist-hg/ / TZ=UTC ../cvs-import/hg-cvs-import ~/dist-cvs/ rpms/ / ~/dist-hg/ mapfile < cvsps.dump mv ~/dist-hg/ / -HEAD/.hg* ~/dist-hg/ / / echo " " >> ~/dist-hg/ /branches rm -rf ~/dist-hg/ -HEAD* rm mapfile One would loop through each releasedir of each package (or at least devel/ FC-6/ FC-5/ FC-4/ FC-3/), hg init the and hg add/commit the branches file. I've converted all of extras package CVS as well as the common module.

Script used to convert: for dir in ; do mkdir /srv/hg/extras/${dir}; cvs -d /cvs/extras/ co rpms/${dir}; for branch in rpms/${dir}/{FC-[3456] ,devel}; do echo  >> /srv/hg/extras/${dir}/branches pushd ${branch}; TZ=UTC cvsps -u -A -q > ../../../cvsps.dump; popd; TZ=UTC ~/sources/cvs-import/hg-cvs-import /cvs/extras/ ${branch} /srv/hg/extras/${dir} mapfile < cvsps.dump; mkdir /srv/hg/extras/${dir}/ ; mv /srv/hg/extras/${dir}-HEAD/.hg* /srv/hg/extras/${dir}/ /; rm -rf /srv/hg/extras/${dir}-*; rm mapfile cvsps.dump; done; pushd /srv/hg/extras/${dir}; hg init; hg add branches; hg commit -m 'add the branches file'; rm branches; popd; rm -rf rpms/${dir} done

Time needed to convert: real   155m0.948s user   84m18.080s sys    41m5.246s

Space after conversion:

= Server Setup =

File system and Permissions
Mercurial can be accessed via http or ssh. In my test server, I created /srv/hg to serve up the repositories. I created a group 'fedora' and set group sticky and write bits on /srv/hg.

Web Configuration
HG web configuration is fairly simple. I've setup /srv/hg/ to be a hgwebdir, in that the repos within /srv/hg can be listed and browsed via http. This is controlled by two files, /srv/hg/hgwebdir.cgi and /srv/hg/hgweb.config. hgwebdir.cgi is a python cgi that calls hg to serve up the repos. hgweb.config is a mapping of names to actual repos paths. Currently I'm listing everything in /srv/hg as a repo by the directory name it has. This gets a bit noisy as each / is listed. URL to view is http://publictest1.fedora.redhat.com One would assume that eventually this would get aliased as say hg.fedoraproject.org/fedora/ or packages/ or something similar.

= Using dist-hg =

Anonymous Access
Anonymous access is achieved via the http interface. hg clone http://publictest1.fedora.redhat.com/extras/mock/fc-6 mock-fc-6 This would "clone" the FC-6 "branch" of mock as the local 'mock-fc-6' directory. This repo can then be used locally.

Write Access
Write or Commit access is achieved via ssh. hg clone ssh://publictest1.fedora.redhat.com//srv/hg/extras/mock/fc-6 mock-fc-6 This would "clone" the FC-6 "branch" of mock as the local 'mock-fc-6' directory. This repo can be used locally until changes are happy, then 'hg push' would commit changes back to the server.

To checkout a package and all of its branches, I have cobbled together a method that uses a 'branches' file to iterate over. In there is  that has one line per release "branch". This script will pull down the set:
 * 1) !/bin/bash

HGSERVER="ssh://publictest1.fedora.redhat.com//srv/hg/extras"

hg clone ${HGSERVER}/${1}

for branch in do hg clone ${HGSERVER}/${1}/${branch} ${1}/${branch} done

hg clone ${HGSERVER}/common ${1}/common Obviously this is a bit rough, but it helps to do the proof-of-concept. Eventually a plugin or something could be written for hg that will do this automatically. Call it with the package name as an argument. This also pulls down the common repo. Eventually script could be modified to take a list of 'branches' to pull down as well, instead of all.

= Work Needed = These are some of the high level items that need to be investigated or implimented. If you wish to help, please contact JesseKeating.

Seems a bust right now. Doesn't integrate into web view, nor seems to work with remote cloning. Most make targets have been converted and tested. Currently done for cvs with /cvs/extras/CVSROOT/tag-check for CVS. Most plague work done, able to pull and build a package from dist-hg
 * Integration of mercurial plugin for a repo of repos - hgforest
 * Makefile changes for hg commands
 * Tagging hook to ensure unique tags across an entire
 * ACL framework
 * Updated scripts for importing srpms (creating new )
 * Updated scripts for "branching" for a release
 * Integration of email notification scripts Notify Extension
 * Modify plague to be able to pull an hg tag to build from.
 * Configure plague to send emails regarding job completion/failure