Fedora Slipstream
From FedoraProject
Contents |
Prereqs
- kopers need their own builders to protect against badness when installing pkgs for builddeps into buildroots
- this now makes fedorapeople more of a critical piece of infrastructure - tied to our build infrastructure in many ways - we may be better off hosting this ELSEWHERE for the actual repos hosted
Where to start
- get a box setup to run as a kojihub/kojidb
- get a koji builder box setup
- get a kopers storage space setup
- work on making a koji scratch build instance be able to handle building pkgs
from arbitrary locations below the baseurl of the kopers storage space + fedora repos
- the above both defined as external repos
- tool to submit an srpm to the above (using their fedora certs, etc) and specify any targets/repos it should use
- tool sucks down resulting rpms
- prompts the user to sign them (or not)
- puts them into the repo they are being built for
Scripts to start on
- sync-to-repo: takes a dir of localpkgs and remote read and write path
- syncs down the repodata from the remote read path and merges the localpkgs into that path and pushes the data back up via the write path.
- I make a srpm of what I want
- it goes to koji for whatever build target
- getting koji to refer to arbitary other locations needs to be thought of here
- Really build target expansion? F-12 + updates + bob-kopers
- those pkgs (if any) come back and I make a repo out of them
- as part of this, create an rpm for the repo that people can install
- the repo shows up on fedorapeople
- .... profit?
- Add static repo list (cron job - like with public git repos)
Design work
KDE RedHat
kde-redhat is a large external repo, ask rdieter how he manages it so far.
All I'm doing really is building stuff, either in koji or my own mock instance. Grabbing those rpms, signing them, stuffing it into repos. Whatever kopers end up being, can't be much worse than what I'm doing now. honestly.
- repos managable by > 1 person (ie, a sig)
- custom buildroots (ie, to use itself or other kopers). we've had repo heirarchies do something like that already kde-unstable depends on kde-testing depends on kde-stable
- easy mirroring, I suppose, would be nice too
- not sure what kind of workflow to get packages into kopers would look like. something bodhi like, or maybe doesn't even have to be that complicated
- maybe we could get some inspiration from how the opensuse buildservice works though I've only spent a couple of hours looking at it.
imo kde redhat is beyond the scope of kopers - it's WAY beyond 1GB per user - skv
Koji Personal Repos
Overview
Koji Personal Repos (KoPeR) is a concept to allow Fedora maintainers to create personal repos of packages to build against and share with others.
Usage Case
- Bob would like to try out a new version of GTK2. Bob does a koper build of the new GTK2 package, and then builds a number of other packages against this. Bob shares his gtk2newness koper with others for validation of the new GTK2
- Jill would like to experiment with rpm using a different compression method for packages. Jill builds a modified version of rpm for her rpm koper and builds a number of other packages with this rpm. Jill shares her rpm koper for further testing of the rpm change.
- Dave would like to experiment with kernel patches and have the repo creation handled for him. Dave continues to add patches to kernel source rpms and builds them for his koper(s). Dave shares his various kernel kopers with the community for early testing and validation. Dave spends more time working on kernel and less time dealing with repo management.
- Fedora Infrastructure would like to create a repo with updates to RHEL packages that it needs in order to manage its systems. It would like other people to be able to use the repos so that they can replicate their environment.
- The KDE SIG wants to have a repository where they can perform major upgrades that their users want but don't follow the Fedora update guidelines. By publishing to a kopers repo the packages don't have to go into the main Fedora repositories but they are able to point to it for people who want to run later versions.
- The KDE SIG wants to test their updates for a Fedora release as a set so they can find regressions between the components and the release. They publish the packages to a kopers repo and have their testers pull updates from there.
Resources
A Fedora maintainer would be alloted up to 1GiB of storage for package repos. Packages can be built for any arch Fedora builds for (but doesn't have to be all of them). koper repos would have repoview content for easy browsing and subscribing to changes.
Work Flow
- Bob requests his account becomes koper capable.
- this seems like a non-issue - but it may mean something for koji
- Bob constructs an srpm of gtk2
- not kopers issue
- Bob issues a command (koper build "gtk2newness" gtk2.src.rpm ?) to build the gtk2 package.
- first this checks the pkg and looks at the License tag to make sure it's not something we cannot trust.
- this just calls koji, ultimately
- Notes about building from SCM rather than SRPM; SCM server needs to be accesible from builders (currently, within Fedora's network)
- Building from SCM is not necessarily as important as being able to share changes via the SCM. if I can clone the SCM repo and you can as well to make changes but we create an SRPM as an intermediate step for build that's not much of a burden.
- Koji creates a buildroot based upon the rawhide collection and any packages in Bob's "gtknewness" koper.
- two issues here: getting koji to use arbitrary repos
- not having the above beat the crap out of fedorapeople with koji constantly hitting it for pkgs/repodata
- Koji builds Bob's gtk2
- $TOOL takes the koji built package and adds it to Bob's "gtknewness" koper, recreating yum repodata and repoview content.
- this means xferring the file back to the system or getting koji to build repodata
- need something in here for SIGNING those pkgs, too.
- Probably want a separate key for each kopers repo.
- why a separate key? if I'm building these for ME and mine there is no reason why I shouldn't use the same key
- Repos can have multiple people adding to them so there needs to be one key for it.
- you could use the same key for all repositories that only you contribute to but then we'd need to keep track of two different types of repository
- Bob proceeds to create srpms for various other packages and builds them in koji against his "gtknewness" koper.
- having koji know the repo exists or to have it know when the repo no longer exists.
- Bob shares his "gtknewness" koper with the world
- this is just advertising a url
- Bob creates a Desktop Live image based on rawhide and his "gtknewness" koper and shares that with the world too.
- Bob fixes bugs with his gtk2 and continues to do builds until everybody is happy
- Bob commits his gtk2 changes to package SCM and does a normal koji build.
- Bob profits.
- These last 4 items don't matter at all they are outside of scope, imo.
Content
Content within ones koper(s) must be content acceptable for distribution within Fedora.
- Manual testing, reaction is one possibility
- Review process for not-yet-in-Fedora-items (just license review probably all that's needed)
Changes Needed
To create a koper system various tools in the Fedora infrastructure would need to have changes made to them.
Koji
In order to build packages against a koper, koji would need to gain the ability to reference external yum repos for (at least) scratch builds.
fedorapeople.org
In order to support kopers, fedorapeople.org would need to be able to:
- Track a separate quota for kopers
- Accept a signal to recreate repodata and repoview content in kopers
Trademark Guidelines
We'd like to enable people to make development like spins of Fedora that include content from kopers. Since all content in kopers has to adhere to Fedora's guidelines it should be safe. We just need terms in the trademark that allow for this kind of spin creation and sharing so that these people don't have to spend time re-branding and can focus more on development.
New Tools
A few new tools might be needed for dealing with kopers.
Tool to build
A script or make target would be needed to initiate a koper build. This is necessary to tell koji what koper to build against, as well as to get the results of the built added to said koper.
Tool to add to koper
A tool would be needed to take the koji scratch build output and add it to ones koper, signaling fedorapeople.org to create repodata and repoview content. It should be able to replace older builds of the same package to save quota space. This tool could also be used to add already built packages to an existing koper without having to go through a koji build.
Version 2.0
Here are some thoughts for future enhancements once we prove that we can walk.
- koper groups. Be able to have a group koper that multiple people work on. This should be made as easy as possible to get the most people as possible working together. Start with the idea of open. Extremely useful for feature driven development.
- koper directory. Have a page off of fedorapeople.org that lists all the (active?) kopers. Easy to walk the filesystem looking for */koper/*. Read from .description files in koper directories for text to display in the directory
- Autogeneration of koper-repo packages for end users to easily install to use said koper. (or maybe register .repo files with pirut to process?)
Comments
If you wish to leave a comment about this concept, you can do here. I'll try to address them as I can.
- MatthiasClasen: The examples make it appear that this is mostly geared towards replacing Fedora packages with own builds. I'd like to see some explanation or guidance on how this will work wrt to versioning (do we require a mandatory suffix for the release, similar to disttags ?). How do we address the concern that bugs reported against such private gtk2 builds will end up in my bugzilla queue ?
- JesseKeating: The same things we have in place today for these kinds of things... not much (to begin with). However it really boils down to a consumer education kind of thing, educating the consumers of kopers that they're keeping all the pieces if it breaks and whatnot. The packages obviously wouldn't be signed by a Fedora key so that could be one method of discovery.
- User:till: Which key will be used to sign the packages or is it planned to not sign them at all?
- User:jkeating: They could be signed with the user's key. Since a user might be able to stuff anything they want into their own repo, I wouldn't feel all that comfortable signing it with a "koji" key. That should be reserved for builds that are produced from repos that koji has ultimate control over.
- User:till: Which key will be used to sign the packages or is it planned to not sign them at all?
- JesseKeating: The same things we have in place today for these kinds of things... not much (to begin with). However it really boils down to a consumer education kind of thing, educating the consumers of kopers that they're keeping all the pieces if it breaks and whatnot. The packages obviously wouldn't be signed by a Fedora key so that could be one method of discovery.
- RussellHarrison: This would be very nice for new packages or packagers. The review process could be streamlined by running some automatic tests (such as rpmlint) on the packages as soon as they're built. Also the ability to use external yum repos would make it much easier to setup a private koji system for making derivative Fedora derived distributions. Say a company internal release or a new live CD.
