The following document explains the key components in Fedora Devshell, and how to extend it for your own nefarious purposes. The Modules section addresses the kinds of actions Fedora Devshell can do, and how they are implemented. The Utility functions highlights some of the key components used as building blocks. The Coding Guidelines are critically important, as they must be followed to in order to submit patches. The Submission Guidelines explain how to get your patches in.
- 1 Modules, Interfaces and Base Classes
- 1.1 Directory
- 1.2 Package
- 1.3 BuildSystem
- 1.4 PackageSource
- 1.5 Fetcher
- 1.6 Port
- 1.7 Build
- 1.8 Mock
- 1.9 Profile
- 2 Utility Functions
- 3 Coding Guidelines
- 4 Submission Guidelines
Modules, Interfaces and Base Classes
Directory is a Base Class for pretty much any class needing persistence on the filesystem. It manages storing metadata attributes in a simple INI-style file under .devshell in each directory. It establishes the link between some directory on the file system and the operations available from the command line, inside the source code, and from any IDE. The initializer takes one optional parameter, a path to the directory on the file system, and it loads up the
Directory from there. If not given,
Directory uses the current working directory.
It exposes these methods:
- is_sysdir_dir(dir) - given some directory, it determines if this is a
Directorymanaged by Fedora Devshell
- load_dir(dir) - a hook to be run when loading some directory that has already been created. do not call this directly, although feel free to override it in subclasses. make sure to call the parent method
- make_dir(dir) - a hook to be run when creating a
Directoryout of some new directory. do not call this directly, although feel free to override it in subclasses. make sure to call the parent method
- close() - a hook to save persisted information when Fedora Devshell exits. safe to call directly, the results should be idempotent. feel free to override it in subclasses, make sure to call the parent method
- rename(new_name) - a hook that renames the directory, assuming it's been changed on the file system to match already. feel free to override it in subclasses, make sure to call the parent method
- move(self, new_loc) - moves the directory to a new location, and calls rename
- copy(self, new_loc): - copies the directory to a new location and calls rename, returns the new
- name - the name of the directory on the file system
- dir - the absolute path to the directory on the file system
- parent - the absolute path to the parent of the directory on the file system
This metaclass registers all directory objects with DirFactory
A subclass of
Directory that will return the correct subclass of
Directory for a given directory on the filesystem, rather than the base class.
Package is an overarching manager for packages that will eventually become RPMs. There should be a one-to-one correspondence with
Package and a source RPM. There will also theoretically be a one-to-one correspondence between
Package and entries in the Fedora CVS repository. It manages a spec file, a list of sources, patches for the actively chosen
PackageSources and other administrative details.
It exposes the following methods:
- add_spec(spec_file) - given some spec file, set the current spec file to be this one.
- get_srpm_name(profile) - given a profile, determine the name of the source RPM
- ver(profile=None) - given a possible profile, determine the upstream version from the current spec file
- copy_source(source_dir) - given some
PackageSource, copy it into this
Packageand add it as a source
- move_source(source_dir) - given some
PackageSource, move it into this
Packageand add it as a source
- add_source(source_dir) - given some
PackageSourcedirectory, assuming it is already inside the
Packagedirectory, add it as a source
- rem_source(source) - remove a
PackageSourcefrom the source list, without deleting it from the file system
- del_source(source) - delete a
PackageSourcefrom the file system and remove it from this
- fetch_sourceballs(profile=None) - with an optional profile for version information, fetch all the source tarballs from the
PackageSources belonging to this package. To do so, it creates symlinks to save space
The following properties:
- spec_file - the name of the current spec file
- pkg_name - the canonical name of the package in the Fedora CVS repository
- sources - a list of the
- sourceballen - a list of all the source tarballs that have been fetched at some point
Possible Package Types
Packages can be managed in all sorts of ways. Fedora currently uses CVS to manage the workflow. Other people might want to use
git. In the future, we will need modules to manage these different workflows
BuildSystem is a Python class of type
MetaBuildSystems are registerd with a
BuildSystemFactory, which can load any
BuildSystem loaded in the run time space. It's initializer takes one argument, a
PackageSource either on the file system or a Python object.
BuildSystem is an interface on top of common build systems such as autotools, cmake, and cabal.
It exposes five methods.
- configure(target, *args) - runs the configure stage of the build system - can be a noop. target points to where to install it on the system, *args are passed to the underlying package source to work on a specific branch.
- build(*args) - runs the build stage - should not be a noop. *args are passed to the underlying package soruce to work on a specific branch.
- install(*args) - runs the install stage - cannot be a noop. *args are passed to the underlying package soruce to work on a specific branch.
- install_source(target, *args) - configures, builds, and installs some source. target points to where to install it on the system, *args are passed to the underlying package source to work on a specific branch.
- gen_spec() - creates a spec file specific to this build system
Cabal is an implementation of a
BuildSystem. It works on top of The Haskell Cabal, which is the build tool used for building Haskell packages. It uses the program
cabal2spec provided in Fedora for generating the spec file.
Possible BuildSystem Types
Currently we only support
Cabal. Plugging in other build systems should be easily feasible, possible options include Autotools , Ant, Python's setuptools, Ruby Gems, CMake, and many other build systems. We may want to look into handling multiple build targets for cross compiling.
PackageSource is a Python class of type
MetaDirectorys are registered with a
DirFactory, which can load any directory from the filesystem as the properly subclassed
Directory in Fedora Devshell.
PackageSource is a single source, be it a tarball or revision control that will go into an RPM Package.
It handles the management of patches between the original upstream source and the downstream sourcetree, generating tarballs and patches, Each implementation manages the storage and generation of these details.
It exposes these methods.
- setup_sourceball(ver=) - sets up a source tarball if needed out of the revision control. ver is the upstream version
- setup_sourceball_w_patches(ver=) - not sure of the behavior of this one yet, it's a placeholder
- src_dir(*args) - a contextmanager that will run a block of code inside a specfic branch or checkout
- src(*args) - a context manager that will run a block of code with a certain branch or checkout set as the active one.
- set_current_src() - a hook to be implemented by the subclass, notifies the subclass when the active source has changed and runs any arbitrary code.
- set_cur_to(*args) - where *args is processed. This parses *args, and sets the source to some source directory accordingly. Must be overridden in a subclass
- branch(*args) - returns the relative path to where a named branch is stored, based on *args.
- branch_dir(*args) - returns the absolute patch of
It exposes these properties
- pkg_src - a relative path where internal files to devshell are stored inside the package source
- pkg_src_dir - the absolute pathname to pkg_src
- branches - a relative path to where branches are stored inside their package source
- branches_dir - the absolute pathname to branches
- sourceball - a relative pathname to the current source tarball equivalent to this package source
- sourceball_loc - the absolute pathname to sourceball
- source - a relative pathname to the currently active sourcetree
- source_dir - the absolute pathname to source
- buildsystem - the type of buildsystem that this sourcetree uses
- builder - a reference to a
BuildSystemthat can build this package
SourceBall is an implementation of
PackageSource. It maintains both an actively edited work tree, and a copy of the original source tree, along with the original sourcetarball for comparison. It can only generate a single stack of patches, on top its one branch, 'orig'.
- *args - can either be 'head' or 'orig'.
PackageSource for all implementations that use
RevisionControl. This is a very boring class.
The only implementation of
RevisionControl. It wraps around the Darcs Distributed Version Control System. It maintains multiple branches based on parameters passed directly to
darcs. It does not yet do patch management.
- *args - can be one of several possibilities.
- beings with 'new' <new-branch-name> - creates a new branch by that name
- 'head' - the top level active source tree
- 'branch' <branch-name> - switches to a named branch
- anything else - passes these parameters to darcs and creates an unnamed patch after the hash of the arguments
new foo branch bar - forks bar as foo new foo --tag 0.8 - forks tag 0.8 as foo --tag 0.8 - switches to tag 0.8 --to-match 'hash 4593894085394890' - switches to some patch identified by some hash
Possible PackageSource Types
SourceBall are the only supported types.
Darcs does branching fundamentally different from Git, Bazaar, and Mercurial. Work will need to go into having the paradigms sit together nicely. Note that we aren't creating an all in one solution, but just using the important bits of each VCS to generate patches to go into RPMs. Ultimately, RPMs must be packaged with Tarballs, the use of
RevisionControl is only to ease and aid the process.
Provides an upstream source for multiple packages. Exposes:
- latest_version(pkg) - returns the latest version of some package by name
- url(pkg, ver) - given a package and version, return a URL where source can be found, namely source control and/or a tarball
- latest_url(pkg) - shortcut function, given a package, return a URL to the latest release
An implementation of
Fetcher for downloading packages from HackageDB - The Haskell Package Repository
Possible Fetcher Types
There are many sources for packages, including Sourceforge, Launchpad, Fedora Hosted, CPan, Python Cheese Shop, and many many others. Implementing
Fetchers is easy, so lets see some more.
Port is analogous to a BSD or Gentoo Port. It combines the above components into a single Module that has an intellegent overview of the whole situation. It can be used to download, setup, configure, install, and package any type of package, so long there is a port to handle it.
It currently exposes:
- close_later(directory) - given some
Directoryobject, mark it for closing later.
Ports may come across some
Directoryobject it does not own, that it will pass on to the outside world. This marks it for deletion once the system is done running. This ensures that all changes to the
NOTE: This will very likely turn into a utility function later on.
This will download any package that is hosted on HackageDB or at darcs.haskell.org, test build it, install it, and generate spec files for it. It combines
Possible Port Types
Theoretically, there could be one port for every package that is being managed. Many do have in fact their own idiosyncronacies that do need to be managed. Despite that, many components in packages are standardized out of other common systems. One goal of ports is to be able to use as much generic code as possible. For example, a KDE port might use a KDEFetcher, CMakeBuilder, SVNSource, and
SourceBall. We may want to look into other ways to declaratively define
Ports, and even include them as a repository somewhere.
A wrapper for
rpmbuild. It subclasses
Directory, and takes an rpmdev-tree (such as ~/rpmbuild, or /usr/src/redhat) as the directory. It can then perform common
rpmbuild operations on
- setup_source(package) - for some
Package, import the spec file, tarballs, and patches into the rpmdev-tree using symlinks to save space and time
- build_quick_rpm(package, profile=None) - given some package, and some optional profile, build a quick rpm using the current system as the buildroot. NB: This package should never be shared publically, it's only good for testing
- build_source_rpm(package, profile=None) - given some package and some optional profile, build an SRPM
- rpmbuild(param, package, profile=None) - runs
rpmbuildwith some parameter, on some package with some optional profile, worker function for
- fetch_rpms(target_dir) - fetches all the RPMs from the rpmdev-tree to some target directory, by moving them. should be run after every invocation of rpmbuild
- fetch_build(package) - given some package, fetches the expanded, possibly built source tree from the rpmdev-tree, and deposits it in the package. this is for doing error debugging, test, and verification
Mock is a simple subclass of a Module, it is merely a worker module with no state of its own. It uses a
Profile for all the stateful bits. It does one thing simplemindedly, it builds a source RPM in mock given some profile and rpmdev-tree for doing temporary work. The initializer takes a
- build_rpm(package) - given some package, use the
Profileto build it cleanly in
Possible Mock Types
Mock is a bit specific purpose. We may want to define a better clean room type builder. There are many technologies for distributing and cleanly building packages, including
Mock, Koji, and many other frameworks. We may want to create a generic RPMBuilder, that
Mock is an implementation of. For example, via a single switch, we could send a package to
Mock on a local system, do a scratch build in Koji, or inject it into a VM that can be sent across a grid using Condor.
Profile is a profile to be used in
Mock. It manages seperate repos of packages built in mock before to be used as a base, version information, vis-a-vis which version of Fedora to use, and other important information about the build profile. It uses its own directory store mock configurations and results from mock.
It exposes the methods:
- configure_from_system(branch) - given some branch (the same as in Fedora CVS), make a new profile based on the local system mock configurations (in
/etc/mock) in the directory.
- dist - eg .fc7 or .olpc2 used in rpmmacro %dist
- distvar - eg: fedora or rhel
- distval - eg: 10 or 3 (string form)
- koji_target - eg: dist-f11
- dist_defines - a list of parameters for an rpm aware function to redefine certain macros to match the profile
- mock_cfg - the name of the mock config/profile to be used to compile packages for this profile
- mock_cfg_dir - directory where profile wide mock settings are kept
- result_dir - where to store results from mock
The following basic rules must be applied. Exceptions can be granted for any reason, provided there's a good reason.
- Basic coding standards will follow PEP 8, except where noted below.
- Code should be as big as your head. If you have a small head or a large monitor, lower your font resolution ;).
- Code should be terse, and haiku like. It should explain a single concept or action clearly, without being overrun by scaffolding.
- Scaffolding should be broken down into utility functions, all exposed in the API. They should be documented clearly.
- Where logical, use context managers. This will isolate the alogrithm from the build up and break down.
- High nesting of if/then statements is not allowed
- High nesting of context managers and for loops are allowed, although extreme nesting is discouraged.
- Read only attributes from
obj.cfgshould be exposed as properties as such:
@property def spec_file(self): '''returns the name of the spec file as it should be accordingto fedora guidelines, good for double checking''' return self.pkg_name + '.spec' @property def pkg_name(self): '''canonical name of the package in a source repository''' return self.cfg['pkg_name']
All patches are welcome. The best way to submit patches initially is via email or mailing lists, or sending a pull request from another git repo. In order to get write access to the public repo, there is a one drink minimum. To demonstrate understanding of the Coding Guidelines, the submitter must have at least one other patch approved before gaining write access. This will ensure that the coder has had ample time to learn the simple guidelines.