From Fedora Project Wiki

Multiple Python configurations

What might we want to vary?

"Debug" versus "standard" build

Python can be built with various debugging options, typically just used when working on Python itself. In my experience "--with-pydebug" roughly halves the speed of the interpreter, but adds lots of extra checking (e.g. reference leaks). It also changes the ABI.

We could produce an entirely separate "debug" python stack, with various debugging checks configured in (breaking ABI, and slowing things down); hope was to make it trivial to run /usr/bin/python2.6-debug (provided you also have the debug build of all of the modules, etc; doesn't work, you need -debug of full stack, I think; you'd have a yum-debug, with -debug of all the stack).

I believe Debian's python packages have worked this way for a year or two; as I understand it they embed the debug build into their "-dbg" subpackages (the equivalent of our "-debuginfo" subpackage)

Configuration option Definition Effect
--with-pydebug Py_DEBUG Implies ABI-breaking other options (see below), and adds additional self-tests throughout libpython
Py_TRACE_REFS (implied by Py_DEBUG): Track a doubly-linked list of all objects. Changes libpython ABI: adds _ob_prev and _ob_next members to PyObject
Py_REF_DEBUG (implied by Py_TRACE_REFS); track total reference count of all objects (_Py_RefTotal), and ensure it never goes negative
--without-pymalloc Avoid dealing with the arena allocator: go straight to malloc/free; changes API meaning
COUNT_ALLOCS Track the number of allocations made per PyTypeObject; changes API meaning
--with-tsc WITH_TSC enable/disable timestamp counter profile
DYNAMIC_EXECUTION_PROFILE Track how many times each opcode is executed
DXPAIRS Track how many times each pair of opcodes is executed
LLTRACE Track all stack operations
USE_STACKCHECK

(See pyconfig.h)

--with-valgrind (not needed if --without-pymalloc)

--enable-profiling ?

Future work: we may want to have with/without LLVM as a build option

"--with-pydebug" is the big one, implying various ABI-breaking changes.

Might want to turn off optimizations, or only optimize lightly, whilst we're at it (e.g. "-Os" ?)

Would need to do as a Feature. Do it for python3 as well, potentially giving 4 subpackages per build.

So for package "coverage" (with C extension modules) you might have the python-coverage.src.rpm emit the following subpackages as part of its build:

  • python-coverage (or "python2-coverage"?)
  • python-dbg-coverage (or "python2-dbg-coverage"?)
  • python3-coverage
  • python3-dbg-coverage
  • python-coverage-debuginfo

whereas for package "setuptools" (without C extension modules) you might have the python-setuptools.src.rpm emit the following subpackages:

  • python-setuptools (or "python2-setuptools"?)
  • python3-setuptools

UCS2 vs UCS4

Is it sane to double the number of configurations, offering UCS2 vs UCS4? I don't think so, but listing it here for completeness. I'm not convinced that the choice of UCS4 is correct; are we merely doing it due to historical precedent?

Multiple minor versions

- advance versions of 2.7 and 3.2

Alternate Implementations

Can we use the following to build out multiple stacks? (e.g. share pure-python modules between Jython and CPython?)

Dealing with multiple python implementations

What would packaging guidelines look like? Generalization of current guidelines to add additional builds.

Expressing variability within Koji

See my idea on this sent to the packaging list before. It was roundly rejected at the time, but I'm still fond of it.

Expressing variability within a single build

We do this at the moment for emitting python2 and python3 from within a single build of a src.rpm. Can it be generalized, so that on a given platform we have a set of python 2 and python 3 runtimes, and all builds emit a set of subpackages automatically?

e.g. thinking alouf here, a "python-meta-build" rpm containing: a /etc/python-runtimes.d/ directory, with each runtime dropping a python-foo.conf file (e.g. "python27-debug-ucs2.conf"). The python-meta-build rpm could have requires on all of the supported runtimes for an OS release, so that e.g. Fedora 15's python-meta-build might have:

Requires: python-devel
Requires: python-debug-devel
Requires: python3-devel
Requires: python3-debug-devel
Requires: python27-devel
Requires: python27-debug-devel
Requires: python32-devel
Requires: python32-debug-devel
Requires: unladen-swallow-devel
Requires: unladen-swallow-debug-devel

and then e.g. python-coverage might have BuildRequires: python-meta-build

We could have separate arch and noarch meta-packages and directories.

python-coverage.spec might then look like this:

%prep
%setup -q -n coverage-%{version}

#FIXME: what to do about running 2to3?

%build
for conf in $(rpm-pyconfig --list) ; do
    $(rpm-pyconfig --exe $conf) setup.py build
done

%install
for conf in $(rpm-pyconfig --list) ; do
    $(rpm-pyconfig --exe $conf) setup.py install -O1 --skip-build --root %{buildroot}
done

%clean
rm -rf %{buildroot}

thus using a tool ("rpm-pyconfig", say) to enumerate the runtimes and query properties on them (by reading the .conf files in /etc/python-runtimes.d/)

It could even support this kind of syntax (somehow need to express the python binary in a way that will be expandable by the tool in its iteration):

%build
rpm-pyconfig --foreach "$PYTHON setup.py build"

%install
rpm-pyconfig --foreach "$PYTHON setup.py  install -O1 --skip-build --root %{buildroot}"

(perhaps with a "--foreach2" and "--foreach3" option?)

Difficult issues with this approach:

  • what do to about python2 versus python3 sources (i.e. when there's a single tarball for python 2 that has 2to3 run on it for python 3)
  • how do we list all of the subpackages, and all of the %files sections; is this doable in rpm as-is? (need to see what the kernel specfile does; this uses macros to great effect)
  • how do we conditionalize? perhaps "python-foo" doesn't build properly against 2.7 - is this analogous to ExcludeArch? What if a module builds everywhere, - except say that the 2.7 build doesn't build on ppc: how does the maintainer control that?


Perhaps something like this: each config file becomes an executable

%prep
%setup -q -n coverage-%{version}

%if 0%{?with_python3}
rm -rf %{py3dir}
cp -a . %{py3dir}
pushd %{py3dir}
2to3 --nobackups --write .
popd
%endif # if with_python3

# Ensure each variant gets its own pristine copy of the sources:
rpm-pyconfig --foreach-2 "cp -a . %{rpm-pyconfig-srcdir}"
rpm-pyconfig --foreach-3 "cp -a %{py3dir} %{rpm-pyconfig-srcdir}"
# perhaps the above could be done by a macro?

%build
for pyconf in $(rpm-pyconfig --list) ; do
    pushd $($pyconf --srcdir)
    $($pyconf --exe) setup.py build
    popd
done

%install
for pyconf in $(rpm-pyconfig --list) ; do
    pushd $($pyconf --srcdir)
    $($pyconf --exe) setup.py install -O1 --skip-build --root %{buildroot}
    popd
done

%clean
rm -rf %{buildroot}


Or even:

%prep
%setup -q -n coverage-%{version}

%if 0%{?with_python3}
rm -rf %{py3dir}
cp -a . %{py3dir}
pushd %{py3dir}
2to3 --nobackups --write .
popd
%endif # if with_python3

rpm-pyconfig --foreach-2 \
    --copy-source-from .

rpm-pyconfig --foreach-3 \
    --copy-source-from %{py3dir}

%build
rpm-pyconfig --foreach --in-srcdir --exe \
    setup.py build

%install
rpm-pyconfig --foreach --in-srcdir --exe \
    setup.py install -O1 --skip-build --root %{buildroot}

%clean
rm -rf %{buildroot}

I've deliberately laid out the above so that each iteration is expressed with the control part on the first line, and the action is expressed on the second line.

Fleshed out and sent to the list here: http://lists.fedoraproject.org/pipermail/python-devel/2010-March/000213.html

Other things we could change

dist-packages rather than site-packages ?

Debian uses "dist-packages" for things coming as part of the distribution, to avoid PyPi users splatting "site-packages"; should we? It seems more logical to me, keeping site-packages for site-specific use.