DaveMalcolm/PythonIdeas

= Multiple Python configurations =

"Debug" versus "standard" build
Python can be built with various debugging options, typically just used when working on Python itself. In my experience "--with-pydebug" roughly halves the speed of the interpreter, but adds lots of extra checking (e.g. reference leaks). It also changes the ABI.

We could produce an entirely separate "debug" python stack, with various debugging checks configured in (breaking ABI, and slowing things down); hope was to make it trivial to run /usr/bin/python2.6-debug (provided you also have the debug build of all of the modules, etc; doesn't work, you need -debug of full stack, I think; you'd have a yum-debug, with -debug of all the stack).

I believe Debian's python packages have worked this way for a year or two; as I understand it they embed the debug build into their "-dbg" subpackages (the equivalent of our "-debuginfo" subpackage)

(See pyconfig.h)

--with-valgrind (not needed if --without-pymalloc)

--enable-profiling ?

Future work: we may want to have with/without LLVM as a build option

"--with-pydebug" is the big one, implying various ABI-breaking changes.

Might want to turn off optimizations, or only optimize lightly, whilst we're at it (e.g. "-Os" ?)

Would need to do as a Feature. Do it for python3 as well, potentially giving 4 subpackages per build.

So for package "coverage" (with C extension modules) you might have the python-coverage.src.rpm emit the following subpackages as part of its build:
 * python-coverage (or "python2-coverage"?)
 * python-dbg-coverage (or "python2-dbg-coverage"?)
 * python3-coverage
 * python3-dbg-coverage
 * python-coverage-debuginfo

whereas for package "setuptools" (without C extension modules) you might have the python-setuptools.src.rpm emit the following subpackages:
 * python-setuptools (or "python2-setuptools"?)
 * python3-setuptools

UCS2 vs UCS4
Is it sane to double the number of configurations, offering UCS2 vs UCS4? I don't think so, but listing it here for completeness. I'm not convinced that the choice of UCS4 is correct; are we merely doing it due to historical precedent?

Multiple minor versions
- advance versions of 2.7 and 3.2

Alternate Implementations
Can we use the following to build out multiple stacks? (e.g. share pure-python modules between Jython and CPython?)
 * Unladen Swallow
 * Jython
 * PyPy
 * http://pypy.org/download.html
 * https://codespeak.net/issue/pypy-dev/issue432 SELinux issue

Dealing with multiple python implementations
What would packaging guidelines look like? Generalization of current guidelines to add additional builds.

Expressing variability within Koji
See my idea on this sent to the packaging list before. It was roundly rejected at the time, but I'm still fond of it.

Expressing variability within a single build
We do this at the moment for emitting python2 and python3 from within a single build of a src.rpm. Can it be generalized, so that on a given platform we have a set of python 2 and python 3 runtimes, and all builds emit a set of subpackages automatically?

e.g. thinking alouf here, a "python-meta-build" rpm containing: a /etc/python-runtimes.d/ directory, with each runtime dropping a python-foo.conf file (e.g. "python27-debug-ucs2.conf"). The python-meta-build rpm could have requires on all of the supported runtimes for an OS release, so that e.g. Fedora 15's python-meta-build might have: Requires: python-devel Requires: python-debug-devel Requires: python3-devel Requires: python3-debug-devel Requires: python27-devel Requires: python27-debug-devel Requires: python32-devel Requires: python32-debug-devel Requires: unladen-swallow-devel Requires: unladen-swallow-debug-devel

and then e.g.  might have

We could have separate arch and noarch meta-packages and directories.

python-coverage.spec might then look like this:

%prep %setup -q -n coverage-%{version}


 * 1) FIXME: what to do about running 2to3?

%build for conf in $(rpm-pyconfig --list) ; do   $(rpm-pyconfig --exe $conf) setup.py build done

%install for conf in $(rpm-pyconfig --list) ; do   $(rpm-pyconfig --exe $conf) setup.py install -O1 --skip-build --root %{buildroot} done

%clean rm -rf %{buildroot} thus using a tool ("rpm-pyconfig", say) to enumerate the runtimes and query properties on them (by reading the .conf files in )

It could even support this kind of syntax (somehow need to express the python binary in a way that will be expandable by the tool in its iteration): %build rpm-pyconfig --foreach "$PYTHON setup.py build"

%install rpm-pyconfig --foreach "$PYTHON setup.py install -O1 --skip-build --root %{buildroot}"

(perhaps with a "--foreach2" and "--foreach3" option?)

Difficult issues with this approach:
 * what do to about python2 versus python3 sources (i.e. when there's a single tarball for python 2 that has 2to3 run on it for python 3)
 * how do we list all of the subpackages, and all of the %files sections; is this doable in rpm as-is? (need to see what the kernel specfile does; this uses macros to great effect)
 * how do we conditionalize? perhaps "python-foo" doesn't build properly against 2.7 - is this analogous to ExcludeArch?  What if a module builds everywhere,  - except say that the 2.7 build doesn't build on ppc: how does the maintainer control that?

Perhaps something like this: each config file becomes an executable %prep %setup -q -n coverage-%{version}

%if 0%{?with_python3} rm -rf %{py3dir} cp -a. %{py3dir} pushd %{py3dir} 2to3 --nobackups --write. popd %endif # if with_python3

rpm-pyconfig --foreach-2 "cp -a . %{rpm-pyconfig-srcdir}" rpm-pyconfig --foreach-3 "cp -a %{py3dir} %{rpm-pyconfig-srcdir}"
 * 1) Ensure each variant gets its own pristine copy of the sources:
 * 1) perhaps the above could be done by a macro?

%build for pyconf in $(rpm-pyconfig --list) ; do   pushd $($pyconf --srcdir) $($pyconf --exe) setup.py build popd done

%install for pyconf in $(rpm-pyconfig --list) ; do   pushd $($pyconf --srcdir) $($pyconf --exe) setup.py install -O1 --skip-build --root %{buildroot} popd done

%clean rm -rf %{buildroot}

Or even: %prep %setup -q -n coverage-%{version}

%if 0%{?with_python3} rm -rf %{py3dir} cp -a. %{py3dir} pushd %{py3dir} 2to3 --nobackups --write. popd %endif # if with_python3

rpm-pyconfig --foreach-2 \ --copy-source-from.

rpm-pyconfig --foreach-3 \ --copy-source-from %{py3dir}

%build rpm-pyconfig --foreach --in-srcdir --exe \ setup.py build

%install rpm-pyconfig --foreach --in-srcdir --exe \ setup.py install -O1 --skip-build --root %{buildroot}

%clean rm -rf %{buildroot}

I've deliberately laid out the above so that each iteration is expressed with the control part on the first line, and the action is expressed on the second line.

Fleshed out and sent to the list here: http://lists.fedoraproject.org/pipermail/python-devel/2010-March/000213.html

= Other things we could change =

rather than ?
Debian uses "dist-packages" for things coming as part of the distribution, to avoid PyPi users splatting "site-packages"; should we? It seems more logical to me, keeping site-packages for site-specific use.