From Fedora Project Wiki
(→‎Porting status: python3-paste added)
m (internal link cleaning)
 
(6 intermediate revisions by 4 users not shown)
Line 17: Line 17:
=== Completed items ===
=== Completed items ===
* The [https://admin.fedoraproject.org/pkgdb/packages/name/python3 python3 srpm] is in Fedora as of 2010-01-13; original package review is [https://bugzilla.redhat.com/show_bug.cgi?id=526126 here], duplicated for the purpose of import [https://bugzilla.redhat.com/show_bug.cgi?id=554799 here].
* The [https://admin.fedoraproject.org/pkgdb/packages/name/python3 python3 srpm] is in Fedora as of 2010-01-13; original package review is [https://bugzilla.redhat.com/show_bug.cgi?id=526126 here], duplicated for the purpose of import [https://bugzilla.redhat.com/show_bug.cgi?id=554799 here].
* Fedora's guidelines on [https://fedoraproject.org/wiki/Packaging:Python how to package Python] have been extended to cover dual python 2 and python 3 stacks
* Fedora's guidelines on [[Packaging:Python|how to package Python]] have been extended to cover dual python 2 and python 3 stacks
* Support for [http://mx.gw.com/pipermail/file/2009/000483.html identifying Python 3.0 and 3.1 .pyc and .pyo bytecode files] has been added to the "file" utility
* Support for [http://mx.gw.com/pipermail/file/2009/000483.html identifying Python 3.0 and 3.1 .pyc and .pyo bytecode files] has been added to the "file" utility
* We have improved [https://bugzilla.redhat.com/show_bug.cgi?id=531117 rpm's handling of python bytecode] so that it can cope with multiple python versions
* We have improved [https://bugzilla.redhat.com/show_bug.cgi?id=531117 rpm's handling of python bytecode] so that it can cope with multiple python versions
Line 36: Line 36:
** python3-setuptools (actually "Distribute"; built as a subpackage of python-setuptools)
** python3-setuptools (actually "Distribute"; built as a subpackage of python-setuptools)
** python3-smbpasswd (built as a subpackage of python-smbpasswd)
** python3-smbpasswd (built as a subpackage of python-smbpasswd)
with [https://fedoraproject.org/wiki/Features/Python3F13#Python_3_already_in_Fedora many more built and ready for Fedora 14]
with [[Features/Python3F13#Python_3_already_in_Fedora|many more built and ready for Fedora 14]]


=== Items still to be done ===
=== Items still to be done ===
* [http://fedoraproject.org/wiki/PackageMaintainers/CompsXml Add a comps group] for "Python 3"
* [[PackageMaintainers/CompsXml|Add a comps group]] for "Python 3"
* As many modules as can be reasonably ported/built.  See [[#Python_3_code_not_yet_in_Fedora|the list]]
* As many modules as can be reasonably ported/built.  See [[#Python_3_code_not_yet_in_Fedora|the list]]


Line 94: Line 94:


=== Porting status ===
=== Porting status ===
Please keep each of these tables sorted alphabetically by python module name
These status tables have been moved to: https://fedoraproject.org/wiki/Python3#Porting_status


==== Python 3 already in Fedora ====
==== Python 3 already in Fedora ====
{|
Moved to: https://fedoraproject.org/wiki/Python3#Python_3_already_in_Fedora
! Python Module !! Fedora Python 2 package !! Upstream status of Python 3 !! Fedora Python 3 package
|-
| || bpython || Should work in 0.9.6.2: http://bitbucket.org/bobf/bpython/issue/84/bpython-doesnt-work-with-python-3 ; needs pyparsing || In Fedora 14 onwards as '''python3-bpython''' subpackage of bpython (was {{bz|590107}})
|-
| || swig || According to http://www.swig.org/news.php, Python 3 support was added to SWIG in 1.3.37 (released 2009-01-15); generated C code appears to be compilable against both Python 2 and Python 3 || Fedora 11 onwards has had a version of "swig" capable of generating code for both Python major-versions.
|-
| beaker || python-beaker ||  || In Fedora 14 onwards as '''python3-beaker''', built as a subpackage of python-beaker
|-
| chardet || python-chardet || Upstream releasing dual-purpose tarballs || In Fedora 13 onwards as '''python3-chardet''' (was RHBZ [https://bugzilla.redhat.com/show_bug.cgi?id=583186 #583186]), the [https://admin.fedoraproject.org/updates/python3-chardet-2.0.1-2.fc13 update] is now stable. Though not yet in the beta release, it will be in the final one.
|-
| cherrypy || python-cherrypy || Python 3 supported as of [http://www.cherrypy.org/wiki/WhatsNewIn32 CherryPy 3.2 (October 2009)], upstream releasing [http://download.cherrypy.org/cherrypy/3.2.0rc1/CherryPy-3.2.0rc1-py3.tar.gz separate tarballs for python 3]|| In Fedora 13 onwards as '''python3-cherrypy''' (was {{bz|579593}}), though F-13 build is only available as [https://admin.fedoraproject.org/updates/python3-cherrypy-3.2.0-0.rc1.r2567.1.fc13 an update].
|-
| coverage || python-coverage || Upstream releasing dual-purpose tarballs || In Fedora 13 onwards as '''python3-coverage''' subpackage of python-coverage (was {{bz|536948}})
|-
| decorator || python-decorator ||  Upstream reports that [http://pypi.python.org/pypi/decorator/3.1.2 3.1.2 ought to work, by running 2to3 on the module decorator.py] || In Fedora 14 onwards as '''python3-decorator''' subpackage of python-decorator
|-
| deltarpm || deltarpm || || In Fedora 13 onwards (the "deltarpm" build emits both "python-deltarpm" and '''"python3-deltarpm'''" subpackages)
|-
| jinja2 || python-jinja2 || Version 2.3 introduced 'experimental' Python 3 support. || In Fedora 14 onwards as '''python3-jinja2''' subpackage of python-jinja2  (was {{bz|537268}})
|-
| lxml|| python-lxml || Upstream releasing dual-purpose tarballs || In Fedora 13 onwards as '''python3-lxml''' subpackage of python-lxml (was {{bz|533290}})
|-
| mako || python-mako || From version 0.3, mako supports python3 || In Fedora 14 onwards as '''python3-mako''' subpackage of python-mako (was {{bz|585036}})
|-
| minimock || python-minimock || [http://groups.google.com/group/minimock-dev/browse_thread/thread/a6cd2d572eec31d0 Asked upstream] they will release dual-purpose tarballs || In Fedora 13 onwards as '''python3-minimock''' subpackage of python-minimock
|-
| mpich2, openmpi || mpi4py || || '''python3-mpi4py-{openmpi,mpich2}''' in Fedora 13 onwards, built as a subpackage of mpi4py.
|-
| paste || python-paste || || In Fedora 15 onwards as '''python3-paste''' subpackage of python-paste
|-
| ply || python-ply || 2 and 3 from same tarball [http://www.dabeaz.com/ply/ from PLY-3.0 onwards]; README states "You should not convert PLY using 2to3 -- it is not necessary and may in fact break the implementation."
|| '''python3-ply''' in Fedora 13 onwards, built as a subpackage of python-ply
|-
| postgresql || || [http://python.projects.postgresql.org/ py-postgresql] || In Fedora 13 onwards  as '''python3-postgresql''' (was {{bz|579280}}), though F-13 build is only available as [https://admin.fedoraproject.org/updates/python3-postgresql-1.0.0-1.fc13 an update]
|-
| psutil || python-psutil || || In Fedora 13 onwards as a '''python3-psutil''' subpackage
|-
| pygments || python-pygments || Upstream reports that [http://dev.pocoo.org/projects/pygments/ticket/448 "Pygments is already ported to Python 3. The same source release can be used for 2.x and 3.x installs"] || In Fedora 14 onwards as a '''python3-pygments'' subpackage of python-pygments (was {{bz|537244}})
|-
| pyke || pyke || || In Fedora 13 onwards as a '''python3-pyke''' subpackage of pyke
|-
| PyQt4 || PyQt4 || || In Fedora 14 onwards as '''python3-PyQt4''' and '''python3-PyQt4-devel''' subpackages of PyQt4 (was {{bz|586196}})
|-
| selinux  || libselinux-python || Generated by SWIG during build of libselinux || In Fedora 14 onwards as '''libselinux-python3''' subpackage of libselinux (was{{bz|582006}})
|-
| semanage  || libsemanage-python || Generated by SWIG during build of libsemanage || In Fedora 14 onwards as '''libsemanage-python3''' subpackage of libsemanage (was {{bz|582007}})
|-
| setuptools || python-setuptools || || In Fedora 13 onwards as a '''python3-setuptools''' subpackage (internally we've switched to the "Distribute" fork)
|-
| sip || sip || [http://freshmeat.net/projects/python-sip/releases/300495 added] (single tarball) || In Fedora 14 onwards as '''python3-sip''' and '''python3-sip-devel''' subpackages of "sip" (was {{bz|545124}})
|-
| smbpasswd || python-smbpasswd || Create own package for p3k; sent patch to upstream || In Fedora 13 onwards as '''python3-smbpasswd''' subpackage (was {{bz|560456}})
|-
| sqlalchemy || python-sqlalchemy || 0.6beta1 has py3k support from a single tarball.||  Packaged for F-14.  Needs nose3 for running unittests of '''python3-sqlalchemy'''.  Due to incompatible API, decided not to push back to F-13.
|}


==== Python 3 code not yet in Fedora ====
==== Python 3 code not yet in Fedora ====
{|
Moved to: https://fedoraproject.org/wiki/Python3#Python_3_code_not_yet_in_Fedora
! Python Module !! Fedora Python 2 package !! Upstream status of Python 3 !! Fedora Python 3 package
|-
| || mod_wsgi || mod_wsgi version 3.0 and later supports Python 3.1 and later: http://code.google.com/p/modwsgi/wiki/SupportForPython3X http://code.google.com/p/modwsgi/wiki/ChangesInVersion0300 ||
|-
| babel || python-babel || || Patch to .spec to emit python2 and python 3 from one build '''needs work''': {{bz|579567}}
|-
| docutils || python-docutils || Website says: "From version 0.6 Docutils is compatible with Python 3, but requires 2to3." Note: Soft dependency on python-imaging which is not yet ported.  We can make python3-docutils not use imaging with reduced functionality compared to the python2 version|| ''Anyone want to help with this?  I'm strapped for time but willing to support it once it's in (toshio)'' '''awaiting review from packager''': {{bz|579567}}
|-
| numpy || numpy || SVN trunk 2.0.0.dev8334 (and some earlier ones) works with Python 3 ||
|-
| PyKDE4  || PyKDE4 (from the kdebindings srpm) || || {{bz|587005}}
|-
| rpm || rpm-python (subpackage of "rpm")|| dmalcolm and pmatilai [http://dmalcolm.livejournal.com/3340.html ported the C extension for librpm to work with both python 2 and 3]; released as [http://www.rpm.org/wiki/Releases/4.8.0 rpm 4.8.0] || '''Needs packaging work''': see {{bz|531543}}
|}


==== To be categorized ====
==== To be categorized ====
{|
Moved to: https://fedoraproject.org/wiki/Python3#To_be_categorized
! Python Module !! Fedora Python 2 package !! Upstream status of Python 3 !! Fedora Python 3 package
|-
| || gedit || ||
|-
| || gdb || ||
|-
| || luma || Depend on other packages, which have to migrate to py3k ||
|-
| BeautifulSoup || python-BeautifulSoup || 3.1.0 adds Python 3 support, but reportedly doesn't work as well as 3.0.8 (See http://www.crummy.com/software/BeautifulSoup/3.1-problems.html http://groups.google.com/group/beautifulsoup/browse_thread/thread/f24882cc17a0625e and http://stackoverflow.com/questions/149585/beautifulsoups-python-3-compatibility ) || Fedora 13 and devel have 3.0.8 for now
|-
| bugzilla || python-bugzilla || ||
|-
| devtools || python-tg-devtools || ||
|-
| genshi || python-genshi || ||
|-
| enum || python-enum || ||
|-
| fedora || python-fedora || ||
|-
| migrate || python-migrate || ||
|-
| MySQLdb  || MySQL-python    || ||
|-
| nose  || python-nose || This is the current nose3 branch [http://bitbucket.org/jpellerin/nose3/ on bitbucket].  Googlecode branch is out-of-date.  This [http://code.google.com/p/python-nose/issues/detail?id=240 upstream ticket], indicates the trunk will move to python3 only, and python2 will be in maintenance mode but upstream has said that the current py2 branch is being worked on by others and the py3 branch is lagging behind.||
|-
| paste.deploy || python-paste-deploy || ||
|-
| paste.script || python-paste-script || ||
|-
| psycopg2  || python-psycopg2 || ||
|-
| py || python-py || 1.3.0 has [http://codespeak.net/py/dist/announce/release-1.3.0.html "...improved compatibility with Python3..."] ||
|-
| pyparsing  || pyparsing || [http://pyparsing.wikispaces.com/News Python 3 support was added in 1.5.2 as a pyparsing_py3.py file]||
|-
| repoze.tm || python-repoze-tm2 || ||
|-
| repoze.what || python-repoze-what || ||
|-
| repoze.what.plugins.sql || python-repoze-what-plugins-sql || ||
|-
| repoze.what.plugins.pylonshq || python-repoze-what-pylons || ||
|-
| repoze.what.plugins.quickstart || python-repoze-what-quickstart || ||
|-
| repoze.who || python-repoze-who || ||
|-
| repoze.who.plugins.friendlyform || python-repoze-who-friendlyform || ||
|-
| repoze.who.plugins.sa || python-repoze-who-plugins-sa || ||
|-
| repoze.who.plugins.testutil || python-repoze-who-testutil || ||
|-
| routes || python-routes || ||
|-
| simplejson || python-simplejson || ||
|-
| sprox || python-sprox || ||
|-
| tg || TurboGears2 || ||
|-
| tgext.admin || python-tgext-admin || ||
|-
| tgext.crud || python-tgext-crud || ||
|-
| tw || python-toscawidgets || ||
|-
| turbojson || python-turbojson || ||
|-
| turbokid || python-turbokid || ||
|-
| tw.forms || python-tw-forms || ||
|-
| virtualenv || python-virtualenv || A forked version for Python 3 is [http://bitbucket.org/ianb/virtualenv/issue/10/python-3-support available], but I'm not sure how "official" it is ||
|-
| weberror || python-weberror || ||
|-
| webflash || python-webflash || ||
|-
| webhelpers || python-webhelpers || ||
|-
| webob || python-webob || ||
|-
| webtest || python-webtest || ||
|-
| zope.sqlalchemy || python-zope-sqlalchemy || ||
|}


==== Python modules that appear to not yet be ready for Python 3 packaging  ====
==== Python modules that appear to not yet be ready for Python 3 packaging  ====
Please let us know if this information is out-of-date.
Moved to: https://fedoraproject.org/wiki/Python3#Python_modules_that_appear_to_not_yet_be_ready_for_Python_3_packaging
 
FIXME: add note about if a python3 version should be packaged
 
{|
! Python Module !! Fedora Python 2 package !! Upstream status of Python 3 !! Fedora Python 3 package
|-
| || blender || Blender 2.5 [http://wiki.blender.org/index.php/BlenderDev/Blender2.5/PythonAPI_31 migrated its embedded version of python from Python 2 to Python 3].
 
However, [http://www.blender.org/bf/2009-planning.png Blender 2.6's schedule] suggests this version might not be regarded as stable in time for Fedora 13
 
(Upstream [http://download.blender.org/release/Blender2.50alpha/ ships with a bundled copy of 3.1]; we would want to build against a system copy of Python 3)
|| Blender 2.6 '''won't be ready in time for Fedora 13'''
|-
| || mod_python || Upstream mod_python has [http://blog.dscpl.com.au/2010/05/modpython-project-soon-to-be-officially.html effectively stopped since February 2007. ..."You also will never see a version of mod_python for Python 3.X as that is going to require a radical rewrite"...] with various suggestions including moving to mod_wsgi ||
|-
| augeas || python-augeas || dmalcolm has [https://www.redhat.com/archives/augeas-devel/2010-April/msg00002.html experimented with porting this to python 3] ||
|-
| cairo || pycairo || Implemented in C, would need some porting (with 2to3c ?) ||
|-
| Cheetah || python-cheetah || As of 2010-02-02, upstream site reports that [http://packages.python.org/Cheetah/roadmap.html#cheetah-v3-0 Python 3.xx support will be in Cheetah v3.0, but that it is "still in planning"]||
|-
| cracklib  || cracklib-python || Looking at cracklib-2.8.16, looks like the module needs some work to port (e.g. module initialization), and uses automake so may need work to build multiple times||
|-
| Crypto || python-crypto || Experimental patch on upstream list http://lists.dlitz.net/pipermail/pycrypto/2010q1/000182.html , however [http://lists.dlitz.net/pipermail/pycrypto/2009q4/000176.html upstream forbids US contributors to the project, which seems to have ruled out the patch] (and rules out dmalcolm from completing the port) Explanation of the [http://www.dlitz.net/software/pycrypto/ ban on US contributors].  If we have any lawyers familiar with the export regulations perhaps we can get the ban lifted?||
|-
| dbus || dbus-python || Patch created by Fedora contributor John Palmieri [http://bugs.freedesktop.org/show_bug.cgi?id=26420 sent upstream], awaiting upstream review|| [https://bugzilla.redhat.com/show_bug.cgi?id=538616 rhbz 538616]
|-
| django || Django || http://wiki.python.org/moin/PortingDjangoTo3k - there's an experimental branch in bitbucket, but as of 2010-04-09 it doesn't appear to have "official" releases ||
|-
| enchant || python-enchant || Uses SWIG with some special-casing; appears to need some porting ||
|-
| gtk || pygtk2 ||  Not yet ready: https://bugzilla.gnome.org/show_bug.cgi?id=566641  Question: should we be using [http://live.gnome.org/GObjectIntrospection GIR] instead?  (Or do both?)|| Need to finish upstream work
|-
| krbV || python-krbV || Implemented in C, would need porting (with 2to3c ?)||
|-
| ldap || python-ldap || Porting [http://www.mail-archive.com/python-ldap-dev@lists.sourceforge.net/msg00620.html discussed on upstream mailing list on 2009-09-03] but appears not to be done yet ||
|-
| libxml2 || libxml2-python || Porting doesn't seem to have started ||
|-
| nss  || python-nss  || Looks like we need to port it, and would be non-trivial ||
|-
| parted || pyparted || Looks like we need to port it ||
|-
| PIL || python-imaging || As of 2010-01-28, upstream website says [http://www.pythonware.com/products/pil/ "The current free version is PIL 1.1.7. This release supports Python 1.5.2 and newer, including 2.5 and 2.6.  A version of 1.1.7 for 3.X will be released later."] A 2010-02-21 mailing list post suggests that [http://mail.python.org/pipermail/image-sig/2010-February/006124.html the port is stalled] ||
|-
| pygi ||  || Fedora contributors David Malcolm and John Palmieri have [https://bugzilla.gnome.org/show_bug.cgi?id=615872 ported pygi to Python 3], patches are awaiting merger upstream  ||
|-
| pylons || python-pylons || As of 2010-04-09, it's [http://wiki.pylonshq.com/display/pylonscommunity/Pylons+Roadmap+to+1.0 on the future roadmap for 1.1]; see also http://pylonshq.com/project/pylonshq/ticket/425 ||
|-
| sphinx || python-sphinx || A port to Python 3 has been [http://groups.google.com/group/sphinx-dev/browse_thread/thread/8d0e4bca8b39f809/98733a0ed9a05c46 proposed as a GSOC 2010 project] ||
|-
| wx || wxPython || As of 2010-02-03, appears not to be ported yet; see http://stackoverflow.com/questions/720806/wxpython-for-python-3-0 ||
|-
| xdg|| pyxdg || Appears to not yet be ported; see {{bz|555620}} ||
|}


=== mod_wsgi and mod_python ===
=== mod_wsgi and mod_python ===
Line 381: Line 169:


=== Follow the "Dive Into Python 3" tutorial ===
=== Follow the "Dive Into Python 3" tutorial ===
If you've installed the python3 rpm, it should be possible for you to follow the tutorial on Python 3 given here: http://diveintopython3.org/
If you've installed the python3 rpm, it should be possible for you to follow the tutorial on Python 3 given here: http://diveintopython3.ep.io/


This is an excellent tutorial, and working through it with our RPMs makes a good test that all is working as expected (obviously it would be a major task to work through the whole book, but at least you'd learn a lot in the process).
This is an excellent tutorial, and working through it with our RPMs makes a good test that all is working as expected (obviously it would be a major task to work through the whole book, but at least you'd learn a lot in the process).
Line 484: Line 272:
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->
[[Category:Python]]

Latest revision as of 20:44, 19 September 2016

Python 3

Summary

A packaged version of python 3.* will be provided within Fedora 13 as an optional component, parallel-installable with the Python 2 stack.

The critical system components that use Python 2 (yum, anaconda) shall continue to use Python 2.

Owner

Current status

  • Targeted release: Fedora 13
  • Last updated: 2010-04-05
  • Percentage of completion: 100% - it's difficult to say when this is done; the more packages we have the better. The big-ticket issues are done, but there are plenty of python3 packages waiting on review - Assistance with package reviews would be most welcome

Completed items

  • The python3 srpm is in Fedora as of 2010-01-13; original package review is here, duplicated for the purpose of import here.
  • Fedora's guidelines on how to package Python have been extended to cover dual python 2 and python 3 stacks
  • Support for identifying Python 3.0 and 3.1 .pyc and .pyo bytecode files has been added to the "file" utility
  • We have improved rpm's handling of python bytecode so that it can cope with multiple python versions
  • We have written new tests for rpmlint to ensure that python bytecode within an RPM package is for the correct version of python, along with other sanity checks.
  • Fedora contributor David Malcolm created the 2to3c tool to help Python developers port their C extensions to Python 3.
  • Python 3 modules available via RPM in Fedora 13:
    • python3-chardet (though currently not yet in the F-13 Beta, it will be available in the final release (update is stable))
    • python3-cherrypy (though currently not in main tree, but available as an update)
    • python3-coverage (built as a subpackage of python-coverage)
    • python3-deltarpm (built in package deltarpm like python-deltarpm)
    • python3-lxml (built as a subpackage of python-lxml)
    • python3-minimock (built as a subpackage of python-minimock)
    • python3-mpi4py-{openmpi,mpich2} (build as a subpackage of mpi4py)
    • python3-ply (built as a subpackage of python-ply)
    • python3-postgresql (though currently not in main tree, but available as an update)
    • python3-psutil (built as a subpackage of python-psutil)
    • python3-pyke (built as a subpackage of pyke)
    • python3-setuptools (actually "Distribute"; built as a subpackage of python-setuptools)
    • python3-smbpasswd (built as a subpackage of python-smbpasswd)

with many more built and ready for Fedora 14

Items still to be done

Detailed Description

Python 3 is intended by upstream to be the future of Python, but we have many critical components that use Python 2. Python 2 and Python 3 are sufficiently different that we need both (try writing "print" in each). Python 2 will be around for a long time.

An interesting summary of Python 3 adoption can be seen here: http://renesd.blogspot.com/2009/09/py3kpython3-more-than-one-year-on-096.html

How to do this? I propose that Fedora shall have separate, parallel-installable Python 2 and Python 3 stacks. I believe we can get things to the point where on a Fedora box you'd be able to install both stacks, and have some processes running python 2 code, and some running python 3, simultaneously.

Where I would draw the line is on having both python 2 and python 3 running within the same _process_: the two libraries share most of their symbol names, but with differing implementations, and the result of trying to dynamically link the two into the same address space would be highly unstable.

As an example, you'd be able to install both mod_python and mod_python3 rpms, but you wouldn't be able to (sanely) configure httpd to have both running simultaneously (I guess we should add a run-time warning for this case)

Benefit to Fedora

Fedora has long been a great platform for doing Python 2 development, but we don't yet have Python 3. Having Python 3 available via rpms will extend our Python coverage.

Note also:

  • Python 3.0 was released almost 10 months ago, on 2008-12-03, and the latest release of the 3.* branch is 3.1.1, released on 2009-08-17.
  • Other distros have python 3, though not necessarily with anything "on top" resembling the full python 2 stack.
  • We have a working, valuable python 2 stack, which is used by critical system components (yum and anaconda): we must not destabilize the python 2 stack.
  • Python 3 is sufficiently different from python 2 that we need them to be independent software stacks.

Scope

A plan for packaging Python 3 and Python 3 modules was posted to fedora-devel-list here: https://www.redhat.com/archives/fedora-devel-list/2009-October/msg00054.html

A rough plan for Fedora 13 might be:

  • get python3 packaged in a manner compatible with the above
  • (persuade /usr/lib/rpm/brp-python-bytecompile to use the correct python when building rpms containing .py files)
  • get rpm bindings working with python3
  • get some useful components working e.g. a web stack: Django, TurboGears etc (though e.g. Django's py3k support is a long way off IIRC); ideas?
    • Web stacks may not be the best thing to focus on since production largely wants to use mod_wsgi... which we won't be able to mix between py2 and py3 stacks. --abadger1999 01:52, 13 November 2009 (UTC)
  • solidify packaging guidelines for python 2 vs python 3 once we've got some experience with the above and hopefully proven the techniques
  • look at porting major components over to python3 (but probably don't actually do this for F13; leave python 2 as the critical component, I suspect): yum (rpm), anaconda
  • add the new packages to the comps file. It's not yet clear what changes to the comps file are required.

At a minimum, we'd want python 3 to be available as an rpm via yum.

Porting status

These status tables have been moved to: https://fedoraproject.org/wiki/Python3#Porting_status

Python 3 already in Fedora

Moved to: https://fedoraproject.org/wiki/Python3#Python_3_already_in_Fedora

Python 3 code not yet in Fedora

Moved to: https://fedoraproject.org/wiki/Python3#Python_3_code_not_yet_in_Fedora

To be categorized

Moved to: https://fedoraproject.org/wiki/Python3#To_be_categorized

Python modules that appear to not yet be ready for Python 3 packaging

Moved to: https://fedoraproject.org/wiki/Python3#Python_modules_that_appear_to_not_yet_be_ready_for_Python_3_packaging

mod_wsgi and mod_python

mod_wsgi and mod_python will break if the python2 and python3 versions are both loaded into the same apache process because the symbols in libpython.so.2 and libpython.so.3 will clash. However, an admin may want to run two apache's on a server, one with the python2 version and one with the python3 versions. We should support this. Giving the admin some default conf files that will only let them load one set of modules is probably what we want. The admin who wants to run two servers will need to customize their configuration to make it work. Note that if the admin installs gets mod_python3 running on the default apache and then installs mod_python, their apps will stop working as apache will start loading the python2 version.

<!-- in mod_wsgi.conf -->
<IfModule !mod_python3.c>
  <IfModule !mod_wsgi_py3.c>
      LoadModule wsgi_module modules/mod_wsgi.so
  </IfModule>
</IfModule>

<!-- in mod_python.conf -->
<IfModule !mod_python3.c>
  <IfModule !mod_wsgi_py3.c>
      LoadModule wsgi_module modules/mod_python.so
  </IfModule>
</IfModule>

<!-- in mod_wsgi_py3.conf -->
<IfModule !mod_python.c>
  <IfModule !mod_wsgi.c>
      LoadModule wsgi_module modules/mod_wsgi_py3.so
  </IfModule>
</IfModule>

<!-- in mod_python3.conf -->
<IfModule !mod_python.c>
  <IfModule !mod_wsgi.c>
      LoadModule wsgi_module modules/mod_python3.so
  </IfModule>
</IfModule>

How To Test

Testing Python 3 functionality

You should be able to test the new python 3 stack using:

$ yum install python3

and then starting the python 3 interpreter using

$ python3

You should then see the Python 3 interpreter:

Python 3.1.1 (r311:74480, Sep 29 2009, 17:01:17) 
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Follow the "Dive Into Python 3" tutorial

If you've installed the python3 rpm, it should be possible for you to follow the tutorial on Python 3 given here: http://diveintopython3.ep.io/

This is an excellent tutorial, and working through it with our RPMs makes a good test that all is working as expected (obviously it would be a major task to work through the whole book, but at least you'd learn a lot in the process).

Ensuring non-brokenness of Python 2 stack

The Python 3 stack should not interfere with the Python 2 stack; the latter is used by many components. The most critical ones here are yum and anaconda.

Please verify that "yum" still works after installing python3.

If there are other python 2 rpms that you are familiar with, please check that they still work after installing the python3 rpm.

Uninstallability of Python 3

  • It ought to be possible for you to remove python3 without affecting other components
    • A better phrasing of this point would be: It ought to be possible to remove python3 without affecting any python2 components. If some app gets ported to python3 upstream in F13, I do think we should allow it.
  • Try uninstalling python 3 using "yum remove python3".
  • Verify that other components still work as expected

Upstream regression test suite

Run Python's regression test suite (within the python3-test subpackage)

/usr/lib/python3.1/test/regrtest.py

as root and verify that only expected errors occur.

As of 2009-10-27, if you run the above command as root, the only failure should be in test_httpservers (appears to be a permissions issue with how we've packaged the test scripts). You may also see an error in test_socket if your machine's hostname isn't set up in DNS/hosts.

If you run using sudo, you may see an extra failure in test_distutils (test_check_environ).

If you run as a non-root user, you'll see additional failures due to tests that expect write access to certain directories

Static testing

Verify that for all subpackages, on all architectures, that the python and python3 packages are independent:

  • verify that there aren't any paths (files/dirs/links) owned by both packages
  • verify that there aren't any "Provides" provided by both packages

A script to automatically verify this can be seen at https://bugzilla.redhat.com/show_bug.cgi?id=526126#c17

User Experience

It should be possible to start a Python 3 interpreter by invoking:

$ python3
Python 3.1.1 (r311:74480, Sep 29 2009, 17:01:17) 
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

The python 3 install should be independent of the python 2 install.

It should be possible to have some processes on the system running python 2 code, and some processes on the system running python 3 code. However you won't be able to run both python 2 and python 3 within the same process.

Dependencies

Getting the core "python3" package into Fedora requires us:

  • to come up with packaging rules for Python 3.
  • to ensure our packaging tools can cope with multiple, parallel python installs:

Building out the python 3 stack beyond the core interpreter component will involve working closely with many different upstream projects. The more we can do this, the better, but it's not necessary for achieving the main goal of having the core python 3 interpreter available via rpm.

rpmbuild changes

rpmbuild currently automatically invokes /usr/lib/rpm/brp-python-bytecompile without arguments, thus using "/usr/bin/python" to byte-compile every .py file that's in a package payload, generating a bytecode file for the 2.6 ABI.

This breaks down if we're to deal with multiple python runtimes:

  • the magic ABI value stored in the .pyc/.pyo file needs to match that of the python binary. .pyc files below /usr/lib/python2.6 need to have the 2.6 magic value, whereas .pyc files below /usr/lib/python3.1 need to have the 3.1 magic value.
  • the syntax of the python language can change (e.g. 2 vs 3), and the compilation can fail with syntax errors if you use the wrong python runtime

Status: I work around this "by hand" in the python3.spec. Need to fix "properly" before building out python3- stack with further packages.

rpmlint changes

I've written a new test for rpmlint to do some of the .pyc checking. See: https://www.zarb.org/pipermail/rpmlint-discuss/2009-October/000775.html (I also uploaded the patch here: http://dmalcolm.fedorapeople.org/rpmlint/add-tests-for-python-bytecode-files.patch )

Status: patch sent upstream, not yet in our Fedora rpmlint rpms.

Contingency Plan

We will be adding all-new RPMs to Fedora. If there's a problem, we can simply choose to not include them without impacting the rest of the Fedora 13 release.

Note: In order to enact this contingency plan, we must do separate packages for python2 and python3. If we keep pursuing the one srpm to build both python2 and python3 route and we have to back out the python3 stack, we also have to backout the python3 changes in the srpms. The same is also true of bindings to C libraries. --abadger1999 02:04, 13 November 2009 (UTC)

Good point. I've conditionalized support throughout the shared specfile changes, adding a with_python3 boolean. Is this an acceptable compromise? --Dmalcolm 18:06, 13 November 2009 (UTC)

Documentation

An excellent tutorial on Python 3 can be seen at http://diveintopython3.org/

Note: We may want to package this since we have the python2 version of diveintopython packaged already.

--Dmalcolm 23:24, 19 November 2009 (UTC) help packaging it would be appreciated! I tried grabbing this from Mercurial, but the toolchain/licensing appeared to have changed greatly since the python2 version of the book

Release Notes

  • Fedora now includes a Python 3 runtime, parallel-installable with our existing Python 2 runtime.

FIXME: obviously this will have to change based on exactly how much of the stack I manage to build out.

Comments and Discussion