From Fedora Project Wiki
m (Toshio moved page Features/Python setuptools 0.7 to Changes/Python setuptools 0.7: Change Proposals have a specified hierarchy)
(Filled out)
Line 1: Line 1:
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
 
  
= Change Proposal Name <!-- The name of your change proposal --> =
+
 
 +
= python-setuptools update to 0.7.x =
  
 
== Summary ==
 
== Summary ==
python-setuptools is used as a buildtime dependency in many packages.  It is also a runtime dependency for some packages where it provides a way to load plugins, load data files, and specify what versions of dependent packages are needed.  In Fedora, we also make use of its multiversioning features to install some backwards compatibility packages.
 
  
The upstream for the python setuptools package has been in a maintenance mode for many years.  Just recently, it has become active and released a new version that is not completely API compatible.  As the older versions won't see bug fixes anymore, we need to move to this new version but there are sure to be some usages that are broken by the new version.
+
Update to a new upstream release of python-setuptools that is not completely compatible with previous releases.
  
 
== Owner ==
 
== Owner ==
<!--
+
* Name: [[User:toshio| Toshio Kuratomi]] with help from [[User:bkabrda| Bohuslav Kabrda]], [[User:ncoghlin| Nick Coghlin]], and the Python SIG
For change proposals to quality as self-contained, owners of all affected packages need to be included here. Alternatively, a SIG can be listed as an owner if it owns all affected packages.
+
* Email: toshio@fedoraproject.org
This should link to your home wiki page so we know who you are.
 
-->
 
* Name: [[User:FASAcountName| Your Name]]
 
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
 
* Email: <your email address so we can contact you, invite you to meetings, etc.>
 
 
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
 
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
 
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
 
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
+
* FESCo shepherd: [[User:FASAccountName| Shepherd name]] <email address>
 
-->
 
-->
  
 
== Current status ==
 
== Current status ==
* Targeted release: [[Releases/<number> | Fedora <number> ]]  
+
* Targeted release: [[Releases/20 | Fedora 20 ]]  
* Last updated: (DATE)
+
* Last updated: June 10 2013
 
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
 
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
 
Bugzilla states meaning as usual:
 
Bugzilla states meaning as usual:
Line 35: Line 29:
  
 
== Detailed Description ==
 
== Detailed Description ==
<!-- Expand on the summary, if appropriateA couple sentences suffices to explain the goal, but the more details you can provide the better. -->
+
 
 +
python-setuptools is used as a buildtime dependency in many packages.  It is also a runtime dependency for some packages where it provides a way to load plugins, load data files, and specify what versions of dependent packages are needed.  In Fedora, we also make use of its multiversioning features to install some backwards compatibility packages.
 +
 
 +
The upstream for the python setuptools package has been in a maintenance mode for many years.  Just recently, it has become active and released a new version that is not completely API compatibleAs the older versions won't see bug fixes anymore, we need to move to this new version but there are sure to be some usages that are broken by the new version.
  
 
== Benefit to Fedora ==
 
== Benefit to Fedora ==
<!-- What is the benefit to the platform? If this is a major capability update, what has changed? If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->
+
 
 +
The upstream activity is due to two competing projects merging: python-seutptools and python-distribute.  The merger is a good thing for the python community as it unites the bugfixes and minor differences that have arisen in these two forks over the years. This makes it easier for upstreams to maintain their software. For us, it updates us to the latest version of the code which means that we'll be able to continue submitting patches as we find bugs in the code rather than having to maintain our own divergencies.
  
 
== Scope ==
 
== Scope ==
<!-- What work do the developers have to accomplish to complete the change in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
 
  
* Proposal owners:
+
=== Proposal owners ===
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
+
 
 +
* Update the python-setuptools package from the current code-base (python-distribute-0.6.x) to  python-setuptools-0.7.x
 +
* Rebuild packages with the new setuptools (Only necessary for testing)
 +
* Areas that may need special attention.  Note that these are areas that we will test to understand the scope of changes required.  It is not a promise to '''thoroughly''' test what is broken due to the update or to fix the breakage (although we will help)
 +
** Test python package installers with the new setuptools (pip, buildout)
 +
** Test python packages that build with setuptools for related ftbfs (we could do a mass rebuild of everything requiring python at runtime or refine our list to the packages that BuildRequire: python-setuptools)
 +
** Test packages where we are using setuptools to build multiversion (example: python-cherrypy2)
 +
** Test python packages that have a runtime requirement on setuptools [Note: The owners of this Change are going to leave most of this to individual package maintainers due to this being highly dependent on what the packages actually do with setuptools at runtime]
 +
* See if our python packaging guidelines need update (particularly the multi-version/python-setuptools/egg guidelines)
 +
* Check that the python-setuptools managed metadata has not changed incompatibly
 +
 
 +
=== Other developers ===
 +
 
 +
Owners of packages which BuildRequire or Require python-setuptools will need to:
 +
 
 +
* Test for buildtime and runtime breakage from new setuptools (as applicable to each package)
 +
* Fix FTBFS breakage from new setuptools
 +
* Fix runtime breakage from new setuptools
  
* Other developers: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
=== Release engineering ===
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
 
  
* Release engineering: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 
 
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuid required?  If a rel-eng ticket exists, add a link here.  -->
 
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuid required?  If a rel-eng ticket exists, add a link here.  -->
  
* Policies and guidelines: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
* Rebuild packages which BuildRequire: python-setuptools with the new setuptools
 +
** Probably not strictly necessary (will confirm when we test whether the package metadata format has changed) but it is good for testing since setuptools is a buildtime dep in many python packages.
 +
 
 +
=== Policies and guidelines ===
 
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
 
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
 +
 +
* Unknown at this time.  This will become apparent when the Change Owners test building with the new setuptools and if multi-version compat packages need any additional changes in order to install and run.
  
 
== Upgrade/compatibility impact ==
 
== Upgrade/compatibility impact ==
<!-- What happens to systems that have had a previous versions of Fedora installed and are updated to the version containing this change? Will anything require manual configuration or data migration? Will any existing functionality be no longer supported? -->
+
* No data will need migrating but user installed python modules that rely on the old version of python-setuptools may no longer work.
 +
* Probably no problems for packages installed with the older python-setuptools but change owners need to check that there are no incompatible changes to the package metadata managed by python-setuptools.
 +
 
 +
== How To Test ==
 +
 
 +
First install the updated python-setuptools package.
 +
 
 +
<pre>
 +
# yum update python-setuptools
 +
# rpm -q python-setuptools |grep 0.7
 +
[should display a python-seutptools 0.7 package]
 +
</pre>
 +
 
 +
=== Check that packages that need setuptools to build still build ===
 +
 
 +
Testing with an upstream tarball/checkout:
 +
 
 +
<pre>
 +
$ grep setuptools setup.py
 +
$ python setup.py install --root `pwd`/tmp/
 +
[This should work with no errors]
 +
</pre>
 +
 
 +
Testing with an rpm package:
 +
 
 +
<pre>
 +
$ fedpkg srpm python-$MODULE
 +
$ fedpkg scratch-build --srpm python-$MODULE*src.rpm
 +
</pre>
 +
 
 +
 
 +
=== Test multiversion python packages ===
 +
 
 +
Building should be the same test as above.
  
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
Runtime testing:
N/A (not a System Wide Change)
 
  
== How To Test ==
+
<pre>
<!-- This does not need to be a full-fledged document. Describe the dimensions of tests that this change implementation is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.
+
[ install both versions of the package]
 +
$ yum install python-cherrypy python-cherrypy2
 +
[test that both versions of the package work]
 +
[Install a package that uses the multiversion package as a dependency]
 +
$ yum install bodhi
 +
[test that the package still works (loads the correct version of the multi-version package)]
 +
</pre>
  
Remember that you are writing this how to for interested testers to use to check out your change implementation - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your change.
+
=== Check that packages that need setuptools to run still run ===
 +
<pre>
 +
$ yum install $PACKAGE
 +
$ rpm -q --requires $PACKAGE|grep python-setuptools
 +
[Displays that $PACKAGE needs python-setuptools]
 +
[Run the program and make use of plugins or other features that need python-setuptools]
 +
</pre>
  
A good "how to test" should answer these four questions:
+
=== Test python package installers with the new setuptools ===
 +
<pre>
 +
$ yum install python-pip
 +
$ pip install --user $OTHER_PACKAGE
 +
[Other python-pip commands]
 +
</pre>
  
0. What special hardware / data / etc. is needed (if any)?
+
Other packages in this category include python-zc-buildout (python-virtualenv if it finally unbundles pip and setuptools).
1. How do I prepare my system to test this change? What packages
 
need to be installed, config files edited, etc.?
 
2. What specific actions do I perform to check that the change is
 
working like it's supposed to?
 
3. What are the expected results of those actions?
 
-->
 
  
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 
N/A (not a System Wide Change)
 
  
 
== User Experience ==
 
== User Experience ==
<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
+
* Since this update includes API changes, developers who use setuptools as a library may notice changes.
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
 
N/A (not a System Wide Change)
+
See the following pages for details:
 +
* http://pythonhosted.org/setuptools/merge.html
 +
* https://pypi.python.org/pypi/setuptools#changes
 +
 
 +
Note that our current pyhton-setuptools package is the distribute fork so the merge changes are relative to that.
  
 
== Dependencies ==
 
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->
 
  
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
To see the dependent packages run these repoquery commands:
N/A (not a System Wide Change)
+
 
 +
<pre>
 +
[Runtime]
 +
$ repoquery -q --whatrequires python-setuptools
 +
[103 packages]
 +
[Buildtime]
 +
$ repoquery --enablerepo rawhide-source --archlist=src --whatrequires python-setuptools > tmp.lst
 +
$ repoquery --enablerepo rawhide-source --archlist=src --whatrequires python-setuptools-devel >> tmp.lst
 +
$ sort tmp.lst | uniq > setuptools-br.lst
 +
[918 packages]
 +
</pre>
 +
 
  
 
== Contingency Plan ==
 
== Contingency Plan ==
 
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
 
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
 
 +
=== Contingency mechanism: (What to do?  Who will do it?) ===
 +
 
 +
Three alternate contingency mechanisms exist:
 +
 
 +
# Someone who has a package that cannot be ported to the new python-setuptools will need to submit a backwards compat package for the older version of python-setuptools.  The Change Owners can commit to reviewing the package but not to being the package owner for the new package.
 +
# The packages that have not been ported can be dropped from the distribution.
 +
# The new setuptools package can be reverted for F20.
 +
 
 +
Deciding which of these to perform depends on the results of testing (which will show how many packages suffer from issues, whether those packages are easy to port, whether the unported packages are essential to the distribution or partially unmaintained, and whether the changes to port other packages to the new setuptools would be hard to revert.)
 +
 
 
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
 
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
 
* Contingency deadline: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 
* Contingency deadline: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
+
 
* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
For option 1, adding a backwards compatible package.  We would want to add that early enough to be able to make sure that the package is usable by the unported packages.  Soon after alpha would be ideal.
 +
 
 +
For option 2, beta freeze -- but we'd need to evaluate that the packages to be dropped are non-essential to the distribution much earlier.  Probable at alpha freeze.
 +
 
 +
For option 3, it depends on how difficult it would be to back out changes for all packages.  At this time, I don't think there will be a widespread need for patching or spec file changes, so this could be done at beta freeze.
 +
 
 +
 
 +
* Blocks release? Probably not.  Neither anaconda nor yum use this directly and it's not a critpath package.  However, we might want to reevaluate this is we find that it blocks building of something essential to the distribution (as it could prevent quick rebuilds of the packages needed for the distribution).
  
 
== Documentation ==
 
== Documentation ==
<!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
 
  
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
Upstream documentation:
N/A (not a System Wide Change)
+
 +
* http://pythonhosted.org/setuptools/merge.html
 +
* https://pypi.python.org/pypi/setuptools#changes
 +
 
 +
Notes from our testing will be added here or a separate wiki page if we encounter incompatibilities in updating.
  
 
== Release Notes ==
 
== Release Notes ==
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
 
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this change, indicate them here.  A link to upstream documentation will often satisfy this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release.
 
  
Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.  
+
* If the update does not cause any porting difficulties I would not release note this -- it will likely be just as invisible to end users as it is to us.
-->
+
 
 +
* If the update shows that the incompatibilities affect a lot of code then we should add something like this:
 +
 
 +
The version of the python-setuptools package has been updated to the 0.7.x series.  This release series merges the setuptools and distribute upstream projects which has introduced a variety of changes to the API and behaviour.  Please see https://fedoraproject.org/wikiChanges/Python_setuptools_0.7#Documentation for more details if this affects code you are writing or deploying.
 +
 
  
 
[[Category:ChangePageIncomplete]]
 
[[Category:ChangePageIncomplete]]

Revision as of 18:37, 10 June 2013


python-setuptools update to 0.7.x

Summary

Update to a new upstream release of python-setuptools that is not completely compatible with previous releases.

Owner

Current status

  • Targeted release: Fedora 20
  • Last updated: June 10 2013
  • Tracker bug: <will be assigned by the Wrangler>

Detailed Description

python-setuptools is used as a buildtime dependency in many packages. It is also a runtime dependency for some packages where it provides a way to load plugins, load data files, and specify what versions of dependent packages are needed. In Fedora, we also make use of its multiversioning features to install some backwards compatibility packages.

The upstream for the python setuptools package has been in a maintenance mode for many years. Just recently, it has become active and released a new version that is not completely API compatible. As the older versions won't see bug fixes anymore, we need to move to this new version but there are sure to be some usages that are broken by the new version.

Benefit to Fedora

The upstream activity is due to two competing projects merging: python-seutptools and python-distribute. The merger is a good thing for the python community as it unites the bugfixes and minor differences that have arisen in these two forks over the years. This makes it easier for upstreams to maintain their software. For us, it updates us to the latest version of the code which means that we'll be able to continue submitting patches as we find bugs in the code rather than having to maintain our own divergencies.

Scope

Proposal owners

  • Update the python-setuptools package from the current code-base (python-distribute-0.6.x) to python-setuptools-0.7.x
  • Rebuild packages with the new setuptools (Only necessary for testing)
  • Areas that may need special attention. Note that these are areas that we will test to understand the scope of changes required. It is not a promise to thoroughly test what is broken due to the update or to fix the breakage (although we will help)
    • Test python package installers with the new setuptools (pip, buildout)
    • Test python packages that build with setuptools for related ftbfs (we could do a mass rebuild of everything requiring python at runtime or refine our list to the packages that BuildRequire: python-setuptools)
    • Test packages where we are using setuptools to build multiversion (example: python-cherrypy2)
    • Test python packages that have a runtime requirement on setuptools [Note: The owners of this Change are going to leave most of this to individual package maintainers due to this being highly dependent on what the packages actually do with setuptools at runtime]
  • See if our python packaging guidelines need update (particularly the multi-version/python-setuptools/egg guidelines)
  • Check that the python-setuptools managed metadata has not changed incompatibly

Other developers

Owners of packages which BuildRequire or Require python-setuptools will need to:

  • Test for buildtime and runtime breakage from new setuptools (as applicable to each package)
  • Fix FTBFS breakage from new setuptools
  • Fix runtime breakage from new setuptools

Release engineering

  • Rebuild packages which BuildRequire: python-setuptools with the new setuptools
    • Probably not strictly necessary (will confirm when we test whether the package metadata format has changed) but it is good for testing since setuptools is a buildtime dep in many python packages.

Policies and guidelines

  • Unknown at this time. This will become apparent when the Change Owners test building with the new setuptools and if multi-version compat packages need any additional changes in order to install and run.

Upgrade/compatibility impact

  • No data will need migrating but user installed python modules that rely on the old version of python-setuptools may no longer work.
  • Probably no problems for packages installed with the older python-setuptools but change owners need to check that there are no incompatible changes to the package metadata managed by python-setuptools.

How To Test

First install the updated python-setuptools package.

# yum update python-setuptools
# rpm -q python-setuptools |grep 0.7
[should display a python-seutptools 0.7 package]

Check that packages that need setuptools to build still build

Testing with an upstream tarball/checkout:

$ grep setuptools setup.py
$ python setup.py install --root `pwd`/tmp/
[This should work with no errors]

Testing with an rpm package:

$ fedpkg srpm python-$MODULE
$ fedpkg scratch-build --srpm python-$MODULE*src.rpm


Test multiversion python packages

Building should be the same test as above.

Runtime testing:

[ install both versions of the package]
$ yum install python-cherrypy python-cherrypy2
[test that both versions of the package work]
[Install a package that uses the multiversion package as a dependency]
$ yum install bodhi
[test that the package still works (loads the correct version of the multi-version package)]

Check that packages that need setuptools to run still run

$ yum install $PACKAGE
$ rpm -q --requires $PACKAGE|grep python-setuptools
[Displays that $PACKAGE needs python-setuptools]
[Run the program and make use of plugins or other features that need python-setuptools]

Test python package installers with the new setuptools

$ yum install python-pip
$ pip install --user $OTHER_PACKAGE
[Other python-pip commands]

Other packages in this category include python-zc-buildout (python-virtualenv if it finally unbundles pip and setuptools).


User Experience

  • Since this update includes API changes, developers who use setuptools as a library may notice changes.

See the following pages for details:

Note that our current pyhton-setuptools package is the distribute fork so the merge changes are relative to that.

Dependencies

To see the dependent packages run these repoquery commands:

[Runtime]
$ repoquery -q --whatrequires python-setuptools
[103 packages]
[Buildtime]
$ repoquery --enablerepo rawhide-source --archlist=src --whatrequires python-setuptools > tmp.lst
$ repoquery --enablerepo rawhide-source --archlist=src --whatrequires python-setuptools-devel >> tmp.lst
$ sort tmp.lst | uniq > setuptools-br.lst
[918 packages]


Contingency Plan

Contingency mechanism: (What to do? Who will do it?)

Three alternate contingency mechanisms exist:

  1. Someone who has a package that cannot be ported to the new python-setuptools will need to submit a backwards compat package for the older version of python-setuptools. The Change Owners can commit to reviewing the package but not to being the package owner for the new package.
  2. The packages that have not been ported can be dropped from the distribution.
  3. The new setuptools package can be reverted for F20.

Deciding which of these to perform depends on the results of testing (which will show how many packages suffer from issues, whether those packages are easy to port, whether the unported packages are essential to the distribution or partially unmaintained, and whether the changes to port other packages to the new setuptools would be hard to revert.)

  • Contingency deadline: N/A (not a System Wide Change)

For option 1, adding a backwards compatible package. We would want to add that early enough to be able to make sure that the package is usable by the unported packages. Soon after alpha would be ideal.

For option 2, beta freeze -- but we'd need to evaluate that the packages to be dropped are non-essential to the distribution much earlier. Probable at alpha freeze.

For option 3, it depends on how difficult it would be to back out changes for all packages. At this time, I don't think there will be a widespread need for patching or spec file changes, so this could be done at beta freeze.


  • Blocks release? Probably not. Neither anaconda nor yum use this directly and it's not a critpath package. However, we might want to reevaluate this is we find that it blocks building of something essential to the distribution (as it could prevent quick rebuilds of the packages needed for the distribution).

Documentation

Upstream documentation:

Notes from our testing will be added here or a separate wiki page if we encounter incompatibilities in updating.

Release Notes

  • If the update does not cause any porting difficulties I would not release note this -- it will likely be just as invisible to end users as it is to us.
  • If the update shows that the incompatibilities affect a lot of code then we should add something like this:

The version of the python-setuptools package has been updated to the 0.7.x series. This release series merges the setuptools and distribute upstream projects which has introduced a variety of changes to the API and behaviour. Please see https://fedoraproject.org/wikiChanges/Python_setuptools_0.7#Documentation for more details if this affects code you are writing or deploying.