From Fedora Project Wiki
 
(11 intermediate revisions by 4 users not shown)
Line 1: Line 1:
<!-- Self Contained or System Wide Change Proposal?
+
= Enabling Python Generators =
Use this guide to determine to which category your proposed change belongs to.
 
 
 
Self Contained Changes are:
 
* changes to isolated/leaf package without the impact on other packages/rest of the distribution
 
* limited scope changes without the impact on other packages/rest of the distribution
 
* coordinated effort within SIG with limited impact outside SIG functional area, accepted by the SIG
 
 
 
System Wide Changes are:
 
* changes that does not fit Self Contained Changes category touching
 
* changes that require coordination within the distribution (for example mass rebuilds, release engineering or other teams effort etc.)
 
* changing system defaults
 
 
 
For Self Contained Changes, sections marked as "REQUIRED FOR SYSTEM WIDE CHANGES" are OPTIONAL but FESCo/Wrangler can request more details (especially in case the change proposal category is
 
improper or updated to System Wide category). For System Wide Changes all fields on this form are required for FESCo acceptance (when applies). 
 
 
 
We request that you maintain the same order of sections so that all of the change proposal pages are uniform.
 
-->
 
 
 
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
 
 
 
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
 
= Change Proposal Name <!-- The name of your change proposal --> =
 
  
 
== Summary ==
 
== Summary ==
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. -->
+
This change enables the ability to choose to use the Python module dependency generator for packages that provide Python Egg/Wheel metadata.
  
 
== Owner ==
 
== Owner ==
<!--
 
For change proposals to qualify as self-contained, owners of all affected packages need to be included here. Alternatively, a SIG can be listed as an owner if it owns all affected packages.
 
This should link to your home wiki page so we know who you are.
 
-->
 
 
* Name: [[User:ignatenkobrain|Igor Gnatenko]], [[User:ngompa|Neal Gompa]]
 
* Name: [[User:ignatenkobrain|Igor Gnatenko]], [[User:ngompa|Neal Gompa]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
 
 
* Email: ignatenkobrain@fedoraproject.org, ngompa13@gmail.com
 
* Email: ignatenkobrain@fedoraproject.org, ngompa13@gmail.com
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
+
* Release notes ticket: [https://pagure.io/fedora-docs/release-notes/issue/99 #99]
 
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
 
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
 
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
 
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
Line 54: Line 27:
 
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
 
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
 
-->
 
-->
* Tracker bug: <will be assigned by the Wrangler>
+
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1541953 #1541953]
  
 
== Detailed Description ==
 
== Detailed Description ==
 +
There is RPM [http://rpm.org/user_doc/dependency_generators.html dependency generator] which is able to automatically add Requires/Provides and other types of dependencies based on egg/wheel metadata. The part which is generating Provides has been used in Fedora since Jun 2016 which means by now all packages which provide egg/wheel metadata have Provides: pythonX.Ydist(xyz) added automatically. With this change proposal we allow people to opt-in for using automatic generation of Requires.
 +
 +
All dependencies which are added are generated out of metadata which means in order to fix them - you need to fix metadata (usually setup.py or requirements.txt).
 +
 +
Other distributions (such as Mageia, OpenMandriva, and PLD) have been using this dependency generator for long time already.
  
 +
In F29 we plan to propose to make generator opt-out.
 
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
 
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
  
 
== Benefit to Fedora ==
 
== Benefit to Fedora ==
 +
* Packagers do not need to specify runtime dependencies manually anymore
 +
* No more errors in runtime due to missing dependencies (see example below)
 +
 +
Packager of python-fasteners forgot to add Requires: python3-monotonic >= 0.1 so when anyone does "import fasteners", it fails with traceback:
 +
 +
<pre>
 +
Traceback (most recent call last):
 +
  File "/builddir/build/BUILD/jenkins-job-builder-2.0.0/tests/yamlparser/test_yamlparser.py", line 42, in test_incorrect_template_dimensions
 +
    reg = registry.ModuleRegistry(config)
 +
  File "/builddir/build/BUILD/jenkins-job-builder-2.0.0/jenkins_jobs/registry.py", line 50, in __init__
 +
    Mod = entrypoint.load()
 +
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2407, in load
 +
    self.require(*args, **kwargs)
 +
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2430, in require
 +
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
 +
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 870, in resolve
 +
    raise DistributionNotFound(req, requirers)
 +
pkg_resources.DistributionNotFound: The 'monotonic>=0.1' distribution was not found and is required by fasteners
 +
</pre>
  
+
With automatic dependency management this should never happen.
 
 
 
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->
 
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->
  
 
== Scope ==
 
== Scope ==
* Proposal owners:
+
* Proposal owners: Add macros to python-rpm-macros package which enables dependency generator.
 
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
 
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
  
* Other developers: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
* Other developers: Packagers are advised to replace their hand-crafted list of dependencies on python libraries with simple macros.
 
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
 
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
  
* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
+
* Release engineering: [https://pagure.io/releng/issue/7276 #7276] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
 
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
 
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
 
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
 
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
Line 79: Line 76:
 
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->
 
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->
  
* Policies and guidelines: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+
* Policies and guidelines: Packaging:Python Guidelines need to reference how to turn on the feature (simple variable override). [https://pagure.io/packaging-committee/issue/740 #740].
 
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
 
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
  
Line 86: Line 83:
  
 
== Upgrade/compatibility impact ==
 
== Upgrade/compatibility impact ==
 +
Some new dependencies might be automatically added, but this is rather good because it fixes real bugs.
 
<!-- What happens to systems that have had a previous versions of Fedora installed and are updated to the version containing this change? Will anything require manual configuration or data migration? Will any existing functionality be no longer supported? -->
 
<!-- What happens to systems that have had a previous versions of Fedora installed and are updated to the version containing this change? Will anything require manual configuration or data migration? Will any existing functionality be no longer supported? -->
  
Line 92: Line 90:
  
 
== How To Test ==
 
== How To Test ==
 +
# Make sure that python-rpm-macros >= 3-25 is installed
 +
# Remove all requires for python modules
 +
# Add following line to spec: `%?python_enable_dependency_generator`
 +
# Rebuild package
 +
 +
Every dependency mentioned in egg/wheel (e.g. generated by setuptools/distutils from setup.py) should be automatically added.
 
<!-- This does not need to be a full-fledged document. Describe the dimensions of tests that this change implementation is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.  
 
<!-- This does not need to be a full-fledged document. Describe the dimensions of tests that this change implementation is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.  
  
Line 105: Line 109:
 
3. What are the expected results of those actions?
 
3. What are the expected results of those actions?
 
-->
 
-->
 
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 
N/A (not a System Wide Change)
 
  
 
== User Experience ==
 
== User Experience ==
Line 143: Line 144:
 
-->
 
-->
  
[[Category:ChangePageIncomplete]]
+
[[Category:ChangeAcceptedF28]]
 
<!-- When your change proposal page is completed and ready for review and announcement -->
 
<!-- When your change proposal page is completed and ready for review and announcement -->
 
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
 
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->

Latest revision as of 14:57, 2 March 2018

Enabling Python Generators

Summary

This change enables the ability to choose to use the Python module dependency generator for packages that provide Python Egg/Wheel metadata.

Owner

Current status

Detailed Description

There is RPM dependency generator which is able to automatically add Requires/Provides and other types of dependencies based on egg/wheel metadata. The part which is generating Provides has been used in Fedora since Jun 2016 which means by now all packages which provide egg/wheel metadata have Provides: pythonX.Ydist(xyz) added automatically. With this change proposal we allow people to opt-in for using automatic generation of Requires.

All dependencies which are added are generated out of metadata which means in order to fix them - you need to fix metadata (usually setup.py or requirements.txt).

Other distributions (such as Mageia, OpenMandriva, and PLD) have been using this dependency generator for long time already.

In F29 we plan to propose to make generator opt-out.

Benefit to Fedora

  • Packagers do not need to specify runtime dependencies manually anymore
  • No more errors in runtime due to missing dependencies (see example below)

Packager of python-fasteners forgot to add Requires: python3-monotonic >= 0.1 so when anyone does "import fasteners", it fails with traceback:

Traceback (most recent call last):
  File "/builddir/build/BUILD/jenkins-job-builder-2.0.0/tests/yamlparser/test_yamlparser.py", line 42, in test_incorrect_template_dimensions
    reg = registry.ModuleRegistry(config)
  File "/builddir/build/BUILD/jenkins-job-builder-2.0.0/jenkins_jobs/registry.py", line 50, in __init__
    Mod = entrypoint.load()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2407, in load
    self.require(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2430, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 870, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'monotonic>=0.1' distribution was not found and is required by fasteners

With automatic dependency management this should never happen.

Scope

  • Proposal owners: Add macros to python-rpm-macros package which enables dependency generator.
  • Other developers: Packagers are advised to replace their hand-crafted list of dependencies on python libraries with simple macros.
  • Release engineering: #7276 (a check of an impact with Release Engineering is needed)
  • Policies and guidelines: Packaging:Python Guidelines need to reference how to turn on the feature (simple variable override). #740.
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Some new dependencies might be automatically added, but this is rather good because it fixes real bugs.

N/A (not a System Wide Change)

How To Test

  1. Make sure that python-rpm-macros >= 3-25 is installed
  2. Remove all requires for python modules
  3. Add following line to spec: %?python_enable_dependency_generator
  4. Rebuild package

Every dependency mentioned in egg/wheel (e.g. generated by setuptools/distutils from setup.py) should be automatically added.

User Experience

N/A (not a System Wide Change)

Dependencies

N/A (not a System Wide Change)

Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
  • Contingency deadline: N/A (not a System Wide Change)
  • Blocks release? N/A (not a System Wide Change), Yes/No
  • Blocks product? product

Documentation

N/A (not a System Wide Change)

Release Notes