From Fedora Project Wiki
m (Thrnciar moved page Changes/Packaging 23+ to Changes/Packaging 22+: Typo in package version, should be 22+ instead of 23+)
(Add trackers)
 
(33 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->


= Packaging 23+ <!-- The name of your change proposal --> =
= Update python-packaging to version >= 22 <!-- The name of your change proposal --> =


== Summary ==
== Summary ==
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->


Update to a new upstream release of python-packaging contains a breaking change. Since version 23+ upstream removed support for [https://github.com/pypa/packaging/pull/407 LegacySpecifier and LegacyVersion] some packages will break. This is a breaking change and projects are encouraged to adhere to the grammar defined in [https://peps.python.org/pep-0508/#complete-grammar PEP 508].  
Update to a new upstream release of python-packaging contains a breaking change. Since version 22+ upstream removed support for [https://github.com/pypa/packaging/pull/407 LegacySpecifier and LegacyVersion] some packages will break. This is a breaking change and projects are encouraged to use versions adherent to [https://peps.python.org/pep-0440/ PEP 440].
 
Note: Currently, we plan to update to 23.0, but this can be updated even further. The change proposal explicitly mentions version 22+ because that version removed support for LegacyVersions and legacy version specifiers.


== Owner ==
== Owner ==
Line 13: Line 15:
This should link to your home wiki page so we know who you are.  
This should link to your home wiki page so we know who you are.  
-->
-->
* Name: [[User:thrnciar| Tomáš Hrnčiar]]  
* Name: [[User:thrnciar| Tomáš Hrnčiar]], [[User:mhroncok| Miro Hrončok]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
* Email: thrnciar@redhat.com
* Email: thrnciar@redhat.com, mhroncok@redhat.com
* Name: [[User:mhroncok| Miro Hrončok]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
* Email: mhroncok@redhat.com


<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
Line 26: Line 25:


== Current status ==
== Current status ==
[[Category:ChangePageIncomplete]]
[[Category:ChangeAcceptedF38]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
Line 44: Line 43:
ON_QA -> change is fully code complete
ON_QA -> change is fully code complete
-->
-->
* FESCo issue: <will be assigned by the Wrangler>
* [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/O5SGZXG5VAGJDV5TAISKZBYHTRI4F5QX/ devel thread]
* Tracker bug: <will be assigned by the Wrangler>
* FESCo issue: [https://pagure.io/fesco/issue/2949 #2949]
* Release notes tracker: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=2166942 #2166942]
* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/966 #966]


== Detailed Description ==
== Detailed Description ==
Line 64: Line 64:
     raise InvalidVersion(f"Invalid version: '{version}'")
     raise InvalidVersion(f"Invalid version: '{version}'")
  packaging.version.InvalidVersion: Invalid version: 'This is a completely random string'
  packaging.version.InvalidVersion: Invalid version: 'This is a completely random string'
This "feature" has been deprecated for nearly two years now.
PyPI has not permitted uploading packages with invalid versions for even more years. The latest versions of pip should be rejecting/erroring out on wheels with such versions as well. The stricter metadata validation helps pip's dependency resolver's logic, along with helping the broader ecology avoid needing to deal with outside-of-standard tooling/behaviours.
Possible failures caused by upgraded python-packaging:
* pinned version of python-packaging
**cekit - https://bugzilla.redhat.com/show_bug.cgi?id=2162433
**python-limits - https://src.fedoraproject.org/rpms/python-limits/pull-request/5
* tests containing unsupported versions need to be adjusted
**copr-backend - https://bugzilla.redhat.com/show_bug.cgi?id=2162436
* some packages provide incompatible versions and python-rpm-generators are unable to parse them. See PRs links below for inspiration on how to fix them.
**pcs - https://bugzilla.redhat.com/show_bug.cgi?id=2166266
**pyodbc - https://src.fedoraproject.org/rpms/pyodbc/pull-request/4
**python-btchip - https://src.fedoraproject.org/rpms/python-btchip/pull-request/1
**python-dipy
**python-dropbox
**python-haversion - https://src.fedoraproject.org/rpms/python-haversion/pull-request/1
**python-lacrosse - https://src.fedoraproject.org/rpms/python-lacrosse/pull-request/1
**python-pdir2 - the sed in `%prep` needs to be [https://src.fedoraproject.org/rpms/python-pdir2/c/bc914bbd3cfb5e3853c666af9c855adc51e8d47c?branch=rawhide adjusted] not to create invalid `>=4.2.*` requirements
**python-pvc
**python-pytest-httpx
See also https://discuss.python.org/t/22782/19 for the exact errors of most of the listed packages.
* other
**python-httplib2 - uses pyparsing in tests but does not BuildRequire it directly - https://src.fedoraproject.org/rpms/python-httplib2/pull-request/9
**obs-service-set_version - tests convert LegacyVersions to RPM syntax and the result is different now (not sure?)
* unknown
**python-google-cloud-bigquery - package does not build in rawhide
=== How to deal with failures in Python RPM dependency generators ===
packaging is used in the Python RPM dependency generators (for `python3dist()` and `python3.11dist()` Provides, Requires and BuildRequires).
When the packaged software has now invalid versions or comparators, you might need to fix the package.
Example problem:
packaging.version.InvalidVersion: Invalid version: 'main'
See if the version of the Python package isn't failing to be determined automatically and provide the missing information, see https://src.fedoraproject.org/rpms/python-haversion/pull-request/1 or https://src.fedoraproject.org/rpms/python-lacrosse/pull-request/1 and https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_version_warning
Example problem:
  packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
    typing-extensions>=4.2.*
                    \~\~\~\~\~^
(Backslashes added to the example as a naïve attempt to escape the dashes form this wiki, ignore them).
Adjust the requirement not to use `.*` in versions when `<`, `<=`, `>`, or `>=` is used, see https://discuss.python.org/t/22782


== Feedback ==
== Feedback ==
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
We are aware that we missed the Change proposal submission deadline, originally we planned to ship this as a regular package update. While doing an impact check we identified ~20 affected packages and we decided to write this Change proposal for better transparency. We believe there is enough time to complete this Change in time for the Completion deadline. We are prepared to postpone this Change to Fedora 39 if FESCo decides so.


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 96: Line 153:
     https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
     https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
-->
-->
Packaging 22+ contains a handwritten parser for parsing requirements and markers. Thanks to this, packaging has dropped a runtime dependency on pyparsing and from now on is not depending on any package on runtime. This will simplify bootstrapping of the next Python.


== Scope ==
== Scope ==
* Proposal owners:
* Proposal owners: update python-packaging to 23.x.x, provide help
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> report problems to the upstream and backport patch to the affected packages. The impact can be tested using [https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/packages/ COPR repository] where Packaging 23+ has been built.
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Release engineering: N/A (not needed for this Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
Line 139: Line 198:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 
# Find the package you want to fix in this [https://copr.fedorainfracloud.org/coprs/thrnciar/python-packaging/packages/ COPR repository] and check the build logs to determine the failure cause.
# Patch package so Provides() provides correct version.
# When patching the package, you can test it using the same copr repository where the latest version of python-packaging has been built.


== User Experience ==
== User Experience ==
Line 152: Line 213:
  - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system.
  - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system.
-->
-->
Regular distro users shouldn't notice any change in python-packaging behaviour, except for packages that use `LegacyVersion` or `LegacySpecifier`. Such packages will fail with `packaging.version.InvalidVersion: Invalid version: 'This is a completely random string'` and should be fixed by their maintainers.


== Dependencies ==
== Dependencies ==
Line 162: Line 224:


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: (What to do?  Who will do it?) revert the update and bump epoch <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: beta freeze <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->




Line 173: Line 235:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
https://github.com/pypa/packaging/blob/main/CHANGELOG.rst


== Release Notes ==
== Release Notes ==

Latest revision as of 15:25, 3 February 2023


Update python-packaging to version >= 22

Summary

Update to a new upstream release of python-packaging contains a breaking change. Since version 22+ upstream removed support for LegacySpecifier and LegacyVersion some packages will break. This is a breaking change and projects are encouraged to use versions adherent to PEP 440.

Note: Currently, we plan to update to 23.0, but this can be updated even further. The change proposal explicitly mentions version 22+ because that version removed support for LegacyVersions and legacy version specifiers.

Owner


Current status

Detailed Description

>>> # before 22.0
>>> packaging.version.parse("This is a completely random string")
<LegacyVersion('This is a completely random string')>
>>> # after 22.0
>>> packaging.version.parse("This is a completely random string")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "[...]/.venv/lib/python3.10/site-packages/packaging/version.py", line 52, in parse
    return Version(version)
  File "[...]/.venv/lib/python3.10/site-packages/packaging/version.py", line 197, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: 'This is a completely random string'

This "feature" has been deprecated for nearly two years now.

PyPI has not permitted uploading packages with invalid versions for even more years. The latest versions of pip should be rejecting/erroring out on wheels with such versions as well. The stricter metadata validation helps pip's dependency resolver's logic, along with helping the broader ecology avoid needing to deal with outside-of-standard tooling/behaviours.

Possible failures caused by upgraded python-packaging:

See also https://discuss.python.org/t/22782/19 for the exact errors of most of the listed packages.

  • unknown
    • python-google-cloud-bigquery - package does not build in rawhide

How to deal with failures in Python RPM dependency generators

packaging is used in the Python RPM dependency generators (for python3dist() and python3.11dist() Provides, Requires and BuildRequires).

When the packaged software has now invalid versions or comparators, you might need to fix the package.

Example problem:

packaging.version.InvalidVersion: Invalid version: 'main'

See if the version of the Python package isn't failing to be determined automatically and provide the missing information, see https://src.fedoraproject.org/rpms/python-haversion/pull-request/1 or https://src.fedoraproject.org/rpms/python-lacrosse/pull-request/1 and https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_version_warning

Example problem:

 packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
    typing-extensions>=4.2.*
                    \~\~\~\~\~^

(Backslashes added to the example as a naïve attempt to escape the dashes form this wiki, ignore them).

Adjust the requirement not to use .* in versions when <, <=, >, or >= is used, see https://discuss.python.org/t/22782

Feedback

We are aware that we missed the Change proposal submission deadline, originally we planned to ship this as a regular package update. While doing an impact check we identified ~20 affected packages and we decided to write this Change proposal for better transparency. We believe there is enough time to complete this Change in time for the Completion deadline. We are prepared to postpone this Change to Fedora 39 if FESCo decides so.

Benefit to Fedora

Packaging 22+ contains a handwritten parser for parsing requirements and markers. Thanks to this, packaging has dropped a runtime dependency on pyparsing and from now on is not depending on any package on runtime. This will simplify bootstrapping of the next Python.

Scope

  • Proposal owners: update python-packaging to 23.x.x, provide help
  • Other developers: report problems to the upstream and backport patch to the affected packages. The impact can be tested using COPR repository where Packaging 23+ has been built.
  • Release engineering: N/A (not needed for this Change)
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Objectives:

Upgrade/compatibility impact

How To Test

  1. Find the package you want to fix in this COPR repository and check the build logs to determine the failure cause.
  2. Patch package so Provides() provides correct version.
  3. When patching the package, you can test it using the same copr repository where the latest version of python-packaging has been built.

User Experience

Regular distro users shouldn't notice any change in python-packaging behaviour, except for packages that use LegacyVersion or LegacySpecifier. Such packages will fail with packaging.version.InvalidVersion: Invalid version: 'This is a completely random string' and should be fixed by their maintainers.

Dependencies

Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) revert the update and bump epoch
  • Contingency deadline: beta freeze
  • Blocks release? No


Documentation

https://github.com/pypa/packaging/blob/main/CHANGELOG.rst

Release Notes