From Fedora Project Wiki
 
(21 intermediate revisions by 3 users not shown)
Line 22: Line 22:
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
= Change Proposal Name <!-- The name of your change proposal --> =
= Change Proposal Name <!-- The name of your change proposal --> =
 
Force the C.UTF-8 locale when Python 3 is run under the C locale.
== Summary ==
== Summary ==
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. -->
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. -->
Force C.UTF-8 when Python 3 is run under the C locale
The standalone Python 3.6 binary will automatically attempt to coerce the C locale to C.UTF-8, unless the new PYTHONALLOWCLOCALE environment variable is set to 1.


== Owner ==
== Owner ==
Line 64: Line 64:


When run under the C locale, Python 3 doesn't work properly on systems where UTF-8 is the correct encoding for interacting with the rest of the system.
When run under the C locale, Python 3 doesn't work properly on systems where UTF-8 is the correct encoding for interacting with the rest of the system.
This proposed change for Python 3 packaged in Fedora assumes the current locale is misconfigured when it detects that "LC_TYPE" refers to the "C" locale, and in that case, prints a warnings to stderr and forces the use of the C.UTF-8 locale instead.


To avoid unintended side effects, this change takes effect *solely* when Python 3 is invoked by the user—nothing changes in cases where CPython is used as a dynamically linked library.
Also, an "escape hatch" environment variable is provided to revert to the old behavior.


This is described in detail by Armin Ronacher in the click documentation: http://click.pocoo.org/5/python3/#python-3-surrogate-handling
The effects of the problem for a popular CLI construction library are explained on Armin Ronacher's blog: http://click.pocoo.org/5/python3/#python-3-surrogate-handling
<br>
The problem is described in detail in PEP (Python Enhancement Proposal) 538: https://www.python.org/dev/peps/pep-0538/


The PEP has no strong opposition upstream, but acceptance is held back by considerations for other systems (e.g. those that do not have a C.UTF-8 locale). These considerations do not apply for Fedora.
We are commited to follow upstream conversation and recommendations around PEP 538.
Also, we hope that testing this in Fedora will help upstream find remaining distro integration issues.


This proposed change for the system Python 3 that assumes the current process is misconfigured when it detects that "LC_CTYPE" refers to the "C" locale, and in that case, prints a warnings to stderr and forces the use of the C.UTF-8 locale instead.
The current version of the PEP specifically encourages backporting to Python 3.6.0: https://www.python.org/dev/peps/pep-0538/#backporting-to-python-3-6-0


To avoid unintended side effects, it *solely* changes the actual python3.6 command line utility - nothing changes for cases where CPython is used as a dynamically linked library.


Behaviour with the patch:
Behaviour with the patch:


$ LANG=C python -c 'import click; cli = click.command()(lambda:None); cli()'


Python detected LC_CTYPE=C. Setting LC_ALL & LANG to C.UTF-8.
$ LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'
 
Python detected LC_CTYPE=C. Setting LC_ALL & LANG to C.UTF-8 (set PYTHONALLOWLOCALE to disable this locale coercion behaviour).


Behaviour without the patch:
Behaviour without the patch:
Line 83: Line 91:
$ LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'
$ LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'


Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 716, in __call__
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
    return self.main(*args, **kwargs)
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 675, in main
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 675, in main
    _verify_python3_env()
    _verify_python3_env()
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env
    'mitigation steps.' + extra)
    'mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.
This system supports the C.UTF-8 locale which is recommended.
You might be able to resolve your issue by exporting the
following environment variables:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8


This system supports the C.UTF-8 locale which is recommended.
== Benefit to Fedora ==
You might be able to resolve your issue by exporting the
following environment variables:


export LC_ALL=C.UTF-8
Linux container technologies like Docker, Kubernetes, and OpenShift are best known for their use in web service development, the related container formats and execution models are also being adopted for Linux command line application development. Technologies like Gnome Flatpak further aim to bring these same techniques to Linux GUI application development.
export LANG=C.UTF-8


== Benefit to Fedora ==
When using Python 3 for application development in these contexts, it isn't uncommon to see various text encoding related errors.
 
One good example is the base Docker image for Fedora which does not have any specific locale set, so the POSIX locale is set by default, which is an alias for the ASCII-based default C locale.


The simplest way to get Python 3 (regardless of the exact version) to behave sensibly in Fedora containers is to run it in the C-UTF-8 locale that the distro provides.
    
    
Another benefit, directly related to packaging rpm's, is that the locale inside mock chroots will be automatically set to C.UTF-8 when the python interpreter is invoked during rpmbuild (e.g. when running tests),
so packagers won't have to rely on various [http://pkgs.fedoraproject.org/cgit/rpms/python-argh.git/tree/python-argh.spec#n106 hacks] in order to export the appropriate locale required for their package to build.
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->


Line 110: Line 125:
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Other developers: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
A patch will be backported to Fedora's [https://admin.fedoraproject.org/pkgdb/package/rpms/python3/ Python 3.6] from the upstream [https://www.python.org/dev/peps/pep-0538/ PEP 538] which targets Python 3.7. The patch that will be backported is explicitly targeting Fedora, as the upstream equivalent one will have to be compatible with Windows, macOS, Solaris etc. The Fedora specific patch is provided by [[User:ncoghlan|Nick Coghlan]], a Python core developer.
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engeneering is needed) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
* Release engineering: No impact with release engineering <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuid required?  include a link to the releng issue.  
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuid required?  include a link to the releng issue.  
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->
* Policies and guidelines: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
* Trademark approval: N/A (not needed for this Change)
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->


== Upgrade/compatibility impact ==
== Upgrade/compatibility impact ==
Line 148: Line 153:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)  
Rawhide users can already test this change by enabling the force_c.utf8_python3  copr repository where a working build exists.
 
dnf copr enable cstratak/force_c.utf8_python3
 
Then install python3-click:
dnf install python3-click
 
And then run this command to check if python3 coerces the locale correctly to c.utf8:
LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'
 
The previous behaviour can also be restored by setting the PYTHONALLOWLOCALE variable to 1, which disables the locale coersion:
PYTHONALLOWLOCALE=1 LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'


== User Experience ==
== User Experience ==
Line 159: Line 176:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
No new dependencies will be added.


== Contingency Plan ==
== Contingency Plan ==


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: Remove the patch from Python 3.6 <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: If users experience any issues this change can be reverted at any time <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks product? product <!-- Applicable for Changes that blocks specific product release/Fedora.next -->
* Blocks product? No <!-- Applicable for Changes that blocks specific product release/Fedora.next -->


== Documentation ==
== Documentation ==

Latest revision as of 10:16, 24 February 2017


Change Proposal Name

Force the C.UTF-8 locale when Python 3 is run under the C locale.

Summary

The standalone Python 3.6 binary will automatically attempt to coerce the C locale to C.UTF-8, unless the new PYTHONALLOWCLOCALE environment variable is set to 1.

Owner

Current status

  • Targeted release: Fedora 26
  • Last updated: 2017-02-24
  • Tracker bug: <will be assigned by the Wrangler>

Detailed Description

When run under the C locale, Python 3 doesn't work properly on systems where UTF-8 is the correct encoding for interacting with the rest of the system. This proposed change for Python 3 packaged in Fedora assumes the current locale is misconfigured when it detects that "LC_TYPE" refers to the "C" locale, and in that case, prints a warnings to stderr and forces the use of the C.UTF-8 locale instead.

To avoid unintended side effects, this change takes effect *solely* when Python 3 is invoked by the user—nothing changes in cases where CPython is used as a dynamically linked library. Also, an "escape hatch" environment variable is provided to revert to the old behavior.

The effects of the problem for a popular CLI construction library are explained on Armin Ronacher's blog: http://click.pocoo.org/5/python3/#python-3-surrogate-handling
The problem is described in detail in PEP (Python Enhancement Proposal) 538: https://www.python.org/dev/peps/pep-0538/

The PEP has no strong opposition upstream, but acceptance is held back by considerations for other systems (e.g. those that do not have a C.UTF-8 locale). These considerations do not apply for Fedora. We are commited to follow upstream conversation and recommendations around PEP 538. Also, we hope that testing this in Fedora will help upstream find remaining distro integration issues.

The current version of the PEP specifically encourages backporting to Python 3.6.0: https://www.python.org/dev/peps/pep-0538/#backporting-to-python-3-6-0


Behaviour with the patch:


$ LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'

Python detected LC_CTYPE=C. Setting LC_ALL & LANG to C.UTF-8 (set PYTHONALLOWLOCALE to disable this locale coercion behaviour).

Behaviour without the patch:

$ LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 675, in main
    _verify_python3_env()
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env
    'mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.
This system supports the C.UTF-8 locale which is recommended.
You might be able to resolve your issue by exporting the
following environment variables:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8

Benefit to Fedora

Linux container technologies like Docker, Kubernetes, and OpenShift are best known for their use in web service development, the related container formats and execution models are also being adopted for Linux command line application development. Technologies like Gnome Flatpak further aim to bring these same techniques to Linux GUI application development.

When using Python 3 for application development in these contexts, it isn't uncommon to see various text encoding related errors.

One good example is the base Docker image for Fedora which does not have any specific locale set, so the POSIX locale is set by default, which is an alias for the ASCII-based default C locale.

The simplest way to get Python 3 (regardless of the exact version) to behave sensibly in Fedora containers is to run it in the C-UTF-8 locale that the distro provides.

Another benefit, directly related to packaging rpm's, is that the locale inside mock chroots will be automatically set to C.UTF-8 when the python interpreter is invoked during rpmbuild (e.g. when running tests), so packagers won't have to rely on various hacks in order to export the appropriate locale required for their package to build.


Scope

  • Proposal owners:

A patch will be backported to Fedora's Python 3.6 from the upstream PEP 538 which targets Python 3.7. The patch that will be backported is explicitly targeting Fedora, as the upstream equivalent one will have to be compatible with Windows, macOS, Solaris etc. The Fedora specific patch is provided by Nick Coghlan, a Python core developer.

  • Release engineering: No impact with release engineering

Upgrade/compatibility impact

N/A (not a System Wide Change)

How To Test

Rawhide users can already test this change by enabling the force_c.utf8_python3 copr repository where a working build exists.

dnf copr enable cstratak/force_c.utf8_python3 

Then install python3-click:

dnf install python3-click

And then run this command to check if python3 coerces the locale correctly to c.utf8:

LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()' 

The previous behaviour can also be restored by setting the PYTHONALLOWLOCALE variable to 1, which disables the locale coersion:

PYTHONALLOWLOCALE=1 LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'

User Experience

N/A (not a System Wide Change)

Dependencies

No new dependencies will be added.

Contingency Plan

  • Contingency mechanism: Remove the patch from Python 3.6
  • Contingency deadline: If users experience any issues this change can be reverted at any time
  • Blocks release? No
  • Blocks product? No

Documentation

N/A (not a System Wide Change)

Release Notes