From Fedora Project Wiki
 
(50 intermediate revisions by 7 users not shown)
Line 1: Line 1:
{{admon/important | Comments and Explanations | The page source contains comments providing guidance to fill out each section. They are invisible when viewing this page. To read it, choose the "edit" link.<br/> '''Copy the source to a ''new page'' before making changes!  DO NOT EDIT THIS TEMPLATE FOR YOUR CHANGE PROPOSAL.'''}}
<!-- Self Contained or System Wide Change Proposal?
<!-- Self Contained or System Wide Change Proposal?
Use this guide to determine to which category your proposed change belongs to.
Use this guide to determine to which category your proposed change belongs to.
Line 34: Line 32:
This should link to your home wiki page so we know who you are.  
This should link to your home wiki page so we know who you are.  
-->
-->
* Name: [[User:Mfabian| Mike Fabian]][[User:Siddhesh| Siddhesh Poyarekar]]
* Name: [[User:Mfabian| Mike Fabian]],[[User:Siddhesh| Siddhesh Poyarekar]], [[User:Codonell| Carlos O’Donell]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
* Email: <mfabian At redhat DOT com, spoyarek AT redhat DOT com
* Email: <mfabian At redhat DOT com>, <spoyarek AT redhat DOT com>, <codonell AT redhat DOT com>
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
Line 47: Line 45:


== Current status ==
== Current status ==
* Targeted release: [[Releases/23 | Fedora 23 ]]  
* Targeted release: [[Releases/24 | Fedora 24 ]]  
* Last updated: (2015-06-17)
* Last updated: (2015-12-08)
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
Bugzilla states meaning as usual:
Bugzilla states meaning as usual:
Line 57: Line 55:
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
-->
-->
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1238406 Bug 1238406]


== Detailed Description ==
== Detailed Description ==
Line 63: Line 61:
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
Currently the file /usr/lib/locale/locale-archive contains
Currently the file /usr/lib/locale/locale-archive contains
*all* locales and is thus huge (103 megabytes).
''all'' locales and is thus huge (103MB).


For small systems (and containers) it would be useful to be able
For small systems (and containers) it would be useful to be able
Line 101: Line 99:


Anaconda then needs to be changed to handle such language packages.
Anaconda then needs to be changed to handle such language packages.
Other [https://fedoraproject.org/wiki/I18N/Locale_Subpackaging implementation options]


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 110: Line 110:


== Scope ==
== Scope ==
* Proposal owners:
* '''Proposal owners:'''
    1. Figure out the best approach to to install/uninstall locales
*# Figure out the best approach to to install/uninstall locales
    2. Make sure that locales added manually by the user are not destroyed (currently they are lost when glibc is updated)
*# Make sure that locales added manually by the user are not destroyed (currently they are lost when glibc is updated)
      
      
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* '''Other developers:''' <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
Anaconda needs to be made aware of the new approach to handle installation/uninstallation of locales
Anaconda needs to be made aware of the new approach to handle installation/uninstallation of locales
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* '''Release engineering:''' <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
I am not sure whether this has affects release engineering, probably the
I am not sure whether this has affects release engineering, probably the
packages in the install image change when parts are split out of glibc-common.
packages in the install image change when parts are split out of glibc-common.
Line 126: Line 126:
Please work with releng prior to feature submission, and ensure that someone is on board to do any process development work and testing; don't just assume that a bullet point in a change puts someone else on the hook  -->
Please work with releng prior to feature submission, and ensure that someone is on board to do any process development work and testing; don't just assume that a bullet point in a change puts someone else on the hook  -->


* Policies and guidelines: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* '''Policies and guidelines:''' <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
No, this change does not require updates to policies and guidelines.
No, this change does not require updates to policies and guidelines.
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->


* Trademark approval: N/A (not needed for this Change)
* '''Trademark approval:''' not needed for this Change
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->


Line 137: Line 137:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
The user should not notice any change after the update. All locales which were installed before
should still be installed after the update.
 
After the update it should be possible to uninstall locales for unneeded languages but this is optional.


== How To Test ==
== How To Test ==
Line 155: Line 158:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
Please visit [https://fedoraproject.org/wiki/QA:Glibc_locale_subpackaging '''How to Test'''] page and participate in Fedora 24 i18n testday
# Try to install/uninstall packages containing locales for some languages.
#* For example, <pre>dnf install glibc-langpack-hi</pre> should install the Hindi locales <pre>locale -a | grep ^hi</pre> should then show that the Hindi loclales are installed.
# Installing/uninstalling the meta packages langpacks-<language> should automatically install/uninstall the respective glibc packages.
#* For example, <pre>dnf install langpacks-hi</pre> should install some stuff useful for Hindi including glibc-langpack-hi. <pre>dnf remove langpacks-hi</pre> should remove the Hindi stuff again and should also remove glibc-langpack-hi.
# There should be a package glibc-all-langpacks, installing that should install all available locales into /usr/lib/locales/locale-archive
#* For example, after <pre>dnf install glibc-all-langpacks</pre> the file <pre>/usr/lib/locales/locale-archive</pre> should exist and all locales available should be installed. I.e. <pre>locale -a</pre> should output a very long list: <pre>$locale -a | wc</pre> should count more then 800 lines.
#* Packages like <pre>glibc-langpack-hi</pre> maybe be installed in addition to <pre>glibc-all-langpacks</pre> but as long as <pre>glibc-all-langpacks</pre> is installed, they are redundant in that case and waste a bit of disk space, only the locales from /usr/lib/locales/locale-archive are used in that case.
# Make sure that other locales, especially custom locales added by the user, are not affected.
#* For example, if you install the locale sources with <pre>dnf install glibc-locale-source</pre> and then create a locale with a custom name <pre>sudo localedef --no-archive -i de_DE -f UTF-8 de_DE.utf8@myspecialgermanlocale</pre>, then a folder <pre>/usr/lib/locale/de_DE.utf8@myspecialgermanlocale/</pre> should exist. <pre>locale -a | grep ^de</pre> should show that such a locale is available and <pre>LC_ALL=de_DE.UTF-8@myspecialgermanlocale locale -k language</pre> should show that the language of this custom locale is "German". As long as the folder name of such a user locale does not conflict with a folder name in any of the glibc-langpack-<language> packages, there should be no problem of installing and removing any of the glibc-langpack-<languages> and/or glibc-all-langpacks packages, that user defined locale should be available no matter how many of the glibc-langpack-<language> or glibc-all-langpacks packages are installed.


== User Experience ==
== User Experience ==
<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
It should be possible to save disk space by having fewer locales installed.


== Dependencies ==
== Dependencies ==
Line 166: Line 178:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
* Anaconda
* dnf-langpacks


== Contingency Plan ==
== Contingency Plan ==


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: Revert changes to glibc packaging <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: Before F23 Beta release eg. Beta freeze. <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks product? product <!-- Applicable for Changes that blocks specific product release/Fedora.next -->
* Blocks product? No <!-- Applicable for Changes that blocks specific product release/Fedora.next -->


== Documentation ==
== Documentation, notes ==
<!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
<!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* QA and validation
** Validated binaries and files should be downloaded and installed.
** No slow compilation should happen on the host.
** For RHEL we want to be able to QA the binary artifacts and ship those. Shipping and compiling from source files is not an option we should consider.


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Compatibility with locale-gen:
N/A (not a System Wide Change)
** Strong consideration should be given to writing a dummy wrapper that provides as much functionality from locale-gen as possible but uses `dnf install foo` in the background to meet the requirements.
** This helps users transition packages from debian/ubuntu to Fedora.
** For example /etc/locale.gen could be parsed and appropriate packages installed to cover the set of locales requested, or with a 1:1 mapping (lots of locale sub-packages) it would be trivial.
** Update of glibc could run locale-gen wrapper.
** Avoid using `locales` sub-package except to install common locale support files and binaries. Ubuntu uses `locales` package for this purpose, but provides `language-pack-*` for languages (like Windows)
 
* Simple for the user to understand.
** Users should install "English" or "German" not "en_US.UTF-8".
** Users know what languages they need but shouldn't care about codesets or any other factors.
** Example: dnf install en-locales;
*** Either `en-locales` is a meta-package that installs all of the English locales, or is a package that contains all the English locales.
*** Probably 217 packages if we split it by lang name before underscore, but that would be way too much. We run into political issues with package counts.
** Example: vi /etc/locale.gen; locale-gen;
*** locale.gen must support simple "en" or "de" lines to install all formats of english or german.
** Example: dnf install locale-source;
*** Installs locale source files.
** Example: dnf install all-locales;
*** Install all binary locales. No locale may be called "all" (same restriction as build-locale-archive).
*** Includes installing a `other-locales` package which contains anything not split out. Adds limit that no locale may be called "other".
 
===In summary:===
 
* The simplest implementation is to split important languages into sub-packages e.g. en-locales, de-locales. Then provide an all-locales metapackage which installs them all. Default to having glibc-common depend on all-locales. Each locale package like en-locales includes the prebuilt en locales. Each locale package is responsible for installing or uninstalling their particular locale from the locale-archive, with _install_langs overriding. This support is put in and tested.
 
* Next work with Anaconda team to support selecting language, and translating that into `dnf install en-locales` or `vi /etc/locale.gen; locale-gen;`
 
* Remove glibc-common dependency on all-locales. At this point user must have a /etc/locale.gen which contains automatically added entries if they were using `dnf install en-locales` or their manual entry.
 
===Notes:===
 
* May need a /etc/sysconfig/locales to track state, but would like to make this  as stateless as possible.
* How does this tie into langpacks dnf support if at all?
 
===Test packages:===
Glibc packages build with copr for F23 and rawhide with locales split into sub-packages for testing are here: [https://copr.fedoraproject.org/coprs/mfabian/glibc/ Test packages]


== Release Notes ==
== Release Notes ==
Line 191: Line 243:
-->
-->


[[Category:ChangePageIncomplete]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- The Wrangler announces the Change to the devel-announce list and changes the category to Category:ChangeAnnounced (no action required) -->  
<!-- The Wrangler announces the Change to the devel-announce list and changes the category to Category:ChangeAnnounced (no action required) -->  
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangeReadyForWrangler-->


<!-- Select proper category, default is Self Contained Change -->
<!-- Select proper category, default is Self Contained Change -->
[[Category:SelfContainedChange]]
<!-- [[Category:SelfContainedChange]] -->
<!-- [[Category:SystemWideChange]] -->
[[Category:ChangeAcceptedF24]]
[[Category:SystemWideChange]]

Latest revision as of 05:38, 12 April 2016


Glibc locale subpackaging

Summary

This change should make it possible to install or uninstall locales individually.

Owner

Current status

Detailed Description

Currently the file /usr/lib/locale/locale-archive contains all locales and is thus huge (103MB).

For small systems (and containers) it would be useful to be able to install only a small number of locales.

Recently we made it possible to install a small number of locales by supplying the rpm-macro “_install_langs”, for example

   rpm -i -D _install_langs="en:de_DE" glibc-common.rpm

will install all English locales and all German locales which start with “de_DE”,

   rpm -i -D _install_langs="en_US.utf8" glibc-common.rpm

will install only the en_US.utf8 locale,

   rpm -i -D _install_langs="POSIX" glibc-common.rpm

will install nothing (but the POSIX/C is still available because it is builtin into glibc).

But this approach works only during an Anaconda based install when Anaconda supplies the _install_langs rpm-macro.

When glibc is updated later, the _install_langs macro will not be supplied on the command line during the update and the default value “all” of “_install_langs” from /usr/lib/rpm/macros will be used and all locales come back during an update.

Therefore, this solution is far from perfect.

It should be made possible to install and uninstall locales individually, for example by having a separate package for the locales for each language. Installing such a package would add these locales to locale-archive, uninstalling it would remove them.

Anaconda then needs to be changed to handle such language packages.

Other implementation options

Benefit to Fedora

This change should make it possible to reduce the minimum size of an installation considerably.

Scope

  • Proposal owners:
    1. Figure out the best approach to to install/uninstall locales
    2. Make sure that locales added manually by the user are not destroyed (currently they are lost when glibc is updated)


  • Other developers:

Anaconda needs to be made aware of the new approach to handle installation/uninstallation of locales

  • Release engineering:

I am not sure whether this has affects release engineering, probably the packages in the install image change when parts are split out of glibc-common.

  • Policies and guidelines:

No, this change does not require updates to policies and guidelines.

  • Trademark approval: not needed for this Change

Upgrade/compatibility impact

The user should not notice any change after the update. All locales which were installed before should still be installed after the update.

After the update it should be possible to uninstall locales for unneeded languages but this is optional.

How To Test

Please visit How to Test page and participate in Fedora 24 i18n testday

  1. Try to install/uninstall packages containing locales for some languages.
    • For example,
      dnf install glibc-langpack-hi
      should install the Hindi locales
      locale -a | grep ^hi
      should then show that the Hindi loclales are installed.
  2. Installing/uninstalling the meta packages langpacks-<language> should automatically install/uninstall the respective glibc packages.
    • For example,
      dnf install langpacks-hi
      should install some stuff useful for Hindi including glibc-langpack-hi.
      dnf remove langpacks-hi
      should remove the Hindi stuff again and should also remove glibc-langpack-hi.
  3. There should be a package glibc-all-langpacks, installing that should install all available locales into /usr/lib/locales/locale-archive
    • For example, after
      dnf install glibc-all-langpacks
      the file
      /usr/lib/locales/locale-archive
      should exist and all locales available should be installed. I.e.
      locale -a
      should output a very long list:
      $locale -a | wc
      should count more then 800 lines.
    • Packages like
      glibc-langpack-hi
      maybe be installed in addition to
      glibc-all-langpacks
      but as long as
      glibc-all-langpacks
      is installed, they are redundant in that case and waste a bit of disk space, only the locales from /usr/lib/locales/locale-archive are used in that case.
  4. Make sure that other locales, especially custom locales added by the user, are not affected.
    • For example, if you install the locale sources with
      dnf install glibc-locale-source
      and then create a locale with a custom name
      sudo localedef --no-archive -i de_DE -f UTF-8 de_DE.utf8@myspecialgermanlocale
      , then a folder
      /usr/lib/locale/de_DE.utf8@myspecialgermanlocale/
      should exist.
      locale -a | grep ^de
      should show that such a locale is available and
      LC_ALL=de_DE.UTF-8@myspecialgermanlocale locale -k language
      should show that the language of this custom locale is "German". As long as the folder name of such a user locale does not conflict with a folder name in any of the glibc-langpack-<language> packages, there should be no problem of installing and removing any of the glibc-langpack-<languages> and/or glibc-all-langpacks packages, that user defined locale should be available no matter how many of the glibc-langpack-<language> or glibc-all-langpacks packages are installed.

User Experience

It should be possible to save disk space by having fewer locales installed.

Dependencies

  • Anaconda
  • dnf-langpacks

Contingency Plan

  • Contingency mechanism: Revert changes to glibc packaging
  • Contingency deadline: Before F23 Beta release eg. Beta freeze.
  • Blocks release? No
  • Blocks product? No

Documentation, notes

  • QA and validation
    • Validated binaries and files should be downloaded and installed.
    • No slow compilation should happen on the host.
    • For RHEL we want to be able to QA the binary artifacts and ship those. Shipping and compiling from source files is not an option we should consider.
  • Compatibility with locale-gen:
    • Strong consideration should be given to writing a dummy wrapper that provides as much functionality from locale-gen as possible but uses dnf install foo in the background to meet the requirements.
    • This helps users transition packages from debian/ubuntu to Fedora.
    • For example /etc/locale.gen could be parsed and appropriate packages installed to cover the set of locales requested, or with a 1:1 mapping (lots of locale sub-packages) it would be trivial.
    • Update of glibc could run locale-gen wrapper.
    • Avoid using locales sub-package except to install common locale support files and binaries. Ubuntu uses locales package for this purpose, but provides language-pack-* for languages (like Windows)
  • Simple for the user to understand.
    • Users should install "English" or "German" not "en_US.UTF-8".
    • Users know what languages they need but shouldn't care about codesets or any other factors.
    • Example: dnf install en-locales;
      • Either en-locales is a meta-package that installs all of the English locales, or is a package that contains all the English locales.
      • Probably 217 packages if we split it by lang name before underscore, but that would be way too much. We run into political issues with package counts.
    • Example: vi /etc/locale.gen; locale-gen;
      • locale.gen must support simple "en" or "de" lines to install all formats of english or german.
    • Example: dnf install locale-source;
      • Installs locale source files.
    • Example: dnf install all-locales;
      • Install all binary locales. No locale may be called "all" (same restriction as build-locale-archive).
      • Includes installing a other-locales package which contains anything not split out. Adds limit that no locale may be called "other".

In summary:

  • The simplest implementation is to split important languages into sub-packages e.g. en-locales, de-locales. Then provide an all-locales metapackage which installs them all. Default to having glibc-common depend on all-locales. Each locale package like en-locales includes the prebuilt en locales. Each locale package is responsible for installing or uninstalling their particular locale from the locale-archive, with _install_langs overriding. This support is put in and tested.
  • Next work with Anaconda team to support selecting language, and translating that into dnf install en-locales or vi /etc/locale.gen; locale-gen;
  • Remove glibc-common dependency on all-locales. At this point user must have a /etc/locale.gen which contains automatically added entries if they were using dnf install en-locales or their manual entry.

Notes:

  • May need a /etc/sysconfig/locales to track state, but would like to make this as stateless as possible.
  • How does this tie into langpacks dnf support if at all?

Test packages:

Glibc packages build with copr for F23 and rawhide with locales split into sub-packages for testing are here: Test packages

Release Notes