From Fedora Project Wiki
(Version normalization)
(Version normalization)
 
(37 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
[[category:Perl]]
 
[[category:Perl]]
  
= Version normalization =
+
== Version normalization ==
 +
'''Problem''': sometimes new version could be in form acceptable for CPAN, but not for RPM.
  
* After takinh over Perl dependency generator
+
'''Example''': package had version 1.00 and new release is 1.000324. It works for CPAN, where version is just a string, but not for RPM.
 +
 
 +
'''Solution''': instead of manual change (also provides might be wrong) can be version normalized by Perl itself (module version can do it).
 +
 
 +
* After taking over Perl dependency generator ✓
 
* Macros for simplified packaging
 
* Macros for simplified packaging
 
** the version translator
 
** the version translator
 +
*** Perl architecture independent (because of upgrading perl): perl-Fedora-VSP ✓
 +
*** C implementation for build-requires required by perl-srpm-macros
 
** the ''perl()'' wrapper
 
** the ''perl()'' wrapper
 
** common snippets for EU::MM, Module::Build, etc.
 
** common snippets for EU::MM, Module::Build, etc.
Line 12: Line 19:
 
** Teach [[Upstream_release_monitoring | Fedora upstream monitoring]] ''cnucnu(1)''
 
** Teach [[Upstream_release_monitoring | Fedora upstream monitoring]] ''cnucnu(1)''
  
= Dependency generator =
+
== Macros for simplified packaging ==
 +
'''Problem''': There is still lot of manual work in specfiles. Better macros could improve packaging. Some packagers might discourage changes, because they prefer lot of ifdefs for different EL releases.
 +
 
 +
'''Solution''': Improvements above could make maintenance easier for future. One macro in macro file can redefine content of all specfiles and rebuild would add all changes.
  
* Take over dependency generator. Assigne: [[User:jplesnik]], [[User:mmaslano]]
+
* Version normalizer (see first paragraph)
* Tests
+
* perl() wrapper
* Tests
+
** This will mask SCL modifications
 +
* Dependency filter abstraction
 +
** One syntax for Fedora and EPEL
 +
** One syntax for SCL
 +
* Provide standard EU::MM, Module::Build, Module::Bild::Tiny, &c. %build and %install implementations
 +
* Standard implementations for CPAN packages
 +
** ''URL''
 +
** ''Sourece0''
 +
** ''%prep''
 +
* Pack files scriptlets editor
 +
** Allow Fatpacker and inc::Module::Build bundler to work on packaged modules
 +
* Mark POD files as documentation
 +
 
 +
== Dependency generator ==
 +
'''Problem''': dependency generator in RPM has lot of incorrect data, which leads to complicated filtering of provides/requires
 +
 
 +
'''Solution''': dependency generator will be removed from RPM and owned by Perl team.
 +
 
 +
'''Comment''': More improvements in main generator are not possible because contyk's smarter scaner has many dependencies. At least contyk's scaner could be used for generating new packages and updates.
 +
 
 +
* Take over dependency generator. Assignee: [[User:jplesnik]], [[User:mmaslano]]. Packaged as ''perl-generators''. ✓
 +
* Tests. Assignee: [[User:jplesnik]]
 
* Support no, base, parent, Moose's with, POE's extensions, Test::Requires, other frameworks, see Module::ScanDeps::scan_chunk()
 
* Support no, base, parent, Moose's with, POE's extensions, Test::Requires, other frameworks, see Module::ScanDeps::scan_chunk()
 
* Receive explicit Requires from spec file to merge them with generated ones
 
* Receive explicit Requires from spec file to merge them with generated ones
 
* Refactoring
 
* Refactoring
 +
** How many dependencies has Module-ScanDeps? Assignee: [[User:psabata]]. Module-ScanDeps reads META.yml. It's not useful. ✓
  
= Fedora-Rebuild =
+
== Fedora-Rebuild ==
 +
'''Problem''': annual Perl rebuild took more than month. No tooling is provided by Fedora nor rpm/yum-tools.
  
 +
'''Solution''': Fedora::Rebuild is able to count build order of packages based on their dependencies.
 +
 +
'''Comment''': rebuilds of every new release should make easier finding bugs in other packages sooner and make rebuild shorter.
 +
 +
* Assignee: [[User:ppisar]]
 
* Optimize pure-perl resolver
 
* Optimize pure-perl resolver
 +
** Finished in git tree. ✓
 +
** Release new version. ✓
 
* Implement [[Features/Hawkey | hawkey]]-based resolver
 
* Implement [[Features/Hawkey | hawkey]]-based resolver
 
** XS wrapper for hawkey
 
** XS wrapper for hawkey
Line 29: Line 69:
 
** YUM repostory downloader
 
** YUM repostory downloader
 
** Augment Fedora-Rebuild to download YUM repository, filter it, and do hawkey queries over it.
 
** Augment Fedora-Rebuild to download YUM repository, filter it, and do hawkey queries over it.
* Copr mode
+
* [[:Category:Copr | Copr]] mode
 
** Copr is missing way how to retrieve binary package made of only exact source package
 
** Copr is missing way how to retrieve binary package made of only exact source package
 
** Select binary packages from Copr's YUM repository
 
** Select binary packages from Copr's YUM repository
 
** Requires YUM downloaded and metadata parser
 
** Requires YUM downloaded and metadata parser
 +
* Mock initialization with multiple groups and packages (to support rebuilding collections)
 +
* Specification file editor for mass-changing source packages.
 +
 +
== Continuous rebuilds ==
 +
'''Problem''': rebuild of new Perl and related packages take a lot of time.
  
= Continuous rebuilds =
+
'''Solution''': continuous rebuilds should show broken packages earlier than in annual rebuild.
  
 
* [http://www.fedmsg.com/en/latest/ Fedmsg] binding to Perl
 
* [http://www.fedmsg.com/en/latest/ Fedmsg] binding to Perl
 
* Use fedmsg to gather updated packages
 
* Use fedmsg to gather updated packages
** I have already GIT based checker
+
** [[User:ppisar]] already has GIT based checker
 
* Use fedmsg to include new packages
 
* Use fedmsg to include new packages
 
* Use fedmsg to exclude retired packages
 
* Use fedmsg to exclude retired packages
 +
* Do periodical scratch boot-strap to evaluate overall package health. Assignee [[user:ppisar]] ✓
 +
** [http://ppisar.fedorapeople.org/perl_rebuild/scratch/latest/index.xhtml logs]
 +
** [http://ppisar.fedorapeople.org/perl_rebuild/scratch/index.xhtml matrix]
  
= Dependency cycle breaker =
+
== Dependency cycle breaker ==
  
 
* YUM repository reader
 
* YUM repository reader
Line 50: Line 98:
 
* Semi-automated recommendation for cycle break-points
 
* Semi-automated recommendation for cycle break-points
  
= YUM repository parser =
+
== YUM repository parser ==
  
 
* [https://en.opensuse.org/openSUSE:Standards_Rpm_Metadata YUM metadata] downloader
 
* [https://en.opensuse.org/openSUSE:Standards_Rpm_Metadata YUM metadata] downloader
Line 57: Line 105:
 
* YUM package downloader
 
* YUM package downloader
  
= Macros for simplified packaging =
+
== Spec file generator ==
 +
'''Problem''': cpanspec is old and not up2date to current guidelines. There is space for improvement of automatic generation of Perl specfiles.
  
* Version normalizer
+
'''Solution''': Push changes to cpanspec and continue in work on Perl::Scan package developed by [[User:psabata]].
* perl() wrapper
 
** This will mask SCL modifications
 
* Dependency filter abstraction
 
** One syntax for Fedora and EPEL
 
** One syntax for SCL
 
* Provide standard EU::MM, Module::Build, Module::Bild::Tiny, &c. %build and %install implementations
 
* Standard implementations for CPAN packages
 
** ''URL''
 
** ''Sourece0''
 
** ''%prep''
 
* Pack files scriptlets editor
 
** Allow Fatpacker and inc::Module::Build bundler to work on packaged modules
 
* Mark POD files as documentation
 
  
= Spec file generator =
+
* The upstream author of ''cpanspec'' has expressed willingness to incorporate some changes, however a complete rewrite of the utility would be more feasiable long-term
 
 
* Fork ''cpan2spec''
 
 
* Static dependency analysis
 
* Static dependency analysis
 +
** Rewrite Perl::Scan without eval. Assignee: [[User:psabata]]
 
* Dynamic dependency analysis
 
* Dynamic dependency analysis
 
* Compare analyses against RPM package
 
* Compare analyses against RPM package
Line 86: Line 121:
 
* False provides checker
 
* False provides checker
 
** RPM package should not provide privately redefined modules
 
** RPM package should not provide privately redefined modules
 +
* Converter into collection spec files which understands Perl spec files
 +
 +
== Fedora Upstream monitoring ==
 +
'''Problem''': Some maintainers don't want to monitor upstream releases. Some upstream releases can't be parsed by Fedora monitoring script. During annual rebuild with new Perl version can updates of modules consume a lot of time.
  
= Fedora Upstream monitoring =
+
'''Solution''': 1/ let's try to persuade monitoring system to fix their issues, 2/ see next point
  
 
* Monitor upstream on our own if Fedora's system will not be improved
 
* Monitor upstream on our own if Fedora's system will not be improved
Line 94: Line 133:
 
** Update them before each mass rebuild
 
** Update them before each mass rebuild
  
= [[Features/DevelopersAssistant | Developer Assistant]] =
+
== Upstream monitoring ==
 
 
* Templates for standard build systems
 
* Templates for web frameworks
 
 
 
= Upstream monitoring =
 
  
 
* Not all packages are registered in [[Upstream_release_monitoring | Fedora Upstream Monitoring service]]
 
* Not all packages are registered in [[Upstream_release_monitoring | Fedora Upstream Monitoring service]]
Line 110: Line 144:
 
** Verify them against Fedora look-aside cache
 
** Verify them against Fedora look-aside cache
 
** Add the check into ''spectool(1)''. Assignee: [[User:ppisar]]
 
** Add the check into ''spectool(1)''. Assignee: [[User:ppisar]]
 +
*** Reported against rpmdevtools as a [https://bugzilla.redhat.com/show_bug.cgi?id=1093712 bug #1093712].
 +
 +
== [[Features/DevelopersAssistant | Developer Assistant]] ==
 +
 +
* Templates for standard build systems
 +
* Templates for web frameworks
  
= Perl Next =
+
== Perl Next ==
  
* While perl is a base package, other CPAN code will go into server or workstation. We need to be ready and implement Next requirements
+
* While perl is a base package, other CPAN code will go into server or workstation. We need to be ready and implement Next requirements. It turned out the the Next was a fiasco. No Perl changes were needed.
* I propose to delay perl 5.20 upgrade to Fedora 22.
+
* Propose to [https://lists.fedoraproject.org/pipermail/perl-devel/2014-March/085719.html delay perl 5.20 upgrade to Fedora 22]. Assignee: [[User:ppisar]] ✓
 +
* Submit Perl 5.20 change to Fedora 22. Assignee: [[User:jplesnik]] ✓
 +
* Rebuild Perl 5.20. Assignee: [[User:jplesnik]] ✓
 +
* Remove Perl from minimal build root. [https://lists.fedoraproject.org/pipermail/devel/2014-October/203821.html Request]. Assignee: [[User:ppisar]]

Latest revision as of 09:54, 15 December 2015


Version normalization

Problem: sometimes new version could be in form acceptable for CPAN, but not for RPM.

Example: package had version 1.00 and new release is 1.000324. It works for CPAN, where version is just a string, but not for RPM.

Solution: instead of manual change (also provides might be wrong) can be version normalized by Perl itself (module version can do it).

  • After taking over Perl dependency generator ✓
  • Macros for simplified packaging
    • the version translator
      • Perl architecture independent (because of upgrading perl): perl-Fedora-VSP ✓
      • C implementation for build-requires required by perl-srpm-macros
    • the perl() wrapper
    • common snippets for EU::MM, Module::Build, etc.
  • version->parse->normal normalization in macros and generator
  • Normalize RPM package version?

Macros for simplified packaging

Problem: There is still lot of manual work in specfiles. Better macros could improve packaging. Some packagers might discourage changes, because they prefer lot of ifdefs for different EL releases.

Solution: Improvements above could make maintenance easier for future. One macro in macro file can redefine content of all specfiles and rebuild would add all changes.

  • Version normalizer (see first paragraph)
  • perl() wrapper
    • This will mask SCL modifications
  • Dependency filter abstraction
    • One syntax for Fedora and EPEL
    • One syntax for SCL
  • Provide standard EU::MM, Module::Build, Module::Bild::Tiny, &c. %build and %install implementations
  • Standard implementations for CPAN packages
    • URL
    • Sourece0
    • %prep
  • Pack files scriptlets editor
    • Allow Fatpacker and inc::Module::Build bundler to work on packaged modules
  • Mark POD files as documentation

Dependency generator

Problem: dependency generator in RPM has lot of incorrect data, which leads to complicated filtering of provides/requires

Solution: dependency generator will be removed from RPM and owned by Perl team.

Comment: More improvements in main generator are not possible because contyk's smarter scaner has many dependencies. At least contyk's scaner could be used for generating new packages and updates.

  • Take over dependency generator. Assignee: User:jplesnik, User:mmaslano. Packaged as perl-generators. ✓
  • Tests. Assignee: User:jplesnik
  • Support no, base, parent, Moose's with, POE's extensions, Test::Requires, other frameworks, see Module::ScanDeps::scan_chunk()
  • Receive explicit Requires from spec file to merge them with generated ones
  • Refactoring
    • How many dependencies has Module-ScanDeps? Assignee: User:psabata. Module-ScanDeps reads META.yml. It's not useful. ✓

Fedora-Rebuild

Problem: annual Perl rebuild took more than month. No tooling is provided by Fedora nor rpm/yum-tools.

Solution: Fedora::Rebuild is able to count build order of packages based on their dependencies.

Comment: rebuilds of every new release should make easier finding bugs in other packages sooner and make rebuild shorter.

  • Assignee: User:ppisar
  • Optimize pure-perl resolver
    • Finished in git tree. ✓
    • Release new version. ✓
  • Implement hawkey-based resolver
    • XS wrapper for hawkey
    • YUM repository filter for Fedora-Rebuild dependency filter
    • YUM repostory downloader
    • Augment Fedora-Rebuild to download YUM repository, filter it, and do hawkey queries over it.
  • Copr mode
    • Copr is missing way how to retrieve binary package made of only exact source package
    • Select binary packages from Copr's YUM repository
    • Requires YUM downloaded and metadata parser
  • Mock initialization with multiple groups and packages (to support rebuilding collections)
  • Specification file editor for mass-changing source packages.

Continuous rebuilds

Problem: rebuild of new Perl and related packages take a lot of time.

Solution: continuous rebuilds should show broken packages earlier than in annual rebuild.

  • Fedmsg binding to Perl
  • Use fedmsg to gather updated packages
  • Use fedmsg to include new packages
  • Use fedmsg to exclude retired packages
  • Do periodical scratch boot-strap to evaluate overall package health. Assignee user:ppisar

Dependency cycle breaker

  • YUM repository reader
    • repoquery --whatprovides is to slow due to reinitialization
  • Gather boot-strapped dependencies from local SRPMs
  • Use graph algorithms to identify cycles
  • Semi-automated recommendation for cycle break-points

YUM repository parser

  • YUM metadata downloader
  • YUM metadata reader
  • YUM metadata filter/writer
  • YUM package downloader

Spec file generator

Problem: cpanspec is old and not up2date to current guidelines. There is space for improvement of automatic generation of Perl specfiles.

Solution: Push changes to cpanspec and continue in work on Perl::Scan package developed by User:psabata.

  • The upstream author of cpanspec has expressed willingness to incorporate some changes, however a complete rewrite of the utility would be more feasiable long-term
  • Static dependency analysis
  • Dynamic dependency analysis
  • Compare analyses against RPM package
    • Comment annotations in spec
    • Structured build-, run-, test- dependencies
  • Differences calculator
    • Useful when upgrading a package
  • False provides checker
    • RPM package should not provide privately redefined modules
  • Converter into collection spec files which understands Perl spec files

Fedora Upstream monitoring

Problem: Some maintainers don't want to monitor upstream releases. Some upstream releases can't be parsed by Fedora monitoring script. During annual rebuild with new Perl version can updates of modules consume a lot of time.

Solution: 1/ let's try to persuade monitoring system to fix their issues, 2/ see next point

  • Monitor upstream on our own if Fedora's system will not be improved
    • The maintainer wants CPAN wildcard
  • Monitor all RPM packages
    • Update them before each mass rebuild

Upstream monitoring

  • Not all packages are registered in Fedora Upstream Monitoring service
  • Not all packagers upgrades
  • Monitor upstream on our own
    • Use CPAN
    • Adjust versions where done in Fedora
    • File bugs or do the upgrade directly or spam perl-devel mailing list
  • CPAN mirrors provides signed digests of CPAN distributions
    • Verify them against Fedora look-aside cache
    • Add the check into spectool(1). Assignee: User:ppisar

Developer Assistant

  • Templates for standard build systems
  • Templates for web frameworks

Perl Next

  • While perl is a base package, other CPAN code will go into server or workstation. We need to be ready and implement Next requirements. It turned out the the Next was a fiasco. No Perl changes were needed.
  • Propose to delay perl 5.20 upgrade to Fedora 22. Assignee: User:ppisar
  • Submit Perl 5.20 change to Fedora 22. Assignee: User:jplesnik ✓
  • Rebuild Perl 5.20. Assignee: User:jplesnik
  • Remove Perl from minimal build root. Request. Assignee: User:ppisar