Features/FullyTestablePerl

From FedoraProject

< Features(Difference between revisions)
Jump to: navigation, search
(Scope)
Line 1: Line 1:
 
<!-- All fields on this form are required to be accepted by FESCo.
 
<!-- All fields on this form are required to be accepted by FESCo.
 
  We also request that you maintain the same order of sections so that all of the feature pages are uniform.  -->
 
  We also request that you maintain the same order of sections so that all of the feature pages are uniform.  -->
 
+
<div class="messagebox" style="float: right; margin-left: 10px;">__TOC__</div>
<!-- The actual name of your feature page should look something like: Features/YourFeatureName.  This keeps all features in the same namespace -->
+
 
+
 
= Feature Name =
 
= Feature Name =
 
Fully Testable Perl.
 
Fully Testable Perl.
 
<div class="messagebox" style="float: right; margin-left: 10px;">__TOC__</div>
 
  
 
== Summary ==
 
== Summary ==
Line 23: Line 19:
 
* Targeted release: [[Releases/{{FedoraVersion||next}} | {{FedoraVersion|long|next}} ]]  
 
* Targeted release: [[Releases/{{FedoraVersion||next}} | {{FedoraVersion|long|next}} ]]  
 
* Last updated: (DATE)
 
* Last updated: (DATE)
* Percentage of completion: XX%
 
  
{{Progress_bar|10}}
+
{{progress_bar|10|width=65%}}
  
 
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
 
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
  
 
== Detailed Description ==
 
== Detailed Description ==
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
 
  
 
Perl has a long history and culture of testing, which has resulted in a very high percentage of the packages on the CPAN (Comprehensive Perl Archive Network) containing significant test suites.  While these test suites are executed at build time, a large number of modern Perl packages (e.g. {{CPAN|Moose}}, {{CPAN|DBIx::Class}}, [[Catalyst]], etc) depend on a significant number of other packages, which may be owned by a different maintainer and updated independently.
 
Perl has a long history and culture of testing, which has resulted in a very high percentage of the packages on the CPAN (Comprehensive Perl Archive Network) containing significant test suites.  While these test suites are executed at build time, a large number of modern Perl packages (e.g. {{CPAN|Moose}}, {{CPAN|DBIx::Class}}, [[Catalyst]], etc) depend on a significant number of other packages, which may be owned by a different maintainer and updated independently.
Line 62: Line 56:
  
 
== Scope ==
 
== Scope ==
{{admon/warning|Preliminary Scope|The exact details of implementation, e.g. where to put the test suites, how to execute them, and how to implement the RPM macros is still evolving; it's also expected to change based on upstream's feedback.  The general principle will remain the same, but the details will evolve.}}
+
{{admon/warning|Don't obsess over the details!|The exact details of implementation, e.g. where to put the test suites, how to execute them, and how to implement the RPM macros is still evolving; it's also expected to change based on upstream's feedback.  The general principle will remain the same, but the details will evolve.}}
  
 
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
 
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
Line 113: Line 107:
 
perl-Sub-Curry-0.8-1.fc8.noarch.rpm  perl-Sub-Curry-tests-0.8-1.fc8.noarch.rpm
 
perl-Sub-Curry-0.8-1.fc8.noarch.rpm  perl-Sub-Curry-tests-0.8-1.fc8.noarch.rpm
 
</pre>
 
</pre>
 +
 +
We can also examine the -tests rpm to ensure it matches what we want:
  
 
<pre>
 
<pre>

Revision as of 08:05, 13 November 2008

Feature Name

Fully Testable Perl.

Summary

Provide the ability to re-test installed Perl packages (primarily CPAN-hosted packages) by automatically providing the test suites of each distribution as their own "perl-Foo-tests" subpackage, and providing a framework to test.

Owner

Current status


Detailed Description

Perl has a long history and culture of testing, which has resulted in a very high percentage of the packages on the CPAN (Comprehensive Perl Archive Network) containing significant test suites. While these test suites are executed at build time, a large number of modern Perl packages (e.g. Moose, DBIx::Class, Catalyst, etc) depend on a significant number of other packages, which may be owned by a different maintainer and updated independently.

That is, the tests results from %check are only valid so long as the environment in which they are run does not change from the set of packages the package is built and tested in. Additionally, there are many reasons one or more tests may be disabled under the buildsystem:

  • Network or $DISPLAY access is required; or some other operation not permitted / possible in the buildsystem is required.
  • Including packages the test requires would cause a circular build dep loop.
  • A highly specific test environment is needed, e.g. a database setup.
  • Packages which cannot be included in Fedora (e.g. DBD::DB2, DBD::Oracle) are required to exercise functionality for testing.

This feature proposes to automatically bundle the package test suite into a -tests subpackage without requiring additional maintainer work (a la debuginfo), which can then be installed to provide the capability to test (or retest) functionality post installation.

Benefit to Fedora

Being able to execute the tests of a Perl dist post-build will leverage existing code (the test suites) to allow the end user to perform sanity checks and rule out problems with the underlying code when tracking down bugs.

The net benefit to Fedora is to create an environment where end users (support, programmers, etc) will be able to execute the test suites of the 900+ CPAN dists Fedora packages natively, to track down bugs, test functionality the buildsystem was unable to provide, or just validate their environment.

Having the test suite used to build the package under koji would allow someone to test their modules without having to move away from the Fedora perl packages. It will:

  • Make finding regressions between multiple packages owned by multiple maintainers easier.
  • Provide the end user with the tools to test, debug and validate their environments with the same test suites originally used to build the modules.
  • Leverage existing RPM functionality to create the -tests subpackages without needing any package maintainer interaction, just as -debuginfo packages are created today.
  • Provide an additional source of documentation (e.g. Moose::Role and MooseX::Types::DateTime explicitly refers one to the test suite as the "best documentation").

Example

For example, Moose is a modern metaobject-based class framework for Perl 5. It's been around for awhile, has a stable user API, and is extremely powerful. It has a slew of dependencies, as well as extensions in the MooseX::* namespace. Sometimes these extensions take advantage of the metaobject system -- those that do can be highly sensitive to backend changes in both Moose and Class::MOP. MooseX::AttributeHelpers is one such extension -- providing alternate attribute metaclass objects for different data structures. Indeed, Moose's build script explicitly checks for certain levels of modules it knows it will break.

Scope

Warning (medium size).png
Don't obsess over the details!
The exact details of implementation, e.g. where to put the test suites, how to execute them, and how to implement the RPM macros is still evolving; it's also expected to change based on upstream's feedback. The general principle will remain the same, but the details will evolve.


There are three main components to this feature.

Perl-QA

We should work with upstream for input on how to retain these tests in a coherent, consistent, usable form. (Upstream in this case would be the perl-qa mailing list.)

RPM macros implementation

The model for this effort is the "behind the scenes" packaging of debugging information into -debuginfo subpackages. While it doesn't appear that we have any sort of generic hook to tie into at this point, there are a couple things that could be done.

  1. Work with upstream to take the %debug_package macro and reimplement it on top of a "generic" packaging hook.
  2. Overload the %debug_package macro to automatically create -tests subpackages for Perl dists.

The first would be cleaner, but would require upstream involvement. Working proof-of-concept rpm macros overloading %debug_package have been created at Features/FullyTestablePerl/RpmMacros.

Test Execution Framework

Create a test-framework: e.g. to test perl-Moose and all its deps, an end user could execute "perl-tests Moose" and step back.

Test Plan

No special hardware is required to test.

Subpackage Generation

TODO.

Testing the subpackage generation will simply require building a Perl package on a system with the right level of macros, etc, installed.

[cweyl@zeus perl-Sub-Curry]$ rpmbuild -ba perl-Sub-Curry.spec 
....<rpmbuild output snipped>....
[cweyl@zeus perl-Sub-Curry]$ ls noarch/
perl-Sub-Curry-0.8-1.fc8.noarch.rpm  perl-Sub-Curry-tests-0.8-1.fc8.noarch.rpm

We can also examine the -tests rpm to ensure it matches what we want:

[cweyl@zeus perl-Sub-Curry]$ rpmquery -pi noarch/perl-Sub-Curry-tests-0.8-1.fc8.noarch.rpm 
Name        : perl-Sub-Curry-tests         Relocations: (not relocatable)
Version     : 0.8                               Vendor: (none)
Release     : 1.fc8                         Build Date: Wed 12 Nov 2008 11:32:46 PM PST
Install Date: (not installed)               Build Host: zeus
Group       : Development/Debug             Source RPM: perl-Sub-Curry-0.8-1.fc8.src.rpm
Size        : 4013                             License: GPL+ or Artistic
Packager    : Chris Weyl <cweyl@alumni.drew.edu>
URL         : http://search.cpan.org/dist/Sub-Curry
Summary     : Test suite for package perl-Sub-Curry
Description :
This package provides the test suite for package perl-Sub-Curry.
[cweyl@zeus perl-Sub-Curry]$ rpmls noarch/perl-Sub-Curry-tests-0.8-1.fc8.noarch.rpm 
drwxr-xr-x  /usr/lib/perl5/tests
drwxr-xr-x  /usr/lib/perl5/tests/5.8.8
drwxr-xr-x  /usr/lib/perl5/tests/5.8.8/vendor_tests
drwxr-xr-x  /usr/lib/perl5/tests/5.8.8/vendor_tests/Sub-Curry
drwxr-xr-x  /usr/lib/perl5/tests/5.8.8/vendor_tests/Sub-Curry/t
-rw-r--r--  /usr/lib/perl5/tests/5.8.8/vendor_tests/Sub-Curry/t/01basic.t
-rw-r--r--  /usr/lib/perl5/tests/5.8.8/vendor_tests/Sub-Curry/t/02spices.t

Testing Framework

TODO.

User Experience

John Q. Fedoraite

TODO.

Package Maintainers

TODO.

Administrators, Developers, etc

TODO.

Dependencies

Aside from the steps detailed under "Scope", above, there are no additional dependencies. As Perl packages are rebuilt, their -tests subpackage will be automatically generated without requiring any intervention by the package maintainer.

Contingency Plan

Failure of this feature to be ready for Fedora 22 only means that this feature will not be available. Existing functionality will not be impaired.

Documentation

No documentation, aside from this page, yet.

Release Notes

Comments and Discussion