User:Mkoncek/Javapackages-validator

Introduction

This effort aims at improving the quality of Fedora Java packages by employing Javapackages validator (JPV) as the TMT test for all Java packages in Fedora.

The tests are executed upon receiving pull requests and validate the package using generic tests based on issues encountered in real life. The tests include sanity checks for artifacts names, trailing metadata paths, archfulnes of certain sub-packages and conforming to certain Fedora-wide Java package policies.

The current approach is to use TMT for testing and this requires configuration files to be added to each package's dist-git repository.

Scope

All Java packages in Fedora.

Benefits

• Close to no manual work required on the part of the maintainer
• Automated sanity checks
• As new Fedora policies are adopted, the testing tool adds tests for them

Adoption

In order to use JPV the maintainers must add an .fmf file (usually named plans/javapackages.fmf) with the following content:

 summary: Run javapackages-specific tests
 discover:
   how: fmf
   url: https://src.fedoraproject.org/tests/javapackages
   ref: main
 execute:
   how: tmt
 environment:
   JP_VALIDATOR_SOURCE_PATH: 
   JP_VALIDATOR_ARGS: org.fedoraproject.javapackages.validator.validators.DefaultValidatorFactory

To be able to invoke tmt tests, the dist-git needs to contain a file .fmf/version with the content:

 1

For better visualization of the test results, there should be another file named ci.fmf with the content:

 resultsdb-testcase: separate

And finally if the maintainer wishes to enable gating on this tests, there should be a file gating.yaml with the content:

 --- !Policy
 product_versions:
   - fedora-*
 decision_contexts:
   - bodhi_update_push_testing
   - bodhi_update_push_stable
 rules:
   - !PassingTestCaseRule {test_case_name: fedora-ci.koji-build./plans/javapackages.functional}

Test ref and compatibility

The test .fmf file may refer to main branch or to versioned branches f<N>. The versioning works similarly to how Fedora itself is released. There are two approaches the maintainers can take to ensure proper package version - test version match:

1. Use explicit versioned ref from the start and bump it as necessary.
2. Use ref: main and branch off with Fedora branching.

Both cases require some work on the part of the maintainer as Fedora releases branch off.

In the first case if the maintainer forgets to bump the reference, their package will not be tested with the latest version and they will miss their package being tested with new tests including cases when Fedora-wide policy regarding Java packages changes.

In the second case, if the maintainer forgets to branch off, later when they rebuild their package in an older version of Fedora, the package will be tested with the current JPV which may include tests for new Fedora changes and cause the tests to fail.

Currently we believe the second approach is less dangerous.

Future

We will consider adding this test to Fedora infrastructure similar to how rpminspect is used after having feedback from the TMT-based approach. This would allow completely zero-work effort on the part of package maintainers.

This effort can also serve as a proof of concept for other package ecosystems.