Infrastructure/Factory2/Focus/WaiverDB

What is a Focus Document?

The Factory 2.0 team produces a confusing number of documents. The first round was about the Problem Statements we were trying to solve. Let’s retroactively call them Problem Documents. The Focus Documents (like this one) focus on some system or some aspect of our solutions that cut across different problems.  The content here doesn’t fit cleanly in one problem statement document, which is why we broke it out.

Background on WaiverDB

In the Factory 2.0 vision, ResultsDB will become the central store for aggregating test results produced by all our disparate testing tools. (See Infrastructure/Factory2/Focus/ResultsDB for more.)

An important characteristic of ResultsDB is that results are immutable once they are recorded. That gives us confidence that we can always go back in time and audit historical test results to understand what testing was done on some artifact and what the outcome was. But sometimes tests fail spuriously, because of infrastructure problems or other known issues with the test, so we need a way for a human to "waive" a failing result: that is, to record in an auditable way that we are ignoring the failed result, and why.

Some testing tools currently include waiving functionality. But as the testing tools proliferate, rather than forcing each tool to reinvent its own scheme for waiving failures, it makes more sense to record waivers in one central place, as a companion to ResultsDB: hence WaiverDB.

Just like ResultsDB, WaiverDB forms part of the solution to the Serialization and Automation problems.

See also:

• WaiverDB source code on pagure.io
• Infrastructure/Factory2/Focus/ResultsDB
• Infrastructure/Factory2/Focus/PolicyEngine
• Factory 2.0 talk at DevConf 2017

Required functionality

• A waiver is tied to a single Result in ResultsDB by referring to its integer result ID.

• For auditability, waiver records are immutable and the timestamp and username are recorded with the waiver. To amend an existing waiver, users can post a new waiver record for the same result. WaiverDB will then consider previous waivers from the same user for the same result to be "obsolete". WaiverDB will exclude obsolete waiver records by default when a caller is looking up waivers for a particular result. Callers can explicitly query for obsolete waivers if they need to see the complete history for auditing purposes.

• Each waiver record will include a flag "waived": true or "waived": false, to support the scenario where a user has previously waived a result ("waived": true) but later change their mind, so they post a new waiver with "waived": false indicating that the result is no longer waived. Note this means when a caller is looking up waivers for a particular result, the caller needs to check not only for the presence of a waiver record, but also whether the waived key is true.
• For authentication WaiverDB will support (at least) Kerberos for Red Hat internal deployment and OpenID Connect/OAuth for Fedora deployment.

• When a new waiver is recorded, WaiverDB will announce it on the relevant message bus (fedmsg for Fedora, UMB internally within Red Hat).

• The WaiverDB API will need to support JSONP and/or CORS headers, so that user interfaces in other applications (Bodhi, Errata Tool) can fetch and post waivers directly using AJAX requests.

Other considerations

• Like all our tools, WaiverDB will be developed upstream in Fedora first. The Fedora QA team has expressed interest in a service for managing waivers, and we anticipate that Bodhi will be one of the first production users of the service.

• The initial version will not include a human-friendly web UI for submitting or viewing waivers, only a JSON API. We anticipate that the workflow tools which currently display results (Bodhi in Fedora, Errata Tool internally) will also grow the ability to post waivers to WaiverDB from the same UI where the failure is being displayed. But we can revisit this decision in future if a UI for WaiverDB is required.

Proposed API

POST /waivers/
{"result_id": 123, "waived": true, "comment": "it broke", "product_version": "rhel-7"}

Records a new waiver for the given result. Revoking your previous waiver ("unwaiving") entails posting a new waiver with "waived": false.

The product_version key re-uses PDC's product version identifiers.

GET /waivers/456
{"id": 456, "result_id": 123, "waived": true, "comment": "it broke", "product_version": "rhel-7", "username": "dcallagh", "timestamp": "2017-03-01T03:08:37.704507"}

The "username" and "timestamp" keys are filled in by WaiverDB when the waiver is posted.

GET /waivers/?result_id=123,124&product_version=rhel-7
[{"id": ...}, ...]

Returns all current waivers for the given set of result IDs, applied to the given product. Should also support filtering by username and timestamp.

This will exclude obsolete waivers by default (that is, waivers where the same user has posted an updated waiver for the same result). An extra query parameter include_obsolete=1 can be passed if the caller wants the complete waiver history for auditing purposes.

Deferred functionality

The following requirements were considered, but are deferred for the initial version of WaiverDB because we are not sure how to implement them or whether they will be needed.

• API to give me all waivers for a certain "item" (e.g. koji build NVR). This is deferred because it would require WaiverDB to keep a complete copy of every waived result, including all its queryable attributes like "item" and "extra data". At that point it suggests that ResultsDB and WaiverDB should be the same thing… As a consequence of deferring this, client code will have to first query ResultsDB for results, retrieve them, and then query WaiverDB for waivers on those results.

• API to waive all results in a ResultsDB Group, or multiple results in a hierarchy (e.g. dist.rpmdiff.*). For now the client will be responsible for waiving specific individual results, but we can expand this in future if there is a need.