Changes/Sqlite Rpmdb

Sqlite RpmDB

Summary

Change format of the RPM database from Berkeley DB to a new Sqlite format.

Owner

• Name: Panu Matilainen Florian Festi
• Email: pmatilai@redhat.com ffesti@redhat.com

Current status

• Targeted release: Fedora 33
• Last updated: 2020-11-20
• FESCo issue: #2360
• Tracker bug: #1818910
• Release notes tracker: #462

Detailed Description

The current rpm database implementation is based on Berkeley DB 5.x, a version which is unmaintained upstream for several years now. Berkeley DB 6.x is license incompatible so moving to that is not an option. In addition, the existing rpmdb implementation is notoriously unreliable as it's not transactional and has no other means to detect inconsistencies either.

Changing to a more sustainable database implementation is long overdue. We propose to change the default rpmdb format to the new sqlite based implementation. Support for current BDB format will be retained in Fedora 33, and phased out to read-only support in Fedora 34.

Benefit to Fedora

• A far more robust rpm database implementation
• Getting rid of Berkeley DB dependency in one of the core components

Scope

• Proposal owners:
  • Once RPM 4.16 lands and passes initial shakedown, change the default rpmdb configuration to sqlite
  • Arrange for automatic database conversion with opt-out possibility (one-shot service on next reboot or similar)
  • Address any bugs and issues in the database backend found by wider testing base
  • Help other developers to address Berkeley DB dependencies

• Other developers:
  • Test for hidden Berkeley DB dependencies in other software, address them as found and needed

• Release engineering: #9308

• Policies and guidelines: Policies and guidelines are not affected

• Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Upgrading

• Ability to upgrade is not affected
• After update, systems will be converted to the sqlite format (on next reboot or similar) unless user overrides configuration to stay on BDB for now

Compatibility

• Container/chroot use-cases will be affected: older rpm versions will be unable to query/manipulate the rpmdb from outside the chroot
• Koji/COPR may need to override the database format (back to) BDB for the time being. Better option would be to use mock bootstrap container, which would solve a whole class of issues going forward.

How To Test

• Rpmdb gets thoroughly exercised as a matter of normal system operation, performing installs, updates, package builds etc
• Of specific interest here is torture testing: forcibly killing rpm in various stages of execution - database should stay consistent and operational (other system state is out of scope)
• Test database conversions from one backend to another (rpmdb --rebuilddb --define "_db_backend <backend>")
• Install/upgrade scenarios:
  • A fresh install has sqlite enabled by default, no database warnings/errors emitted during rpm operation
  • System upgraded from Fedora < 33 has been converted to sqlite rpmdb after the post-upgrade reboot without specifc user intervention, no database warnings/errors emitted during normal operation

User Experience

• In normal operation, users should see little or no change
• Behavior in error situations is much more robust: forcibly killed transaction no longer causes database inconsistency or corruption

Dependencies

• This change depends on RPM 4.16, support for sqlite rpmdb is not present in older versions
• RPM will grow a new dependency on sqlite-libs
• Technically the rpmdb format is an internal implementation detail of RPM and the data is only accessible through the librpm API, but some software is making assumptions both about the format and/or in particular, file naming. These are being tracked at https://bugzilla.redhat.com/show_bug.cgi?id=1766120
• A new systemd service (rpmdb-rebuild) is introduced, it can be used to flag the database for a maintenance rebuild at boot and is used to perform the automatic BDB -> Sqlite conversion on first boot after upgrade.

Contingency Plan

• Contingency mechanism:
  • Revert the default database back to Berkeley DB backend in the package. The actual conversion can use the same mechanism as in the other direction.
  • The rpm-team does not expect problems with the database backend itself, but we are aware that postponing may be needed due to infrastructure or other tooling not being ready, primarily due to inability to access the database from older releases.

• Contingency deadline: Beta freeze
• Blocks release? Yes

Documentation

• | RPM 4.16 release notes

Release Notes

• RPM database default changes from Berkeley DB to Sqlite in this release. The conversion will take place automatically on the first boot after distribution upgrade via rpmdb-rebuild systemd service.
• Users who have a particular need to stay on Berkeley DB backend can do still so in this release by overriding the configuration manually (eg. echo %_db_backend bdb > /etc/rpm/macros.db) before rebooting, or convert back at any later time. This is discouraged however, support for Berkeley DB will be reduced to read-only in the next release.
• In some circumstances [*] users may see messages like "warning: Found bdb Packages database while attempting sqlite backend: using bdb backend." This is a harmless indication that rpm configuration and what's on disk disagree. It can be silenced either by running rpmdb --rebuilddb to convert the database to match configuration, or by overriding configuration to match what is on disk (see above).
  • Unfortunately, this warning appears after pretty normal upgrade, see https://bugzilla.redhat.com/show_bug.cgi?id=1836108

[*] At least non-bootstrap mock roots where inside and outside rpm database default differs, (rawhide) users who haven't rebooted since the default changed.