Revision as of 18:32, 3 July 2017

NSS Default File Format SQL

Summary

Change the NSS library default to use the sqlite based data storage, when applications don't specify their preferred storage file format.

Owner

Name: Kai Engert
Email: kengert@redhat.com
Release notes owner:

Current status

Targeted release: Fedora 27
Last updated: 2017-07-03
Tracker bug: <will be assigned by the Wrangler>

Detailed Description

Applications that use the NSS library often use a database for storage of keys, certificates and trust. NSS supports two different file formats, one called DBM (based on berkeley DB files) and another one called SQL (based on sqlite DB files).

Today's default file format used by NSS, used when applications omit the type parameter, is the older DBM file format, which forbids parallel access to the storage. The suggestion is to change the default file format to SQL, which allows parallel access to the storage.

Applications, or users using the NSS command line utilities, often provide the database storage location using a simple directory path parameter. Some might not be aware, or forget, that the parameter can be prefixed with a type modifier, either "dbm:" or "sql:".

As a result, when not providing this parameter, the file format used will be the fragile DBM file format. This is particuarly problematic, if a user attempts to modify the NSS storage using command line tools, while another process, such as a daemon, is running concurrently, which also accesses the same database in the DBM file format. This often results in corrupted database storage, which cannot be recovered.

By changing the default, all applications that currently use the DBM file format, will automatically be migrated to the SQL file format. NSS has the ability to discover if a storage location (a directory) contains the DBM file format. If configured to use the modern SQL format, NSS will automatically perform a one-time conversion from the DBM to the SQL format.

The same applies to the NSS command line utilities. If the NSS library default is changed to SQL, the NSS tools will also trigger the one-time conversion, or access the already converted files.

Benefit to Fedora

The benefit will be less data corruption, because the SQL file format explicitly allows parallel access to the storage files by multiple applications.

NSS has been supporting this new file format for many years, and several applications have already opted in to explicitly use the new file format, so it can be considered stable.

Scope

Proposal owners:

A small downstream patch needs to be applied to the NSS library package, which changes the library default.

Other developers:

It's up to developers of NSS applications, if they accept the new default and an automatic conversion, or if they prefer to continue to use the classic DBM storage format. Although not recommended, developers can easily do so, by adding a "dbm:" prefix to the storage parameter they provide to NSS at NSS library initialization time.

Release engineering: #Releng issue number (a check of an impact with Release Engineering is needed)

No help should be necessary. No mass rebuild necessary.

- List of deliverables: No deliverables.

Policies and guidelines: N/A

Trademark approval: N/A

Upgrade/compatibility impact

Data will be migrated automatically.

How To Test

Each application or server softare that uses NSS should be tested for successful data migration. Ideally, an application/server configuration should be used, which already contains data from a previous Fedora version. It should be tested that after migration to the new Fedora version, the software continues to operate as expected. In particular, this means that the software still has access to the locally configure private keys, certificates, trust settings which the application/server has configured. (This only applies to applications that perform this kind of operation.)

User Experience

The result changes will be restricted to the file system level, and to applications that use the NSS library default.

Previously, the following filenames were used for NSS storage:

* cert8.db
* key3.db
* secmod.db

After the change, the following filenames will be used:

* cert9.db
* key4.db
* pkcs11.txt

Applications that migrate data from an older version will contain both sets of files in their storage. However, after the conversion, the old files will no longer be used, they can be considered a one-time backup.

Applications that are executed for the first time with the new library default already effective, will only get the new storage files.

Dependencies

No dependencies.

Contingency Plan

Contingency mechanism: (What to do? Who will do it?) Revert the shipped configuration, NSS maintainer.
Contingency deadline: beta freeze
Blocks release? No
Blocks product? No

Documentation

N/A (not a System Wide Change)

Release Notes

@@ Line 1: / Line 1: @@
-{{admon/important | Comments and Explanations | The page source contains comments providing guidance to fill out each section. They are invisible when viewing this page. To read it, choose the "view source" link.<br/> '''Copy the source to a ''new page'' before making changes!  DO NOT EDIT THIS TEMPLATE FOR YOUR CHANGE PROPOSAL.'''}}
+= NSS Default File Format SQL =
-<!-- Self Contained or System Wide Change Proposal?
-Use this guide to determine to which category your proposed change belongs to.
-Self Contained Changes are:
-* changes to isolated/leaf package without the impact on other packages/rest of the distribution
-* limited scope changes without the impact on other packages/rest of the distribution
-* coordinated effort within SIG with limited impact outside SIG functional area, accepted by the SIG
-System Wide Changes are:
-* changes that does not fit Self Contained Changes category touching
-* changes that require coordination within the distribution (for example mass rebuilds, release engineering or other teams effort etc.)
-* changing system defaults
-For Self Contained Changes, sections marked as "REQUIRED FOR SYSTEM WIDE CHANGES" are OPTIONAL but FESCo/Wrangler can request more details (especially in case the change proposal category is
-improper or updated to System Wide category). For System Wide Changes all fields on this form are required for FESCo acceptance (when applies).
-We request that you maintain the same order of sections so that all of the change proposal pages are uniform.
--->
-<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
-<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
-= Change Proposal Name <!-- The name of your change proposal --> =
 == Summary ==
-<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. -->
+Change the NSS library default to use the sqlite based data storage, when applications don't specify their preferred storage file format.
 == Owner ==
-<!--
+* Name: [[User:kengert| Kai Engert]]
-For change proposals to qualify as self-contained, owners of all affected packages need to be included here. Alternatively, a SIG can be listed as an owner if it owns all affected packages.
+* Email: kengert@redhat.com
-This should link to your home wiki page so we know who you are.
--->
-* Name: [[User:FASAcountName| Your Name]]
-<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
-* Email: <your email address so we can contact you, invite you to meetings, etc.>
 * Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
 <!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
@@ Line 46: / Line 17: @@
 == Current status ==
-* Targeted release: [[Releases/<number> | Fedora <number> ]]
+* Targeted release: [[Releases/27 | Fedora 27 ]]
 * Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}
 <!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page
@@ Line 59: / Line 30: @@
 == Detailed Description ==
+Applications that use the NSS library often use a database for storage of keys, certificates and trust. NSS supports two different file formats, one called DBM (based on berkeley DB files) and another one called SQL (based on sqlite DB files).
+Today's default file format used by NSS, used when applications omit the type parameter, is the older DBM file format, which forbids parallel access to the storage.
+The suggestion is to change the default file format to SQL, which allows parallel access to the storage.
+Applications, or users using the NSS command line utilities, often provide the database storage location using a simple directory path parameter. Some might not be aware, or forget, that the parameter can be prefixed with a type modifier, either "dbm:" or "sql:".
-<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
+As a result, when not providing this parameter, the file format used will be the fragile DBM file format.
+This is particuarly problematic, if a user attempts to modify the NSS storage using command line tools, while another process, such as a daemon, is running concurrently, which also accesses the same database in the DBM file format. This often results in corrupted database storage, which cannot be recovered.
+By changing the default, all applications that currently use the DBM file format, will automatically be migrated to the SQL file format. NSS has the ability to discover if a storage location (a directory) contains the DBM file format. If configured to use the modern SQL format, NSS will automatically perform a one-time conversion from the DBM to the SQL format.
+The same applies to the NSS command line utilities. If the NSS library default is changed to SQL, the NSS tools will also trigger the one-time conversion, or access the already converted files.
 == Benefit to Fedora ==
+The benefit will be less data corruption, because the SQL file format explicitly allows parallel access to the storage files by multiple applications.
+NSS has been supporting this new file format for many years, and several applications have already opted in to explicitly use the new file format, so it can be considered stable.
-<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->
 == Scope ==
 * Proposal owners:
-<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
+A small downstream patch needs to be applied to the NSS library package, which changes the library default.
-* Other developers: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Other developers:
-<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
+It's up to developers of NSS applications, if they accept the new default and an automatic conversion, or if they prefer to continue to use the classic DBM storage format. Although not recommended, developers can easily do so, by adding a "dbm:" prefix to the storage parameter they provide to NSS at NSS library initialization time.
 * Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE AS WELL AS FOR SELF CONTAINED CHANGES -->
-<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.
+No help should be necessary. No mass rebuild necessary.
-The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
-** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->
-* Policies and guidelines: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: No deliverables.
-<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
-* Trademark approval: N/A (not needed for this Change)
+* Policies and guidelines: N/A
-<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
+* Trademark approval: N/A
 == Upgrade/compatibility impact ==
-<!-- What happens to systems that have had a previous versions of Fedora installed and are updated to the version containing this change? Will anything require manual configuration or data migration? Will any existing functionality be no longer supported? -->
+Data will be migrated automatically.
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-N/A (not a System Wide Change)
 == How To Test ==
-<!-- This does not need to be a full-fledged document. Describe the dimensions of tests that this change implementation is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.
+Each application or server softare that uses NSS should be tested for successful data migration. Ideally, an application/server configuration should be used, which already contains data from a previous Fedora version. It should be tested that after migration to the new Fedora version, the software continues to operate as expected. In particular, this means that the software still has access to the locally configure private keys, certificates, trust settings which the application/server has configured. (This only applies to applications that perform this kind of operation.)
-Remember that you are writing this how to for interested testers to use to check out your change implementation - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your change.
+== User Experience ==
+The result changes will be restricted to the file system level, and to applications that use the NSS library default.
-A good "how to test" should answer these four questions:
+Previously, the following filenames were used for NSS storage:
+ * cert8.db
+ * key3.db
+ * secmod.db
-. What special hardware / data / etc. is needed (if any)?
+After the change, the following filenames will be used:
-. How do I prepare my system to test this change? What packages
+ * cert9.db
-need to be installed, config files edited, etc.?
+ * key4.db
-. What specific actions do I perform to check that the change is
+ * pkcs11.txt
-working like it's supposed to?
-. What are the expected results of those actions?
--->
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+Applications that migrate data from an older version will contain both sets of files in their storage. However, after the conversion, the old files will no longer be used, they can be considered a one-time backup.
-N/A (not a System Wide Change)
-== User Experience ==
+Applications that are executed for the first time with the new library default already effective, will only get the new storage files.
-<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-N/A (not a System Wide Change)
 == Dependencies ==
-<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->
+No dependencies.
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-N/A (not a System Wide Change)
 == Contingency Plan ==
+* Contingency mechanism: (What to do?  Who will do it?) Revert the shipped configuration, NSS maintainer.
-<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
+* Contingency deadline: beta freeze
-* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Blocks release? No
-<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
+* Blocks product? No
-* Contingency deadline: N/A (not a System Wide Change)  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
-* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-* Blocks product? product <!-- Applicable for Changes that blocks specific product release/Fedora.next -->
 == Documentation ==

Search

Changes/NSSDefaultFileFormatSql: Difference between revisions