Editing Test Day:Test Day:2024-07-09 IPA HSM

<!-- Make sure the pages is named as "Test Day:YYYY-MM-DD topic" -->

{{Infobox_group
| name = IPA HSM Test DAy
| image = [[File:test-days-banner.svg|300px|link=QA/Test Days]]
| date = '''2024-07-09 to 2024-07-11'''
| time = all day
| website = [[QA/Test Days]]
| matrix = {{matrix|#test-day:fedoraproject.org}}
| fedora_mailing_list = test
}}

{{admon/note | Can't make the date? | If you come to this page after the test day is completed, your testing is still valuable, and you can use the information on this page to test, file any bugs you find, and add your results to the results section. If this page is more than a month old when you arrive here, please check the [[QA/Test_Days|current schedule]] and see if a similar but more recent Test Day is planned or has already happened.}}

== What to test? ==

<!-- Describe in detail what this test day is about and why would users want to participate in it. What makes this interesting for them? What's new and exciting in your software or a feature? -->

This [[QA/Test Days|Test Day]] will focus on '''FIXME'''

== Who's available ==

The following cast of characters will be available testing, workarounds, bug fixes, and general discussion:
* Development - [[User:Developer1|rcritten]] (rcritten)
* Fedora QA - [[User:Sumantro Mukherjee|sumantrom]] (sumantrom)

You can chat with me on [https://docs.fedoraproject.org/en-US/project/communications/ Matrix]. See the infobox on top of the page to learn where to join.

== Prerequisite for Test Day == 

* A virtual machine or a bare metal machine
* An installation of Fedora 40 (ideally Server). Make sure to fully update your system. If installing a fresh system, it's recommended to use the latest [https://www.happyassassin.net/nightlies.html nightly image].

== What to test ==

This will focus on testing IPA support for generating and storing CA private keys on a Hardware Security Module (HSM).

There are two supported HSMs: the nCipher nShield Connect XC (High) and the Thales TCT Luna Network HSM Luna-T7. Firmware versions can vary so only specific ones are supported.

Using softhsm2 as an HSM is usable for testing. It is not recommended for production because it is not a truly networked HSM and the private keys live on a file system (protected yes but not at a hardware level). Because it is not networked, users will need to carefully synchronize the token files whenever any private key generation is done to ensure the contents are identical.

== How to test? ==

=== Install freeIPA packages ===
# dnf -y install freeipa-server-dns

=== Pre-configure the HSM ===

If you are using softhsm2, grant read access to the tokens: 

  # usermod pkiuser -a -G ods

=== Set up environment variables on each machine/VM ===

  # export TOKEN_PASSWORD=password
  # export ADMIN_PASSWORD=password
  # export DM_PASSWORD=password

If using a supported hardware HSM ensure that it is working properly and have the token name and PKCS#11 library path handy.

=== In between tests ===

To re-use test machines in between installations:

On replica (if there is one)
  # ipa server-del $HOSTNAME
  # ipa-server-install –uninstall -U

On the initial IPA server

  # ipa-server-install –uninstall -U

If using softhsm2 you will also need to delete and re-create the token. To delete the token:

  # softhsm2-util --delete-token --token ipa_token

This should return the machine(s) to the pre-installed state.

Visit the '''[http://testdays.fedoraproject.org/events/191 results page]''' and click on the column title links to see the tests that need to be run: most column titles are links to a specific test case. Follow the instructions there, then enter your results by clicking the ''Enter result'' button for the test.

== Test Cases ==

=== Test Case 1 ===

Install a basic IPA server with HSM

How to test:

==== Install the freeipa packages ====
  # dnf install freeipa-server freeipa-server-dns softhsm -y

==== Rename the hostname with the domain to be used with ipa ====
  # hostnamectl hostname ipa.example.test
  # echo “<ip-address> ipa.example.test ” >> /etc/hosts

==== Create softhsm token ====
  # runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token

==== Install the IPA server ====
  # ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD

==== Ensure that certificate stored with the hsm token ====
  # certutil -L -d /etc/pki/pki-tomcat/alias -h ipa_token

  ipa_token:ocspSigningCert cert-pki-ca                    	u,u,u
  ipa_token:subsystemCert cert-pki-ca                      	u,u,u
  ipa_token:auditSigningCert cert-pki-ca                   	u,u,Pu
  ipa_token:caSigningCert cert-pki-ca                      	CTu,Cu,Cu

==== Basic IPA Sanity ====

Test that basic things within IPA work.

  # kinit admin
  # ipa user-add --first tim --last user --password tuser
  # id tuser
  # kinit tuser (and reset password)
  # ipa user-show tuser

=== Test Case 2 ===

Install an IPA server and replica with HSM

How to test:

Install the freeipa packages on both machines
  # dnf install freeipa-server freeipa-server-dns softhsm -y

Rename the hostname with the domain to be used with ipa

server:
  # hostnamectl hostname ipa.example.test
  # echo “<ip-address> ipa.example.test ” >> /etc/hosts

==== Create softhsm token on ipa.example.test only ====
  # runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token

==== Install the IPA server ====

  # ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD

==== Identify the token directory on ipa.example.test ====

  # ls -1tr /var/lib/softhsm/tokens/ | tail -1

This will return a UUID like e373ded4-8763-29e9-dff9-e41f6930297e

==== Copy token data to replica ====
  # export token=”<UUID>”
  # rsync -avp $IPA_SERVER_IP:/var/lib/softhsm/tokens/${token} /var/lib/softhsm/tokens/

==== Add a DNS server to ipa.example.test (it will make things easier) ====

  # ipa-dns-install --no-forwarders --auto-reverse

==== Add the replica IP information to DNS (on the IPA server) ====

  # kinit admin
  # ipa dnsrecord-add example.test. replica --a-rec=$REPLICA_IP

==== Set replica hostname ====

  # hostnamectl hostname replica.example.test

==== Configure the replica to use the IPA server DNS ====

  # resolvectl dns eth0 $IPA_SERVER_IP:53

==== Install ipa-replica ====

  # ipa-replica-install --domain example.test --realm EXAMPLE.TEST --admin-password $ADMIN_PASSWORD -U -N --setup-ca --token-password $TOKEN_PASSWORD

==== Verify that the certificate serial numbers are the same. Run this on both machines. ====

  # certutil -L -d /etc/pki/pki-tomcat/alias -h ipa_token -n 'ipa_token:subsystemCert cert-pki-ca' |grep -A1 'Serial Number:'

        Serial Number: 4 (0x4)

=== Test Case 3 ===

Install an IPA server with a KRA

How to test:

==== Install the freeipa packages ====

  # dnf install freeipa-server freeipa-server-dns softhsm -y

==== Rename the hostname with the domain to be used with ipa ====

  # hostnamectl hostname ipa.example.test
  # echo “<ip-address> ipa.example.test” >> /etc/hosts

==== Create softhsm token ====
  # runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token

==== Install the IPA server ====

  # ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD --setup-kra

==== Ensure that certificate stored with the hsm token (note the kra certs) ====

  # certutil -L -d /etc/pki/pki-tomcat/alias -h ipa_token

==== Verify that the KRA is functional ====

  # kinit admin
  # ipa vault-add test
  # ipa vault-archive test --data Zm9vCg==
  # ipa vault-retrieve test

=== Test Case 4 ===

Install an IPA server and replica with KRA

How to test:

==== Install the freeipa packages ====

  # dnf install freeipa-server freeipa-server-dns softhsm -y

==== Rename the hostname with the domain to be used with ipa ====

  # hostnamectl hostname ipa.example.test
  # echo “<ip-address> ipa.example.test” >> /etc/hosts

==== Create softhsm token ====

  # runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token

==== Install IPA server with a KRA ====

  # ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD --setup-kra

==== Identify the token directory on ipa.example.test ====

  # ls -1tr /var/lib/softhsm/tokens/ | tail -1

This will return a UUID like e373ded4-8763-29e9-dff9-e41f6930297e

==== Copy token data to replica ====
  # export token=”<UUID>”
  # rsync -avp $IPA_SERVER_IP:/var/lib/softhsm/tokens/${token} /var/lib/softhsm/tokens/

==== Add a DNS server to ipa.example.test (it will make things easier) ====

  # ipa-dns-install --no-forwarders --auto-reverse

==== Add the replica IP information to DNS (on the IPA server) ====

  # kinit admin
  # ipa dnsrecord-add example.test. replica --a-rec=$REPLICA_IP

==== Set replica hostname ====

  # hostnamectl hostname replica.example.test

==== Configure the replica to use the IPA server DNS ====

  # resolvectl dns eth0 $IPA_SERVER_IP:53

==== Install an IPA replica with a KRA ====

  # ipa-replica-install --domain example.test --realm EXAMPLE.TEST --admin-password $ADMIN_PASSWORD -U -N --setup-ca --token-password $TOKEN_PASSWORD --setup-kra

==== Verify that the KRA is functional ====

This vault can be created on either machine. Please verify that the vault is accessible on both.

  # kinit admin
  # ipa vault-add test
  # ipa vault-archive test --data Zm9vCg==
  # ipa vault-retrieve test

=== Test Case 5 ===

Install an IPA server with HSM and renew a certificate outside the grace period

How to test:

==== Install the freeipa packages ====

  # dnf install freeipa-server freeipa-server-dns softhsm -y

==== Rename the hostname with the domain to be used with ipa ====

  # hostnamectl hostname ipa.example.test
  # echo “<ip-address> ipa.example.test” >> /etc/hosts

==== Create softhsm token ====

  # runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token

==== Install the IPA server ====

  # ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD

==== Move date to within the expiration grace period ====

  # date -s +2years+11months+20days
  # ipactl restart
  # sleep 90

==== Force issuance of new certs ====

  # ipa-cert-fix (answer yes)

It will take a bit for new certs to be issued and for certmonger to notice. To monitor it:
watch -n 5 'getcert list | grep status'

==== Expected results ====

The expired certificates (all but the CA cert) will be re-issued. As you monitor using getcert list you may see the certificates go through different states including:
SUBMITTING, GENERATING_CSR,  POST_SAVED_CERT, NEED_TO_SUBMIT and/or NEED_TO_SAVE_CERT

==== Return date to current time ====

Uninstall the IPA server prior to moving time backwards.

 # date +s +2years+11months+20days 

=== Test Case 6 ===

Install an IPA server with HSM and renew a certificate inside the grace period

How to test:

==== Install the freeipa packages ====
  # dnf install freeipa-server freeipa-server-dns softhsm -y

==== Rename the hostname with the domain to be used with ipa ====
  # hostnamectl hostname ipa.example.test
  # echo “<ip-address> ipa.example.test” >> /etc/hosts

==== Create softhsm token ====

  # runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token

==== Install the IPA server ====

  # ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD

==== Move date to near the end of the grace period ====

  # date -s +1years+11months+20days

==== Monitor the renewal ====

It will take a bit for new certs to be issued and for certmonger to notice. To monitor it:

  # watch -n 5 'getcert list | grep status'

The expired certificates (all but the CA cert) will be re-issued. As you monitor using getcert list you may see the certificates go through different states including:
SUBMITTING, GENERATING_CSR,  POST_SAVED_CERT, NEED_TO_SUBMIT and/or NEED_TO_SAVE_CERT

If one certificate fails to renew with CA_UNREACHABLE wait until all of the certs are either in this state or MONITORING. Then restart certmonger and run the watch again. Certificate renewal can be bumpy as lots of service restarts happen and the renewals can step on one another.

== Reporting bugs ==

<!--
If there are multiple places where to report bugs (e.g. general GNOME bugs go into upstream GNOME GitLab, and only packaging issues or non-GNOME bugs go to Red Hat Bugzilla), mention it and advise users where to file which issues.
-->

Perhaps you've found an already-reported bug. Please look at:
* [https://qa.fedoraproject.org/blockerbugs/milestone/41/beta/buglist Fedora 41 Beta blocker bugs]
* [https://qa.fedoraproject.org/blockerbugs/milestone/41/final/buglist Fedora 41 Final blocker bugs]

All new bugs should be reported into the [https://pagure.io/freeipa/issues upstream bug tracker]. A less-preferred alternative is to file them into [https://issues.redhat.com Red Hat JIRA], in most cases against the <code>ipa</code> component. 

{{admon/tip | We really need bug reports! | Please note that just mentioning your problem into the comments section on the results page is not very helpful. Very often those problems only happen in specific circumstances, or with specific steps taken. We need the logs and screenshots, and we need to be able to ask you followup questions. Please file bug reports, it's much more useful than a short comment. Thank you!}}

When filing the bug, it's very helpful to include:
* exact steps you've performed (and whether you can reproduce it again)
* screenshots or videos, if applicable
* system journal (log), which you can retrieve by <code>journalctl -b > journal.txt</code>
* all output in a terminal, if started from a terminal
* your system description

If you are unsure about exactly how to file the report or what other information to include, just [[#Who's_available|ask us]].

Please make sure to link to the bug when submitting your test result, thanks!

== Test Results ==

Test results will be exported here once the test day is over. See [[#How_to_test?|How to test?]] section for information how to submit results and see the live results.



[[Category:Fedora 41 Test Days]]