The hmaccalc package uses NSS's digesting functions, and it isn't dealing with certificate databases, private keys, and the like, so there's very little that affects it except for being able to read its input files. Testing it amounts to using it to generate key checksums, and then checking them, verifying that it works as expected with the right key, and fails as expected with the wrong key. Interoperability with the unkeyed *sum tools is also of interest.
We test with the test vectors from the specification at build-time, but if you want to give it a workout anyway, here's the rundown for sha512hmac, with some help from sha512sum (repeating for sha1hmac/sha1sum, sha256hmac/sha256sum, and sha384hmac/sha384sum for completeness):
1. Select a group of files to checksum.
FILESTOCHECK="$HOME/Desktop/*"
A2. Select a key to use.
dd if=/dev/urandom bs=512 count=1 of=keyfile1
A3. Generate HMAC values over the files.
sha512hmac -k keyfile1 $FILESTOCHECK > sums
A4. Verify the HMACs using the right key.
sha512hmac -k keyfile1 -c sums
A5. Try to verify the HMACs using the wrong key, and see that it fails.
dd if=/dev/urandom bs=512 count=1 of=keyfile2 sha512hmac -k keyfile2 -c sums
B1. Generate an unkeyed digest using sha512sum.
sha512sum -k keyfile1 $FILESTOCHECK > sums
B2. Verify the unkeyed digest using sha512hmac.
sha512hmac -u -c sums
C1. Generate an unkeyed digest using sha512hmac.
sha512hmac -u $FILESTOCHECK > sums
C2. Verify the unkeyed digest using sha512sum.
sha512sum -c sums
D1. Repeat all of the tests from part A, adding "-t 260" to each
invocation of sha512sum to instruct it to truncate its results to 80 bits before outputting them or using them for comparison when told to check existing files. (We use "260" here because it's half of 512, plus 4. Half of the usual output length is the lower-bound on values that hmaccalc will accept, and we want to try comparing using both shorter and longer values. For the other tools, the value would have to be different -- 84 for sha1hmac, 132 for sha256hmac, 196 for sha384hmac.) sha512hmac -t 260 -k keyfile1 $FILESTOCHECK > sums sha512hmac -t 260 -k keyfile1 -c sums
D2. Attempt to verify checksums using the same key, but with a different
(or no) truncation size. Each attempt should fail. sha512hmac -t 256 -k keyfile1 -c sums sha512hmac -t 264 -k keyfile1 -c sums sha512hmac -k keyfile1 -c sums