`sun/security/pkcs11` tests failing on RHEL8 systems

[sxa]

Please set the title to indicate the test name and machine name where known.

To make it easy for the infrastructure team to repeat and diagnose, please
answer the following questions:

• test suite/name (e.g, BUILD_LIST, TARGET, CUSTOM_TARGET)? TARGET of jdk_security3_0 and jdk_security3_1
• a link into recent Test_ job on https://ci.adoptopenjdk.net which showed the failure: https://ci.adoptopenjdk.net/job/Test_openjdk17_hs_extended.openjdk_x86-64_linux/102/testReport/
• Hyperlink to re-run in Grinder: Full run without limiting to 2 test suites
• Is there an existing issue elsewhere covering this? Not to my knowledge although Two jdk8 security tests consistently failed on test-osuosl-centos74-ppc64le-1 #2625 covers a problem with two pkcs11 tests on Centos7/ppc64le
• Which machine(s) does it work on? As far as I know, all except the one on the next line, although I'm grinding on an s390x RHEL8 machine at G#6520
• Which machine(s) does it fail on? test-aws-rhel8-x64-1: G#6519

Any other details: Split out from #2030 as that issue is resolved
See also #2360 and #1829 which are related to issues on this RHEL8 system.

[sxa]

Failures occurred on the s390x RHEL8 system too ... Will assign myself so I can discuss with the Red Hat internal team.

[sxa]

Upstream issue: https://bugs.openjdk.org/browse/JDK-8295343

[sxa]

@smlambert What would you propose as the next action here?

[smlambert]

Exclude the tests in our problem lists under https://bugs.openjdk.org/browse/JDK-8295343 (and no we currently do not support by OS sub-flavours and supporting such granular exclusions is currently not in plan, unless somehow jtreg already offers that feature), and try to see an expedited fix to JDK-8295343 to be able to re-enable the tests quickly.

[sxa]

See also https://ci.adoptium.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_2/86/testReport/ and https://ci.adoptium.net/job/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0/82/testReport/ which ran on test-equinix_esxi-ubuntu2204-x64-1 and test-equinix_esxi-ubuntu2204-x64-2, suggesting that Ubuntu 22.04 may be subject to the same issues as they both failed the following in the jdk_security3 suites:

• sun/security/pkcs11/Secmod/GetPrivateKey.java.GetPrivateKey
• sun/security/pkcs11/Secmod/JksSetPrivateKey.java.JksSetPrivateKey
• sun/security/pkcs11/fips/TestTLS12.java.TestTLS12

New grinders to determine problematic tests

• RHEL8/s390x and 6899
• RHEL8/x64 (See below for list)
• Ubunu22/x64
• Ubuntu22/x64 container - JDK20

RHEL8/x64 failures (JDK8):

• javax/net/ssl/ServerName/EndingDotHostname.java.EndingDotHostname SSLHandshakeException: Received fatal alert: certificate_unknown
• sun/security/pkcs11/Signature/ByteBuffers.java.ByteBuffers RSA key must be at least 1023 bits
• sun/security/pkcs11/Signature/ReinitSignature.java.ReinitSignature RSA key must be at least 1023 bits
• sun/security/pkcs11/Signature/TestRSAKeyLength.java.TestRSAKeyLength RSA key must be at least 1023 bits
• sun/security/pkcs11/fips/TestTLS12.java.TestTLS12 PKCS11Exception: CKR_ARGUMENTS_BAD
• sun/security/pkcs11/rsa/TestKeyFactory.java.TestKeyFactory Could not create RSA public key
• sun/security/pkcs11/rsa/TestSignatures.java.TestSignatures Could not create RSA public key
• sun/security/rsa/TestKeyFactory.java.TestKeyFactory Exception: Keys not equal
• sun/security/rsa/pss/TestPSSKeySupport.java.TestPSSKeySupport Exception: Keys not equal
• sun/security/tools/keytool/autotest.sh.autotest exit code 1

On test-docker-ubuntu2204-x64-1 (JDK20):

• javax/net/ssl/ServerName/EndingDotHostname.java.EndingDotHostname SSLHandshakeException: Received fatal alert: certificate_unknown
• sun/security/rsa/TestKeyFactory.java.TestKeyFactory Exception: Keys not equal
• sun/security/rsa/pss/TestPSSKeySupport.java.TestPSSKeySupport Exception: Keys not equal

On test-equinix_esxi-ubuntu2204-x64-1 (JDK8):

• javax/net/ssl/ServerName/EndingDotHostname.java.EndingDotHostname
• sun/security/pkcs11/Secmod/GetPrivateKey.java.GetPrivateKey
• sun/security/pkcs11/Secmod/JksSetPrivateKey.java.JksSetPrivateKey
• sun/security/pkcs11/fips/TestTLS12.java.TestTLS12
• sun/security/rsa/TestKeyFactory.java.TestKeyFactory
• sun/security/rsa/pss/TestPSSKeySupport.java.TestPSSKeySupport

Ubuntu 20.04: https://ci.adoptium.net/job/Grinder/6905/ (Same three failures as Ubuntu 22.04)

RHEL8/x64 JDK20: https://ci.adoptium.net/job/Grinder/6909/console - PASSED

Ubuntu 22.04 x64 JDK20: All passed: https://ci.adoptium.net/job/Grinder/6904/

(I was assuming we'd have a nice simple list of things to exclude ... :-) )

On RHEL8/x64: JDK8 (14 failures) JDK11 (16 failures) JDK17 (12 failures) JDK19 (12 failures) JDK20 Nightly (0 failures!) (Note - failures can be /2 since each one is there twice for the _0 and _1 suites)

Test	JDK8	JDK11	JDK17	JDK19	JDK20
pkcs11/Signature/ByteBuffers [*]	❌	❌	✅	✅	✅
pkcs11/Signature/ReinitSignature [*]	❌	❌	✅	✅	✅
pkcs11/Signature/TestRSAKeyLength [*]	❌	❌	❌	❌	n/a
pkcs11/fips/TestTLS12	❌	✅	n/a	n/a	n/a
tools/autotest.sh	❌	❌	n/a	n/a	n/a
pkcs11/rsa/TestKeyFactory [+]	❌	❌	❌	❌	✅
pkcs11/rsa/TestSignatures	❌	❌	❌	❌	✅
pkcs11/KeyStore/Basic.java	✅	❌	❌	❌	n/a
pkcs11/KeyPair/TestKeyPairGenerator	n/a	❌	❌	❌	n/a
tools/NssTest	n/a	n/a	❌	❌	n/a
lib/cacerts/VerifyCACerts	n/a	n/a	n/a	❌	n/a
X509TrustManager/Distrust.java	n/a	n/a	n/a	❌	n/a

[*] - the top 3 in this table are the ones definitely related to the enforced minimum key lengths in RHEL8, and suggests a fix may have gone into to 17+
[+] - Note that while sun/security/pkcs11/rsa/TestKeyFactory.java.TestKeyFactory is a failure, sun/security/pkcs11/rsa/TestKeyFactory passed