Solve problem DBeaver, openssl s_client cannot terminate ssl cert on java application but web browser can

Few days ago My devops lead invite me for case that Devops team have update ssl certificate. They already test after update ssl certificate with browser before a week that old certificate will expired. At that time most of user can open and access to web browser and ok browser show secure connection.

I need to tell story. This application belong to Analytic team and handle by DevOps of Analytic team. One month ago no one left in DevOps of Analytic team before they left they short hand over to My DevOps team. If you are techies this is hardest for do anything for application or system or upgrade but often happen with everyone and we can’t avoid. So we need to help together to move on and support.

So issue is web browser can terminate ssl certificate and show secure connection but DBeaver for query PrestoDB error with “unable to find valid certifaction path to requested target” or “unable to get local issuer certificate” or “unable to verify the first certificate”.

Day1

After I got problem from my DevOps lead. I start to check with browser and ok It can show secure connection.

I still don’t have information to connect database with DBeaver. I try to test with https://www.ssllabs.com/ssltest but it can not resolve because domain name map internal ip.

• I start with request for server key ca-bundled and private key.
• I checking with script.

#!/bin/bash

cat $1 $2 > tls.crt
cp $3 tls.key


openssl x509 -in tls.crt -text -noout
openssl rsa -in tls.key -check

# Store modulus of tls.crt
crt_modulus=$(openssl x509 -noout -modulus -in tls.crt | openssl md5)

# Store modulus of tls.key
key_modulus=$(openssl rsa -noout -modulus -in tls.key | openssl md5)

# Compare the results
if [ "$crt_modulus" = "$key_modulus" ]; then
    echo "Match: Private key corresponds to the certificate"
else
    echo "No match: Private key does not correspond to the certificate"
fi

./genkey.bash STAR_tdg-analytics-platform_io_2024.crt STAR_tdg-analytics-platform_io_2024.crt private_key.txt

• Result

• It mean server ca-bundle have valid for date and private key ok and match with server and ca-bundle.
• My devops lead try to re update and redeploy ssl certificate but user still same problem.
• I try to search and found that why Browser can terminate but other tools may can’t. refer to https://pavolkutaj.medium.com/unable-to-verify-the-first-certificate-with-openssl-47eecb652a9b
• I try to check with openssl s_client

openssl s_client -connect <internal-domain.io>:443 -servername <internal-domain.io>

• Result have error same like show with DBeaver app.

• So I test with -CAfile STAR_tdg-analytics-platform_io_2024.crt

openssl s_client -connect <internal-domain.io>:443 -servername <internal-domain.io> -CAfile STAR_tdg-analytics-platform_io_2024.crt

• Result still error.
• Now I think ca-bundle or (intermediate and root) may be wrong.
• I try to download intermediate and root ca file from https://support.sectigo.com/articles/Knowledge/Sectigo-Intermediate-Certificates

• I create new ca-bundle.

cp SHA-2\ Root\ \ USERTrust\ RSA\ Certification\ Authority.crt ca-bundle-fix.crt
cat SectigoRSADomainValidationSecureServerCA.crt >> ca-bundle-fix.crt 

• I test again new ca-bundle then result fine.

openssl s_client -connect <internal-domain.io>:443 -servername <internal-domain.io> -CAfile ca-bundle-fix.crt

• Now I think high possibility that old ca-bundle will wrong or not up to date.
• I test again with old ca-bundle then error

openssl verify STAR_tdg-analytics-platform_io_2024.crt

• But from new ca-bundle fine.

openssl verify ca-bundle-fix.crt

• I test with more option for old ca-bundle with server then still error.

openssl verify -CAfile STAR_tdg-analytics-platform_io_2024.ca-bundle  STAR_tdg-analytics-platform_io_2024.crt

• But new ca-bundle with server fine.

openssl verify -CAfile ca-bundle-fix.crt  STAR_tdg-analytics-platform_io_2024.crt 

Finally I think ca-bundle wrong and I send new ca-bundle to my Devops lead to upgrade ssl certificate.

Day2

• Devops lead try to update ssl cert but this time error when redeploy to deployment and pod crashloopbackoff.
• We have meeting for pod crashloopbackoff. Devops lead show step to update ssl to secret and redeploy for java application.
• I just know step for update ssl that need to convert from our cert to JKS file.

openssl pkcs12 -export -in <server.crt> -inkey key.key -out keystore.p12 -name tdg-analytics-platform -CAfile <new-ca-bunlde> -caname root

keytool -importkeystore -srckeystore keystore.p12 -srcstoretype PKCS12 -destkeystore keystore.jks -deststoretype JKS
(this step it will recommend to run:
keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.jks -deststoretype pkcs12
)

keytool -list -v -keystore keystore.jks

base64 -i keystore.jks > keystore.yaml

cat keystore.yaml

• Error in log deployment show about “java.io.IOException: Invalid keystore format”.
• I try to remove step recommend so new step will be.

openssl pkcs12 -export -in <server.crt> -inkey key.key -out keystore.p12 -name tdg-analytics-platform -CAfile <new-ca-bunlde> -caname root
keytool -importkeystore -srckeystore keystore.p12 -srcstoretype PKCS12 -destkeystore keystore.jks -deststoretype JKS
keytool -list -v -keystore keystore.jks
base64 -i keystore.jks > keystore.yaml
cat keystore.yaml

• Now I can deploy to Deployment and pod running as well.

Day3

• User tell us that still error.
• I still have same error too with.

openssl s_client -connect <internal-domain.io>:443 -servername <internal-domain.io>

• I try to review again for step to create keystore.jks after a hour. I try many thing but this step is ok.

# server + new ca-bundle
cat STAR_tdg-analytics-platform_io_2024.crt ca-bundle-fix.crt > tls.crt
cp private_key.txt tls.key

# no need seprarate -in with -CAfile <new-ca-bunlde>
openssl pkcs12 -export -in tls.crt -inkey tls.key -out keystore.p12 -name tdg-analytics-platform -caname root
keytool -importkeystore -srckeystore keystore.p12 -srcstoretype PKCS12 -destkeystore keystore.jks -deststoretype JKS
keytool -list -v -keystore keystore.jks
base64 -i keystore.jks > keystore.yaml
cat keystore.yaml

• Then I test

openssl s_client -connect <internal-domain.io>:443 -servername <internal-domain.io>

• Result fine

• OH!!!!! I think old ca-bundle may can use?
• I try old ca-bundle.

# server + new ca-bundle
cat STAR_tdg-analytics-platform_io_2024.crt STAR_tdg-analytics-platform_io_2024.ca-bundle > tls.crt
cp private_key.txt tls.key

openssl pkcs12 -export -in tls.crt -inkey tls.key -out keystore.p12 -name tdg-analytics-platform -caname root
keytool -importkeystore -srckeystore keystore.p12 -srcstoretype PKCS12 -destkeystore keystore.jks -deststoretype JKS
keytool -list -v -keystore keystore.jks
base64 -i keystore.jks > keystore.yaml
cat keystore.yaml

OH!!!!! result is fine too.

Now I think problem not because ca-bundle file. but problem with step to create “keystore.jks”

• Now User can connect DBeaver for query PrestoDB
• Problem and issue have been solved.

Today our issue finish in time. I have great experience in my DevOps skill problem solving so much. Thank you our DevOps team for being and help together.

Sometime I almost give up to find solution. But when I can clear problem I’m very happy.

I hope this solution may help some one.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Credit : TrueDigitalGroup

— — — — — — — — — — — — — — — — — — — — — — — — — — — — —