Skip to content

Configuring ESM for TLS

Transport Layer Security (TLS) is used to provide network security and privacy. In addition to providing encryption services, TLS uses trusted certificates to perform client and server authentication, and it uses message authentication codes to ensure data integrity.

Assumptions in this guide

Assumption 1 - This guide assumes the deployment of ESM to a Linux system. The commands for Windows are the same but the commands will need to be updated to reflect the different path structures.

Assumption 2 - This guide assumes that all of the machines in your environment can talk to each other on the short hostname. If you are unable to ping the ESM server from the agents without using the FQDN then you will either need to modify the hostname in the instructions of modify the /etc/hosts file on all of the machines in your environment.

Shutdown your environment before starting this guide

Before implementing the steps in this configuration guide it is important to stop all of the ESM agents and the ESM server in your environment.

Configure ESM Server for TLS

  1. Navigate to the root of your ESM Server installation directory

  2. Generate a private key, a certificate signing request, and a certificate, and add the certificate to the key chain:

./jre/bin/keytool -genkey -keysize 2048 -alias mymachine -validity 36500 -keyalg RSA -dname "CN=localhost,O=MyOrg,L=London,S=UK,C=UK" -keypass changeit -storepass changeit -keystore esm-core/glassfish/domains/domain1/config/keystore.jks
  1. Export the generated certificate for deployment to the ESM agents:
./jre/bin/keytool -export -alias mymachine -file mymachine.cer -keypass changeit -storepass changeit -keystore esm-core/glassfish/domains/domain1/config/keystore.jks

  1. Modify the domain.xml file within following location from the root of your ESM server installation directory esm-core/glassfish/domains/domain1/config/:

  2. Replace lines 221 and 384 with the following updating the cert-nickname value with the alias of the certificate that you generated in step 2 above:

<ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="mymachine">\</ssl>

Configure ESM Agent for TLS

  1. Navigate to the root of your ESM Agent installation directory

  2. In the root of your ESM agent installation, create a certs folder.

  3. Copy the mymachine.cer file created in step 3 above into the certs folder created in the step above
  4. Import the certificate into the agent's trusted certificate keystore 
    ./jre/bin/keytool -import -trustcacerts -noprompt -file certs/mymachine.cer -alias mymachine -storepass changeit -keystore jre/lib/security/cacerts
  5. Verify that the certificate was successfully added to the keystore 
    ./jre/bin/keytool -list -v -storepass changeit -keystore jre/lib/security/cacerts
  6. Upcate the following configuration option in conf/esmconfig.sh 
    ESMUSESSL=true
  7. Restart the ESM agent

Additional TLS Tasks

Verify certificate is in the keystore

On the ESM server:

./jre/bin/keytool -list -v -storepass changeit -keystore esm-core/glassfish/domains/domain1/config/keystore.jks

Delete certificate from the keychain

On the ESM Server machine:

jre/bin/keytool -delete -alias localhost -storepass changeit -keystore esm-core/glassfish/domains/domain1/config/keystore.jks

Common Errors

Incorrect hostname or certificate used in generating the certificate

WARNING: java.net.ConnectException: Connection refused (Connection refused)
  javax.xml.ws.WebServiceException: java.net.ConnectException: Connection refused (Connection refused)
  at com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:210)
  at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:160)
  at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:93)
  at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
  at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
  at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
  at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
  at com.sun.xml.ws.client.Stub.process(Stub.java:222)
  at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
  at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
  at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
  at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:118)
  at com.sun.proxy.$Proxy32.sendPackage(Unknown Source)
  at com.boemska.d$1.run(ESMAgentWorker.java:712)
  at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.ConnectException: Connection refused (Connection refused)
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:589)
  at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
  at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
  at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
  at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
  at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
  at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:307)
  at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:298)
  at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:281)
  at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:96)
  at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.setNewClient(AbstractDelegateHttpsURLConnection.java:80)
  at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:705)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1585)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
  at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
  at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
  at com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:207)

The domain.xml file is not updated correctly

The following error may be seen when the domain.xml file is not updated correctly on the ESM server. 

SEVERE: Unable to reach the server.
com.sun.xml.ws.wsdl.parser.InaccessibleWSDLException: 2 counts of InaccessibleWSDLException.
javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names present
javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names present

at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.tryWithMex(RuntimeWSDLParser.java:172)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:144)
at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:264)
at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:227)
at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:175)
at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:104)
at javax.xml.ws.Service.<init>(Service.java:77)
at com.boemska.ws.ESMServiceService.<init>(ESMServiceService.java:58)
at com.boemska.d.<init>(ESMAgentWorker.java:217)
at com.boemska.d.a(ESMAgentWorker.java:154)
at com.boemska.Main.main(Main.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.tanukisoftware.wrapper.WrapperStartStopApp.run(WrapperStartStopApp.java:238)
at java.lang.Thread.run(Thread.java:748)

Incomplete or incorrect Agent setup

SEVERE: Unable to reach the server.
com.sun.xml.ws.wsdl.parser.InaccessibleWSDLException: 2 counts of InaccessibleWSDLException.

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.tryWithMex(RuntimeWSDLParser.java:172)
at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:144)
at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:264)
at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:227)
at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:175)
at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:104)
at javax.xml.ws.Service.<init>(Service.java:77)
at com.boemska.ws.ESMServiceService.<init>(ESMServiceService.java:58)
at com.boemska.d.<init>(ESMAgentWorker.java:217)
at com.boemska.d.a(ESMAgentWorker.java:154)
at com.boemska.Main.main(Main.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.tanukisoftware.wrapper.WrapperStartStopApp.run(WrapperStartStopApp.java:238)
at java.lang.Thread.run(Thread.java:748)