Apache SSL config: Many SSL warnings in Glassfish server log (SSLEngine is null)

[eaquigley]

---

Author Name: Kevin Condon (@kcondon)
Original Redmine Issue: 4092, https://redmine.hmdc.harvard.edu/issues/4092
Original Date: 2014-06-10
Original Assignee: Philip Durbin

---

Since fronting demo with Apache to support Shibboleth/SSL, we are seeing numerous glassfish server log errors and at one point some people could not get to the site, though this last issue may not be related.

The repeated warnings are:

[2014-06-09T14:28:29.688-0400] [glassfish 4.0] [WARNING] [] [org.glassfish.grizzly.http.server.util.RequestUtils] [tid: _ThreadID=45 _ThreadName=jk-connector(1)] [timeMillis: 1402338509688] [levelValue: 900] [[
  Unable to populate SSL attributes
java.lang.IllegalStateException: SSLEngine is null
        at org.glassfish.grizzly.ssl.SSLSupportImpl.<init>(SSLSupportImpl.java:87)
        at org.glassfish.grizzly.http.server.util.RequestUtils.populateSSLAttributes(RequestUtils.java:85)
        at org.apache.catalina.connector.Request.populateSSLAttributes(Request.java:4580)
        at org.apache.catalina.connector.Request.getAttributeNames(Request.java:1412)
        at org.apache.catalina.connector.RequestFacade.getAttributeNames(RequestFacade.java:367)
        at org.jboss.weld.context.beanstore.http.RequestBeanStore.getAttributeNames(RequestBeanStore.java:48)
        at org.jboss.weld.context.beanstore.AttributeBeanStore.getPrefixedAttributeNames(AttributeBeanStore.java:207)
        at org.jboss.weld.context.beanstore.AttributeBeanStore.attach(AttributeBeanStore.java:106)
        at org.jboss.weld.context.http.HttpRequestContextImpl.associate(HttpRequestContextImpl.java:52)
        at org.jboss.weld.context.http.HttpRequestContextImpl.associate(HttpRequestContextImpl.java:37)
        at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:190)
        at org.apache.catalina.core.StandardContext.fireRequestInitializedEvent(StandardContext.java:5225)
        at org.apache.catalina.core.StandardHostValve.preInvoke(StandardHostValve.java:647)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:166)
        at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:357)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:260)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:188)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:191)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:168)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:189)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:838)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:564)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:544)
        at java.lang.Thread.run(Thread.java:744)
]]

There are some web posts on this error but I did not see an immediate answer, though discussion centered around gfish/apache config and apache not passing cert info along to gfish:
https://java.net/jira/browse/GLASSFISH-20694?page=com.atlassian.streams.streams-jira-plugin%3Aactivity-stream-issue-tab

---

Redmine related issue(s): 1096

---

[eaquigley]

---

Original Redmine Comment
Author Name: Philip Durbin (@pdurbin)
Original Date: 2014-06-10T18:40:14Z

---

Kevin Condon wrote:

There are some web posts on this error but I did not see an immediate answer, though discussion centered around gfish/apache config and apache not passing cert info along to gfish:
https://java.net/jira/browse/GLASSFISH-20694

Yes, I did notice this and it's easy to replicate by hitting a SWORD URL with curl like this:

curl -u user1:user1 https://dataverse-demo.iq.harvard.edu/dvn/api/data-deposit/v1/swordv2/service-document

I did leave that jira bug number in this commit:

progress on fronting glassfish with apache #1096 #2657 · 71cd4ef · IQSS/dataverse - 71cd4ef

[pdurbin]

Due to #647 we've moved API testing from the demo site to http://apitest.dataverse.org

Here's a new URL for replicating the SSL error:

curl -u pete:pete https://apitest.dataverse.org/dvn/api/data-deposit/v1/swordv2/service-document

[pdurbin]

A couple of times I've tried adding variations on SSLOptions +StdEnvVars +ExportCertData from https://java.net/jira/browse/GLASSFISH-20694 to /etc/httpd/conf.d/ssl.conf but it doesn't seem to help.

For now I'm going with asadmin set-log-levels org.glassfish.grizzly.http.server.util.RequestUtils=SEVERE as a workaround mentioned in that same JIRA ticket. (To verify the workaround is in place: asadmin list-log-levels | grep org.glassfish.grizzly.http.server.util.RequestUtils.)

This only hides the error, of course. Perhaps we could move this issue to a different milestone to come up with a better fix. Sending to QA for comment.

[kcondon]

Not sure what you are looking for: what log level do we want in prod? Generally good idea not to have lots of warnings in logs but if after a reasonable amount of effort it cannot be resolved then not sure what to do.

As for the schedule, that's not my call but hiding the errors makes it less obtrusive for now. Passing back to dev.

[esotiri]

tried suggestions on https://java.net/jira/si/jira.issueviews:issue-html/GLASSFISH-20694/GLASSFISH-20694.html but still see the same in the log.

[pdurbin]

if after a reasonable amount of effort it cannot be resolved

I feel like I've put a reasonable amount of effort in. Someone else is welcome to take a crack at it. My proposed workaround as I mentioned at #643 (comment) is (sadly) to hide the errors. I'm willing to document this in the Installer's Guide. I'll put this in the "in review" milestone for a decision.

[pdurbin]

@esotiri now that Glassfish 4.1 is out, are you interested in seeing if upgrading makes the errors go away?

[kcondon]

This error really fills up the logs and would be a showstopper for production in my opinion.

[pdurbin]

This error really fills up the logs and would be a showstopper for production in my opinion.

Can we try Glassfish 4.1? 1000+ bug fixes: https://pbs.twimg.com/media/BypemmWIMAIFaQN.jpg

[mercecrosas]

did yo talk to Bob and Ellen about our recent Glassfish update in
Consilience?

Mercè Crosas, Ph.D.
Director of Data Science, IQSS
Harvard University
http://iq.harvard.edu/merce-crosas

On Thu, Oct 16, 2014 at 3:57 PM, Philip Durbin [email protected]
wrote:

This error really fills up the logs and would be a showstopper for
production in my opinion.

Can we try Glassfish 4.1?

Reply to this email directly or view it on GitHub
#643 (comment).

[pdurbin]

@kcondon as we discussed a couple weeks ago, I simply documented the workaround for hiding these errors. If you do a build you'll find them here: https://dvn-build.hmdc.harvard.edu/guides/Installers/dataverse-installer-main.html#sslengine-is-null-workaround

Moving to QA.

[esotiri]

applied workaround on dvn-build.