Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update 5.5 -> 5.6 fails with both payara 5.2021.5 and 5.2021.10 #8376

Closed
lmaylein opened this issue Jan 28, 2022 · 9 comments
Closed

Update 5.5 -> 5.6 fails with both payara 5.2021.5 and 5.2021.10 #8376

lmaylein opened this issue Jan 28, 2022 · 9 comments

Comments

@lmaylein
Copy link
Contributor

I have tried to update from 5.5 to 5.6 according to https://github.com/IQSS/dataverse/releases/tag/v5.6.

When I install payara version 5.2021.5 and copy domain1 to it, the start of payara fails:

Jan 28 13:10:31 heidata4 systemd: Starting GlassFish Server v5...
Jan 28 13:10:33 heidata4 asadmin: Waiting for domain1 to start ..Error starting domain domain1.
Jan 28 13:10:33 heidata4 asadmin: The server exited prematurely with exit code 1.
Jan 28 13:10:33 heidata4 asadmin: Before it died, it produced the following output:
Jan 28 13:10:33 heidata4 asadmin: Launching Payara Server on Felix platform
Jan 28 13:10:33 heidata4 asadmin: ERROR: Bundle fish.payara.server.internal.core.glassfish [218] Error starting file:/usr/local/payara5/glassfish/modules/glassfish.jar (org.osgi.framework.BundleException: Unable to resolve fish.payara.server.internal.core.glassfish [218](R 218.0): missing requirement [fish.payara.server.internal.core.glassfish [218](R 218.0)] osgi.wiring.package; (&(osgi.wiring.package=org.osgi.framework)(version>=1.9.0)(!(version>=2.0.0))) Unresolved requirements: [[fish.payara.server.internal.core.glassfish [218](R 218.0)] osgi.wiring.package; (&(osgi.wiring.package=org.osgi.framework)(version>=1.9.0)(!(version>=2.0.0)))])
Jan 28 13:10:33 heidata4 asadmin: Jan 28, 2022 1:10:32 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner createBundleProvisioner
Jan 28 13:10:33 heidata4 asadmin: INFO: Create bundle provisioner class = class com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.
Jan 28 13:10:33 heidata4 asadmin: Jan 28, 2022 1:10:32 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner uninstall
Jan 28 13:10:33 heidata4 asadmin: INFO: Uninstalled bundle 407 installed from /usr/local/payara5/glassfish/modules/rest-client-ssl.jar.
Jan 28 13:10:33 heidata4 asadmin: Jan 28, 2022 1:10:32 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner uninstall
Jan 28 13:10:33 heidata4 asadmin: INFO: Uninstalled bundle 411 installed from /usr/local/payara5/glassfish/modules/security-connector-oauth2-client.jar.
Jan 28 13:10:33 heidata4 asadmin: Jan 28, 2022 1:10:32 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner uninstall
Jan 28 13:10:33 heidata4 asadmin: INFO: Uninstalled bundle 409 installed from /usr/local/payara5/glassfish/modules/org.osgi.util.function.jar.
Jan 28 13:10:33 heidata4 asadmin: Jan 28, 2022 1:10:32 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner uninstall
Jan 28 13:10:33 heidata4 asadmin: INFO: Uninstalled bundle 406 installed from /usr/local/payara5/glassfish/modules/security-connector-oidc-client.jar.
Jan 28 13:10:33 heidata4 asadmin: Jan 28, 2022 1:10:32 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner uninstall
Jan 28 13:10:33 heidata4 asadmin: INFO: Uninstalled bundle 410 installed from /usr/local/payara5/glassfish/modules/security-connectors-api.jar.
Jan 28 13:10:33 heidata4 asadmin: Jan 28, 2022 1:10:32 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner uninstall
Jan 28 13:10:33 heidata4 asadmin: INFO: Uninstalled bundle 408 installed from /usr/local/payara5/glassfish/modules/org.osgi.util.promise.jar.
Jan 28 13:10:33 heidata4 asadmin: org.osgi.framework.BundleException: Unable to resolve fish.payara.server.internal.core.glassfish [218](R 218.0): missing requirement [fish.payara.server.internal.core.glassfish [218](R 218.0)] osgi.wiring.package; (&(osgi.wiring.package=org.osgi.framework)(version>=1.9.0)(!(version>=2.0.0))) Unresolved requirements: [[fish.payara.server.internal.core.glassfish [218](R 218.0)] osgi.wiring.package; (&(osgi.wiring.package=org.osgi.framework)(version>=1.9.0)(!(version>=2.0.0)))]
Jan 28 13:10:33 heidata4 asadmin: at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:4368)
Jan 28 13:10:33 heidata4 asadmin: at org.apache.felix.framework.Felix.startBundle(Felix.java:2281)
Jan 28 13:10:33 heidata4 asadmin: at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1539)
Jan 28 13:10:33 heidata4 asadmin: at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
Jan 28 13:10:33 heidata4 asadmin: at java.base/java.lang.Thread.run(Thread.java:829)
Jan 28 13:10:33 heidata4 asadmin: Exception in thread "main" java.lang.reflect.InvocationTargetException
Jan 28 13:10:33 heidata4 asadmin: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Jan 28 13:10:33 heidata4 asadmin: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Jan 28 13:10:33 heidata4 asadmin: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Jan 28 13:10:33 heidata4 asadmin: at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Jan 28 13:10:33 heidata4 asadmin: at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:119)
Jan 28 13:10:33 heidata4 asadmin: at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:54)
Jan 28 13:10:33 heidata4 asadmin: Caused by: org.glassfish.embeddable.GlassFishException: org.glassfish.embeddable.GlassFishException: No GlassFishRuntime available
Jan 28 13:10:33 heidata4 asadmin: at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishRuntimeBuilder.build(OSGiGlassFishRuntimeBuilder.java:170)
Jan 28 13:10:33 heidata4 asadmin: at org.glassfish.embeddable.GlassFishRuntime._bootstrap(GlassFishRuntime.java:157)
Jan 28 13:10:33 heidata4 asadmin: at org.glassfish.embeddable.GlassFishRuntime.bootstrap(GlassFishRuntime.java:110)
Jan 28 13:10:33 heidata4 asadmin: at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:134)
Jan 28 13:10:33 heidata4 asadmin: ... 6 more
Jan 28 13:10:33 heidata4 asadmin: Caused by: org.glassfish.embeddable.GlassFishException: No GlassFishRuntime available
Jan 28 13:10:33 heidata4 asadmin: at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishRuntimeBuilder.getGlassFishRuntime(OSGiGlassFishRuntimeBuilder.java:208)
Jan 28 13:10:33 heidata4 asadmin: at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishRuntimeBuilder.build(OSGiGlassFishRuntimeBuilder.java:168)
Jan 28 13:10:33 heidata4 asadmin: ... 9 more
Jan 28 13:10:33 heidata4 asadmin: Error stopping framework: java.lang.NullPointerException
Jan 28 13:10:33 heidata4 asadmin: java.lang.NullPointerException
Jan 28 13:10:33 heidata4 asadmin: at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher$1.run(GlassFishMain.java:222)
Jan 28 13:10:33 heidata4 asadmin: Command start-domain failed.
Jan 28 13:10:33 heidata4 systemd: payara.service: control process exited, code=exited status=1
Jan 28 13:10:33 heidata4 systemd: Failed to start GlassFish Server v5.
Jan 28 13:10:33 heidata4 systemd: Unit payara.service entered failed state.
Jan 28 13:10:33 heidata4 systemd: payara.service failed.

At the next attempt with payara 5.2021.10 payara can be started, but afterwards the deploy of dataverse-5.6.war fails.

The directory glassfish/lib is writable for the dataverse/payara user.

/usr/local/payara5/glassfish/bin/asadmin deploy /tmp/dataverse-5.6.war 
remote failure: Error occurred during deployment: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.7.9.payara-p1): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: Access denied to execute this method : setURL
Error Code: 0. Please see server.log for more details.
Command deploy failed.

Same, if I copy postgresql-42.2.2.jar (from our payara 5.2020.x installation) manually into glassfish/lib before.

server.log shows:

...
[2022-01-28T12:54:52.971+0100] [Payara 5.2021.10] [INFO] [] [javax.enterprise.system.tools.deployment.dol] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370892971] [levelValue: 800] [[
  visiting unvisited references]]

[2022-01-28T12:54:53.451+0100] [Payara 5.2021.10] [INFO] [] [javax.enterprise.system.tools.deployment.dol] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370893451] [levelValue: 800] [[
  visiting unvisited references]]

[2022-01-28T12:54:54.787+0100] [Payara 5.2021.10] [WARNING] [] [javax.enterprise.web.util] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370894787] [levelValue: 900] [[
  The web application [unknown] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.]]

[2022-01-28T12:54:54.787+0100] [Payara 5.2021.10] [WARNING] [] [javax.enterprise.web.util] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370894787] [levelValue: 900] [[
  The web application [unknown] registered the JDBC driver [org.apache.solr.client.solrj.io.sql.DriverImpl] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.]]

[2022-01-28T12:54:55.025+0100] [Payara 5.2021.10] [INFO] [] [org.eclipse.persistence.session./file:/usr/local/payara5/glassfish/domains/domain1/applications/dataverse-5.6/WEB-INF/classes/_VDCNet-ejbPU] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370895025] [levelValue: 800] [[
  EclipseLink, version: Eclipse Persistence Services - 2.7.9.payara-p1]]

[2022-01-28T12:54:55.047+0100] [Payara 5.2021.10] [SEVERE] [] [javax.enterprise.resource.resourceadapter.com.sun.gjc.util] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370895047] [levelValue: 1000] [[
  RAR5103 : Error setting java bean value : [dvndbdvndb]]]

[2022-01-28T12:54:55.048+0100] [Payara 5.2021.10] [SEVERE] [] [javax.enterprise.resource.resourceadapter.com.sun.gjc.util] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370895048] [levelValue: 1000] [[
  
java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.sun.gjc.util.MethodExecutor.runMethod(MethodExecutor.java:142)
	at com.sun.gjc.common.DataSourceObjectBuilder.constructDataSourceObject(DataSourceObjectBuilder.java:118)
	at com.sun.gjc.spi.ManagedConnectionFactoryImpl.getDataSource(ManagedConnectionFactoryImpl.java:1384)
	at com.sun.gjc.spi.CPManagedConnectionFactory.getDataSource(CPManagedConnectionFactory.java:91)
	at com.sun.gjc.spi.CPManagedConnectionFactory.createManagedConnection(CPManagedConnectionFactory.java:123)
	at com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator.createResource(LocalTxConnectorAllocator.java:87)
	at com.sun.enterprise.resource.pool.ConnectionPool.createSingleResource(ConnectionPool.java:919)
	at com.sun.enterprise.resource.pool.ConnectionPool.createResource(ConnectionPool.java:1209)
	at com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.addResource(RWLockDataStructure.java:100)
	at com.sun.enterprise.resource.pool.ConnectionPool.addResource(ConnectionPool.java:288)
	at com.sun.enterprise.resource.pool.ConnectionPool.createResourceAndAddToPool(ConnectionPool.java:1532)
	at com.sun.enterprise.resource.pool.ConnectionPool.createResources(ConnectionPool.java:957)
	at com.sun.enterprise.resource.pool.ConnectionPool.initPool(ConnectionPool.java:236)
	at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:527)
	at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:388)
	at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:244)
	at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:171)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:360)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:307)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:196)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:171)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:166)
	at com.sun.gjc.spi.base.AbstractDataSource.getConnection(AbstractDataSource.java:113)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:138)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:172)
	at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:225)
	at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:809)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:261)
	at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:773)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:222)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:330)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:350)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:313)
	at org.glassfish.persistence.jpa.JPADeployer$2.visitPUD(JPADeployer.java:507)
	at org.glassfish.persistence.jpa.JPADeployer$PersistenceUnitDescriptorIterator.iteratePUDs(JPADeployer.java:571)
	at org.glassfish.persistence.jpa.JPADeployer.iterateInitializedPUsAtApplicationPrepare(JPADeployer.java:553)
	at org.glassfish.persistence.jpa.JPADeployer.event(JPADeployer.java:454)
	at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:131)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:564)
	at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:588)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:120)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
	at org.glassfish.admin.rest.resources.admin.CommandResource.executeCommand(CommandResource.java:409)
	at org.glassfish.admin.rest.resources.admin.CommandResource.execCommandSimpInMultOut(CommandResource.java:236)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356)
	at org.glassfish.admin.rest.adapter.JerseyContainerCommandService$3.service(JerseyContainerCommandService.java:179)
	at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:189)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
	at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: URL invalid dvndbdvndb
	at org.postgresql.ds.common.BaseDataSource.setUrl(BaseDataSource.java:1273)
	at org.postgresql.ds.common.BaseDataSource.setURL(BaseDataSource.java:1289)
	... 103 more
]]

[2022-01-28T12:54:55.052+0100] [Payara 5.2021.10] [WARNING] [] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.resource.allocator] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370895052] [levelValue: 900] [[
  RAR5038:Unexpected exception while creating resource for pool __SYSTEM/pools/__datasource_definition/dataverse-5.6/java:app/jdbc/dataverse. Exception : javax.resource.ResourceException: Access denied to execute this method : setURL]]

[2022-01-28T12:54:55.053+0100] [Payara 5.2021.10] [WARNING] [] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370895053] [levelValue: 900] [[
  RAR5117 : Failed to obtain/create connection from connection pool [ __SYSTEM/pools/__datasource_definition/dataverse-5.6/java:app/jdbc/dataverse ]. Reason : com.sun.appserv.connectors.internal.api.PoolingException: Access denied to execute this method : setURL]]

[2022-01-28T12:54:55.054+0100] [Payara 5.2021.10] [WARNING] [] [javax.enterprise.resource.resourceadapter.com.sun.gjc.spi] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370895054] [levelValue: 900] [[
  RAR5114 : Error allocating connection : [Error in allocating a connection. Cause: Access denied to execute this method : setURL]]]

[2022-01-28T12:54:55.057+0100] [Payara 5.2021.10] [SEVERE] [] [org.eclipse.persistence.session./file:/usr/local/payara5/glassfish/domains/domain1/applications/dataverse-5.6/WEB-INF/classes/_VDCNet-ejbPU.ejb] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370895057] [levelValue: 1000] [[
  
Local Exception Stack: 
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.7.9.payara-p1): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: Access denied to execute this method : setURL
Error Code: 0
	at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:318)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:150)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:172)
	at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:225)
	at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:809)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:261)
	at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:773)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:222)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:330)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:350)
	at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:313)
	at org.glassfish.persistence.jpa.JPADeployer$2.visitPUD(JPADeployer.java:507)
	at org.glassfish.persistence.jpa.JPADeployer$PersistenceUnitDescriptorIterator.iteratePUDs(JPADeployer.java:571)
	at org.glassfish.persistence.jpa.JPADeployer.iterateInitializedPUsAtApplicationPrepare(JPADeployer.java:553)
	at org.glassfish.persistence.jpa.JPADeployer.event(JPADeployer.java:454)
	at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:131)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:564)
	at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:588)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:120)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
	at org.glassfish.admin.rest.resources.admin.CommandResource.executeCommand(CommandResource.java:409)
	at org.glassfish.admin.rest.resources.admin.CommandResource.execCommandSimpInMultOut(CommandResource.java:236)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356)
	at org.glassfish.admin.rest.adapter.JerseyContainerCommandService$3.service(JerseyContainerCommandService.java:179)
	at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:189)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
	at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Error in allocating a connection. Cause: Access denied to execute this method : setURL
	at com.sun.gjc.spi.base.AbstractDataSource.getConnection(AbstractDataSource.java:119)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:138)
	... 75 more
Caused by: javax.resource.spi.ResourceAllocationException: Error in allocating a connection. Cause: Access denied to execute this method : setURL
	at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:319)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:196)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:171)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:166)
	at com.sun.gjc.spi.base.AbstractDataSource.getConnection(AbstractDataSource.java:113)
	... 76 more
Caused by: com.sun.appserv.connectors.internal.api.PoolingException: Access denied to execute this method : setURL
	at com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.addResource(RWLockDataStructure.java:108)
	at com.sun.enterprise.resource.pool.ConnectionPool.addResource(ConnectionPool.java:288)
	at com.sun.enterprise.resource.pool.ConnectionPool.createResourceAndAddToPool(ConnectionPool.java:1532)
	at com.sun.enterprise.resource.pool.ConnectionPool.createResources(ConnectionPool.java:957)
	at com.sun.enterprise.resource.pool.ConnectionPool.initPool(ConnectionPool.java:236)
	at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:527)
	at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:388)
	at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:244)
	at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:171)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:360)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:307)
	... 80 more
Caused by: com.sun.appserv.connectors.internal.api.PoolingException: Access denied to execute this method : setURL
	at com.sun.enterprise.resource.pool.ConnectionPool.createSingleResource(ConnectionPool.java:932)
	at com.sun.enterprise.resource.pool.ConnectionPool.createResource(ConnectionPool.java:1209)
	at com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.addResource(RWLockDataStructure.java:100)
	... 90 more
Caused by: com.sun.appserv.connectors.internal.api.PoolingException: Access denied to execute this method : setURL
	at com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator.createResource(LocalTxConnectorAllocator.java:110)
	at com.sun.enterprise.resource.pool.ConnectionPool.createSingleResource(ConnectionPool.java:919)
	... 92 more
Caused by: javax.resource.ResourceException: Access denied to execute this method : setURL
	at com.sun.gjc.util.MethodExecutor.runMethod(MethodExecutor.java:157)
	at com.sun.gjc.common.DataSourceObjectBuilder.constructDataSourceObject(DataSourceObjectBuilder.java:118)
	at com.sun.gjc.spi.ManagedConnectionFactoryImpl.getDataSource(ManagedConnectionFactoryImpl.java:1384)
	at com.sun.gjc.spi.CPManagedConnectionFactory.getDataSource(CPManagedConnectionFactory.java:91)
	at com.sun.gjc.spi.CPManagedConnectionFactory.createManagedConnection(CPManagedConnectionFactory.java:123)
	at com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator.createResource(LocalTxConnectorAllocator.java:87)
	... 93 more
]]

[2022-01-28T12:54:55.059+0100] [Payara 5.2021.10] [SEVERE] [] [javax.enterprise.system.core] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370895059] [levelValue: 1000] [[
  Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.7.9.payara-p1): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: Access denied to execute this method : setURL
Error Code: 0]]

[2022-01-28T12:54:55.060+0100] [Payara 5.2021.10] [WARNING] [] [javax.enterprise.web.util] [tid: _ThreadID=106 _ThreadName=admin-thread-pool::admin-listener(2)] [timeMillis: 1643370895060] [levelValue: 900] [[
  The web application [unknown] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.]]
@qqmyers
Copy link
Member

qqmyers commented Jan 28, 2022

I haven't read line by line above but the line "URL invalid dvndbdvndb" might suggest a typo somewhere?
Regardless, one thing to be aware of for payara-5.2021.8+ is #8064 / #8230 - upgrades can have unreadable entries for the timers which won't allow deployment to succeed. That shouldn't affect 5.2021.5 but could be an error for your 5.2021.10 attempt. The error log entries above aren't what I saw though, so you must have some other issue as well. (FWIW - QDR is running 5.2021.8 and I believe UVA just updated to 5.2021.10 (both DV 5.9) successfully after using the steps in the issues above.)

@lmaylein
Copy link
Contributor Author

I haven't read line by line above but the line "URL invalid dvndbdvndb" might suggest a typo somewhere?

I noticed this issue as well. But the setting in domain.xml is definitely correct and domain.xml was only copied together with the directory domain1.

With our last updates glassfish/lib was never writable for the dataverse/payara user. What is the correct jar file for the ODBC driver?

@lmaylein
Copy link
Contributor Author

lmaylein commented Apr 1, 2022

If I understood #8064 correctly, it would probably help to install payara 5.2021.10 or higher and remove the contents of <domain>/lib/databases before deploying the Dataverse release 5.6. However, it is not entirely clear to me what the effects of this removal are. Where are the database credentials stored? In previous versions they were directly in domain.xml.
In https://guides.dataverse.org/en/latest/installation/installation-main.html I found:

Here is a list of the password aliases that are set by the installation process and entered into Payara’s domain.xml file:
    dataverse.db.password
    doi_password_alias

But I couldn't find such entries in domain.xml.
After all: At least dataverse.db.password seems to exist.

bin/asadmin list-password-aliases
dataverse.db.password
Command list-password-aliases executed successfully.

Would this be a safe update path:

  • undeploy dataverse 5.5
  • stop payara
  • move the current payara directory out of the way
  • unzip payara 5.2021.10 or higher
  • replace domain1 with a copy of the old directory
  • remove domain1/generated, domain1/osgi-cache and contents of domain1/lib/databases
  • start payara
  • deploy dataverse 5.6

Which payara version would you currently recommend?

Do I need a postgresql-x.y.z.jar in glassfish/lib? There is such a file in our running Payara instance, but not in the new Payara releases. I am not quite clear where this file comes from. I can't find anything about it in the Dataverse installation guide.

Many thanks.

@qqmyers
Copy link
Member

qqmyers commented Apr 1, 2022

AFAIK, the recommendation is still 5.2021.5, moving to 5.2021.6 (.5 has a bug if you use asadmin set-log-levels). From my personal testing at various institutions, it looks like .8, .10, 5.2022.1 are all fine.

The note about the lib/databases directory when going to 5.2021.8 or above relates only to the EJBTimers Dataverse uses. As with the osgi-cache and generated dirs, it appears that payara/Dataverse will end up recreating everything needed in those dirs at a restart. (The point of deleting the lib/databases dir is to remove EJBTimer info written in an older, incompatible db format.)

At some point, the postgres driver file started being included in the war and no longer has to be added to glassfish/lib.

In general, the release guide is the place for step-by-step instructions for upgrades with the install guide focusing on new installs. (If you find outdated info (like the doi password being in the clear), please send an issue/PR.)

FWIW: My process for upgrading payara looks like yours (usually have a /usr/local/payara5 symbolic link that I redirect rather than moving payaras around) with the following additions:
Copy/chown the following files:

  • /usr/local/payara5/dataverse_admin_passwd
  • /usr/local/payara5/glassfish/.gfclient

chown to dataverse:

  • /usr/local/payara5/glassfish/lib

On one system, we also have the .aws directory (used for s3 stores) in a place where we have to copy/chown it as well.

@lmaylein
Copy link
Contributor Author

Unfortunately, our next update attempt failed. With Payara 5.2021.6 we have the same problem as with 5.2021.10. Payara can be installed, but afterwards the deployment of Dataverse 5.6 fails. Apparently something has changed in the database access between 5.5 and 5.6.

# /usr/local/payara5/glassfish/bin/asadmin deploy /tmp/dataverse-5.6.war 
remote failure: Error occurred during deployment: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.7.7.payara-p3): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Error in allocating a connection. Cause: Access denied to execute this method : setURL
Error Code: 0. Please see server.log for more details.
Command deploy failed.

The server log shows:


[2022-04-12T13:21:10.721+0200] [Payara 5.2021.6] [INFO] [] [org.eclipse.persistence.session./file:/usr/local/payara5/glassfish/domains/domain1/applications/dataverse-5.6/WEB-INF/classes/_VDCNet-ejbPU] [tid: _ThreadID=116 _ThreadName=admin-thread-pool::admin-listener(3)] [timeMillis: 1649762470721] [levelValue: 800] [[
  EclipseLink, version: Eclipse Persistence Services - 2.7.7.payara-p3]]

[2022-04-12T13:21:10.743+0200] [Payara 5.2021.6] [SEVERE] [] [javax.enterprise.resource.resourceadapter.com.sun.gjc.util] [tid: _ThreadID=116 _ThreadName=admin-thread-pool::admin-listener(3)] [timeMillis: 1649762470743] [levelValue: 1000] [[
  RAR5103 : Error setting java bean value : [dvndbdvndb]]]

[2022-04-12T13:21:10.744+0200] [Payara 5.2021.6] [SEVERE] [] [javax.enterprise.resource.resourceadapter.com.sun.gjc.util] [tid: _ThreadID=116 _ThreadName=admin-thread-pool::admin-listener(3)] [timeMillis: 1649762470744] [levelValue: 1000] [[
  
java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.sun.gjc.util.MethodExecutor.runMethod(MethodExecutor.java:142)
        at com.sun.gjc.common.DataSourceObjectBuilder.constructDataSourceObject(DataSourceObjectBuilder.java:118)
        at com.sun.gjc.spi.ManagedConnectionFactoryImpl.getDataSource(ManagedConnectionFactoryImpl.java:1384)
        at com.sun.gjc.spi.CPManagedConnectionFactory.getDataSource(CPManagedConnectionFactory.java:91)
        at com.sun.gjc.spi.CPManagedConnectionFactory.createManagedConnection(CPManagedConnectionFactory.java:123)
        at com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator.createResource(LocalTxConnectorAllocator.java:87)
        at com.sun.enterprise.resource.pool.ConnectionPool.createSingleResource(ConnectionPool.java:919)
        at com.sun.enterprise.resource.pool.ConnectionPool.createResource(ConnectionPool.java:1209)
        at com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.addResource(RWLockDataStructure.java:98)
        at com.sun.enterprise.resource.pool.ConnectionPool.addResource(ConnectionPool.java:288)
        at com.sun.enterprise.resource.pool.ConnectionPool.createResourceAndAddToPool(ConnectionPool.java:1532)
        at com.sun.enterprise.resource.pool.ConnectionPool.createResources(ConnectionPool.java:957)
        at com.sun.enterprise.resource.pool.ConnectionPool.initPool(ConnectionPool.java:236)
        at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:527)
        at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:388)
        at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:244)
        at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:171)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:360)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:307)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:196)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:171)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:166)
        at com.sun.gjc.spi.base.AbstractDataSource.getConnection(AbstractDataSource.java:113)
        at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:138)
        at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:172)
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:225)
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:809)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:256)
        at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:772)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:222)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:330)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:350)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:313)
        at org.glassfish.persistence.jpa.JPADeployer$2.visitPUD(JPADeployer.java:507)
        at org.glassfish.persistence.jpa.JPADeployer$PersistenceUnitDescriptorIterator.iteratePUDs(JPADeployer.java:571)
        at org.glassfish.persistence.jpa.JPADeployer.iterateInitializedPUsAtApplicationPrepare(JPADeployer.java:553)
        at org.glassfish.persistence.jpa.JPADeployer.event(JPADeployer.java:454)
        at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:131)
        at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:564)
        at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:576)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:120)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
        at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
        at org.glassfish.admin.rest.resources.admin.CommandResource.executeCommand(CommandResource.java:409)
        at org.glassfish.admin.rest.resources.admin.CommandResource.execCommandSimpInMultOut(CommandResource.java:236)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356)
        at org.glassfish.admin.rest.adapter.JerseyContainerCommandService$3.service(JerseyContainerCommandService.java:179)
        at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:189)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: URL invalid dvndbdvndb
        at org.postgresql.ds.common.BaseDataSource.setUrl(BaseDataSource.java:1273)
        at org.postgresql.ds.common.BaseDataSource.setURL(BaseDataSource.java:1289)
        ... 103 more
]]

It looks to us like the URL (setURL) for database access is composed incorrectly.

We have now looked through the old release notes again and found the following in version 5.3:

<payara install path>/bin/asadmin create-system-properties "dataverse.db.user=${DB_USER}"
<payara install path>/bin/asadmin create-system-properties "dataverse.db.host=${DB_HOST}"
<payara install path>/bin/asadmin create-system-properties "dataverse.db.port=${DB_PORT}"
<payara install path>/bin/asadmin create-system-properties "dataverse.db.name=${DB_NAME}"

In our case, it looks like this:


# /usr/local/payara5/glassfish/bin/asadmin list-system-properties
The target server contains following 2 system properties
dataverse.db.user=dvnapp
dataverse.db.name=dvndb
Command list-system-properties executed successfully.

# bin/asadmin list-password-aliases
dataverse.db.password
Command list-password-aliases executed successfully.

Additionally in our domain.xml there are these two lines:

      <system-property name="dataverse.db.user" value="dvnapp"></system-property>
      <system-property name="dataverse.db.name" value="dvndb"></system-property>

Now when we want to run create-system-properties, we get the following message:

# /bin/asadmin create-system-properties "dataverse.db.host=127.0.0.1"
Traceback (most recent call last):
  File "/bin/asadmin", line 260, in <module>
    autoscale = boto.connect_autoscale()
  File "/usr/lib/python2.7/site-packages/boto/__init__.py", line 208, in connect_autoscale
    **kwargs)
  File "/usr/lib/python2.7/site-packages/boto/ec2/autoscale/__init__.py", line 115, in __init__
    profile_name=profile_name)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1100, in __init__
    provider=provider)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 569, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/usr/lib/python2.7/site-packages/boto/auth.py", line 997, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials

What options do we have to set the database parameters correctly?
Can the two lines for database name and database user in domain.xml be deleted?
We are grateful for any information.

@pdurbin
Copy link
Member

pdurbin commented Apr 15, 2022

Caused by: java.lang.IllegalArgumentException: URL invalid dvndbdvndb
at org.postgresql.ds.common.BaseDataSource.setUrl(BaseDataSource.java:1273)

Like @qqmyers pointed out, the "URL invalid" error and strange "dvndbdvndb" value are worrisome.

When I run asadmin list-system-properties on my laptop I get 5 properties instead of the 2 you get, including host, which seems important when setting a URL:

The target server contains following 5 system properties
dataverse.db.user=dvnapp
dataverse.db.host=localhost
dataverse.db.port=5432
dataverse.db.name=dvndb
dataverse.jsf.refresh-period=1
Command list-system-properties executed successfully.

We have now looked through the old release notes again and found the following in version 5.3:

/bin/asadmin create-system-properties "dataverse.db.user=${DB_USER}"
/bin/asadmin create-system-properties "dataverse.db.host=${DB_HOST}"
/bin/asadmin create-system-properties "dataverse.db.port=${DB_PORT}"
/bin/asadmin create-system-properties "dataverse.db.name=${DB_NAME}"

Yes, I think these are important. It's strange to me that they aren't already defined for you. As you say, this was in the 5.3 release notes: https://github.com/IQSS/dataverse/releases/tag/v5.3

If asadmin create-system-properties isn't working for you, that's also strange.

I locally, I just ran asadmin create-system-properties "foo=bar" on my machine and the key/value pair was added to my domain.xml:

<system-property name="foo" value="bar"></system-property>

So maybe you need to edit your domain.xml manually? Here are the db-related properties from my domain.xml:

  <system-property name="dataverse.db.user" value="dvnapp"></system-property>
  <system-property name="dataverse.db.host" value="localhost"></system-property>
  <system-property name="dataverse.db.port" value="5432"></system-property>
  <system-property name="dataverse.db.name" value="dvndb"></system-property>

I hope this helps. Please do feel free to open a support ticket about this by emailing [email protected].

@lmaylein
Copy link
Contributor Author

Oh no. It should be bin/asadmin, not /bin/asadmin. The latter is a completely different tool (Amazon web service).
We will try it again shortly. Sorry.

@pdurbin
Copy link
Member

pdurbin commented Apr 26, 2022

@lmaylein since you closed this, I assume you're all set! Great! 🎉

@lmaylein
Copy link
Contributor Author

@pdurbin Yes, at least on our test system the update has now worked without any problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants