Payara6: failing integration tests

[donsizemore]

Current state of integration tests on Payara 6.2023.2 running @poikilotherm's Payara6 branch:

"[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.564 s <<< FAILURE! - in edu.harvard.iq.dataverse.api.DownloadFilesIT", "[ERROR] edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8  Time elapsed: 1.033 s  <<< FAILURE!", "java.lang.AssertionError: ", "Expected status code <201> doesn't match actual status code <400>.", "", "\tat edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8(DownloadFilesIT.java:426)", "", "[INFO] ", "[INFO] Results:", "[INFO] ", 
"[ERROR] Failures: ", "[ERROR]   DatasetsIT.testArchivalStatusAPI:3021 expected:<200> but was:<403>", 
"[ERROR]   DatasetsIT.testFileChecksum:1310 expected:<201> but was:<400>", 
"[ERROR]   DatasetsIT.testPrivateUrl:903 expected:<201> but was:<400>", 
"[ERROR]   DataversesIT.testMalformedFacetQueryString:305", 
"[ERROR]   DownloadFilesIT.downloadFilenameUtf8:426 Expected status code <201> doesn't match actual status code <400>.", "", 
"[ERROR]   InReviewWorkflowIT.testCuratorSendsCommentsToAuthor:206 XML path error.summary doesn't match.", "Expected: Couldn't update dataset edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException: Dataset cannot be edited due to In Review dataset lock.", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "", 
"[ERROR]   SearchIT.testGeospatialSearchInvalid:1267 Expected status code <400> doesn't match actual status code <500>.", "", 
"[ERROR]   SwordIT.testCreateAndDeleteDatasetInRoot:437 Expected status code <201> doesn't match actual status code <400>.", "",
"[ERROR]   SwordIT.testCreateDataverseCreateDatasetUploadFileDownloadFileEditTitle:250 XML path error.summary doesn't match.", "Expected: user user60a5e34c user60a5e34c is not authorized to modify dataset with global ID doi:10.5072/FK2/9KJJRE", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "", 
"[ERROR]   SwordIT.testDeleteFiles:793 expected:<201> but was:<400>", 
"[ERROR] Errors: ", "[ERROR]   SearchIT.testDatasetThumbnail:422 » IllegalArgument Cannot get property 'dataF..."

[poikilotherm]

I probably need to do the rebase again. Something went sideways during the merge.

[donsizemore]

Oliver created his squeaky clean Payara 6 branch and:

"[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 25.51 s <<< FAILURE! - in edu.harvard.iq.dataverse.api.DownloadFilesIT", 
"[ERROR] edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8  Time elapsed: 0.891 s  <<< FAILURE!", "java.lang.AssertionError: ", "Expected status code <201> doesn't match actual status code <400>.", "", "\tat edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8(DownloadFilesIT.java:426)", "", "[INFO] ", "[INFO] Results:", "[INFO] ", 
"[ERROR] Failures: ", 
"[ERROR]   DatasetsIT.testFileChecksum:1310 expected:<201> but was:<400>", 
"[ERROR]   DatasetsIT.testPrivateUrl:903 expected:<201> but was:<400>", 
"[ERROR]   DownloadFilesIT.downloadFilenameUtf8:426 Expected status code <201> doesn't match actual status code <400>.", "", 
"[ERROR]   InReviewWorkflowIT.testCuratorSendsCommentsToAuthor:206 XML path error.summary doesn't match.", "Expected: Couldn't update dataset edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException: Dataset cannot be edited due to In Review dataset lock.", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "",
"[ERROR]   SwordIT.testCreateAndDeleteDatasetInRoot:437 Expected status code <201> doesn't match actual status code <400>.", "", 
"[ERROR]   SwordIT.testCreateDataverseCreateDatasetUploadFileDownloadFileEditTitle:250 XML path error.summary doesn't match.", "Expected: user user7cd63767 user7cd63767 is not authorized to modify dataset with global ID doi:10.5072/FK2/KCABFV", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "", 
"[ERROR]   SwordIT.testDeleteFiles:793 expected:<201> but was:<400>", 
"[ERROR] Errors: ", "[ERROR]   SearchIT.testDatasetThumbnail:422 » IllegalArgument Cannot get property 'dataF..."

• Big sweep: migrate codebase to Jakarta EE 10 and use Payara 6 now (third iteration) #9685

[pdurbin]

Oliver updated his branch but these last two runs failed before the API tests could be executed:

• https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/45/
• https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/46/

[pdurbin]

@donsizemore and I talked about this. The API test suite is partially executing and this is flagged as a failure. This means the report never gets pushed so we have to try to figure out which API tests failed from the console log.

It's hard to read the errors from the console log.

Here's a quick one liner that got me closer to being able to read them:

curl -s https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/46/consoleText | grep "Results:" | sed 's/", /\n/g' | sed 's/^"//g'

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   DatasetsIT.testFileChecksum:1310 expected:<201> but was:<400>
[ERROR]   DatasetsIT.testPrivateUrl:903 expected:<201> but was:<400>
[ERROR]   DownloadFilesIT.downloadFilenameUtf8:426 Expected status code <201> doesn't match actual status code <400>.

[ERROR]   InReviewWorkflowIT.testCuratorSendsCommentsToAuthor:206 XML path error.summary doesn't match.
Expected: Couldn't update dataset edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException: Dataset cannot be edited due to In Review dataset lock.
  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='

[ERROR]   SwordIT.testCreateAndDeleteDatasetInRoot:437 Expected status code <201> doesn't match actual status code <400>.

[ERROR]   SwordIT.testCreateDataverseCreateDatasetUploadFileDownloadFileEditTitle:250 XML path error.summary doesn't match.
Expected: user user647feb5b user647feb5b is not authorized to modify dataset with global ID doi:10.5072/FK2/3OI2W6
  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='

[ERROR]   SwordIT.testDeleteFiles:793 expected:<201> but was:<400>
[ERROR] Errors: 
[ERROR]   SearchIT.testDatasetThumbnail:422 » IllegalArgument Cannot get property 'dataF...
[INFO] 
[ERROR] Tests run: 197, Failures: 7, Errors: 1, Skipped: 8
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10:38 min
[INFO] Finished at: 2023-07-18T22:17:31Z
[INFO] ------------------------------------------------------------------------

I'm pretty sure the same tests are failing.

[donsizemore]

@pdurbin well, the API test suite is fully executing, but we set any_errors_fatal so Ansible bails at that step, and doesn't complete the surefire reporting stuff. Without any_errors_fatal Ansible would carry on, finish its run successfully, and Jenkins would see a check. But yes, the errors will always show in the console log.

[cmbz]

Sized, labeled, and moved to Sprint Ready as per conversation with @scolapasta

[pdurbin]

We're definitely suffering from problem related to the SWORD API that we saw with the last attempted Payara 6 upgrade. Here's the commit I put in last time: d86e220

I basically flagged it as an incompatible change:

• To upload files, the SWORD API now requires "Content-Disposition: attachment; filename=example.zip" rather than "Content-Disposition: filename=example.zip".

Here are some comments I had added:

// Under Payara 5 we could send "Content-Disposition: filename=example.zip"
// Under Payara 6 now must send "Content-Disposition: attachment; filename=example.zip"
// Otherwise we get "Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='"
// Use req.getHeader("Content-Disposition") to see what the client is sending.

Here's my original writeup of the problem and the solution above: #8305 (comment)

I'll look around for any more new information but we might have to simply accept this backward incompatibility.

[pdurbin]

As I mentioned in Slack, I'm no longer sure what's going on.

• In Jenkins, we see errors that seem to implicate the platform change that breaks SWORD, as mentioned above: Payara6: failing integration tests #9457 (comment)
• On my work laptop using Docker, all SWORD tests are passing.
• On my home laptop using non-Docker, all SWORD tests are passing.
• On GitHub Actions, all SWORD tests are passing but other tests are failing (details below). We're new to running the API tests suite in GitHub Actions so a good follow up test would be to see how the "develop" branch fares in GitHub Actions.

Here are the results from the Payara 6 branch as of 26e66d8. Notice that there are a lot of 500 errors. If these 500 errors are due to Payara falling over, we might see different tests fail on a subsequent run. From https://github.com/pdurbin/dataverse-api-test-runner/actions/runs/5659573350/job/15333705090

[INFO] Results:
[INFO]
Error: Failures:
Error: AuxiliaryFilesIT.testUploadAuxFiles:69 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileExportDdi:2331 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileTermsOfUseAndAccess:2889 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFilesWORequestAccess:2960 Expected status code <409> doesn't match actual status code <400>.

Error: DataversesIT.testMalformedFacetQueryString:289
Error: DeleteUsersIT.testDeleteUserWithFileAccessRequests:517 Expected status code <200> doesn't match actual status code <400>.

Error: DownloadFilesIT.downloadAllFilesRestricted:261 Expected status code <200> doesn't match actual status code <400>.

Error: HarvestingClientsIT.testHarvestingClientRun:234 Last harvest not reported a success (took 0 seconds) expected: but was:
Error: HarvestingServerIT.testMultiRecordOaiSet:595->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: HarvestingServerIT.testSingleRecordOaiSet:443->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: LinkIT.testDeepLinks:169 Expected status code <200> doesn't match actual status code <500>.

Error: MakeDataCountApiIT.testMakeDataCountGetMetric:61 Expected status code <200> doesn't match actual status code <400>.

Error: SearchIT.testGeospatialSearch:1233 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testGeospatialSearchInvalid:1274 Expected status code <400> doesn't match actual status code <500>.

Error: SearchIT.testIdentifier:753 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testNestedSubtree:801 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testSubtreePermissions:987 Expected status code <200> doesn't match actual status code <500>.

Error: Errors:
Error: DataversesIT.testImportDDI:536 » IllegalArgument Cannot get property 'total_co...
Error: DataversesIT.testMoveDataverse:380 » IllegalArgument Cannot get property 'tota...
Error: FilesIT.testAccessFacet:1111 » IllegalArgument Cannot get property 'total_coun...
Error: SearchIT.testAdditionalDatasetContent6300:243 » IllegalArgument Cannot get pro...
Error: SearchIT.testCuratorCardDataversePopulation:917 » IllegalArgument Cannot get p...
Error: SearchIT.testDatasetThumbnail:346 » IllegalArgument Cannot get property 'items...
Error: SearchIT.testSearchCitation:183 » IllegalArgument Cannot get property 'items' ...
Error: SearchIT.testSearchDynamicMetadataFields:292 » IllegalArgument Cannot get prop...
Error: SearchIT.testSearchPermisions:77 » IllegalArgument Cannot get property 'total_...
[INFO]
Error: Tests run: 209, Failures: 17, Errors: 9, Skipped: 8
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:55 min
[INFO] Finished at: 2023-07-25T17:18:34Z
[INFO] ------------------------------------------------------------------------

Update, here another GitHub Actions run, from https://github.com/pdurbin/dataverse-api-test-runner/actions/runs/5660040551/job/15334873565

[INFO] Results:
[INFO]
Error: Failures:
Error: AuxiliaryFilesIT.testUploadAuxFiles:69 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileExportDdi:2331 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileTermsOfUseAndAccess:2889 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFilesWORequestAccess:2960 Expected status code <409> doesn't match actual status code <400>.

Error: DataversesIT.testMalformedFacetQueryString:289
Error: DeleteUsersIT.testDeleteUserWithFileAccessRequests:517 Expected status code <200> doesn't match actual status code <400>.

Error: DownloadFilesIT.downloadAllFilesRestricted:261 Expected status code <200> doesn't match actual status code <400>.

Error: HarvestingClientsIT.testHarvestingClientRun:234 Last harvest not reported a success (took 0 seconds) expected: but was:
Error: HarvestingServerIT.testMultiRecordOaiSet:595->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: HarvestingServerIT.testSingleRecordOaiSet:443->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: LinkIT.testDeepLinks:169 Expected status code <200> doesn't match actual status code <500>.

Error: MakeDataCountApiIT.testMakeDataCountGetMetric:61 Expected status code <200> doesn't match actual status code <400>.

Error: SearchIT.testGeospatialSearch:1233 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testGeospatialSearchInvalid:1274 Expected status code <400> doesn't match actual status code <500>.

Error: SearchIT.testIdentifier:753 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testNestedSubtree:801 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testSubtreePermissions:987 Expected status code <200> doesn't match actual status code <500>.

Error: Errors:
Error: DataversesIT.testImportDDI:536 » IllegalArgument Cannot get property 'total_co...
Error: DataversesIT.testMoveDataverse:380 » IllegalArgument Cannot get property 'tota...
Error: FilesIT.testAccessFacet:1111 » IllegalArgument Cannot get property 'total_coun...
Error: SearchIT.testAdditionalDatasetContent6300:243 » IllegalArgument Cannot get pro...
Error: SearchIT.testCuratorCardDataversePopulation:917 » IllegalArgument Cannot get p...
Error: SearchIT.testDatasetThumbnail:346 » IllegalArgument Cannot get property 'items...
Error: SearchIT.testSearchCitation:183 » IllegalArgument Cannot get property 'items' ...
Error: SearchIT.testSearchDynamicMetadataFields:292 » IllegalArgument Cannot get prop...
Error: SearchIT.testSearchPermisions:77 » IllegalArgument Cannot get property 'total_...
[INFO]
Error: Tests run: 209, Failures: 17, Errors: 9, Skipped: 8
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:49 min
[INFO] Finished at: 2023-07-25T18:04:03Z
[INFO] ------------------------------------------------------------------------

[pdurbin]

For kicks I tried running the entire API test suite against Payara 6 running on my home laptop (no Docker). Surprisingly, only one test failed!

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] HarvestingServerIT.testMultiRecordOaiSet:606 Wrong number of items on the first ListIdentifiers page expected:<2> but was:<5>
[INFO]
[ERROR] Tests run: 209, Failures: 1, Errors: 0, Skipped: 8
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:19 min
[INFO] Finished at: 2023-07-25T16:28:32-04:00
[INFO] ------------------------------------------------------------------------

... and it turns out that test is likely due to some config that we should document. For now, see here:

• Extra configuration for the automated harvest tests to work under Jenkins  gdcc/dataverse-ansible#264

But what does that mean? Is Jenkins wrong? 😬

[donsizemore]

Jenkins may well be wrong. Note that, in contrast, I can run the (passing) integration test suite on RHEL9 instead of RHEL8, and get some handful of test failures. Storage, system libraries, CPU/RAM seem to affect results.

[pdurbin]

• We're new to running the API tests suite in GitHub Actions so a good follow up test would be to see how the "develop" branch fares in GitHub Actions.

I just ran the tests on the "develop" branch (04385e7) and I'm seeing very similar failures from the GitHub Actions environment (see below). From https://github.com/pdurbin/dataverse-api-test-runner/actions/runs/5663434708/job/15345161727

[INFO] Results:
[INFO]
Error: Failures:
Error: AuxiliaryFilesIT.testUploadAuxFiles:69 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileExportDdi:2396 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFileTermsOfUseAndAccess:2954 Expected status code <200> doesn't match actual status code <400>.

Error: DatasetsIT.testRestrictFilesWORequestAccess:3025 Expected status code <409> doesn't match actual status code <400>.

Error: DataversesIT.testMalformedFacetQueryString:291
Error: DeleteUsersIT.testDeleteUserWithFileAccessRequests:517 Expected status code <200> doesn't match actual status code <400>.

Error: DownloadFilesIT.downloadAllFilesRestricted:261 Expected status code <200> doesn't match actual status code <400>.

Error: HarvestingClientsIT.testHarvestingClientRun:234 Last harvest not reported a success (took 0 seconds) expected: but was:
Error: HarvestingServerIT.testMultiRecordOaiSet:595->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: HarvestingServerIT.testSingleRecordOaiSet:443->validateOaiVerbResponse:150 expected:<[ListIdentifiers]> but was:<[[]]>
Error: LinkIT.testDeepLinks:169 Expected status code <200> doesn't match actual status code <500>.

Error: MakeDataCountApiIT.testMakeDataCountGetMetric:61 Expected status code <200> doesn't match actual status code <400>.

Error: SearchIT.testGeospatialSearch:1234 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testGeospatialSearchInvalid:1275 Expected status code <400> doesn't match actual status code <500>.

Error: SearchIT.testIdentifier:754 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testNestedSubtree:802 Expected status code <200> doesn't match actual status code <500>.

Error: SearchIT.testSubtreePermissions:988 Expected status code <200> doesn't match actual status code <500>.

Error: Errors:
Error: DataversesIT.testImportDDI:531 » IllegalArgument Cannot get property 'total_co...
Error: DataversesIT.testMoveDataverse:382 » IllegalArgument Cannot get property 'tota...
Error: FilesIT.testAccessFacet:1113 » IllegalArgument Cannot get property 'total_coun...
Error: SearchIT.testAdditionalDatasetContent6300:244 » IllegalArgument Cannot get pro...
Error: SearchIT.testCuratorCardDataversePopulation:918 » IllegalArgument Cannot get p...
Error: SearchIT.testDatasetThumbnail:347 » IllegalArgument Cannot get property 'items...
Error: SearchIT.testSearchCitation:184 » IllegalArgument Cannot get property 'items' ...
Error: SearchIT.testSearchDynamicMetadataFields:293 » IllegalArgument Cannot get prop...
Error: SearchIT.testSearchPermisions:78 » IllegalArgument Cannot get property 'total_...
[INFO]
Error: Tests run: 210, Failures: 17, Errors: 9, Skipped: 8
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:33 min
[INFO] Finished at: 2023-07-26T01:17:57Z
[INFO] ------------------------------------------------------------------------

[pdurbin]

Lots of activity today.

The bottom line is that for whatever reason Jenkins ALMOST got all the tests to run. In https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/51/ we saw a single failure, which should be fixed by this PR:

• #9700 increase wait time for indexing on failing test #9703

So I merged the latest from develop into #9685, resolved merge conflicts, etc.

But then Payara wouldn't start due to this issue:

• Bug Report: Payara doesn't run on latest JDK patch levels / FISH-7665 payara/Payara#6344

So I also pushed the suggested fix into the Payara 6 branch: da5193f

Will Jenkins pass this time?!? Stay tuned! This is the job to watch: https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/52/

Why was it failing before? That's unknown. 🤔 @donsizemore did spin me up my own Payara 6 server and I got the full API test suite to run there. Again, now we need to make sure Jenkins is happy. If Jenkins ain't happy, ain't nobody happy!

[pdurbin]

We're definitely suffering from problem related to the SWORD API that we saw with the last attempted Payara 6 upgrade. Here's the commit I put in last time: d86e220

I basically flagged it as an incompatible change:

* To upload files, the SWORD API now requires "Content-Disposition: attachment; filename=example.zip" rather than "Content-Disposition: filename=example.zip".

I was wrong about this. The SWORD fix I mentioned above is already in the Payara 6 branch under this commit: 89182c1

I like Oliver's idea in #9685 - "Should we preserve backward compatibility of the SWORD API by inserting attachment; for the user if it isn't supplied?" I'm planning on poking around with this.

[donsizemore]

@pdurbin Promise you won't kill me: in that branch I had bumped the AWS AMI to Rocky 8.8, meaning to merge it to develop. Aside from that difference in minor release (though binary compatibility is a tenet of RHEL), the only other difference is that Ansible didn't automatically fire off the test suite on your server - giving garbage collection a chance to do its thing. If Payara6#52 had trouble (I'll check tomorrow) I can bump the AMI, but... we're really close!

[pdurbin]

Yes! Very close. I think all the API tests passed but the next (hopefully small) hurdle is to get the job to not error out. I opened this issue:

• Support Payara 6 when rsyncing files gdcc/dataverse-ansible#306

[pdurbin]

Good news! The last two builds succeeded with all API tests passing:

I'm especially glad about the second build because I pushed a fix to the Payara 6 branch to keep SWORD working as-is. The details are in 0221006 but I reverted the workaround and put in what I think is a better fix. We can talk about it more as we review that branch/PR.

[pdurbin]

I'm also testing the Payara 6 branch by manually running a job at https://github.com/pdurbin/dataverse-api-test-runner/actions/workflows/payara6.yml

It uses GitHub Actions to test Dataverse running in Docker.

Since both Jenkins and GitHub Actions are showing passing tests, I'm closing this issue.