Pulls no longer work due to 'Unknown cursor format'

[batkinson]

Software versions

Briefcase v1.16.1
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)
CIMS v5.7 - a custom Aggregate-compatible server

Problem description

After updating to Briefcase 1.16.1 to try and avoid #767, we found that the software would no longer successfully pull forms. This essentially boxed us into having data managers use one version of the software to pull, 1.15.x, and another to export - 1.16.1.

Steps to reproduce the problem

The reproduction steps depend on a non-ODK server, but I believe this is a known issue since a very similar issue was fixed for Ona servers in 5c210e2.

• Attempt to pull forms from the server
• The pull fails with the following:

Start pulling form and submissions
Start downloading form
Form downloaded
Start getting form manifest
Got the form manifest
Start getting submission IDs

Expected behavior

The pull succeed as in previous versions of the software.

Other information

The underlying issue is that the cursors used in the Collect/Aggregate protocol were previously treated as being opaque and uninterpreted, which was explicitly stated in various places. When Briefcase was modified to break this contract and assume a specific structure of the cursor values to implement pull resume any server implementation previously compatible broke. Unfortunately, no fallback implementation was provided so non-Aggregate servers fail to function with the user presented with the rather uninformative message above.

Strangely, when the fix for Ona was added, it was made specific to Ona's format and again did not provide a general fallback.

The relevant stack trace:

12:49:31.646 [ForkJoinPool-4-worker-1] ERROR org.opendatakit.briefcase.reused.job.JobsRunner - Error running Job
org.opendatakit.briefcase.reused.BriefcaseException: Unknown cursor format
	at org.opendatakit.briefcase.pull.aggregate.Cursor.lambda$from$3(Cursor.java:85)
	at java.util.Optional.orElseThrow(Optional.java:290)
	at org.opendatakit.briefcase.pull.aggregate.Cursor.from(Cursor.java:85)
	at java.util.Optional.map(Optional.java:215)
	at org.opendatakit.briefcase.pull.aggregate.InstanceIdBatchGetter.parseBatch(InstanceIdBatchGetter.java:69)
	at org.opendatakit.briefcase.reused.http.response.Success.map(Success.java:75)
	at org.opendatakit.briefcase.pull.aggregate.InstanceIdBatchGetter.fetchNext(InstanceIdBatchGetter.java:60)
	at org.opendatakit.briefcase.pull.aggregate.InstanceIdBatchGetter.<init>(InstanceIdBatchGetter.java:49)
	at org.opendatakit.briefcase.pull.aggregate.PullFromAggregate.getInstanceIdBatches(PullFromAggregate.java:325)
	at org.opendatakit.briefcase.pull.aggregate.PullFromAggregate.getSubmissionIds(PullFromAggregate.java:244)
	at org.opendatakit.briefcase.pull.aggregate.PullFromAggregate.lambda$pull$11(PullFromAggregate.java:109)
	at org.opendatakit.briefcase.reused.job.Job.lambda$thenAccept$8(Job.java:134)
	at org.opendatakit.briefcase.reused.job.JobsRunner.lambda$null$1(JobsRunner.java:65)
	at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

[batkinson]

In case it helps, the way I solved this issue was to add a fallback that simply treats cursors as opaque strings, similar to how the system worked before. You can see the way we fixed 1.16.1 to work here: cims-bioko@91ca88a

[ggalmazor]

Thanks, @batkinson! I'll take your OpaqueCursor and we'll hopefully release a v1.16.2 asap.

[ggalmazor]

Also, I agree that an opaque cursor implementation should have been provided as a fallback in v1.16.

I'm not familiar with the kind of cursors you're using, but let me know if you think the "start pull from date/whatever" feature would be useful for your users. I know that we will eventually want that for Central as well.

[batkinson]

Thanks @ggalmazor. I will look up the format and will share it with you. IIRC, it's a formatted submission date in UTC using a sort-friendly format. I suspect that it would be fairly simple to implement the feature with our scheme, but implementing it without coupling briefcase to implementation details every non-ODK system might be a challenge. It probably would have been better to add or extend the wire protocol rather than to reinterpret parts of it. In that case, other systems could add support by augmenting their systems and not the other way around.

[batkinson]

Yep, I remembered correctly. The cursor format we are using is just the the submitted date of the submission in the format 'yyyy-mm-dd hh:mm:ss.fffffffff'. This format is the one used by java.sql.Timestamp.toString().

[ggalmazor]

Thanks, Brent!

[ggalmazor]

Just wanted to mention that we've just released v1.16.2 with this fix, @batkinson

More info at https://forum.opendatakit.org/t/odk-briefcase-v1-16/20742