submissions.csv.zip can be downloaded, but submissions.csv cannot

[matthew-white]

On the forum, we've heard of two similar cases in which the submissions.csv.zip can be downloaded, but the submissions.csv cannot. Both cases involved an encrypted project, a form with hundreds of fields, and <1,000 submissions. I'd be interested to see whether we could reproduce it ourselves. In both cases, the users were able to download the submissions.csv if they filtered the submissions down to a smaller number.

• https://forum.getodk.org/t/odk-central-v1-2-primary-data-table-csv-download/34419
• https://forum.getodk.org/t/unable-to-download-submissions-from-two-forms-of-encrypted-project/39655

One part of this that I don't understand is that the users reported that at least in some cases, they saw no error message at all:

there is no error message, only the download never starts after having been requested in the Decrypt and Download window (even when trying several times as suggested).

The first screenshot shows the message after entering the password and clicking "CSV". After waiting 2-3 minutes, we got another error message, as shown in the second screenshot.
Sometimes, the first situation last longer, 5-10 minutes, and nothing happens.

I'm not surprised by an error message after 2–3 minutes: I'm guessing that's an nginx timeout based on the timing and the message. However, I am surprised that there would be cases in which there is no response at all.

[alxndrsn]

@matthew-white both of the forum reports mention encrypted projects/submissions, this may be a factor.

For reference, I haven't had problems downloading an unencrypted submissions.csv for a form with 2500 questions & ~1330 submissions, either locally or from a getodk.cloud server. You can try at https://dev.getodk.cloud/#/projects/29/forms/25000_questions/submissions, although the UI looks slightly different to the screenshots at https://forum.getodk.org/t/odk-central-v1-2-primary-data-table-csv-download/34419.

I have on one occasion seen the "something went wrong" error in the UI as at https://forum.getodk.org/t/unable-to-download-submissions-from-two-forms-of-encrypted-project/39655, although my download started ok.

[matthew-white]

both of the forum reports mention encrypted projects/submissions, this may be a factor.

I strongly suspect so, all the more so given that you weren't able to reproduce with an unencrypted submissions.csv.

the UI looks slightly different to the screenshots at https://forum.getodk.org/t/odk-central-v1-2-primary-data-table-csv-download/34419.

Yes, we replaced the dropdown in that screenshot with the current modal as part of v1.4, when we added more download options. (Before v1.4, you would only see a modal if you had a passphrase to enter; now, you always see a modal.)

I have on one occasion seen the "something went wrong" error in the UI as at https://forum.getodk.org/t/unable-to-download-submissions-from-two-forms-of-encrypted-project/39655, although my download started ok.

Very interesting, that's definitely unexpected. If you see that again, could you see what's logged in the console? That modal has an unusual setup: to start the download, we submit a form from an iframe. We use a form rather than a simple download link in order to pass the passphrase via the request body rather than as a query parameter. We use an iframe so that the form submission doesn't change the main page. Once the form is submitted, we check for a Problem response in the iframe on a regular basis. We display the "something went wrong" error if there's a response in the iframe that isn't JSON (for example, an nginx error). We also log the error to the console.

[matthew-white]

Another data point. @yanokwa encountered this issue while:

downloading encrypted submissions when a form has a few repeats and media. All data tables and All data and attachments work great but Main data table never starts.

You can repro at https://test.getodk.cloud/#/projects/467/forms/my_form_2/submissions.

(The passphrase for the repro is passphrase.)

A couple of things that I'm noticing:

• This form only has 24 fields. From above, the other cases we've heard about had more fields. The repro has >1,000 submissions. The main CSV file is 339 kB.
• The form has media, but both All data tables and All data and attachments work great. So the issue seems to center on the CSV-only case.