“HTTP Error 431 Request Header Fields Too Large” on large file uploads

[mariusschroeterCP]

Describe the bug
The larger a file, the more upload resources are requested by the server for the upload via POST with the TUS protocol. As a result, the size of the Upload-Concat header grows proportionally to the file size in the final request and can therefore exceed the maximum permitted header limit of the server or proxy, which triggers the “HTTP 431 Request Header Fields Too Large” error.

You can increase the chunk size but this only postpones the problem.

To Reproduce
Define the Uploader with a small chunk size (like 8mb) and upload a large file (probably bigger than 8gb). Check the last Post Request Upload-Concat Header.

Expected behavior
The implementation of the TUS protocol in GitHub - tus/tus-js-client: A pure JavaScript client for the tus resumable upload protocol takes a different approach. Here, only as many upload resources are requested as parallel, simultaneous uploads are to be carried out. PATCH requests are used to repeatedly append chunks to the same uploadUrl, which is made possible by setting the upload offset. With Uploady, however, Upload-Offset is always 0, and the server requests a new upload resource for the next chunk. As a result, the size of the Upload-Concat header in the final request no longer depends on the file size, which in our view is recommendable and conforms to the TUS specification. It should also have a positive effect on the number of possible preflight requests.

Versions
"@rpldy/tus-uploady": "^1.8.1"

[yoavniran]

thanks for the detailed issue description @mariusschroeterCP .
I will try to investigate and provide a fix soon

[github-actions]

It's been a while. Waiting for an update... 🧐

[yoavniran]

Glad to update that I have the fix/improvement in the code finally.
Required a fairly massive change to the tus-sender, but now the code is simpler and better.

Now need to update & add tests and clean up.

Hope that i won't take too long but difficult with very limited time...

[yoavniran]

Hey @mariusschroeterCP
Finally got around to polish off this one.

Published v1.9.0-rc.2 with the fix.

If you can, give it a try and let me know if it solves the reported issue and that everything else is working before I publish 1.9.0 officially in a few days.

[pulmer-cp]

Many thanks for the quick realisation @yoavniran . Works great.