POC: Direct upload to S3 (Uppy + Minio)

[phwissmann]

An alternative to having a tus backend, Uppy allows writing directly to an S3 backend (Minio) https://uppy.io/docs/guides/choosing-uploader/#i-want-to-upload-to-aws-s3-or-s3-compatible-storage-directly
The AWS API seems to mandate md5 checksum verification, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html. This is also the case for Minio, see https://min.io/docs/minio/linux/developers/dotnet/API.html#putobjectasync-putobjectargs-args
This would simplify the setup and additionally ensure the data integrity. Questions regarding performance remain.
Goal:

Create a POC with Uppy, Companion and Minio and measure performance. Compare with tusd upload.

[phwissmann]

Stability issues have been resolved

Source -> Target	Rate
VM -> NAS	3.5 Gb/s
Transfer Computer -> VM (S3)	1.5 Gb/s
Transfer Computer -> VM (tusd)	?
Transfer Computer -> NAS	3.5 Gb/s

[phwissmann]

First Observations:

• Upload from Transfer Computer to Minio with storage local to the VM:
  • peak of 1.4 Gb/s reached, but unstable.
  • Network interface crashes from time to time
  • Limited by the VMs network speed (not in the 10Gb/s network)
  • Enabling checksum reduces the speed potentially
• Upload from Transfer Computer to Minio with NAS mounted storage: 800 Mb/s
  • Unclear what limits the data transfer
    • Minio shouldn't run on a NAS https://blog.min.io/no-san-nas/
  • rsync from VM local storage to NAS mounted storage at 3.5 Gb/s
• Copying via Windows explorer reaches rates of 3.5Gb/s, i.e 440 MB/s

[phwissmann]

Currently blocked until further tests with 10Gbit connection can be done