Extra configuration for HttpDataSink

[lucian-torje-siemens]

Feature Request

HttpDataSink supports at the moment only HTTP POST and it changes the initial endpoint by appending the part name.

In order to make it compatible with upload AWS S3 presigned URLs it needs extra adjustments, which are part of this feature request.

Which Areas Would Be Affected?

Dataplane HTTP extension

Why Is the Feature Desired?

We need this feature for small files uploads in cloud (< 50 MB).
The flow is:

• create upload presign URL
• use http data source dataplane to get the data
• use http data sink dataplane to push the data to the created presign URL

Solution Proposal

In case we want to use it for uploading data to an upload presign URL it needs to be changed to support HTTP PUT and not to modify the endpoint (unfortunately a single presign URL can be used for the all content).

We propose adding 3 new asset properties:

• httpVerb- possible values:
  - POST (default) - use HTTP POST to send data
  - PUT - use HTTP PUT to send data
• additionalHeaders - make it possible to add extra headers like Azure x-ms-blob-type:BlockBlob
  - added not as String bu as Json key-value object like:
  "dataDestination": { "properties": { "endpoint": "https://qwerty.blob.core.windows.net/test/test.dat?sv=2021-06-08&ss=qwerty&srt=qwerty&sp=qaz&se=2022-06-20T17:19:22Z&st=2022-06-20T09:19:22Z&spr=https&sig=qwerty", "name": "", "type": "HttpData", "additionalHeaders" : {"Content-Type" : "application/octet-stream","x-ms-blob-type": "BlockBlob"} }
  and a new config property:
• edc.dataplane.http.sink.partition.size - used to setup the partition size (5 by default as hardcoded now) - see HttpDataSinkFactory

Limitations

In order to enable multipart cloud data upload (for big files > 50MB) a different solution should be used instead - AWS/Azure extension (which require cloud credentials) or a solution similar to - see AWS doc and Boto example.

see CloudHttpDataSink

When to use

This solution is usable in case of:

• no cloud direct access (wrapped access like when using Integrated Data Lake service offered by Mindsphere)
• data small in size (< 50 MB)

Type of Issue

new feature

Checklist

• assigned appropriate label?
• Do NOT select a milestone or an assignee!