Docs: Blob transfer Architectural Decision Record #233

ouphi · 2022-04-21T14:24:52Z

Added ADR explaining how Blob transfer should work.
Added sequence diagram to explain the data transfer flow.

The goal of this document is to explain what should be implemented. This document does not represents what is already implemented.
For example, right now DPF uses the storage account key (SHARED_KEY), but the SAS token is not used yet (not written and read from Vault).

github-actions · 2022-04-21T14:38:03Z

Unit Test Results

    386 files -     6     386 suites - 6 8m 42s ⏱️ - 9m 43s
12 065 tests - 130 12 046 ✔️ - 131 18 💤 ±0 1 ❌ +1
12 126 runs - 130 12 107 ✔️ - 131 18 💤 ±0 1 ❌ +1

For more details on these failures, see this check.

Results for commit 6f41446. ± Comparison against base commit 6065693.

♻️ This comment has been updated with latest results.

docs/developer/decision-records/2022-04-21-dpf-blob-data-transfer/README.md

…fer/README.md Co-authored-by: Izabela Kulakowska <[email protected]>

docs/developer/decision-records/2022-04-21-dpf-blob-data-transfer/README.md

…aSpaceConnector into docs/231-blob-transfer-adr

codecov-commenter · 2022-04-26T08:22:17Z

Codecov Report

Merging #233 (6f41446) into docs/1183-blob-transfer-adr (6065693) will not change coverage.
The diff coverage is n/a.

@@                      Coverage Diff                       @@
##             docs/1183-blob-transfer-adr     #233   +/-   ##
==============================================================
  Coverage                          58.61%   58.61%           
  Complexity                          2731     2731           
==============================================================
  Files                                698      698           
  Lines                              15444    15444           
  Branches                            1045     1045           
==============================================================
  Hits                                9053     9053           
  Misses                              5967     5967           
  Partials                             424      424

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6065693...6f41446. Read the comment docs.

…aSpaceConnector into docs/231-blob-transfer-adr

…key.

chrislomonico

resolved open comment

chrislomonico · 2022-05-04T06:27:00Z

@algattik @ouphi what is remaining in approving this PR and merging?

algattik · 2022-05-04T09:30:46Z

12 and 13 (now 13 and 14) should be separate loops, these are not synchronised

algattik · 2022-05-04T09:31:43Z

in (16) "deletes the container containing the blob and the SAS token" the token is not in the container. We should add an arrow (17) whereby the consumer deletes the Sas token in its vault.
"The ObjectStorageProvisioner is responsible for deprovisioning the container" and who is responsible for deprovisioning the SAS token?

algattik · 2022-05-04T09:34:04Z

nit: it would make the puml clearer to have service activation boxes across 2/3/4/13 for Consumer, across 5/6 for Provider and across 6/7/8/9/10/11 for DPF and across 14/15 for Client, across 15/16/(new)17 for Consumer

algattik · 2022-05-04T09:39:09Z

Pushed some proofreading fixes

…Azure blob storage.

* Docs: Blob transfer Architectural Decision Record (#233) Added Azure blob transfer Architectural Decision Record. * Updated changelog. * Fixed PR number in changelog. * Applied suggestion. Co-authored-by: Paul Latzelsperger <[email protected]> * Applied suggestions. Co-authored-by: Paul Latzelsperger <[email protected]> * Update docs/developer/decision-records/2022-04-21-dpf-blob-data-transfer/README.md Co-authored-by: Paul Latzelsperger <[email protected]> * Update docs/developer/decision-records/2022-04-21-dpf-blob-data-transfer/README.md Co-authored-by: Paul Latzelsperger <[email protected]> * Update docs/developer/decision-records/2022-04-21-dpf-blob-data-transfer/README.md Co-authored-by: Paul Latzelsperger <[email protected]> * Moved client poll. * Update docs/developer/decision-records/2022-04-21-dpf-blob-data-transfer/README.md Co-authored-by: Paul Latzelsperger <[email protected]> * Made 17 and 18 substep of 16. Co-authored-by: Paul Latzelsperger <[email protected]>

ouphi added 3 commits April 21, 2022 15:42

Added puml describing the blob transfer.

5738374

Added sequence diagram explanations.

64351eb

Fixed syntax.

a927898

ouphi changed the title ~~Docs/231 blob transfer adr~~ Blob transfer ADR Apr 21, 2022

ouphi added 2 commits April 21, 2022 16:48

Added description.

91a39f9

Added ObjectStorageProvisioner role.

44b4e9f

ouphi changed the title ~~Blob transfer ADR~~ Blob transfer Architectural Decision Record Apr 21, 2022

ouphi added 5 commits April 22, 2022 10:42

Added info about data stream.

58917d7

Explained how the storage account access key is retrieved.

108ee97

Improved example description.

9b2619c

Added information about checking transfer status.

4766eb1

Added classes location.

9514daf

ouphi marked this pull request as ready for review April 26, 2022 07:43

ouphi added 3 commits April 26, 2022 09:43

Small improvement.

ef2ccba

Add changelog.

2fad0b0

Merged main.

3c69b16

ouphi requested a review from paullatzelsperger as a code owner April 26, 2022 07:46

ouphi removed the request for review from paullatzelsperger April 26, 2022 07:50

ouphi changed the base branch from main to docs/1183-blob-transfer-adr April 26, 2022 07:51

Izzzu reviewed Apr 26, 2022

View reviewed changes

docs/developer/decision-records/2022-04-21-dpf-blob-data-transfer/README.md Outdated Show resolved Hide resolved

ouphi changed the title ~~Blob transfer Architectural Decision Record~~ Docs: Blob transfer Architectural Decision Record Apr 26, 2022

Update docs/developer/decision-records/2022-04-21-dpf-blob-data-trans…

be817a6

…fer/README.md Co-authored-by: Izabela Kulakowska <[email protected]>

zeier mentioned this pull request Apr 26, 2022

Data transfer Across Azure Storage Accounts #244

Closed

6 tasks

zeier linked an issue Apr 26, 2022 that may be closed by this pull request

End-to-end blob storage transfer is not working #231

Closed

7 tasks

Izzzu reviewed Apr 26, 2022

View reviewed changes

docs/developer/decision-records/2022-04-21-dpf-blob-data-transfer/README.md Outdated Show resolved Hide resolved

ouphi added 2 commits April 26, 2022 10:21

Added .complete to sequence diagram.

613e067

Merge branch 'docs/231-blob-transfer-adr' of github.com:agera-edc/Dat…

27cf3dc

…aSpaceConnector into docs/231-blob-transfer-adr

zeier requested a review from algattik April 26, 2022 08:44

ouphi added 13 commits May 3, 2022 12:09

Added when consumer get the process_id.

b2f0f11

Merge branch 'docs/231-blob-transfer-adr' of github.com:agera-edc/Dat…

b643cd3

…aSpaceConnector into docs/231-blob-transfer-adr

Linked issues.

909fedd

Added information about managedResources=false.

298da77

Added info in step 1.

b7c3c49

Limit to HTTPS.

957e906

Specified that SAS token cannot be created without a storage account …

122a1b1

…key.

Added contract negotiation information.

c1bbb22

Added deletion of sas token.

b5bb70b

Specified assetId and contractId.

8952b59

Added asset id information.

43010e0

Added explanation about keystore.

9ffb909

Fixed typos.

96419a2

ouphi requested a review from algattik May 3, 2022 14:03

chrislomonico approved these changes May 4, 2022

View reviewed changes

Proofreading

6c5003c

ouphi added 2 commits May 4, 2022 12:13

Added storage and deletion of sas token in the puml.

50e6ab5

Added activate boxes.

6f41446

algattik approved these changes May 4, 2022

View reviewed changes

ouphi added 3 commits May 4, 2022 16:46

Fixed responsible class for deleting sas token.

9f8e4fd

Fixed loops.

3d5d407

Specified that the document is valid if the 2 participants are using …

58fc42e

…Azure blob storage.

ouphi merged commit 7ce17a2 into docs/1183-blob-transfer-adr May 4, 2022

ouphi deleted the docs/231-blob-transfer-adr branch May 4, 2022 14:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs: Blob transfer Architectural Decision Record #233

Docs: Blob transfer Architectural Decision Record #233

ouphi commented Apr 21, 2022 •

edited

Loading

github-actions bot commented Apr 21, 2022 •

edited

Loading

codecov-commenter commented Apr 26, 2022 •

edited

Loading

chrislomonico left a comment

chrislomonico commented May 4, 2022

algattik commented May 4, 2022 •

edited

Loading

algattik commented May 4, 2022 •

edited

Loading

algattik commented May 4, 2022

algattik commented May 4, 2022

Docs: Blob transfer Architectural Decision Record #233

Docs: Blob transfer Architectural Decision Record #233

Conversation

ouphi commented Apr 21, 2022 • edited Loading

github-actions bot commented Apr 21, 2022 • edited Loading

Unit Test Results

codecov-commenter commented Apr 26, 2022 • edited Loading

Codecov Report

chrislomonico left a comment

Choose a reason for hiding this comment

chrislomonico commented May 4, 2022

algattik commented May 4, 2022 • edited Loading

algattik commented May 4, 2022 • edited Loading

algattik commented May 4, 2022

algattik commented May 4, 2022

ouphi commented Apr 21, 2022 •

edited

Loading

github-actions bot commented Apr 21, 2022 •

edited

Loading

codecov-commenter commented Apr 26, 2022 •

edited

Loading

algattik commented May 4, 2022 •

edited

Loading

algattik commented May 4, 2022 •

edited

Loading