Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/1183/231 azure e2e test #251

Merged
merged 94 commits into from
May 2, 2022

Conversation

algattik
Copy link

@algattik algattik commented Apr 29, 2022

What this PR changes/adds

Fixes end-to-end azure storage copy process, and adds a system test to verify it.

The EDC provides the DPF with a storage access signature (rather than storage account key) for the sink. The storage access signature is passed via key vault.

Why it does that

Further notes

Linked Issue(s)

Relates to #231

Checklist

  • added appropriate tests?
  • performed checkstyle check locally?
  • added/updated copyright headers?
  • documented public classes/methods?
  • added/updated relevant documentation?
  • added relevant details to the changelog? (skip with label no-changelog)
  • formatted title correctly? (take a look at the CONTRIBUTING and styleguide for details)

cpeeyush and others added 30 commits April 21, 2022 14:48
* Use Datamanagment API to initiate file transfer

* Update system test

* Remove Api Module

* Resolve checkstyle warning

* tmp

* Revert "tmp"

This reverts commit f97672e.
@algattik
Copy link
Author

@algattik It looks like Readme in data-plane azure extension should be also updated, because it mentions in the example usage sharedKey in the properties:
https://github.com/agera-edc/DataSpaceConnector/tree/main/extensions/azure/data-plane/storage

Good point, I've updated the README and removed the example.

@algattik algattik requested review from Izzzu and cpeeyush and removed request for paullatzelsperger April 30, 2022 08:14
Copy link

@cpeeyush cpeeyush left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with suggestions.

@algattik algattik requested a review from ouphi as a code owner May 2, 2022 09:06
@algattik
Copy link
Author

algattik commented May 2, 2022

Test was not running in CI. Adding it

…nnector/system/tests/local/BlobTransferIntegrationTest.java

Co-authored-by: Peeyush Chandel <[email protected]>
@cpeeyush
Copy link

cpeeyush commented May 2, 2022

Cosmos tests are failing do we already have a bug/story in backlog for that ?
//cc @Izzzu

@algattik algattik removed the request for review from ouphi May 2, 2022 10:34
@algattik algattik merged commit f19d01b into feature/1183-azure-e2e-test May 2, 2022
algattik added a commit that referenced this pull request May 5, 2022
cpeeyush pushed a commit that referenced this pull request May 5, 2022
* Feature/1183/231 system test fixture (#248)

* Feature/1183/231 azure e2e test (#251)

* Removed in-memory deps

* Update CHANGELOG.md

* Update TransferRequestFactory.java
cpeeyush pushed a commit that referenced this pull request May 5, 2022
cpeeyush added a commit that referenced this pull request May 13, 2022
* Feature/1183/231 azure e2e test (#251)

* merging changes from feature/1183/231-azure-e2e-test

* Create new E2E test for Azure Data Factory

* Fix import errors

* Fix import errors

* Updated test- Fix allow test to run on windows

* Add azure resource manager dependencies

* Update test

* update version

* Update test

* Update test with vault for runtime

* Adding vault support

* Move azure data factory system test to separate module

* .

* testFixtures

* rebased

* reuse base test class

* generic transfer simulation

* generic transfer

* Moved FileTransferRequestFactory

* blob-common

* Fixed storage blob code

* load test data

* Remove file extension

* Check output blob

* Aligned blob store changes with #235

* Fix UTs

* Adding data factory modules

* Update data-plane modules

* Update to fix ADF extension code after merge

* Update DataFactoryLinked Service to use sas token for source

* Update test

* Use sas token after update

* Successful ADF run

* Add Storage key in keyvault

* Update test to use keys from vault

* Update to write .complete blob

* Increase test timeout

* Update test

* Update terraform output to have provider and consumer storage account names

* Add keyvayult name in output

* User data from runtime properties instead of env

* Update environment variables

* Add clean up

* Resolve conflicts

* Update

* Fix tests

* Update ADF integration test

* Update test clean up in system tests

* Moving common test utils to a test fixture

* checkstyle validations

* Update UTs

* Fix tests

* rebase and update action

* Update to fix diff

* Add permissions to actions for accessing secrets

* update action env

* terraform fmt

* Add positive test

* Improve logging

* Include more test

* Update Test

* Update test to use BlobStoreApiImpl

* Update

* Removed unused dependencies

* Added TransferSimulationConfiguration configurable timeout

* refactor common code

* move common code into test fixtures

* removed publish

* Upload artifact

Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Marc Gomez <[email protected]>
Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Peeyush Chandel <[email protected]>
cpeeyush added a commit that referenced this pull request May 13, 2022
* Feature/1183/231 azure e2e test (#251)

* merging changes from feature/1183/231-azure-e2e-test

* Create new E2E test for Azure Data Factory

* Fix import errors

* Fix import errors

* Updated test- Fix allow test to run on windows

* Add azure resource manager dependencies

* Update test

* update version

* Update test

* Update test with vault for runtime

* Adding vault support

* Move azure data factory system test to separate module

* .

* testFixtures

* rebased

* reuse base test class

* generic transfer simulation

* generic transfer

* Moved FileTransferRequestFactory

* blob-common

* Fixed storage blob code

* load test data

* Remove file extension

* Check output blob

* Aligned blob store changes with #235

* Fix UTs

* Adding data factory modules

* Update data-plane modules

* Update to fix ADF extension code after merge

* Update DataFactoryLinked Service to use sas token for source

* Update test

* Use sas token after update

* Successful ADF run

* Add Storage key in keyvault

* Update test to use keys from vault

* Update to write .complete blob

* Increase test timeout

* Update test

* Update terraform output to have provider and consumer storage account names

* Add keyvayult name in output

* User data from runtime properties instead of env

* Update environment variables

* Add clean up

* Resolve conflicts

* Update

* Fix tests

* Update ADF integration test

* Update test clean up in system tests

* Moving common test utils to a test fixture

* checkstyle validations

* Update UTs

* Fix tests

* rebase and update action

* Update to fix diff

* Add permissions to actions for accessing secrets

* update action env

* terraform fmt

* Add positive test

* Improve logging

* Include more test

* Update Test

* Update test to use BlobStoreApiImpl

* Update

* Removed unused dependencies

* Added TransferSimulationConfiguration configurable timeout

* refactor common code

* move common code into test fixtures

* removed publish

* Upload artifact

Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Marc Gomez <[email protected]>
Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Peeyush Chandel <[email protected]>
cpeeyush added a commit that referenced this pull request May 18, 2022
* Feature/1183/231 azure e2e test (#251)

* merging changes from feature/1183/231-azure-e2e-test

* Create new E2E test for Azure Data Factory

* Fix import errors

* Fix import errors

* Updated test- Fix allow test to run on windows

* Add azure resource manager dependencies

* Update test

* update version

* Update test

* Update test with vault for runtime

* Adding vault support

* Move azure data factory system test to separate module

* .

* testFixtures

* rebased

* reuse base test class

* generic transfer simulation

* generic transfer

* Moved FileTransferRequestFactory

* blob-common

* Fixed storage blob code

* load test data

* Remove file extension

* Check output blob

* Aligned blob store changes with #235

* Fix UTs

* Adding data factory modules

* Update data-plane modules

* Update to fix ADF extension code after merge

* Update DataFactoryLinked Service to use sas token for source

* Update test

* Use sas token after update

* Successful ADF run

* Add Storage key in keyvault

* Update test to use keys from vault

* Update to write .complete blob

* Increase test timeout

* Update test

* Update terraform output to have provider and consumer storage account names

* Add keyvayult name in output

* User data from runtime properties instead of env

* Update environment variables

* Add clean up

* Resolve conflicts

* Update

* Fix tests

* Update ADF integration test

* Update test clean up in system tests

* Moving common test utils to a test fixture

* checkstyle validations

* Update UTs

* Fix tests

* rebase and update action

* Update to fix diff

* Add permissions to actions for accessing secrets

* update action env

* terraform fmt

* Add positive test

* Improve logging

* Include more test

* Update Test

* Update test to use BlobStoreApiImpl

* Update

* Removed unused dependencies

* Added TransferSimulationConfiguration configurable timeout

* refactor common code

* move common code into test fixtures

* removed publish

* Upload artifact

Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Marc Gomez <[email protected]>
Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Peeyush Chandel <[email protected]>
algattik added a commit that referenced this pull request May 19, 2022
…on records (eclipse-edc#910)

* DPF Azure extension: Azure Data factory and cloud testing

DPF Azure extension: Azure Data factory and cloud testing

DPF Azure extension: Azure Data factory and cloud testing

DPF Azure extension: Azure Data factory and cloud testing

Move changes to verify.yml

Test to verify automatic github env creation

Revert "Test to verify automatic github env creation"

This reverts commit 88c96bf.

Remove unused dependency

Fix checkstyle errors

Fix checkstyle errors

Update to use StatusResult instead of TransferResult

Co-Authored-By: Peeyush Chandel <[email protected]>
Co-Authored-By: Izabela Kulakowska <[email protected]>
Co-Authored-By: Paul Latzelsperger <[email protected]>

* Checkstyle

* Update name of S3 module.

* Update dependencies

* Update dependencies

* SystemTests: Azure Data Factory end to end system tests (#260)

* Feature/1183/231 azure e2e test (#251)

* merging changes from feature/1183/231-azure-e2e-test

* Create new E2E test for Azure Data Factory

* Fix import errors

* Fix import errors

* Updated test- Fix allow test to run on windows

* Add azure resource manager dependencies

* Update test

* update version

* Update test

* Update test with vault for runtime

* Adding vault support

* Move azure data factory system test to separate module

* .

* testFixtures

* rebased

* reuse base test class

* generic transfer simulation

* generic transfer

* Moved FileTransferRequestFactory

* blob-common

* Fixed storage blob code

* load test data

* Remove file extension

* Check output blob

* Aligned blob store changes with #235

* Fix UTs

* Adding data factory modules

* Update data-plane modules

* Update to fix ADF extension code after merge

* Update DataFactoryLinked Service to use sas token for source

* Update test

* Use sas token after update

* Successful ADF run

* Add Storage key in keyvault

* Update test to use keys from vault

* Update to write .complete blob

* Increase test timeout

* Update test

* Update terraform output to have provider and consumer storage account names

* Add keyvayult name in output

* User data from runtime properties instead of env

* Update environment variables

* Add clean up

* Resolve conflicts

* Update

* Fix tests

* Update ADF integration test

* Update test clean up in system tests

* Moving common test utils to a test fixture

* checkstyle validations

* Update UTs

* Fix tests

* rebase and update action

* Update to fix diff

* Add permissions to actions for accessing secrets

* update action env

* terraform fmt

* Add positive test

* Improve logging

* Include more test

* Update Test

* Update test to use BlobStoreApiImpl

* Update

* Removed unused dependencies

* Added TransferSimulationConfiguration configurable timeout

* refactor common code

* move common code into test fixtures

* removed publish

* Upload artifact

Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Marc Gomez <[email protected]>
Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Peeyush Chandel <[email protected]>

* Resolve conflicts and data factory system tests separate runtime for consumer participant

* Fix checkstyle

Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Izabela Kulakowska <[email protected]>
Co-authored-by: Paul Latzelsperger <[email protected]>
Co-authored-by: Marc Gomez <[email protected]>
Co-authored-by: Alexandre Gattiker <[email protected]>
Co-authored-by: Peeyush Chandel <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants