Feature/1183/231 azure e2e test #251
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Use Datamanagment API to initiate file transfer * Update system test * Remove Api Module * Resolve checkstyle warning * tmp * Revert "tmp" This reverts commit f97672e.
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
Good point, I've updated the README and removed the example. |
algattik
requested review from
Izzzu and
cpeeyush
and removed request for
paullatzelsperger
cpeeyush
reviewed
May 2, 2022
...test/java/org/eclipse/dataspaceconnector/system/tests/local/BlobTransferIntegrationTest.java
Outdated
Show resolved
Hide resolved
cpeeyush
reviewed
May 2, 2022
system-tests/runtimes/azure-storage-transfer-consumer/build.gradle.kts
Outdated
Show resolved
Hide resolved
cpeeyush
reviewed
May 2, 2022
system-tests/runtimes/azure-storage-transfer-provider/build.gradle.kts
Outdated
Show resolved
Hide resolved
…adle.kts Co-authored-by: Peeyush Chandel <[email protected]>
…adle.kts Co-authored-by: Peeyush Chandel <[email protected]>
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
|
Test was not running in CI. Adding it |
Izzzu
approved these changes
May 2, 2022
…nnector/system/tests/local/BlobTransferIntegrationTest.java Co-authored-by: Peeyush Chandel <[email protected]>
|
Cosmos tests are failing do we already have a bug/story in backlog for that ? |
algattik
added a commit
that referenced
this pull request
May 5, 2022
cpeeyush
pushed a commit
that referenced
this pull request
May 5, 2022
cpeeyush
added a commit
that referenced
this pull request
May 13, 2022
* Feature/1183/231 azure e2e test (#251) * merging changes from feature/1183/231-azure-e2e-test * Create new E2E test for Azure Data Factory * Fix import errors * Fix import errors * Updated test- Fix allow test to run on windows * Add azure resource manager dependencies * Update test * update version * Update test * Update test with vault for runtime * Adding vault support * Move azure data factory system test to separate module * . * testFixtures * rebased * reuse base test class * generic transfer simulation * generic transfer * Moved FileTransferRequestFactory * blob-common * Fixed storage blob code * load test data * Remove file extension * Check output blob * Aligned blob store changes with #235 * Fix UTs * Adding data factory modules * Update data-plane modules * Update to fix ADF extension code after merge * Update DataFactoryLinked Service to use sas token for source * Update test * Use sas token after update * Successful ADF run * Add Storage key in keyvault * Update test to use keys from vault * Update to write .complete blob * Increase test timeout * Update test * Update terraform output to have provider and consumer storage account names * Add keyvayult name in output * User data from runtime properties instead of env * Update environment variables * Add clean up * Resolve conflicts * Update * Fix tests * Update ADF integration test * Update test clean up in system tests * Moving common test utils to a test fixture * checkstyle validations * Update UTs * Fix tests * rebase and update action * Update to fix diff * Add permissions to actions for accessing secrets * update action env * terraform fmt * Add positive test * Improve logging * Include more test * Update Test * Update test to use BlobStoreApiImpl * Update * Removed unused dependencies * Added TransferSimulationConfiguration configurable timeout * refactor common code * move common code into test fixtures * removed publish * Upload artifact Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]>
cpeeyush
added a commit
that referenced
this pull request
May 13, 2022
* Feature/1183/231 azure e2e test (#251) * merging changes from feature/1183/231-azure-e2e-test * Create new E2E test for Azure Data Factory * Fix import errors * Fix import errors * Updated test- Fix allow test to run on windows * Add azure resource manager dependencies * Update test * update version * Update test * Update test with vault for runtime * Adding vault support * Move azure data factory system test to separate module * . * testFixtures * rebased * reuse base test class * generic transfer simulation * generic transfer * Moved FileTransferRequestFactory * blob-common * Fixed storage blob code * load test data * Remove file extension * Check output blob * Aligned blob store changes with #235 * Fix UTs * Adding data factory modules * Update data-plane modules * Update to fix ADF extension code after merge * Update DataFactoryLinked Service to use sas token for source * Update test * Use sas token after update * Successful ADF run * Add Storage key in keyvault * Update test to use keys from vault * Update to write .complete blob * Increase test timeout * Update test * Update terraform output to have provider and consumer storage account names * Add keyvayult name in output * User data from runtime properties instead of env * Update environment variables * Add clean up * Resolve conflicts * Update * Fix tests * Update ADF integration test * Update test clean up in system tests * Moving common test utils to a test fixture * checkstyle validations * Update UTs * Fix tests * rebase and update action * Update to fix diff * Add permissions to actions for accessing secrets * update action env * terraform fmt * Add positive test * Improve logging * Include more test * Update Test * Update test to use BlobStoreApiImpl * Update * Removed unused dependencies * Added TransferSimulationConfiguration configurable timeout * refactor common code * move common code into test fixtures * removed publish * Upload artifact Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]>
cpeeyush
added a commit
that referenced
this pull request
May 18, 2022
* Feature/1183/231 azure e2e test (#251) * merging changes from feature/1183/231-azure-e2e-test * Create new E2E test for Azure Data Factory * Fix import errors * Fix import errors * Updated test- Fix allow test to run on windows * Add azure resource manager dependencies * Update test * update version * Update test * Update test with vault for runtime * Adding vault support * Move azure data factory system test to separate module * . * testFixtures * rebased * reuse base test class * generic transfer simulation * generic transfer * Moved FileTransferRequestFactory * blob-common * Fixed storage blob code * load test data * Remove file extension * Check output blob * Aligned blob store changes with #235 * Fix UTs * Adding data factory modules * Update data-plane modules * Update to fix ADF extension code after merge * Update DataFactoryLinked Service to use sas token for source * Update test * Use sas token after update * Successful ADF run * Add Storage key in keyvault * Update test to use keys from vault * Update to write .complete blob * Increase test timeout * Update test * Update terraform output to have provider and consumer storage account names * Add keyvayult name in output * User data from runtime properties instead of env * Update environment variables * Add clean up * Resolve conflicts * Update * Fix tests * Update ADF integration test * Update test clean up in system tests * Moving common test utils to a test fixture * checkstyle validations * Update UTs * Fix tests * rebase and update action * Update to fix diff * Add permissions to actions for accessing secrets * update action env * terraform fmt * Add positive test * Improve logging * Include more test * Update Test * Update test to use BlobStoreApiImpl * Update * Removed unused dependencies * Added TransferSimulationConfiguration configurable timeout * refactor common code * move common code into test fixtures * removed publish * Upload artifact Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]>
algattik
added a commit
that referenced
this pull request
May 19, 2022
…on records (eclipse-edc#910) * DPF Azure extension: Azure Data factory and cloud testing DPF Azure extension: Azure Data factory and cloud testing DPF Azure extension: Azure Data factory and cloud testing DPF Azure extension: Azure Data factory and cloud testing Move changes to verify.yml Test to verify automatic github env creation Revert "Test to verify automatic github env creation" This reverts commit 88c96bf. Remove unused dependency Fix checkstyle errors Fix checkstyle errors Update to use StatusResult instead of TransferResult Co-Authored-By: Peeyush Chandel <[email protected]> Co-Authored-By: Izabela Kulakowska <[email protected]> Co-Authored-By: Paul Latzelsperger <[email protected]> * Checkstyle * Update name of S3 module. * Update dependencies * Update dependencies * SystemTests: Azure Data Factory end to end system tests (#260) * Feature/1183/231 azure e2e test (#251) * merging changes from feature/1183/231-azure-e2e-test * Create new E2E test for Azure Data Factory * Fix import errors * Fix import errors * Updated test- Fix allow test to run on windows * Add azure resource manager dependencies * Update test * update version * Update test * Update test with vault for runtime * Adding vault support * Move azure data factory system test to separate module * . * testFixtures * rebased * reuse base test class * generic transfer simulation * generic transfer * Moved FileTransferRequestFactory * blob-common * Fixed storage blob code * load test data * Remove file extension * Check output blob * Aligned blob store changes with #235 * Fix UTs * Adding data factory modules * Update data-plane modules * Update to fix ADF extension code after merge * Update DataFactoryLinked Service to use sas token for source * Update test * Use sas token after update * Successful ADF run * Add Storage key in keyvault * Update test to use keys from vault * Update to write .complete blob * Increase test timeout * Update test * Update terraform output to have provider and consumer storage account names * Add keyvayult name in output * User data from runtime properties instead of env * Update environment variables * Add clean up * Resolve conflicts * Update * Fix tests * Update ADF integration test * Update test clean up in system tests * Moving common test utils to a test fixture * checkstyle validations * Update UTs * Fix tests * rebase and update action * Update to fix diff * Add permissions to actions for accessing secrets * update action env * terraform fmt * Add positive test * Improve logging * Include more test * Update Test * Update test to use BlobStoreApiImpl * Update * Removed unused dependencies * Added TransferSimulationConfiguration configurable timeout * refactor common code * move common code into test fixtures * removed publish * Upload artifact Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]> * Resolve conflicts and data factory system tests separate runtime for consumer participant * Fix checkstyle Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Izabela Kulakowska <[email protected]> Co-authored-by: Paul Latzelsperger <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR changes/adds
Fixes end-to-end azure storage copy process, and adds a system test to verify it.
The EDC provides the DPF with a storage access signature (rather than storage account key) for the sink. The storage access signature is passed via key vault.
Why it does that
Further notes
Linked Issue(s)
Relates to #231
Checklist
no-changelog)