-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/1183/231 azure e2e test #251
Feature/1183/231 azure e2e test #251
Conversation
* Use Datamanagment API to initiate file transfer * Update system test * Remove Api Module * Resolve checkstyle warning * tmp * Revert "tmp" This reverts commit f97672e.
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
Good point, I've updated the README and removed the example. |
...test/java/org/eclipse/dataspaceconnector/system/tests/local/BlobTransferIntegrationTest.java
Outdated
Show resolved
Hide resolved
system-tests/runtimes/azure-storage-transfer-consumer/build.gradle.kts
Outdated
Show resolved
Hide resolved
system-tests/runtimes/azure-storage-transfer-provider/build.gradle.kts
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved with suggestions.
…adle.kts Co-authored-by: Peeyush Chandel <[email protected]>
…adle.kts Co-authored-by: Peeyush Chandel <[email protected]>
…agera-edc/DataSpaceConnector into feature/1183/231-azure-e2e-test
Test was not running in CI. Adding it |
…nnector/system/tests/local/BlobTransferIntegrationTest.java Co-authored-by: Peeyush Chandel <[email protected]>
Cosmos tests are failing do we already have a bug/story in backlog for that ? |
* Feature/1183/231 azure e2e test (#251) * merging changes from feature/1183/231-azure-e2e-test * Create new E2E test for Azure Data Factory * Fix import errors * Fix import errors * Updated test- Fix allow test to run on windows * Add azure resource manager dependencies * Update test * update version * Update test * Update test with vault for runtime * Adding vault support * Move azure data factory system test to separate module * . * testFixtures * rebased * reuse base test class * generic transfer simulation * generic transfer * Moved FileTransferRequestFactory * blob-common * Fixed storage blob code * load test data * Remove file extension * Check output blob * Aligned blob store changes with #235 * Fix UTs * Adding data factory modules * Update data-plane modules * Update to fix ADF extension code after merge * Update DataFactoryLinked Service to use sas token for source * Update test * Use sas token after update * Successful ADF run * Add Storage key in keyvault * Update test to use keys from vault * Update to write .complete blob * Increase test timeout * Update test * Update terraform output to have provider and consumer storage account names * Add keyvayult name in output * User data from runtime properties instead of env * Update environment variables * Add clean up * Resolve conflicts * Update * Fix tests * Update ADF integration test * Update test clean up in system tests * Moving common test utils to a test fixture * checkstyle validations * Update UTs * Fix tests * rebase and update action * Update to fix diff * Add permissions to actions for accessing secrets * update action env * terraform fmt * Add positive test * Improve logging * Include more test * Update Test * Update test to use BlobStoreApiImpl * Update * Removed unused dependencies * Added TransferSimulationConfiguration configurable timeout * refactor common code * move common code into test fixtures * removed publish * Upload artifact Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]>
* Feature/1183/231 azure e2e test (#251) * merging changes from feature/1183/231-azure-e2e-test * Create new E2E test for Azure Data Factory * Fix import errors * Fix import errors * Updated test- Fix allow test to run on windows * Add azure resource manager dependencies * Update test * update version * Update test * Update test with vault for runtime * Adding vault support * Move azure data factory system test to separate module * . * testFixtures * rebased * reuse base test class * generic transfer simulation * generic transfer * Moved FileTransferRequestFactory * blob-common * Fixed storage blob code * load test data * Remove file extension * Check output blob * Aligned blob store changes with #235 * Fix UTs * Adding data factory modules * Update data-plane modules * Update to fix ADF extension code after merge * Update DataFactoryLinked Service to use sas token for source * Update test * Use sas token after update * Successful ADF run * Add Storage key in keyvault * Update test to use keys from vault * Update to write .complete blob * Increase test timeout * Update test * Update terraform output to have provider and consumer storage account names * Add keyvayult name in output * User data from runtime properties instead of env * Update environment variables * Add clean up * Resolve conflicts * Update * Fix tests * Update ADF integration test * Update test clean up in system tests * Moving common test utils to a test fixture * checkstyle validations * Update UTs * Fix tests * rebase and update action * Update to fix diff * Add permissions to actions for accessing secrets * update action env * terraform fmt * Add positive test * Improve logging * Include more test * Update Test * Update test to use BlobStoreApiImpl * Update * Removed unused dependencies * Added TransferSimulationConfiguration configurable timeout * refactor common code * move common code into test fixtures * removed publish * Upload artifact Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]>
* Feature/1183/231 azure e2e test (#251) * merging changes from feature/1183/231-azure-e2e-test * Create new E2E test for Azure Data Factory * Fix import errors * Fix import errors * Updated test- Fix allow test to run on windows * Add azure resource manager dependencies * Update test * update version * Update test * Update test with vault for runtime * Adding vault support * Move azure data factory system test to separate module * . * testFixtures * rebased * reuse base test class * generic transfer simulation * generic transfer * Moved FileTransferRequestFactory * blob-common * Fixed storage blob code * load test data * Remove file extension * Check output blob * Aligned blob store changes with #235 * Fix UTs * Adding data factory modules * Update data-plane modules * Update to fix ADF extension code after merge * Update DataFactoryLinked Service to use sas token for source * Update test * Use sas token after update * Successful ADF run * Add Storage key in keyvault * Update test to use keys from vault * Update to write .complete blob * Increase test timeout * Update test * Update terraform output to have provider and consumer storage account names * Add keyvayult name in output * User data from runtime properties instead of env * Update environment variables * Add clean up * Resolve conflicts * Update * Fix tests * Update ADF integration test * Update test clean up in system tests * Moving common test utils to a test fixture * checkstyle validations * Update UTs * Fix tests * rebase and update action * Update to fix diff * Add permissions to actions for accessing secrets * update action env * terraform fmt * Add positive test * Improve logging * Include more test * Update Test * Update test to use BlobStoreApiImpl * Update * Removed unused dependencies * Added TransferSimulationConfiguration configurable timeout * refactor common code * move common code into test fixtures * removed publish * Upload artifact Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]>
…on records (eclipse-edc#910) * DPF Azure extension: Azure Data factory and cloud testing DPF Azure extension: Azure Data factory and cloud testing DPF Azure extension: Azure Data factory and cloud testing DPF Azure extension: Azure Data factory and cloud testing Move changes to verify.yml Test to verify automatic github env creation Revert "Test to verify automatic github env creation" This reverts commit 88c96bf. Remove unused dependency Fix checkstyle errors Fix checkstyle errors Update to use StatusResult instead of TransferResult Co-Authored-By: Peeyush Chandel <[email protected]> Co-Authored-By: Izabela Kulakowska <[email protected]> Co-Authored-By: Paul Latzelsperger <[email protected]> * Checkstyle * Update name of S3 module. * Update dependencies * Update dependencies * SystemTests: Azure Data Factory end to end system tests (#260) * Feature/1183/231 azure e2e test (#251) * merging changes from feature/1183/231-azure-e2e-test * Create new E2E test for Azure Data Factory * Fix import errors * Fix import errors * Updated test- Fix allow test to run on windows * Add azure resource manager dependencies * Update test * update version * Update test * Update test with vault for runtime * Adding vault support * Move azure data factory system test to separate module * . * testFixtures * rebased * reuse base test class * generic transfer simulation * generic transfer * Moved FileTransferRequestFactory * blob-common * Fixed storage blob code * load test data * Remove file extension * Check output blob * Aligned blob store changes with #235 * Fix UTs * Adding data factory modules * Update data-plane modules * Update to fix ADF extension code after merge * Update DataFactoryLinked Service to use sas token for source * Update test * Use sas token after update * Successful ADF run * Add Storage key in keyvault * Update test to use keys from vault * Update to write .complete blob * Increase test timeout * Update test * Update terraform output to have provider and consumer storage account names * Add keyvayult name in output * User data from runtime properties instead of env * Update environment variables * Add clean up * Resolve conflicts * Update * Fix tests * Update ADF integration test * Update test clean up in system tests * Moving common test utils to a test fixture * checkstyle validations * Update UTs * Fix tests * rebase and update action * Update to fix diff * Add permissions to actions for accessing secrets * update action env * terraform fmt * Add positive test * Improve logging * Include more test * Update Test * Update test to use BlobStoreApiImpl * Update * Removed unused dependencies * Added TransferSimulationConfiguration configurable timeout * refactor common code * move common code into test fixtures * removed publish * Upload artifact Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]> * Resolve conflicts and data factory system tests separate runtime for consumer participant * Fix checkstyle Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Izabela Kulakowska <[email protected]> Co-authored-by: Paul Latzelsperger <[email protected]> Co-authored-by: Marc Gomez <[email protected]> Co-authored-by: Alexandre Gattiker <[email protected]> Co-authored-by: Peeyush Chandel <[email protected]>
What this PR changes/adds
Fixes end-to-end azure storage copy process, and adds a system test to verify it.
The EDC provides the DPF with a storage access signature (rather than storage account key) for the sink. The storage access signature is passed via key vault.
Why it does that
Further notes
Linked Issue(s)
Relates to #231
Checklist
no-changelog
)