Samples: add automated test to sample 04.0-file-transfer

docs/developer/decision-records/2022-04-21-dpf-blob-data-transfer/README.md

@@ -27,7 +27,7 @@ This sequence diagram describes the flow if the 2 participants are using Azure b
 
 The sequence starts from the client triggering the transfer on the consumer side and finishes when the consumer deprovisions its resources.
 
-![blob-transfer](architecture/data-transfer/diagrams/blob-transfer.png)
+![blob-transfer](../../../architecture/data-transfer/diagrams/blob-transfer.png)
 
 1. The client calls the data management API to trigger a transfer process. The requested asset is identified by the `assetId` and the `contractId` from previous contract negotiation. The client get the `PROCESS_ID` corresponding to the `transferProcess`. This `PROCESS_ID` will be used to get the transfer status. For now, `managedResources` needs to be set to true, to make sure that the consumer provisions the blob container. `managedResources=false` would be used if the client wants to use a pre-existing container without creating a new one, but this feature is not supported yet.  
 2. Consumer gets the destination storage account access key in its Vault.  

extensions/filesystem/configuration-fs/src/main/java/org/eclipse/dataspaceconnector/configuration/fs/FsConfigurationExtension.java

@@ -37,7 +37,7 @@
 public class FsConfigurationExtension implements ConfigurationExtension {
 
     @EdcSetting
-    private static final String CONFIG_LOCATION = propOrEnv("edc.fs.config", "dataspaceconnector-configuration.properties");
+    private static final String FS_CONFIG = "edc.fs.config";
 
     private Config config;
     private Path configFile;
@@ -62,9 +62,11 @@ public String name() {
 
     @Override
     public void initialize(Monitor monitor) {
-        var configPath = configFile != null ? configFile : Paths.get(FsConfigurationExtension.CONFIG_LOCATION);
+        var configLocation = propOrEnv(FS_CONFIG, "dataspaceconnector-configuration.properties");
+        var configPath = configFile != null ? configFile : Paths.get(configLocation);
+
         if (!Files.exists(configPath)) {
-            monitor.info(format("Configuration file does not exist: %s. Ignoring.", FsConfigurationExtension.CONFIG_LOCATION));
+            monitor.info(format("Configuration file does not exist: %s. Ignoring.", configLocation));
             return;
         }
 

extensions/junit/src/main/java/org/eclipse/dataspaceconnector/junit/extensions/EdcRuntimeExtension.java

@@ -44,7 +44,7 @@
  * is used to determine the runtime classpath of the module to run. The runtime obtains a classpath determined by the
  * Gradle build.
  * <p>
- * This extension attaches a EDC runtime to the {@link BeforeTestExecutionCallback} and
+ * This extension attaches an EDC runtime to the {@link BeforeTestExecutionCallback} and
  * {@link AfterTestExecutionCallback} lifecycle hooks. Parameter injection of runtime services is supported.
  */
 public class EdcRuntimeExtension extends EdcExtension {

extensions/junit/src/main/java/org/eclipse/dataspaceconnector/junit/testfixtures/TestUtils.java

@@ -33,6 +33,7 @@
 public class TestUtils {
     public static final int MAX_TCP_PORT = 65_535;
     public static final String GRADLE_WRAPPER;
+    private static File buildRoot = null;
     private static final String GRADLE_WRAPPER_UNIX = "gradlew";
     private static final String GRADLE_WRAPPER_WINDOWS = "gradlew.bat";
 
@@ -145,21 +146,25 @@ public static OkHttpClient testOkHttpClient() {
 
     /**
      * Utility method to locate the Gradle project root.
+     * Search for build root will be done only once and cached for subsequent calls.
      *
      * @return The Gradle project root directory.
      */
     public static File findBuildRoot() {
+        // Use cached value if already existing.
+        if (buildRoot != null) return buildRoot;
+
         File canonicalFile;
         try {
             canonicalFile = new File(".").getCanonicalFile();
         } catch (IOException e) {
             throw new IllegalStateException("Could not resolve current directory.", e);
         }
-        var root = findBuildRoot(canonicalFile);
-        if (root == null) {
+        buildRoot = findBuildRoot(canonicalFile);
+        if (buildRoot == null) {
             throw new IllegalStateException("Could not find " + GRADLE_WRAPPER + " in parent directories.");
         }
-        return root;
+        return buildRoot;
     }
 
     private static File findBuildRoot(File path) {

samples/04.0-file-transfer/README.md

@@ -194,6 +194,12 @@ curl -X POST -H "Content-Type: application/json" -H "X-Api-Key: password" -d @sa
 
 In the response we'll get a UUID that we can use to get the contract agreement negotiated between provider and consumer.
 
+Sample output:
+
+```json
+{"id":"5a6b7e22-dc7d-4135-bc98-4cc5fd1dd1ed"}
+```
+
 ### 3. Look up the contract agreement ID
 
 After calling the endpoint for initiating a contract negotiation, we get a UUID as the response. This UUID is the ID of
@@ -249,6 +255,12 @@ curl -X POST -H "Content-Type: application/json" -H "X-Api-Key: password" -d @sa
 Again, we will get a UUID in the response. This time, this is the ID of the `TransferProcess` created on the consumer
 side, because like the contract negotiation, the data transfer is handled in a state machine and performed asynchronously.
 
+Sample output:
+
+```json
+{"id":"deeed974-8a43-4fd5-93ad-e1b8c26bfa44"}
+```
+
 Since transferring a file does not require any resource provisioning on either side, the transfer will be very quick and
 most likely already done by the time you read the UUID.
 

samples/04.0-file-transfer/filetransfer.json

@@ -1,4 +1,5 @@
 {
+  "edctype": "dataspaceconnector:datarequest",
   "protocol": "ids-multipart",
   "assetId": "test-document",
   "contractId": "{agreement ID}",

samples/04.0-file-transfer/integration-tests/build.gradle.kts

@@ -0,0 +1,31 @@
+/*
+ *  Copyright (c) 2022 Microsoft Corporation
+ *
+ *  This program and the accompanying materials are made available under the
+ *  terms of the Apache License, Version 2.0 which is available at
+ *  https://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  SPDX-License-Identifier: Apache-2.0
+ *
+ *  Contributors:
+ *       Microsoft Corporation - initial test implementation for sample
+ *
+ */
+
+plugins {
+    `java-library`
+}
+
+val restAssured: String by project
+val awaitility: String by project
+
+
+dependencies {
+    testImplementation(project(":extensions:junit"))
+    testImplementation(testFixtures(project(":common:util")))
+    testImplementation("io.rest-assured:rest-assured:${restAssured}")
+    testImplementation("org.awaitility:awaitility:${awaitility}")
+
+    testCompileOnly(project(":samples:04.0-file-transfer:consumer"))
+    testCompileOnly(project(":samples:04.0-file-transfer:provider"))
+}

samples/04.0-file-transfer/integration-tests/src/test/java/org/eclipse/dataspaceconnector/extension/sample/test/FileTransferSampleTest.java

@@ -0,0 +1,239 @@
+/*
+ *  Copyright (c) 2022 Microsoft Corporation
+ *
+ *  This program and the accompanying materials are made available under the
+ *  terms of the Apache License, Version 2.0 which is available at
+ *  https://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  SPDX-License-Identifier: Apache-2.0
+ *
+ *  Contributors:
+ *       Microsoft Corporation - initial test implementation for sample
+ *
+ */
+
+package org.eclipse.dataspaceconnector.extension.sample.test;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import io.restassured.RestAssured;
+import io.restassured.http.ContentType;
+import io.restassured.path.json.JsonPath;
+import org.apache.http.HttpStatus;
+import org.eclipse.dataspaceconnector.common.util.junit.annotations.EndToEndTest;
+import org.eclipse.dataspaceconnector.junit.extensions.EdcRuntimeExtension;
+import org.eclipse.dataspaceconnector.junit.testfixtures.TestUtils;
+import org.eclipse.dataspaceconnector.spi.types.domain.DataAddress;
+import org.eclipse.dataspaceconnector.spi.types.domain.transfer.DataRequest;
+import org.jetbrains.annotations.NotNull;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.awaitility.Awaitility.await;
+import static org.hamcrest.Matchers.emptyString;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.not;
+
+@EndToEndTest
+public class FileTransferSampleTest {
+
+    static final ObjectMapper MAPPER = new ObjectMapper();
+    static final String INITIATE_CONTRACT_NEGOTIATION_URI = "http://localhost:9192/api/v1/data/contractnegotiations";
+    static final String LOOK_UP_CONTRACT_AGREEMENT_URI = "http://localhost:9192/api/v1/data/contractnegotiations/{id}";
+    static final String INITIATE_TRANSFER_PROCESS_URI = "http://localhost:9192/api/v1/data/transferprocess";
+    static final String CONTRACT_OFFER_FILE_PATH = "samples/04.0-file-transfer/contractoffer.json";
+    static final String CONSUMER_CONFIG_PROPERTIES_FILE_PATH = "samples/04.0-file-transfer/consumer/config.properties";
+    static final String PROVIDER_CONFIG_PROPERTIES_FILE_PATH = "samples/04.0-file-transfer/provider/config.properties";
+    static final String TRANSFER_FILE_PATH = "samples/04.0-file-transfer/filetransfer.json";
+    // Reuse an already existing file for the test. Could be set to any other existing file in the repository.
+    static final String SAMPLE_ASSET_FILE_PATH = TRANSFER_FILE_PATH;
+    static final String DESTINATION_FILE_PATH = "samples/04.0-file-transfer/consumer/requested.test.txt";
+    static final Duration TIMEOUT = Duration.ofSeconds(15);
+    static final Duration POLL_INTERVAL = Duration.ofMillis(500);
+    static final String API_KEY_HEADER_KEY = "X-Api-Key";
+    static final String API_KEY_HEADER_VALUE = "password";
+    @RegisterExtension
+    static EdcRuntimeExtension provider = new EdcRuntimeExtension(
+            ":samples:04.0-file-transfer:provider",
+            "provider",
+            Map.of(
+                    // Override 'edc.samples.04.asset.path' implicitly set via property 'edc.fs.config'.
+                    "edc.samples.04.asset.path", getFileFromRelativePath(SAMPLE_ASSET_FILE_PATH).getAbsolutePath(),
+                    "edc.fs.config", getFileFromRelativePath(PROVIDER_CONFIG_PROPERTIES_FILE_PATH).getAbsolutePath()
+            )
+    );
+    @RegisterExtension
+    static EdcRuntimeExtension consumer = new EdcRuntimeExtension(
+            ":samples:04.0-file-transfer:consumer",
+            "consumer",
+            Map.of(
+                    "edc.fs.config", getFileFromRelativePath(CONSUMER_CONFIG_PROPERTIES_FILE_PATH).getAbsolutePath()
+            )
+    );
+    static final File DESTINATION_FILE = getFileFromRelativePath(FileTransferSampleTest.DESTINATION_FILE_PATH);
+    static final File SAMPLE_ASSET_FILE = getFileFromRelativePath(FileTransferSampleTest.SAMPLE_ASSET_FILE_PATH);
+    String contractNegotiationId;
+    String contractAgreementId;
+
+    /**
+     * Resolves a {@link File} instance from a relative path.
+     */
+    @NotNull
+    static File getFileFromRelativePath(String relativePath) {
+        return new File(TestUtils.findBuildRoot(), relativePath);
+    }
+
+    /**
+     * Run all sample steps in one single test.
+     * Note: Sample steps cannot be separated into single tests because {@link EdcRuntimeExtension}
+     * runs before each single test.
+     */
+    @Test
+    void runSampleSteps() throws Exception {
+        assertTestPrerequisites();
+
+        initiateContractNegotiation();
+        lookUpContractAgreementId();
+        requestTransferFile();
+        assertDestinationFileContent();
+
+        cleanTemporaryTestFiles();
+    }
+
+    /**
+     * Assert that prerequisites are fulfilled before running the test.
+     * This assertion checks only whether the file to be copied is not existing already.
+     */
+    void assertTestPrerequisites() {
+        assertThat(DESTINATION_FILE).doesNotExist();
+    }
+
+    /**
+     * Remove files created while running the tests.
+     * The copied file will be deleted.
+     */
+    @SuppressWarnings("ResultOfMethodCallIgnored")
+    void cleanTemporaryTestFiles() {
+        DESTINATION_FILE.delete();
+    }
+
+    /**
+     * Assert that the file to be copied exists at the expected location.
+     * This method waits a duration which is defined in {@link FileTransferSampleTest#TIMEOUT}.
+     */
+    void assertDestinationFileContent() {
+        await().atMost(TIMEOUT).pollInterval(POLL_INTERVAL).untilAsserted(()
+                -> assertThat(DESTINATION_FILE).hasSameBinaryContentAs(SAMPLE_ASSET_FILE));
+    }
+
+    /**
+     * Assert that a POST request to initiate a contract negotiation is successful.
+     * This method corresponds to the command in the sample: {@code curl -X POST -H "Content-Type: application/json" -H "X-Api-Key: password" -d @samples/04.0-file-transfer/contractoffer.json "http://localhost:9192/api/v1/data/contractnegotiations"}
+     */
+    void initiateContractNegotiation() {
+        contractNegotiationId = RestAssured
+            .given()
+                .headers(API_KEY_HEADER_KEY, API_KEY_HEADER_VALUE)
+                .contentType(ContentType.JSON)
+                .body(new File(TestUtils.findBuildRoot(), CONTRACT_OFFER_FILE_PATH))
+            .when()
+                .post(INITIATE_CONTRACT_NEGOTIATION_URI)
+            .then()
+                .statusCode(HttpStatus.SC_OK)
+                .body("id", not(emptyString()))
+                .extract()
+                .jsonPath()
+                .get("id");
+    }
+
+    /**
+     * Assert that a GET request to look up a contract agreement is successful.
+     * This method corresponds to the command in the sample: {@code curl -X GET -H 'X-Api-Key: password' "http://localhost:9192/api/v1/data/contractnegotiations/{UUID}"}
+     */
+    void lookUpContractAgreementId() {
+        // Wait for transfer to be completed.
+        await().atMost(TIMEOUT).pollInterval(POLL_INTERVAL)
+                .untilAsserted(() ->
+                contractAgreementId = RestAssured
+                    .given()
+                        .headers(API_KEY_HEADER_KEY, API_KEY_HEADER_VALUE)
+                    .when()
+                        .get(LOOK_UP_CONTRACT_AGREEMENT_URI, contractNegotiationId)
+                    .then()
+                        .statusCode(HttpStatus.SC_OK)
+                        .body("state", equalTo("CONFIRMED"))
+                        .body("contractAgreementId", not(emptyString()))
+                        .extract().body().jsonPath().getString("contractAgreementId")
+        );
+    }
+
+    /**
+     * Assert that a POST request to initiate transfer process is successful.
+     * This method corresponds to the command in the sample: {@code curl -X POST -H "Content-Type: application/json" -H "X-Api-Key: password" -d @samples/04.0-file-transfer/filetransfer.json "http://localhost:9192/api/v1/data/transferprocess"}
+     *
+     * @throws IOException Thrown if there was an error accessing the transfer request file defined in {@link FileTransferSampleTest#TRANSFER_FILE_PATH}.
+     */
+    void requestTransferFile() throws IOException {
+        File transferJsonFile = getFileFromRelativePath(TRANSFER_FILE_PATH);
+        DataRequest sampleDataRequest = readAndUpdateDataRequestFromJsonFile(transferJsonFile, contractAgreementId);
+
+        JsonPath jsonPath = RestAssured
+                .given()
+                    .headers(API_KEY_HEADER_KEY, API_KEY_HEADER_VALUE)
+                    .contentType(ContentType.JSON)
+                    .body(sampleDataRequest)
+                .when()
+                    .post(INITIATE_TRANSFER_PROCESS_URI)
+                .then()
+                    .statusCode(HttpStatus.SC_OK)
+                    .body("id", not(emptyString()))
+                    .extract()
+                    .jsonPath();
+
+        String transferProcessId = jsonPath.get("id");
+
+        assertThat(transferProcessId).isNotEmpty();
+    }
+
+    /**
+     * Reads a transfer request file with changed value for contract agreement ID and file destination path.
+     *
+     * @param transferJsonFile A {@link File} instance pointing to a JSON transfer request file.
+     * @param contractAgreementId This string containing a UUID will be used as value for the contract agreement ID.
+     * @return An instance of {@link DataRequest} with changed values for contract agreement ID and file destination path.
+     * @throws IOException Thrown if there was an error accessing the file given in transferJsonFile.
+     */
+    static DataRequest readAndUpdateDataRequestFromJsonFile(File transferJsonFile, String contractAgreementId) throws IOException {
+        // convert JSON file to map
+        DataRequest sampleDataRequest = MAPPER.readValue(transferJsonFile, DataRequest.class);
+
+        var changedAddressProperties = sampleDataRequest.getDataDestination().getProperties();
+        changedAddressProperties.put("path", DESTINATION_FILE.getAbsolutePath());
+
+        DataAddress newDataDestination = DataAddress.Builder.newInstance()
+                .properties(changedAddressProperties)
+                .build();
+
+        return DataRequest.Builder.newInstance()
+                // copy unchanged values from JSON file
+                .id(sampleDataRequest.getId())
+                .processId(sampleDataRequest.getProcessId())
+                .connectorAddress(sampleDataRequest.getConnectorAddress())
+                .protocol(sampleDataRequest.getProtocol())
+                .connectorId(sampleDataRequest.getConnectorId())
+                .assetId(sampleDataRequest.getAssetId())
+                .destinationType(sampleDataRequest.getDestinationType())
+                .transferType(sampleDataRequest.getTransferType())
+                .managedResources(sampleDataRequest.isManagedResources())
+                .properties(sampleDataRequest.getProperties())
+                // set changed values
+                .contractId(contractAgreementId)
+                .dataDestination(newDataDestination)
+                .build();
+    }
+}

settings.gradle.kts

@@ -171,6 +171,7 @@ include(":samples:03-configuration")
 include(":samples:04.0-file-transfer:consumer")
 include(":samples:04.0-file-transfer:provider")
 include(":samples:04.0-file-transfer:transfer-file")
+include(":samples:04.0-file-transfer:integration-tests")
 
 include(":samples:04.1-file-transfer-listener:consumer")
 include(":samples:04.1-file-transfer-listener:listener")
@@ -204,3 +205,4 @@ include(":system-tests:runtimes:azure-data-factory-transfer-consumer")
 include(":system-tests:runtimes:azure-storage-transfer-consumer")
 include(":system-tests:azure-tests")
 include(":system-tests:azure-data-factory-tests")
+