testing processor

Author: yuunlimm

apps/nextra/next.config.mjs

@@ -461,6 +461,13 @@ export default withBundleAnalyzer(
           "/en/build/indexer/indexer-sdk/documentation/advanced-tutorials/txn-scripts",
         permanent: true,
       },
+      {
+        source:
+          "/indexer/indexer-sdk/documentation/advanced-tutorials/processor-test",
+        destination:
+          "/en/build/indexer/indexer-sdk/documentation/advanced-tutorials/processor-test",
+        permanent: true,
+      },
       {
         source: "/indexer/txn-stream/labs-hosted",
         destination: "/en/build/indexer/api/labs-hosted",

apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/advanced-tutorials/_meta.tsx

@@ -6,4 +6,7 @@ export default {
   "txn-script": {
     title: "Generating Transactions with Move Scripts",
   },
+  "processor-test": {
+    title: "Testing Your Processor",
+  },
 };

apps/nextra/pages/en/build/indexer/indexer-sdk/documentation/advanced-tutorials/processor-test.mdx

@@ -0,0 +1,172 @@
+---
+title: "Testing Processor"
+---
+
+# Processor Test Runbook
+
+### What Are We Testing With This?
+
+- **Transaction correctness**: Ensure that each transaction is processed and stored accurately.
+- **Schema consistency**: Verify that the database schema is correctly set up and maintained throughout the tests.
+
+### General Flow
+
+1. Prepare testing transactions (refer to prior documentation).
+2. Update dependencies as needed.
+3. Import new transactions.
+4. Write test cases.
+5. Generate expected database output and validate.
+6. Merge.
+
+Ensure tests pass whenever there are updates in the processor code.
+
+
+## Prerequisites
+
+1. Ensure Docker is running for PostgreSQL container support.
+2. Identify the transactions to test.
+3. Import necessary modules, see example:
+    ```rust
+    use aptos_indexer_testing_framework::{
+        database::{PostgresTestDatabase, TestDatabase},
+        sdk_test_context::SdkTestContext,
+    };
+    ```
+
+> 💡 **Key Considerations**:
+>
+> - Each test runs in an isolated environment using a PostgreSQL container to prevent interference.
+> - Proper handling of versions ensures transactions are processed and validated in the correct order.
+> - Validation logic must detect changes or issues by comparing processor output with the expected baseline.
+
+
+---
+
+
+## Steps to Write a Test
+
+### 1. Set Up the Test Environment
+
+```rust
+let (diff_flag, custom_output_path) = get_test_config();
+let output_path = custom_output_path.unwrap_or_else(|| format!("{}/imported_mainnet_txns", DEFAULT_OUTPUT_FOLDER));
+
+// Replace with your test transaction
+let (db, mut test_context) = setup_test_environment(&[IMPORTED_MAINNET_TXNS_438536688_ANS_CURRENT_ANS_LOOKUP_V2]).await;
+```
+
+### 2. Configure the Processor
+
+```rust
+let db_url = db.get_db_url();
+let (indexer_processor_config, processor_name) =
+    setup_ans_processor_config(&test_context, &db_url);
+```
+
+### 3. Create the Processor
+
+```rust
+let processor = DefaultProcessor::new(indexer_processor_config)
+    .await
+    .expect("Failed to create processor");
+```
+
+### 4. Query the Databasefaw
+
+Set up a query to load data from the local database and compare it with expected results.
+
+### 5. Run the Processor Test
+
+```rust
+#[derive(Debug, Clone)]
+pub struct TestArgs {
+    pub generate_output: bool,
+    pub output_path: Option<String>,
+}
+```
+
+Run the processor and verify the outputs using `SdkTestContext.run`.
+
+---
+
+## Tips
+1. **Isolate Tests**: Use Docker containers for database isolation.
+2. **Handle Non-Deterministic Fields**: Use helpers like `remove_inserted_at` to clean up timestamps before validation.
+3. **Enable Debugging**: Use `eprintln!` for detailed error logging.
+
+
+## FAQ
+
+### What Types of Tests Does It Support?
+
+- Database (PostgreSQL) schema output diff.
+
+### How Does It Work?
+
+#### Step-by-Step Flow:
+
+1. Setup Test Configuration: Define the parameters in the CLI command.
+2. Specify Transactions to Import: List the transactions to test.
+3. Initialize the Test Environment: Set up the test database and context.
+4. Optional - Deserialize Transactions: Convert transactions into `Transaction` structs for processing.
+5. Configure the Processor: Define the database URL and processor settings.
+6. Run the Processor Test: Execute the processor and validate correctness.
+
+### What Is `TestContext`?
+
+`TestContext` is a struct that manages:
+
+- `transaction_batches`: A collection of transaction batches.
+- `postgres_container`: A PostgreSQL container for test isolation.
+
+It initializes and manages the database and transaction context for tests.
+
+#### What Does `TestContext.run` Do?
+
+This function executes the processor, applies validation logic, and optionally generates output files.
+
+#### Key Features:
+
+- Flexible Validation: Accepts a user-provided verification function.
+- Multi-Table Support: Handles data across multiple tables.
+- Retries: Uses exponential backoff and timeout for retries.
+- Optional File Generation: Controlled by a flag.
+
+#### Example Usage:
+
+```rust
+pub async fn run<F>(
+    &mut self,
+    processor: &impl ProcessorTrait,
+    txn_version: u64,
+    generate_files: bool,             // Flag to control file generation
+    output_path: String,              // Output path
+    custom_file_name: Option<String>, // Custom file name
+    verification_f: F,                // Verification function
+) -> anyhow::Result<HashMap<String, Value>>
+where
+```
+
+### How to Generate Expected DB Output?
+
+Run the following command:
+
+```bash
+cargo test sdk_tests -- --nocapture generate-output
+```
+
+Supported Test Args:
+
+1. `generate-output`
+2. `output_path`
+
+---
+
+## Additional Notes
+
+- **Adapting to Other Databases**:
+  - Replace PostgreSQL-specific code with relevant database APIs (e.g., MySQL).
+  - Update schema initialization and query methods.
+- **Docker Installation**: Follow the [Docker setup guide](https://docs.docker.com/get-docker/).
+- **Referencing Existing Tests**:
+  - Example: [Event Processor Tests](https://github.com/aptos-labs/aptos-indexer-processors/blob/main/rust/integration-tests/src/sdk_tests/events_processor_tests.rs#L139).