OpenSearch Migrations Engine

Table of Contents

1. Overview
2. Key Features
3. Supported Versions and Platforms
4. Issue Tracking
5. User Guide Documentation
6. Getting Started
   • Local Deployment
   • AWS Deployment
7. Continuous Integration and Deployment
8. Contributing
9. Security
10. License
11. Acknowledgments

Overview

The OpenSearch Migrations Engine is a comprehensive set of tools designed to facilitate upgrades, migrations, and comparisons for OpenSearch and Elasticsearch clusters. This project aims to simplify the process of moving between different versions and platforms while ensuring data integrity and performance.

Key Features

• Upgrade and Migration Support: Provides tools for migrating between different versions of Elasticsearch and OpenSearch.

  • Metadata Migration: Migrate essential cluster components such as configuration, settings, templates, and aliases.
  • Multi-Version Upgrade: Easily migrate across major versions (e.g., from Elasticsearch 6.8 to OpenSearch 2.15), skipping intermediate upgrades and reducing time and risk.
  • Downgrade Support: Downgrade to an earlier version if needed (e.g., from Elasticsearch 7.17 to 7.10.2).
  • Existing Data Migration with Reindex-from-Snapshot: Migrate indices and documents using snapshots, updating your data to the latest Lucene version quickly without impacting the target cluster.
  • Live Traffic Capture with Capture-and-Replay: Capture live traffic from the source cluster and replay it on the target cluster for validation. This ensures the target cluster can handle real-world traffic patterns before fully migrating.

• Zero-Downtime Migration with Live Traffic Routing: Tools to seamlessly switch client traffic between clusters while keeping services fully operational.

• Migration Rollback: Keep your source cluster synchronized during the migration, allowing you to monitor the target cluster's performance before fully committing to the switch. You can safely revert if needed.

• User-Friendly Interface via Migration Console: Command Line Interface (CLI) that guides you through each migration step.

• Flexible Deployment Options:

  • AWS Deployment: Fully automated deployment to AWS.
  • Local Docker Deployment: Run the solution locally in a container for testing and development.
  • Contribute to add more deployment options.

Supported Migration Paths and Platforms

Migration Paths

Source Version	Target Version	Status
Elasticsearch 5.6	OpenSearch 1.3	Supported
Elasticsearch 5.6	OpenSearch 2.19	Supported
Elasticsearch 6.8	OpenSearch 1.3	Supported
Elasticsearch 6.8	OpenSearch 2.19	Supported
Elasticsearch 7.10.2	OpenSearch 1.3	Supported
Elasticsearch 7.10.2	OpenSearch 2.19	Supported
Elasticsearch 7.17	OpenSearch 1.3	Supported
Elasticsearch 7.17	OpenSearch 2.19	Supported
OpenSearch 1.3	OpenSearch 2.19	Supported
Elasticsearch 8.x	OpenSearch 2.x	Prioritized
Elasticsearch 2.3	OpenSearch 2.x	Prioritized
Elasitcsearch 1.5	OpenSearch 2.x	Prioritized
OpenSearch 2.x	OpenSearch 2.x	Requested

Note that testing is done on specific minor versions, but any minor versions within a listed major version are expected to work.

Platforms

• Self-managed (cloud provider hosted)
• Self-managed (on-premises)
• Managed cloud offerings (e.g., Amazon OpenSearch, Amazon OpenSearch Serverless)

Performance Limitations

A performance test was performed on 03/10/25 alongside PR 1337 to identify general indexing throughput with Reindex-From-Snapshot and Traffic Replay. While Reindex-From-Snapshot includes periods of ingestion inactivity (e.g. while downloading the shard data), this was not factored into the ingestion rates. Test used docs with an average uncompressed size 2.39 KB (source doc and index command) which corresponded to 1.54 KB of primary shard size per doc on OS 2.17 with default settings

For real-word use, throughput can be increased by vertically scaling Reindex-From-Snapshot and Traffic Replay instances or horizontally scaling the Reindex-From-Snapshot workers that are running on AWS Fargate. Outside the exception indicated, all cases are CPU limited for throughput. All cases were using uncompressed network traffic generated by the applications, results will vary if using client compression.

Throughput here is measured by the rate of increase in primary shard data with bulk ingestion on the target cluster alongside the uncompressed size of the source data ingested.

Tests were ran with and without the Type Mapping Sanitization Transformer.

Test Results

Service	vCPU	Memory (GB)	Type Mapping Sanitization	Peak Docs Ingested per minute	Primary Shard Data Ingestion Rate (MBps)	Uncompressed Source Data Ingestion Rate (MBps)
Reindex-From-Snapshot	2	4	Disabled	590,000	15.1	23.5
Reindex-From-Snapshot	2	4	Enabled	546,000	14.0	21.7
Traffic Replay	8	48	Disabled	1,694,000 [1]	43.5 [1]	67.5 [1]
Traffic Replay	8	48	Enabled	1,645,000	42.2	65.5

[1] Network Bandwidth Limitations Observed

Issue Tracking

We encourage users to open bugs and feature requests in this GitHub repository.

Encountering a compatibility issue or missing feature?

• Search existing issues to see if it’s already reported. If it is, feel free to upvote and comment.
• Can’t find it? Create a new issue to let us know.

For issue prioritization and management, the migrations team uses Jira, but uses GitHub issues for community intake:

https://opensearch.atlassian.net/

User Guide Documentation

User guide documentation is available in the OpenSearch Migrations Wiki.

Getting Started

Local Deployment

Refer to the Development Guide for more details.

AWS Deployment

To deploy the solution on AWS, follow the steps outlined in Migration Assistant for Amazon OpenSearch Service, specifically deploying the solution.

Continuous Integration and Deployment

We use a combination of GitHub actions and Jenkins so that we can publish releases on a weekly basis and allow users to provide attestation for migration tooling.

Jenkins pipelines are available here

Contributing

Please read CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.

Please refer to the Development Guide for details on building and deploying.

Security

See SECURITY.md for information about reporting security vulnerabilities.

License

This project is licensed under the Apache-2.0 License - see the LICENSE file for details.

Acknowledgments

• OpenSearch Community
• Contributors and maintainers

For more detailed information about specific components, please refer to the README files in the respective directories.