Skip to content

Commit 33c1ca1

Browse files
authored
Update chain-indexing-overview-for-rpc-providers.md
Doc updates based on @jennijuju feedack.
1 parent 47e421f commit 33c1ca1

File tree

1 file changed

+32
-16
lines changed

1 file changed

+32
-16
lines changed

chain/index/chain-indexing-overview-for-rpc-providers.MD

+32-16
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,9 @@
22

33
## Introduction
44

5-
This document is for RPC providers and node operators who serve RPC requests walking through the configuration changes, migration flow and operations/maintenance work needed to enable, backfill and maintain the `ChainIndexer`.
5+
This document is for RPC providers and node operators who serve RPC requests walking through the configuration changes, migration flow and operations/maintenance work needed to enable, backfill and maintain the [`ChainIndexer`](chainindex-indexing#system).
66

7-
**Note: If you are a Storage Provider or node operator who does not serve RPC requests, you can skip this document as the `ChainIndexer` is already disabled by default**.
8-
7+
**Note: If you are a Storage Provider or node operator who does not serve RPC requests (i.e, if `Fevm.EnableEthRPC = false`), you can skip this document as the `ChainIndexer` is already disabled by default**.
98

109
## ChainIndexer Config
1110
### Enablement
@@ -66,36 +65,38 @@ DisableHistoricFilterAPI = false
6665
DatabasePath = ""
6766
```
6867

69-
## Migration Guide
68+
## Upgrade Steps
69+
70+
> **Note:** One can upgrade/downgrade between [pre-ChainIndexer](#previous-indexing-system) and [with-ChainIndexer](#chainindex-indexing-system) Lotus versions without conflict because they persist state to different directories and don't rely on each other. No backup is necessary (but extra backups don't hurt).
7071
71-
Migrating to the new `ChainIndexer` involves several steps to ensure a smooth transition:
72+
Upgrading to the new `ChainIndexer` involves these steps:
7273

7374
1. **Stop the Lotus Node**
7475
- Stop your Lotus node before starting the upgrade and migration process.
7576
2. **Update Configuration**
7677
- Modify your Lotus configuration to enable the `ChainIndexer` as described in the [`ChainIndexer Config` section above](#chainindexer-config).
77-
7878
3. **Restart Lotus Node**
7979
- Restart your Lotus node with the new configuration.
8080
- The `ChainIndexer` will begin indexing **real-time chain state changes** immediately in the `${LOTUS_PATH}/chainindex` directory.
81-
- **However, it will not automatically index any historical chain state (i.e., any previously existing chain state). To perform backfilling, please see the [`Backfilling` section below](#backfilling).**
81+
- **However, it will not automatically index any historical chain state (i.e., any previously existing chain state). To perform backfilling, please see the [`Backfill` section below](#backfill).**
82+
83+
> **Note:** It's recommended to keep the [pre-ChainIndexer](#previous-indexing-system) state directory around until you've confirmed you don't need to [downgrade](#downgrade). After sustained successful operation after the upgrade, the [pre-ChainIndexer](#previous-indexing-system) state directory can be removed to reclaim disk space.
8284
83-
Note: The ChainIndexer will write state to `${LOTUS_HOME}/chainindex` rather than the [removed option](#removed-options) for `Events.DatabasePath` (which defaulted to `${LOTUS_HOME}/sqllite`). It's recommended to keep the legacy directory around until you've confirmed you don't need to [rollback](#rollback).
8485
## Backfill
85-
To index historical chain state (i.e., **"backfilling"**), you can use the following tools:
86+
There is no migration from [pre-ChainIndexer indices](#previous-indexing-system) to the [ChainIndex](#chainindex-indexing-system) because the [pre-ChainIndexer indices](#previous-indexing-system) are not fully reliable. Instead one needs to index historical chain state (i.e., backfill) using one of the following tools.
87+
88+
> **Note:** The ChainIndex will consume ~10GB of storage per month of tipsets (e.g., ~86400 epochs). Ensure you have sufficient disk space before proceeding.
8689
8790
### `ChainValidateIndex` JSON RPC API
8891

8992
Please refer to the [Lotus API documentation](https://github.com/filecoin-project/lotus/blob/master/documentation/en/api-v1-unstable-methods.md) for detailed documentation of the `ChainValidateIndex` JSON RPC API.
9093

9194
The `ChainValidateIndex` JSON RPC API serves a dual purpose: it validates/diagnoses the integrity of the index at a specific epoch (i.e., it ensures consistency between indexed data and actual chain state), while also providing the option to backfill the `ChainIndexer` if it does not have data for the specified epoch.
9295

93-
9496
The `ChainValidateIndex` RPC API is available for use once the Lotus daemon has started with `ChainIndexer` [enabled](#enablement).
9597

9698
Here are some examples of how to use the `ChainValidateIndex` JSON RPC API for validating/ backfilling the index:
9799

98-
99100
1) Validating the index for an epoch that is a NULL round:
100101

101102
```bash
@@ -191,12 +192,27 @@ The command validates the chain index entries for each epoch in the specified ra
191192

192193
You can learn about how to use the tool with `lotus-shed chainindex validate-backfill -h`.
193194

194-
## Rollback
195+
## Downgrade
195196

196-
In case you need to rollback to the previous indexing system (`EthTxIndex`, `MsgIndex`, and `EventIndex`), follow these steps:
197+
In case you need to downgrade to the [previous indexing system](#previous indexing system), follow these steps:
197198

198199
1. Stop your Lotus node.
199-
2. Build your Lotus binary for the rollback version which has the implementation of the old `EthTxIndex`, `MsgIndex`, and `EventIndex` Indices.
200-
4. Ensure that you've set the correct config for the existing `EthTxIndex`, `MsgIndex`, and `EventIndex` indices in the `config.toml` file.
200+
2. Build your Lotus binary for the rollback version which has the implementation of the old `EthTxHashLookup`, `MsgIndex`, and `EventIndex` indices.
201+
4. Ensure that you've set the correct config for the existing `EthTxHashLookup`, `MsgIndex`, and `EventIndex` indices in the `config.toml` file.
201202
5. Restart your Lotus node.
202-
6. Backfill the `EthTxIndex`, `MsgIndex`, and `EventIndex` indices using the `lotus-shed index backfill-*` CLI tooling available in the previous indexing system for the epoch range in [epochs between the upgrade to `ChainIndexer` and the rollback of `ChainIndexer`].
203+
6. Backfill the `EthTxHashLookup`, `MsgIndex`, and `EventIndex` indices using the `lotus-shed index backfill-*` CLI tooling available in the [previous indexing system](#previous-indexing-system) for the epoch range in [epochs between the upgrade to `ChainIndexer` and the rollback of `ChainIndexer`].
204+
205+
## Terminology
206+
### Previous Indexing System
207+
* This corresponds to the indexing system used in Lotus versions before v1.31.0.
208+
* It has been replaced by the [ChainIndex](#chainindex-indexing-system).
209+
* It was composed of three indexers using three separate databases: [`EthTxHashLookup`](https://github.com/filecoin-project/lotus/blob/master/chain/ethhashlookup/eth_transaction_hash_lookup.go), [`MsgIndex`](https://github.com/filecoin-project/lotus/blob/master/chain/index/msgindex.go), and [`EventIndex`](https://github.com/filecoin-project/lotus/blob/master/chain/events/filter/index.go).
210+
* It persisted state to the [removed option](#removed-options) for `Events.DatabasePath`, which defaulted to `${LOTUS_HOME}/sqllite`.
211+
* It had CLI backfill tooling: `lotus-shed index backfill-*`
212+
213+
### ChainIndex Indexing System
214+
* This corresponds to the indexing system used in Lotus versions v1.31.0 onwards.
215+
* It replaced the [previous indexing system](#previous-indexing-system).
216+
* It is composed of a single indexer, [`ChainIndexer`](https://github.com/filecoin-project/lotus/blob/master/chain/index/indexer.go), using a single database for transactions, messages, and events.
217+
* It persists state to `${LOTUS_HOME}/chainindex`.
218+
* It has this CLI backfill tooling: [`lotus-shed chainindex validate-backfill`](#lotus-shed-chainindex-validate-backfill-cli-tool)

0 commit comments

Comments
 (0)