Skip to content

Commit b12811d

Browse files
deblasish5law
deblasis
and
h5law
authored
[CLI][Localnet][Bug] - [localnet_client_debug] Error 137 - Issue #607 (#608)
## Description This PR improves the keybase initialization by: - doing the work only if the `private_keys.yaml` has been changed - creating a backup of the keybase - loading it (which is way cheaper than parsing the yaml and rehydrating the keybase from scratch) ### Before At 1:46 I run `make localnet_client_debug` and you can see it takes a long time plus it's a very expensive operation https://user-images.githubusercontent.com/29378614/227202479-1320fdf4-f851-4a58-884e-a36b3c385b84.mp4 ### After At 1:12 I run `make localnet_client_debug` and it's basically instantaneous with no further spikes in CPU/RAM usage https://user-images.githubusercontent.com/29378614/227202494-a1eff819-e2d9-4881-a2e5-4b28448d574c.mp4 Where is the magic? We do the expensive operation only when needed and that means when the `private_keys.yaml` has changed. This is handled by a small binary that calculates the hash of the yaml file, compares it with the one we have already (if any) and if needed, it recreates a backup of the keybase that can be rehydrated from the debug client. Sloppy recording but it shows the whole process: https://user-images.githubusercontent.com/29378614/227204654-7b7f3ea8-6db5-473c-ac6c-75be13f60553.mp4 ## Issue Fixes #607 ## Type of change Please mark the relevant option(s): - [x] New feature, functionality or library - [x] Bug fix - [ ] Code health or cleanup - [ ] Major breaking change - [ ] Documentation - [ ] Other <!-- add details here if it a different type of change --> ## List of changes - Updated debug keybase initialization - Created small binary to help with the integrity check ## Testing - [x] `make develop_test` - [x] [LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md) w/ all of the steps outlined in the `README` <!-- REMOVE this comment block after following the instructions If you added additional tests or infrastructure, describe it here. Bonus points for images and videos or gifs. --> ## Required Checklist - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have added, or updated, [`godoc` format comments](https://go.dev/blog/godoc) on touched members (see: [tip.golang.org/doc/comment](https://tip.golang.org/doc/comment)) - [x] I have tested my changes using the available tooling - [x] I have updated the corresponding CHANGELOG ### If Applicable Checklist - [ ] I have updated the corresponding README(s); local and/or global - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have added, or updated, [mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding README(s) - [ ] I have added, or updated, documentation and [mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*` if I updated `shared/*`README(s) --------- Co-authored-by: harry <[email protected]>
1 parent a326c9e commit b12811d

File tree

8 files changed

+243
-98
lines changed

8 files changed

+243
-98
lines changed

‎Makefile

+5
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,7 @@ develop_start: ## Run all of the make commands necessary to develop on the proje
116116
make go_clean_deps && \
117117
make mockgen && \
118118
make generate_rpc_openapi && \
119+
make refresh_debug_keybase && \
119120
make build
120121

121122
.PHONY: develop_test
@@ -218,6 +219,10 @@ docker_loki_install: docker_check ## Installs the loki docker driver
218219
docker_loki_check:
219220
if [ `docker plugin ls | grep loki: | wc -l` -eq 0 ]; then make docker_loki_install; fi
220221

222+
.PHONY: refresh_debug_keybase
223+
refresh_debug_keybase: ## Refreshes the debug keybase with the latest keys from the private_keys.yaml file if necessary
224+
go run build/debug_keybase/main.go ./build/localnet/manifests/private-keys.yaml ./build/debug_keybase
225+
221226
.PHONY: clean_mocks
222227
clean_mocks: ## Use `clean_mocks` to delete mocks before recreating them. Also useful to cleanup code that was generated from a different branch
223228
$(eval modules_dir = "shared/modules")

‎app/client/keybase/debug/keystore.go

+27-93
Original file line numberDiff line numberDiff line change
@@ -4,26 +4,17 @@ package debug
44

55
import (
66
"fmt"
7+
"io/ioutil"
78
"os"
9+
"path/filepath"
810

9-
"github.com/korovkin/limiter"
11+
"github.com/dgraph-io/badger/v3"
1012
"github.com/pokt-network/pocket/app/client/keybase"
1113
"github.com/pokt-network/pocket/build"
1214
"github.com/pokt-network/pocket/logger"
13-
cryptoPocket "github.com/pokt-network/pocket/shared/crypto"
14-
"gopkg.in/yaml.v2"
1515
)
1616

17-
const (
18-
// NOTE: This is the number of validators in the private-keys.yaml manifest file
19-
numValidators = 999
20-
debugKeybaseSuffix = "/.pocket/keys"
21-
22-
// Increasing this number is linearly proportional to the amount of RAM required for the debug client to start and import
23-
// pre-generated keys into the keybase. Beware that might cause OOM and process can exit with 137 status code.
24-
// 4 threads takes 350-400MiB from a few tests which sounds acceptable.
25-
debugKeybaseImportConcurrencyLimit = 4
26-
)
17+
const debugKeybaseSuffix = "/.pocket/keys"
2718

2819
var (
2920
// TODO: Allow users to override this value via `datadir` flag or env var or config file
@@ -45,81 +36,17 @@ func init() {
4536
}
4637

4738
func initializeDebugKeybase() error {
48-
var (
49-
validatorKeysPairMap map[string]string
50-
err error
51-
)
52-
53-
validatorKeysPairMap, err = parseValidatorPrivateKeysFromEmbeddedYaml()
54-
55-
if err != nil {
56-
return err
57-
}
58-
5939
// Create/Open the keybase at `$HOME/.pocket/keys`
6040
kb, err := keybase.NewKeybase(debugKeybasePath)
6141
if err != nil {
6242
return err
6343
}
6444
db := kb.GetBadgerDB()
6545

66-
// Add the keys if the keybase contains less than 999
67-
curAddr, _, err := kb.GetAll()
68-
if err != nil {
46+
if err := restoreBadgerDB(build.DebugKeybaseBackup, db); err != nil {
6947
return err
7048
}
7149

72-
// Add validator addresses if not present
73-
if len(curAddr) < numValidators {
74-
logger.Global.Debug().Msgf(fmt.Sprintf("Debug keybase initializing... Adding %d validator keys to the keybase", numValidators-len(curAddr)))
75-
76-
// Use writebatch to speed up bulk insert
77-
wb := db.NewWriteBatch()
78-
79-
// Create a channel to receive errors from goroutines
80-
errCh := make(chan error, numValidators)
81-
82-
limit := limiter.NewConcurrencyLimiter(debugKeybaseImportConcurrencyLimit)
83-
84-
for _, privHexString := range validatorKeysPairMap {
85-
limit.Execute(func() {
86-
// Import the keys into the keybase with no passphrase or hint as these are for debug purposes
87-
keyPair, err := cryptoPocket.CreateNewKeyFromString(privHexString, "", "")
88-
if err != nil {
89-
errCh <- err
90-
return
91-
}
92-
93-
// Use key address as key in DB
94-
addrKey := keyPair.GetAddressBytes()
95-
96-
// Encode KeyPair into []byte for value
97-
keypairBz, err := keyPair.Marshal()
98-
if err != nil {
99-
errCh <- err
100-
return
101-
}
102-
if err := wb.Set(addrKey, keypairBz); err != nil {
103-
errCh <- err
104-
return
105-
}
106-
})
107-
}
108-
109-
limit.WaitAndClose()
110-
111-
// Check if any goroutines returned an error
112-
select {
113-
case err := <-errCh:
114-
return err
115-
default:
116-
}
117-
118-
if err := wb.Flush(); err != nil {
119-
return err
120-
}
121-
}
122-
12350
// Close DB connection
12451
if err := kb.Stop(); err != nil {
12552
return err
@@ -128,24 +55,31 @@ func initializeDebugKeybase() error {
12855
return nil
12956
}
13057

131-
// parseValidatorPrivateKeysFromEmbeddedYaml fetches the validator private keys from the embedded build/localnet/manifests/private-keys.yaml manifest file.
132-
func parseValidatorPrivateKeysFromEmbeddedYaml() (map[string]string, error) {
58+
func restoreBadgerDB(backupData []byte, db *badger.DB) error {
59+
logger.Global.Debug().Msg("Debug keybase initializing... Restoring from the embedded backup file...")
13360

134-
// Parse the YAML file and load into the config struct
135-
var config struct {
136-
ApiVersion string `yaml:"apiVersion"`
137-
Kind string `yaml:"kind"`
138-
MetaData map[string]string `yaml:"metadata"`
139-
Type string `yaml:"type"`
140-
StringData map[string]string `yaml:"stringData"`
61+
// Create a temporary directory to store the backup data
62+
tempDir, err := ioutil.TempDir("", "badgerdb-restore")
63+
if err != nil {
64+
return err
14165
}
142-
if err := yaml.Unmarshal(build.PrivateKeysFile, &config); err != nil {
143-
return nil, err
66+
defer os.RemoveAll(tempDir)
67+
68+
// Write the backup data to a file in the temporary directory
69+
backupFilePath := filepath.Join(tempDir, "backup")
70+
if err := ioutil.WriteFile(backupFilePath, backupData, 0644); err != nil {
71+
return err
14472
}
145-
validatorKeysMap := make(map[string]string)
14673

147-
for id, privHexString := range config.StringData {
148-
validatorKeysMap[id] = privHexString
74+
backupFile, err := os.Open(backupFilePath)
75+
if err != nil {
76+
return err
14977
}
150-
return validatorKeysMap, nil
78+
defer backupFile.Close()
79+
80+
if err := db.Load(backupFile, 4); err != nil {
81+
return err
82+
}
83+
84+
return nil
15185
}

‎app/docs/CHANGELOG.md

+4
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
77

88
## [Unreleased]
99

10+
## [0.0.0.4] - 2023-03-24
11+
12+
- Updated debug keystore initialization to use an embedded backup instead of the yaml file that has to be rehydrated every time.
13+
1014
## [0.0.0.3] - 2023-03-20
1115

1216
- Adds message routing type field labels to debug CLI actions

‎build/debug.go

+5-5
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,13 @@ package build
44

55
import _ "embed"
66

7-
// PrivateKeysFile is the pre-generated manifest file for LocalNet debugging
7+
// DebugKeybaseBackup is a backup of the pre-loaded debug keybase sourced from the manifest file for LocalNet debugging
88
//
9-
//go:embed localnet/manifests/private-keys.yaml
10-
var PrivateKeysFile []byte
9+
//go:embed debug_keybase/debug_keybase.bak
10+
var DebugKeybaseBackup []byte
1111

1212
func init() {
13-
if len(PrivateKeysFile) == 0 {
14-
panic("PrivateKeysFile is empty")
13+
if len(DebugKeybaseBackup) == 0 {
14+
panic("DebugKeybaseBackup is empty")
1515
}
1616
}

‎build/debug_keybase/cedffe1d784ca397062169b565abc056.md5

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
This file is used to check if the keybase dump is in sync with the YAML file. Its name is the MD5 hash of the private_keys.yaml

‎build/debug_keybase/debug_keybase.bak

481 KB
Binary file not shown.

0 commit comments

Comments
 (0)