Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[debug client] Limit key import concurrency to 4 #591

Merged
merged 7 commits into from
Mar 20, 2023
Merged

[debug client] Limit key import concurrency to 4 #591

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4113cf2
limit key import concurrency to 4
okdas Mar 16, 2023
426f341
switch to const
okdas Mar 16, 2023
1a306e0
add changelog
okdas Mar 16, 2023
e4ceef4
Update app/client/keybase/debug/keystore.go
okdas Mar 17, 2023
87b47e7
add comment regarding concurrency limit
okdas Mar 17, 2023
428dec3
add pvc for a client pod
okdas Mar 17, 2023
7d50fe8
added changelog for build dir
okdas Mar 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions app/client/doc/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.0.0.22] - 2023-03-17

- Added a limit on concurrent key imports for debug client to avoid OOM.

## [0.0.0.21] - 2023-03-14

- Simplifies the debug CLI tooling by embedding private-keys.yaml manifest
Expand Down
20 changes: 10 additions & 10 deletions app/client/keybase/debug/keystore.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ package debug
import (
"fmt"
"os"
"sync"

"github.com/korovkin/limiter"
"github.com/pokt-network/pocket/app/client/keybase"
"github.com/pokt-network/pocket/build"
"github.com/pokt-network/pocket/logger"
Expand All @@ -18,6 +18,11 @@ const (
// NOTE: This is the number of validators in the private-keys.yaml manifest file
numValidators = 999
debugKeybaseSuffix = "/.pocket/keys"

// Increasing this number is linearly proportional to the amount of RAM required for the debug client to start and import
// pre-generated keys into the keybase. Beware that might cause OOM and process can exit with 137 status code.
// 4 threads takes 350-400MiB from a few tests which sounds acceptable.
debugKeybaseImportConcurrencyLimit = 4
)

var (
Expand Down Expand Up @@ -74,14 +79,10 @@ func initializeDebugKeybase() error {
// Create a channel to receive errors from goroutines
errCh := make(chan error, numValidators)

// Create a WaitGroup to wait for all goroutines to finish
var wg sync.WaitGroup
wg.Add(numValidators)
limit := limiter.NewConcurrencyLimiter(debugKeybaseImportConcurrencyLimit)

for _, privHexString := range validatorKeysPairMap {
go func(privHexString string) {
defer wg.Done()

limit.Execute(func() {
// Import the keys into the keybase with no passphrase or hint as these are for debug purposes
keyPair, err := cryptoPocket.CreateNewKeyFromString(privHexString, "", "")
if err != nil {
Expand All @@ -102,11 +103,10 @@ func initializeDebugKeybase() error {
errCh <- err
return
}
}(privHexString)
})
}

// Wait for all goroutines to finish
wg.Wait()
limit.WaitAndClose()

// Check if any goroutines returned an error
select {
Expand Down
5 changes: 5 additions & 0 deletions build/docs/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.0.0.24] - 2023-03-17

- Added resource limits and PVC for debug client pod.
- Added `procps` to the client pod image.

## [0.0.0.23] - 2023-03-14

- Simplifies the debug CLI tooling by embedding private-keys.yaml manifest
Expand Down
1 change: 1 addition & 0 deletions build/localnet/Tiltfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ docker_build_with_restart(
"client-image",
root_dir,
dockerfile_contents="""FROM debian:bullseye
RUN apt-get update && apt-get install -y procps
WORKDIR /
COPY bin/client-linux /usr/local/bin/client
""",
Expand Down
23 changes: 23 additions & 0 deletions build/localnet/manifests/cli-client.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,13 @@ spec:
image: client-image
imagePullPolicy: IfNotPresent
command: ["sleep", "infinity"]
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "512Mi"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the client process die quick instead of crashing control plane when memory is full.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you check if this works with docker-compose client?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you check if this works with docker-compose client?

Yep, it worked for me!

cpu: "4"
env:
- name: POCKET_P2P_IS_CLIENT_ONLY
value: "true"
Expand Down Expand Up @@ -74,6 +81,8 @@ spec:
name: config-volume
- mountPath: /var/pocket/genesis
name: genesis-volume
- mountPath: /home/root/.pocket
name: datadir
volumes:
- name: config-volume
configMap:
Expand All @@ -83,3 +92,17 @@ spec:
configMap:
name: v1-localnet-genesis
defaultMode: 420
- name: datadir
persistentVolumeClaim:
claimName: pocket-v1-cli-client-datadir
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pocket-v1-cli-client-datadir
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
9 changes: 5 additions & 4 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ require (
github.com/getkin/kin-openapi v0.107.0
github.com/jackc/pgconn v1.13.0
github.com/jordanorelli/lexnum v0.0.0-20141216151731-460eeb125754
github.com/korovkin/limiter v0.0.0-20230307205149-3d4b2b34c99d
github.com/labstack/echo/v4 v4.9.1
github.com/libp2p/go-libp2p v0.25.1
github.com/libp2p/go-libp2p-pubsub v0.9.2
Expand All @@ -37,7 +38,7 @@ require (
github.com/rs/zerolog v1.27.0
github.com/spf13/cobra v1.6.0
github.com/spf13/viper v1.13.0
golang.org/x/term v0.3.0
golang.org/x/term v0.5.0
gopkg.in/yaml.v2 v2.4.0
k8s.io/api v0.26.1
k8s.io/apimachinery v0.26.1
Expand Down Expand Up @@ -69,7 +70,7 @@ require (
github.com/prometheus/client_golang v1.14.0
github.com/sirupsen/logrus v1.9.0 // indirect
go.opencensus.io v0.23.0 // indirect
golang.org/x/net v0.4.0 // indirect
golang.org/x/net v0.7.0 // indirect
gotest.tools v2.2.0+incompatible // indirect
)

Expand Down Expand Up @@ -206,8 +207,8 @@ require (
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasttemplate v1.2.2 // indirect
golang.org/x/mod v0.7.0 // indirect
golang.org/x/sys v0.3.0 // indirect
golang.org/x/text v0.5.0 // indirect
golang.org/x/sys v0.5.0 // indirect
golang.org/x/text v0.7.0 // indirect
golang.org/x/time v0.0.0-20220411224347-583f2d630306 // indirect
golang.org/x/tools v0.3.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
Expand Down
18 changes: 10 additions & 8 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,8 @@ github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxv
github.com/koron/go-ssdp v0.0.0-20191105050749-2e1c40ed0b5d/go.mod h1:5Ky9EC2xfoUKUor0Hjgi2BJhCSXJfMOFlmyYrVKGQMk=
github.com/koron/go-ssdp v0.0.3 h1:JivLMY45N76b4p/vsWGOKewBQu6uf39y8l+AQ7sDKx8=
github.com/koron/go-ssdp v0.0.3/go.mod h1:b2MxI6yh02pKrsyNoQUsk4+YNikaGhe4894J+Q5lDvA=
github.com/korovkin/limiter v0.0.0-20230307205149-3d4b2b34c99d h1:7CfsXfFpCG1wrUpuyOzG8+vpL1ZqH2goz23wZ9pboGE=
github.com/korovkin/limiter v0.0.0-20230307205149-3d4b2b34c99d/go.mod h1:3NeYeWwAOTnDChps1fD7YGD/uWzp+tqmShgjhhMIHDM=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
Expand Down Expand Up @@ -912,8 +914,8 @@ golang.org/x/net v0.0.0-20210726213435-c6fcb2dbf985/go.mod h1:9nx3DQGgdP8bBQD5qx
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.4.0 h1:Q5QPcMlvfxFTAPV0+07Xz/MpK9NTXu2VDUuy0FeMfaU=
golang.org/x/net v0.4.0/go.mod h1:MBQ8lrhLObU/6UmLb4fmbmk5OcyYmqtbGd/9yIeKjEE=
golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
Expand Down Expand Up @@ -1023,13 +1025,13 @@ golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20221010170243-090e33056c14/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.3.0 h1:w8ZOecv6NaNa/zC8944JTU3vz4u6Lagfk4RPQxv92NQ=
golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.3.0 h1:qoo4akIqOcDME5bhc/NgxUdovd6BSS2uMsVjB56q1xI=
golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA=
golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
Expand All @@ -1038,8 +1040,8 @@ golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.5.0 h1:OLmvp0KP+FVG99Ct/qFiL/Fhk4zp4QQnZ7b2U+5piUM=
golang.org/x/text v0.5.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
Expand Down