Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: pull various fixes #127

Merged
merged 4 commits into from
Dec 12, 2024
Merged

*: pull various fixes #127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e2110c5
go.mod: update Prometheus dep
GiedriusS Dec 12, 2024
7599cb5
[Receive] Fix race condition when adding multiple new tenants at once…
jnyi Dec 3, 2024
ba04d87
queryfrontend: add httpgrpc server
GiedriusS Nov 15, 2024
f3ee849
thanos: fix build
GiedriusS Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
- [#7852](https://github.com/thanos-io/thanos/pull/7852) Query Frontend: pass "stats" parameter forward to queriers and fix Prometheus stats merging.
- [#7832](https://github.com/thanos-io/thanos/pull/7832) Query Frontend: Fix cache keys for dynamic split intervals.
- [#7885](https://github.com/thanos-io/thanos/pull/7885) Store: Return chunks to the pool after completing a Series call.
- [#7893](https://github.com/thanos-io/thanos/pull/7893) Sidecar: Fix retrieval of external labels for Prometheus v3.0.0.
- [#7903](https://github.com/thanos-io/thanos/pull/7903) Query: Fix panic on regex store matchers.
- [#7915](https://github.com/thanos-io/thanos/pull/7915) Store: Close block series client at the end to not reuse chunk buffer
- [#7941](https://github.com/thanos-io/thanos/pull/7941) Receive: Fix race condition when adding multiple new tenants, see [issue-7892](https://github.com/thanos-io/thanos/issues/7892).

### Added
- [#7763](https://github.com/thanos-io/thanos/pull/7763) Ruler: use native histograms for client latency metrics.
Expand Down
9 changes: 7 additions & 2 deletions cmd/thanos/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"github.com/pkg/errors"

"github.com/prometheus/common/model"
"github.com/prometheus/prometheus/model/labels"

Check failure on line 20 in cmd/thanos/config.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:

"github.com/thanos-io/thanos/pkg/extkingpin"
"github.com/thanos-io/thanos/pkg/shipper"
Expand All @@ -32,10 +32,15 @@
maxConnectionAge time.Duration
}

func (gc *grpcConfig) registerFlag(cmd extkingpin.FlagClause) *grpcConfig {
func (gc *grpcConfig) registerFlag(cmd extkingpin.FlagClause, optionalListener bool) *grpcConfig {
var addr = "0.0.0.0:10901"
if optionalListener {
addr = ""
}
cmd.Flag("grpc-address",
"Listen ip:port address for gRPC endpoints (StoreAPI). Make sure this address is routable from other components.").
Default("0.0.0.0:10901").StringVar(&gc.bindAddress)
Default(addr).StringVar(&gc.bindAddress)

cmd.Flag("grpc-server-tls-cert",
"TLS Certificate for gRPC server, leave blank to disable TLS").
Default("").StringVar(&gc.tlsSrvCert)
Expand Down
2 changes: 1 addition & 1 deletion cmd/thanos/query.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,11 @@
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/common/route"
"github.com/prometheus/prometheus/discovery"

Check failure on line 26 in cmd/thanos/query.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:
"github.com/prometheus/prometheus/discovery/file"

Check failure on line 27 in cmd/thanos/query.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:
"github.com/prometheus/prometheus/discovery/targetgroup"

Check failure on line 28 in cmd/thanos/query.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:
"github.com/prometheus/prometheus/model/labels"
"github.com/prometheus/prometheus/promql"

Check failure on line 30 in cmd/thanos/query.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:
"github.com/thanos-io/promql-engine/api"

apiv1 "github.com/thanos-io/thanos/pkg/api/query"
Expand Down Expand Up @@ -83,7 +83,7 @@
httpBindAddr, httpGracePeriod, httpTLSConfig := extkingpin.RegisterHTTPFlags(cmd)

var grpcServerConfig grpcConfig
grpcServerConfig.registerFlag(cmd)
grpcServerConfig.registerFlag(cmd, false)

secure := cmd.Flag("grpc-client-tls-secure", "Use TLS when talking to the gRPC server").Default("false").Bool()
skipVerify := cmd.Flag("grpc-client-tls-skip-verify", "Disable TLS certificate verification i.e self signed, signed by fake CA").Default("false").Bool()
Expand Down
91 changes: 66 additions & 25 deletions cmd/thanos/query_frontend.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"github.com/opentracing/opentracing-go"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/prometheus/promql/parser"

Check failure on line 19 in cmd/thanos/query_frontend.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:
"github.com/thanos-io/promql-engine/execution/parse"
"github.com/weaveworks/common/user"
"gopkg.in/yaml.v2"
Expand All @@ -34,9 +34,11 @@
"github.com/thanos-io/thanos/pkg/logging"
"github.com/thanos-io/thanos/pkg/prober"
"github.com/thanos-io/thanos/pkg/queryfrontend"
grpcserver "github.com/thanos-io/thanos/pkg/server/grpc"
httpserver "github.com/thanos-io/thanos/pkg/server/http"
"github.com/thanos-io/thanos/pkg/server/http/middleware"
"github.com/thanos-io/thanos/pkg/tenancy"
"github.com/thanos-io/thanos/pkg/tls"
"github.com/thanos-io/thanos/pkg/tracing"
)

Expand Down Expand Up @@ -67,6 +69,9 @@

cfg.http.registerFlag(cmd)

var grpcServerConfig grpcConfig
grpcServerConfig.registerFlag(cmd, true)

cmd.Flag("web.disable-cors", "Whether to disable CORS headers to be set by Thanos. By default Thanos sets CORS headers to be allowed by all.").
Default("false").BoolVar(&cfg.webDisableCORS)

Expand Down Expand Up @@ -171,7 +176,7 @@
return errors.Wrap(err, "error while parsing config for request logging")
}

return runQueryFrontend(g, logger, reg, tracer, httpLogOpts, cfg, comp)
return runQueryFrontend(g, logger, reg, tracer, httpLogOpts, cfg, comp, grpcServerConfig)
})
}

Expand Down Expand Up @@ -237,6 +242,7 @@
httpLogOpts []logging.Option,
cfg *queryFrontendConfig,
comp component.Component,
grpcServerConfig grpcConfig,
) error {
tenantHeaderProvided := cfg.TenantHeader != "" && cfg.TenantHeader != tenancy.DefaultTenantHeader
// If tenant header is set and different from the default tenant header, add it to the list of org id headers.
Expand Down Expand Up @@ -328,15 +334,43 @@
}

httpProbe := prober.NewHTTP()
grpcProbe := prober.NewGRPC()

statusProber := prober.Combine(
httpProbe,
grpcProbe,
prober.NewInstrumentation(comp, logger, extprom.WrapRegistererWithPrefix("thanos_", reg)),
)

// Configure Request Logging for HTTP calls.
logMiddleware := logging.NewHTTPServerMiddleware(logger, httpLogOpts...)
ins := extpromhttp.NewTenantInstrumentationMiddleware(cfg.TenantHeader, cfg.DefaultTenant, reg, nil)

instr := func(f http.HandlerFunc) http.HandlerFunc {
hf := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
orgId := extractOrgId(cfg, r)
name := "query-frontend"
if !cfg.webDisableCORS {
api.SetCORS(w)
}
middleware.RequestID(
tracing.HTTPMiddleware(
tracer,
name,
logger,
ins.NewHandler(
name,
logMiddleware.HTTPMiddleware(name, f),
),
// Cortex frontend middlewares require orgID.
),
).ServeHTTP(w, r.WithContext(user.InjectOrgID(r.Context(), orgId)))
})
return hf
}

instrumentedHandler := instr(handler.ServeHTTP)

// Start metrics HTTP server.
{
srv := httpserver.New(logger, reg, comp, httpProbe,
Expand All @@ -345,29 +379,7 @@
httpserver.WithTLSConfig(cfg.http.tlsConfig),
)

instr := func(f http.HandlerFunc) http.HandlerFunc {
hf := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
orgId := extractOrgId(cfg, r)
name := "query-frontend"
if !cfg.webDisableCORS {
api.SetCORS(w)
}
middleware.RequestID(
tracing.HTTPMiddleware(
tracer,
name,
logger,
ins.NewHandler(
name,
logMiddleware.HTTPMiddleware(name, f),
),
// Cortex frontend middlewares require orgID.
),
).ServeHTTP(w, r.WithContext(user.InjectOrgID(r.Context(), orgId)))
})
return hf
}
srv.Handle("/", instr(handler.ServeHTTP))
srv.Handle("/", instrumentedHandler)

g.Add(func() error {
statusProber.Healthy()
Expand All @@ -381,8 +393,37 @@
})
}

if grpcServerConfig.bindAddress != "" {
level.Info(logger).Log("msg", "starting gRPC server", "address", grpcServerConfig.bindAddress)
tlsCfg, err := tls.NewServerConfig(log.With(logger, "protocol", "gRPC"), grpcServerConfig.tlsSrvCert, grpcServerConfig.tlsSrvKey, grpcServerConfig.tlsSrvClientCA)
if err != nil {
return errors.Wrap(err, "setup gRPC server")
}

httpgrpcServer := queryfrontend.NewHTTPGRPCServer(instrumentedHandler)

s := grpcserver.New(logger, reg, tracer, nil, nil, comp, grpcProbe,
grpcserver.WithServer(queryfrontend.RegisterHTTPGRPCServer(httpgrpcServer)),
grpcserver.WithListen(grpcServerConfig.bindAddress),
grpcserver.WithGracePeriod(grpcServerConfig.gracePeriod),
grpcserver.WithMaxConnAge(grpcServerConfig.maxConnectionAge),
grpcserver.WithTLSConfig(tlsCfg),
)

g.Add(func() error {
statusProber.Ready()

return s.ListenAndServe()
}, func(error) {
statusProber.NotReady(err)
s.Shutdown(err)
})
} else {
level.Info(logger).Log("msg", "gRPC server is disabled")
statusProber.Ready()
}

level.Info(logger).Log("msg", "starting query frontend")
statusProber.Ready()
return nil
}

Expand Down
2 changes: 1 addition & 1 deletion cmd/thanos/receive.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/common/model"
"github.com/prometheus/prometheus/model/labels"
"github.com/prometheus/prometheus/model/relabel"

Check failure on line 27 in cmd/thanos/receive.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:
"github.com/prometheus/prometheus/tsdb"
"github.com/prometheus/prometheus/tsdb/wlog"
"google.golang.org/grpc"
Expand Down Expand Up @@ -895,7 +895,7 @@

func (rc *receiveConfig) registerFlag(cmd extkingpin.FlagClause) {
rc.httpBindAddr, rc.httpGracePeriod, rc.httpTLSConfig = extkingpin.RegisterHTTPFlags(cmd)
rc.grpcConfig.registerFlag(cmd)
rc.grpcConfig.registerFlag(cmd, false)
rc.storeRateLimits.RegisterFlags(cmd)

cmd.Flag("remote-write.address", "Address to listen on for remote write requests.").
Expand Down
2 changes: 1 addition & 1 deletion cmd/thanos/rule.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,13 @@
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/common/model"
"github.com/prometheus/common/route"
"github.com/prometheus/prometheus/config"

Check failure on line 31 in cmd/thanos/rule.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:
"github.com/prometheus/prometheus/model/labels"
"github.com/prometheus/prometheus/model/relabel"
"github.com/prometheus/prometheus/notifier"

Check failure on line 34 in cmd/thanos/rule.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:
"github.com/prometheus/prometheus/promql"
"github.com/prometheus/prometheus/promql/parser"
"github.com/prometheus/prometheus/rules"

Check failure on line 37 in cmd/thanos/rule.go

View workflow job for this annotation

GitHub Actions / Go build with -tags=stringlabels 

github.com/vinted/[email protected]: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/3bba655407d43cfd7c196f49d3d65584cfc10694c56e16474ff62a1333c7d575: exit status 128:
"github.com/prometheus/prometheus/storage"
"github.com/prometheus/prometheus/storage/remote"
"github.com/prometheus/prometheus/tsdb"
Expand Down Expand Up @@ -120,7 +120,7 @@

func (rc *ruleConfig) registerFlag(cmd extkingpin.FlagClause) {
rc.http.registerFlag(cmd)
rc.grpc.registerFlag(cmd)
rc.grpc.registerFlag(cmd, false)
rc.web.registerFlag(cmd)
rc.shipper.registerFlag(cmd)
rc.query.registerFlag(cmd)
Expand Down
2 changes: 1 addition & 1 deletion cmd/thanos/sidecar.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -551,7 +551,7 @@ type sidecarConfig struct {

func (sc *sidecarConfig) registerFlag(cmd extkingpin.FlagClause) {
sc.http.registerFlag(cmd)
sc.grpc.registerFlag(cmd)
sc.grpc.registerFlag(cmd, false)
sc.prometheus.registerFlag(cmd)
sc.tsdb.registerFlag(cmd)
sc.reloader.registerFlag(cmd)
Expand Down
2 changes: 1 addition & 1 deletion cmd/thanos/store.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ type storeConfig struct {

func (sc *storeConfig) registerFlag(cmd extkingpin.FlagClause) {
sc.httpConfig = *sc.httpConfig.registerFlag(cmd)
sc.grpcConfig = *sc.grpcConfig.registerFlag(cmd)
sc.grpcConfig = *sc.grpcConfig.registerFlag(cmd, false)
sc.storeRateLimits.RegisterFlags(cmd)

cmd.Flag("data-dir", "Local data directory used for caching purposes (index-header, in-mem cache items and meta.jsons). If removed, no data will be lost, just store will have to rebuild the cache. NOTE: Putting raw blocks here will not cause the store to read them. For such use cases use Prometheus + sidecar. Ignored if --no-cache-index-header option is specified.").
Expand Down
2 changes: 1 addition & 1 deletion docs/sharding.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Queries against store gateway which are touching large number of blocks (no matt

# Relabelling

Similar to [promtail](https://grafana.com/docs/loki/latest/send-data/promtail/configuration/#relabel_configs) this config follows native [Prometheus relabel-config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) syntax.
Similar to [promtail](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#relabel_configs) this config follows native [Prometheus relabel-config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) syntax.

Currently, thanos only supports the following relabel actions:

Expand Down
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ require (
github.com/gobwas/pool v0.2.1 // indirect
github.com/gobwas/ws v1.2.1 // indirect
github.com/gofrs/flock v0.8.1 // indirect
github.com/gogo/googleapis v1.4.0 // indirect
github.com/gogo/googleapis v1.4.0
github.com/google/go-cmp v0.6.0
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/pprof v0.0.0-20240827171923-fa2c70bbbfe5 // indirect
Expand Down Expand Up @@ -285,7 +285,7 @@ replace (
// Required by Cortex https://github.com/cortexproject/cortex/pull/3051.
github.com/bradfitz/gomemcache => github.com/themihai/gomemcache v0.0.0-20180902122335-24332e2d58ab

github.com/prometheus/prometheus => github.com/vinted/prometheus v1.8.2-0.20241104091239-2b97618294c8
github.com/prometheus/prometheus => github.com/vinted/prometheus v0.0.0-20241212114003-53a808bbaf43

// Pin kuberesolver/v5 to support new grpc version. Need to upgrade kuberesolver version on weaveworks/common.
github.com/sercand/kuberesolver/v4 => github.com/sercand/kuberesolver/v5 v5.1.1
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2271,8 +2271,8 @@ github.com/uber/jaeger-client-go v2.30.0+incompatible/go.mod h1:WVhlPFC8FDjOFMMW
github.com/uber/jaeger-lib v2.2.0+incompatible/go.mod h1:ComeNDZlWwrWnDv8aPp0Ba6+uUTzImX/AauajbLI56U=
github.com/uber/jaeger-lib v2.4.1+incompatible h1:td4jdvLcExb4cBISKIpHuGoVXh+dVKhn2Um6rjCsSsg=
github.com/uber/jaeger-lib v2.4.1+incompatible/go.mod h1:ComeNDZlWwrWnDv8aPp0Ba6+uUTzImX/AauajbLI56U=
github.com/vinted/prometheus v1.8.2-0.20241104091239-2b97618294c8 h1:yLIYtyb+wGxhvSDYGV7ySkXXMx+S/TQj6dlzsWMG35w=
github.com/vinted/prometheus v1.8.2-0.20241104091239-2b97618294c8/go.mod h1:n3/PY5be8xgYe+DUCjKdK0eSmDSQBQ6ZOoIUGmO9vEY=
github.com/vinted/prometheus v0.0.0-20241212114003-53a808bbaf43 h1:3PZq7phU4tTEF8It55AHuM8mAx18edD9guvlWiAHyz8=
github.com/vinted/prometheus v0.0.0-20241212114003-53a808bbaf43/go.mod h1:n3/PY5be8xgYe+DUCjKdK0eSmDSQBQ6ZOoIUGmO9vEY=
github.com/vultr/govultr/v2 v2.17.2 h1:gej/rwr91Puc/tgh+j33p/BLR16UrIPnSr+AIwYWZQs=
github.com/vultr/govultr/v2 v2.17.2/go.mod h1:ZFOKGWmgjytfyjeyAdhQlSWwTjh2ig+X49cAp50dzXI=
github.com/weaveworks/common v0.0.0-20230728070032-dd9e68f319d5 h1:nORobjToZAvi54wcuUXLq+XG2Rsr0XEizy5aHBHvqWQ=
Expand Down
Loading
Loading