Reconciliation error upgrading enterprise search with TLS disabled

[rmoesbergen]

Bug Report

What did you do?
I'm trying to upgrade enterprise search from 8.3.3 to 8.4.3 using the ECK operator. The cluster runs fine with version 8.3.3, but the operator refuses to upgrade complaining about a missing secret. The cluster has TLS disabled, using this guide: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-service-mesh-istio.html

What did you expect to see?
An upgraded enterprise search deployment.

What did you see instead? Under which circumstances?

The controller errors out with a 'Reconciliation failure', see the logs below.
I suspect this is because I have disabled TLS for both elasticsearch and enterprise search in favor of a cluster-wide istio installation.

Environment

• ECK version: 2.5.0

• Kubernetes information:

We use GKE, version 1.23.

$ kubectl version --short
Client Version: v1.25.4
Kustomize Version: v4.5.7
Server Version: v1.23.12-gke.100

• Resource definition:

The appsearch cluster:

apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: ens
  namespace: elastic
spec:
  config:
    elasticsearch.ssl.enabled: false
    ent_search.external_url: https://ens.<something>
    kibana.external_url: https://kibana.<something>
    kibana.host: http://kibana-kb-http:5601
  count: 2
  elasticsearchRef:
    name: my-elastic
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  version: 8.4.3

And the elasticsearch cluster:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: my-elastic
  namespace: elastic
spec:
  http:
    tls:
      certificate: {}
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - count: 2
    name: default
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300"
          traffic.sidecar.istio.io/includeInboundPorts: '*'
  transport:
    tls:
      certificate: {}
  version: 8.4.3

• Logs:

{"log.level":"info","@timestamp":"2022-11-21T07:56:36.123Z","log.logger":"enterprisesearch-controller","message":"Starting reconciliation run","service.version":"2.5.0+642f9ecd","service.type":"eck","ecs.version":"1.4.0","iteration":"37","namespace":"elastic","ent_name":"ens"}
{"log.level":"info","@timestamp":"2022-11-21T07:56:36.124Z","log.logger":"enterprisesearch-controller","message":"Enabling read-only mode for version upgrade","service.version":"2.5.0+642f9ecd","service.type":"eck","ecs.version":"1.4.0","iteration":"37","namespace":"elastic","ent_name":"ens","namespace":"elastic","ent_name":"ens","target_version":"8.4.3"}
{"log.level":"info","@timestamp":"2022-11-21T07:56:36.124Z","log.logger":"enterprisesearch-controller","message":"Ending reconciliation run","service.version":"2.5.0+642f9ecd","service.type":"eck","ecs.version":"1.4.0","iteration":"37","namespace":"elastic","ent_name":"ens","took":0.000714316}
{"log.level":"error","@timestamp":"2022-11-21T07:56:36.124Z","log.logger":"manager.eck-operator","message":"Reconciler error","service.version":"2.5.0+642f9ecd","service.type":"eck","ecs.version":"1.4.0","controller":"enterprisesearch-controller","object":{"name":"ens","namespace":"elastic"},"namespace":"elastic","name":"ens","reconcileID":"dd2c8f1d-d3da-46c9-b20f-a66ef1743ebf","error":"version upgrade: Secret \"ens-ent-http-certs-internal\" not found","errorCauses":[{"error":"version upgrade: Secret \"ens-ent-http-certs-internal\" not found"}],"error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}

[thbkrkr]

Apologies, this is a bug in ECK.

During an EnterpriseSearch upgrade, ECK calls its API to set the read-only mode setting and we missed adjusting the HTTP client used for this call based on whether or not TLS is enabled.

cloud-on-k8s/pkg/controller/enterprisesearch/version_upgrade.go

Lines 167 to 176 in 1509894

	func (r *VersionUpgrade) setReadOnlyMode(ctx context.Context, enabled bool) error {
	httpClient := r.httpClient
	if httpClient == nil {
	// build an HTTP client to reach the Enterprise Search service
	tlsCerts, err := r.retrieveTLSCerts()
	if err != nil {
	return err
	}
	httpClient = apmhttp.WrapClient(
	commonhttp.Client(r.dialer, tlsCerts, 0),

[rmoesbergen]

Thanks, good to know it's not my fault :) Looking forward to the fix, and thanks for looking into this!

[thbkrkr]

A workaround for you may be to enable TLS, upgrade and disable TLS again.

[rmoesbergen]

I'm afraid that's not feasible since it would throw Elasticsearch off the service-mesh and cause a lot of downtime. I'll just wait for the fix to be released.