Cannot configure metrics monitoring on Beats CRD with secretName (ECK 2.5)

[scrossan]

Bug Report

What did you do?

I set up stack monitoring as per ECK docs in a central, management k8s cluster which is intended to receive log and metric data from various environmental k8s clusters (i.e. sandbox, dev, prod), and this went well. I can view monitoring data for our production cluster (ES, Kibana, and Beats) in a new monitoring cluster. In this case I am populating elasticsearchRefs using the names of Elastic resources since the production and monitoring clusters are deployed in the same k8s cluster.

I then wanted to configure stack monitoring in Beats deployed to our environmental k8s clusters to ship monitoring data to the monitoring ES cluster that had been deployed while following the above doc link, by following the docs to connect to an external monitoring cluster. Since these Beats are deployed in k8s clusters external to the one our production & monitoring ES clusters are deployed to, we have to use k8s-external URLs to connect.

I added the following snippet to Beats CRDs in these environmental k8s clusters:

spec:
  monitoring:
    logs:
      elasticsearchRefs:
      - secretName: elastic-monitoring-settings
    metrics:
      elasticsearchRefs:
      - secretName: elastic-monitoring-settings

elastic-monitoring-settings is a secret containing keys as per the example given in the doc link to connect to an external cluster, and there is no CA cert defined (since our endpoints are using certs issued by a well-known CA).

What did you expect to see?

I had expected to see Beats daemonsets being updated to deploy 2 sidecars alongside the main pod - one for Metricbeat, and the other for Filebeat - each shipping monitoring log/metric data to our monitoring cluster, just as I had observed in our production cluster when I configured stack monitoring on it using the names of Elastic resources.

What did you see instead? Under which circumstances?

First of all, there were no changes made to the Beat daemonset at all. I noted that the Beat resource was annotated with this but otherwise, nothing happens:

association.k8s.elastic.co/es-conf-4009089187: '{"authSecretName":"elastic-monitoring-settings","authSecretKey":"password","isServiceAccount":false,"caCertProvided":false,"caSecretName":"","url":"https://URL:443","version":"8.5.3"}'

While debugging I tried removing the monitoring.metrics object in the Beats spec, and I saw that the daemonset was updated and a Filebeat sidecar was deployed into the Beat pod. I verified that it was able to ship Beat logs to the monitoring cluster (the logs for the sidecar confirmed it was connected to our monitoring cluster).

I then tried removing monitoring.logs and re-adding monitoring.metrics, and ECK does not change the Beat daemonset, i.e. the Filebeat sidecar remains (since I had just had logs monitoring configured) and no Metricbeat sidecar is added into the daemonset.

If I then remove the monitoring block entirely, ECK will remove the Filebeat sidecar and the association annotation. Then, finally, if I re-add monitoring.metrics (and not monitoring.logs), then the association annotation is re-added onto the Beat resource but the daemonset is not updated and no changes are made to the running Beat pods (but a Metricbeat sidecar should be added).

It seems like monitoring.metrics does not work with an unmanaged secret containing connection details, although monitoring.logs works fine, and additionally the presence of monitoring.metrics with a secretName will prevent ECK making any other updates to the Beat.

Environment

• ECK version: 2.5.0+642f9ecd

• Kubernetes information: GKE v1.23.12-gke.1600

• Resource definition: https://gist.github.com/scrossan/59cc60fb3b10d530c9bb140bb42f2af7#file-beat-yml

• Logs: Logs for the cases I described above can be found in this gist: https://gist.github.com/scrossan/59cc60fb3b10d530c9bb140bb42f2af7

[pebrc]

I have not been able to reproduce this issue with a recent version of the ECK operator.

[scrossan]

Hey @pebrc, could you share how you tried to reproduce this?

I can reproduce it in a local Kubernetes environment (deployed using Colima), with ECK 2.6.1, trying to configure monitoring to ship to a remote monitoring cluster.

Here's the Beat spec I used: https://gist.github.com/scrossan/130c787e20f17236e92407c154bd085e

colima start --kubernetes
helm upgrade --install elastic-operator elastic/eck-operator --version 2.6.1 --set config.logVerbosity=2
kubectl apply beat.yml

If you comment out the spec.monitoring block, a Filebeat pod is deployed and ships Kubernetes pod logs to the configured ES cluster. If you then uncomment spec.monitoring.logs, it works as expected and the pod restarts with a logging sidecar. I can view Filebeat's logs in the monitoring cluster. If you then uncomment spec.monitoring.metrics, nothing happens, and the pod is not restarted with an additional metrics sidecar. That's the problem that I'm having.

[pebrc]

I was able to reproduce your issue. What I had missed originally was that you used references to external Elasticsearch and Kibana also for Beats outside of stack monitoring, which is what broke your stack monitoring for the metrics because we need a Elasticsearch cluster UUID to configure Metricbeat. I will raise a PR to fix the issue.