Elastic Agent/Fleet container certificate update does not work for UBI images

[b2ronn]

from website instructions
https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-elastic-agent-fleet-quickstart.html

---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-quickstart
  namespace: elastic
spec:
  version: 8.0.1
  kibanaRef:
    name: kibana-quickstart
  elasticsearchRefs:
  - name: elasticsearch-quickstart
  mode: fleet
  fleetServerEnabled: true
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-quickstart
  namespace: elastic
spec:
  version: 8.0.1
  kibanaRef:
    name: kibana-quickstart
  fleetServerRef:
    name: fleet-server-quickstart
  mode: fleet
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-quickstart
  namespace: elastic
spec:
  version: 8.0.1
  count: 1
  elasticsearchRef:
    name: elasticsearch-quickstart
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-quickstart-es-http.elastic.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-quickstart-agent-http.elastic.svc:8220"]
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-quickstart
  namespace: elastic
spec:
  version: 8.0.1
  nodeSets:
  - name: elastic
    count: 1
    config:
      node.store.allow_mmap: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - pods
  - nodes
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: elastic
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
- kind: ServiceAccount
  name: elastic-agent
  namespace: elastic
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io

then
oc adm policy add-scc-to-user anyuid -z apm-server -n elastic

but the fleet-server and elastic-agent give errors
bash: line 4: update-ca-certificates: command not found
ps: OpenShift version 4.9.10
Elasticsearch (ECK) Operator 2.0.0 provided by Elastic

[pebrc]

Which version of ECK are you running?

[b2ronn]

Elasticsearch (ECK) Operator 2.0.0 provided by Elastic

[pebrc]

This is unfortunately a known issue in ECK 2.0 https://www.elastic.co/guide/en/cloud-on-k8s/2.0/release-highlights-2.0.0.html#k8s-200-known-issues

#5323 (comment) describes a workaround.

Sorry about the inconvenience. We are shipping a fix in the next release

I just realised you are running 8.0.1 where it should work. Let me try to reproduce this.

[b2ronn]

if in deployment fleet-server-quickstart-agent change
update-ca-certificates
to
update-ca-trust
then the server starts successfully.
I understand that the deployment is controlled by the operator and can be changed, but I checked it for the test.

[pebrc]

I was able to reproduce the problem. The issue is that on OCP when using the certified Red Hat operator version of ECK we are using UBI based images by default.

We added a workaround in #5268 to address a base image change in the Elastic Agent images. So as of 8.0/7.17 we are using update-ca-certificates matching the new Ubuntu image. However on OCP with the certified operator this is incorrect as UBI is RHEL/CentOS based so update-ca-trust would be the right call.

@b2ronn to work around your problem two options come to mind:

1. use the non-UBI images for Elastics Agent by setting spec.image: docker.elastic.co/beats/elastic-agent:8.0.1
2. override the container command as follows:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
...
spec:
  ...
  deployment:
    ...
    podTemplate:
      spec:
        ...
        containers:
        - name: agent
          command:
          - bash
          - -c 
          - |
            #!/usr/bin/env bash
            set -e
            if [[ -f /mnt/elastic-internal/elasticsearch-association/elastic/elasticsearch-quickstart/certs/ca.crt ]]; then
              cp /mnt/elastic-internal/elasticsearch-association/elastic/elasticsearch-quickstart/certs/ca.crt /etc/pki/ca-trust/source/anchors
              update-ca-trust
            fi
            /usr/bin/tini -- /usr/local/bin/docker-entrypoint -e