Implement mTLS resources for CloudWatch Agent client

[musa-asad]

Description of the issue

The Helm charts for the CloudWatch Agent do not include a server-side Certificate Authority, client certificate, and client key when the CloudWatch Agent behaves as a client for mutual TLS (mTLS). Without these mTLS resources, the agent cannot properly connect to the Target Allocator server, as it lacks the required certificates and keys to authenticate the mTLS connection.

Description of changes

Important

Co-PRs: 1) aws/amazon-cloudwatch-agent#1510; 2) aws/amazon-cloudwatch-agent-operator#284

• Created server-side Certificate Authority in certmanager.yaml.
• Created Secret in the certmanager.yaml containing the CloudWatch Agent client certificate, key, and CA certificate, for mutual TLS authentication.
• Used genSignedCert to generate the CloudWatch Agent client certificate and key.
• Created Secret containing the CloudWatch Agent client certificate and key, for use by the agent.
• Added volume mount for the CloudWatch Agent client certificate and key in the custom resource.
• Added volume for the CloudWatch Agent client certificate and key in the custom resource.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

1. Created EKS cluster using custom agent (Add default TLS client cert and key paths for Prometheus input and receiver aws/amazon-cloudwatch-agent#1510), operator, and target allocator images (Implement mTLS resources and configuration for Target Allocator server aws/amazon-cloudwatch-agent-operator#284).
2. Ran helm upgrade --install --debug amazon-cloudwatch-observability helm-charts/charts/amazon-cloudwatch-observability --set clusterName=<cluster_name> --set region=us-west-2 --namespace amazon-cloudwatch --create-namespace and edited values.yaml with a custom agent and prometheus configuration.

Secrets

Volumes

Volume Mounts

[lisguo]

is there a reason why we can't use "amazon-cloudwatch-observability-agent-cert" ?

I see that we specify the target allocator service as a dns name we can make calls to?

Are we using this outbound cert specifically for target allocator?

[musa-asad]

We do use "amazon-cloudwatch-observability-agent-cert" for the server cert and key for the TA server: https://github.com/aws/amazon-cloudwatch-agent-operator/blob/d6c5c6d9aa983222ac3557fe1c3bb2c367fff615/internal/manifests/targetallocator/volume.go#L47.

The "amazon-cloudwatch-observability-agent-outbound-cert" is for the client cert and key for the agent client. We could technically use "amazon-cloudwatch-observability-agent-server-cert", but I avoided doing so due to separation of concerns.

[okankoAMZ]

Could you elaborate on what you meant by

I avoided doing so due to separation of concerns.

[musa-asad]

The server-cert is used to generate server cert and keys when the CloudWatch Agent operates as a server for FluentBit.

The outbound-cert is used to generate client cert and keys when the CloudWatch Agent operates as a client for the Target Allocator.

We are able to use the server-cert to generate client cert and keys as well when the CloudWatch Agent operates as a client, but it's preferable to separate that functionality to the outbound-cert.

[musa-asad]

Updated the naming from outbound-cert to ta-client-cert.