Allow predicates to be disabled on a case-by-case basis via annotation.

[naemono]

Closes #2092

This change will allow ECK user to disable certain upgrade predicates, by name, via an annotation on the Elasticsearch CRD, thus allowing clusters to be "forced" to be upgraded in scenarios where the default predicates (that disallow upgrade because of a set of rules) are too restrictive.

Use Case 1:

Cluster is a "red" state because of an un-assignable shard. This cluster would never be allowed to be upgraded with the standard set of upgrade predicates in place.

If the following annotation was added to the cluster

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: testing
  annotations:
    eck.k8s.elastic.co/disable-upgrade-predicates: "only_restart_healthy_node_if_green_or_yellow"
spec:
  version: 7.15.1

And the version was bumped to 7.15.2, then this cluster's nodes would be allowed to be upgraded in this scenario, even with the cluster in a "red" state.

Use Case 2:

A test Elasticsearch cluster has been setup, and regardless of any state of the cluster, a user wants to force an upgrade.

With the following annotation, and a version bump to 7.15.2, all predicates will be disabled, allowing an upgrade to proceed regardless of the state of the cluster:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: testing
  annotations:
    eck.k8s.elastic.co/disable-upgrade-predicates: "*"
spec:
  version: 7.15.1

Discussion Points

• Do we want to allow anything other than disabling one predicate, or disabling all predicates? Such as disabling 2 or more predicates specifically? The way this is designed at the moment, this wouldn't be allowed.. This limit was assumed, as there appeared to be a 256 character limit on the values of annotations, but since kubectl.kubernetes.io/last-applied-configuration clearly uses more characters than that, this assumption was wrong. A comma separated list should suffice here for > 1 predicate. I'll get this change into this PR.

Todo

• Write documentation
• Add e2e test(s)

[naemono]

run full pr build

[barkbay]

A comma separated list should suffice here for > 1 predicate. I'll get this change into this PR.

👍 I think it would be great to be able to disable more than one replica.

[barkbay]

I left a few comments but LGTM.
Since we have some documentation here I would maybe request a review from @alaudazzi

[naemono]

will merge after review by @alaudazzi, as suggested.

[alaudazzi]

Left a few suggestions in the Asciidoc file, otherwise LGTM.