HA-Kubernetes mangles data when running mixed apiserver versions (upgrades)

[justinsb]

We saw an interesting edge case during a user's 1.5 -> 1.6 upgrade. The installation tool (kops) did kubectl apply with a 1.6 manifest, but the 1.6 specific fields were removed from the deployment at some stage - in particular the tolerations.

The kubectl.kubernetes.io/last-applied-configuration reflected the 1.6 fields, but the toleration fields were missing from the spec itself.

I hypothesize that a 1.5 kube-controller-manager was the leader, and it did e.g. an UpdateStatus at some stage during the update.

How can we prevent this happening? The ideas I've had are that we could either report the lowest API version across the cluster, or we could use a protobuf unknown-fields approach so that we don't mangle extra fields. Neither of those are great options.

[erictune]

Justin, can you please add a sig label. We are trying to standarize on sig rather than area labels. Or if there no identifiable SIG for it, say so.

[justinsb]

Sorry @erictune - fixed!

[bgrant0607]

Apply is kubectl.

[bgrant0607]

And I consider this a feature request rather than a bug.

cc @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-cli-feature-requests

[bgrant0607]

This is basically a manifestation of the problem discussed in #4855.

New fields shouldn't be exposed until all components are upgraded and there is no risk of rollback.

We can't store unknown fields for reasons discussed in #30819.

However, all components, but especially kubelet, really should be using patch to write status.

[justinsb]

Edit: this "crossed in the mail" with bgrant's previous reply, but I think the patch issue still holds.

So I think there are two things here:

1. kubectl apply will apply a N+1 manifest, reflect that in last-applied-configuration, but if it was applied to version N apiserver, some fields may be missing. Even after the upgrade, kubectl apply will believe all is well.
2. Mixing versions of components, even within our version skew, can lose fields that are added if a version N component updates a version N+1 object. I believe this even applies to patch commands, if the apiserver is at version N.

I am happy to split out the kubectl issue if desired. The API issue is more problematic though.

[mml]

@justinsb Looking forward to seeing a design doc for this when it's ready.
/sig archictecture

[justinsb]

@mml I think sig-apimachinery should own this; it's too fundamental an issue I believe.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[bgrant0607]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[bgrant0607]

/remove-lifecycle stale
/lifecycle frozen

[liggitt]

fyi, with the guidance in https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version (and the sweeps done in #72169 and #72651), n-1 skew among API servers will no longer drop data from beta+ fields supported in version n, as long as those fields existed in alpha level at version n