design clean support for "image pull inside kubernetes"

[cgwalters]

Breaking this issue out from discussion in openshift/os#657

We need to make it easy for a Kubernetes node to pull container images for host OS updates in the same way kubelet is doing so. In particular, I want to support pulling from e.g. an in-cluster registry.

There are three major aspects to this:

• Configuring pull secrets; this can be done today by injecting REGISTRY_AUTH_FILE in the environment. We should also add CLI flags that just wrap the global skopeo --authfile argument.
• Ensuring that we trust the service CA
• Joining the pod network - we need to use cluster DNS at least. Digging into things, it seems at least crio has special support for forking a copy of its binary into a designated cgroup. Presumably this aids applying separate cpu/network/io limits to image pulls.

[cgwalters]

My strawman would be that we expose in this library something like:

pub struct ImagePullConfiguration {
  pub auth_file: Option<String>, // maps to --authfile
  pub certificate_authority: Option<?>,  // maps to `--src-cert-dir`
  pub network_config: ???,
  pub systemd_config: Option<Vec<String>>, // direct options passed to `systemd-run`, see also https://github.com/coreos/rpm-ostree/blob/main/rust/src/isolation.rs
}

or so, and then implement it by extending how we fork skopeo.

While we're here, I actually also really want to support systemd-run skopeo when run as root so we can use e.g. -p DynamicUser=yes to completely drop privileges.

[cgwalters]

containers-image-proxy now has a config struct.

It's still missing the certificate bits though, and that only allows us to configure for pulls from inside a cluster, it doesn't actually demonstrate it working.

[mkenigs]

Trying to understand the conclusion about networking from openshift/os#657 - will we need to do anything special for networking? After debugging a node and oc logging in I can run skopeo inspect docker://image-registry.openshift-image-registry.svc:5000/openshift/cli --username kubeadmin --password $(oc whoami -t) So I'm assuming it'd also work with just --authfile?

Or do we need to fork into a cgroup for resource limits?

[mkenigs]

Would we want to copy or factor out somewhere the systemd-run code from https://github.com/coreos/rpm-ostree/blob/main/rust/src/isolation.rs?

[cgwalters]

Sorry, information on this is split across several places. It turns out that I was confused by DNS - openshift/os#657 (comment)

There is no need to switch to a separate network namespace, we can contact the kube service IP addresses from the host. It's just DNS that is special cased.

Or to state this another way, I think the only remaining blocker for the "pull host updates from internal registry" is correctly setting up the certificates.

[mkenigs]

Do you still want to add the systemd-run support here or do that separately?

[cgwalters]

The basic support for that landed in containers/containers-image-proxy-rs#15 and I tried to use it in coreos/rpm-ostree#3239 but it's sadly blocking on selinux right now 😢

[cgwalters]

One thing that came out of a discussion on this is that kubelet doesn't use its own service account to pull containers or anything like that - the credentials are per-pod. The pod can have its own credentials to pull. See https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

Hence, it likely makes sense to use the MCD's service account.

See also https://docs.openshift.com/container-platform/4.9/registry/accessing-the-registry.html

In some recovery scenarios, it'd be nice to streamline using the kube:admin's credentials when logged into a node via ssh or so. Think e.g. hacking on a node to install a debug kernel via rpm-ostree rebase directly.

--

https://docs.openshift.com/container-platform/4.9/authentication/bound-service-account-tokens.html

For internal registry: https://github.com/openshift/cluster-image-registry-operator/blob/2b82360000454e7e02b6f3b46d1354546dffccbc/bindata/nodecadaemon.yaml
(Yet another load-bearing bash script run as root!)

[mkenigs]

Is this correct then?

• Add an ImagePullSecret to MCD service account (https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account)
• Inside the MCD pod, use the MCD service account token and kubectl proxy to get the ImagePullSecret from the Kubernetes API (https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#using-kubectl-proxy)
• Copy the pull secret to something like /rootfs/etc/machine-config-daemon/authfile
• Run rpm-ostree rebase --authfile /etc/machine-config-daemon/authfile

[cgwalters]

I think but need to verify that we should already have a pull secret associated with the pod.

[cgwalters]

Calling this one fixed! We had the relevant test pass in openshift/machine-config-operator#2886