Appending /etc/hosts is triggering master reboot cycles

[wking]

Since #56, this operator has been adding content to /etc/hosts. But that file is managed by the machine-config daemon, and when the MCD detects the altered content, it reboots the node. The node comes back up, and tries to restore the expected pods, and presumably the whole cycle repeats again. This makes it harder for external clients to connect to the Kubernetes and OpenShift API servers (openshift/origin#21612), and that shows up in failed aws-e2e CI runs as random, unrelated flakes. I'm not sure what the /etc/hosts additions are for, but can you find another approach that accomplishes the same goal? Or find some way to ask the MCD to append your content, instead of reaching around the MCD and touching the file directly?

CC @abhinavdahiya, @ironcladlou, @pravisankar

[wking]

Possible resolution is for the MCD to stop caring about /etc/hosts: openshift/machine-config-operator#225.

[Miciah]

I'm not sure what the /etc/hosts additions are for, but can you find another approach that accomplishes the same goal? Or find some way to ask the MCD to append your content, instead of reaching around the MCD and touching the file directly?

We update /etc/hosts to add the registry's name and address so that the container runtime can resolve it. We considered using MCD, but we did not know a way to use it to update the file without triggering a reboot. We did not realize that MCD would trigger a reboot anyway if we modified /etc/hosts ourselves.

[wking]

We update /etc/hosts to add the registry's name and address so that the container runtime can resolve it.

Well, with openshift/machine-config-operator#225 landed, the MCD is out of the business of touching /etc/hosts (it had been injecting this). Can you take over exclusive /etc/hosts management in this operator? It seems like a reasonable fit for the DNS operator, but I'm not familiar enough to know.

[ironcladlou]

@knobunc @smarterclayton any concerns with cluster-dns-operator assuming management responsibilities for /etc/hosts for now? I don't know of any other components who need an interface.

[ironcladlou]

@wking want to close this one out as resolved by
openshift/machine-config-operator#225?

[wking]

Works for me, just so y'all know you're officially in charge of the file now ;).

[knobunc]

@wking cool. Thanks... we just have a different hack to manage the same thing... but a little more dynamically :-)