Stop including debugging utilities in the Conduit images

[briansmith]

Let's remove the debugging utilities from all the Conduit Docker images.

Currently we have Dockerfile-base that is based on jessie-slim and which installs the following utilities:

• curl
• dnsutils
• iptables
• jq
• nghttp2

We do this because we want to help with debugging networking issues. However:

• In Kubernetes, all the containers in the same pod share a networking namespace. This means that it is possible to have a separate debugging container that, when injected into the pod, gets the same network access. This means that when we need to do debugging, we can "just" inject this kind of base container into the pod and then re-roll the pod. We don't need these tools to be in every pod by default.
• Conduit is supposed to be lightweight. Primarily we're interested in memory and networking overhead, but the size of the containers matters too. Removing these from the container images will reduce the container sizes, making Conduit even more lightweight.
• Having utilities like curl around inside the containers increases the capabilities of anybody to pivot (etc.) after exploiting any vulnerability that allows code execution. Removing these utilities from the containers will make it harder to do terrible things post-exploitation.
• For security reasons, we are going to lock down the capabilities of the containers such that it is unlikely that these utilities are going to work in the containers anyway. In particular, none of our containers except the proxy init container should be able to spawn processes in the first place. Thus the utilities won't work
• We need to optimize the docker build process to make development ergonomics better.

/cc @klingerf @olix0r

[olix0r]

Conduit is supposed to be lightweight. Primarily we're interested in memory and networking overhead, but the size of the containers matters too. Removing these from the container images will reduce the container sizes, making Conduit even more lightweight.

I'm not swayed by this. Once this image is downloaded, it should basically never change and there only needs to be one copy per host. And we're practically talking about... a few mb? I agree smaller is better, but not absolutely.

We need to optimize the docker build process to make development ergonomics better.

The utilities you are proposing to remove are absolutely essential to debugging during development. We'll need to maintain an image like this one way or another.

---

While we ultimately need to build more of these debugging capabilities into the proxy itself, we need a solid debugging story continually.

Though, I agree that from the security we should minimize the surface area of these containers, at least in release builds. But even in release environments, we need to provide a predictable environment where users can diagnose issues the way we would.

Perhaps we could do something like the following:

• maintain a diagnostic container image; initially this is just base, but later it could have diagnostic tools (i.e. that wrap grpc bindings);
• add a flag to inject and install to add the diagnostic container image;
• add iptables to proxy-init directly

The downside to this approach is that one has to know beforehand that they want to debug something. When you're actually trying to attach to something that's broken, it can be a real burden to get debug tools into that environment in that moment.

Another option would be that we could use the RUNTIME_IMAGE build-arg that proxy/Dockerfile uses to change the base image for releases -- but this isn't particularly good for non-developers.

[briansmith]

The downside to this approach is that one has to know beforehand that they want to debug something. When you're actually trying to attach to something that's broken, it can be a real burden to get debug tools into that environment in that moment.

Yes, that's the big downside.

Note that the debugging of pods without any utlities like this is
kubernetes/community#649 and will be supported via kubectl debug -c debug-shell --image=debian target-pod -- bash in Kubernetes in the (near?) future.

The main issue I am trying to solve is protecting the private keys for TLS, by locking down the proxy container. In particular we don't want to have an exploit of the proxy using curl to send the TLS private key to some external system. Basically curl and similar tools are a huge liability.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.