container: Use docker-archive: to stream image #22
Conversation
It turns out that for whatever reasons (presumably mostly historical) in skopeo, while `oci-archive:` is backed by just writing everything to a tempdir and then tarring it up, the `docker-archive:` format is actually streaming. And it's *way* nicer and more efficient to stream. Then the last piece of this puzzle is that I realized we can use a named pipe i.e. `mkfifo` to avoid having the containers/image stack crash because it wants to invoke `realpath()` on the provided target. By simply giving our pipe a name, it can do so (though it's pointless) but this way we work with shipped versions of skopeo and don't need to wait months for patches to propagate. Also as a bonus, here skopeo ends up decompressing the layer for us, so we don't need to e.g. handle gzip vs zstd in our code.
cgwalters
force-pushed
the
async-skopeo-fetch
branch
from
c70c41d to
3ecb887
Compare
|
OK rebased 🏄 and lifting draft! |
|
(OK actually I was wrong it seems, |
|
But to be clear I think it's still worth merging this because we at least re-gain the "incrementally parse streamed tarfile" code which is definitely closer to the desired end state. |
| // Is there a better way to do this? | ||
| let worker = async move { | ||
| let (import, input_copydriver) = tokio::join!(import, input_copydriver); | ||
| let _: () = import?.context("Import worker")?; |
There was a problem hiding this comment.
Just wondering what the purpose of these unused variables are. Is it just for adding annotations for the compiler?
There was a problem hiding this comment.
It's confusing because there's Result<Result<T>> stuff involved here and I want to be careful we're fully unwrapping these and handling the chain.
It's too easy to do e.g. let _ = foo? when it should be let _ = foo??.
There was a problem hiding this comment.
To elaborate further by doing let _: () and adding the explicit () typing we make it a compile error if the result of ? isn't ().
Depends: #20 (mostly to avoid conflicts)
container: Use docker-archive: to stream image
It turns out that for whatever reasons (presumably mostly historical)
in skopeo, while
oci-archive:is backed by just writing everythingto a tempdir and then tarring it up, the
docker-archive:formatis actually streaming.
And it's way nicer and more efficient to stream. Then the
last piece of this puzzle is that I realized we can use a named pipe
i.e.
mkfifoto avoid having the containers/image stack crashbecause it wants to invoke
realpath()on the provided target.By simply giving our pipe a name, it can do so (though it's
pointless) but this way we work with shipped versions of skopeo
and don't need to wait months for patches to propagate.
Also as a bonus, here skopeo ends up decompressing the layer for us,
so we don't need to e.g. handle gzip vs zstd in our code.