Enable estargz-based lazy pulling on registry cache importer

[ktock]

Currently, stargz-snapshotter performs lazy pulling for estargz-fromatted base images but doesn't for estargz-formatted remote (inline) caches. This is because registry cache importer doesn't pass snapshot labels to stargz-snapshotter. This commit fixes this issue.

Since this commit, if remote cache is exported as estargz (currenly this only happens for inline caches) and stargz-snapshotter is enabled, remote caches aren't unlazied even when running commands on it.
Example of exporting estargz-formatted inline cache:

buildctl build --progress=plain --frontend=dockerfile.v0 --local context=/tmp/ctx --local dockerfile=/tmp/ctx \
               --output type=image,name=registry2-buildkit:5000/sample1/img:1,push=true,compression=estargz,force-compression=true,oci-mediatype=true \
               --export-cache type=inline

[tonistiigi]

Not a request for this PR but I noticed that this works now for the inline cache but it doesn't work for the standalone cache manifests and local cache. Although the descriptors in the cache manifest seem to have the required stargz annotations.

[tonistiigi]

@ktock Any ideas what the issue with #2648 (comment) might be? Do you think it is something simple that could be fixed for the upcoming release or something more fundamental?

[ktock]

@tonistiigi

If you use estargz image as the base image, standalone cache can be lazily pulled as well.

For example, the second build of the following performs lazy pulling for the registry cache.

# mkdir -p /tmp/ctx1
# cat <<EOF > /tmp/ctx1/Dockerfile
FROM ghcr.io/stargz-containers/ubuntu:20.04-esgz
RUN echo hello > /hello
EOF
# buildctl build --progress=plain --frontend=dockerfile.v0 --local context=/tmp/ctx1 --local dockerfile=/tmp/ctx1 \
               --output type=oci,dest=/tmp/out.tar,compression=estargz,oci-mediatype=true \
               --export-cache type=registry,ref=registry2-buildkit:5000/cache1/img:1-cache

(clear cache here)

# mkdir -p /tmp/ctx2
# cat <<EOF > /tmp/ctx2/Dockerfile
FROM ghcr.io/stargz-containers/ubuntu:20.04-esgz
RUN echo hello > /hello
RUN echo hi > /hi
EOF
# buildctl build --progress=plain --frontend=dockerfile.v0 --local context=/tmp/ctx2 --local dockerfile=/tmp/ctx2 \
               --output type=image,name=registry2-buildkit:5000/sample1/img:1,push=true,compression=estargz,oci-mediatype=true \
               --import-cache type=registry,ref=registry2-buildkit:5000/cache1/img:1-cache

Registry/local cache exporter always try to export blobs with gzip if gzip-compressed blobs exist in the cache.

buildkit/solver/llbsolver/solver.go

Line 266 in ff205f9

ResolveRemotes: workerRefResolver(cacheconfig.RefConfig{Compression: compression.New(compression.Default)}, false, g), // TODO: make configurable

So, if you use gzip base image, registry cache exporter exports the base image as gzip even if you specify force-compression=true to --output option.
I'll submit a PR to enable compression= option to --export-cache.