Image metadata without affecting the CAS

[vishvananda]

It would be nice to be able to add metadata to an image (or a layer) without affecting the CAS id for the image. This allows for images to be built reproducibly but stills allows for keeping useful information with the image like the build date, build software versioning, build machine, etc.

This could go into a specific directory (.meta/) inside the image that can be ignored when generating the content id, or it could be shipped in a separate filesystem structure that is outside the "data" filesystem for the image.

[vishvananda]

I note that there is some discussion in #22 about where signatures should be stored so that they don't disrupt the CAS hash as well. It seems that this is a general issue.

[vbatts]

Like the annotations field?

On Mon, May 2, 2016 at 5:03 PM Vish Ishaya [email protected] wrote:

I note that there is some discussion in #22
#22 about where
signatures should be stored so that they don't disrupt the CAS hash as
well. It seems that this is a general issue.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#59 (comment)

[wking]

On Mon, May 02, 2016 at 01:56:02PM -0700, Vish Ishaya wrote:

It would be nice to be able to add metadata to an image (or a layer)
without affecting the CAS id for the image.

I like TUF's ‘custom’ on the signed name ↔ hash assertion 1. You
could use that for, among other things, expiration timestamps 2.

[vishvananda]

@vbatts interesting, I missed that addition. It depends on whether the json for the image is part of the CAS or lives separately. Does the image itself have a reproducible ID or is that outside the scope of the spec? I suppose annotations could work if the image ID were generated by filtering out the annotations field and taking a sha of the json file.

My goal would be to have two different build machines building an image from the same source, throwing their own annotations in, but still producing the same image id.

[wking]

On Mon, May 02, 2016 at 02:13:15PM -0700, Vish Ishaya wrote:
“I suppose annotations could work if the image ID were generated by
filtering out the annotations field and taking a sha of the json
file.”

^ this makes annotations unsignable and breaks content-addressability
for the image, so I'd rather not ;).

“My goal would be to have two different build machines building an
image from the same source, throwing their own annotations in, but
still producing the same image id.”

This is going to be hard. For one thing, you'd need to use
compatible, stable tar and JSON implementations. But it would be cool
if you got it working [1](assuming your mutated annotations were
living above the image level of the CAS hierarchy).

 Subject: Re: OCI Bundle Digests Summary
 Date: Thu, 15 Oct 2015 16:52:42 -0700
 Message-ID: <[email protected]>

[vishvananda]

@wking I currently have tar and json producing stable results for a custom build system so I'm not so worried there. I understand the desire to keep everything content-addressible, but it is also important to have build metadata available. In this case I would think that annotations are not the right place for this information because it is useful at the very least to determine that multiple builds have produced the same manifest.json, and for image ids to remain stable if nothing has changed.

Maybe what we actually want is for build servers to be able to sign existing manifests and put their metadata somewhere else. If there was a way to search these metadata signatures, you could potentially see all build servers that produced the same image along with their metadata.

[vishvananda]

I think the ideal solution is essentially something akin to a git annotated tag. Each build server could create a signed tag with their build metadata. Then there could be some way to find all annotated tags for a given image id.

Then a user could download an image based on the sha of the tag (which would be unique per build) or the sha of the image id. The runtime (docker, rkt, etc.) could resolve tags to the sha of the underlying image for display purposes so it would be easy to determine if multiple machines are running the same image even if they downloaded different "tags"

[wking]

On Mon, May 02, 2016 at 03:14:41PM -0700, Vish Ishaya wrote:

I think the ideal solution is essentially something akin to a git
annotated tag.

This sounds like “custom metadata in the signed name ↔ hash assertion”
1 to me, so +1.

[philips]

@vishvananda Happy to consider a concrete proposal. What is the use case?

[stevvooe]

Complex hash funnels typically lead to fragile and insecure behavior. It makes implementations fragile and creates an opening for unverified data to creep through. Anything that requires pre-processing before calculating a hash is suspect.

The right approach to handling ephemeral metadata is to appropriately layer it. Only change fields at the level that corresponds to the build. For example, rather than pushing build version to the image config, keep it at the tag-level.

An annotated tag-like object is probably the right balance. The key here is keeping the lowest layers (image config, layer file) as simple as possible then tag these hashed artifacts with metadata.

[philips]

@stevvooe Yep, +1 on all points.

I would be OK with considering the schema for this "external" metadata in this repo. So, if @vishvananda comes up with a good use case and wants to drive the work I am OK with it.

[philips]

@vishvananda Want to try and start on something next week so we can consider it for v0.2.0?

[vishvananda]

My current thinking is that this should be merged with the idea of signing. And that signed objects are canonical json that live in the cas as a reference to another object. Something along the lines of:

{ "signature": {...}, "metadata": {...}, "object": "some-cas-image-id" }

The runtime software (rkt, docker) can choose to treat this as a "tag" (in the sense of git annotated tags) on the underlying object. The object in this case would be the image json, but other objects could theoretically be signed as well.

[wking]

On Mon, May 09, 2016 at 09:37:21AM -0700, Vish Ishaya wrote:

The runtime software (rkt, docker) can choose to treat this as a
"tag" (in the sense of git annotated tags)…

A crucial piece of annotated tags that your JSON sketch misses is the
name being assigned. In 1 I have a similar sketch that includes a
‘name’ property, which allows you to say “I assert that the manifest
at sha256:ca6e… is ‘debian 8.0’”.

[vishvananda]

@wking agreed. Both "name" and "signature" could be embedded in metadata and interpreted by tools, but I think it makes sense to treat those two as special and split them out of the generic metadata.

[philips]

I am putting this as a post-v1.0.0 milestone as it feels like a nice to have with other dependencies like #22.

[philips]

cc @gtank

[vbatts]

@vishvananda I think this can be down now with the image-layout, and an index (old manifest-list). I'm inclined to close this issue out.

[vishvananda]

Yeah custom metadata in the index should work. That was my plan. I haven't tested it but I don't see any reason. Why it shouldn't.

…

On Thu, Mar 9, 2017, 8:38 AM Vincent Batts ***@***.***> wrote: @vishvananda <https://github.com/vishvananda> I think this can be down now with the image-layout, and an index (old manifest-list). I'm inclined to close this issue out. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#59 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIrjmzMDdLgvh9TV-V9V5-EWrXghKTTks5rkCrzgaJpZM4IVtPV> .

[vbatts]

fixed by #533