net/http: support content negotiation

[kevinburke]

Content negotiation is, roughly, the process of figuring out what content type the response should take, based on an Accept header present in the request.

An example might be an image server that figures out which image format to send to the client, or an API that wants to return HTML to browsers but JSON to command line clients.

It's tricky to get right because the client may accept multiple content types, and the server may have multiple types available, and it can be difficult to match these correctly. I think this is a good fit for Go standard (or adjacent) libraries because:

• there's a formal specification for how it should behave: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
• it's annoying to implement yourself, correctly; you have to write a mini-parser.
• it would take one function to implement, which makes it annoying to import an entire third party library for (assuming you find the right one)

I've seen people hack around this in various ways:

• checking whether the Accept header contains a certain string
• checking for the first matching value,
• returning different content types based on the User-Agent
• requiring different URI's to get different content.

There's a sample implementation here with a pretty good API: https://godoc.org/github.com/golang/gddo/httputil#NegotiateContentType

// NegotiateContentType returns the best offered content type for the request's
// Accept header. If two offers match with equal weight, then the more specific
// offer is preferred.  For example, text/* trumps */*. If two offers match
// with equal weight and specificity, then the offer earlier in the list is
// preferred. If no offers match, then defaultOffer is returned.
func NegotiateContentType(r *http.Request, offers []string, defaultOffer string) string

offers are content-types that the server can respond with, and can include wildcards like text/* or */*. defaultOffer is the default content type, if nothing in offers matches. The returned value never has wildcards.

So you'd call it with something like

availableTypes := []string{"application/json", "text/plain", "text/html", "text/*"}
ctype: = NegotiateContentType(req, availableTypes, "application/json")

If the first value in offers is text/* and the client requests text/plain, NegotiateContentType will return text/plain. This is why you have to have a default - you can't just return the first value in offers because it might include a wildcard.

In terms of where it could live, I'm guessing that net/http is frozen at this point. Maybe one of the packages in x/net would be a good fit? There is also a similar function for parsing Accept-Language headers in x/text/language. Open to ideas.

[quentinmit]

Funny you should mention this, I was just looking for this for use in x/perf. Your proposed API is reasonable, but you should explicitly document what happens in exceptional circumstances:

• What happens when multiple content types end up with the same q? I think I would expect the first one in offers to be chosen.
• What happen if the Accept header(s) cannot be parsed?

[kevinburke]

What happens when multiple content types end up with the same q? I think I would expect the first one in offers to be chosen.

That sounds reasonable to me.

What happen if the Accept header(s) cannot be parsed?

The spec includes a grammar but doesn't really say what happens if you can't match it, other than you should return a 406 if you can't match anything. I think we should just return the default?

[kevinburke]

@bradfitz suggested (if we want to go forward with this proposal) it could live in net/http/httputil.

[bradfitz]

I also mentioned that it seems like defaultOffer string could go away and we say that the first element in the slice was the default.

[kevinburke]

Sorry, I tried to edit the description to cover that case.

I also mentioned that it seems like defaultOffer string could go away and we say that the first element in the slice was the default.

I think the problem with this would be, if you had "text/*" as the first element in offers, it would match if the client sent text/plain and text/plain would be returned from the function.

If we return the first value in offers as the default, then the response value would contain a wildcard ("text/*"), which it doesn't in any other circumstance. Maybe that's okay.

[jimmyfrasche]

@kevinburke What's wrong with returning text/plain if the first offer is text/*? I don't see why it has to return the exact string passed in—don't you need to know the negotiated type to set the header in the response?

Also, if negotiation fails does it return the empty string? I'd rather have an error to distinguish between expected failure, negotiation failed, and unexpected failure, bad request.

[bradfitz]

@kevinburke, any response to @jimmyfrasche? This seems fine if the simplified signature works. You want to prepare a CL?

[bradfitz]

Also, we like to make sure we append "; charset=utf-8" to Content-Types to be explicit. Can we make sure that that's still automatic?

[cellfish]

offers are content-types that the server can respond with, and can include wildcards like text/* or */*

I don't think wildcards should be allowed in the offers. The server knows exactly what content types it supports and returning a wildcard content type seems weird. Nor do I think the RFC says wildcards are OK in the content-type header.
Content-Type (https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17) mentions media-type (https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7) that does not have wild cards.
Accept (https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1) mentions media-range which includes wild cards.

I agree with @jimmyfrasche that returning an error seems better so that the caller can know the reason if a default (first offered) is returned. maybe even not return a default at all. If there is a default the caller can handle that rather than build it as part of this method. For example there is a difference between not finding a match in an existing accept header and when the accept header is missing IMO.

Since the RFC for Accept supports multiple parameters (such as charset) how about returning something more than just a string like:

type NegotiatedContentType struct {
   ContentType string
   Params map[string]string
}

func NegotiateContentType(r *http.Request, offers []string) (NegotiatedContentType, error)

[bradfitz]

@kevinburke, what's the status here?

It can also live in x/net if you want.

It's probably too late for Go 1.10, though.

[kevinburke]

It's too late and I'm buried in work stuff unfortunately, I probably won't have time to try to offer an implementation. I remember when I tried to implement it I ran into non-trivial problems with the design and some kinds of inputs.

[bwplotka]

👋🏽 We (Prometheus team) are finding even more use cases for a common content negotiation logic, so instead of providing local one for Prometheus, I went ahead and proposed something as a x/net/http/httpcontent package here.

Discussion here was quite rich and useful (thanks for so many ideas and implementations!), so I attempted to summarize all findings and aspects in the design doc. Feel free to give feedback in CL, doc comment or here!

It felt like the one of the main debates here was if we should do the generic string based or rich structured signature. I went for something that (hopefully) has the most chances to end up in the standard non-experimental package one day, so string-based. I also attempted to make one function that supports all "Accept" like headers that follow RFC 2616 Accept format. Trade-offs are discussed here.

Proposed implementation (https://go-review.googlesource.com/c/net/+/642956):

// Negotiate returns the best offered content for the provided accepted values.
// Accept values is expected to follow the Accept header format as defined in
// https://datatracker.ietf.org/doc/html/rfc2616#section-14.1. Negotiate is expected
// to also support similar Accept-* HTTP 1.1. headers like -Charset, -Encoding
// -Language, etc., as long as they follow a similar (even if simplified) format.
//
// If two offers match with equal quality factor, then the more specific
// offer is preferred.  For example, text/* trumps */*. If two offers match
// with equal weight and specificity, then the offer earlier in the list is
// preferred. If no offers match, then the empty string is returned.
func Negotiate(accepts []string, offers []string) string

Example use:

// On client side:
r, _ := http.NewRequest("", "http://example.com", nil)
r.Header.Set("Accept", "text/*;q=0.3, text/html;q=0.7, text/html;level=1, text/html;level=2;q=0.4, */*;q=0.5")

// In your ServeHTTP handler:
negotiated := httpcontent.Negotiate(r.Header.Values("Accept"), []string{"text/html", "text/html;level=1"})
if negotiated == "" {
   // Can't agree on the content, return 415 or try default content type...
}

fmt.Println(negotiated)

// Output:
//  text/html;level=1

[gopherbot]

Change https://go.dev/cl/642956 mentions this issue: x/net/http/httpcontent: Add content Negotiate function.

[neild]

This proposal was accepted some years ago. We have a proposed implementation now in https://go.dev/cl/642956.

I'm not sure which package the accepted proposal was adding the new function to. The issue title references "x/net/http" (which does not exist), #19307 (comment) mentions "net/http/httputil" (which does, but we kind of regret it these days).

As I understand our current stance on x/ repo packages, we are trying to avoid adding new x/ packages for public use that don't have a path to inclusion in the standard library. If it's worth it for the Go project to provide and support a package, we should do so in std.

Given that, I see a few possibilities here:

1. Add the NegotiateContent function to net/http. The net/http package already has too large an API surface, but this function stands alone and doesn't complicate the rest of the package.
2. Add the NegotiateContent function to net/http/httputil. I'm not a fan of this option, since we've generally settled on ...util packages as a grab-bag of miscellaneous functionality being a bad idea. httputil is mostly used these days for httputil.ReverseProxy, and we really should move that to a better-named package of its own.
3. Add a net/http/httpcontent package. This is the cleanest separation, and provides a good place to add a matcher type (as proposed in net/http: support content negotiation #19307 (comment)) if we want one later. The bar for new packages in std is generally pretty high, though.
4. Add it in golang.org/x/net/http/httpcontent, my comments above about x/ repos notwithstanding.
5. Unaccept the proposal and decide that this should be a third-party package.

I don't really have any strong opinion here (other than I don't like putting it in net/http/httputil).

[mitar]

I would just go with net/http. When I was learning Go it was much easier for me just to go through all functions in the package to learn what is there. Only later I learned about httputil and x packages.

[seankhliao]

would putting it under mime be out of place considering it operates on mime types?

[neild]

It's actually more general than MIME types; RFC 9110 defines four headers (Accept, Accept-Charset, Accept-Encoding, and Accept-Language) that all use the same negotiation algorithm. And the negotiation is part of the HTTP spec rather than the MIME spec.

Given that these headers are defined in the core HTTP RFCs, perhaps net/http is the best place. "If it's in RFC 9110, it belongs in net/http (if we support it at all)" is fairly principled.

[neild]

Requesting re-review by proposal committee as a final check, given the time since this was accepted.

Specific proposal:

We add a function to net/http which performs content type negotiation as specified by RFC 9110:

// NegotiateContent returns the best content to offer from a set of possible
// values, based on the preferences represented by the accept values.
// For example, NegotiateContent can be used on the HTTP servers to find the
// best "Content-Type" to provide to HTTP user agents, based on the "Accept"
// request header. This is also known as a proactive negotiation (or "server-driven"
// negotiation).
//
// NegotiateContent may be used to negotiate several fields, e.g.:
//
//	Response field    Request header
//	--------------    --------------
//	Content-Type      Accept
//	Content-Charset 	Accept-Charset
//	Content-Encoding  Accept-Encoding
//	Content-Language  Accept-Language
//
// The accepts parameter is the appropriate request header value(s),
// and is interpreted as in RFC 9110 section 12.
//
// The offers parameter is a list of possible values to offer.
//
// If no offers match, it returns the empty string.
// If more than one offer matches with equal weight and specificity,
// it returns the one earliest in the list.
// If no accept values are provided (representing no preference case), and offers
// are non-empty, the first offer is returned.
//
// NegotiateContent only considers the first 32 accept values to
// avoid DOS attacks.
func NegotiateContent(accepts []string, offers []string) string

The function signature varies slightly from the original proposal above. In particular, the offers may not include wildcards and there is no default offer. The lack of wildcards is because the server presumably knows the complete set of content types it can offer, and the lack of a default offer is because it is unnecessary--NegotiateContent returns the best offer, or the empty string if none are suitable.

I propose to add the function to net/http, on the grounds that content type negotiation is specified as a core HTTP feature in RFC 9110 and there doesn't seem to be a better place for it. (See above discussion for some alternatives.)

[rsc]

Pushing back into the proposal process for a quick check.

[seankhliao]

is the []string in accepts a single value per string, or possibly multiple values joined as in a single header line?

[rsc]

Have all remaining concerns about this proposal been addressed?

We add a function to net/http which performs content type negotiation as specified by RFC 9110:

// NegotiateContent returns the best content to offer from a set of possible
// values, based on the preferences represented by the accept values.
// For example, NegotiateContent can be used on the HTTP servers to find the
// best "Content-Type" to provide to HTTP user agents, based on the "Accept"
// request header. This is also known as a proactive negotiation (or "server-driven"
// negotiation).
//
// NegotiateContent may be used to negotiate several fields, e.g.:
//
//	Response field    Request header
//	--------------    --------------
//	Content-Type      Accept
//	Content-Charset 	Accept-Charset
//	Content-Encoding  Accept-Encoding
//	Content-Language  Accept-Language
//
// The accepts parameter is the appropriate request header value(s),
// and is interpreted as in RFC 9110 section 12.
//
// The offers parameter is a list of possible values to offer.
//
// If no offers match, it returns the empty string.
// If more than one offer matches with equal weight and specificity,
// it returns the one earliest in the list.
// If no accept values are provided (representing no preference case), and offers
// are non-empty, the first offer is returned.
//
// NegotiateContent only considers the first 32 accept values to
// avoid DOS attacks.
func NegotiateContent(accepts []string, offers []string) string

The function signature varies slightly from the original proposal above. In particular, the offers may not include wildcards and there is no default offer. The lack of wildcards is because the server presumably knows the complete set of content types it can offer, and the lack of a default offer is because it is unnecessary--NegotiateContent returns the best offer, or the empty string if none are suitable.

I propose to add the function to net/http, on the grounds that content type negotiation is specified as a core HTTP feature in RFC 9110 and there doesn't seem to be a better place for it. (See above discussion for some alternatives.)

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[mitar]

Just to make sure if I get it right, the proposal is then closer to gddo's NegotiateContentEncoding, only that it does not take request but the list? (And not NegotiateContentType.) I think this is great.

[neild]

I haven't looked at gddo's function in detail, but based on the docs I think the differences are that the proposed NegotiateContent:

• takes a []string rather than a *Request, so you can use it with various headers
• does not have a defaultOffer parameter, using offers[0] as the default
• does not support wildcards in the offers list, because they don't seem useful (you can't offer text/*, you need to pick a specific content type)

[mitar]

does not support wildcards in the

That is for NegotiateContentType. Not its NegotiateContentEncoding.

But I otherwise agree with your analysis. Thanks.

[aclements]

Based on the discussion above, this proposal seems like a likely accept.
— aclements for the proposal review group

We add a function to net/http which performs content type negotiation as specified by RFC 9110:

// NegotiateContent returns the best content to offer from a set of possible
// values, based on the preferences represented by the accept values.
// For example, NegotiateContent can be used on the HTTP servers to find the
// best "Content-Type" to provide to HTTP user agents, based on the "Accept"
// request header. This is also known as a proactive negotiation (or "server-driven"
// negotiation).
//
// NegotiateContent may be used to negotiate several fields, e.g.:
//
//	Response field    Request header
//	--------------    --------------
//	Content-Type      Accept
//	Content-Charset 	Accept-Charset
//	Content-Encoding  Accept-Encoding
//	Content-Language  Accept-Language
//
// The accepts parameter is the appropriate request header value(s),
// and is interpreted as in RFC 9110 section 12.
//
// The offers parameter is a list of possible values to offer.
//
// If no offers match, it returns the empty string.
// If more than one offer matches with equal weight and specificity,
// it returns the one earliest in the list.
// If no accept values are provided (representing no preference case), and offers
// are non-empty, the first offer is returned.
//
// NegotiateContent only considers the first 32 accept values to
// avoid DOS attacks.
func NegotiateContent(accepts []string, offers []string) string

The function signature varies slightly from the original proposal above. In particular, the offers may not include wildcards and there is no default offer. The lack of wildcards is because the server presumably knows the complete set of content types it can offer, and the lack of a default offer is because it is unnecessary--NegotiateContent returns the best offer, or the empty string if none are suitable.

I propose to add the function to net/http, on the grounds that content type negotiation is specified as a core HTTP feature in RFC 9110 and there doesn't seem to be a better place for it. (See above discussion for some alternatives.)

[aclements]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— aclements for the proposal review group

We add a function to net/http which performs content type negotiation as specified by RFC 9110:

// NegotiateContent returns the best content to offer from a set of possible
// values, based on the preferences represented by the accept values.
// For example, NegotiateContent can be used on the HTTP servers to find the
// best "Content-Type" to provide to HTTP user agents, based on the "Accept"
// request header. This is also known as a proactive negotiation (or "server-driven"
// negotiation).
//
// NegotiateContent may be used to negotiate several fields, e.g.:
//
//	Response field    Request header
//	--------------    --------------
//	Content-Type      Accept
//	Content-Charset 	Accept-Charset
//	Content-Encoding  Accept-Encoding
//	Content-Language  Accept-Language
//
// The accepts parameter is the appropriate request header value(s),
// and is interpreted as in RFC 9110 section 12.
//
// The offers parameter is a list of possible values to offer.
//
// If no offers match, it returns the empty string.
// If more than one offer matches with equal weight and specificity,
// it returns the one earliest in the list.
// If no accept values are provided (representing no preference case), and offers
// are non-empty, the first offer is returned.
//
// NegotiateContent only considers the first 32 accept values to
// avoid DOS attacks.
func NegotiateContent(accepts []string, offers []string) string

The function signature varies slightly from the original proposal above. In particular, the offers may not include wildcards and there is no default offer. The lack of wildcards is because the server presumably knows the complete set of content types it can offer, and the lack of a default offer is because it is unnecessary--NegotiateContent returns the best offer, or the empty string if none are suitable.

I propose to add the function to net/http, on the grounds that content type negotiation is specified as a core HTTP feature in RFC 9110 and there doesn't seem to be a better place for it. (See above discussion for some alternatives.)