Refine Specifications Definitions data model

[dontcallmedom]

Bikeshed defines the contract that specifications need to follow to have their definitions included in the bikeshed list of references, which ReSpec imports as well.

Reffy aims to provide a new source of data for these definitions with more up-to-date and accurate data (given e.g. its capability to parse ReSpec live-generated editors draft), and while there are a few remaining issues (#333, #334, #335), the gist of the diff between what Reffy finds and what Shepherd finds comes from rules that Shepherd applies that aren't described in the contract.

This issue aims at gathering input from @tabatkins, @marcoscaceres @sidvishnoi and @plinss on refinements to the spec definition parsing contract, to make sure what reffy produces aligns with what editing tools outputs and what editing tools would want to consume for cross-references.

Analyzing the diff between reffy and shepherd, the following aspects need clarification:

• in dfn-type dfn, shepherd seems to convert curly quotes (’) into straight ones (') - is that something we want to preserve?
• likewise, quotes that surround the whole linking text get removed by shepherd in many cases (e.g. enum values in general, HTML definitions e.g. "non-blocking"), but not all (e.g. "strict-origin-when-cross-origin") - it would be best to clarify the intent in the parsing contract
  • with dfn-type enum-value, bikeshed includes a version of the linking text which includes the surrounding " , whereas respec only has the "-less version; shepherd strips them (per the above), but it would seem better to align the editing tools here
• right now, for functions (incl methods, constructors), bikeshed automatically exports definitions of the method with and without arguments; in respec, this is left to the discretion of spec editors, which create inconsistent results; in practice, I'm not sure the versions with arguments help anyone.
• both bikeshed and respec allows to apply type-dfn to other elements, and when applied to headings that get auto-numbered, currently following the spec definition contract also extracts the section number (which is of course not wanted); shepherd seems to be stripping it, but maybe instead the spec tools should turn <h3 data-type=dfn>Foo</h3> into <h3>4.2 <dfn>Foo</dfn></h3>?

[tabatkins]

in practice, I'm not sure the versions with arguments help anyone.

The versions with arguments are required when you have overloads, which are distinct entities in WebIDL and must be independently linkable. It's unfortunate that this ends up causing a lot of annoyance and confusion, but I don't see a way around it besides "Bikeshed doesn't support overloads".

(Technically overloads must be unique according to argument types, not names, but that would be even worse to deal with for spec authors and autolinks, and in practice argument names almost always naturally uniquify an overload set.)

[marcoscaceres]

We've had the same struggles with overloads... I've been considering using something like C++ name mangling to address it.

Another thing that's come up recently is making sure the letter casing of exported terms in retained. I think bikeshed's data normalizes to lowercase, which can be confusing for things like sRGB, which shows up as "srgb"... similarly, utf-8 and so on. This makes it hard to suggest "how to cite" certain concepts properly, for example: https://respec.org/xref/?term=srgb

About the headings, we could definitely add the dfns... but maybe they should be a new dfn-type?... however, there are often times when definitions are included in a header: e.g., <h2>The <dfn>Whatever</dfn> interface</h2>... so we'd have to be careful not to double wrap things.

[dontcallmedom]

Another thing that's come up recently is making sure the letter casing of exported terms in retained. I think bikeshed's data normalizes to lowercase, which can be confusing for things like sRGB, which shows up as "srgb"... similarly, utf-8 and so on.

I noticed that as well in #332 - right now, the definition extraction from Rreffy keep the case of the extracted terms as-is. I think it should be on the consumer side that, based on the type of the definition, case-sensitivity should be taken into account (e.g. CSS properties are case insensitive, IDL names aren't).

[dontcallmedom]

The versions with arguments are required when you have overloads, which are distinct entities in WebIDL and must be independently linkable. It's unfortunate that this ends up causing a lot of annoyance and confusion, but I don't see a way around it besides "Bikeshed doesn't support overloads".

That makes a lot more sense now, thanks for the explanation. I wonder if we could limit that behavior to methods that actually have overloads. Now that WebIDL requires overloads to be defined in a single place, I believe ReSpec/Bikeshed could detect this and expose the arguments only in that case. Since overloads are rather exceptional in any case, I think special-casing them would be OK.

List of overloads known in Reffy

{
  "https://html.spec.whatwg.org/multipage/": [
    "HTMLInputElement.setRangeText",
    "HTMLSelectElement.remove",
    "HTMLTextAreaElement.setRangeText",
    "CanvasTransform.setTransform",
    "CanvasDrawPath.fill",
    "CanvasDrawPath.stroke",
    "CanvasDrawPath.clip",
    "CanvasDrawPath.isPointInPath",
    "CanvasDrawPath.isPointInStroke",
    "CanvasUserInterface.drawFocusIfNeeded",
    "CanvasUserInterface.scrollPathIntoView",
    "CanvasDrawImage.drawImage",
    "CanvasImageData.createImageData",
    "CanvasImageData.putImageData",
    "DataTransferItemList.add",
    "Window.alert",
    "Window.postMessage",
    "WindowOrWorkerGlobalScope.createImageBitmap",
    "WebSocket.send",
    "MessagePort.postMessage",
    "DedicatedWorkerGlobalScope.postMessage",
    "Worker.postMessage"
  ],
  "https://wicg.github.io/cookie-store/": [
    "CookieStore.get",
    "CookieStore.getAll",
    "CookieStore.set",
    "CookieStore.delete"
  ],
  "https://wicg.github.io/web-locks/": [
    "LockManager.request"
  ],
  "https://www.khronos.org/registry/webgl/specs/latest/1.0/": [
    "WebGLRenderingContextOverloads.bufferData",
    "WebGLRenderingContextOverloads.texImage2D",
    "WebGLRenderingContextOverloads.texSubImage2D"
  ],
  "https://www.khronos.org/registry/webgl/specs/latest/2.0/": [
    "WebGL2RenderingContextBase.texImage3D",
    "WebGL2RenderingContextBase.texSubImage3D",
    "WebGL2RenderingContextBase.compressedTexImage3D",
    "WebGL2RenderingContextBase.compressedTexSubImage3D",
    "WebGL2RenderingContextOverloads.bufferData",
    "WebGL2RenderingContextOverloads.bufferSubData",
    "WebGL2RenderingContextOverloads.texImage2D",
    "WebGL2RenderingContextOverloads.texSubImage2D",
    "WebGL2RenderingContextOverloads.compressedTexImage2D",
    "WebGL2RenderingContextOverloads.compressedTexSubImage2D",
    "WebGL2RenderingContextOverloads.readPixels"
  ],
  "https://www.w3.org/TR/presentation-api/": [
    "PresentationConnection.send"
  ],
  "https://www.w3.org/TR/service-workers-1/": [
    "ServiceWorker.postMessage",
    "Client.postMessage"
  ],
  "https://www.w3.org/TR/wasm-js-api-1/": [
    "WebAssembly.instantiate"
  ],
  "https://www.w3.org/TR/webaudio/": [
    "AudioNode.connect",
    "AudioNode.disconnect"
  ],
  "https://www.w3.org/TR/webrtc/": [
    "RTCPeerConnection.createOffer",
    "RTCPeerConnection.createAnswer",
    "RTCPeerConnection.setLocalDescription",
    "RTCPeerConnection.setRemoteDescription",
    "RTCPeerConnection.addIceCandidate",
    "RTCDataChannel.send"
  ],
  "https://xhr.spec.whatwg.org/": [
    "XMLHttpRequest.open",
    "FormData.append",
    "FormData.set"
  ]
}

(at least in the case of RTCDataChannel.send and WebSocket.send, they could be rewritten as Union)

[dontcallmedom]

Re case sensitivity, some related discussions on this: speced/bikeshed#861 speced/bikeshed#1705

[marcoscaceres]

For overloads, I'm happy to add support for whatever, but just need a clear specification is needed in terms of markup.

[dontcallmedom]

we should confirm if an empty data-lt is the right convention to "hide" a definition (as was done e.g. in whatwg/html#1715)

[marcoscaceres]

We can explicitly data-no-export="" partials.

[marcoscaceres]

(I need to check, but I think ReSpec never defines partials... I recall we always "xref" them and assume they are defined in another specification).

[dontcallmedom]

* in `dfn-type`  `dfn`, shepherd seems to convert curly quotes (`’`) into straight ones (`'`) - is that something we want to preserve?

I see that this (and presumably some other of the points I identified) are actually done in bikeshed import of shepherd rather than shepherd directly https://github.com/tabatkins/bikeshed/blob/master/bikeshed/update/updateCrossRefs.py

[tabatkins]

I wonder if we could limit that behavior to methods that actually have overloads. Now that WebIDL requires overloads to be defined in a single place, I believe ReSpec/Bikeshed could detect this and expose the arguments only in that case. Since overloads are rather exceptional in any case, I think special-casing them would be OK.

Ooooh, I didn't realize that restriction was in place now. Yeah, in that case I'm totally in favor of only doing argument stuff when an overload is actually present, and in all other cases just dfn'ing the argless term.

[tidoust]

Back to refining the specifications definitions data model, I note the need to clarify the "comma-separated value" rule, as noted in #229 (comment)

[dontcallmedom]

Another potential addition to the data model that we discussed the other day with @marcoscaceres : it would probably be useful to have algorithm dfn-type, and to be able to annotate it optionally with the expected type of its input/output.

(the latter would be useful in ensuring that users of a given algorithm update their input/output when the said algorithm evolves)

[tidoust]

Closing this issue as I believe all points raised here got raised in the speced/spec-dfn-contract repo, where they should get addressed.