Does deterministic profile have threads instructions?

[eqrion]

From appendix/profiles.rst

The deterministic profile excludes all rules marked \exprofiles{\PROFDET}. It defines a sub-language that does not exhibit any incidental non-deterministic behaviour:

    All [:ref:`NaN <syntax-nan>`](https://github.com/WebAssembly/relaxed-simd/blob/main/document/core/appendix/profiles.rst#id22) values [:ref:`generated <aux-nans>`](https://github.com/WebAssembly/relaxed-simd/blob/main/document/core/appendix/profiles.rst#id24) by [:ref:`floating-point instructions <syntax-instr-numeric>`](https://github.com/WebAssembly/relaxed-simd/blob/main/document/core/appendix/profiles.rst#id26) are canonical and positive.

Even under this profile, the [|MEMORYGROW|](https://github.com/WebAssembly/relaxed-simd/blob/main/document/core/appendix/profiles.rst#id32) and [|TABLEGROW|](https://github.com/WebAssembly/relaxed-simd/blob/main/document/core/appendix/profiles.rst#id34) instructions technically remain [:ref:`non-deterministic <exec-memory.grow>`](https://github.com/WebAssembly/relaxed-simd/blob/main/document/core/appendix/profiles.rst#id28), in order to be able to indicate resource exhaustion.

When threads is merged in the future, is this expected to include shared memory and instructions that operate on it?

If so, then deterministic may not be the best name as that's a huge source of non-determinism. Not sure on a better name though.

If not, then what is the expected ecosystem that will be adopting the 'deterministic' profile? At least for browsers, we'd maybe have an opt-in preference for this for variants like Tor. Other than that, I had heard that having a deterministic variants of these relaxed-simd instructions could be valuable for cloud platforms but I would guess they would also like to have shared memory.

cc'ing @rossberg as I think this came from the profiles proposal originally.

[rossberg]

That would be part of the work for finalising the threads proposal, I'd say.

One option is that the deterministic profile simply forbids shared memories (atomic instructions are fine when applied to non-shared memory, which I believe we decided to allow, so no reason to forbid them).

The least intrusive option would be to merely rule out configurations with more than one thread. Then declaring a shared memory would still be fine, i.e., nothing is taken away from the language, but a host adopting this profile isn't allowed to actually create threads. That would defer the real question to a Post-MVP "Wasm threads" proposal, which seems adequate.

In the future we may also think about specifying a deterministic scheduling model, but it's unclear at this point whether that has any practical value.

[conrad-watt]

I hope, if we add the deterministic profile, that some runtime is actually going to make use of the mechanism (otherwise why are we adding it!?). So we should find a representative of a runtime fitting this description and ask them what their expectations are. If we can't find such a person, that would somewhat call into question the utility of "deterministic" as a profile altogether. @lukewagner - do you have any thoughts about who to ask?

[rossberg]

Wasmtime has a mode to normalise NaNs, which is sort of the Wasm 1.0/2.0 version of deterministic mode. That is needed e.g. by Consensus-based systems running Wasm, such as blockchains. Dfinity is using it, for example. So the need is definitely real.

[conrad-watt]

I guess a consensus system really would want to just forbid the creation of threads, which fits the sketch above. I'm not sure how this fits in a post-MVP threads world with thread.spawn though. Maybe if we make the spawn instruction fallible as has been discussed, it could be required to always fail in deterministic mode, similar to the way we'll need to make memory.grow always succeed/fail.

[eqrion]

similar to the way we'll need to make memory.grow always succeed/fail.

Does the current deterministic profile also restrict these?

[eqrion]

I agree that this is sort of a threads proposal issue, but it does affect the name we choose here so I think it's worth discussing so we don't paint ourselves into a corner.

I guess trying to come at it from an ecosystem perspective there are really three different ones I've heard from:

1. consensus/blockchain systems
2. large cloud platforms
3. browsers with fingerprint resisting

(1): would want no threads, deterministic instruction execution rules, and something around resource exhaustion(?).
(2): would just want deterministic instruction execution rules
(3): is somewhere between (1) and (2). I believe Tor disables shared memory and would want deterministic instructions execution rules, but stock Firefox with resist fingerprinting settings may just want deterministic instruction rules. We also may just not care about this ecosystem enough to have a profile for it.

Ignoring (3) you arrive at the following profiles:

1. full - {}
2. numeric-determinism - {no-relaxed-simd-nondeterminism no-nan-nondeterminism}
3. fully-deterministic - numeric-determinism U {no-threads, no-resource-exhaustion?}

I would say that numeric-determinism is the more important ecosystem at this point and I'd start with that being the one relaxed-simd uses.

[lukewagner]

A 4th use case is for the deterministic profile is build environments that want to ensure reproducible builds by construction using wasm.

Since the deterministic profile is more-widely deployed in production and since, until we add threads, the deterministic profile is the same as the numeric-determinism profile, I'd suggest we start with just the deterministic profile and postpone discussing the addition of more-refined profiles.

[conrad-watt]

until we add threads, the deterministic profile is the same as the numeric-determinism profile

I think there's some difference as deterministic would likely include deterministic memory.grow while the numeric version wouldn't. The numeric version is interesting as it would be (almost) the bare minimum needed to navigate around the specific additional issues caused by relaxed SIMD, so it might make sense as an initial approach while we hash out the details of other "determinisms".

[rossberg]

Yeah, let's not over-engineer it. If there is a need for a no-threads profile in the future, then that should be added as a separate profile — profiles ought to compose, so deterministic no-threads could naturally be formed from the two.

I don't think we can meaningfully make the likes of memory.grow deterministic on the language level, because that would require locking down how much memory is available. Only a given host or platform can possibly define that. Resource exhaustion is a totally different class of non-determinism that not much can be done about in general.

[eqrion]

@rossberg How do you feel about renaming the profile to numeric-determinism (or numeric-deterministic) then, as it won't encompass shared-memory determinism?

[rossberg]

@eqrion, with the threads proposal, deterministic mode will have to disallow the creation of additional threads. That at least is what the consensus, reproducible builds, and similar scenarios need. Do we have as strong a use case for numerics-only determinism?

[eqrion]

@rossberg My impression was that numeric-determinism would be useful for large cloud platforms that want to minimize architectural differences while still allowing multithreading.

In the end, we'll probably want both of these if we're trying to faithfully model real ecosystems out there. I personally think it'd be better to start with the conservative and smaller scoped numeric-determinism, but we could start in the other direction and end in the same place too.

[rossberg]

Yeah, we know immediate users for full determinism. The other seems more hypothetical, so perhaps let's wait until somebody actually requests it.

[eqrion]

As you said above, wasmtime has a canonicalize-nans option, which is the 1.0/2.0 version of numeric-determinism. So it's very much not hypothetical.

Again, I'm fine if you want to start with full determinism. But if we want to model what exists out there faithfully, we'll want numeric-determinism eventually.

[rossberg]

Well, in 1.0/2.0 the two interpretations are indistinguishable. But for 3.0, at least the customers of that option that I'm aware of, need the "full determinism" interpretation.

[eqrion]

wasmtime has support for threading and the option is only to canonicalize-nans, it's not a full determinism option. I'm also not sure why you're saying there's no customer that would use this as the cloud platform use case mentioned by Luke was one of the arguments that pushed us in the direction of profiles in the last in-person CG meeting.

[rossberg]

Hm, interesting point. I assume that the platforms in question simply disable the threads extension, since Wasmtime allows that. That is exactly the kind of ad-hoc fragmentation I'd like to prevent with standardised profiles. But to correctly support their existing use case, we'll then need to provide either (1) a no-threads profile, or (2) a full determinism profile.

I understand the potential clouds use case, but do we actually have a concrete client for it today? I'm a bit wary of profile inflation, so I wouldn't want to set bad precedence right away by starting off with multiple versions of determinism.

Also, why would they care about numerics specifically? If there was a reason to under-specify other, non-numeric instructions in future versions of Wasm, they'd probably want those fixed as well. That is, the distinction isn't between numeric and others, but between inherent (like threads) and accidental (implementation-dependent behaviour) non-determinism. So even if we want a profile for only the latter, the name should probably be a reflection of that.

[penzn]

I think it is worthwhile to separate deterministic relaxed-simd (as it is specified today) from NaN sanitization. NaN produced on one architecture is guaranteed to be treated as a NaN on a different one, while relaxed ops can produce different numeric results. Making relaxed math strict would ensure numeric (floating point) compatibility, unlike sanitizing NaNs, which is only necessary to ensure compatibility of a non-FP function consuming the values (cryptographic hash, for example).

A naive question, however. It looks like a large part of the discussion is multithreading and memory semantics, which are strictly speaking out of scope of relaxed-simd. Maybe detach profiles from this proposal for the time being? It would be still possible to do strict lowering without profiles, this spec allows for it, and we can just continue the discussion in the context of the profiles proposal.

[conrad-watt]

but do we actually have a concrete client for it today?

If this is the bar we're putting on the introduction of a profile (which isn't unreasonable), I'm a bit concerned that we have next to no input from any concrete client for any use case here - we're essentially trying to guess what such a client would want.

Maybe detach profiles from this proposal for the time being?

I'd personally be fine seeing this proposal stripped of its "profile" component, and then have the spec reinterpreted as the "full" profile when we eventually work out how a "deterministic" profile should work. My expectation is that anyone today who wants numeric determinism in their Wasm program already has to avoid/handle floating point NaNs, so avoiding relaxed SIMD (which is already explicitly signalled as "only use these if you know what you're doing") isn't too problematic since the analogous deterministic SIMD instructions/lowerings are available.

EDIT: I know we decided to couple the deterministic profile in a previous meeting, but it wouldn't be unreasonable to revisit that decision given the knots we're tying ourselves in

[rossberg]

My expectation is that anyone today who wants numeric determinism in their Wasm program already has to avoid/handle floating point NaNs, so avoiding relaxed SIMD (which is already explicitly signalled as "only use these if you know what you're doing") isn't too problematic

It's not the users who need the profile, it's the platforms. For example, a blockchain absolutely needs a well-defined deterministic semantics. It cannot rely on users/producers to "do the right thing".

[conrad-watt]

It's not the users who need the profile, it's the platforms. For example, a blockchain absolutely needs a well-defined deterministic semantics. It cannot rely on users/producers to "do the right thing".

I don't think relaxed SIMD is introducing a new class of problem for these platforms in comparison to existing things they need to be careful about like NaN bits. I can see the potential value of the deterministic profile but we seem to be rushing its design without much user input purely because we tied it to relaxed SIMD phase 4. At the very least we're finding ourselves having to map out how the currently sketched deterministic profile (mainly about numerics) would extend to threads and we have two conjectured use-cases that would want different choices.

[eqrion]

@rossberg

Also, why would they care about numerics specifically? If there was a reason to under-specify other, non-numeric instructions in future versions of Wasm, they'd probably want those fixed as well. That is, the distinction isn't between numeric and others, but between inherent (like threads) and accidental (implementation-dependent behaviour) non-determinism. So even if we want a profile for only the latter, the name should probably be a reflection of that.

Yeah, naming things is hard. I agree that it's really about the determinism of instructions in the absence of threads. I thought about 'instruction-determinism', but loads/stores on shared memory are 'instructions'. 'local-determinism' or 'nonshared-determinism' could be other options.

The point in favor of 'numeric-determinism' for me is that it covers the relaxed-SIMD and FP cases together, is unambiguous, and is easy to say. I understand it wouldn't cover potential non-numeric sources of non-determinism, but do we ever expect to add any? We wouldn't ever want impl-defined behavior in control flow, reference instructions, etc. It's also clear it doesn't cover resource exhaustion.

But either way, the name isn't super important to me.

Hm, interesting point. I assume that the platforms in question simply disable the threads extension, since Wasmtime allows that. That is exactly the kind of ad-hoc fragmentation I'd like to prevent with standardised profiles. But to correctly support their existing use case, we'll then need to provide either (1) a no-threads profile, or (2) a full determinism profile.

I understand the potential clouds use case, but do we actually have a concrete client for it today? I'm a bit wary of profile inflation, so I wouldn't want to set bad precedence right away by starting off with multiple versions of determinism.

My understanding of the 'cloud ecosystem use-case' is that they would still want to support threads while having deterministic behavior for the numeric instructions. But I work in browsers, so I cannot represent this beyond what I've heard in previous CG meetings.

It would be helpful to have more representation here. I would also like to hear more about the full determinism that platforms like blockchains or reproducible builds rely on as well. Resource exhaustion seems like a significant hole in those systems and I'm curious how they address it. I wouldn't want us to specify a profile that they don't exactly follow.

@penzn

I think it is worthwhile to separate deterministic relaxed-simd (as it is specified today) from NaN sanitization. NaN produced on one architecture is guaranteed to be treated as a NaN on a different one, while relaxed ops can produce different numeric results. Making relaxed math strict would ensure numeric (floating point) compatibility, unlike sanitizing NaNs, which is only necessary to ensure compatibility of a non-FP function consuming the values (cryptographic hash, for example).

It seems likely to me that any platform that would adopt a profile with deterministic relaxed-simd would likely also want canonical NaN's as well. They'd both be motivated by having no implementation defined behavior in numerics to worry about on their platform. If you care enough to have deterministic relaxed-simd, you probably care enough to have deterministic NaN's.

@conrad-watt

I'd personally be fine seeing this proposal stripped of its "profile" component, and then have the spec reinterpreted as the "full" profile when we eventually work out how a "deterministic" profile should work

I don't want to block relaxed-simd here, so I'm fine if that were to happen. However I'd also like to see if we can come up with some consensus on this before the next CG meeting so that we could avoid having to undo/redo the profiles work in this repo.

[eqrion]

@lukewagner

Since the deterministic profile is more-widely deployed in production and since, until we add threads, the deterministic profile is the same as the numeric-determinism profile, I'd suggest we start with just the deterministic profile and postpone discussing the addition of more-refined profiles.

I do agree with the sentiment here, however threads is potentially moving to phase 4 in the next CG meeting and so the difference in profiles becomes real pretty soon.

[penzn]

It seems likely to me that any platform that would adopt a profile with deterministic relaxed-simd would likely also want canonical NaN's as well. They'd both be motivated by having no implementation defined behavior in numerics to worry about on their platform. If you care enough to have deterministic relaxed-simd, you probably care enough to have deterministic NaN's.

I think it is important to distinguish between integer and FP numerics in this case. As far as FP math is concerned all (non-signaling) NaNs are equivalent, there is no difference in numeric behavior based on their exact bits. If the platform is only concerned about getting the same numbers from the same inputs to the same floating-point computation, then they won't necessarily care what the NaN bits are. Basically, a suspended mathematical computation can be resumed on a different architecture and produce the same result without requiring identical NaN representation.

The differences in output start to happen when we treat NaNs as non-FP values, because as a 32/64 bit integer float NaN actually covers a range of values. There are two obvious ways this would be a problem: fingerprinting, in case NaN bits actually convey useful information, and when FP values are part of an input into a hash. It is easy to imagine that every environment requiring single NaN would want deterministic math as well, but I am not sure about the other way around.

Anyway, that is also out of scope of relaxed-simd 😄 Though I think we should probably continue this discussion elsewhere.

[rossberg]

I'd personally be fine seeing this proposal stripped of its "profile" component, and then have the spec reinterpreted as the "full" profile when we eventually work out how a "deterministic" profile should work

I don't want to block relaxed-simd here, so I'm fine if that were to happen.

I'm rather concerned about the direction this discussion is taking. May I remind everybody that we had a lengthy and frustrating discussion about this a year ago, and were only able to reach consensus on Relaxed SIMD by agreeing to include a deterministic mode? That's why we are here. If you now propose rolling back that consensus then I have no idea were we stand.

At the very least we're finding ourselves having to map out how the currently sketched deterministic profile (mainly about numerics) would extend to threads and we have two conjectured use-cases that would want different choices.

Right, but that ought to be a problem for the threads proposal, the Relaxed SIMD proposal can remain entirely agnostic to that choice (up to possibly naming, which isn't semantically relevant), so should be able to move ahead.

As for threads, as I said above, I see only two viable choices to serve the existing use cases: either interpret and adjust the deterministic profile to also forbid spawning threads, or introduce a separate no-threads profile that can be combined with (then less-than-)deterministic mode. I'm fine with either.

Resource exhaustion seems like a significant hole in those systems and I'm curious how they address it.

I can only tell for sure for Dfinity's platform, where it is defined by the platform semantics, essentially as part of the fuel model for execution that is needed anyway. I assume other blockchains behave similarly. For other use cases, like reproducible builds, it may not matter too much, since at worst it causes a build to fail and that can be retried. But here I'm guessing.

[eqrion]

Right, but that ought to be a problem for the threads proposal, the Relaxed SIMD proposal can remain entirely agnostic to that choice (up to possibly naming, which isn't semantically relevant), so should be able to move ahead.

Threads and relaxed simd both want to move to phase 4 in the next CG meeting. So if we defer this to threads then we'll need to delay that proposal instead.

As for threads, as I said above, I see only two viable choices to serve the existing use cases: either interpret and adjust the deterministic profile to also forbid spawning threads, or introduce a separate no-threads profile that can be combined with (then less-than-)deterministic mode. I'm fine with either.

I would be fine and prefer the second option, to have them split out from each other. I think that provides for the use-cases discussed in this thread well.

[conrad-watt]

I'd prefer the second option - I also think a name change like "functional-determinism" or "numeric-determinism" would better reflect the scope of such a profile, but I wouldn't push for this too hard.

[titzer]

Just as a datapoint, the threads proposal includes ~70 instructions, including read-modify-write variants of memory accesses, as well as futex operations. For engines and platforms that can't or won't offer threads and want to conform to a no-threads profile, it would simplify them a lot if the profile disallows those instructions.

[dtig]

Right, but that ought to be a problem for the threads proposal, the Relaxed SIMD proposal can remain entirely agnostic to that choice (up to possibly naming, which isn't semantically relevant), so should be able to move ahead.

Threads and relaxed simd both want to move to phase 4 in the next CG meeting. So if we defer this to threads then we'll need to delay that proposal instead.

Late to this thread, but in my mind proposals and profiles are somewhat orthogonal, i.e. that advancing a proposal to Phase 4 doesn't necessarily mean that we can't introduce a profile that restricts the instructions introduced in some way in the future. I'd also like to push for the position that if we're adding new profiles, they should reflect a real world environment that we have feedback from.

I think any proposal that first tries to use the profiles syntax will probably run into similar discussions, if the syntax in the profiles proposal is something the CG wants to adopt, then for relaxed-simd specifically, scoping the name to indicate exactly what the profile restricts (numeric-determinism or something else) makes sense. The only difference between threads and relaxed-simd here is that existing engines that have provided feedback to the CG, have a need for a restricted profile in some environments that doesn't introduce the type of non-determinism that the relaxed-simd proposal introduces.

Stepping back a little, I also think the risk here to experiment with the proposed syntax or other approaches are very small. The profiles proposal has no implications on implementations at this time except for a way to represent what part of the spec implementations support, so if we run into cases where the existing syntax doesn't address the use cases, we can revisit the infrastructure, propose new syntax, or think of other ways to factor the spec when needed.

As for threads, as I said above, I see only two viable choices to serve the existing use cases: either interpret and adjust the deterministic profile to also forbid spawning threads, or introduce a separate no-threads profile that can be combined with (then less-than-)deterministic mode. I'm fine with either.

I would be fine and prefer the second option, to have them split out from each other. I think that provides for the use-cases discussed in this thread well.