WatIDL - strawman text format for wasm interfaces #64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wanderer
changed the title
bjorn3
reviewed
Jul 7, 2019
|
I love this idea! We've been working in something similar (but a bit closer to the current existing types of WebAssembly) wasm-interfaces (See interfaces proposal in WebAssembly Design repo) I would love to hear more thoughts about the Wat proposal! |
sbc100
reviewed
Jul 8, 2019
| ``` | ||
| (func (export "multi-vale") | ||
| (result $result1 u64) | ||
| (result $result2 f32) |
There was a problem hiding this comment.
Would it make sense to use the same syntax as the multi-val proposal here: https://github.com/WebAssembly/multi-value/blob/master/proposals/multi-value/Overview.md#functions-with-multiple-return-values
There was a problem hiding this comment.
i thought this was valid syntax for the multi-val proposal. Well at least wabt excepts it as valid 😛 That said if we can find the authoritative definition some where, lets follow that!
There was a problem hiding this comment.
(param i32) (param i32) (result i32) (result i32), (param i32 i32) (result i32 i32), and permutations of both, are all valid. (result $name xyz) is not valid, but I think it's fine for documentation purposes here.
watidl/watidl.md
Outdated
| (result u64) | ||
| ) | ||
|
|
||
| (func (export "not-static") |
There was a problem hiding this comment.
Maybe leave func with is current meaning and use "method" when you want an implicit "this" for 0?
There was a problem hiding this comment.
yep i see how that part is confusing. Method sounds like an accurate description.
watidl/wasi-core.watidl
Outdated
|
|
||
| ;; Move the offset of a file descriptor. | ||
| ;; Note: This is similar to lseek in POSIX. | ||
| (func (export "seek") |
There was a problem hiding this comment.
So is "fd" a separate WASI interface or just part of wasi-core? Right now we just have wasi-unstable as a single interface. If we wanted to literally translate the current wasi-unstable as it stands would be just one interface?
Maybe this isn't a precise representation of wasi-unstable as it is today, but if one wanted to do that presumably one wouldn't try to mode file functions as methods on the 'fd' object like this? This seems like add OO-style when there was none previously.
There was a problem hiding this comment.
So is "fd" a separate WASI interface or just part of wasi-core?
Currently its part of WASI core, but I think there some plans to break up wasi-core into several modules. So this is trying to reflect that future. I was thinking of adding (interface $fd (export "wasi-unstable") to explicitly show that all of the imports are under the "wasi-unstable" module namespace. In the future it would be nice to have multiple modules we could use so I imagine common types would be imported from "wasi-core" or just "wasi" base file descriptor methods would be imported from "wasi-fd" ect.
Maybe this isn't a precise representation of wasi-unstable as it is today
I think it would be nice if this could represent wasi-unstable today. I think we can do that by adding some notation to that says "all these interfaces have the same module namespace". That way we can introduce this notation without having to make other changes.
but if one wanted to do that presumably one wouldn't try to mode file functions as methods on the 'fd' object like this?
So there is another idea here. Yes its very OO-style centric and ideally we can having binding that allow wasm programs to have two different modes of interacting with the interface 1) is the current ADT pattern or 2) have some sort of dynamic dispatching which would allow for virtualization.
The OO-style would suggest number # 2 but can also express # 1 But if don't have some OO-style notation then how would represent # 2 in the future?
sunfishcode
reviewed
Jul 8, 2019
watidl/watidl.md
Outdated
|
|
||
| (interface $a | ||
| (func (export "factory") | ||
| (result $the_result $b) |
There was a problem hiding this comment.
Could you describe what this means in more detail? Would each instance of an interface have its own copy of the exported global variables declared in an interface?
There was a problem hiding this comment.
Would each instance of an interface have its own copy of the exported global variables declared in an interface?
thats a good question, and it might show that there is some inconsistency here with what interfaces are. i'll think about this more
| (param u64 u64) | ||
| (result u64) | ||
| ) | ||
| ) |
There was a problem hiding this comment.
Do you intend this to interact with interfaces being first-class? Specifically, are you intending to allow for dynamic polymorphism, where a base class can have multiple derived classes which override methods in the base class, and method dispatch is determined by the dynamic type of an object?
There was a problem hiding this comment.
so i think i want to back away form interfaces being first-class. I think things will be more simple of interfaces just represent what a module can import and export. I'll try to itterate on this in the next couple of days. But instead of having the OO-style stuff squished on top of an interface (this was capn-proto style) we can put them in to stucts ie
(interface $wasi
;; all the base methods for fd's such as
(struct $file_descriptor
(field $close (func))
(field $datasync (func))
)
...
)
watidl/wasi-core.watidl
Outdated
| @@ -0,0 +1,1076 @@ | |||
| (type $size_t u64) | |||
There was a problem hiding this comment.
This is interesting :-). I assume the motivation for using 64-bit here is to be generic between 32-bit and 64-bit. I think that's reasonable, especially since most uses of size_t in wasi_unstable are replaced here by bindings types like string and array. However, it'd be good to add a comment about this here.
added 2 commits
July 8, 2019 18:25
|
Following up on today's meeting: Here's the issue we have for "Wasm Interfaces" which are WAT syntax with only Wasm core types. We also have a working set of semantics for validating and merging them, but we are of course open to change and compromise for standardization! @wanderer I have some questions about the |
|
I vote for punting on trying to model any kind of dynamic dispatch in the WASI interfaces until we have a real requirement for it (and primitives that might support it). Keeping this simple and static at the WASI level (at least for now) makes sense to me. |
|
@MarkMcCaskey I updated the syntax and removed
@sbc100 what do you think about |
I don't feel super strongly about it, it just seems that its not needed for the current wasi_unstable. I don't see any problem with modeling |
one option could be to explicitly have a |
Hmm, that's a good point -- having data types definitely makes it more useful for things like generating code. The reason we decided to make something like our current proposal of "Wasm Interfaces" is that we needed a way to allow arbitrary, user-defined ABIs to be combined and modules validated against them to improve the usability of our Wasm package manager and the usability of other programs that interact with lots of less common ABIs. The assumption we made is that if a module fits this interface then it was probably written in a higher level language with a library providing the definitions or even an entire wrapper (which could be generated by ⛄️ bindings), so things like passing the correct non-core Wasm types are already taken care of. The way we handled changes in data types (which are not represented in our interface text format) with identical imports/exports is just to version the interfaces. Hopefully that gives context into where we're coming from. To get more context on this: is there a reason this IDL proposal is specific to WASI? It makes sense in the context of unblocking WASI's split into submodules (which I'm super excited about!), but isn't this IDL more widely applicable? |
|
@sbc100 okie I removed method for now. Maybe we can reconsider it once we have the story on dynamic dispatching.
@MarkMcCaskey yes i agree that it could be more widely applicable. I think we will probably eventual merge some efforts with the webild-binding work. |
This was referenced Jul 18, 2019
Closed
sunfishcode
added a commit
that referenced
this pull request
Aug 15, 2019
This adds a description of the current wasi_unstable API, using a syntax which anticipates the "module types" syntax [recently proposed to the CG](https://github.com/WebAssembly/meetings/blob/master/2019/CG-08-06.md#discuss-new-proposal-that-introduces-types-for-modules-and-instances-and-text-format-thereof-as-initially-discussed-in-design1289), though it does a few extensions which can be easily lowered, and which mostly anticipate the [Interface Types](WebAssembly/interface-types#47 (comment)). proposal. This is derived from [the WatIDL proposal](#64), though it differs in a few ways: - It doesn't yet do anything special for capabilities, for now. File descriptors are just integers, for now. But the idea is that we could add OCAP concepts later. - It uses a new `(flags ...)` construct in place of structs of Bool fields, to describe flags values. This allows us to explicitly declare the underlying wasm type for flags values. - Types used by the API are declared outside of the moduletype, because we're using wat syntax as much as possible, and there, `(type ...)` inside a module declares an entry in the type section, which isn't what's intended here. The extensions to module types are: - It uses `string`, `array`, and `{s,u}{8,16,32,64}` types, which are expected to be provided in some form by the Interface Types proposal. For now though, they can all be lowered in straightforward ways, by assuming that `string` is always just UTF-8. - It adds `typename` declarations, which can name any wasm type, and of the extra types mentioned in the previous item, or `struct`, `enum`, or `flags`. - It declares functions with multiple return values, which for now will need to be lowered to output (pointer) parameters.
sunfishcode
added a commit
that referenced
this pull request
Aug 15, 2019
This adds a description of the current wasi_unstable API, using a syntax which anticipates the "module types" syntax [recently proposed to the CG](https://github.com/WebAssembly/meetings/blob/master/2019/CG-08-06.md#discuss-new-proposal-that-introduces-types-for-modules-and-instances-and-text-format-thereof-as-initially-discussed-in-design1289), though it does a few extensions which can be easily lowered, and which mostly anticipate the [Interface Types](WebAssembly/interface-types#47 (comment)) proposal. This is derived from [the WatIDL proposal](#64), though it differs in a few ways: - It doesn't yet do anything special for capabilities, for now. File descriptors are just integers, for now. But the idea is that we could add OCAP concepts later. - It uses a new `(flags ...)` construct in place of structs of Bool fields, to describe flags values. This allows us to explicitly declare the underlying wasm type for flags values. - Types used by the API are declared outside of the moduletype, because we're using wat syntax as much as possible, and there, `(type ...)` inside a module declares an entry in the type section, which isn't what's intended here. The extensions to module types are: - It uses `string`, `array`, and `{s,u}{8,16,32,64}` types, which are expected to be provided in some form by the Interface Types proposal. For now though, they can all be lowered in straightforward ways, by assuming that `string` is always just UTF-8. - It adds `typename` declarations, which can name any wasm type, and of the extra types mentioned in the previous item, or `struct`, `enum`, or `flags`. - It declares functions with multiple return values, which for now will need to be lowered to output (pointer) parameters.
sunfishcode
added a commit
to sunfishcode/WASI
that referenced
this pull request
Aug 15, 2019
This sketches out a description of the current wasi_unstable API, using a syntax which anticipates the "module types" syntax [recently proposed to the CG](https://github.com/WebAssembly/meetings/blob/master/2019/CG-08-06.md#discuss-new-proposal-that-introduces-types-for-modules-and-instances-and-text-format-thereof-as-initially-discussed-in-design1289), though it does use a few extensions which can be easily lowered, and which mostly anticipate the [Interface Types](WebAssembly/interface-types#47 (comment)) proposal. This is derived from [the WatIDL proposal](WebAssembly#64), though it differs in a few ways: - It doesn't yet do anything special for capabilities, for now. File descriptors are just integers, for now. But the idea is that we could add OCAP concepts later. - It uses a new `(flags ...)` construct in place of structs of Bool fields, to describe flags values. This allows us to explicitly declare the underlying wasm type for flags values. - Types used by the API are declared outside of the moduletype, because we're using wat syntax as much as possible, and there, `(type ...)` inside a module declares an entry in the type section, which isn't what's intended here. The extensions to module types are: - It uses `string`, `array`, and `{s,u}{8,16,32,64}` types, which are expected to be provided in some form by the Interface Types proposal. For now though, they can all be lowered in straightforward ways, by assuming that `string` is always just UTF-8. - It adds `typename` declarations, which can name any wasm type, and of the extra types mentioned in the previous item, or `struct`, `enum`, or `flags`. - It declares functions with multiple return values, which for now will need to be lowered to output (pointer) parameters.
pchickey
pushed a commit
to bytecodealliance/lucet
that referenced
this pull request
Sep 4, 2019
original commit, from https://github.com/sunfishcode/wasi: commit 383385700a13acc2bc4ad2ac7e78a445d9053d21 (HEAD, refs/remotes/sunfishcode/moduletype) Author: Dan Gohman <[email protected]> Date: Wed Aug 14 22:42:24 2019 -0700 A wasi_unstable description based on module types. This sketches out a description of the current wasi_unstable API, using a syntax which anticipates the "module types" syntax [recently proposed to the CG](https://github.com/WebAssembly/meetings/blob/master/2019/CG-08-06.md#discuss-new-proposal-that-introduces-types-for-modules-and-instances-and-text-format-thereof-as-initially-discussed-in-design1289), though it does use a few extensions which can be easily lowered, and which mostly anticipate the [Interface Types](WebAssembly/interface-types#47 (comment)) proposal. This is derived from [the WatIDL proposal](WebAssembly/WASI#64), though it differs in a few ways: - It doesn't yet do anything special for capabilities, for now. File descriptors are just integers, for now. But the idea is that we could add OCAP concepts later. - It uses a new `(flags ...)` construct in place of structs of Bool fields, to describe flags values. This allows us to explicitly declare the underlying wasm type for flags values. - Types used by the API are declared outside of the moduletype, because we're using wat syntax as much as possible, and there, `(type ...)` inside a module declares an entry in the type section, which isn't what's intended here. The extensions to module types are: - It uses `string`, `array`, and `{s,u}{8,16,32,64}` types, which are expected to be provided in some form by the Interface Types proposal. For now though, they can all be lowered in straightforward ways, by assuming that `string` is always just UTF-8. - It adds `typename` declarations, which can name any wasm type, and of the extra types mentioned in the previous item, or `struct`, `enum`, or `flags`. - It declares functions with multiple return values, which for now will need to be lowered to output (pointer) parameters.
pchickey
pushed a commit
to bytecodealliance/lucet
that referenced
this pull request
Sep 11, 2019
* lucet-idl: competent sexpr parser * lucet-idl: start implementing grammar from https://github.com/WebAssembly/WASI/pull/74/files * lucet-idl: add dan's wasi_unstable.wati as a test case original commit, from https://github.com/sunfishcode/wasi: commit 383385700a13acc2bc4ad2ac7e78a445d9053d21 (HEAD, refs/remotes/sunfishcode/moduletype) Author: Dan Gohman <[email protected]> Date: Wed Aug 14 22:42:24 2019 -0700 A wasi_unstable description based on module types. This sketches out a description of the current wasi_unstable API, using a syntax which anticipates the "module types" syntax [recently proposed to the CG](https://github.com/WebAssembly/meetings/blob/master/2019/CG-08-06.md#discuss-new-proposal-that-introduces-types-for-modules-and-instances-and-text-format-thereof-as-initially-discussed-in-design1289), though it does use a few extensions which can be easily lowered, and which mostly anticipate the [Interface Types](WebAssembly/interface-types#47 (comment)) proposal. This is derived from [the WatIDL proposal](WebAssembly/WASI#64), though it differs in a few ways: - It doesn't yet do anything special for capabilities, for now. File descriptors are just integers, for now. But the idea is that we could add OCAP concepts later. - It uses a new `(flags ...)` construct in place of structs of Bool fields, to describe flags values. This allows us to explicitly declare the underlying wasm type for flags values. - Types used by the API are declared outside of the moduletype, because we're using wat syntax as much as possible, and there, `(type ...)` inside a module declares an entry in the type section, which isn't what's intended here. The extensions to module types are: - It uses `string`, `array`, and `{s,u}{8,16,32,64}` types, which are expected to be provided in some form by the Interface Types proposal. For now though, they can all be lowered in straightforward ways, by assuming that `string` is always just UTF-8. - It adds `typename` declarations, which can name any wasm type, and of the extra types mentioned in the previous item, or `struct`, `enum`, or `flags`. - It declares functions with multiple return values, which for now will need to be lowered to output (pointer) parameters. * lucet-idl: enough infra to feed wasi_unstable.wati through parser * lucet-idl: fix typo in wasi_unstable lexer found it! cheers * lucet-idl: better error reporting * lucet-idl: parse a bunch more of wati * lucet-idl: @interface is an Annot("interface") * lucet-idl: switch from wasi_unstable.wati to multiple .witx * lucet-idl: interface-types idl files are now called .witx webassembly interface types has claimed the `.wit` file extension for describing just module interfaces. this idl is currently the `.wit` syntax, plus extensions. in lieu of a better name, dan and I decided `.witx` will work for the time being. * lucet-idl: renames of stuff in witx, plus use stmts * lucet-idl: typenames.witx had an extra close paren * lucet-idl: correct path to typenames in wasi_unstable.witx * lucet-idl: change parser to own everything. start implementing use * lucet-idl: witx use directive works! * lucet-idl: better error messages in parser through lookahead * lucet-idl: parser makes it through wasi_unstable * lucet-idl: wasi_unstable.witx: fix two syntax errors in * lucet-idl: witx use decls are now like #pragma once so we dont have to care about cycles, and diamond dependencies will be handled correctly * lucet-idl: witx use resolution moved to toplevel.rs * lucet-idl: make an AST for witx * lucet-idl: rename interfacetypes to witx * lucet-idl witx: file name goes in location, location throughout parse tree * lucet-idl witx: validator is filling out * lucet-idl witx: typenames.witx reorg so names are defined before use this restriction makes it possible to perform validation linearly, which is in line with validation elsewhere in wasm * lucet-idl witx: validation of all datatypes complete * lucet-idl witx: finish validation * lucet-idl witx: fix tests * lucet-idl: fix tests * lucet-idl witx: syntax derives Eq. mock the filesystem to test use more tests required * lucet-idl witx: some fixups, plenty of toplevel tests * lucet-idl witx: add docs, refactor error name, resolve warnings * lucet-idl: add test case to load wasi_unstable.witx * lucet-idl: rustfmt * lucet-idl witx: review comments * lucet-idl witx: really fix the path thing this time * lucet-idl witx: simplify paths one more time not my favorite corner of std!!
sunfishcode
added a commit
that referenced
this pull request
Sep 12, 2019
This sketches out a description of the current wasi_unstable API, using a syntax which anticipates the "module types" syntax [recently proposed to the CG](https://github.com/WebAssembly/meetings/blob/master/2019/CG-08-06.md#discuss-new-proposal-that-introduces-types-for-modules-and-instances-and-text-format-thereof-as-initially-discussed-in-design1289), though it does use a few extensions which can be easily lowered, and which mostly anticipate the [Interface Types](WebAssembly/interface-types#47 (comment)) proposal. This is derived from [the WatIDL proposal](#64), though it differs in a few ways: - It doesn't yet do anything special for capabilities, for now. File descriptors are just integers, for now. But the idea is that we could add OCAP concepts later. - It uses a new `(flags ...)` construct in place of structs of Bool fields, to describe flags values. This allows us to explicitly declare the underlying wasm type for flags values. - Types used by the API are declared outside of the moduletype, because we're using wat syntax as much as possible, and there, `(type ...)` inside a module declares an entry in the type section, which isn't what's intended here. The extensions to module types are: - It uses `string`, `array`, and `{s,u}{8,16,32,64}` types, which are expected to be provided in some form by the Interface Types proposal. For now though, they can all be lowered in straightforward ways, by assuming that `string` is always just UTF-8. - It adds `typename` declarations, which can name any wasm type, and of the extra types mentioned in the previous item, or `struct`, `enum`, or `flags`. - It declares functions with multiple return values, which for now will need to be lowered to output (pointer) parameters.
sunfishcode
added a commit
that referenced
this pull request
Sep 12, 2019
This sketches out a description of the current wasi_unstable API, using a syntax which anticipates the "module types" syntax [recently proposed to the CG](https://github.com/WebAssembly/meetings/blob/master/2019/CG-08-06.md#discuss-new-proposal-that-introduces-types-for-modules-and-instances-and-text-format-thereof-as-initially-discussed-in-design1289), though it does use a few extensions which can be easily lowered, and which mostly anticipate the [Interface Types](WebAssembly/interface-types#47 (comment)) proposal. This is derived from [the WatIDL proposal](#64), though it differs in a few ways: - It doesn't yet do anything special for capabilities, for now. File descriptors are just integers, for now. But the idea is that we could add OCAP concepts later. - It uses a new `(flags ...)` construct in place of structs of Bool fields, to describe flags values. This allows us to explicitly declare the underlying wasm type for flags values. - Types used by the API are declared outside of the moduletype, because we're using wat syntax as much as possible, and there, `(type ...)` inside a module declares an entry in the type section, which isn't what's intended here. The extensions to module types are: - It uses `string`, `array`, and `{s,u}{8,16,32,64}` types, which are expected to be provided in some form by the Interface Types proposal. For now though, they can all be lowered in straightforward ways, by assuming that `string` is always just UTF-8. - It adds `typename` declarations, which can name any wasm type, and of the extra types mentioned in the previous item, or `struct`, `enum`, or `flags`. - It declares functions with multiple return values, which for now will need to be lowered to output (pointer) parameters.
|
Thanks @wanderer for your work here! Much of the contents of this WatIDL PR helped shape and contribute to #74 and the witx file format, which has now landed. While witx does currently have a smaller scope than watidl here, it's just a beginning, and we can now work on extending it to better support OCAP in parallel with our other WASI API design work. |
DeepInThought
added a commit
to DeepInThought/lucet
that referenced
this pull request
Jan 23, 2020
* lucet-idl: moved half of datatype validation & layout code to new side * lucet-idl: datatype validation & layout calculated in new stuff * lucet-idl: port in func validator * lucet-idl: factor module names into their own validator put all validators behind a single module name * lucet-idl: port module datatype tests into new code * lucet-idl: build out cursor, module validation for funcs * lucet-idl: finish up module tests in env * lucet-idl: add package-level validation and tests to new side * lucet-idl: move new repr/cursor stuff to root of crate. codegen disabled! * lucet-idl: a PackageRepr is a Package now unboxing it fixed our lifetime problems * lucet-idl: c generator turned on again and it looks SO MUCH NICER dont you think?? * lucet-idl: build out idiomatic func defintion in C! * lucet-idl: rust datatypes are generated * lucet-idl: add rust guest abi import * lucet-idl: rust backend ported to new cursors * lucet-idl-test: fixup most issues * lucet-idl: move atomtype/abitype parsing from lexer to validator * lucet-idl: parse tree uses &'syntax str rather than String one less clone is good * lucet-idl: reject out-slice bindings cant support these till we make an allocation protocol * lucet-idl: refactor rust backend to be more reasonable * lucet-idl-test: render harnesses for both rust guest and host * bump cargo.lock * lucet-idl-test: many fixes, very nearly works * lucet-idl-test: fix testing! we can now round-trip test the example idl successfully! * lucet-idl: get rid of `io` binding syntax, just support `inout` * lucet-idl: only derive Eq and Ord on structs that do not contain floats * lucet-idl: simplification in parser for func return value / whereclauses * lucet-idl: explain c backend unreachable * lucet-idl: make all of the cursor structs Copy and eliminate a ton of clones as a result * lucet-idl: use a while let * lucet-idl: host func name is a rust-specific thing * lucet-idl: validationerror derives Fail * lucet-idl: prelude tests use PackageBuilder which makes the thing being tested less obvious, but the invariant does have to hold across these two modules. not a great design tbh * lucet-idl: derive default unless contains enum * lucet-idl: use MaybeUninit instead of null for struct member offset assertions * lucet-idl: get rid of Deref impls of Datatype rust should have a non-deref way to delegate method calls to a member * lucet-idl: many code review fixes * delete empty file * lucet-idl: get rid of tons of unnecessary refs/clones of Location * lucet-idl: fixup atom/abi relationships allow i64 abi to represent smaller types * lucet-idl: fix error messages in binding validation tests and add an extra test case or two * lucet-idl-test: fix stub * lucet-idl-test: fixes * lucet-idl: separate parse trees for package-level and syntax-level decls rather than a recursive parse tree. * lucet-idl: finish fixing parser split-up * lucet-idl-test: split test plan into separate module; fixes * lucet-idl: render rust cast to boolean as != 0 * lucet-idl-test: implement trivial test plan * lucet-idl: func validator review comments * lucet-idl: proptest found bug in parser :) * bump cargo.lock * Update the bincode crate to version 1.1 (bytecodealliance#274) Using an old version prevents applications using the lucet crate from also using bincode 1.1. * delete faerie submodule * upgrade to faerie 0.11, and latest cranelift, target-lexicon, wasmparser etc * cargo install with debug to avoid full release build times * Fixes bytecodealliance#288 - Remove cranelift-codegen from the dependencies of the lucet-runtime components such as lucet-module * Fix a few typos * lucet-idl: parse and validate `witx` format (bytecodealliance#275) * lucet-idl: competent sexpr parser * lucet-idl: start implementing grammar from https://github.com/WebAssembly/WASI/pull/74/files * lucet-idl: add dan's wasi_unstable.wati as a test case original commit, from https://github.com/sunfishcode/wasi: commit 383385700a13acc2bc4ad2ac7e78a445d9053d21 (HEAD, refs/remotes/sunfishcode/moduletype) Author: Dan Gohman <[email protected]> Date: Wed Aug 14 22:42:24 2019 -0700 A wasi_unstable description based on module types. This sketches out a description of the current wasi_unstable API, using a syntax which anticipates the "module types" syntax [recently proposed to the CG](https://github.com/WebAssembly/meetings/blob/master/2019/CG-08-06.md#discuss-new-proposal-that-introduces-types-for-modules-and-instances-and-text-format-thereof-as-initially-discussed-in-design1289), though it does use a few extensions which can be easily lowered, and which mostly anticipate the [Interface Types](WebAssembly/interface-types#47 (comment)) proposal. This is derived from [the WatIDL proposal](WebAssembly/WASI#64), though it differs in a few ways: - It doesn't yet do anything special for capabilities, for now. File descriptors are just integers, for now. But the idea is that we could add OCAP concepts later. - It uses a new `(flags ...)` construct in place of structs of Bool fields, to describe flags values. This allows us to explicitly declare the underlying wasm type for flags values. - Types used by the API are declared outside of the moduletype, because we're using wat syntax as much as possible, and there, `(type ...)` inside a module declares an entry in the type section, which isn't what's intended here. The extensions to module types are: - It uses `string`, `array`, and `{s,u}{8,16,32,64}` types, which are expected to be provided in some form by the Interface Types proposal. For now though, they can all be lowered in straightforward ways, by assuming that `string` is always just UTF-8. - It adds `typename` declarations, which can name any wasm type, and of the extra types mentioned in the previous item, or `struct`, `enum`, or `flags`. - It declares functions with multiple return values, which for now will need to be lowered to output (pointer) parameters. * lucet-idl: enough infra to feed wasi_unstable.wati through parser * lucet-idl: fix typo in wasi_unstable lexer found it! cheers * lucet-idl: better error reporting * lucet-idl: parse a bunch more of wati * lucet-idl: @interface is an Annot("interface") * lucet-idl: switch from wasi_unstable.wati to multiple .witx * lucet-idl: interface-types idl files are now called .witx webassembly interface types has claimed the `.wit` file extension for describing just module interfaces. this idl is currently the `.wit` syntax, plus extensions. in lieu of a better name, dan and I decided `.witx` will work for the time being. * lucet-idl: renames of stuff in witx, plus use stmts * lucet-idl: typenames.witx had an extra close paren * lucet-idl: correct path to typenames in wasi_unstable.witx * lucet-idl: change parser to own everything. start implementing use * lucet-idl: witx use directive works! * lucet-idl: better error messages in parser through lookahead * lucet-idl: parser makes it through wasi_unstable * lucet-idl: wasi_unstable.witx: fix two syntax errors in * lucet-idl: witx use decls are now like #pragma once so we dont have to care about cycles, and diamond dependencies will be handled correctly * lucet-idl: witx use resolution moved to toplevel.rs * lucet-idl: make an AST for witx * lucet-idl: rename interfacetypes to witx * lucet-idl witx: file name goes in location, location throughout parse tree * lucet-idl witx: validator is filling out * lucet-idl witx: typenames.witx reorg so names are defined before use this restriction makes it possible to perform validation linearly, which is in line with validation elsewhere in wasm * lucet-idl witx: validation of all datatypes complete * lucet-idl witx: finish validation * lucet-idl witx: fix tests * lucet-idl: fix tests * lucet-idl witx: syntax derives Eq. mock the filesystem to test use more tests required * lucet-idl witx: some fixups, plenty of toplevel tests * lucet-idl witx: add docs, refactor error name, resolve warnings * lucet-idl: add test case to load wasi_unstable.witx * lucet-idl: rustfmt * lucet-idl witx: review comments * lucet-idl witx: really fix the path thing this time * lucet-idl witx: simplify paths one more time not my favorite corner of std!! * split lucet-idl and witx-frontend into two separate crates * Exit dev shell scripts on first command exiting with non-zero return (bytecodealliance#298) * Exit dev shell scripts on first command exiting with non-zero return Fixes issue bytecodealliance#297 * Create stages for Travis * Remove set -e from devenv scripts * add version info to modules version information is comprised of both the current crate version and the current git commit hash (if available). the current git commit hash is only used in release builds to avoid too much furstration in typical development workflows using tools like "git commit --amend" or "git rebase", or just making non-conflicting spot changes to only one of lucetc or lucet-runtime * review comments * Fixes bytecodealliance#299 - WasiCtx should open dev/null read write (bytecodealliance#302) * need to go through the ELF view to read the tables slice too we were going through the mapper to read each table, but incorrectly assumed the pointer in the artifact to look at didn't need translation too * add a CallStackBuilder abstraction to tease apart Context::init also adds documentation on what the guest stack looks like when initialized * upgrade to cranelift 0.43.1, wasmparser 0.39.1 * cranelift 0.44.0 * lucet-benchmarks: use module-level constant to explicitly pin OptLevel * lucet-validate: initial commit * lucet-validate: build up moduletype from parser * lucet-validate: print out type sig of all imports, start func sig * lucet-validate: better api to witx crate is in a branch * upgrade to cranelift 0.43.1, wasmparser 0.39.1 * lucet-validate: calculate module type of InterfaceFunc * lucet-validate: use release of wasmparser 0.39.1 * lucet-validate: parse module type of wasi sdk test program * lucet-validate: use wasi crate to validate against spec * lucet-validate: cranelift 0.44.0 * lucet-wasi-sdk: correct signature of `main` in test c code * lucet-validate: use witx spec from WASI repo as a submodule * lucet-validate: wip... starting to validate sigs against witx * lucet-validate: all lucet-wasi test cases now validate * move the stack guard page between the stack and heap where it was protecting against stack underflow into globals, which are extremely unlikely. Far more likely, are stack overflows into the instance's heap, where we accidentally caught those via heap guard space instead. If an instance's `Limits` had the number of heap guard pages as 0, which is permissible, an instance with full utilization would then map a page that had been acting as a stack guard page as read/write and remove the guard-ness of that page. This commit moves a page to be an outside-of-heap dedicated stack guard page. * adjust signal handler to not always be so stack-hungry Because the Lucet signal handler is run on any arbitrary thread that recives a signal, if the signal handler is installed anywhere in the process, it may be run on a signal stack we didn't set up. This means we are still bound to be careful about how much stack is present, which will often be SIGSTKSZ (observed to be ~8kb on Debian-likes, at least). `handle_signal` is the entrypoint for any signal in the process, so its call frame, locals, etc, are active for any path through the signal handler. For other paths through the signal handler that may involve subsequent calls, and in debug builds of lucet-runtime, the body of the `SignalBehavior::Default` arm would include several `UContext` local variables, as well as a `State` local which itself contains another `UContext`. x86_64 `UContext` are large, resulting in this match arm being responsible for ~5kb of stack usage in `handle_signal`, that is only _actually_ used in this match arm. So, by moving the arm to a closure and immediately calling it, we force Rust to defer that stack usage in debug builds to only be used when the closure is called. In release builds, the closure is likely inlined, but all intermediate copies of UContext _probably_ go away and the entire issue is _probably_ avoided. A subsequent commit will introduce tests around signal handler stack use for these various paths. * switch sigprocmask to pthread_sigmask `pthread_sigmask` is the safe choice for a multithreaded environment. Per `sigprocmask(2)`: > The use of sigprocmask() is unspecified in a multithreaded process; see pthread_sigmask(3). * switch to nightly-2019-09-25, the most recent with all components * lucet-validate: witx_moduletype is now part of witx crate * makefile: add lucet-validate to tests * lucet-validate: latest witx branch, where module types renamed core types * use goblin 0.0.24 everywhere * [lucet-idl] parameterize C test with `WASI_SDK` env var * use lucet-validate in lucetc * lucetc: use lucet-validate Validator * changes to lucetc threaded throughout repo and lucet-wasi-sdk validates that imports work according to witx! * delete witx-frontend: subsumed by wasi repo * upgrade to witx 0.3.0 * delete lucet-validate's wasi-sdk test: that testis now inside lucet-wasi-sdk * lucet-validate: test harness uses new api * Make WASI `poll` test more deterministic, but disable pending debugging (bytecodealliance#320) - Removes a sleep that wasn't relevant for testing poll - Timeout step waits on an in-process pipe rather than stdin, which should make its behavior more consistent across environments - Temporarily ignores the test while we debug its behavior in certain CI environments * address review comments * lucet-validate: add readme * [lucet-validate] ignore non-c/wat files rather than failing test Ephemeral files left around by editors were causing a panic. * package lucet-validate, including wasi unstable witx * rename `lucet-analyze->lucet-objdump`; add debs for it and `lucetc` * skip Lucet build in the container step on travis * 0.2.0 * make bump-global-version.sh gnu sed-friendly -i.previous should be usable on either linux sed or macos sed, but the space leads to linux sed interpreting it as `-i`, which is a valid flag on its own, and `.previous`, as a sed command. * lucetc: fix --opt-level {none, speed, speed_and_size} (bytecodealliance#328) * cargo publish requires versions on dev dependencies to publish * lucet-wasi/README.md doesn't exist any more, don't try to install it (bytecodealliance#330) * dockerfile: use wask-sdk 7 * lucet-wasi-sdk: fix test to reflect behavior of clang-9 * the runtime's module version check was restrictive enough to be annoying In the case that the runtime is a debug or from-cargo version, there's no commit hash, so a build from any lucetc that can include a commit hash would immediately fail the version check. Since lucetc may be installed by `cargo install`, which will build release binaries, this is more likely than initially expected. * grammar * rustfmt * configure CI using Github Actions there are two workflows: * workflows/main.yml defines the Test and Lint jobs, which run in parallel on each push and PR. * Test takes about 19 minutes to complete. * Lint takes about 45 seconds to run `make indent-check`. * workflows/fuzz.yml defines the Fuzz job, which runs nightly at midnight. It only runs on the master branch, so we need to merge this PR to test it. The fuzz smoke-test passes, so I expect that this will work. We will probably have to add some system to report failures it discovers. We may want to run it more often, as well. there is one disabled workflow: * workflows/mac-ci.yml.disabled will run the CI, except for fuzzing, under Mac OS. It is disabled because tests fail on Mac OS at the moment. It should be enabled as part of fixing Mac OS. there is one Action: * actions/test specifies (in action.yml) a job that builds the Dockerfile at the repo root and runs `make test` in the container. The Test job uses the Dockerfile in the repo, but not the devenv scripts to run it. Actions only runs Docker via the This means the devenv scripts themselves have no CI under Actions. Once we have Mac OS support enabled again, there is a lot less reason to maintain the complex set of devenv scripts, which do a lot of fancy and difficult-to-understand stuff to allow using `lucetc`, `lucet-wasi` and other executables on the Mac (or windows, I guess). We could instead document the one-liner invocations of Docker to reproduce our standard developer environment locally. * bump lucet crates to 0.2.1 * Improve lucetc error messages (bytecodealliance#269) * move off debug repr for error reporting this still prints the underlying error, like a wabt error in the case of invalid webassembly text, but does so in a more readable way * Timeouts (bytecodealliance#150) * add termination mechanism and use it to implement timeouts The termination model is described in the `execution` module-level docs, but to reiterate: when in guest code, terminate with a SIGALRM sent immediately, and when in host code, terminate by setting a flag (setting `execution_domain` to `Terminated`) and checking that flag when exiting a hostcall. Hostcalls currently have their safety, with respect to termination, implemented by running the whole hostcall in `uninterruptable`, which effectively makes all hostcalls critical sections. In the future, we may want to expose the idea of interruptable hostcalls, both to avoid overhead of synchronizing `KillState` when entering an exiting hostcalls, and to avoid delayed termination. In the extreme, even though this adds a tool for timeout-style logic, it is at worst a request to stop execution, which requires hostcalls to complete in a reasonable amount of time to guarantee that timeouts are always witnessed promptly. * [lucetc] add very basic JSON formatting for errors This is meant to be a starting point for more semantic error reporting, but in order to get there we'll need to do a broader refactor of our error types * [lucet-runtime] relax unnecessary restrictions on some functions - The `Send + Sync` restriction on the downcast methods was leftover from before the `State` refactor lifted those restrictions from `YieldedVal`s - The mutable borrow on `Instance::get_embed_ctx_mut()` is checked at runtime by the `RefCell`s inside the underlying map. This patch gives us fewer compile-time guarantees, but holding a mutable borrow on the entire instance just to get at an embedding context is way too strict * tests run again * fix the number of magic globals that wasi-sdk clang creates * fix lucet-wasi-sdk test for wask-sdk-7 * allow KillState's `exit_guest_region` to be name-mangled * Bump versions to 0.3.0 (bytecodealliance#340) * 0.3.0 * lucetc error messages include the underlying error from wabt * lucet-runtime supports timeouts * lucetc can emit errors as json * lucet-runtime removes unnecessary restrictions on some functions wabt update is necessary to avoid linker issues with multiple wabt in use at once * [lucet-runtime] add `is_yielded`, etc methods for instances Previously a client would have to keep around the `RunResult` to know which state an instance is in. * [lucetc] json errors report as an array to support multiple errors We don't have the capability to produce multiple errors yet, so this only returns an array of a single element. Returning multiple errors will require refactoring of our code as well as upstream dependencies. * add (currently-failing) test for moving yielded instances * fix the sending-yielded-instances bug in horrendous fashion * move guest data from the guest stack to Instance * run the benchmarks in debug mode for `make test` * [lucet-runtime-internals] refactor guest-specific context data This ended up changing around the interface to initializing and swapping contexts in some important ways. Notably, a pointer to the parent context that `lucet_context_bootstrap` swaps into when the entrypoint function returns is no longer written into the guest stack, but rather is a field on `Context`. This field gets updated in the `to` context when calling `Context::swap()`, so that we can swap `to` a context from different `from` contexts without trying to always swap back to the original context. This also gets us ready to abstract away instance/timeout-specific data from the `Context` struct by making the use of a backstop callback function explicit but optional. * [lucet-benchmarks] slightly simplify a setup iter * bump crate versions to 0.3.1 Keeping this as a minor semver because API changes were restricted to `lucet-runtime-internals` functions that are not exported via `lucet-runtime`. * [lucetc] restore output for the underlying cause of `lucetc::Error`s * update cranelift to current master (0.46.1) This is in preparation for adding an interface to customize machine-dependent features in cranelift's codegen * delete .travis.yml * [lucetc] add CPU feature flags for code generation This introduces a couple new command-line arguments and the associated programmatic APIs for selecting optional CPU features at compile time. Previously, we used `cranelift-native` to automatically detect and use the features available on the host, but this can cause problems when compiling for a different CPU, even if the target triple is constant. Now, we can enable or disable individual features using `--target-feature=+sse3,+avx,-popcnt`, for example. For convenience, we can also specify `--target-cpu=haswell`, for example, which enables a fixed set of individual features. The default behavior of auto-detection and use of features remains the same for now. * [lucetc] add `native` and `sandybridge` CPU profiles This required some reworking of how `CpuFeatures` is represented. * fix rustfmt * crank up the fuzz * Include CPU features in module data (bytecodealliance#351) * write cpu features into modules and verify compatibility when loading * build spectests to ward away evil bit gremlins * bump to 0.4.0 * Use app_from_crate!() instead of hardcoding clap base parameters * Add include_str!("Cargo.toml") to update macros if Cargo.toml changes * bump to 0.4.1 and refine some dependencies There were some problems when trying to publish the crates due to parity-wasm versions differing in our crates and Cranelift's. I went through and updated and/or fixed the versions of some crates that look like they could've caused trouble. * use crates.io version of wasmonkey, and fix parity-wasm version * Replace lucet-wasi with wasi-common + lucet wrappers * lucet-wasi: use pch/working branch of wasi-common * lucet-wasi: export our own __wasi_exitcode_t alias, use master upstream * lucet-wasi: re-export wasi-common's error * lucet-wasi: export start symbol * use nix 0.15 everywhere wasi-common depends on nix 0.15, so upgrade everywhere using 0.13.1 * latest wasi-common master * pin wasi-common to f4ac1299b2001b575cfc99f5544a4a96355a6bff in cargo.toml * Apply suggestions from code review Thank you Adam for a helpful and thorough code review! Co-Authored-By: Adam C. Foltzer <[email protected]> * heap_u32_mut: require a mutable heap (bytecodealliance#359) * replace lucet_runtime ensure_linked in lucet-wasi fixes bytecodealliance#360 where the runtime's C API was being discarded by the linker. * Make it a Bytecode Alliance project * read function names from custom name section * minor grammar fix in runtime docs * specify signature size for module data * pr review: use exported names * pr review: use SecondaryMap for names * delete lucet-idl. superceded by witx * remove lucet-idl from bump-global-version * add `#[lucet_hostcall]` attr macro; deprecate `lucet_hostcalls!` The attribute macro is much more flexible syntactically, and in particular allows hostcall implementations to be properly `rustfmt`ed rather than being stuck in the invocation of a macro. * bump to 0.4.2 Deprecating an existing API and adding a new one are both minor semver changes * redefine `lucet_hostcalls!` in terms of `#[lucet_hostcall]` Prompted by @jedisct1's comment, this will hopefully stave off bitrot as we deprecate `lucet_hostcalls!`. Also tweaks a comment in response to @data-pup's review. Thank you both! * remove some unnecessary cloning in `#[lucet_hostcall]` * use `lucet_runtime` rather than `$crate` in `lucet_hostcalls!` * upgrade cranelift, wasi-common, witx deps, and friends * pull in latest Cranelift release 0.51 * upgrade to wasi-common 0.7, which fixes a lot of the WasiCtxBuilder stuff that required many extra unwraps * pull in latest WASI repo. This moves the preview0 stuff to another directory. Biggest change here is that the lucetc deb, which includes the wasi witx files, has a slightly different file layout now. * transitive upgrades to target-lexicon, faerie * lucet-validate: correct deb paths * CI testing: add cargo-deb packaging * CI: run `make package` on a parallel runner, so CI doesnt get longer * Use actions/checkout@v1 to support submodules checkout This commit fixes the CI by setting `actions/checkout` at version `v1`. The current master of `actions/checkout` now obsoleted the `with: submodules: true` input. See [actions/checkout/releases/tag/v2-beta] for more info. [actions/checkout/releases/tag/v2-beta]: https://github.com/actions/checkout/releases/tag/v2-beta * show unsupported global import in error message * misc. instance lints * replace empty constructor w/ default impl * we can derive default * keep `new()` but defined in terms of default * Make yield methods take `&mut Vmctx`; update semver This prevents resources like embedder contexts or heap views from living across yield points, which is important for safety since the host can modify the data underlying those resources while the instance is suspended. Since this is a semver breaking change, this patch also updates the package versions. * Fixes bytecodealliance#276 - Allow use of aligned heaps * add blank rustfmt config file * into_iter() -> iter() for arrays (bytecodealliance#382) This was previously accepted by the compiler but is being phased out; it will become a hard error in a future rustc release. See rust-lang/rust#66145 > * switch rustc version back to stable; add note about callback safety * add `make test-full` target; make `test` more focused The benchmark and fuzz targets in particular take quite a long time to run, and are usually not the focus of development. They're still worth running in CI, though, to catch bitrot. * clean up symbol data code * move bindings tests into file in tests/ * fix test assertion * ModuleDecls cleanup (bytecodealliance#391) * Addresses two lint warnings about redundant field names, using the shorthand notation instead. * Moves `*_name_for` helper functions nested inside of `declare_funcs` out of a loop body. This is just a small readability fix for our loop. * add `--target` option to lucetc Currently, `lucetc` assumes that the machine that it's compiling for (the "target") is the same as the machine on which `lucetc` is running ("the host"). While this is a reasonable assumption to make, many future uses of `lucetc` may require these machines to be distinct. Towards that end, let's introduce a `--target` option for specifying the machine `lucetc` is supposed to be generating code for, and thread that information through to all the places that require it. * [lucet-runtime] export the `KillSwitch` type from the public API It was already possible to use via the `Instance::kill_switch()` method, but couldn't be named in a type signature or a type declaration. * Fixes bytecodealliance#357: Migrate from using the failure to thiserror + anyhow. (bytecodealliance#374) Co-authored-by: Pat Hickey <[email protected]> * use exact dependencies for crates within the repo Most of our thinking about semver for this project is concerned with the more "public-facing" crates like `lucetc`, `lucet-runtime`, and `lucet-wasi`, as opposed to "implementation detail" crates like `lucet-module`, `lucet-runtime-internals`, etc. As the project matures, we are paying more attention to semver for all of the crates, but this change makes the released sets of packages more consistent. * Fix bad merge in cpu_features.rs. ISA lookup must be invoked on target (bytecodealliance#399) Co-authored-by: Pat Hickey <[email protected]> Co-authored-by: Frank Denis <[email protected]> Co-authored-by: awortman-fastly <[email protected]> Co-authored-by: Shravan Narayan <[email protected]> Co-authored-by: Gabor Greif <[email protected]> Co-authored-by: Eric Kilmer <[email protected]> Co-authored-by: Adam C. Foltzer <[email protected]> Co-authored-by: Tyler McMullen <[email protected]> Co-authored-by: Till Schneidereit <[email protected]> Co-authored-by: data-pup <[email protected]> Co-authored-by: Jakub Konka <[email protected]> Co-authored-by: Nathan Froyd <[email protected]> Co-authored-by: iximeow <[email protected]> Co-authored-by: Tanya L. Crenshaw <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WatIDL
/ˈwädl/🦆WatIDL is strawman text format for Wasm interfaces. This is an attempt to aggregate feedback from PR #58
A quick overview
I didn't full flush out how the binding would look like but I think it will be fairly straight since the is a pretty straight forward mapping between watIDL type and webIDL types.