arm64-specific: wasm fails to instantiate: out of bounds memory access

[davidmdm]

Describe the bug
Running a wasm binary returns the error:

error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
wasm stack trace:
        .time.__dataIO_.big8(i32) i32
        .github.ghproxy.top_BurntSushi_toml_internal.init(i32) i32
        .runtime.doInit1(i32) i32
        .runtime.main(i32) i32
        .wasm_pc_f_loop()
        ._rt0_wasm_wasip1()

To Reproduce
Running on Darwin/arm64 M2 chip

set -x

if [ ! -d "yoke" ]; then
  git clone --depth=1 https://github.com/yokecd/yoke.git
fi

(
  cd yoke
  GOOS=wasip1 GOARCH=wasm go build -o ../redis-installer.wasm ./examples/redis
)

# will run successfully
go run github.com/tetratelabs/wazero/cmd/[email protected] run ./redis-installer.wasm

# will fail with aforementioned error
go run github.com/tetratelabs/wazero/cmd/[email protected] run ./redis-installer.wasm

Expected behavior
Expected same output as [email protected]

Environment (please complete the relevant information):

• Go version: 1.24.0
• wazero Version: 1.9.0
• Host architecture: arm64
• Runtime mode: compiler

[evacchi]

can you run this with amd64 emulation to see if it's specific to arm64? this should do it:

GOARCH=amd64 go run github.com/tetratelabs/wazero/cmd/[email protected] run ./redis-installer.wasm

[davidmdm]

@evacchi it works under amd64 emulation. So its a bug specific to arm64!

[davidmdm]

Perhaps still a trampoline issue? I remember when certain programs would hang in version 1.7.x and that a fix was made to the arm64 backend then. Perhaps an off-by-one of sorts is causing a memory out of bounds issue?

[evacchi]

@davidmdm out of curiosity, have you tried against Go 1.23? let's rule out something related to #2375

[davidmdm]

@evacchi Just ran against a version with the wasm produced by go1.23.6 and it produces an identical error.
I checked the new wasm did not match the one produced by go1.24.0 as well to be sure.

[ncruces]

I don't think that's what @evacchi asked for. Don't build the Wasm with 1.23, rather use 1.23 to go run wazero.

[davidmdm]

@ncruces Good catch. I got that confused.

@evacchi Same result unfortunately!

GOTOOLCHAIN=go1.23.6 go run github.com/tetratelabs/wazero/cmd/[email protected] run ./redis-installer.wasm

go: downloading go1.23.6 (darwin/arm64)
error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
wasm stack trace:
        .time.__dataIO_.big8(i32) i32
        .github.ghproxy.top_BurntSushi_toml_internal.init(i32) i32
        .runtime.doInit1(i32) i32
        .runtime.main(i32) i32
        .wasm_pc_f_loop()
        ._rt0_wasm_wasip1()
exit status 1

[evacchi]

interesting, so unrelated issue

[evacchi]

FWIW 😅

❯ go run github.com/tetratelabs/wazero/cmd/[email protected] run -env GODEBUG='inittrace=1' ./redis-installer.wasm
init runtime @0.006 ms, 0.024 ms clock, 0 bytes, 0 allocs
init crypto/internal/fips140deps/cpu @0.16 ms, 0.004 ms clock, 0 bytes, 0 allocs
init errors @0.18 ms, 0.005 ms clock, 0 bytes, 0 allocs
init iter @0.20 ms, 0.003 ms clock, 16 bytes, 1 allocs
init sync @0.21 ms, 0.007 ms clock, 0 bytes, 0 allocs
init internal/godebug @0.23 ms, 0.15 ms clock, 312 bytes, 7 allocs
init encoding/base32 @0.40 ms, 0.007 ms clock, 640 bytes, 2 allocs
init crypto @0.42 ms, 0.004 ms clock, 160 bytes, 1 allocs
init hash/crc32 @0.43 ms, 0.026 ms clock, 1264 bytes, 13 allocs
init syscall @0.47 ms, 0.044 ms clock, 272 bytes, 3 allocs
init time @0.52 ms, 0.008 ms clock, 256 bytes, 2 allocs
init context @0.54 ms, 0.010 ms clock, 112 bytes, 1 allocs
init github.com/BurntSushi/toml/internal @0.56 ms, 0.034 ms clock, 0 bytes, 0 allocs
error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
wasm stack trace:
        .time.Time.absSec(i32) i32
        .time.Time.Month(i32) i32
        .time.__Time_.Month(i32) i32
        .github.ghproxy.top_google_go_cmp_cmp_internal_diff.init(i32) i32
        .runtime.doInit1(i32) i32
        .runtime.main(i32) i32
        .wasm_pc_f_loop()
        ._rt0_wasm_wasip1()
exit status 1

[evacchi]

another "interesting" note, commenting out these lines in the frontend let it run correctly to end:

wazero/internal/engine/wazevo/frontend/lower.go

Lines 3668 to 3670 in c96893d

	exitIfNZ := builder.AllocateInstruction()
	exitIfNZ.AsExitIfTrueWithCode(c.execCtxPtrValue, cmp.Return(), wazevoapi.ExitCodeMemoryOutOfBounds)
	builder.InsertInstruction(exitIfNZ)

[evacchi]

❯ wasm-opt ../redis-installer.wasm -o ../redis-installer-opt.wasm -Os -all

❯ go run ./cmd/wazero run ../repro/redis-installer-opt.wasm
t.Kind == 0
panic: unknown type kind

goroutine 1 [running]:
reflect.(*abiSeq).regAssign(0x0, 0xa, 0x6259a0)
        /opt/homebrew/Cellar/go/1.24.0/libexec/src/reflect/abi.go:247 +0x59
strconv.Itoa(...)
        /opt/homebrew/Cellar/go/1.24.0/libexec/src/strconv/itoa.go:35
github.com/davidmdm/ansi.Esc({0x246ce08, 0x1, 0x1})
        /Users/evacchi/.local/go/pkg/mod/github.com/davidmdm/[email protected]/style.go:15 +0x5
github.com/davidmdm/ansi.init()
        /Users/evacchi/.local/go/pkg/mod/github.com/davidmdm/[email protected]/style.go:20 +0x8
exit status 2

🤪

[evacchi]

probably a red herring, I think most of what wasm-opt did in this is case is removing DWARF symbols, and with -env GODEBUG=inittrace=1 it still crashes with OOB

error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
wasm stack trace:
        .$2211(i32) i32
        .$2183(i32) i32
        .$2184(i32) i32
        .$2254(i32) i32
        .$2321(i32) i32
        .$47976(i32) i32
        .$1098(i32) i32
        .$951(i32) i32
        .$1667()
        .$1671()

[evacchi]

ok here's what I have learned so far. In function #2320 (big8) register (x11) gets assigned a value that's higher than the given bound:

     x15 = 0x0000000028d90004
     x14 = 0x0000000002800000

and it "correctly" fails.
this is not even a scaling issue because it would be still 0x28d9000 > 0x2800000

the value is loaded waaay earlier in x10 from [x10] and then eventually copied to x11 (it goes through a bunch of spills and reloads too):

    0x1285db1f4: b      0x1285db1f4
    0x1285db1f8: add    x10, x12, x10
    0x1285db1fc: ldur   x10, [x10]

before the load, x10 = 0x000001402776adf8

in post-regalloc, this appears to be block L148:

L148:
        add x10, x12, x10
        ldr x10, [x10]

if I elide the bound check it is hard to track what truly is different because many branches obviously disappear (the bound checks themselves), some loads fold UXTW into the addressing mode, and all registers are thus different

[davidmdm]

I don't have the technical expertise to add anything, but I just wanted to let you know that I appreciate your updates!

[evacchi]

upon further inspection GODEBUG=initrace=1 causes function big8 to terminate correctly, so now I suspect the issue might be we jump to an incorrect offset. In fact, I added an explicit breakpoint at the basic block that contains the failing check (L148) and we never get to that block.

From what I can tell from the generated code, that block has only 1 or 2 predecessors, and neither are explicitly present: but there is a table macro instruction br_table_sequence (in fact big8 expands to a large function with a br_table and a lot of nested blocks).

So my suspect now is that something goes wrong in that jump table...; indeed it is a big jump table:

br_table 0 0 0 0 0 0 0 0 1 2 3 3 4 4 5 5 5 6 7 7 8 8 9 9 10 10 10 10 10 10 10 11 12 12 13 13 13 14 14 15 15 15 15 16 17 17 18 18 19 19 20 20 21 22 23 24

this in turn might be related to ResolveRelativeAddress and insertConditionalJumpTrampoline -- but that might not explain why this won't break when GODEBUG=initrace=1 and instead it breaks some place else (although still in the time package...)

[evacchi]

ok, some degree of progress, big8 can't be that big, and in fact with a little help from our friend GOSSAFUNC

GOSSAFUNC="time.(*dataIO).big8" GOOS=wasip1 GOARCH=wasm go build -o ../redis-installer.wasm ./examples/redis

we see there is a bunch of inlining happening. I tried a flag to disable inlining earlier and I didn't get any success... but it turns out the flag has changed and the one I supplied was just being ignored :D

so here's the right incantation:

GOOS=wasip1 GOARCH=wasm go build -gcflags='all=-N -l' -o ../redis-installer.wasm ./examples/redis

surprise:

❯ go run ./cmd/wazero run ../repro/redis-installer.wasm
panic: value method time.Time.Local called using nil *Time pointer

goroutine 1 [running]:
time.(*Time).Local(0x0)
        <autogenerated>:1 +0x7
github.com/BurntSushi/toml/internal.init.func1()
        /Users/evacchi/.local/go/pkg/mod/github.com/!burnt!sushi/[email protected]/internal/tz.go:32 +0x2
github.com/BurntSushi/toml/internal.init()
        /Users/evacchi/.local/go/pkg/mod/github.com/!burnt!sushi/[email protected]/internal/tz.go:32 +0x3
exit status 2

and with GODEBUG='inittrace=1':

❯ go run ./cmd/wazero run -env GODEBUG='inittrace=1' ../repro/redis-installer.wasm
init runtime @0 ms, 0.033 ms clock, 0 bytes, 0 allocs
init crypto/internal/fips140deps/cpu @0.17 ms, 0.004 ms clock, 0 bytes, 0 allocs
init errors @0.19 ms, 0.024 ms clock, 16 bytes, 1 allocs
init internal/oserror @0.23 ms, 0.009 ms clock, 80 bytes, 5 allocs
init iter @0.26 ms, 0.005 ms clock, 16 bytes, 1 allocs
init math/rand/v2 @0.27 ms, 0.005 ms clock, 16 bytes, 1 allocs
init path @0.29 ms, 0.003 ms clock, 16 bytes, 1 allocs
init strconv @0.31 ms, 0.006 ms clock, 32 bytes, 2 allocs
init sync @0.32 ms, 0.010 ms clock, 0 bytes, 0 allocs
init internal/godebug @0.35 ms, 0.23 ms clock, 312 bytes, 7 allocs
init io @0.60 ms, 0.010 ms clock, 144 bytes, 9 allocs
init encoding/base32 @0.62 ms, 0.016 ms clock, 640 bytes, 2 allocs
init crypto @0.64 ms, 0.003 ms clock, 160 bytes, 1 allocs
init hash/crc32 @0.66 ms, 0.055 ms clock, 1264 bytes, 13 allocs
init math/rand @0.72 ms, 0.007 ms clock, 96 bytes, 2 allocs
init syscall @0.74 ms, 0.046 ms clock, 272 bytes, 3 allocs
init time @0.80 ms, 0.017 ms clock, 384 bytes, 8 allocs
init context @0.83 ms, 0.010 ms clock, 128 bytes, 2 allocs
init github.com/BurntSushi/toml/internal @0.85 ms, 0.032 ms clock, 48 bytes, 3 allocs
error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
wasm stack trace:
        .math_rand.__rngSource_.Uint64(i32) i32
        .github.ghproxy.top_google_go_cmp_cmp_internal_diff.init(i32) i32
        .runtime.doInit1(i32) i32
        .runtime.doInit(i32) i32
        .wasm_pc_f_loop()
        ._rt0_wasm_wasip1()
exit status 1

this is great news because I think the root cause of the issue is the same (same weird difference in crashing behavior when inittrace=1), but at the same time the crash with no inittrace is a nil value which might be slightly easier to pinpoint!

[evacchi]

so I have some very interesting news, my old patch does not seem to have the bug #2169
this means:

1. the bug is still with our friends, the relocations/trampoline islands
2. the bug has likely to do with the fact there are a lot of these islands and a lot of functions

in fact, while the inlined Wasm binary is ~68MB the -gcflags='all=-N -l' is obviously larger with way more functions: 104MB

EDIT: it's not a lot of islands, it's only 4:

Trampoline interval=67108861 (0x3fffffd), trampoline size = 1033020 (0xfc33c)
trampoline offset: 67119992 0x(4002b78)
trampoline offset: 134220684 0x(8000b8c)
trampoline offset: 201339748 0x(c003364)
trampoline offset: 268437428 0x(100007b4)