invalid table access only with Go 1.24 and amd64

[anuraaga]

Describe the bug

go-re2 usage crashes only with Go 1.24 on amd64. I guess this must be some interaction with different codegen in 1.24 on that platform. I believe it is happening when invoking a wasi host function and may be trampoline related. It is crashing right away with no concurrency or anything special.

❯ GOARCH=amd64 ~/tmp/go/bin/go test .
--- FAIL: TestBadCompile (1.34s)
panic: wasm error: invalid table access
        wasm stack trace:
                libcre2-noopt.so.fwrite(i32,i32,i32,i32) i32
                        0xc6b: wasisdk:/v25.0/build/sysroot/wasi-libc-wasm32-wasip1-threads/libc-top-half/musl/src/stdio/fwrite.c:10:36 (inlined)
                               wasisdk:/v25.0/build/sysroot/wasi-libc-wasm32-wasip1-threads/libc-top-half/musl/src/stdio/fwrite.c:33:6
                libcre2-noopt.so.absl::lts_20240722::log_internal::WriteToStderr(std::__2::basic_string_view<char, std::__2::char_traits<char>>, absl::lts_20240722::LogSeverity)(i32,i32)
                        0x38259: /re2/abseil-cpp/absl/log/internal/globals.cc:79:3
                libcre2-noopt.so.void absl::lts_20240722::base_internal::CallOnceImpl<absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)::'lambda'()>(std::__2::atomic<unsigned int>*, absl::lts_20240722::base_internal::SchedulingMode, absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)::'lambda'()&&)(i32)
                        0x3d0e7: /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:104:7 (inlined)
                                 /opt/wasi-sdk/share/wasi-sysroot/include/wasm32-wasip1-threads/c++/v1/__type_traits/invoke.h:149:25 (inlined)
                                 /opt/wasi-sdk/share/wasi-sysroot/include/wasm32-wasip1-threads/c++/v1/__functional/invoke.h:28:10 (inlined)
                                 /re2/abseil-cpp/absl/base/call_once.h:182:5
                libcre2-noopt.so.absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)(i32,i32)
                        0x3cfd2: /re2/abseil-cpp/absl/base/call_once.h:216:5 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:99:5
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogToSinks(absl::lts_20240722::LogEntry const&, absl::lts_20240722::Span<absl::lts_20240722::LogSink*>, bool)(i32,i32,i32)
                        0x3caec: /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:260:13 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:201:9 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:281:31
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::SendToLog()(i32)
                        0x472cd: /re2/abseil-cpp/absl/log/internal/log_message.cc:551:3
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::Flush()(i32)
                        0x471c5: /re2/abseil-cpp/absl/log/internal/log_message.cc:465:3
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::~LogMessage()(i32) i32
                        0x46f9f: /re2/abseil-cpp/absl/log/internal/log_message.cc:299:3
                libcre2-noopt.so.re2::RE2::Init(std::__2::basic_string_view<char, std::__2::char_traits<char>>, re2::RE2::Options const&)(i32,i32,i32)
                        0x6fd7a: /re2/re2/re2.cc:237:7
                libcre2-noopt.so.re2::RE2::RE2(std::__2::basic_string_view<char, std::__2::char_traits<char>>, re2::RE2::Options const&)(i32,i32,i32) i32
                        0x705b6: /re2/re2/re2.cc:158:3
                libcre2-noopt.so.cre2_new(i32,i32,i32) i32
                        0x756c0: /cre2/cre2.cpp:135:27 [recovered]
        panic: wasm error: invalid table access
        wasm stack trace:
                libcre2-noopt.so.fwrite(i32,i32,i32,i32) i32
                        0xc6b: wasisdk:/v25.0/build/sysroot/wasi-libc-wasm32-wasip1-threads/libc-top-half/musl/src/stdio/fwrite.c:10:36 (inlined)
                               wasisdk:/v25.0/build/sysroot/wasi-libc-wasm32-wasip1-threads/libc-top-half/musl/src/stdio/fwrite.c:33:6
                libcre2-noopt.so.absl::lts_20240722::log_internal::WriteToStderr(std::__2::basic_string_view<char, std::__2::char_traits<char>>, absl::lts_20240722::LogSeverity)(i32,i32)
                        0x38259: /re2/abseil-cpp/absl/log/internal/globals.cc:79:3
                libcre2-noopt.so.void absl::lts_20240722::base_internal::CallOnceImpl<absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)::'lambda'()>(std::__2::atomic<unsigned int>*, absl::lts_20240722::base_internal::SchedulingMode, absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)::'lambda'()&&)(i32)
                        0x3d0e7: /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:104:7 (inlined)
                                 /opt/wasi-sdk/share/wasi-sysroot/include/wasm32-wasip1-threads/c++/v1/__type_traits/invoke.h:149:25 (inlined)
                                 /opt/wasi-sdk/share/wasi-sysroot/include/wasm32-wasip1-threads/c++/v1/__functional/invoke.h:28:10 (inlined)
                                 /re2/abseil-cpp/absl/base/call_once.h:182:5
                libcre2-noopt.so.absl::lts_20240722::log_internal::(anonymous namespace)::StderrLogSink::Send(absl::lts_20240722::LogEntry const&)(i32,i32)
                        0x3cfd2: /re2/abseil-cpp/absl/base/call_once.h:216:5 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:99:5
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogToSinks(absl::lts_20240722::LogEntry const&, absl::lts_20240722::Span<absl::lts_20240722::LogSink*>, bool)(i32,i32,i32)
                        0x3caec: /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:260:13 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:201:9 (inlined)
                                 /re2/abseil-cpp/absl/log/internal/log_sink_set.cc:281:31
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::SendToLog()(i32)
                        0x472cd: /re2/abseil-cpp/absl/log/internal/log_message.cc:551:3
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::Flush()(i32)
                        0x471c5: /re2/abseil-cpp/absl/log/internal/log_message.cc:465:3
                libcre2-noopt.so.absl::lts_20240722::log_internal::LogMessage::~LogMessage()(i32) i32
                        0x46f9f: /re2/abseil-cpp/absl/log/internal/log_message.cc:299:3
                libcre2-noopt.so.re2::RE2::Init(std::__2::basic_string_view<char, std::__2::char_traits<char>>, re2::RE2::Options const&)(i32,i32,i32)
                        0x6fd7a: /re2/re2/re2.cc:237:7
                libcre2-noopt.so.re2::RE2::RE2(std::__2::basic_string_view<char, std::__2::char_traits<char>>, re2::RE2::Options const&)(i32,i32,i32) i32
                        0x705b6: /re2/re2/re2.cc:158:3
                libcre2-noopt.so.cre2_new(i32,i32,i32) i32
                        0x756c0: /cre2/cre2.cpp:135:27

goroutine 10 [running]:
testing.tRunner.func1.2({0x3b8d7e0, 0xc001a76ec0})
        /Users/anuraag/tmp/go/src/testing/testing.go:1734 +0x21c
testing.tRunner.func1()
        /Users/anuraag/tmp/go/src/testing/testing.go:1737 +0x35e
panic({0x3b8d7e0?, 0xc001a76ec0?})
        /Users/anuraag/tmp/go/src/runtime/panic.go:787 +0x132
github.com/wasilibs/go-re2/internal.newRE(0xc0009e7e00, {0x3b487f8?, 0x0?}, {0x0?, 0x0?, 0x0?, 0x0?})
        /Users/anuraag/git/go-re2/internal/re2_wazero.go:301 +0x269
github.com/wasilibs/go-re2/internal.Compile({0x3b487f8, 0x1}, {0x28?, 0x93?, 0xd7?, 0x3?})
        /Users/anuraag/git/go-re2/internal/re2.go:67 +0x165
github.com/wasilibs/go-re2.Compile(...)
        /Users/anuraag/git/go-re2/re2.go:46
github.com/wasilibs/go-re2.compileTest(0xc000083dc0, {0x3b487f8, 0x1}, {0x3ae0295, 0x2c})
        /Users/anuraag/git/go-re2/all_test.go:51 +0x5a
github.com/wasilibs/go-re2.TestBadCompile(0xc000083dc0)
        /Users/anuraag/git/go-re2/all_test.go:71 +0x45
testing.tRunner(0xc000083dc0, 0x3bdf4b0)
        /Users/anuraag/tmp/go/src/testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 1
        /Users/anuraag/tmp/go/src/testing/testing.go:1851 +0x413
FAIL    github.com/wasilibs/go-re2      2.604s
FAIL

To Reproduce
Description of the host and wasm code that reproduces the behavior.
Smoothest debugging will be if you can share a repository with the
actual code.

With go 1.24, run go-re2 tests. debug-build branch has unattested debug binary, main will crash without symbols

git clone https://github.com/anuraaga/go-re2
cd go-re2
git checkout debug-build
GOARCH=amd64 go test .

Expected behavior

Doesn't crash

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the relevant information):

• Go version: [1.24.0]
• wazero Version: [1.8.2]
• Host architecture: [amd64]
• Runtime mode: [compiler]

Additional context

Originally reported in wasilibs/go-re2#190

Dump with host function logging - while I thought it may be only the first host function call that is affected, it appears there is one clock_time_get call that succeeds before the failing one within fwrite.

https://gist.github.com/anuraaga/83ef5402670e51f492bcd4b6f25014e8

[anuraaga]

For reference, I do get a crash on wazero tests using rosetta with go 1.24 that doesn't happen on 1.23

ok      github.com/tetratelabs/wazero/imports/assemblyscript    (cached)
ok      github.com/tetratelabs/wazero/imports/assemblyscript/example    (cached)
ok      github.com/tetratelabs/wazero/imports/emscripten        (cached)
--- FAIL: Test_fdReaddir_ls (1.98s)
    --- FAIL: Test_fdReaddir_ls/tinygo (0.06s)
        --- FAIL: Test_fdReaddir_ls/tinygo/directory_with_tons_of_entries (0.01s)
            require.go:331: expected no error, but was wasm error: out of bounds memory access
                wasm stack trace:
                        main.runtime.sliceAppend(i32,i32,i32,i32,i32,i32,i32)
                        main.main.printFileNames(i32,i32)
                        main._start():
                /Users/anuraag/git/wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:268
                /Users/anuraag/git/wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:238
                /Users/anuraag/git/wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:148
FAIL
FAIL    github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1    14.352s
ok      github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1/example    (cached)

But it doesn't seem to happen if I run on a github runner so maybe just rosetta-specific (above error happens on native too)

https://github.com/anuraaga/wazero/actions/runs/13302580071/job/37146554276?pr=2

[corani]

But it doesn't seem to happen if I run on a github runner so maybe just rosetta-specific (above error happens on native too)

Interesting, I tried running gitleaks (where I first reported the issue: gitleaks/gitleaks#1751) in a GitHub Action, and that indeed works. However, locally I'm on WSL2 so this isn't Rosetta-specific (though apparently not reproducible for everyone).

[corani]

FYI, on WSL2 with both go-1.24.0 and gotip-1.25-5ff7a634e this fails (but works with go-1.23.6):

$ go test ./imports/wasi_snapshot_preview1/
--- FAIL: Test_fdReaddir_ls (0.59s)
    --- FAIL: Test_fdReaddir_ls/tinygo (0.03s)
        --- FAIL: Test_fdReaddir_ls/tinygo/empty_directory (0.03s)
            require.go:331: expected no error, but was wasm error: unreachable
                wasm stack trace:
                        main.runtime._panic(i32,i32)
                        main._start(): panic: fdopendir ./a-: errno 21

                /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:268
                /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:238
                /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:104
        --- FAIL: Test_fdReaddir_ls/tinygo/directory_with_entries (0.00s)
            require.go:331: expected no error, but was wasm error: unreachable
                wasm stack trace:
                        main.runtime._panic(i32,i32)
                        main._start(): panic: fdopendir .: errno 21

                /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:268
                /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:238
                /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:118
        --- FAIL: Test_fdReaddir_ls/tinygo/directory_with_tons_of_entries (0.00s)
            require.go:331: expected no error, but was wasm error: unreachable
                wasm stack trace:
                        main.runtime._panic(i32,i32)
                        main._start(): panic: fdopendir .: errno 21

                /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:268
                /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:238
                /wazero/imports/wasi_snapshot_preview1/wasi_stdlib_test.go:148
FAIL
FAIL    github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1    9.651s
FAIL

[anuraaga]

We have found this commit from Go to be the cause of the issues

golang/go@601ea46#diff-ab6c974d2f954725a8b1b04f85fb0f1d6aa453692188e727b47ffc8ff202603bL46

Currently we call the runtime's memmove directly but this can cause problems with certain changes to the implementation. The solution to this and forwards compatibility will be to vendor in Go 1.23's memmove implementation and call it instead of the runtime's.

[corani]

We have found this commit from Go to be the cause of the issues

Does that mean it's a compiler bug? Or is memmove being used in a sketchy way here? 😉

[ncruces]

There's a lot of unsafe here, some of which breaks unsafe rules (particularly around the non-moving collector assumption), some go:linkname, and more. Especially in the compiler.

The point isn't that the compiler is impervious to Go runtime changes. The point is that you can self contain those breakages to wazero, and have a reasonably safe and performant way to run external code.

[corani]

Thanks for the quick turnaround on this one! Much appreciated 🙇‍♂