[Investigation] The MSYS2 + meson + Python crash issue

[lazka]

Another issue (old one: #11864) to collect some information and what was tried so far.

I've created a small repo for reproducing the issue: https://github.com/lazka/python-crash-test

Any ideas regarding what we could try welcome.

The issue

meson fails with STATUS_ACCESS_VIOLATION sometimes like this when being called from ninja, and we haven't found a way to reproduce it locally:

FAILED: libmy-shared-lib6.dll.p/libmy-shared-lib6.dll.symbols 
"D:/a/_temp/msys64/ucrt64/bin/meson" "--internal" "symbolextractor" "D:/a/python-crash-test/python-crash-test/project/_build" libmy-shared-lib6.dll "libmy-shared-lib6.dll.a" libmy-shared-lib6.dll.p/libmy-shared-lib6.dll.symbols 

This below is now out of date, see the followup answers for the cause

When has it started

• When we updated from Python 3.9 to 3.10

Where it failed so far

• It happened in:

  • MINGW64: https://github.com/msys2/msys2-tests/actions/runs/5117107910/jobs/9199890976
  • UCRT64: https://github.com/lazka/python-crash-test/actions/runs/5156416563/jobs/9287390420#step:4:68
  • CLANG64: https://gitlab.gnome.org/GNOME/gobject-introspection/-/issues/456

• It happend with:

  • GHA Windows Server 2022
  • GHA Windows Server 2019
  • Gitlab CI using Windows Server 2016

Where it hasn't failed so far

• In MINGW32 / CLANG32
• Locally, outside of CI

What makes the error go away

• Running everything via powershell (and just setting PATH), and not from bash
• Setting MSYS=winjitdebug
• Downgrading to our last Python 3.9 mingw version (I created a rebuild for the last version we had)
• Using the official CPython 3.9 installed via setup-python
• Using the official CPython 3.11 installed via setup-python

What doesn't make the error go away

• Disabling Python forward-slash-path mode via MSYSTEM= doesn't help.
• Using the official CPython 3.10 installed via setup-python can also fail: https://github.com/lazka/python-crash-test/actions/runs/5157613189/jobs/9290253297 (with both 2019 and 2022)

Bisecting:

Testing with official CPython builds:

• NO-ERROR: 3.9.13, 3.10.0-alpha.1, 3.10.0-alpha.2, 3.10.0-alpha.3 <-> 3.11.0-alpha.4, 3.11.0-alpha.5, 3.11.0-alpha.6, 3.11.0-alpha.7, 3.11.3
• ERROR: 3.10.0-alpha.4, 3.10.0-alpha.5, 3.10.0-alpha.7, 3.10.11, 3.11.0-alpha.1, 3.11.0-alpha.3

Error started: v3.10.0a3...v3.10.0a4
Fixed: v3.11.0a3...v3.11.0a4

Bisect, see next post.

[lazka]

Short summary of bisecting the issue with upstream CPython in GHA:

• First bad commit (in the range v3.10.0a3...v3.10.0a4): python/cpython@4101018
• First fixed commit (in the range v3.11.0a3...v3.11.0a4): python/cpython@ed57b36

[lazka]

I've reduced the reproducer to no longer include MSYS2 now: https://github.com/lazka/python-crash-test/blob/70349faa496cdf19c4bd5152435542542eb7a14f/.github/workflows/main.yml

[lazka]

I've filed a bug report: python/cpython#105400 but I don't expect much from it, but who knows.

I'm also not sure how to proceed on the MSYS2 side of things. Prioritizing a 3.11 update would be one option.

[sskras]

Wow, impressive. I tried to reproduce this in cmd by adding MINGW64 tools to the PATH with no success:

Did I miss something?

[Biswa96]

Did I miss something?

Yes, this issue does not happen "Locally, outside of CI".

[sskras]

OK. Is it possible to know the actual CI setup (VMM name/version, the host CPUID, the host OS) ?

[eli-schwartz]

It's a Github Actions runner defined at https://github.com/lazka/python-crash-test/blob/70349faa496cdf19c4bd5152435542542eb7a14f/.github/workflows/main.yml

https://github.com/lazka/python-crash-test/actions/runs/5190923933/jobs/9358053943

[sskras]

@eli-schwartz, sure, I saw the defined action. Well, this says thing about the Guest:
https://github.com/actions/runner-images/releases/tag/win22%2F20230517.1

But no configuration of the lower levels – the VMM/hypervisor, the Host OS/hardware.

EDIT: Anyway, I do not even understand it it's possible to download/build the exact disk image of the Guest OS to try it at locally.

[eli-schwartz]

I have no idea how people generally find out information about the host hardware and hypervisor used on Github's servers, and would not assume that someone who ran a workflow could find that out.

[jeremyd2019]

Interesting that it's specifically SEM_NOGPFAULTERRORBOX... the other relevant flag that we are losing by setting winjitdebug is SEM_FAILCRITICALERRORS and that's the one whose absence results in message boxes for missing exports. I don't know if there's a reasonable way to only set one and not the other for child processes though, as a workaround to keep CI from hanging on missing exports while still working around this issue. It's probably not worth a new MSYS option and additional patching of msys2-runtime over...

[kasper93]

Small update. I've reviewed meson's symbolextractor.py code. While there are few things that could be improved, it is not related to this issue.

One important thing I noticed that the issue happens only with llvm-nm which is preferred by meson, it does not happen with mingw's nm. With this python-crash-test reproducer.

I think next step would be to remove meson and ninja (if possible) from the reproducer. I tested with script that only runs llvm-nm on the library. The reproducer is pretty minimal, good job on that, but removing components like meson would really help to focus on important parts.

But my fear is that since it doesn't happen with Python 3.11 there will not be much interest (if at all) to find the root cause of this. I think during process exit some resources/fd are not in a valid state which causes the issue.

[anarazel]

Interesting that it's specifically SEM_NOGPFAULTERRORBOX...

There's a lot tied into that one these days... Windows error reporting, JIT debugging, etc only happen when SEM_NOGPFAULTERRORBOX is not set.

I don't know if there's a reasonable way to only set one and not the other for child processes though

I hit this issue without mingw or llvm-nm being involved (meson running C binaries compiled with msvc, via msbuild, not ninja). SEM_NOGPFAULTERRORBOX was the relevant factor, SEM_FAILCRITICALERRORS did not matter. Unfortunately the failure rates in my case are too low to make for good test case, and I hadn't been able to increase them meaningfully.

From writing this up for postgres:

The problem:

Occasionally ecpg test files would fail to compile, exiting with -1073741819:
C:\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5): error MSB8066: Custom build for 'C:\cirrus\build\meson-private\custom_target.rule' exited with code -1073741819. [c:\cirrus\build\src\interfaces\ecpg\test\sql\3701597@@twophase(dot)c(at)cus(dot)vcxproj]

-1073741819 is 0xc0000005, which in turn is STATUS_ACCESS_VIOLATION, i.e. a
segfault. This happens in roughly 1/3 of the builds, but with "streaks" of not
happening and more frequently happening.
...
The problem even persisted when replacing meson's sys.exit() with os._exit(),
which indeed just calls _exit().

Which to me strongly hints that the issue is somewhere in the windows runtime.

Possibly the explanation for why the issue is visible in CI but not, so far, locally, is due to different version of the windows runtime being used.

[lb90]

Hi! Writing here since that question may also help with this issue

I am experiencing a deadlock in python when running meson test --gdb. So I tried building mingw-w64-x86_64-python with options=('debug' '!strip'), but after installing the package python doesn't work anymore. Actually even the PKGBUILD check() fails. Anybody knows why? EDIT: solved by removing --with-pydebug

Regarding this issue, I'd try installing a vectored exception handler which simply calls Sleep() with a super long timeout. Then, when python hangs due to the exception handler, attach gdb (or lldb) to the process via SSH and gather a stacktrace. Vectored exception handlers always run on top of the normal stack, no unwind is done, so the backtrack would give us all the infos we need. Not entirely sure if handlers are also run during process teardown (ExitProcess), but most probably yes.

[kasper93]

Would it be possible to update Python package to 3.11? It is already at 3.11.4 and 3.10 is in "security fixes" only stage. Arch also provides 3.11 by default, but of course they have also other version available in repo.

[lazka]

I can confirm that a mingw build of 3.11 also doesn't crash: msys2-contrib/cpython-mingw#139 (comment)

[Biswa96]

Should this be closed?

[lazka]

Let's hope it doesn't come back :)