Arbitrary MSYS2 Python failures on Windows builds

[vintagepc]

Description

I use MSYS2 to build Windows binaries of QEMU in my actions. Currently, I'm getting arbitrary python failures during the build process - sometimes the build works, and sometimes it fails despite there being no changes in the affected region.

Here is an example of a failed build:
https://github.com/vintagepc/MINI404/actions/runs/3404920110/jobs/5662636923#step:1:2

And this build completed successfully:
https://github.com/vintagepc/MINI404/actions/runs/3404850675/jobs/5662389192

Both builds ran within a few minutes of each other.

There were no differences to the build configuration or code between the two except to fix a warning (in a file that isn't even compiled until much, much later) The build.yml change is for a different job entirely.
vintagepc/MINI404@1fc2742

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 10.15
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

Build logs linked above.

Current runner version: '2.298.2'
Operating System
Microsoft Windows Server 2022
10.0.20348
Datacenter
Runner Image
Image: windows-2022
Version: 20221027.1

Is it regression?

OS 10.0.20348/ Runner image 20221019.1

Expected behavior

Build to complete successfully

Actual behavior

Python command to generate keymaps seems to be arbitrarily failing or passing with no detailed error. This area of the code hasn't been changed in months and comes directly from upstream QEMU.

https://github.com/vintagepc/MINI404/actions/runs/3381838302/jobs/5624470117 is an example of a build with a python failue elsewhere in the process, also in code that hasn't been altered.

Repro steps

See build_msys section in this workflow: https://github.com/vintagepc/MINI404/blob/MINI404/.github/workflows/build.yml

Summarily:

1. Install MSYS and required packages
2. Checkout MINI404 repository with autoCRLF false
3. configure and build per workflow commands.

[al-cheb]

Hey @vintagepc.
We will take a look at it.

[ddobranic]

hello @vintagepc! in successful build log i can see checkout step for 114-fix-workflows differs from checkout step from unsuccessful build

[vintagepc]

Yes, that is the diff linked and noted in the initial report - however none of these changes impact this region of the code.

I should add I have some private forks of this repository that have also started experiencing similar failures during the msys build.

[eli-schwartz]

Yeah, there seems to be a general issue with msys2 python crashing unpredictably for no obvious reason, just in GitHub Actions. It's definitely not a problem with the projects running the workflows, because... restarting the workflow with the same context causes it to pass.

[vintagepc]

That's some interesting reading. Glad to hear it's not just me and others are also experiencing this problem in GA

[ddobranic]

@vintagepc do you still have the same issue? maybe using the previous version of msys2 could help? please check example here

[vintagepc]

As far as I know the problem still exists, but I was able to work around it by force-downgrading to an older MSYS2 python package. However, this is not a permanent fix, as MSYS doesn't support this and the old packages might be removed or become incompatible at any time.

Note my action still uses setup-msys@v2, not the v2.5 mentioned in the linked issue.

[ddobranic]

@vintagepc since we cannot do more for now, i will close the issue. please feel free to reopen if you have any additional questions or concerns.