libyaml extension build fails on Python 3.11.0a7

[wimglenn]

Trouble building in 3.11 (gcc 11.2.0 on Linux):

$ python setup.py --with-libyaml bdist_wheel
running bdist_wheel
running build
running build_py
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
skipping 'yaml/_yaml.c' Cython extension (up-to-date)
building 'yaml._yaml' extension
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/wglenn/git/pyyaml/.venv/include -I/usr/local/include/python3.11 -c yaml/_yaml.c -o build/temp.linux-x86_64-cpython-311/yaml/_yaml.o
yaml/_yaml.c: In function ‘__Pyx_PyBytes_Equals’:
yaml/_yaml.c:25363:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
25363 |             hash1 = ((PyBytesObject*)s1)->ob_shash;
      |             ^~~~~
In file included from /usr/local/include/python3.11/bytesobject.h:62,
                 from /usr/local/include/python3.11/Python.h:50,
                 from yaml/_yaml.c:6:
/usr/local/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
    7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
      |                                   ^~~~~~~~
yaml/_yaml.c:25364:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
25364 |             hash2 = ((PyBytesObject*)s2)->ob_shash;
      |             ^~~~~
In file included from /usr/local/include/python3.11/bytesobject.h:62,
                 from /usr/local/include/python3.11/Python.h:50,
                 from yaml/_yaml.c:6:
/usr/local/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
    7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
      |                                   ^~~~~~~~
yaml/_yaml.c: In function ‘__Pyx_AddTraceback’:
yaml/_yaml.c:438:62: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
  438 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
      |                                                              ^~
yaml/_yaml.c:26009:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
26009 |     __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command 'gcc' failed with exit code 1

[nitzmahone]

This is a known Cython codegen issue due to some C API changes made in 3.11- Cython already has a fix merged and backported to 0.29.x (cython/cython@afc00fc), but they haven't released it yet- I suspect they're waiting for the imminent 3.11 beta 1 to drop before releasing an update.

I've verified that things build and work correctly on 3.11.0a7 if you install Cython from https://github.com/cython/cython/tree/0.29.x manually and do a non-isolated build with pip or your wheel builder of choice.

[ssbarnea]

Still there is no pre-build wheel for 3.11 uploaded to https://pypi.org/project/PyYAML/#files and that is a problem as py311 release is just around the corner.

I observed that GHA pipelines are ~3x slower on py311 on ansible-lint and I suspect that the reason for that is because pyyaml is installed without the compiled extension.

Yep, instead of running for ~11min, it takes ~35min on average.

As py311 already reached rc1, the big question is when we can have a new release that includes compiled extension for py311?

[masklinn]

Still there is no pre-build wheel for 3.11 uploaded to https://pypi.org/project/PyYAML/#files and that is a problem as py311 release is just around the corner.

It's still a while away, 3.11.0 is slated for October 3rd (PEP 664).

[bristea]

If pyyaml would have being a python application that is not listed as a dependency by any other package from pypi, we could easily argue that there is plenty of time. But given that in reality is one of the critical packages, I would say that this should make priority because otherwise those that depend on it will not have any chance of meeting the deadlines for the release.

Imagine what will happen if we discover that the compiled version crashes in some cases with py311 after python release? Without building it early enough in the process, we lose the ability to have one working (tested) version when new python is released.

[cognifloyd]

What work is left to build the 3.11 wheel? It looks like cython had had several releases, so that shouldn't be the issue anymore. What else is blocking this?

[nitzmahone]

3.11 wheels for PyYAML 6.0 are up.

For future reference- we'll probably never publish release wheels targeting a new major Python version before it hits rc1- we've gotten burned too many times by unstable ABI issues during the betas, and there's no way to re-publish those on PyPI without doing a new release that bumps the version number. The RC windows are usually several months long, which is a good enough balance IMO between catching potential real-world issues not caught by the test suite and exposing ourselves to late-breaking ABI changes in RCs (which Python core team says should never happen).