Segfaults while passing a PyCapsule for a cython function with numpy array / memory view input parameters

[ashwinvis]

See various attempts here: https://github.com/ashwinvis/cython_capi#summary-of-the-different-attempts-in-this-repo

Each directory is a different attempt complete with a Makefile. To get started with spectacular segfaults compare using_cpython_pycapsule_class (using integer parameters - working example) and using_cpython_pycapsule_class_array (using array parameters not working).

cc: @paugier

[serge-sans-paille]

@ashwinvis you're using a numpy array signature for a capsule, which is not supported. Please read http://serge-sans-paille.github.io/pythran-stories/the-capsule-corporation.html#capsule-and-numpy for the proper way to do that.

Note: I'll try to issue an error message in that case.

[serge-sans-paille]

@ashwinvis can you try with #790 ?

[ashwinvis]

OK, so only native C types are supported for a capsule. No numpy or memory-view types. Got it.

[ashwinvis]

using_api_keyword gives error messages 👍 , but not with using_cpython_pycapsule_class_array - it builds and segfaults.

(pythran) cython_capi/using_api_keyword » make 
python setup_add.py
WARNING: Disabling color, you really want to install colorlog.
WARNING: Pythran support disabled for module: scipy.special
WARNING: Pythran support disabled for module: scipy
Compiling add.pyx because it changed.
Compiling add_int.pyx because it changed.
[1/2] Cythonizing add.pyx
[2/2] Cythonizing add_int.pyx
running build_ext
building 'add' extension
C compiler: gcc -pthread -fno-strict-aliasing -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC

creating build
creating build/temp.linux-x86_64-2.7
compile options: '-I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c'
gcc: add.c
In file included from /scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0,
                 from /scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from add.c:544:
/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^~~~~~~
gcc -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now build/temp.linux-x86_64-2.7/add.o -L/usr/lib -lpython2.7 -o /home/avmo/src/sandbox/cython_capi/using_api_keyword/add.so
building 'add_int' extension
C compiler: gcc -pthread -fno-strict-aliasing -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC

compile options: '-I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c'
gcc: add_int.c
gcc -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now build/temp.linux-x86_64-2.7/add_int.o -L/usr/lib -lpython2.7 -o /home/avmo/src/sandbox/cython_capi/using_api_keyword/add_int.so
pythran add_pythran.py -v
WARNING: Disabling color, you really want to install colorlog.
WARNING: Pythran support disabled for module: scipy.special
WARNING: Pythran support disabled for module: scipy
CRITICAL: I am in trouble. Your input file does not seem to match Pythran's constraints...
E: Invalid argument type `float[][]' for capsule 'add_capsule'.

(pythran) cython_capi/using_cpython_pycapsule_class_array » make build 
python setup.py build_ext -i
WARNING: Disabling color, you really want to install colorlog.
WARNING: Pythran support disabled for module: scipy.special
WARNING: Pythran support disabled for module: scipy
Compiling twice.pyx because it changed.
[1/1] Cythonizing twice.pyx
warning: twice.pyx:49:13: Use boundscheck(False) for faster access
running build_ext
building 'twice' extension
C compiler: gcc -pthread -fno-strict-aliasing -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC

creating build
creating build/temp.linux-x86_64-2.7
compile options: '-I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c'
gcc: twice.cpp
In file included from /scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1809:0,
                 from /scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from twice.cpp:559:
/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^~~~~~~
g++ -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now build/temp.linux-x86_64-2.7/twice.o -L/usr/lib -lpython2.7 -o /home/avmo/src/sandbox/cython_capi/using_cpython_pycapsule_class_array/twice.so
pythran -v call_capsule_pythran.py
WARNING: Disabling color, you really want to install colorlog.
WARNING: Pythran support disabled for module: scipy.special
WARNING: Pythran support disabled for module: scipy
INFO: customize UnixCCompiler
running build_ext
running build_src
build_src
building extension "call_capsule_pythran" sources
build_src: building npy-pkg config files
new_compiler returns distutils.unixccompiler.UnixCCompiler
INFO: customize UnixCCompiler
customize UnixCCompiler using build_ext
********************************************************************************
distutils.unixccompiler.UnixCCompiler
linker_exe    = ['gcc', '-pthread']
compiler_so   = ['gcc', '-pthread', '-fno-strict-aliasing', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt', '-DNDEBUG', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt', '-fPIC']
archiver      = ['ar', 'rc']
preprocessor  = ['gcc', '-pthread', '-E']
linker_so     = ['gcc', '-pthread', '-shared', '-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now']
compiler_cxx  = ['g++', '-pthread']
ranlib        = None
compiler      = ['gcc', '-pthread', '-fno-strict-aliasing', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt', '-DNDEBUG', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt']
libraries     = []
library_dirs  = ['/usr/lib']
include_dirs  = ['/usr/include/python2.7']
********************************************************************************
new_compiler returns distutils.unixccompiler.UnixCCompiler
INFO: customize UnixCCompiler
customize UnixCCompiler using build_ext
********************************************************************************
distutils.unixccompiler.UnixCCompiler
linker_exe    = ['gcc', '-pthread']
compiler_so   = ['gcc', '-pthread', '-fno-strict-aliasing', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt', '-DNDEBUG', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt', '-fPIC']
archiver      = ['ar', 'rc']
preprocessor  = ['gcc', '-pthread', '-E']
linker_so     = ['gcc', '-pthread', '-shared', '-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now']
compiler_cxx  = ['g++', '-pthread']
ranlib        = None
compiler      = ['gcc', '-pthread', '-fno-strict-aliasing', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt', '-DNDEBUG', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt']
libraries     = []
library_dirs  = ['/usr/lib']
include_dirs  = ['/usr/include/python2.7']
********************************************************************************
building 'call_capsule_pythran' extension
compiling C++ sources
C compiler: g++ -pthread -fno-strict-aliasing -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -fPIC

creating /tmp/tmpAvSbZt/tmp
compile options: '-DUSE_GMP -DENABLE_PYTHON_MODULE -D__PYTHRAN__=2 -DPYTHRAN_BLAS_BLAS -I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/pythran-0.8.4.post0-py2.7.egg/pythran -I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/pythran-0.8.4.post0-py2.7.egg/pythran/pythonic/patch -I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I/scratch/avmo/opt/pythran/include -I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c'
extra options: '-std=c++11 -fno-math-errno -w -fvisibility=hidden -fno-wrapv'
g++: /tmp/tmpPB4SSa.cpp
exec_command(['g++', '-pthread', '-fno-strict-aliasing', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt', '-DNDEBUG', '-march=x86-64', '-mtune=generic', '-O2', '-pipe', '-fstack-protector-strong', '-fno-plt', '-fPIC', '-DUSE_GMP', '-DENABLE_PYTHON_MODULE', '-D__PYTHRAN__=2', '-DPYTHRAN_BLAS_BLAS', '-I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/pythran-0.8.4.post0-py2.7.egg/pythran', '-I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/pythran-0.8.4.post0-py2.7.egg/pythran/pythonic/patch', '-I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include', '-I/usr/local/include', '-I/usr/include', '-I/scratch/avmo/opt/pythran/include', '-I/scratch/avmo/opt/pythran/lib/python2.7/site-packages/numpy/core/include', '-I/usr/include/python2.7', '-c', '/tmp/tmpPB4SSa.cpp', '-o', '/tmp/tmpAvSbZt/tmp/tmpPB4SSa.o', '-std=c++11', '-fno-math-errno', '-w', '-fvisibility=hidden', '-fno-wrapv'],)
Retaining cwd: /home/avmo/src/sandbox/cython_capi/using_cpython_pycapsule_class_array
_preserve_environment([])
_update_environment(...)
_update_environment(...)
g++ -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now /tmp/tmpAvSbZt/tmp/tmpPB4SSa.o -L/usr/lib64 -L/usr/lib -lgmp -lgmpxx -lcblas -lblas -lpython2.7 -o /tmp/tmpIgK6NN/call_capsule_pythran.so -fvisibility=hidden -Wl,-strip-all
exec_command(['g++', '-pthread', '-shared', '-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now', '/tmp/tmpAvSbZt/tmp/tmpPB4SSa.o', '-L/usr/lib64', '-L/usr/lib', '-lgmp', '-lgmpxx', '-lcblas', '-lblas', '-lpython2.7', '-o', '/tmp/tmpIgK6NN/call_capsule_pythran.so', '-fvisibility=hidden', '-Wl,-strip-all'],)
Retaining cwd: /home/avmo/src/sandbox/cython_capi/using_cpython_pycapsule_class_array
_preserve_environment([])
_update_environment(...)
_update_environment(...)
INFO: Generated module: call_capsule_pythran
INFO: Output: /home/avmo/src/sandbox/cython_capi/using_cpython_pycapsule_class_array/call_capsule_pythran.so
(pythran) cython_capi/using_cpython_pycapsule_class_array » make test 
python -Wall test.py


capsule twice_static
make: *** [Makefile:16: test] Segmentation fault (core dumped)

[serge-sans-paille]

@ashwinvis yep, the function accepted as parameter has to be a capsule itself, so the same type constaints apply, at least for the arguments. i'll enforce that.

[ashwinvis]

Btw, I tried a bit with Pythran and Cython with just pointers, but it is not straightforward to create capsules from Cython functions, especially with pointer types as arguments. https://github.com/ashwinvis/cython_capi/tree/master/using_cpython_pycapsule_pointers

Perhaps (and if you don't mind) it would be useful, if you can write a working example Pythran+Cython/ctypes with pointers away from the context of scipy.

[serge-sans-paille]

@ashwinvis after some thoughts, I decided to remove the checks on capsule arguments. The checks are valid in your case, but I can see a situation where a Pythran function would use a pythran-generated capsule. And in that situation, it is perfectly legit to pass arbitrary types as parameters.

That's the issue with capsule: they are opaque and if you don't use them with care, you just get a segfault.
I'll try to update the reference on capsule usage with more examples though :-)