CI is failing with pytest-related import errors during the collection stage

[joshuacwnewton]

We started getting errors for two separate sets of OSs:

Tests / Windows (Native)

=========================== short test summary info ===========================
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\matplotlib\tests - OSError: The baseline image directory does not exist. This is most likely because the test data is not installed. You may need to install matplotlib from source to get the test data.
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\mpl_toolkits\axes_grid1\tests - OSError: The baseline image directory does not exist. This is most likely because the test data is not installed. You may need to install matplotlib from source to get the test data.
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\mpl_toolkits\axisartist\tests - OSError: The baseline image directory does not exist. This is most likely because the test data is not installed. You may need to install matplotlib from source to get the test data.
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\mpl_toolkits\mplot3d\tests - OSError: The baseline image directory does not exist. This is most likely because the test data is not installed. You may need to install matplotlib from source to get the test data.
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\numpy - ModuleNotFoundError: No module named 'hypothesis'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\pandas - ModuleNotFoundError: No module named 'hypothesis'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\scipy - ModuleNotFoundError: No module named 'hypothesis'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_cte.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_ddl.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_dialect.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_insert.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_reflection.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_results.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_select.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_sequence.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_types.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sqlalchemy\testing\suite\test_update_delete.py - AttributeError: 'NoneType' object has no attribute 'fixture'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\envs\venv_sct\Lib\site-packages\sympy - ImportError: hypothesis is a required dependency to run the SymPy test suite. Install it with 'pip install hypothesis' or 'conda install -c conda-forge hypothesis'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\pkgs\anaconda-anon-usage-0.4.4-py312hfc23b7f_100\info\test\tests - ModuleNotFoundError: No module named 'conda'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\pkgs\boltons-23.0.0-py312haa95532_0\info\test\tests - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', 'C:\\Users\\runneradmin\\sct_6.4.dev0\\python\\envs\\venv_sct\\Lib\\site-packages\\wandb\\vendor\\promise-2.3.0\\tests\\conftest.py', WindowsPath('C:/Users/runneradmin/sct_6.4.dev0/python/pkgs/boltons-23.0.0-py312haa95532_0/info/test/tests/conftest.py'))
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\pkgs\conda-package-handling-2.3.0-py312haa95532_0\info\test\tests - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', 'C:\\Users\\runneradmin\\sct_6.4.dev0\\python\\envs\\venv_sct\\Lib\\site-packages\\wandb\\vendor\\promise-2.3.0\\tests\\conftest.py', WindowsPath('C:/Users/runneradmin/sct_6.4.dev0/python/pkgs/conda-package-handling-2.3.0-py312haa95532_0/info/test/tests/conftest.py'))
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\pkgs\cryptography-42.0.5-py312h89fc84f_1\info\test\tests - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', 'C:\\Users\\runneradmin\\sct_6.4.dev0\\python\\envs\\venv_sct\\Lib\\site-packages\\wandb\\vendor\\promise-2.3.0\\tests\\conftest.py', WindowsPath('C:/Users/runneradmin/sct_6.4.dev0/python/pkgs/cryptography-42.0.5-py312h89fc84f_1/info/test/tests/conftest.py'))
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\pkgs\menuinst-2.1.1-py312h5da7b33_0\info\test\tests - ModuleNotFoundError: No module named 'menuinst'
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\pkgs\tqdm-4.66.4-py312hfc267ef_0\info\test\tests - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', 'C:\\Users\\runneradmin\\sct_6.4.dev0\\python\\envs\\venv_sct\\Lib\\site-packages\\wandb\\vendor\\promise-2.3.0\\tests\\conftest.py', WindowsPath('C:/Users/runneradmin/sct_6.4.dev0/python/pkgs/tqdm-4.66.4-py312hfc267ef_0/info/test/tests/conftest.py'))
ERROR C:\Users\runneradmin\sct_6.4.dev0\python\pkgs\truststore-0.8.0-py312haa95532_0\info\test\tests - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', 'C:\\Users\\runneradmin\\sct_6.4.dev0\\python\\envs\\venv_sct\\Lib\\site-packages\\wandb\\vendor\\promise-2.3.0\\tests\\conftest.py', WindowsPath('C:/Users/runneradmin/sct_6.4.dev0/python/pkgs/truststore-0.8.0-py312haa95532_0/info/test/tests/conftest.py'))
!!!!!!!!!!!!!!!!!! Interrupted: 25 errors during collection !!!!!!!!!!!!!!!!!!!

Create release / Ubuntu

=========================== short test summary info ============================
ERROR ../../../sct_0.0/python/pkgs/anaconda-anon-usage-0.4.4-py312hfc0e8ea_100/info/test/tests - ModuleNotFoundError: No module named 'conda'
ERROR ../../../sct_0.0/python/pkgs/conda-package-handling-2.3.0-py312h06a4308_0/info/test/tests - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', '/home/runner/sct_0.0/python/pkgs/boltons-23.0.0-py312h06a4308_0/info/test/tests/conftest.py', PosixPath('/home/runner/sct_0.0/python/pkgs/conda-package-handling-2.3.0-py312h06a4308_0/info/test/tests/conftest.py'))
ERROR ../../../sct_0.0/python/pkgs/cryptography-42.0.5-py312hdda0065_1/info/test/tests - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', '/home/runner/sct_0.0/python/pkgs/boltons-23.0.0-py312h06a4308_0/info/test/tests/conftest.py', PosixPath('/home/runner/sct_0.0/python/pkgs/cryptography-42.0.5-py312hdda0065_1/info/test/tests/conftest.py'))
ERROR ../../../sct_0.0/python/pkgs/menuinst-2.1.1-py312h06a4308_0/info/test/tests - ModuleNotFoundError: No module named 'menuinst'
ERROR ../../../sct_0.0/python/pkgs/tqdm-4.66.4-py312he106c6f_0/info/test/tests - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', '/home/runner/sct_0.0/python/pkgs/boltons-23.0.0-py312h06a4308_0/info/test/tests/conftest.py', PosixPath('/home/runner/sct_0.0/python/pkgs/tqdm-4.66.4-py312he106c6f_0/info/test/tests/conftest.py'))
ERROR ../../../sct_0.0/python/pkgs/truststore-0.8.0-py312h06a4308_0/info/test/tests - _pytest.pathlib.ImportPathMismatchError: ('tests.conftest', '/home/runner/sct_0.0/python/pkgs/boltons-23.0.0-py312h06a4308_0/info/test/tests/conftest.py', PosixPath('/home/runner/sct_0.0/python/pkgs/truststore-0.8.0-py312h06a4308_0/info/test/tests/conftest.py'))
!!!!!!!!!!!!!!!!!!! Interrupted: 6 errors during collection !!!!!!!!!!!!!!!!!!!!

I'll try to get to the bottom of which package upgrade is causing these issues, and report upstream if necessary.

[joshuacwnewton]

Aha! I think I know the issue here.

• The test runs that use .ci.sh (where we explicitly specify testing/cli and testing/api) work fine: https://github.com/spinalcordtoolbox/spinalcordtoolbox/actions/runs/10039572922
• However, the test runs that invoke sct_testing (while implicitly relying on setup.cfg to configure pytest) end up running the tests for our dependencies.

spinalcordtoolbox/setup.cfg

Line 3 in 9cd5700

python_files = testing/**/test_*.py

And, of course, this fails miserably because we are not set up to run our dependencies' tests whatsoever.

So, I will have to figure out how pytest has changed its config to parse tests differently. 🤔