Retry results broken w/ pytest 8.2.2 and unittest.TestCase

[andrewasheridan]

Results now show a generic

>   call = pytest.CallInfo.from_call(lambda: hook.pytest_runtest_call(item=item), when="call")
E   AssertionError

instead of the actual test.

• potentially related: pytest 8.2.2 breaks pytest-rerunfailures for tests that inherit unittest.TestCase pytest-dev/pytest#12424

To reproduce:

test_foo.py:

from unittest import TestCase


def foo():
    return "foo"


class TestFoo(TestCase):

    def test_foo(self):
        assert foo() == "bar"

Correctly shows failed, but assertion error points to pytest-retry code w/ pytest 8.2.2

❯ pytest --retries=1 -- test_foo.py
================================================================= test session starts =================================================================
platform darwin -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0
rootdir: /Users/e5703838/Projects/scratch
plugins: retry-1.6.3
collected 1 item                                                                                                                                      

test_foo.py RF                                                                                                                                  [100%]

====================================================================== FAILURES =======================================================================
__________________________________________________________________ TestFoo.test_foo ___________________________________________________________________

>   call = pytest.CallInfo.from_call(lambda: hook.pytest_runtest_call(item=item), when="call")
E   AssertionError

venv/lib/python3.12/site-packages/pytest_retry/retry_plugin.py:238: AssertionError

========================================================== the following tests were retried ===========================================================
        test_foo failed on attempt 1! Retrying!
        Traceback (most recent call last):
          File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/case.py", line 58, in testPartExecutor
            yield
        AssertionError: assert 'foo' == 'bar'
          
          - bar
          + foo

        test_foo failed after 2 attempts!
        Traceback (most recent call last):
          File "/Users/e5703838/Projects/scratch/venv/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
            result: Optional[TResult] = func()
                                        ^^^^^^
        AssertionError

============================================================== end of test retry report ===============================================================

=============================================================== short test summary info ===============================================================
FAILED test_foo.py::TestFoo::test_foo - AssertionError
============================================================ 1 failed, 1 retried in 0.02s =============================================================

vs incorrectly shows passed, but correctly points to actual test code w/ pytest 8.2.1

❯ pytest --retries=1 -- test_foo.py
================================================================= test session starts =================================================================
platform darwin -- Python 3.12.3, pytest-8.2.1, pluggy-1.5.0
rootdir: /Users/e5703838/Projects/scratch
plugins: retry-1.6.3
collected 1 item                                                                                                                                      

test_foo.py R.E                                                                                                                                 [100%]

======================================================================= ERRORS ========================================================================
________________________________________________________ ERROR at teardown of TestFoo.test_foo ________________________________________________________

self = <test_foo.TestFoo testMethod=test_foo>

    def test_foo(self):
>       assert foo() == "bar"
E       AssertionError: assert 'foo' == 'bar'
E         
E         - bar
E         + foo

test_foo.py:11: AssertionError

========================================================== the following tests were retried ===========================================================
        test_foo failed on attempt 1! Retrying!
        Traceback (most recent call last):
          File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/case.py", line 58, in testPartExecutor
            yield
        AssertionError: assert 'foo' == 'bar'
          
          - bar
          + foo

        test_foo passed on attempt 2!
        

============================================================== end of test retry report ===============================================================

=============================================================== short test summary info ===============================================================
ERROR test_foo.py::TestFoo::test_foo - AssertionError: assert 'foo' == 'bar'
======================================================== 1 passed, 1 error, 1 retried in 0.02s ========================================================

Noticed this b/c in Jenkins it just shows this:

Thanks for your attention and the helpful plugin :)

[str0zzapreti]

Thanks for reporting this. I don't think it has anything specifically to do with the new release, as I see the same type of behavior locally using older versions of Pytest. Just to be clear, compatibility with legacy unittest support in Pytest was never part of my scope when I designed this plugin, and I actually don't have a ton of familiarity with that particular subsystem. I do know that unittest cases function quite differently under the hood from standard Pytest cases and I suspect the issue is actually more related to this architectural difference, as indicated in the unittest compatibility page of the Pytest docs: https://docs.pytest.org/en/7.1.x/how-to/unittest.html

Due to architectural differences between the two frameworks, setup and teardown for unittest-based tests is performed during the call phase of testing instead of in pytest’s standard setup and teardown stages. This can be important to understand in some situations, particularly when reasoning about errors. For example, if a unittest-based suite exhibits errors during setup, pytest will report no errors during its setup phase and will instead raise the error during call.

With a bit of cursory debugging, it's clear that pytest-retry's current method for retrying individual tests is incompatible with the unittest implementation. As soon as I add a unittest.TestCase inheritance to the test class, the call object for subsequent attempts no longer includes any exception or result data regardless of whether the attempt failed, so my current solution for determining the test outcome isn't sufficient here. I'll need to do a more detailed investigation into the test runner process for unittest cases as I suspect I might need to add a separate control flow for when unittest is detected. The way you worded your report seems to imply that your unittest cases were working with an older version of Pytest and/or pytest-retry. If so, could you tell me what version that was? It would certainly help determine what changes led to the issue.

I won't make any promises here as unittest compatibility isn't particularly high on my priority list for this plugin, but I'll try to come up with an implementation as long as it doesn't compromise the design too much.

[andrewasheridan]

Thanks again for looking into it.

To be honest I only noticed the incorrect 'passing' messaging while typing up this issue.

Checking now I see that as far back as pytest 7.4.4 and retry 1.0.0 the test above results in "1 passed, 1 error, 1 retried", even though that test can never pass.

If changed to a not-unittest test the report is correct (1 failed, 1 retried)

So I would guess it's not due to a recent change.

All testing was on Python 3.12