No test report when pytest can't repr() a test failure in --showlocals mode

[yozlet]

When using pytest, the test adapter will report a blank result (no success or failure) if a test fails and pytest raises an exception when reporting the test failure.

Here's example_tests.py:

class Thing:
    def __repr__(self):
        raise NotImplementedError

def test_the_thing():
    t = Thing()
    assert t == 3

def test_something_works():
    assert 1 == 1

Expected output: Test Explorer shows one pass, one fail.

Actual output: Test Explorer shows no test results; the icons to the left of the tests in the tree go back to the square-in-circle. Clicking the test nodes does nothing. The Python Test Adapter Log just shows a normal execution that completes.

I'm not completely sure what's happening here because when running pytest on the command line, the tests execute fully, and the exception caused by the __repr__ failure is contained within the output:

$ pytest example_tests.py

========================================= test session starts =========================================
platform darwin -- Python 3.6.8, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
Django settings:settings.pytest (from ini file)
rootdir: /Users/yoz/Work, inifile: pytest.ini
plugins: mock-1.10.1, metadata-1.8.0, jest-0.3.0, django-3.4.8, cov-2.6.1
collected 2 items

example_tests.py F.                                                                             [100%]

============================================== FAILURES ===============================================
___________________________________________ test_the_thing ____________________________________________

    def test_the_thing():
        t = Thing()
>       assert t == 3
E       assert <[NotImplementedError("") raised in repr()] Thing object at 0x10d949d30> == 3

example_tests.py:7: AssertionError
================================= 1 failed, 1 passed in 0.24 seconds ==================================

[kondratyev-nv]

@yozlet Thank you for the issue! I'll take a look.

[kondratyev-nv]

@yozlet Just a small update on this issue. I have tried to reproduce it on my Linux machine, and everything worked as expected. A little bit later I will try to reproduce this on Mac and CI.

[yozlet]

Ah, here's the differentiator - the --showlocals argument:

"python.unitTest.pyTestArgs": ["--showlocals", "-vv"]

Without --showlocals, the Test Adapter works fine.

Trouble is, --showlocals is just massively useful for fixing tests quickly without needing to run the debugger. It's so useful that maybe you could do something with the data it reports? (Though that's a separate feature request.)

[kondratyev-nv]

It looks like this is a pytest bug happening when using -l (--showlocals) and -vv together. I've created the issue pytest-dev/pytest#5089 and look forward to what pytest devs thinking about it.

[kondratyev-nv]

@yozlet The issue in pytest has already been fixed. I will close this issue since it has nothing to do with the extension. Please, recheck this case after the new release of pytest. If there would be any other problems, feel free to create a new issue.

[yozlet]

Many thanks for chasing this up and fixing it on the pytest side - that's far beyond the call of duty!