Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

short test summary and -vv #12713

Closed
ewjoachim opened this issue Aug 14, 2024 · 2 comments · Fixed by #13061
Closed

short test summary and -vv #12713

ewjoachim opened this issue Aug 14, 2024 · 2 comments · Fixed by #13061
Labels
topic: reporting related to terminal output and user-facing messages and errors type: enhancement new feature or API change, should be merged into features branch

Comments

@ewjoachim
Copy link

ewjoachim commented Aug 14, 2024

a detailed description of the bug or problem you are having

(I've done my best to try and see if this is a duplicate and I'm sorry if it is, but it seems it's not)

Following #11777 (8.2.0) | changelog:

Text is no longer truncated in the short test summary info section when -vv is given.

The short summary can become completely impractical with -vv because each failure message could be hundreds of lines long.

It seems to be a strange choice given #11777 is about the text being truncated in:

E       assert 50 == 49
E        +  where 50 = len({0: 0, 1: 1, 2: 2, 3: 3, ...})

and I don't see the original poster complaining about the short summary, but both were changed?

Was it the intended change ? Is there something we can do to keep the original -vv behaviour for the test output, and keep the "short test summary" short ? It's actually quite useful in CI that the last thing printed is the list of tests that failed and not why each of them failed: it's a summary and the full output is available above anyway.

I think the decision a lot of versions ago to add an excerpt of the failure reason to the short summary was a good compromise: it used otherwise unused space to add more useful info. But if that excerpt start taking more screen real estate space (potentially 100x more) than the summary itself, I'm not sure I understand why we would want that?

Maybe we could introduce a different "letter" (or a different option or something) to -r to get the list of tests without any kind of explanation of why they failed? Maybe the fix for #11777 should have affected the "failure explanation" and not the short summary?

output of pip list from the virtual environment you are using

$ pip list
pip list
Package   Version
--------- -------
iniconfig 2.0.0
packaging 24.1
pip       24.0
pluggy    1.5.0
pytest    8.3.2

pytest and operating system versions

pytest==8.3.2
Ubuntu 22.04.4 LTS

minimal example if possible

def test_foo():
    assert "a\n" * 50 == ""
$ pytest -vv
============================================================== test session starts ===============================================================
platform linux -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0 -- /home/joachim/other-src/pytest-verbose/.venv/bin/python3.12
cachedir: .pytest_cache
rootdir: /home/joachim/other-src/pytest-verbose
collected 1 item                                                                                                                                 

test_verbose.py::test_foo FAILED                                                                                                           [100%]

==================================================================== FAILURES ====================================================================
____________________________________________________________________ test_foo ____________________________________________________________________

    def test_foo():
>       assert "a\n" * 50 == ""
E       AssertionError: assert 'a\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\n' == ''
E         
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a

test_verbose.py:2: AssertionError
============================================================ short test summary info =============================================================
FAILED test_verbose.py::test_foo - AssertionError: assert 'a\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\n' == ''
  
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
=============================================================== 1 failed in 0.01s ================================================================
@Zac-HD Zac-HD added type: enhancement new feature or API change, should be merged into features branch topic: reporting related to terminal output and user-facing messages and errors labels Oct 30, 2024
@kenny-y-dev
Copy link
Contributor

I took a look at this and it seems that the fix implemented for #11777 may have misunderstood the ask in the original issue; the issue referenced -vv still truncating sequences when an int comparison assert based on len() failed, though the PR only adjusted the behavior of the short test summary.

It appears that the issue demonstrated in #11777 is still present -- I do agree with @ewjoachim that the short test summary should still be brief at the end of the output for quick visual parsing, even if there is extended verbose output above. I'll take a deeper dive to see exactly why the sequence truncation is still occurring.

@ewjoachim
Copy link
Author

Oh, I had missed your PR. Just saw it now that it's merged. Thank you very much for taking the time :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: reporting related to terminal output and user-facing messages and errors type: enhancement new feature or API change, should be merged into features branch
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants