Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

terminal: default to fE with -r (reportchars) #6524

Merged
merged 1 commit into from
Jan 29, 2020
Merged

terminal: default to fE with -r (reportchars) #6524

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog/6454.bugfix.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`--disable-warnings` is honored with `-ra` and `-rA`.
1 change: 1 addition & 0 deletions changelog/6454.feature.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Changed default for `-r` to `fE`, which displays failures and errors in the :ref:`short test summary <pytest.detailed_failed_tests_usage>`. `-rN` can be used to disable it (the old behavior).
8 changes: 6 additions & 2 deletions doc/en/usage.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,11 +169,11 @@ option you make sure a trace is shown.
Detailed summary report
-----------------------



The ``-r`` flag can be used to display a "short test summary info" at the end of the test session,
making it easy in large test suites to get a clear picture of all failures, skips, xfails, etc.

It defaults to ``fE`` to list failures and errors.

Example:

.. code-block:: python
Expand Down Expand Up @@ -261,8 +261,12 @@ Here is the full list of available characters that can be used:
- ``X`` - xpassed
- ``p`` - passed
- ``P`` - passed with output

Special characters for (de)selection of groups:

- ``a`` - all except ``pP``
- ``A`` - all
- ``N`` - none, this can be used to display nothing (since ``fE`` is the default)

More than one character can be used, so for example to only see failed and skipped tests, you can execute:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could also somehow mention that the effective default is wfE without --disable-warnings actually? (#5066)

Copy link
Member

@nicoddemus nicoddemus Jan 22, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, that got me thinking and I think -rw should be removed from the docs.

Since we have merged pytest-warnings (which seems like decades ago), -rw is not really a thing, because independently of what you put in -r (except N now in this PR) warnings are always displayed:

No -r option:

λ pytest .tmp\test-reporting.py
...
========================= warnings summary ==========================
test-reporting.py::test_warn
  D:\projects\pytest\.tmp\test-reporting.py:28: UserWarning: some warning
    warnings.warn(UserWarning('some warning'))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
====================== short test summary info ======================
FAILED .tmp\test-reporting.py::test_fail[0] - assert 0
FAILED .tmp\test-reporting.py::test_fail[1] - assert 0
FAILED .tmp\test-reporting.py::test_fail[2] - assert 0
=== 3 failed, 4 passed, 3 skipped, 3 xfailed, 1 warning in 0.11s ====

Some -r combination (note the missing w):

λ pytest .tmp\test-reporting.py -rXfs
...
========================= warnings summary ==========================
test-reporting.py::test_warn
  D:\projects\pytest\.tmp\test-reporting.py:28: UserWarning: some warning
    warnings.warn(UserWarning('some warning'))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
====================== short test summary info ======================
FAILED .tmp\test-reporting.py::test_fail[0] - assert 0
FAILED .tmp\test-reporting.py::test_fail[1] - assert 0
FAILED .tmp\test-reporting.py::test_fail[2] - assert 0
SKIPPED [3] test-reporting.py:15: nas
=== 3 failed, 4 passed, 3 skipped, 3 xfailed, 1 warning in 0.12s ====

So the only way to skip the warnings summary section is by using either -rN or --disable-warnings, as of now.

Warnings being controlled in -r is historical left-over, so much that warnings summary and short test summary are separate today.

I propose then:

  • Remove -rw from the docs, given it doesn't do anything (or mention that it is there for historical reasons but has no effect).
  • Make -rN not affect warnings at all, warnings can already be disabled with --disable-warnings. No backward compatibility here to consider because N is new, and the only way to hide warnings is with --disable-warnings anyway. This also makes using -rN produce the exact same behavior as before: hide short test summary, but warnings summary is still shown.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nicoddemus have you looked at #5066 again?
There we wanted something like -r-w and remove/deprecate the option completely even.

This comment was marked as duplicate.

Sign in to view

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh I didn't, even thought you mentioned it, my bad.

I've re-read that thread, and I still think what I wrote above holds better, specially because warnings and short summaries are reported and dealt separately. It also means we wouldn't need to deprecate anything really.

Expand Down
32 changes: 19 additions & 13 deletions src/_pytest/terminal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@

REPORT_COLLECTING_RESOLUTION = 0.5

_REPORTCHARS_DEFAULT = "fE"


class MoreQuietAction(argparse.Action):
"""
Expand Down Expand Up @@ -88,12 +90,13 @@ def pytest_addoption(parser):
"-r",
action="store",
dest="reportchars",
default="",
default=_REPORTCHARS_DEFAULT,
metavar="chars",
help="show extra test summary info as specified by chars: (f)ailed, "
"(E)rror, (s)kipped, (x)failed, (X)passed, "
"(p)assed, (P)assed with output, (a)ll except passed (p/P), or (A)ll. "
"(w)arnings are enabled by default (see --disable-warnings).",
"(w)arnings are enabled by default (see --disable-warnings), "
"'N' can be used to reset the list. (default: 'fE').",
)
group._addoption(
"--disable-warnings",
Expand Down Expand Up @@ -166,24 +169,27 @@ def mywriter(tags, args):


def getreportopt(config: Config) -> str:
reportopts = ""
reportchars = config.option.reportchars
if not config.option.disable_warnings and "w" not in reportchars:
reportchars += "w"
elif config.option.disable_warnings and "w" in reportchars:
reportchars = reportchars.replace("w", "")
aliases = {"F", "S"}

old_aliases = {"F", "S"}
reportopts = ""
for char in reportchars:
# handle old aliases
if char in aliases:
if char in old_aliases:
char = char.lower()
if char == "a":
reportopts = "sxXwEf"
reportopts = "sxXEf"
elif char == "A":
reportopts = "PpsxXwEf"
break
reportopts = "PpsxXEf"
elif char == "N":
reportopts = ""
elif char not in reportopts:
reportopts += char

if not config.option.disable_warnings and "w" not in reportopts:
reportopts = "w" + reportopts
elif config.option.disable_warnings and "w" in reportopts:
reportopts = reportopts.replace("w", "")

return reportopts


Expand Down
51 changes: 38 additions & 13 deletions testing/test_terminal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -813,9 +813,9 @@ def test():
def test_fail_extra_reporting(testdir, monkeypatch):
monkeypatch.setenv("COLUMNS", "80")
testdir.makepyfile("def test_this(): assert 0, 'this_failed' * 100")
result = testdir.runpytest()
result = testdir.runpytest("-rN")
result.stdout.no_fnmatch_line("*short test summary*")
result = testdir.runpytest("-rf")
result = testdir.runpytest()
result.stdout.fnmatch_lines(
[
"*test summary*",
Expand Down Expand Up @@ -984,37 +984,62 @@ def test_this(i):


def test_getreportopt():
from _pytest.terminal import _REPORTCHARS_DEFAULT

class Config:
class Option:
reportchars = ""
disable_warnings = True
reportchars = _REPORTCHARS_DEFAULT
disable_warnings = False

option = Option()

config = Config()

assert _REPORTCHARS_DEFAULT == "fE"

# Default.
assert getreportopt(config) == "wfE"

config.option.reportchars = "sf"
assert getreportopt(config) == "sf"
assert getreportopt(config) == "wsf"

config.option.reportchars = "sfxw"
assert getreportopt(config) == "sfx"
assert getreportopt(config) == "sfxw"

config.option.reportchars = "a"
assert getreportopt(config) == "wsxXEf"

config.option.reportchars = "N"
assert getreportopt(config) == "w"

config.option.reportchars = "NwfE"
assert getreportopt(config) == "wfE"

config.option.reportchars = "NfENx"
assert getreportopt(config) == "wx"

# Now with --disable-warnings.
config.option.disable_warnings = False
config.option.disable_warnings = True
config.option.reportchars = "a"
assert getreportopt(config) == "sxXwEf" # NOTE: "w" included!
assert getreportopt(config) == "sxXEf"

config.option.reportchars = "sfx"
assert getreportopt(config) == "sfxw"
assert getreportopt(config) == "sfx"

config.option.reportchars = "sfxw"
assert getreportopt(config) == "sfxw"
assert getreportopt(config) == "sfx"

config.option.reportchars = "a"
assert getreportopt(config) == "sxXwEf" # NOTE: "w" included!
assert getreportopt(config) == "sxXEf"

config.option.reportchars = "A"
assert getreportopt(config) == "PpsxXwEf"
assert getreportopt(config) == "PpsxXEf"

config.option.reportchars = "AN"
assert getreportopt(config) == ""

config.option.reportchars = "NwfE"
assert getreportopt(config) == "fE"


def test_terminalreporter_reportopt_addopts(testdir):
Expand Down Expand Up @@ -1131,7 +1156,7 @@ def test_func():
)
for tbopt in ["long", "short", "no"]:
print("testing --tb=%s..." % tbopt)
result = testdir.runpytest("--tb=%s" % tbopt)
result = testdir.runpytest("-rN", "--tb=%s" % tbopt)
s = result.stdout.str()
if tbopt == "long":
assert "print(6*7)" in s
Expand Down