Regression: doctests that used to pass are now failing

[Paulius-Maruska]

Problem: Doctests that are trying to showcase logging features of the code no longer pass. When running under pytest. They still do pass, when using unittest or doctest runner, but fail when using pytest.

The minimal code to reproduce the bug can be found in this gist:
https://gist.github.com/Paulius-Maruska/055eede98347ee306939d43c35cfe459

[pytestbot]

GitMate.io thinks possibly related issues are #758 (Doctests), #481 (Running doctests with -s fails?), #2955 (Fails or pass depending on capture mode!), #950 (Doctests fail when another test uses the fixture between runs.), and #1164 (test_something fails).

[RonnyPfannschmidt]

it seems like toxic doctests that just hook into the global logging setup where indeed a collateral to this
i believe its sufficient to disable the logging plugin of pytest to keep those tests running

my personal opinion is, its perfectly fine that doctests that modify random global state break on occasion, its simply really bad test design and had it coming

[Paulius-Maruska]

Well, I don't disagree with you, this is a rather bad case of doc string. But that doc string is written for documentation purposes. It is important to have that example in the docs. It is also important to somehow verify, that these examples in the docs are correct.

PS: current workaround - run your doctests separately using python -m doctest.

[nicoddemus]

Thanks @Paulius-Maruska for sharing the workaround. I'm closing this for now, feel free to follow up with more questions.