Feature request: Skip doctest conditionally

[akaihola]

Subject: Unit tests can be skipped conditionally using the @unittest.skipIf() decorator. A similar feature would be useful for doctests when run by Sphinx.

Example use cases

• different tests for specific versions of dependencies
• skip tests in absence of optional dependencies
• skip tests in absence of a network or VPN
• skip tests in absence of some hardware

The feature was also mentioned on Stack Overflow in the Sphinx doctest: conditional test skipping question.

Proposed syntax

.. doctest::
   :skipif: <condition>

   >>> test()
   <output>

.. testcode::
   :skipif: <condition>

   test()

.. testoutput::
   :skipif: <condition>

   <output>

Global setup

The code in the doctest_global_setup and doctest_global_cleanup configuration settings should be run before and after the condition. This would enable users to store complex conditions in a global variable and avoid repetition in doctest definitions.

Expected results

When a <condition> evaluates to a true value, the test or the test output assertion should be skipped.

[akaihola]

In my patch above, I simply return an empty list from the run() method of the doctest directive classes in case the expression in the new :skipif: option evaluates to a true value.

I noticed that in the :pyversion: directive, a SKIP doctest flag is injected instead. However, this method only seems to be usable for the doctest:: directive but not for testcode:: and testoutput::. For this reason, I adopted the empty list method for all of the directives. As a side effect, conditional skipping can also be used for testsetup:: and testcleanup::.

If the proposed feature and my approach make sense, I'll proceed to writing tests and documentation next.

[tk0miya]

+0; I don't know doctest module so much. So I don't have any opinion. But it seems worthy.

ping @shimizukawa

[shimizukawa]

+0. I am anxious about whether it will be too many options.
However, it seems good to have an option to exclude doctest under specific conditions from testing.

[tk0miya]

Okay, let's go forward.
@akaihola could you send us a PR please? It seems your patch is enough good. I will merge it soon if
you'll add testcode for it.

Thanks,

[tk0miya]

Note:
We'll call feature freeze for 1.8 in this weekend. It will be released as 1.8 if the patch will be merged until then.

[akaihola]

@tk0miya, thanks for the note, I'll try to find time to work on this ASAP.

[akaihola]

@tk0miya, I added both tests and documentation for the :skipif: option, and created pull request #5307.