Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make @composite strategies collected by pytest act as failing tests #1665

Closed
Zac-HD opened this issue Nov 1, 2018 · 0 comments
Closed

Make @composite strategies collected by pytest act as failing tests #1665

Zac-HD opened this issue Nov 1, 2018 · 0 comments
Assignees
Labels
legibility make errors helpful and Hypothesis grokable

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Nov 1, 2018

From #1658, occasionally users use @composite when they should be using @given(st.data()). Unfortunately, this silently goes wrong:

from hypothesis.strategies import composite

@composite
def test_data_factory(draw):
    foo = draw(custom_objects)
    assert foo.check_bar()

Pytest will happily collect test_data_factory, execute it, and report it as a passing test... even if you replace the function body with assert False! This is because calling the function returns a LazyStrategy object, and the function isn't actually executed until later. More generally, it's because @composite should not be used to define test functions!

So it would be good to add a check to our pytest plugin, in pytest_collection_modifyitems, that replaces the collected item.obj with a function that will (a) fail, and (b) do so with a useful error suggesting that the user either rename the strategy or if intending to execute it as a test use @given(st.data()) instead.

@Zac-HD Zac-HD added the legibility make errors helpful and Hypothesis grokable label Nov 1, 2018
@Zac-HD Zac-HD self-assigned this Nov 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
legibility make errors helpful and Hypothesis grokable
Projects
None yet
Development

No branches or pull requests

1 participant