Re-consider where helper files are sourced in load_all()

[berndbischl]

Your NEWS state this

load_all() now sources all test helpers if you use testthat. This makes it much easier to interactively run tests (#1125). load_all() also correctly handles unix and windows subdirectories within R (@gaborcsardi, #1102)

In my setup I have a helper.R file, in tests/testthat which creates a few (data) objects I want to use in my unit tests (among other stuff).

You can reproduce the problem by creating a basically empty (but valid) package, then adding such a file under tests/testthat

data(Sonar, package = "mlbench")
x = Sonar

This now breaks, after the new devtools update

> load_all()
Loading minipkg
Error in eval(expr, envir, enclos) : object 'Sonar' not found

mlbench would usually be in my SUGGESTS (and I want it to be there) but even adding it to DEPENDS and adding and explicit call to library("mlbench") in the helper.R does not remove the problem.

This now breaks our complete devel workflow.

Any advice?

[jimhester]

The problem is the helpers are sourced in the package environment, but data() by default loads the objects in the global environment. Explicitly loading into the calling environment fixes the issue. (see mlr-org/mlr#835)

@hadley maybe we should add a data() shim which makes this the default?

[berndbischl]

Explicitly loading into the calling environment fixes the issue.

Cool that you are even fixing our package for us :) Thx a lot!

@hadley maybe we should add a data() shim which makes this the default?

I would seriously suggest that. What happened on our side was that I wanted to show a student assistant how the mlr unit tests work. Apparently at that point in time we got the new devtools version on her laptop. After calling load_all on the package, and seeing the error, we had 3 technically competent people sitting around the laptop for 30 min thinking "wtf is happening".
The error is extremely "unintuitive" and very hard to debug. I basically had to guess what happened, and downgrade devtools to figure this out.

[imanuelcostigan]

I have a package that creates some option settings through the .onAttach hook. My test helper files require these options. Unfortunately, devtools runs the package hooks after sourcing the test helpers and breaks things for me. Why can't the test helper files be loaded after the package is attached by load_all()?

[imanuelcostigan]

@hadley would you be open to me submitting a PR with the effect of sourcing the test helper files after running hooks?

PS - check() passes without error after making this change.

[hadley]

Sure

[berndbischl]

one other question from my side, that i just noticed.

in the past, i also created a file called helper_zzz.R
this eg set a seed for my unit tests. and this seed should ONLY be set for the unit tests.
now it is always used (when i call load_all on the package)

so i guess my question is:
how can i do "state changes" in my helper files, which are only executed when i am in "testthat unit testing mode" ?

[hadley]

@berndbischl maybe condition on interactive()?

[jennybc]

It feels like part of the pain around test helpers is that they are used for setup but there's no similarly official way to do corresponding tear down. I have a faux test file named test-zz-clean-up.R that does this. Is that awful?

[hadley]

What are you typically tearing down?

[jennybc]

OK this will be very specific. But I remove an OAuth token from an environment internal to the package. I delete some local files. I delete some things from Google Drive.

[hadley]

Maybe what we need for the non-idempotent stuff is to simply source a setup.R before running the tests and a teardown.R after the tests?

[jennybc]

Is there something like on.exit() in this context?

[hadley]

No, but that would be another approach - if you wanted to keep setup and tear down in the same file together. Could have something that allowed you to register setup and teardown code for each run.

I'm thinking something like:

register_test_something(
  setup = file.create("x"),
  teardown = unlink("x")
)

[berndbischl]

@berndbischl maybe condition on interactive()?

we also run our unit tests in non-interactive mode via shell scripts / from the console. actually we do that most often.

[pitakakariki]

@berndbischl maybe condition on interactive()?

I think interactive() returns TRUE whether you've called load_all() or test().

[hadley]

Moved to pkgload

[lock]

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/