Incorrectly up-to-date dynamic targets

[wlandau]

Prework

• Read and abide by drake's code of conduct.
• Search for duplicates among the existing issues, both open and closed.
• Advanced users: verify that the bug still persists in the current development version (i.e. remotes::install_github("ropensci/drake")) and mention the SHA-1 hash of the Git commit you install.

Description

Under some circumstances, dynamic targets incorrectly show as up-to-date. This bug is unpredictable and extremely difficult to reproduce, but some of the surrounding circumstances seem to be:

• The pipeline uses dynamic branching with hundreds of sub-targets.
• There is a complete set of old sub-targets in the cache from a previous make().
• make()s use caching = "worker" with a parallel backend on a cluster.
• make()s are interrupted, either by abruptly terminating the workers (e.g. qdel or scancel) or abruptly terminating the master (e.g. kill -9 ...) or both.
• The dynamic target somehow gets incorrectly finalized using the old set of sub-targets instead of staying outdated.

cc @richardbayes

[wlandau]

6fb84de is a preemptive patch to

1. Before finalizing a dynamic target, double-check that all the sub-targets we expected to run actually ran. (Uses the sub-target keys in the dynamic progress namespace from If a dynamic target fails, can I avoid remaking those subtargets that succeeded? #1209.)
2. To prevent network file system lag from messing with (1), wrap the dynamic progress namespace key into the HPC checksum.

@richardbayes, let's try 6fb84de and see if the problem goes away.

[wlandau]

Turns out I misjudged the cause of the problem. Here is a reprex from @richardbayes, and it has nothing to do with HPC.

library(drake)

my_fun <- function(x) x

plan <- drake_plan(
  x = seq_len(5),
  y = target(my_fun(x), dynamic = map(x))
)

make(plan)
#> ▶ target x
#> ▶ dynamic y
#> > subtarget y_0b3474bd
#> > subtarget y_b2a5c9b8
#> > subtarget y_71f311ad
#> > subtarget y_98cf3c11
#> > subtarget y_0a86c9cb
#> ■ finalize y

readd(y)
#> [1] 1 2 3 4 5

my_fun <- function(x) x ^ 2

plan <- drake_plan(
  x = seq_len(3),
  y = target(my_fun(x), dynamic = map(x))
)

make(plan)
#> ▶ target x
#> ▶ dynamic y
#> > subtarget y_0b3474bd
#> > subtarget y_b2a5c9b8
#> > subtarget y_71f311ad
#> ■ finalize y

readd(y)
#> [1] 1 4 9

plan <- drake_plan(
  x = seq_len(5),
  y = target(my_fun(x), dynamic = map(x))
)
make(plan)
#> ▶ target x
#> ■ finalize y

readd(y)
#> [1] 1 4 9 4 5

^{Created on 2020-05-22 by the reprex package (v0.3.0)}

[wlandau]

I think we have to explicitly invalidate old unused sub-targets when we finalize a dynamic target. I was avoiding this because I thought it would be slow, but finalization does not actually take very long.