filelock error when working with drake plan

[mattwarkentin]

Hi,

Seemingly out of nowhere I keep getting the following error when I try to run any function that acts on my drake plan (e.g. visualization):

plan <- drake_plan()
vis_drake_graph(plan)

Error in filelock::lock(private$lock_file) : 
  Cannot lock file: '/Matt/.drake/drake/history/lock': Operation not supported
Error in filelock::unlock(lock) : 
  VECTOR_ELT() can only be applied to a 'list', not a 'closure'

I had a working plan, then this error started showing up. I worked backwards to slowly try to figure out what the issue was by removing functions and targets in the plan. Even clearing the entire global environment, resetting R, and creating an empty plan doesn't work...

I honestly have no idea why this started happening. I have even tried deleting the .drake directory, resetting R, and trying again but it still happens. Any idea what this error means? How to fix it?

[wlandau]

This is an instance of #1232. If you remove .drake/drake/history and set history = FALSE for the next make(), the error should go away. Unfortunately, I am having trouble reproducing this general issue myself, and it is coming from a package outside drake (filelock), so I think that's all we can do for now.

[mattwarkentin]

Thanks for the response, but I had actually never ran make() yet for this project. I am still defining functions and targets, visualizing the structure as I go.

I tried your suggestions of deleting .drake/drake/history, but I still get the same error when I simply load my functions (by sourceing R scripts), define my plan by running plan <- drake_plan(...), and try to run vis_drake_graph(plan) or outdated(plan). Same error as before.

This seems to be independent of make() as I have never used this yet. Any other suggestions?

[mattwarkentin]

I now ran make(plan, history = FALSE) and it made my targets seemingly without issue. Then I tried to readd() in a target value or visualize the network again and I keep getting the filelock error for both.

[wlandau]

vis_drake_graph() might touch history because it goes through drake_config(). I am surprised that readd() is causing that, but it could be an easier way to narrow down the issue. Would you post a reproducible example and include your sessionInfo()?

[mattwarkentin]

I can't really reproduce this issue. But I have seemingly narrowed it down to possibly a permissions issue. The original directory is located within a mounted volume on my Mac (network drive at my institute), and not the internal storage on my Mac.

For some reason when I copied the whole directory over to my local machine it worked...so could it be a permissions issue? Within the network drive, the project is in a folder that I have full read/write permissions so I can't imagine why I would be limited in any way.

I really am at a loss for why this doesn't work when the directory is opened from its normal location.

[mattwarkentin]

Session Info

R version 4.0.0 (2020-04-24)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.5

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
 [1] ggtext_0.1.0      gghighlight_0.3.0 ggforce_0.3.1    
 [4] assertthat_0.2.1  progress_1.2.2    here_0.1         
 [7] fs_1.4.1          glue_1.4.1        rhdf5_2.32.0     
[10] keras_2.3.0.0     reticulate_1.16   lubridate_1.7.9  
[13] vroom_1.2.1       readxl_1.3.1      openxlsx_4.1.5   
[16] yardstick_0.0.6   workflows_0.1.1   tune_0.1.0       
[19] rsample_0.0.7     recipes_0.1.12    parsnip_0.1.1    
[22] infer_0.5.2       dials_0.0.7       scales_1.1.1     
[25] broom_0.5.6       tidymodels_0.1.0  forcats_0.5.0    
[28] stringr_1.4.0     dplyr_1.0.0       purrr_0.3.4      
[31] readr_1.3.1       tidyr_1.1.0       tibble_3.0.1     
[34] ggplot2_3.3.2     tidyverse_1.3.0   drake_7.12.2     
[37] dotenv_1.0.2      conflicted_1.0.4  testthat_2.3.2   
[40] devtools_2.3.0    usethis_1.6.1    

loaded via a namespace (and not attached):
  [1] tidyselect_1.1.0     lme4_1.1-23         
  [3] htmlwidgets_1.5.1    grid_4.0.0          
  [5] pROC_1.16.2          munsell_0.5.0       
  [7] base64url_1.4        codetools_0.2-16    
  [9] statmod_1.4.34       DT_0.13             
 [11] future_1.17.0        miniUI_0.1.1.1      
 [13] withr_2.2.0          colorspace_1.4-1    
 [15] filelock_1.0.2       knitr_1.28          
 [17] prompt_1.0.0         rstudioapi_0.11     
 [19] stats4_4.0.0         tensorflow_2.2.0    
 [21] bayesplot_1.7.2      listenv_0.8.0       
 [23] rstan_2.19.3         polyclip_1.10-0     
 [25] farver_2.0.3         bit64_0.9-7         
 [27] DiceDesign_1.8-1     txtq_0.2.0          
 [29] rprojroot_1.3-2      vctrs_0.3.1         
 [31] generics_0.0.2       ipred_0.9-9         
 [33] xfun_0.15            R6_2.4.1            
 [35] markdown_1.1         rstanarm_2.19.3     
 [37] lhs_1.0.2            promises_1.1.1      
 [39] nnet_7.3-14          gtable_0.3.0        
 [41] globals_0.12.5       processx_3.4.2      
 [43] timeDate_3043.102    rlang_0.4.6         
 [45] clisymbols_1.2.0     zeallot_0.1.0       
 [47] splines_4.0.0        inline_0.3.15       
 [49] reshape2_1.4.4       modelr_0.1.8        
 [51] tidytext_0.2.4       threejs_0.3.3       
 [53] crosstalk_1.1.0.1    backports_1.1.8     
 [55] httpuv_1.5.4         rsconnect_0.8.16    
 [57] tokenizers_0.2.1     gridtext_0.1.1      
 [59] tools_4.0.0          lava_1.6.7          
 [61] ellipsis_0.3.1       sessioninfo_1.1.1   
 [63] ggridges_0.5.2       Rcpp_1.0.4.6        
 [65] plyr_1.8.6           base64enc_0.1-3     
 [67] ps_1.3.3             prettyunits_1.1.1   
 [69] rpart_4.1-15         zoo_1.8-8           
 [71] haven_2.3.1          furrr_0.1.0         
 [73] magrittr_1.5         colourpicker_1.0    
 [75] reprex_0.3.0         GPfit_1.0-8         
 [77] SnowballC_0.7.0      whisker_0.4         
 [79] packrat_0.5.0        storr_1.2.1         
 [81] matrixStats_0.56.0   pkgload_1.1.0       
 [83] tidyposterior_0.0.3  hms_0.5.3           
 [85] shinyjs_1.1          mime_0.9            
 [87] xtable_1.8-4         tidypredict_0.4.5   
 [89] shinystan_2.5.0      tfruns_1.4          
 [91] gridExtra_2.3        rstantools_2.1.0    
 [93] compiler_4.0.0       crayon_1.3.4        
 [95] minqa_1.2.4          StanHeaders_2.21.0-5
 [97] htmltools_0.5.0      later_1.1.0.1       
 [99] RcppParallel_5.0.1   DBI_1.1.0           
[101] tweenr_1.0.1         dbplyr_1.4.4        
[103] MASS_7.3-51.6        boot_1.3-25         
[105] Matrix_1.2-18        cli_2.0.2           
[107] parallel_4.0.0       gower_0.2.1         
[109] igraph_1.2.5         pkgconfig_2.0.3     
[111] xml2_1.3.2           foreach_1.5.0       
[113] dygraphs_1.1.1.6     prodlim_2019.11.13  
[115] rvest_0.3.5          janeaustenr_0.1.5   
[117] callr_3.4.3          digest_0.6.25       
[119] cellranger_1.1.0     shiny_1.4.0.2       
[121] gtools_3.8.2         nloptr_1.2.2.1      
[123] lifecycle_0.2.0      nlme_3.1-148        
[125] jsonlite_1.6.1       Rhdf5lib_1.10.0     
[127] desc_1.2.0           fansi_0.4.1         
[129] pillar_1.4.4         lattice_0.20-41     
[131] loo_2.2.0            fastmap_1.0.1       
[133] httr_1.4.1           pkgbuild_1.0.8      
[135] survival_3.2-3       xts_0.12-0          
[137] remotes_2.1.1        zip_2.0.4           
[139] shinythemes_1.1.2    iterators_1.0.12    
[141] bit_1.1-15.2         class_7.3-17        
[143] stringi_1.4.6        blob_1.2.1          
[145] memoise_1.1.0

[wlandau]

Thanks. And next time you see the error, would you post a traceback too?

[mattwarkentin]

## Load packages ----
source("_packages.R")

## Source R files ----
purrr::walk(fs::dir_ls("./R", glob = "*.R"), source)

vis_drake_graph(plan)

19: filelock::unlock(lock)
18: private$txtq_exclusive({
        file_create(private$db_file)
        if (!file.exists(private$head_file)) {
            private$txtq_set_head(0)
        }
        if (!file.exists(private$total_file)) {
            private$txtq_set_total(0)
        }
    })
17: private$txtq_establish(path)
16: .subset2(public_bind_env, "initialize")(...)
15: R6_txtq$new(path = path)
14: txtq::txtq(history_path)
13: history_queue(history_path)
12: recover_default_history(path)
11: methods::initRefFields(.Object, classDef, selfEnv, list(...))
10: initialize(value, ...)
9: initialize(value, ...)
8: methods::new(def, ...)
7: refclass_decorated_storr$new(storr = storr, driver = storr$driver, 
       default_namespace = storr$default_namespace, envir = storr$envir, 
       hash_algorithm = hash_algorithm, digest = digest, history = recover_default_history(path), 
       ht_encode_path = ht_new(), ht_decode_path = ht_new(), ht_encode_namespaced = ht_new(), 
       ht_decode_namespaced = ht_new(), ht_hash = ht_new(), ht_keys = ht_keys(digest), 
       path = path, path_return = file.path(path, "drake", "return"), 
       path_tmp = file.path(path, "drake", "tmp"))
6: decorate_storr(storr::storr_rds(path = path))
5: drake_fetch_rds(path = path)
4: this_cache_(path = path)
3: drake::drake_cache()
2: drake_config2(...)
1: vis_drake_graph(plan)

[wlandau]

Okay, that really helps. Does txtq::txtq(".drake/drake/history") give the same error? If so, would you be willing to upload that history folder here as a zip archive? In the meantime, I will try a workaround in the txtq package, which I also maintain.

[mattwarkentin]

txtq::txtq(".drake/drake/history")

Error in filelock::lock(private$lock_file) : 
  Cannot lock file: '/Volumes/hung_lab/shared/Matt/deep-learning/.drake/drake/history/lock': Operation not supported
Error in filelock::unlock(lock) : 
  VECTOR_ELT() can only be applied to a 'list', not a 'closure'

6: filelock::unlock(lock)
5: private$txtq_exclusive({
       file_create(private$db_file)
       if (!file.exists(private$head_file)) {
           private$txtq_set_head(0)
       }
       if (!file.exists(private$total_file)) {
           private$txtq_set_total(0)
       }
   })
4: private$txtq_establish(path)
3: .subset2(public_bind_env, "initialize")(...)
2: R6_txtq$new(path = path)
1: txtq::txtq(".drake/drake/history")

[mattwarkentin]

I will upload the history folder, one sec.

[mattwarkentin]

I think this is what you wanted. This is a zip of .drake/drake/history

history.zip

[wlandau]

Thanks for that. Odd, but I cannot reproduce the error with txtq::txtq("history") with either development txtq or CRAN txtq. What happens if you run this?

library(filelock)
lock <- lock("history/lock")
unlock(lock)

[wlandau]

Also worth checking file.access("history/lock", mode = x) for x = 0, 1, 2, and 4. If for some reason you do not have permission to write the file, that might cause problems.

[mattwarkentin]

library(filelock)
lock <- lock(".drake/drake/history/lock")
unlock(lock)

Error in lock(".drake/drake/history/lock") : 
  Cannot lock file: '/Volumes/hung_lab/shared/Matt/deep-learning/.drake/drake/history/lock': Operation not supported
Error in unlock(lock) : 
  VECTOR_ELT() can only be applied to a 'list', not a 'closure'

Regarding permission:

fs::file_access(".drake/drake/history/lock", c("read", "write", "execute"))

.drake/drake/history/lock 
                     TRUE 

[wlandau]

I think we have narrowed down the issue as much as we can here. I think the next thing to do is ask Gabor about it on the filelock issue tracker. Would you post a new issue to https://github.com/r-lib/filelock and cc me?

By the way, thank you so much for helping us get to the bottom of this issue. I have heard other reports of this happening (which sometimes could be avoided by suppressing history) but it is super tough to reproduce on my end.

[mattwarkentin]

No problem, I'll post the issue over on filelock GitHub right now.

Not a problem; happy to help any way I can. Thanks a lot for your help. I am fairly new to {drake} which makes me feel even more helpless when this error showed up. Thanks for being so responsive and working toward a solution.

[wlandau]

You know what? After reading the discussions of file systems at r-lib/filelock#26, I think I need to revisit wlandau/txtq#18.

[wlandau]

NB the fix requires txtq version 0.2.1, which is only on GitHub now (remotes::install_github("wlandau/txtq")).

[wlandau]

@pydupont confirmed the patch works, so I am closing. Please let me know if it persists for you.

[mattwarkentin]

@wlandau I also confirm everything seems to be working perfectly after installing the dev versions of txtq and drake. Thanks so much for all of your hard work.