'Stopped early on line...' when data is large

[Yiguan]

Hi,

Here is a weird issue (probably a bug) when using fread on a large data set. I have reproduced the issue on different computers and systems.

This is my code:

library(data.table)
N_large = 5000000

ii = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\tbb\tccccccccc\tdddddd\tee\tffff\tg\thhhhhh\tiiii\tjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj\tkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk\tllllll\tmmmmmmmmmmmmmm\tnnnnnnnnn\toooooooo\tppppppp\tqqqqqqq'

d_large <- fread(text=rep(ii,N_large), header=FALSE, sep='\t', verbose=TRUE)

I alway get a warning message:

Warning message:
In fread(text = rep(ii, N_large), header = FALSE, sep = "\t", verbose = TRUE) :
  Stopped early on line 4836675. Expected 17 fields but found 0. Consider fill=TRUE and comment.char=. First discarded non-empty line: <<aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa    bb      ccccccccc       dddddd  ee      ffff    g       hhhhhh  iiii    jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj     kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk     llllll  mmmmmmmmmmmmmm  nnnnnnnnn       oooooooo        ppppppp qqqqqqq>>

It stopped on line 4836675 and the following lines were not read in. If I use fill=TRUE, the warning message disappears but an empty line inserts in the data table and the total rows will be N_large + 1. I also tried read.table() and I didn't see such an issue.

However, when data is small, e.g. N_large = 500, no errors, no warnings, fread works well.

---

Here is my R info:

R version 3.6.0 (2019-04-26)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Scientific Linux 7.9 (Nitrogen)

Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8
 [5] LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_GB.UTF-8
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] data.table_1.13.0

loaded via a namespace (and not attached):
[1] compiler_3.6.0

Here is the fread verbose output:

  omp_get_num_procs()            24
  R_DATATABLE_NUM_PROCS_PERCENT  unset (default 50)
  R_DATATABLE_NUM_THREADS        unset
  R_DATATABLE_THROTTLE           unset (default 1024)
  omp_get_thread_limit()         2147483647
  omp_get_max_threads()          24
  OMP_THREAD_LIMIT               unset
  OMP_NUM_THREADS                unset
  RestoreAfterFork               true
  data.table is using 12 threads with throttle==1024. See ?setDTthreads.
Input contains no \n. Taking this to be a filename to open
[01] Check arguments
  Using 12 threads (omp_get_max_threads()=24, nth=12)
  NAstrings = [<<NA>>]
  None of the NAstrings look like numbers.
  show progress = 1
  0/1 column will be read as integer
[02] Opening the file
  Opening file /tmp/RtmpokgBEl/file331c321bc62b
  File opened, size = 2.063GB (2215000001 bytes).
  Memory mapped ok
[03] Detect and skip BOM
[04] Arrange mmap to be \0 terminated
  \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal.
[05] Skipping initial rows if needed
  Positioned on line 1 starting: <<aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa>>
[06] Detect separator, quoting rule, and ncolumns
  Using supplied sep '\t'
  sep=0x9  with 100 lines of 17 fields using quote rule 0
  Detected 17 columns on line 1. This line is either column names or first data row. Line starts as: <<aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa>>
  Quote rule picked = 0
  fill=false and the most number of columns found is 17
[07] Detect column types, good nrow estimate and whether first row is column names
  'header' changed by user from 'auto' to false
  Number of sampling jump points = 100 because (2215000000 bytes from row 1 to eof) / (2 * 44300 jump0size) == 25000
  Type codes (jump 000)    : CCCCCCCCCCCCCCCCC  Quote rule 0
  Type codes (jump 100)    : CCCCCCCCCCCCCCCCC  Quote rule 0
  =====
  Sampled 10050 rows (handled \n inside quoted fields) at 101 jump points
  Bytes from first data row on line 1 to the end of last row: 2215000000
  Line length: mean=443.00 sd=0.01 min=442 max=443
  Estimated number of rows: 2215000000 / 443.00 = 5000002
  Initial alloc = 5500002 rows (5000002 + 9%) using bytes/max(mean-2*sd,min) clamped between [1.1*estn, 2.0*estn]
  =====
[08] Assign column names
[09] Apply user overrides on column types
  After 0 type and 0 drop user overrides : CCCCCCCCCCCCCCCCC
[10] Allocate memory for the datatable
  Allocating 17 column slots (17 - 0 dropped) with 5500002 rows
[11] Read the data
  jumps=[0..2112), chunk_size=1048768, total_size=2215000000
|--------------------------------------------------|
|================================================  Restarting team from jump 2043. nSwept==0 quoteRule==1
  jumps=[2043..2112), chunk_size=1048768, total_size=2215000000
  Restarting team from jump 2043. nSwept==0 quoteRule==2
  jumps=[2043..2112), chunk_size=1048768, total_size=2215000000
  Restarting team from jump 2043. nSwept==0 quoteRule==3
  jumps=[2043..2112), chunk_size=1048768, total_size=2215000000
==|
Read 4836674 rows x 17 columns from 2.063GB (2215000001 bytes) file in 00:19.425 wall clock time
[12] Finalizing the datatable
  Type counts:
        17 : string    'C'
=============================
   0.000s (  0%) Memory map 2.063GB file
   0.014s (  0%) sep='\t' ncol=17 and header detection
   0.000s (  0%) Column type detection using 10050 sample rows
   2.082s ( 11%) Allocation of 5500002 rows x 17 cols (0.697GB) of which 4836674 ( 88%) rows used
  17.329s ( 89%) Reading 2112 chunks (0 swept) of 1.000MB (each chunk 2290 rows) using 12 threads
   +    0.660s (  3%) Parse to row-major thread buffers (grown 0 times)
   +   12.294s ( 63%) Transpose
   +    4.375s ( 23%) Waiting
   0.000s (  0%) Rereading 0 columns due to out-of-sample type exceptions
  19.425s        Total

[ben-schwen]

Ty for the report. I just ran your example on current dev and 1.14.2.

The goods news: It seems that the issue is fixed in current dev, while it still stops early on 1.14.2.

The bads news: I cannot find the issue/PR in NEWS which is responsible for fixing it.

[ben-schwen]

Ok bisection shows that #4802 fixed your example.

Does the error appear on reading from a file or using fread(text=)?

[ben-schwen]

Checking the line in tmpfile with using cat(text, file=tmp, sep="\n") shows indeed that line 4836674 is an empty line.

Seems like you found a mysterious bug in base::cat, reported this to R bug tracker (https://bugs.r-project.org/show_bug.cgi?id=18340).

N_large = 5000000
ii = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\tbb\tccccccccc\tdddddd\tee\tffff\tg\thhhhhh\tiiii\tjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj\tkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk\tllllll\tmmmmmmmmmmmmmm\tnnnnnnnnn\toooooooo\tppppppp\tqqqqqqq'
tmp = tempfile()
cat(rep(ii,N_large), file=tmp, sep="\n")
system(command = sprintf("cat %s | uniq -c", tmp))
# 4836674 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa bb      ccccccccc       dddddd  ee      ffff    g       hhhhhh  iiii    jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj     kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk llllll  mmmmmmmmmmmmmm  nnnnnnnnn       oooooooo        ppppppp     qqqqqqq
#       1 
#  163326 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa bb      ccccccccc       dddddd  ee      ffff    g       hhhhhh  iiii    jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj     kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk llllll  mmmmmmmmmmmmmm  nnnnnnnnn       oooooooo        ppppppp     qqqqqqq

[ben-schwen]

This is fixed in current devel of R. There is also a separate fix for data.table in #4802 so I'm closing here.

Feel free to reopen if the issue reoccurs.

[rbutleriii]

This can still happen when the number of columns is highly variable. For instance this file which has some rows with 3 columns and some with >3000. Is this still the R base issue?

> annot = fread("AdultBrain.genes.annot", header=F, fill=T)
Warning message:
In fread("AdultBrain.genes.annot", header = F, fill = T) :
  Stopped early on line 496. Expected 1008 fields but found 1074. Consider fill=TRUE and comment.char=. First discarded non-empty line: <<ENSG00000036448        8:1993155:2093380       rs13272630      rs149509713      rs112324702     rs13264775      rs55913329      rs66739903      rs35389636      rs151271118     rs1562925        rs62488537      rs9650539       rs115417135     rs11778096      rs17740775      rs17740799       rs17668154      rs17668166      rs6983981       rs187829154     rs7000148       rs7004042       rs2600487        rs2701901       rs12677463      rs6559207       rs187395437     rs4735980       rs4735981rs71516150      rs6559208       rs34510005      rs143661164     rs2701899       rs62487807      rs62487810       rs145992668     rs138692710     rs142828803     rs146094290     rs187725070     rs193298979     rs75669442       rs7>>

R version 4.2.2 (2022-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblas-r0.3.3.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] stringi_1.7.12    data.table_1.14.8

loaded via a namespace (and not attached):
[1] compiler_4.2.2    tools_4.2.2       R.methodsS3_1.8.2 R.utils_2.12.2    R.oo_1.25.0

[ben-schwen]

@rbutleriii thanks for your example which is also not working on current dev.

There is already a PR #5119 tackling this issue. Using the version of the PR I can read your file with fread("Adult_brain.genes.annot", fill=3435L) after getting to this estimate (I skipped the discarded non-empty lines)

Warning message:
In fread("~/Downloads/Adult_brain.genes.annot", fill = TRUE) :
  Stopped early on line 496. Expected 1008 fields but found 1074. Consider fill=1074 or even higher ncol estimate. First discarded non-empty line:

Warning message:
In fread("~/Downloads/Adult_brain.genes.annot", fill = 1074) :
  Stopped early on line 517. Expected 1074 fields but found 1358. Consider fill=1358 or even higher ncol estimate. First discarded non-empty line:

Warning message:
In fread("~/Downloads/Adult_brain.genes.annot", fill = 1358) :
  Stopped early on line 542. Expected 1358 fields but found 1525. Consider fill=1525 or even higher ncol estimate. First discarded non-empty line:

Warning message:
In fread("~/Downloads/Adult_brain.genes.annot", fill = 1525) :
  Stopped early on line 968. Expected 1525 fields but found 3435. Consider fill=3435 or even higher ncol estimate. First discarded non-empty line:

[Yunuuuu]

is it possible to prevent warning: stop early?

[Yunuuuu]

I have used fread in FAERS data and Drugs@FDA, all failed to read the whole file, even set fill = TRUE

[ben-schwen]

@Yunuuuu I'm sorry but without a reproducible example, I'm afraid we cannot help. The branch in #5119 might solve your problem by using an integer guess for the number of columns.

[Yunuuuu]

Thanks for your reply, I uploaded a file Products.txt downloaded from Drugs@FDA. However, I am unable to use a branch solely on GitHub because I intend to submit my package to Bioconductor, which requires the use of packages hosted on CRAN or Bioconductor's repositories. I finally choose import vroom to read these files

# file path downloaded from above
data.table::fread("path")

data.table::fread("Products.txt", fill = TRUE)
#> Warning in data.table::fread("Products.txt", fill = TRUE): Stopped early on
#> line 35466. Expected 8 fields but found 9. Consider fill=TRUE and
#> comment.char=. First discarded non-empty line: <<206029 001 TABLET;ORAL EQ 1MG
#> BASE 0 PITAVASTATIN CALCIUM PITAVASTATIN CALCIUM >>
#>        ApplNo ProductNo                           Form        Strength
#>         <int>     <int>                         <char>          <char>
#>     1:      4         4      SOLUTION/DROPS;OPHTHALMIC              1%
#>     2:    159         1                    TABLET;ORAL           500MG
#>     3:    552         1           INJECTABLE;INJECTION 20,000 UNITS/ML
#>     4:    552         2           INJECTABLE;INJECTION 40,000 UNITS/ML
#>     5:    552         3           INJECTABLE;INJECTION  5,000 UNITS/ML
#>    ---                                                                
#> 35460: 206026         1                 GAS;INHALATION             N/A
#> 35461: 206028         1 CAPSULE, EXTENDED RELEASE;ORAL             7MG
#> 35462: 206028         2 CAPSULE, EXTENDED RELEASE;ORAL            14MG
#> 35463: 206028         3 CAPSULE, EXTENDED RELEASE;ORAL            21MG
#> 35464: 206028         4 CAPSULE, EXTENDED RELEASE;ORAL            28MG
#>        ReferenceDrug                DrugName                ActiveIngredient
#>                <int>                  <char>                          <char>
#>     1:             0               PAREDRINE HYDROXYAMPHETAMINE HYDROBROMIDE
#>     2:             0           SULFAPYRIDINE                   SULFAPYRIDINE
#>     3:             0        LIQUAEMIN SODIUM                  HEPARIN SODIUM
#>     4:             0        LIQUAEMIN SODIUM                  HEPARIN SODIUM
#>     5:             0        LIQUAEMIN SODIUM                  HEPARIN SODIUM
#>    ---                                                                      
#> 35460:             0             HELIUM, USP                          HELIUM
#> 35461:             0 MEMANTINE HYDROCHLORIDE         MEMANTINE HYDROCHLORIDE
#> 35462:             0 MEMANTINE HYDROCHLORIDE         MEMANTINE HYDROCHLORIDE
#> 35463:             0 MEMANTINE HYDROCHLORIDE         MEMANTINE HYDROCHLORIDE
#> 35464:             0 MEMANTINE HYDROCHLORIDE         MEMANTINE HYDROCHLORIDE
#>        ReferenceStandard
#>                    <int>
#>     1:                 0
#>     2:                 0
#>     3:                 0
#>     4:                 0
#>     5:                 0
#>    ---                  
#> 35460:                NA
#> 35461:                 0
#> 35462:                 0
#> 35463:                 0
#> 35464:                 0

^{Created on 2023-11-07 with reprex v2.0.2}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.1 (2023-06-16)
#>  os       Ubuntu 22.04.3 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language en
#>  collate  C.UTF-8
#>  ctype    C.UTF-8
#>  tz       Asia/Shanghai
#>  date     2023-11-07
#>  pandoc   2.9.2.1 @ /usr/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date (UTC) lib source
#>  cli           3.6.1   2023-03-23 [1] CRAN (R 4.3.1)
#>  data.table    1.14.9  2023-11-03 [1] Github (Rdatatable/data.table@e6076b0)
#>  digest        0.6.33  2023-07-07 [1] CRAN (R 4.3.1)
#>  evaluate      0.21    2023-05-05 [1] CRAN (R 4.3.1)
#>  fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.1)
#>  fs            1.6.3   2023-07-20 [1] CRAN (R 4.3.1)
#>  glue          1.6.2   2022-02-24 [1] CRAN (R 4.3.1)
#>  htmltools     0.5.5   2023-03-23 [1] CRAN (R 4.3.1)
#>  knitr         1.43    2023-05-25 [1] CRAN (R 4.3.1)
#>  lifecycle     1.0.3   2022-10-07 [1] CRAN (R 4.3.1)
#>  magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.3.1)
#>  purrr         1.0.1   2023-01-10 [1] CRAN (R 4.3.1)
#>  R.cache       0.16.0  2022-07-21 [1] CRAN (R 4.3.1)
#>  R.methodsS3   1.8.2   2022-06-13 [1] CRAN (R 4.3.1)
#>  R.oo          1.25.0  2022-06-12 [1] CRAN (R 4.3.1)
#>  R.utils       2.12.2  2022-11-11 [1] CRAN (R 4.3.1)
#>  reprex        2.0.2   2022-08-17 [1] CRAN (R 4.3.1)
#>  rlang         1.1.1   2023-04-28 [1] CRAN (R 4.3.1)
#>  rmarkdown     2.23    2023-07-01 [1] CRAN (R 4.3.1)
#>  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.1)
#>  styler        1.10.1  2023-06-05 [1] CRAN (R 4.3.1)
#>  vctrs         0.6.3   2023-06-14 [1] CRAN (R 4.3.1)
#>  withr         2.5.0   2022-03-03 [1] CRAN (R 4.3.1)
#>  xfun          0.39    2023-04-20 [1] CRAN (R 4.3.1)
#>  yaml          2.3.7   2023-01-23 [1] CRAN (R 4.3.1)
#> 
#>  [1] /home/yun/Rlibrary/4.3
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

[ben-schwen]

@Yunuuuu Ty. Your example works with the branch in #5119 with fread("Products.txt", fill=9) where fill=9 is also the first provided guess. However, it might take some time until this feature makes its way to the version on CRAN.

In fread("Products.txt", fill = TRUE) :
  Stopped early on line 35466. Expected 8 fields but found 9. Consider fill=9 or even higher ncol estimate. First discarded non-empty line: <<206029	001	TABLET;ORAL	EQ 1MG BASE	0	PITAVASTATIN CALCIUM	PITAVASTATIN CALCIUM		>>

[Yunuuuu]

Thank you, @ben-schwen, for your assistance! I appreciate your help in testing the example using the branch in #5119. It's great to hear that it works and fills the purpose well. I understand that it may take some time for this feature to be included in the version available on CRAN. Nevertheless, your contribution is valuable, and I look forward to seeing this feature implemented in a future update. Presently, I shall avail myself of the vroom utility to facilitate the package integrated into Bioconductor, though I want to solely employ data.table for this purpose.