Importing DataInterpolations.jl takes more than 8 seconds #129

nrontsis · 2022-11-18T14:43:47Z

I am not sure if github issues is the place for this. I am opening this following this post that suggested "From now on we should treat compile time issues and compile time regressions just like we treat performance issues".

On my machine (Macbook 16 inch 2019, MacOS 13.0, Julia 1.8.2, DataInterpolations v3.10.1) I get:

julia> @time using DataInterpolations
  8.627113 seconds (14.34 M allocations: 898.668 MiB, 4.95% gc time, 26.15% compilation time: 69% of which was recompilation)

ChrisRackauckas · 2022-11-22T13:31:19Z

Can you do @time_imports using DataInterpolations? That will show what exactly is taking all of the time.

nrontsis · 2022-11-22T14:50:27Z

Sure:

julia> @time_imports using DataInterpolations
      0.4 ms  Compat
     72.0 ms  ChainRulesCore
      3.7 ms  DocStringExtensions 57.11% compilation time
      0.2 ms  SnoopPrecompile
     57.0 ms  RecipesBase
      3.7 ms  StaticArraysCore
      3.9 ms  ArrayInterfaceCore
     36.3 ms  MacroTools
      0.3 ms  ZygoteRules
      0.4 ms  Adapt
      0.5 ms  ArrayInterfaceStaticArraysCore
    165.5 ms  FillArrays
      0.2 ms  DataValueInterfaces
      1.5 ms  DataAPI
      0.2 ms  IteratorInterfaceExtensions
      0.1 ms  TableTraits
     16.5 ms  Tables
      4.1 ms  GPUArraysCore
     35.2 ms  RecursiveArrayTools
      0.2 ms  Reexport
    668.6 ms  StaticArrays
      0.3 ms  Requires
     20.1 ms  FiniteDiff 54.03% compilation time (14% recompilation)
     13.9 ms  IrrationalConstants
      1.1 ms  DiffRules
      3.3 ms  DiffResults
     30.2 ms  Preferences
      2.2 ms  OpenLibm_jll
      0.5 ms  NaNMath
      0.8 ms  ChangesOfVariables
      1.5 ms  InverseFunctions
      1.0 ms  LogExpFunctions
      0.4 ms  JLLWrappers
     46.7 ms  CompilerSupportLibraries_jll
     48.1 ms  OpenSpecFun_jll 5.09% compilation time
     23.1 ms  SpecialFunctions
      0.4 ms  CommonSubexpressions
    148.6 ms  ForwardDiff
      6.1 ms  NLSolversBase
      1.1 ms  PositiveFactorizations
      0.2 ms  UnPack
      0.5 ms  Parameters
      7.1 ms  LineSearches
     92.1 ms  DataStructures
      1.1 ms  SortingAlgorithms
     11.3 ms  Missings
      0.4 ms  StatsAPI
     33.3 ms  StatsBase
     27.7 ms  Optim
      0.3 ms  TermInterface
      0.4 ms  ExprTools
    185.7 ms  TimerOutputs
      0.3 ms  AutoHashEquals
      0.8 ms  ConstructionBase
     44.4 ms  Setfield
     11.5 ms  SplittablesBase
      0.6 ms  ArgCheck
     41.2 ms  InitialValues
    741.1 ms  BangBang 93.70% compilation time (60% recompilation)
     11.4 ms  MicroCollections
      3.6 ms  Referenceables
     93.2 ms  Baselet
      0.2 ms  CompositionsBase
      0.1 ms  DefineSingletons
     86.2 ms  Transducers 38.60% compilation time
     33.8 ms  ThreadsX
    265.2 ms  Metatheory 88.79% compilation time
     35.9 ms  IntervalSets
      2.4 ms  CompositeTypes
    149.2 ms  DomainSets
     25.7 ms  AbstractTrees
      0.1 ms  IfElse
      8.1 ms  Combinatorics
    393.0 ms  MutableArithmetics
     92.1 ms  MultivariatePolynomials
     36.6 ms  DynamicPolynomials
      1.8 ms  Bijections
      1.4 ms  PreallocationTools
     32.2 ms  LabelledArrays
    311.3 ms  SymbolicUtils
      1.3 ms  RuntimeGeneratedFunctions
      0.2 ms  CommonSolve
     17.9 ms  FunctionWrappers
      1.3 ms  FunctionWrappersWrappers
      0.4 ms  EnumX
    251.1 ms  SciMLBase
      0.2 ms  TreeViews
     31.5 ms  RandomExtensions
      6.8 ms  GroupsCore
    424.4 ms  AbstractAlgebra
      0.2 ms  IntegerMathUtils
     13.8 ms  Primes
      5.7 ms  Groebner
     30.0 ms  PDMats
    299.5 ms  Rmath_jll 91.47% compilation time (97% recompilation)
     97.8 ms  Rmath 89.93% compilation time
      3.4 ms  Calculus
    123.2 ms  DualNumbers
      1.6 ms  HypergeometricFunctions
      9.2 ms  StatsFuns
      4.6 ms  QuadGK
      2.2 ms  DensityInterface
    293.5 ms  Distributions
      1.2 ms  LaTeXStrings
      1.0 ms  Formatting
     58.1 ms  Latexify 31.86% compilation time (100% recompilation)
     42.9 ms  LambertW 40.29% compilation time
    913.7 ms  Symbolics 0.64% compilation time
    290.6 ms  MLStyle
      0.2 ms  Memoize
      0.2 ms  Underscores
     27.7 ms  LeastSquaresOptim
     28.8 ms  Lazy
      2.5 ms  RegularizationTools
    274.9 ms  DataInterpolations

ChrisRackauckas · 2022-11-22T16:16:09Z

Looks like it's mostly the symbolics stack. @YingboMa @shashi so you're aware.

GlenHertz · 2023-01-09T16:25:36Z

In my project the using MyProject time went from 12 seconds to 6 seconds when I removed Symbolics (these lines) from DataInterpolations:

https://github.com/PumasAI/DataInterpolations.jl/blob/d31cb1cd1e23cd7df4849d2cf60d774aa752d3b6/src/DataInterpolations.jl#L39-L41

Does DataInterpolations need to depend on Symbolics?

ChrisRackauckas · 2023-01-09T16:26:43Z

We can make it a weak dependency with v1.9.

GlenHertz · 2023-01-09T16:28:16Z

I was running SnoopCompile to find invalidations in my project and most of them were to do with isequal definitions in Symbolics.

ChrisRackauckas · 2023-01-09T16:38:47Z

Open an issue and share it?

GlenHertz · 2023-01-09T19:12:39Z

Added the weak dependency on 1.9 (while still being backwards compat with 1.8) with the help of @oscardssmith at #131

GlenHertz · 2023-01-09T20:13:27Z

Continued the invalidations issue in Symbolics here:
JuliaSymbolics/SymbolicUtils.jl#496

baggepinnen · 2023-01-18T14:17:24Z

The deps that arise from (I believe) the CurveFit interpolation are quite heavy

In particular MLStyle which, if Symbolics is already loaded, contributes the majority of the loading time

julia> @time_imports using DataInterpolations
    436.9 ms  MLStyle
      0.2 ms  Memoize
      0.2 ms  Underscores
     21.2 ms  LeastSquaresOptim
     27.3 ms  Lazy
      2.0 ms  RegularizationTools
    432.6 ms  DataInterpolations

nrontsis · 2023-03-21T17:02:34Z

Do you think we can close this now?

In the v4.0.0 release I get:

$ docker run --rm -it julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.8) pkg> add https://github.com/PumasAI/DataInterpolations.jl

julia> @time_imports using DataInterpolations
      9.7 ms  DocStringExtensions 30.09% compilation time
     30.1 ms  Preferences
      0.2 ms  SnoopPrecompile
     53.7 ms  RecipesBase
      4.3 ms  StaticArraysCore
      0.5 ms  Requires
    134.6 ms  ArrayInterface 78.50% compilation time (2% recompilation)
      1.1 ms  SymbolicIndexingInterface
      0.9 ms  Compat
     79.0 ms  ChainRulesCore
     41.1 ms  MacroTools
      0.4 ms  ZygoteRules
     17.1 ms  Adapt 59.95% compilation time
    197.6 ms  FillArrays
      0.2 ms  DataValueInterfaces
      1.8 ms  DataAPI
      0.2 ms  IteratorInterfaceExtensions
      0.2 ms  TableTraits
      8.8 ms  OrderedCollections
     33.4 ms  Tables
      4.7 ms  GPUArraysCore
     65.4 ms  RecursiveArrayTools 16.59% compilation time
     66.3 ms  DataInterpolations 67.38% compilation time

Either way, thanks so much for all the awesome work 😍

ChrisRackauckas · 2023-03-21T21:47:52Z

I wonder what's pulling in FillArrays... but it's not us so that's a problem for another time. Cheers.

nrontsis · 2023-03-22T10:46:50Z

I wonder what's pulling in FillArrays.

I think it's RecursiveArrayTools

julia> PkgDependency.tree("DataInterpolations")
 DataInterpolations v4.0.0
━━━━━━━━━━━━━━━━━━━━━━━━━━━
            │
            ├── RecipesBase v1.3.3
            │   └── SnoopPrecompile v1.0.3
            │       └── Preferences v1.3.0
            │           └── TOML v1.0.0
            ├── RecursiveArrayTools v2.38.0
            │   ├── StaticArraysCore v1.4.0
            │   ├── GPUArraysCore v0.1.4
            │   │   └── Adapt v3.6.1
            │   │       └── Requires v1.3.0
            │   ├── IteratorInterfaceExtensions v1.0.0
            │   ├── ZygoteRules v0.2.3
            │   │   ├── ChainRulesCore v1.15.7
            │   │   │   └── Compat v4.6.1
            │   │   └── MacroTools v0.5.10
            │   ├── Tables v1.10.1
            │   │   ├── DataAPI v1.14.0
            │   │   ├── OrderedCollections v1.4.1
            │   │   ├── IteratorInterfaceExtensions v1.0.0 (*)
            │   │   ├── DataValueInterfaces v1.0.0
            │   │   └── TableTraits v1.0.1
            │   │       └── IteratorInterfaceExtensions v1.0.0 (*)
            │   ├── DocStringExtensions v0.9.3
            │   ├── ArrayInterface v7.2.1
            │   │   ├── Requires v1.3.0 (*)
            │   │   ├── SnoopPrecompile v1.0.3 (*)
            │   │   └── Adapt v3.6.1 (*)
            │   ├── SymbolicIndexingInterface v0.2.2
            │   │   └── DocStringExtensions v0.9.3 (*)
            │   ├── RecipesBase v1.3.3 (*)
            │   ├── ChainRulesCore v1.15.7 (*)
            │   ├── Requires v1.3.0 (*)
            │   ├── FillArrays v0.13.9
            │   └── Adapt v3.6.1 (*)
            ├── Requires v1.3.0 (*)
            └── Reexport v1.2.2

ChrisRackauckas · 2023-03-25T15:35:03Z

FillArrays is addressed in SciML/RecursiveArrayTools.jl#257

It was added in b3ed973 but in SciML/DataInterpolations.jl#129 it has been identified as one of the main import time contributions. It's only necessary for the Zygote rules, and not even ChainRules but specifically `Zygote.@adjoint`, so they are simply moved to a Zygote extension and use FillArrays from Zygote to achieve the same goal.

ChrisRackauckas closed this as completed Mar 21, 2023

ChrisRackauckas mentioned this issue Mar 25, 2023

Remove FillArrays dependency to reduce import times SciML/RecursiveArrayTools.jl#257

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Importing DataInterpolations.jl takes more than 8 seconds #129

Importing DataInterpolations.jl takes more than 8 seconds #129

nrontsis commented Nov 18, 2022 •

edited

Loading

ChrisRackauckas commented Nov 22, 2022

nrontsis commented Nov 22, 2022

ChrisRackauckas commented Nov 22, 2022

GlenHertz commented Jan 9, 2023 •

edited

Loading

ChrisRackauckas commented Jan 9, 2023

GlenHertz commented Jan 9, 2023 •

edited

Loading

ChrisRackauckas commented Jan 9, 2023

GlenHertz commented Jan 9, 2023

GlenHertz commented Jan 9, 2023

baggepinnen commented Jan 18, 2023

nrontsis commented Mar 21, 2023 •

edited

Loading

ChrisRackauckas commented Mar 21, 2023

nrontsis commented Mar 22, 2023

ChrisRackauckas commented Mar 25, 2023

Importing DataInterpolations.jl takes more than 8 seconds #129

Importing DataInterpolations.jl takes more than 8 seconds #129

Comments

nrontsis commented Nov 18, 2022 • edited Loading

ChrisRackauckas commented Nov 22, 2022

nrontsis commented Nov 22, 2022

ChrisRackauckas commented Nov 22, 2022

GlenHertz commented Jan 9, 2023 • edited Loading

ChrisRackauckas commented Jan 9, 2023

GlenHertz commented Jan 9, 2023 • edited Loading

ChrisRackauckas commented Jan 9, 2023

GlenHertz commented Jan 9, 2023

GlenHertz commented Jan 9, 2023

baggepinnen commented Jan 18, 2023

nrontsis commented Mar 21, 2023 • edited Loading

ChrisRackauckas commented Mar 21, 2023

nrontsis commented Mar 22, 2023

ChrisRackauckas commented Mar 25, 2023

nrontsis commented Nov 18, 2022 •

edited

Loading

GlenHertz commented Jan 9, 2023 •

edited

Loading

GlenHertz commented Jan 9, 2023 •

edited

Loading

nrontsis commented Mar 21, 2023 •

edited

Loading