Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installing xarray-leaflet causes jupyter server to be really slow to start #79

Closed
yuvipanda opened this issue Jan 12, 2023 · 9 comments · Fixed by #80
Closed

Installing xarray-leaflet causes jupyter server to be really slow to start #79

yuvipanda opened this issue Jan 12, 2023 · 9 comments · Fixed by #80

Comments

@yuvipanda
Copy link

On a fresh install (no .pyc files), just importing xarray_leaflet takes about 9s:

(notebook) jovyan@jupyter-yuvipanda:~$ time python -c 'import xarray_leaflet'


real    0m9.275s
user    0m8.100s
sys     0m1.134s

As this is imported each time jupyter server is started, the user can not see their notebook until this initialization is complete. When coupled with other slow loading extensions (such as panel holoviz/panel#4271), this can cause server starts to time out in JupyterHubs. There is no way to disable this (jupyter server will still import your package even if it is disabled), so the only way to get a faster startup is to uninstall xarray-leaflet completely.

I think a majority of the import time comes from rioxarray

(notebook) jovyan@jupyter-yuvipanda:~$ time python -c 'import rioxarray'

real    0m7.707s
user    0m5.787s
sys     0m0.912s
yuvipanda added a commit to yuvipanda/pangeo-docker-images that referenced this issue Jan 12, 2023
Panel and Xarray-Leaflet are heavy enough imports that
without .pyc files, they sometimes together take as much as
15s to import?! This causes jupyterhub to fail startup in
some cases.

The longer term fix is in panel and xarray-leaflet (
see holoviz/panel#4271,
xarray-contrib/xarray_leaflet#79).

In the meantime, leaving the .pyc files in place doesn't
increase the image size by much, but makes startup definitely
much faster!

Ref 2i2c-org/infrastructure#2047
@yuvipanda
Copy link
Author

I opened jupyter-server/jupyter_server#1171 to make this information available more easily.

@davidbrochart
Copy link
Collaborator

Thanks for reporting the issue @yuvipanda.
Maybe we could implement something like lazy importing in xarray-leaflet?

@yuvipanda
Copy link
Author

@davidbrochart yep that would seem to be the solution

@davidbrochart
Copy link
Collaborator

Actually xarray-leaflet doesn't import rioxarray. It depends on it and it uses properties of xarray's DataArray that exist because the array was opened with rioxarray, but that's external to xarray-leaflet.
Also, I don't think I can implement lazy importing anyway, since enabling xarray-leaflet only consists of importing it (it registers a DataArray accessor).
If rioxarray is the culprit, it might be worth opening an issue there? Although I'm not sure they can do a lot either, it's probably just the time to load GDAL in memory.

@yuvipanda
Copy link
Author

sorry i misspoke, i meant https://github.com/xarray-contrib/xarray_leaflet/blob/main/xarray_leaflet/xarray_leaflet.py#L16 and rasterio as one of the culprits, not rioxarray. I think IPython is the other big culprit - moving those to be imported just before use might help?

@davidbrochart
Copy link
Collaborator

Actually there is not need to import LeafMap in __init__. One could e.g. import xarray_leaflet.map instead of import xarray_leaflet. That would be a breaking change though.

@yuvipanda
Copy link
Author

@davidbrochart right, I'm not sure if a breaking change is desired - I don't know how far along the xarray-leaflet project is.

Another option is to split the serverextension into its own module perhaps? It could ship by default with xarray-leaflet, maybe even as part of the same package, but just be a different module?

@davidbrochart
Copy link
Collaborator

Another option is to split the serverextension into its own module perhaps?

Thanks, that's what I did in #80, and I released v0.2.2 with these changes.

@yuvipanda
Copy link
Author

Thank you so much for your quick work on this, @davidbrochart!

yuvipanda added a commit to yuvipanda/pangeo-docker-images that referenced this issue Jan 19, 2023
Brings in xarray-contrib/xarray_leaflet#79,
which speeds up container startup quite a bit!
yuvipanda added a commit to yuvipanda/pangeo-docker-images that referenced this issue Jan 19, 2023
Brings in xarray-contrib/xarray_leaflet#79,
which speeds up container startup quite a bit!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants