Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming add remfile #1761

Merged
merged 25 commits into from
Jan 13, 2024
Merged
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
6739cf3
refactor streaming tutorial to expose Python code
bendichter Aug 17, 2023
12ad86f
add fsspec to docs requirements
bendichter Aug 17, 2023
64e05d6
move fsspec to requirements-dev.txt
bendichter Aug 17, 2023
50ffa58
move fsspec to environment-ros3.yml
bendichter Aug 17, 2023
793526d
flake8
bendichter Aug 17, 2023
d824431
add required dependencies
bendichter Aug 17, 2023
4cad986
add remfile to streaming.py tutorial
bendichter Aug 17, 2023
5045815
adjust language
bendichter Aug 17, 2023
614375a
Merge branch 'dev' into streaming_add_remfile2
bendichter Aug 17, 2023
bab2f9e
move remfile to pip install
bendichter Aug 18, 2023
d5d0439
Merge branch 'dev' into streaming_add_remfile2
rly Oct 22, 2023
65c34aa
Update CHANGELOG.md
rly Oct 22, 2023
a942988
Merge branch 'dev' into streaming_add_remfile2
rly Nov 27, 2023
31c5f5f
* change tutorial to use correct h5py.File object
bendichter Nov 27, 2023
3a03c49
update CHANGELOG.md
bendichter Nov 27, 2023
77331a2
Update src/pynwb/__init__.py
bendichter Nov 27, 2023
ea72283
Update environment-ros3.yml
bendichter Nov 27, 2023
59fe561
Update docs/gallery/advanced_io/streaming.py
bendichter Nov 27, 2023
d34a526
Update docs/gallery/advanced_io/streaming.py
bendichter Nov 27, 2023
2a70ca9
Merge branch 'dev' into streaming_add_remfile2
bendichter Nov 27, 2023
b3591ed
Merge branch 'dev' into streaming_add_remfile2
bendichter Nov 29, 2023
bae8d30
Merge branch 'dev' into streaming_add_remfile2
rly Dec 2, 2023
7974499
Merge branch 'dev' into streaming_add_remfile2
rly Jan 13, 2024
c16736b
Update streaming.py
rly Jan 13, 2024
226f41d
Update streaming.py
rly Jan 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'dev' into streaming_add_remfile2
# Conflicts:
#	docs/gallery/advanced_io/streaming.py
#	environment-ros3.yml
bendichter committed Aug 17, 2023
commit 614375ac4be5d11a750275b016ac5512a7c3fbcd
21 changes: 11 additions & 10 deletions docs/gallery/advanced_io/streaming.py
Original file line number Diff line number Diff line change
@@ -140,17 +140,18 @@
nwbfile = io.read()
print(nwbfile.acquisition["lick_times"].time_series["lick_left_times"].data[:])

################################################
##################################################
# Which streaming method to choose?
# ---------------------------------
# When choosing a streaming method, consider these comparisons:
#
# 1. fsspec and remfile are easier to install than ros3, requiring a simple pip-install that works across platforms.
# 2. fsspec and remfile support caching, which will dramatically speed up repeated requests for the same region of data.
# 3. fsspec and remfile automatically retries when s3 fails to return due to brief connection errors.
# 4. fsspec works with other storage backends. remfile and ros3 only work with s3.
# 5. fsspec and remfile work with other types of files. ros3 is a driver designed specifically for HDF5 files.
# 6. In our hands, remfile is fastest, then fsspec, then ros3, though this could vary by file and access pattern.
# 7. fsspec and ros3 are well-supported and popular community tool. remfile is much newer and not as well adopted.
#
# fsspec has many advantages over ros3:
#
# 1. fsspec is easier to install
# 2. fsspec supports caching, which will dramatically speed up repeated requests for the
# same region of data
# 3. fsspec automatically retries when s3 fails to return.
# 4. fsspec works with other storage backends and
# 5. fsspec works with other types of files.
# 6. In our hands, fsspec is faster out-of-the-box.
#
# For these reasons, we would recommend use fsspec for most Python users.
You are viewing a condensed version of this merge commit. You can view the full changes here.