Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use multithreaded load / save per entry #51

Merged
merged 6 commits into from
Feb 20, 2020
Merged

Conversation

haampie
Copy link
Contributor

@haampie haampie commented Feb 20, 2020

Loading 10GB of data on a laptop with 4 cores goes from

julia> example()
 43.730908 seconds (11.57 k allocations: 9.866 GiB, 2.12% gc time)

to

julia> example()
 14.883994 seconds (11.60 k allocations: 9.866 GiB, 1.79% gc time)

Saving is probably even more interesting, that took 6mins, I can check if it goes to 1.5 minutes with this PR

Copy link
Member

@oxinabox oxinabox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. still some issues.
But you got the locks in the right places which is important

Can we do this in a way that maintains compatibility with Julia 1.0?
Does Compat.jl have support for making @spawn into @async or some such?

@haampie
Copy link
Contributor Author

haampie commented Feb 20, 2020

I'll take a look at Julia 1.0 compat a bit later

@codecov
Copy link

codecov bot commented Feb 20, 2020

Codecov Report

Merging #51 into master will increase coverage by 0.15%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #51      +/-   ##
==========================================
+ Coverage   96.00%   96.15%   +0.15%     
==========================================
  Files           6        6              
  Lines         150      156       +6     
==========================================
+ Hits          144      150       +6     
  Misses          6        6              

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9531c5f...b6cf74c. Read the comment docs.

Copy link
Member

@oxinabox oxinabox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
can you bump the minor version so we can tag a release?

@oxinabox oxinabox merged commit 249fb66 into invenia:master Feb 20, 2020
@oxinabox
Copy link
Member

Thanks!

@haampie haampie mentioned this pull request Feb 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants