Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating a subdirectory for each neuron becomes an issue with larger models #8

Open
albertsgarde opened this issue Nov 1, 2023 · 0 comments

Comments

@albertsgarde
Copy link

albertsgarde commented Nov 1, 2023

At least on linux, the filesystem isn't really built for 10000s of subdirectories.
You run into issues like the ls command taking 10 seconds to run or taking minutes to get the size of the directory.
This is a pretty big problem when running this on large models is already a large undertaking.

I would suggest one of two solutions

  1. A flat file structure where all graph files are in the same directory (maybe one subdirectory for each type of graph file) in order to minimize the number of subdirectories. I'm not actually sure this would fix the problems I mentioned. This should be tested.
  2. Allow grouping the graphs into single files, e.g. tar balls. This could be configured by the user. Graphs could be grouped by model, layer og just by 512 neurons or something.
@albertsgarde albertsgarde changed the title Creating a subdirectory for each neuron becomes and issue with larger models Creating a subdirectory for each neuron becomes an issue with larger models Nov 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant