You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At least on linux, the filesystem isn't really built for 10000s of subdirectories.
You run into issues like the ls command taking 10 seconds to run or taking minutes to get the size of the directory.
This is a pretty big problem when running this on large models is already a large undertaking.
I would suggest one of two solutions
A flat file structure where all graph files are in the same directory (maybe one subdirectory for each type of graph file) in order to minimize the number of subdirectories. I'm not actually sure this would fix the problems I mentioned. This should be tested.
Allow grouping the graphs into single files, e.g. tar balls. This could be configured by the user. Graphs could be grouped by model, layer og just by 512 neurons or something.
The text was updated successfully, but these errors were encountered:
albertsgarde
changed the title
Creating a subdirectory for each neuron becomes and issue with larger models
Creating a subdirectory for each neuron becomes an issue with larger models
Nov 1, 2023
At least on linux, the filesystem isn't really built for 10000s of subdirectories.
You run into issues like the
ls
command taking 10 seconds to run or taking minutes to get the size of the directory.This is a pretty big problem when running this on large models is already a large undertaking.
I would suggest one of two solutions
The text was updated successfully, but these errors were encountered: