Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-enable n_jobs in multi-comparison #3747

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

alejoe91
Copy link
Member

@alejoe91 alejoe91 commented Mar 6, 2025

@paul-aparicio I realized that the n_jobs parameter was actually not used.

This PR adds it back and updates the parallelization using the ProcessPoolExecutor. This will run each comparison in parallel.
A simple test shows that the results are >10x faster (on my machine - 8 CPUs).

Can you test this PR? you just need to add the n_jobs=-1 to the compare_multiple_sorters

@alejoe91 alejoe91 added the comparison Related to comparison module label Mar 6, 2025
@alejoe91 alejoe91 requested a review from samuelgarcia March 6, 2025 14:26
@paul-aparicio
Copy link

Absolutely! Thank you so much

Comment on lines +118 to +129
job_list = []
job_names = []
for i in range(len(self.object_list)):
for j in range(i + 1, len(self.object_list)):
if self.name_list is not None:
name_i = self.name_list[i]
name_j = self.name_list[j]
else:
name_i = "object i"
name_j = "object j"
job_list.append((i, j))
job_names.append((name_i, name_j))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactor this

@paul-aparicio
Copy link

Hi, just to make sure I am doing the right thing (sorry for lack of experience), I should use gh pr checkout 3747 and then rerun the analysis with n_jobs=-1 and agreement_method='distance', is that correct?

@alejoe91
Copy link
Member Author

As long as you have spikeinterface installed from sources, yes! You can also try the other agreement method (counts), since this will parallelize over comparisons across sorters

@paul-aparicio
Copy link

Hi, I completed my tests. I did not find much difference, maybe I downloaded the incorrect patch? I tested agreement_method='counts' (took ~7.5 hours to complete both before and after) and agreement_method='count' (took ~7.75 hours to compete before and after the change). I am supposed to patch with #3747? Thanks!

@alejoe91
Copy link
Member Author

alejoe91 commented Mar 14, 2025

How did you install from this PR? My suggestion is:

pip uninstall -y spikeinterface
git clone https://github.com/alejoe91/spikeinterface.git
git checkout multicomp-parallel
cd spikeinterface
pip install -e .
cd ..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comparison Related to comparison module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants