Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve options for running jobs in parallel #57

Open
2 tasks
nicolas-kuechler opened this issue Aug 30, 2022 · 0 comments
Open
2 tasks

Improve options for running jobs in parallel #57

nicolas-kuechler opened this issue Aug 30, 2022 · 0 comments

Comments

@nicolas-kuechler
Copy link
Owner

Currently, experiments run in parallel but jobs within an experiment run sequentially.

This can lead to a lot of code duplication in suite designs to leverage parallelization.
An experiment is copied and something that is a $FACTOR$ is treated as a constant in two separate experiments.

(note, for single instance experiments running on Euler, this limitation is less of a problem because jobs can be executed in parallel on Euler)

We could investigate whether it is possible to define on an experiment level: n_env or something similar that defines how many times the environment defined in host_types is duplicated.

Options:

  • do a preprocessing step on the design to split the runs into separate experiments and then do not change anything in the execution (would have to ensure that the run_id is preserved, otherwise the results will overlap
  • have some functionality in the execution that checks that we only schedule runs with run_id % n_env == x or something. I think such a choice may be better suited because splitting experiments and preserving run_id could be complex.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant