Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save intermediate models after each stage #16

Open
onadegibert opened this issue Apr 25, 2023 · 1 comment
Open

Save intermediate models after each stage #16

onadegibert opened this issue Apr 25, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@onadegibert
Copy link

It would be nice to save models after each training stage.

For example, for multilingual training, if you put a higher percentage of data from language X in step Y, it would be nice to have the resulting model of that Y stage to evaluate for that X language.

@onadegibert onadegibert added the enhancement New feature or request label Apr 25, 2023
@jelmervdl
Copy link
Contributor

A sensible way of implementing this would probably be by having the trainer stop at the end of a stage (i.e. feeding it end-of-file), making a copy of the model files, and then restart the trainer with the next stage.

Stopping/restarting is already expected to be safe since the whole resume-training functionality relies on that. And to make a backup of the model files at that point, we can add a little config entry that describes which files are model files.

I'm not completely sure whether fine-tuning is best done in the same OpusTrainer config as the initial model training. I can imagine a scenario where you have a base model, and then want to "fork" training into multiple directions, i.e. finetune that base model with different dataset (or mixes). And then you end up with a sort of dependency graph that makes more sense for something like Makefiles or (sorry) Snakemake. And from those tools you can then of course call OpusTrainer again to wrap the actual training.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants