Save intermediate models after each stage

[onadegibert]

It would be nice to save models after each training stage.

For example, for multilingual training, if you put a higher percentage of data from language X in step Y, it would be nice to have the resulting model of that Y stage to evaluate for that X language.

[jelmervdl]

A sensible way of implementing this would probably be by having the trainer stop at the end of a stage (i.e. feeding it end-of-file), making a copy of the model files, and then restart the trainer with the next stage.

Stopping/restarting is already expected to be safe since the whole resume-training functionality relies on that. And to make a backup of the model files at that point, we can add a little config entry that describes which files are model files.

I'm not completely sure whether fine-tuning is best done in the same OpusTrainer config as the initial model training. I can imagine a scenario where you have a base model, and then want to "fork" training into multiple directions, i.e. finetune that base model with different dataset (or mixes). And then you end up with a sort of dependency graph that makes more sense for something like Makefiles or (sorry) Snakemake. And from those tools you can then of course call OpusTrainer again to wrap the actual training.