A more general support for QA and summarization (review and feedback needed)

[ekurtulus]

With this update, I am aiming for generalizing our training framework. I need feedback and review on the structure that I propose. Accordingly, I will make the codebase fully functional and extend it. Additionally, I added more papers to the docs side.

see this issue

I think we need support not only for QA but also summarization. In this PR, I wanted to extend the existing datasets, add support for seq2seq models like T5, PolyLoss and few other modifications. Normally, I was also planning to add support for Sharpness Aware Training, but it requires overriding the Huggingface Trainer's _inner_training_loop function. I can do that but I am sure if it is something we want. If we decide to do so, I can submit another PR.

Furthermore, if the structure of this commit is approved, I can add more tasks, a script for synthetic data generation through data augmentation, and more.

[yk]

you have some merge conflict markers in your files

[sanagno]

Looks good, after the merge! Do you know which models use the SAM optimizer?

[andrewm4894]

i think this means you might need to update from upstream/master

[andrewm4894]

@ekurtulus it might be cleaner if you move your work onto a feature branch in your fork. That way you can keep your main in sync with LAION-AI/Open-Assistant:main and might make it easier for then updating your feature branch from main. I find this approach a bit easier to work with given how often main is changing with so much work going on.

[ekurtulus]

Looks good, after the merge! Do you know which models use the SAM optimizer?

None. It is an optimizer you use in conjunction with other optimizers. Here is the link to the paper. There are papers claiming it to be increasing generalization like this one.

[sanagno]

Looks great, If you can merge the final conflicts and add a default accuracy metric as before would be great!

[yk]

@ekurtulus did you close this on purpose?

[ekurtulus]

@ekurtulus did you close this on purpose?

Yes, I realized that I was not using pre-commit files and not branching properly. While trying to set everything up properly, I had to delete and re-fork the main repository. So, it was closed automatically. I will submit a new PR.

[ekurtulus]

Please see PR 619.