-
Notifications
You must be signed in to change notification settings - Fork 495
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[lightning] Add support to continue training #828
Labels
Comments
In progress, co-development with @Schludel. |
TODO: Update total steps of OneCycleLR with new total steps in if-statement |
To decide: Should we continue using the OneCycleLR in case we continue training the network or should we rather use a simple decaying lr scheduler instead that uses the last learning rate from the previous lr scheduler? |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Desired functionality
The training of a model can be continued. The
fit
function can be called a second (arbitrary number) of time(s) to tune the model performance.Current state
test_continue_training
added undertest_utils.py
continue_training
is supported byfit(...)
Required changes
Currently the learning rate scheduler throws a ValueError, indicating that the number of steps is above the originally defined number of steps (which makes sense). The new max_step number likely needs to be adjusted somewhere.
Further, the learning rate scheduler of the model likely needs to be switched when continuing training.
The text was updated successfully, but these errors were encountered: