Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[lightning] Add support to continue training #828

Open
karl-richter opened this issue Oct 14, 2022 · 3 comments
Open

[lightning] Add support to continue training #828

karl-richter opened this issue Oct 14, 2022 · 3 comments
Assignees
Labels
priority:P2 Medium priority task Tasks (not an epic, not a bug)

Comments

@karl-richter
Copy link
Collaborator

Note: This feature depends on the introduction of Lightning as model wrapper.

Desired functionality
The training of a model can be continued. The fit function can be called a second (arbitrary number) of time(s) to tune the model performance.

Current state

  • Test test_continue_training added under test_utils.py
  • Parameter continue_training is supported by fit(...)

Required changes
Currently the learning rate scheduler throws a ValueError, indicating that the number of steps is above the originally defined number of steps (which makes sense). The new max_step number likely needs to be adjusted somewhere.
Further, the learning rate scheduler of the model likely needs to be switched when continuing training.

@karl-richter
Copy link
Collaborator Author

In progress, co-development with @Schludel.

@Schludel
Copy link

Schludel commented Dec 6, 2022

TODO: Update total steps of OneCycleLR with new total steps in if-statement

@karl-richter
Copy link
Collaborator Author

To decide: Should we continue using the OneCycleLR in case we continue training the network or should we rather use a simple decaying lr scheduler instead that uses the last learning rate from the previous lr scheduler?
Todo: Run experiments to explore

@ourownstory ourownstory added priority:P2 Medium priority and removed enhancement labels Feb 17, 2023
@noxan noxan added the task Tasks (not an epic, not a bug) label Feb 24, 2023
@ourownstory ourownstory removed this from the Release 0.6.0 milestone Apr 7, 2023
@ourownstory ourownstory added this to V1.0 Apr 7, 2023
@ourownstory ourownstory moved this to New Features in V1.0 Apr 7, 2023
@ourownstory ourownstory removed this from V1.0 Apr 17, 2023
@ourownstory ourownstory added this to V2.0 Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority:P2 Medium priority task Tasks (not an epic, not a bug)
Projects
Status: No status
Development

No branches or pull requests

4 participants