ship log pruning: optionally reduce disk usage of ship log files during operation by periodically trimming logs #327
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spoonincode
changed the title
arhag
requested changes
Jun 8, 2022
* note on truncate() that it leaves the end of the log without a trailer even if in prune mode * vacuum() no longer stomps on the version field * prune() & vacuum() are deliberate in checking their preconditions
* restore validation that the caller of write_entry() filled out the header's size correctly * consider suffix in deciding when to prune to ensure very pessimistic log entries will get pruned * tweak vacuum hole punching range to ensure nothing is missed
arhag
reviewed
Jun 9, 2022
* change vacuum status message to occur every 5 seconds like in blog * fix some goofs in the transition to a config struct
arhag
approved these changes
Jun 14, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a new option,
state-history-log-retain-blocks, which when set will periodically reduce the disk usage of the state history log files to only that of the given number of most recent blocks. This allows operators to limit the disk usage of state history without stopping and restarting nodeos.Pruning is performed by "poking holes" in the log files; in other words making them sparse files. This may have implications if trimmed log files are tarballed/copied/etc with an application that does not recognize sparse files: the files may take up more disk space than expected on their destination if an application is not spare file aware. Logs can be "vacuumed" (converted back in to an unpruned log that doesn't contain holes) by emitting the
state-history-log-retain-blocksoption.The pruning is only periodically performed so in practice a small number of additional blocks (no more than a few megabytes worth of data) will be available beyond the configured number.