Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow the option to store blobs to filesystem as they're being removed from database #8513

Open
3 of 5 tasks
rolfyone opened this issue Aug 19, 2024 · 2 comments
Open
3 of 5 tasks
Labels
enhancement 🕵️‍♀️ New feature or request

Comments

@rolfyone
Copy link
Contributor

rolfyone commented Aug 19, 2024

Blobs are currently stored for the mandatory time in database and then deleted.

Rather than simply deleting them, it'd be helpful to be able to write the blobs to filesystem.

I could imagine keeping this relatively simple and just specifying something like --data-storage-store-old-blobs-path=/data and we'd then start writing all blobs to the filesystem before they get archived.

Theres a tool already in use https://github.com/base-org/blob-archiver that basically dumps the blobs api output per block, we could do similar re. file naming etc potentially if its not too crazy.. Remembering there'll be 7200 slots per day so we probably want output in some kind of structure.

Work is split into following tasks:

@david-ry4n
Copy link
Contributor

david-ry4n commented Sep 24, 2024

Moved plan to main description.

@tbenr
Copy link
Contributor

tbenr commented Sep 24, 2024

The plan makes sense to me! bare in mind that you used fixes xxx in the first PR description, so once you merge that it will automatically close this one, which instead seems like an "epic" tracking several steps.
I'd move the checkboxes in the main issue description so we can keep it open and track the status.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement 🕵️‍♀️ New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants