Ingesters should handle TSDB corruption on startup

[aknuds1]

Is your feature request related to a problem? Please describe.

It sometimes happens that an ingester fails to start due to detecting TSDB corruption. The typical result of this will be that the ingester in question crashloops, and an engineer has to step in and clean up the corrupted data.

The corruption might for example have occurred because of physical disk failure in the data center.

An example log message from when this happens:

unable to open TSDB for user <redacted>: failed to open TSDB: /data/tsdb/<redacted>: /data/tsdb/<redacted>/chunks_head/<redacted>: invalid magic number 0 

This can also appear in combination with corrupted shipper files, example log:

failed to parse /data/tsdb/<redacted>/thanos.shipper.json as JSON: "": unexpected end of JSON input

Describe the solution you'd like

I would like some automatic cleanup of corrupted TSDB files in ingesters, so it won't be required for engineers to step in and clean up manually.

According to @pracucci dealing with TSDB corruption could be tricky, but corrupted thanos.shipper.json files should be easy to solve.