feat: three phased startup order

[koivunej]

Problem

Initial logical size calculation could still hinder our fast startup efforts in #4397. See #4183. In deployment of 2023-06-06 about a 200 initial logical sizes were calculated on hosts which took the longest to complete initial load (12s).

Summary of changes

Implements the three step/tier initialization ordering described in #4397:

1. load local tenants
2. do initial logical sizes per walreceivers for 10s
3. background tasks

Ordering is controlled by:

• waiting on utils::completion::Barriers on background tasks
• having one attempt for each Timeline to do initial logical size calculation
• pageserver/src/bin/pageserver.rs releasing background jobs after timeout or completion of initial logical size calculation

The timeout is there just to safeguard in case a legitimate non-broken timeline initial logical size calculation goes long. The timeout is configurable, by default 10s, which I think would be fine for production systems. In the test cases I've been looking at, it seems that these steps are completed as fast as possible.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[koivunej]

Only a few test failures :) (44 -> 1 -> hopefully to 0 with f92d0ae)

[github-actions]

1000 tests run: 960 passed, 0 failed, 40 skipped (full report)

---

Flaky tests (2)

Postgres 15

• test_remote_storage_upload_queue_retries[local_fs]: ✅ debug
• test_threshold_based_eviction: ✅ debug

_{The comment gets automatically updated with the latest test results
76d0ef1 at 2023-06-06T16:58:09.039Z :recycle:}

[koivunej]

IMO while the tenant::mgr::TENANTS is in state Uninitialized, we should fail with some HTTP status that indicates come back later, or, not expose the HTTP API at all.

Could do that on a separate PR, but sounds trivial to delay.

[koivunej]

IMO while the tenant::mgr::TENANTS is in state Uninitialized, we should fail with some HTTP status that indicates come back later, or, not expose the HTTP API at all.

Could do that on a separate PR, but sounds trivial to delay.

We already delay, because the init_tenant_mgr is done inside a BACKGROUND_RUNTIME.block_on so it will block after having bound to the http port. After init_tenant_mgr the TENANTS is Open. Later we will start the hyper+routerify to actually accept new connections, so we are all good already.

[koivunej]

The limiters to this is other background tasks (initial compaction), eviction task, there is no semaphore for limiting these. Then on next round metrics collection will have the logical sizes. I don't think there's any problem here.

The problem is that metrics collection is N(=1) tenant at a time right now. So, if there is at least one tenant without an active compute, then this PR will delay background task launch needlessly, until metrics collection does get_current_logical_size() on that tenant's timelines. Due to the N=1, that might take a long time.

Discussed this off-github, metrics collection doesn't wait for anything (except mutexes, http POST), so all timelines which haven't yet had their logical sizes computed will get them queued up practically at the same time.

[problame]

Created follow-up to my comment about tenant::mgr Uninitialized handling: #4433

EDIT: now I read #4399 (comment) so this is not a pressing issue.

[koivunej]

Is the 10s value a random value or based on the reconnect timeout (+ backoff?) of the active computes?

Oops forgot to answer to this. It's just a guess, Harrison-Stetson method. Safekeepers push updates to storage_broker every 1s or so, so we'd have some chances to get these, and then to connect to safekeepers, and then to initiate the init logical size calculations. Hopefully the first seconds doing the tenant loading will allow us to see the first updates on storage_broker, and then the logical sizes will be awaiting on the barrier.

When the 10s runs out, we might not have finished every init size calculation, but at least the tasks are queued up, delaying background tasks "naturally".

[koivunej]

I don't think the cloud-e2e timeout is caused by the changes, because the pageservers are not restarted, and background tasks get to start immediatedly because the pageservers are empty.