CI failure in test_restart_compute test

[hlinnaka]

This happened as part of PR #405, but seems unrelated to the code changes:

https://app.circleci.com/pipelines/github/zenithdb/zenith/1098/workflows/a752b1e3-2771-405d-92d4-6231d002357d/jobs/5938

=================================== FAILURES ===================================
__________________________ test_restart_compute[True] __________________________
batch_others/test_restart_compute.py:57: in test_restart_compute
    with closing(pg.connect()) as conn:
fixtures/zenith_fixtures.py:106: in connect
    conn = psycopg2.connect(self.connstr(**kwargs))
../../.local/share/virtualenvs/test_runner-sNYEMCQ_/lib/python3.8/site-packages/psycopg2/__init__.py:122: in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
E   psycopg2.OperationalError: server closed the connection unexpectedly
E   	This probably means the server terminated abnormally
E   	before or while processing the request.

[hlinnaka]

@knizhnik, is this the same or related as issue #395 ?

[kelvich]

Looks like that is caused by this morning fixes to #395. @arssher looking investigating problem itself and meanwhile we decided to rollback that commits and commit fixed version later.

[knizhnik]

@knizhnik, is this the same or related as issue #395 ?

The fixes I have made this morning seems to be very small an obvious:

1. Align prev LSN on 8
2. Update walkeeper state not only on the disk, but also in memory
   I can not reproduce the problem locally. May be the problem is caused by not updated version of postgres submodule...

[arssher]

Failure happens due to could not receive data from WAL stream error in walproposer during recovery which forces it to restart over and over again while trying to fetch WAL from the most advanced safekeeper (so WAL doesn't get to pageserver => get_page_at_lsn hangs => auth fails after timeout). I suspect this is caused by incorrect COPY protocol termination (there should be a dance like CopyDone followed by one or two CommandCompletes which we don't do) by replication.rs, but not exactly sure.

Rolling back yesterday 0ee2e16 was quite a stupid action from me; that commit doesn't have to do anything with this (#409) problem, it is a floating bug and it is just a coincidence that test passed on my laptop and CI after revert. Anyway, 1) 0ee2e16 is itself incorrect 2) the whole thing is rewritten in #315. Today I can't reproduce could not receive data from WAL stream as well. I think it is better now to get back to the plan of merging #366, then #315 and then reconsidering this, otherwise we are kind of searching for new bugs among known ones.

[kelvich]

I think it is better now to get back to the plan of merging #366, then #315 and then reconsidering this, otherwise we are kind of searching for new bugs among known ones.

+1

[kelvich]

Main ticket for related activities is #395, closing as dup