Restart based test for divergence

[mhofman]

What is the Problem Being Solved?

With the addition of #4133 and #4256, we now have an integration test that can detect simple causes of divergence (closing #3444).

However in the past we've encountered other triggers for divergence between nodes, in particular when a node restarts, e.g.:

• Divergence between nodes after restart during IBC channel open #4159
• mailbox device nondeterminism after kernel restart #3471
• consensus failure due to comms get(initialized) check #3726

Description of the Design

A testnet with at least 4 nodes, with loadgen applied, where one node at a time gets randomly restarted. For best results, the loadgen should use a random RPC node which will get restarted.

Security Considerations

None

Test Plan

Probably part of a standalone nightly test.

[ivanlei]

From @mhofman in slack --

Yeah the issues about hunting down divergences now have a better story given the improvements I've made to replay tools. I think we no longer need to use the loadgen or instagoric as the means to detect these. We could still use them, but mostly to generate transcripts that could be used as input to the replay tools.