Joint Consensus

[Hoverbear]

This PR enables Joint Consensus as described by the Raft paper (https://raft.github.io/raft.pdf).

As of this commit it is possible to undergo arbitrary peer membership changes in a safe way.

Notably, this gives us a resilient "Replace Node" functionality, which is able to progress in the situation of a loss of both the old (removed) and new (added) peers go down. This is not possible with our previous one-at-a-time strategy.

Unfortunately, this feature is fairly large in scope. Thankfully it's mostly testing code!

There is some new API surface, notably the Raft::begin_membership_change function.

There is also some moved API surface. The RaftLog::applied_to function has been moved to Raft::commit_apply. The old function, while it still works, carries a deprection warning. In the future we should make it a function available to only Raft::applied_to.

Several APIs have changed to return errors instead of panic or otherwise ignore non-critical errors. In most cases you can just do old_call().ok() to have the same behavior.

Then, in order to facilitate the needs in testing, some additional API was added to the Network harness, which now can dispatch messages and not automatically send responses, allowing time to inspect the state of peers.

Finally, many tests were added to ensure Joint Consensus can work.

[BusyJay]

I don't think disable removing leader can solve the problem at all. Because the leader can be down at any time, in which case, all new peers may not be able to elect a leader successfully if they are all not part of the old configuration.

[Hoverbear]

@BusyJay I thought about this more last night too, and I agree it doesn't solve the problem.

[Hoverbear]

PTAL again pre-merge. Please pay particular attention to documentation.

[zhangjinpeng87]

@BusyJay @hicqu PTAL again.

[hicqu]

LGTM. PTAL @nolouch @BusyJay

[nolouch]

LGTM