Abstract
State machine replication protocols, like MultiPaxos and Raft, are at the heart of numerous distributed systems.
To tolerate machine failures, these protocols must replace failed machines with new machines, a process known as reconﬁguration.
Reconﬁguration has become increasingly important over time as the need for frequent reconﬁguration has grown.
Despite this, reconﬁguration has largely been neglected in the literature.
In this paper, we present Matchmaker Paxos and Matchmaker MultiPaxos, a reconﬁgurable consensus and state machine replication protocol respectively.
Our protocols can perform a reconﬁguration with little to no impact on the latency or throughput of command processing; they can perform a reconﬁguration in a few milliseconds; and they present a framework that can be generalized to other replication protocols in a way that previous reconﬁguration techniques can not.
We provide proofs of correctness for the protocols and optimizations, and present empirical results from an open source implementation showing that throughput and latency do not change signiﬁcantly during a reconﬁguration.
Introduction
Many distributed systems [4,6,7,14,16] rely on a state machine replication protocol, like MultiPaxos [17] or Raft [34], to keep multiple replicas of their data in sync.
Over time, machines fail, and if too many machines in a state machine replication protocol fail, the protocol grinds to a halt.
Thus, state machine replication protocols have to replace failed machines with new machines as the protocol runs, a process known as reconﬁguration.
Reconﬁguration is an essential component of state machine replication.
It is not an optimization or an afterthought.
Without a reconﬁguration protocol in place, a state machine replication protocol will inevitably stop working; it’s just a matter of when.
Despite this, reconﬁguration has largely been neglected by current academic literature.
Researchers have invented dozens of state machine replication protocols, yet many papers either discuss reconﬁguration brieﬂy with no evaluation [31, 36–38], propose theoretically safe but inefﬁcient reconﬁguration protocols [18, 26], or do not discuss reconﬁguration at all [2,3,19,28,29].
Ignoring reconﬁguration has never been ideal, but we have largely been able to get away with it.
Historically, state machine replication protocols were deployed on a ﬁxed set of machines, and reconﬁguration was used only to replace failed machines with new machines – an infrequent occurrence.
This made it easy to leave reconﬁguration out of sight, out of mind.
Recently however, systems have become increasingly elastic, and the need for frequent reconﬁguration has grown.
These elastic systems don’t just perform reconﬁgurations reactively when machines fail; they reconﬁgure proactively .
For example, cloud databases can proactively request more resources to handle workload spikes, and orchestration tools like Kubernetes [15] are making it easier to build these types of elastic systems.
Similarly, in environments with short-lived cloud instances—as with serverless computing and spot instances— and in mobile edge and Internet of Things settings, protocols must adapt to a changing set of machines much more frequently.
This frequent need for reconﬁguration makes it hard to ignore reconﬁguration any longer.
In this paper, we present a reconﬁgurable consensus protocol and a reconﬁgurable state machine replication protocol:
Matchmaker Paxos and Matchmaker MultiPaxos.
In a nutshell, our protocols work by leveraging two key design ideas.
• The ﬁrst is to decouple reconﬁguration from the standard processing path.
Many replication protocols [23, 26,31,34] have machines that are responsible for both processing commands and for orchestrating reconﬁgurations.
By contrast, Matchmaker Paxos introduces a set of distinguished matchmaker machines that are solely responsible for managing reconﬁgurations and operate off of the critical path.
These matchmakers act as a source of truth; they always know the current conﬁguration.
• The second design point is to reconﬁgure across rounds, a technique known as vertical reconﬁguration [22].
With vertical reconﬁguration, every round of consensus can execute using a different conﬁguration.
At the beginning of every round, the
Paxos
leader queries the matchmakers to discover the older conﬁgurations that were used in previous rounds, and it simultaneously sends the matchmakers the conﬁguration it intends to use in the current round.
In this way, the matchmakers act as a registry for conﬁgurations.
Leaders simultaneously query the past and update the present.
This matchmaking phase requires a single round trip of communication and happens rarely.
We also introduce a number of novel protocol optimizations to perform the matchmaking completely off the critical path to avoid
degrading performance.
Moreover, the protocol employs a garbage collection protocol to delete old conﬁgurations stored on the matchmakers.
Our protocols have the following desirable properties.
Little to No Performance Degradation.
Matchmaker
MultiPaxos can perform a reconﬁguration without significantly degrading the throughput or latency of processing client commands.
For example, we show that reconﬁguration has less than a 4% effect on the median of throughput and latency measurements (Section 7).
Note that Matchmaker
MultiPaxos is not the ﬁrst protocol to achieve this [27].
Quick Reconﬁguration.
Matchmaker MultiPaxos can perform a reconﬁguration quickly.
Reconﬁguring to a new set of machines takes one round trip of communication in the normal case (Section 4).
Empirically, this requires only a few milliseconds within a single data center (Section 7).
It takes slightly longer to shut down the old machines, but empirically this takes only ﬁve milliseconds within a data center (Section 7).
Generality Replication protocols based on classical MultiPaxos assume a totally ordered log of chosen commands and reconﬁgure across log entries, known as horizontal reconﬁguration .
However, many state machine replication protocols do not replicate a log [2,18,31,38,39,41].
These protocols cannot use horizontal reconﬁguration.
However, while none of these protocols have logs, they all have rounds and can implement vertical reconﬁguration.
This allows Matchmaker
Paxos and Matchmaker MultiPaxos to serve as a foundation on top of which reconﬁguration protocols can be built for these other non-log based protocols.
Theoretical Insights.
Matchmaker Paxos generalizes Vertical Paxos [22], it is the ﬁrst protocol to achieve the theoretical lower bound on Fast Paxos [19] quorum sizes, and it corrects errors in DPaxos [33] (Section 6).
Proven Safe.
We describe Matchmaker Paxos and Matchmaker MultiPaxos precisely and prove that both are safe (Sections 3, 4, 5, A, B).
Unfortunately, this is not often done for all reconﬁguration protocols [30,36–38].
Background 2. System Model
Throughout the paper, we assume an asynchronous network model in which messages can be arbitrarily dropped, delayed, and reordered.
We assume machines can fail by crashing but do not act maliciously.
We assume that machines operate at arbitrary speeds, and we do not assume clock synchronization.
We assume a discovery service that nodes can use to ﬁnd each other, but do not require that this service be strongly consistent.
A node can safely communicate with outdated nodes.
A system like DNS would sufﬁce.
Every protocol discussed in this paper assumes (for liveness) that at most f machines will fail for some conﬁgurable f .
All the protocols discussed in this paper are safe, but due to the FLP impossibility result [10], none of the protocols are guaranteed to be fully live (unless the network is synchronous).
2. Paxos
A consensus protocol is a protocol that selects a single value from a set of proposed values.
Paxos [17, 20] is one of the oldest and most popular consensus protocols.
A Paxos deployment that tolerates f faults consists of an arbitrary number of clients, at least f + 1 nodes called proposers , and 2 f +nodes called acceptors , as illustrated in Figure 1.
To reach consensus on a value, an execution of Paxos is divided into a number of rounds, each round having two phases: Phase 1 and Phase 2.
Every round is orchestrated by a single predetermined proposer.
The set of rounds can be any unbounded, totally ordered set.
It is common to let the set of rounds be the set of lexicographically ordered integer pairs ( r , id ) where r is an integer and id is a unique proposer id, where a proposer is responsible for executing every round that contains its id. c
[Equation]
p
[Equation]
a
[Equation]
aa
Clients f +
Proposers 2 f +
Acceptors 22 (a) Phasecccppaaa
Clients f +
Proposers 2 f +
Acceptors 44  (b) Phase
When a proposer executes a round, say round
i , it attempts to get some value x chosen in that round.
Paxos is a consensus protocol, so it must only choose a single value.
Thus, Paxos must ensure that if a value x is chosen in round i , then no other value besides x can ever be chosen in any round less than i .
This is the purpose of Paxos’ two phases.
In Phaseof round i , the proposer contacts the acceptors to (a) learn of any value that may have already been chosen in any round less than i and (b) prevent any new values from being chosen in any round less than i .
In Phase 2, the proposer proposes a value to the acceptors, and the acceptors vote on whether or not to choose it.
In Phase 2, the proposer will only propose a value x if it ensured through Phase 1 that no other value has been or will be chosen in a previous round.
More concretely, Paxos executes as follows, as illustrated in Figure 1.
When a client wants to propose a value x , it sends x to a proposer p .
Upon receiving x , p begins executing one round of Paxos, say round i .
First, it executes Phase 1.
It sends
P HASE 1A (cid:104) i (cid:105) messages to the acceptors.
An acceptor ignores a P HASE 1A (cid:104) i (cid:105) message if it has already received a message in a larger round.
Otherwise, it replies with a P HASE 1B (cid:104) i , vr , vv (cid:105) message containing the largest round vr in which the acceptor voted and the value it voted for, vv .
If the acceptor hasn’t voted yet, then vr = − 1 and vv = null .
When the proposer receives P HASE 1 B messages from a majority of the acceptors,
Phase 1 ends and Phase 2 begins.
At the start of Phase 2, the proposer uses the P HASE 1B messages that it received in Phase 1 to select a value x such that no value other than x has been or will be chosen in any round less than i .
Speciﬁcally x is the vote value associated with the largest received vote round, or any value if no acceptor had voted (see [20] for details).
Then, the proposer sends P HASE 2A (cid:104) i , x (cid:105) messages to the acceptors.
An acceptor ignores a P HASE 2A (cid:104) i , x (cid:105) message if it has already received a message in a larger round.
Otherwise, it votes for x and sends back a P HASE 2B (cid:104) i (cid:105) message to the proposer.
If a majority of acceptors vote for the value, then the value is chosen, and the proposer informs the client.
Proposer and acceptor pseudocode (with modiﬁcations for Matchmaker Paxos) are shown in Algorithm 3 and Algorithm 2.
2. Flexible Paxos
Paxos deploys a set of 2 f + 1 acceptors, and proposers communicate with at least a majority of the acceptors in Phaseand in Phase 2.
Flexible Paxos [13] is a Paxos variant that eschews the notion of a majority for that of an arbitrary quorum .
Speciﬁcally, Flexible Paxos introduces the notion of a conﬁguration C =
( A ; P 1; P 2 ) .
A is a set of acceptors.
Pand P 2 are sets of quorums , where each quorum is a subset of A .
A conﬁguration satisﬁes the property that every quorum in P 1 (known as a Phase 1 quorum ) intersects every quorum in P 2 (known as a Phase 2 quorum ).
For a conﬁguration to tolerate f failures, there must exist some Phase 1 quorum and some Phase 2 quorum of non-failed machines despite an arbitrary set of f failures.
Flexible Paxos is identical to Paxos with the exception that proposers now communicate with an arbitrary Phasequorum in Phase 1 and an arbitrary Phase 2 quorum in Phase 2.
In the remainder of this paper, we assume that all protocols operate using quorums from an arbitrary conﬁguration rather than majorities from a ﬁxed set of 2 f + 1 acceptors.
Matchmaker Paxos
We now present Matchmaker Paxos.
To ease understanding, we ﬁrst describe a simpliﬁed version of Matchmaker Paxos that is easy to understand but is also naively inefﬁcient.
We then upgrade the protocol to the complete, efﬁcient version by way of a number of optimizations.
3. Overview and Intuition
Matchmaker Paxos is largely identical to Paxos.
Like Paxos,
a
Matchmaker Paxos deployment includes an arbitrary number of clients, a set of at least f + 1 proposers, and some set of acceptors, as illustrated in Figure 2.
Paxos assumes that a single, ﬁxed conﬁguration of acceptors is used for every round.
The big difference between Paxos and Matchmaker
Paxos is that Matchmaker Paxos allows every round to have a different conﬁguration of acceptors.
Round 0 may use some conﬁguration C 0 , while round 1 may use some completely different conﬁguration C 1 .
This idea was ﬁrst introduced by
Vertical Paxos [22].
cccppmmmaaabbb
Clients f +
Proposers 2 f +
Matchmakers
C 0 Acceptors
C 1 Acceptors  2    6 6 72/
Matchmaking Phase 4/
Phase6/
[Figure]
Recall from Section 2 that a Paxos proposer in round i executes Phase 1 in order to (1) learn of any value that may have been chosen in a round less than i and (2) prevent any new values from being chosen in any round less than i .
To do so, the proposer contacts the ﬁxed set of acceptors.
A
Matchmaker Paxos proposer must also execute Phase 1 to establish that these two properties hold.
The difference is that there is no longer a single ﬁxed conﬁguration of acceptors to contact.
Instead, a Matchmaker Paxos proposer has to contact all of the conﬁgurations used in rounds less than i .
However, every round can use a different conﬁguration of acceptors, so how does the proposer of round i know which acceptors to contact in Phase 1?
To resolve this question, a
Matchmaker Paxos deployment also includes a set of 2 f +matchmakers .
The protocol executes as follows, as illustrated in Figure 2.
(1) A client proposes a value x by sending it to a proposer ( p 1 in Figure 2).
(2,3) When a proposer receives a value x , it begins executing the protocol in some round i .
It selects a conﬁguration C i and sends C i to the matchmakers.
The matchmakers reply with the conﬁgurations used in previous rounds.
We call this the Matchmaking phase .
In Figure 2, the proposer executes in round 1 and selects conﬁguration Cconsisting of acceptors b 1 , b 2 , and b 3 .
The matchmakers reply with the conﬁguration C 0 consisting of acceptors a 1 , a 2 , and a 3 .
The proposer then executes Phase 1 of Paxos with the prior conﬁgurations that it received during the Matchmaking Phase.
In Figure 2, the proposer executes Phase 1 with conﬁguration C 0 .
The proposer then executes Phase 2 with the conﬁguration C i to get the value x chosen.
In Figure 2, the proposer executes Phase 2 with conﬁguration C 1 .
(8) Finally, the proposer informs the client that x was chosen.
At ﬁrst, the extra round trip of communication with the matchmakers and the large number of conﬁgurations in Phase 1 make Matchmaker Paxos look slow.
This is for ease of explanation.
Later, we will eliminate these costs (Section 3.
– Section 3.6).
3. Details
Every matchmaker maintains a log L of conﬁgurations indexed by round.
That is, L [ i ] stores the conﬁguration of round i .
When a proposer receives a request x from a client and begins executing round i , it ﬁrst selects a conﬁguration
C i to use in round i .
It then sends a M ATCH A (cid:104) i , C i (cid:105) message to all of the matchmakers.
When a matchmaker receives a M ATCH A (cid:104) i , C i (cid:105) message, it checks to see if it had previously received a M ATCH A (cid:104) j , C j (cid:105) message for some round j ≥ i .
If so, the matchmaker ignores the M ATCH A (cid:104) i , C i (cid:105) message.
Otherwise, it inserts C i in log entry i and computes the set H i of previous conﬁgurations in the log: H i = { ( j , C j ) | j < i , C j ∈ L } .
It then replies to the proposer with a M ATCH B (cid:104) i , H i (cid:105) message.
Matchmaker pseudocode is given in Algorithm 1.
An example execution of a matchmaker is illustrated in Figure 3.
State: a log L indexed by round, initially empty 1: upon receiving M ATCH
A (cid:104) i , C i (cid:105) from proposer p do 2: if ∃ a conﬁguration C j in round j ≥ i in L then 3: ignore the M ATCH A (cid:104) i , C i (cid:105) message 4: else 5:
H i ← { ( j , C j ) | C j ∈ L } 6:
L [ i ] ← C i 7: send M ATCH B (cid:104) i , H i (cid:105) to p
When the proposer in round i receives M ATCH B (cid:104) i ,
H 1 i (cid:105) , ... , M ATCH B (cid:104) i , H f + 1 i (cid:105) from f + 1 matchmakers, it computes H i = ∪ f + 1 j = 1 H ji .
For example, with f = 1 and i = 2, if the proposer in round 2 receives M ATCH B (cid:104) 2 , { ( 0 , C 0 ) }(cid:105) and    (a)
C 0   (b)
C
C 2 (c)
C
C
C 3(d) in any round less than 0.
(c)
The matchmaker then receives M ATCH A (cid:104) 2 , C 2 (cid:105) .
It inserts C 2 in log entry 2 and returns M ATCH B (cid:104) 2 , { ( 0 , C 0 ) }(cid:105) .
(d) It then receives M ATCH A (cid:104) 3 , C 3 (cid:105) , inserts C 3 in log entry 3, and returns
M ATCH B (cid:104) 3 , { ( 0 , C 0 ) , ( 2 , C 2 ) }(cid:105) .
At this point, if the matchmaker were to receive M ATCH A (cid:104) 1 , C 1 (cid:105) , it would ignore it.
Algorithm 2 Acceptor Pseudocode
State: the largest seen round r , initially −
State: the largest round vr voted in, initially −
State: the value vv voted for in round vr , initially null 1: upon receiving P HASE 1A (cid:104) i (cid:105) from p with i > r do 2: r ← i 3: send P HASE 1B (cid:104) i , vr , vv (cid:105) to p 4: upon receiving P HASE 2A (cid:104) i , x (cid:105) from p with i ≥ r do 5: r , vr , vv ← i , i , x 6: send P HASE 2B (cid:104) i (cid:105) to p
M ATCH B (cid:104) 2 , { ( 1 , C 1 ) }(cid:105) , it computes H 2 = { ( 0 , C 0 ) , ( 1 , C 1 ) } .
Note that every round is statically assigned to a single proposer and that a proposer selects a single conﬁguration for a round, so if two matchmakers return conﬁgurations for the same round, they are guaranteed to be the same.
The proposer then ends the Matchmaking phase and begins
Phase 1.
It sends P HASE 1A messages to every acceptor in every conﬁguration in H i and waits to receive P HASE 1B messages from a Phase 1 quorum from every conﬁguration.
Using the previous example, the proposer sends P HASE 1A messages to every acceptor in C 0 and C 1 and waits for P HASE 1B messages from a Phase 1 quorum of C 0 and a Phase 1 quorum of C 1 .
The proposer then runs Phase 2 with C i .
Acceptor and proposer pseudocode are shown in Algorithm 2 and Algorithm 3 respectively.
To keep things simple, we assume that round numbers are integers, but generalizing to an arbitrary totally ordered set is straightforward.
A
Matchmaker Paxos acceptor is identical to a Paxos acceptor.
A Matchmaker Paxos proposer is nearly identical to a Flexible
Paxos proposer with the exception of the Matchmaking phase and the conﬁgurations used in Phase 1 and Phase 2.
For clarity of exposition, we omit straightforward details surrounding re-sending dropped messages and nacking ignored messages.
State: a value x , initially null
State: a round i , initially −
State: the conﬁguration C
i for round
i , initially null
State: the prior conﬁgurations H i for round
i , initially null 1: upon receiving value y from a client do 2: i ← next largest round owned by this proposer 3: x ← y 4:
C i ← an arbitrary conﬁguration 5: send M ATCH A (cid:104) i , C i (cid:105) to all of the matchmakers 6: upon receiving M ATCH B (cid:104) i , H 1 i (cid:105) ,..., M ATCH B (cid:104) i , H f + 1 i (cid:105) from f + 1 matchmakers do 7:
H i ← (cid:83) f
+ 1 j = 1 H ji 8: send P HASE 1A (cid:104) i (cid:105) to every acceptor in H i 9: upon receiving P HASE 1B (cid:104) i , − , −(cid:105) from a Phase 1 quorum from every conﬁguration in H i do 10: k ← the largest vr in any P HASE 1B (cid:104) i , vr , vv (cid:105) 11: if k (cid:54) = − 1 then 12: x ← the corresponding vv in round k 13: send P HASE 2A (cid:104) i , x (cid:105) to every acceptor in C i 14: upon receiving P HASE 2B (cid:104) i (cid:105) from a Phase 2 quorum do 15: x is chosen, inform the client
3. Proof of Safety
We now prove that Matchmaker Paxos is safe; i.e. every execution of Matchmaker Paxos chooses at most one value.
Proof.
Our proof is based on the Paxos safety proof in [19].
We prove, for every round i , the statement P ( i ) : “if a proposer proposes a value v in round i (i.e. sends a P HASE 2A message for value v in round i ), then no value other than v has been or will be chosen in any round less than i .”
At most one value is ever proposed in a given round, so at most one value is ever chosen in a given round.
Thus, P ( i ) sufﬁces to prove that
Matchmaker Paxos is safe for the following reason.
Assume for contradiction that Matchmaker Paxos chooses distinct values x and y in rounds j
and i with j < i .
Some proposer must have proposed y in round i , so P ( i ) ensures us that no value other than y could have been chosen in round j .
But, x was chosen, a contradiction.
We prove P ( i ) by strong induction on i .
P ( 0 ) is vacuous because there are no rounds less than 0.
For the general case P ( i ) , we assume P ( 0 ) ,..., P ( i − 1 ) .
We perform a case analysis on the proposer’s pseudocode (Algorithm 3).
Either k is − 1 or it is not (line 11).
First, assume it is not.
In this case, the proposer proposes x , the value proposed in round k (line 12).
We perform a case analysis on round j to show that no value other than x has been or will be chosen in any round j < i .
That is, we show P ( i ) .
Case 1: j > k .
We show that no value has been or will be chosen in round j .
Recall that at the end of the Matchmaking phase, the proposer computed the set H i of prior conﬁgurations using responses from a set M i of f + 1 matchmakers.
Either H i contains a conﬁguration C j in round j or it doesn’t.
First, suppose it does.
Then, the proposer sent
P HASE 1A (cid:104) i (cid:105) messages to all of the acceptors in C j .
A Phase 1 quorum of these acceptors, say Q , all received P HASE 1A (cid:104) i (cid:105) messages and replied with P HASE 1B messages.
Thus, every acceptor in Q set its round r to i , and in doing so, promised to never vote in any round less than i .
Moreover, none of the acceptors in Q had voted in any round greater than k .
So, every acceptor in Q has not voted and never will vote in round j .
For a value v (cid:48) to be chosen in round j , it must receive votes from some Phase 2 quorum Q (cid:48) of round j acceptors.
But, Q and Q (cid:48) necessarily intersect, so this is impossible.
Thus, no value has been or will be chosen in round j .
Now suppose that H i does not contain a conﬁguration for round j .
H i is the union of f + 1 M ATCH B messages from the f + 1 matchmakers in M i .
Thus, if H i does not contain a conﬁguration for round j , then none of the M ATCH B messages did either.
This means that for every matchmaker m ∈ M i , when m received M ATCH A (cid:104) i , C i (cid:105) , it did not contain a conﬁguration for round j in its log.
Moreover, by processing the M ATCH A (cid:104) i , C i (cid:105) request, the matchmaker is guaranteed to never process a M ATCH A (cid:104) j , C j (cid:105) request in the future.
Thus, every matchmaker in M i has not processed a M ATCH
A request in round j and never will.
For a value to be chosen in round j , the proposer executing round j must ﬁrst receive replies from f + 1 matchmakers, say M j , in round j .
But, M i and M j necessarily intersect, so this is impossible.
Thus, no value has been or will be chosen in round j .
Case 2: j = k .
In a given round, at most one value is proposed, let alone chosen.
x is the value proposed in round k , so no other value could be chosen in round k .
Case 3: j < k .
By induction, P ( k ) states that no value other than x has been or will be chosen in any round less than k .
This includes round j .
Finally, if k is − 1, then we are in the same situation as in
Case 1.
No value has or will be chosen in a round j < i .
3. Garbage Collection (How)
We’ve discussed how a proposer can change its round and introduce a new conﬁguration.
Now, we explain how to shut down old conﬁgurations.
At the beginning of round i , a proposer p executes the Matchmaking phase and computes a set
H i of conﬁgurations in rounds less than i .
The proposer then executes Phase 1 with the acceptors in these conﬁgurations.
Assume H i contains a conﬁguration C j for a round j < i .
If we prematurely shut down the acceptors in C j , then proposer p will get stuck in Phase 1, waiting for P HASE 1B messages from a quorum of nodes that have been shut down.
Therefore, we cannot shut down the acceptors in a conﬁguration C j until we are sure that the matchmakers will never again return C j during the Matchmaking phase.
Thus, we extend Matchmaker Paxos to allow matchmakers to garbage collect conﬁgurations from their logs, ensuring that the garbage collected conﬁgurations will not be returned during any future Matchmaking phase.
More concretely, a proposer p can now send a G ARBAGE A (cid:104) i (cid:105) command to the matchmakers informing them to garbage collect all conﬁgurations in rounds less than i .
When a matchmaker receives a G ARBAGE A (cid:104) i (cid:105) message, it deletes log entry L [ j ] for every round j < i .
It then updates a garbage collection watermark w to the maximum of w and i and sends back a G ARBAGE B (cid:104) i (cid:105) message to the proposer.
See Algorithm 4.
1: upon receiving G ARBAGE A (cid:104) i (cid:105) from proposer p do 2: delete L [ j ] for all j < i .
3: w ← max ( w , i ) 4: send G ARBAGE B (cid:104) i (cid:105) to p 5: upon receiving M ATCH
A (cid:104) i , C i (cid:105) from proposer p do 6: if i < w or ∃ C j in round j ≥ i in L then 7: ignore the M ATCH A (cid:104) i , C i (cid:105) message 8: else 9:
H i ← { ( j , C j ) | C j ∈ L } 10:
L [ i ] ← C i 11: send M ATCH B (cid:104) i , w , H i (cid:105) to p
We also update the Matchmaking phase in three ways.
First, a matchmaker ignores a M ATCH A (cid:104) i , C i (cid:105) message if i has been garbage collected (i.e. if i < w ).
Second, a matchmaker returns its garbage collection watermark w in every M ATCH B that it sends.
Third, when a proposer receives M ATCH B (cid:104) i , w 1 , H 1 i (cid:105) , ... , M ATCH B (cid:104) i , w f + 1 , H f + 1 i (cid:105) from f + 1 matchmakers, it again computes H i = ∪ f + 1 j = 1 H ji .
It then computes w = max f + 1 j = 1 w j and prunes every conﬁguration in H i in a round less than w .
In other words, if any of the f + 1 matchmakers have garbage collected round j , then the proposer also garbage collects round j .
Once a proposer receives G ARBAGE B (cid:104) i (cid:105) messages from at least f + 1 matchmakers M , it is guaranteed that all future
Matchmaking phases will not include any conﬁguration in any round less than i .
Why? Consider a future Matchmaking phase run with f + 1 matchmakers M (cid:48) .
M and M (cid:48) intersect, so some matchmaker in the intersection has a garbage collection watermark at least as large as i .
Thus, once a conﬁguration has been garbage collected by f + 1 matchmakers, we can shut down the acceptors in the conﬁguration.
3. Garbage Collection (When)
Once a conﬁguration has been garbage collected, it is safe to shut it down, but when is it safe to garbage collect a conﬁguration?
It is not always safe.
For example, if we prematurely garbage collect conﬁguration C j in round j , a future proposer in round i > j may not learn about a value v chosen in round j and then erroneously get a value other than v chosen in round i .
There are three situations in which it is safe for a proposer p i in round i to issue a G ARBAGE A (cid:104) i (cid:105) command.
We explain the three situations and provide intuition on why they are safe.
Later, we’ll see that all three scenarios are important for
Matchmaker MultiPaxos.
See Section A for a safety proof.
Scenario 1.
If the proposer p i gets a value x chosen in round i , then it can safely issue a G ARBAGE A (cid:104) i (cid:105) command.
Why?
When a proposer p j in round j > i executes Phase 1, it will learn about the value x and propose x in Phase 2.
But ﬁrst, it must establish that no value other than x has been or will be chosen in any round less than j .
This is P ( j ) from the safety proof in Section 3.3.
The proposer p i already established this fact for all rounds less than i (this is P ( i ) ), so any communication with the conﬁgurations in these rounds is redundant.
Thus, we can garbage collect them.
Scenario 2.
If the proposer p i executes Phase 1 in round i and ﬁnds k = − 1 (see Algorithm 3), then it can safely issue a G ARBAGE A (cid:104) i (cid:105) command.
Recall that if k = − 1, then no value has been or will be chosen in any round less than i .
This situation is similar to Scenario 1.
Any future proposer p j in round j > i does not have to redundantly communicate with the conﬁgurations in rounds less than i since p i already established that no value has been chosen in these rounds.
Scenario 3.
If the proposer p i learns that a value x has already been chosen and has been stored on f + 1 non-acceptor machines (e.g., f + 1 proposers), then the proposer can safely issue a G ARBAGE A (cid:104) i (cid:105) command after it informs a Phasequorum of acceptors in C i of this fact.
Any future proposer p j in round j > i will contact a Phase 1 quorum of C i and encounter at least one acceptor that knows the value x has already been chosen.
When this acceptor informs p j that a value x has already been chosen, p j stops executing the protocol entirely and simply fetches the value x from one of the f + 1 machines that store the value.
Note that storing the value on f + 1 machines ensures that some machine will store the value despite f failures.
The decision of exactly which f + 1 machines is not important.
Later, we’ll extend this garbage collection protocol to
Matchmaker MultiPaxos (Section 4) and see empirically that matchmakers usually return just a single conﬁguration (Section 7).
3. Optimizations
We now present a couple of protocol optimizations.
First, note that a proposer can proactively run the Matchmaking phase in round i before it hears from a client.
This is similar to proactively executing Phase 1, a standard optimization [12].
We call this optimization proactive matchmaking .
Second, assume that the proposer in round i has executed the Matchmaking phase and Phase 1.
Through Phase 1, it ﬁnds that k = − 1 and thus learns that no value has been chosen in any round less than i (see the safety proof above).
Assume that before executing Phase 2 in round
i , the proposer decides to perform a reconﬁguration.
To perform the reconﬁguration, the proposer stops executing round i and begins executing the next round i + 1 1 .
Typically to perform the reconﬁguration, the proposer would have to execute the Matchmaking phase,
Phase 1, and Phase 2 in round i + 1.
However, in this case, after executing the Matchmaking phase in round i + 1, the proposer can skip Phase 1 and proceed directly to Phase 2.
Why?
The proposer established in round i that no value has been or will be chosen in any round less than i .
Moreover, because it did not run Phase 2 in round i , it also knows that no value has been or will be chosen in round i .
Together, these imply that no value has been or will be chosen in any round less than i
+ 1.
Normally, the proposer would run Phasein round i + 1 to establish this fact, but since it has already established it, it can instead proceed directly to Phase 2.
We call this optimization Phase 1 bypassing .
Phase 1 Bypassing depends on a proposer being the leader of round
i and the leader of the next round i + 1.
We can construct a set of rounds such that this is always the case.
Let the set of rounds be the set of lexicographically ordered tuples ( r , id , s ) where r and s are both integers and id is a proposer id.
A proposer is responsible for all the rounds that contain its id. With this set of rounds, the proposer p in round ( r , p , s ) always owns the next round ( r , p , s + 1 ) .
For example given two proposers a and b , we have the following ordering on rounds: ( 0 , a , 0 ) < ( 0 , a , 1 ) < ( 0 , a , 2 ) <
( 0 , a , 3 ) < ··· ( 0 , b , 0 ) < ( 0 , b , 1 ) < ( 0 , b , 2 ) < ( 0 , b , 3 ) < ··· ( 1 , a , 0 ) < ( 1 , a , 1 ) <
( 1 , a , 2 ) <
( 1 , a , 3 ) < ···
We assume this round scheme throughout the rest of the paper.
In the next section, we’ll see that this optimization is essential for implementing Matchmaker MultiPaxos with good performance.
Also note that this optimization is not particular to
Matchmaker Paxos.
Paxos and MultiPaxos can both take advantage of this optimization.
Matchmaker MultiPaxos 4. MultiPaxos
First, we summarize MultiPaxos.
Whereas Paxos is a consensus protocol that agrees on a single value, MultiPaxos [17,40] is a state machine replication protocol that agrees on a sequence, or “log” of values.
MultiPaxos manages multiple replicas of a state machine.
Clients send state machine commands to MultiPaxos, MultiPaxos places the commands in a totally ordered log, and state machine replicas execute the commands in log order.
By beginning in the same initial state and executing the same commands in the same order, all deterministic state machine replicas are kept in sync.
cccppaaarr
Clients f +
Proposers
Conﬁguration C of Acceptors f
+
Replicas
To agree on a log of commands, MultiPaxos implements one instance of Paxos for every log entry.
The i th instance of
Paxos chooses the command in log entry i .
More concretely, a MultiPaxos deployment that tolerates f faults consists of an arbitrary number of clients, at least f + 1 proposers, a conﬁguration C of acceptors which can tolerate f failures, and at least f + 1 replicas, as illustrated in Figure 4.
One of the proposers is elected leader in some round, say round i .
We assume the leader knows that log entries up to and including log entry k c have already been chosen (e.g., by communicating with the replicas).
We call this log entry the commit index .
The leader then runs Phase 1 of Paxos in round i for every log entry.
Note that even though there are an inﬁnite number of log entries larger than k c , the leader can execute Phase 1 using a ﬁnite amount of information.
In particular, the leader sends a single P HASE 1A (cid:104) i (cid:105) message that acts as the P HASE 1A message for every log entry larger than k c .
Also, an acceptor replies with a P HASE 1B (cid:104) i , vr , vv (cid:105) message only for log entries in which the acceptor has voted.
The inﬁnitely many log entries in which the acceptor has not yet voted do not yield an explicit P HASE 1B message.
a b c
[Equation]
Region 1:
[Figure]
Region 2:
[Figure]
Region 3:
[Figure]
The leader’s knowledge about the log after Phase 1 can be characterized by the commit index k c and a pending index k p with k c ≤ k p , as shown in Figure 5.
The commit index and pending index divide the log into three regions: a preﬁx of chosen log entries (Region 1), a sufﬁx of unchosen log entries (Region 3), and a middle region of pending log entries (Region 2).
More speciﬁcally:
• Region 1 [ 0 , k c ] : The leader knows that a command has been chosen in every log entry less than or equal to k c .
• Region 3 [ k p + 1 , ∞ ) :
The leader knows that no command has been chosen (in any round less than i ) in any log entry larger than k p .
• Region
[ k c + 1 , k p ] : If there is a command that may have already been chosen, then it appears between k c and k p .
Region 2 may also contain some log entries in which the leader knows (from executing a previous round) that a value has already been chosen, and it may contain some log entries in which the leader knows (from counting votes in Phase 1) that no value has been chosen (we call these “holes”).
After Phase 1, the leader sends a P HASE 2A message for every unchosen log entry in Region 2, proposing a “no-op” command for the holes.
Simultaneously, the leader begins accepting client requests.
When a client wants to propose a state machine command, it sends the command to the leader.
The leader assigns log entries to commands in increasing order, beginning at k p + 1.
It then runs Phase 2 of Paxos to get the command chosen in that entry in round i .
Once the leader learns that a command has been chosen in a given log entry, it informs the replicas.
Replicas insert chosen commands into their logs and execute the logs in preﬁx order, sending the results of execution back to the clients.
This execution is illustrated in Figure 4.
It is critical to note that a leader performs Phase 1 of Paxos only once per round , not once per command .
In other words,
Phase 1 is not performed during normal operation.
It is performed only when the leader fails and a new leader is elected in a larger round, an uncommon occurrence.
4. Matchmaker MultiPaxos
We ﬁrst extend Matchmaker Paxos to Matchmaker MultiPaxos with proactive matchmaking but without Phase 1 bypassing or garbage collection.
We’ll see how to incorporate these two momentarily in Section 4.4.
The extension from
Matchmaker Paxos to Matchmaker MultiPaxos is analogous to the extension of Paxos to MultiPaxos.
Matchmaker MultiPaxos reaches consensus on a totally ordered log of state machine commands, one log entry at a time, using one instance of Matchmaker Paxos for every log entry.
More concretely, a Matchmaker MultiPaxos deployment consists of an arbitrary number of clients, at least f + 1 proposers, a set of 2 f + 1 matchmakers, a dynamic set of acceptors (one conﬁguration per round which can tolerate f failures), and a set of at least f + 1 state machine replicas.
We assume, as is standard, that a leader election algorithm is used to select one of the proposers as a stable leader in some round, say round i .
The leader selects a conﬁguration
C i of acceptors that it will use for every log entry.
The mechanism by which the conﬁguration is chosen is an orthogonal concern.
A system administrator, for example, could send the conﬁguration to the leader, or the conﬁguration could be read from an external service.
Throughout the paper, we do not depend any speciﬁc mechanism by which a conﬁguration is chosen.
We assume that proposers use some unspeciﬁed abstract process to select conﬁgurations.
The leader then executes the Matchmaking phase in the same way as in Matchmaker Paxos (i.e. it sends
M ATCH A (cid:104) i , C i (cid:105) messages to the matchmakers and awaits
M ATCH B (cid:104) i , H i (cid:105) responses).
After the Matchmaking phase completes, the leader executes Phase 1 for every log entry.
This is identical to MultiPaxos, except that the leader uses the conﬁgurations returned by the matchmakers rather than assuming a ﬁxed conﬁguration.
Note that proactive matchmaking allows the leader to execute the Matchmaking phase and Phase 1 before receiving any client requests.
The leader then enters Phase 2 and operates exactly as it would in MultiPaxos.
It executes Phase 2 with C i for the log entries in Region 2.
Moreover, when it receives a state machine command from a client, it assigns the command a log entry in Region 3, runs Phase 2 with the acceptors in
C i , and informs the replicas when the command is chosen.
Replicas execute commands in log order and send the results of executing commands back to the clients.
4. Discussion
To reconﬁgure from some old conﬁguration C old in round i to some new conﬁguration C new , the Matchmaker MultiPaxos leader of round i simply advances to round i + 1 and selects the new conﬁguration C new .
The new conﬁguration is active immediately after the Matchmaking phase, a one round trip delay.
Note that the acceptors in the new conﬁguration C new do not have to undergo any sort of warm up or bootstrapping and do not have to contact any other acceptors in any other conﬁguration.
The new conﬁguration is active immediately, but it is not safe to deactivate the acceptors in the old conﬁguration immediately, as we saw in Section 3.5.
We extend Matchmaker
Paxos’s garbage collection to Matchmaker MultiPaxos momentarily.
Also note that Matchmaker MultiPaxos does not perform the Matchmaking phase or Phase 1 on the critical path of normal execution.
Similar to how MultiPaxos executes Phase 1 only once per leader change (and not once per command),
Matchmaker MultiPaxos runs the Matchmaking phase and
Phase 1 only when a new leader is elected or when a leader changes its round (e.g., when a leader transitions from round cccppmmmaaabbb 12 a b b c c d (a) Matchmaking cccppmmmaaabbb  a (b) Phasecccppmmmaaabbb
5 6a b b c c d (c) Phase does not introduce any overheads.
In the normal case, Matchmaker
MultiPaxos deploys a single stable leader that changes rounds only to perform a reconﬁguration.
Changing from one leader to another only happens after a leader has failed.
Furthermore, conﬁgurations do not have to be unique across rounds.
The leader in round i is free to re-use a conﬁguration C j that was used in some round j < i .
Finally, because Matchmaker MultiPaxos deploys more nodes than MultiPaxos, the mean time to failure is decreased, and it will take less time to reach f failures.
However, this mean time to failure is many orders of magnitude larger than the time required to perform a reconﬁguration.
As long as failed machines are replaced via reconﬁguration in a reasonable amount of time, it is unlikely to experience f or more failures.
4. Optimization
Ideally, Matchmaker MultiPaxos’ performance would be unaffected by a reconﬁguration.
The latency of every client request and the protocol’s overall throughput would remain constant throughout a reconﬁguration.
Matchmaker MultiPaxos as we’ve described it so far, however, does not meet this ideal.
During a reconﬁguration, a leader must temporarily stop processing client commands and wait for the reconﬁguration to ﬁnish before resuming normal operation.
This is illustrated in Figure 6.
Figure 6 shows a leader p 1 reconﬁguring from a conﬁguration of acceptors C old consisting of acceptors a 1 , a 2 , and a 3 in round i to a new conﬁguration of acceptors C new consisting of acceptors b 1 , b 2 , and b 3 in round i + 1.
While the leader performs the reconﬁguration, clients continue to send state machine commands to the leader.
We consider such a command and perform a case analysis on when the command arrives at the leader to see whether or not the command has to be stalled.
Case 1: Matchmaking (Figure 6a) .
If the leader receives a command during the Matchmaking phase, then the leader can process the command as normal in round i using the acceptors in C old .
Even though the leader is executing the
Matchmaking phase in round i + 1 and is communicating with the matchmakers, the acceptors in C old are oblivious to this and can process commands in Phase 2 in round i .
Case 2: Phase 1 (Figure 6b) .
If the leader receives a command during Phase 1, then the leader cannot process the command.
It must delay the processing of the command until Phase 1 ﬁnishes.
Here’s why.
Once an acceptor in C old receives a P HASE 1A (cid:104) i + 1 (cid:105) message, it will reject any future commands in rounds less than i + 1, so the leader is unable to send the command to C old .
The leader also cannot send the command to C new in round i + 1 because it has not yet ﬁnished executing Phase 1.
Case 3: Phase 2 (Figure 6c) .
If the leader receives a command during Phase 2, then the leader can send the command to the new acceptors in C new in round i + 1.
This is the normal case of execution.
In summary, any commands received during Phase 1 of a reconﬁguration are delayed.
Fortunately, we can eliminate this problem by using Phase 1 bypassing.
Consider a leader performing a reconﬁguration from C i in round i to C i + 1 in round i + 1.
At the end of the Matchmaking phase and at the beginning of Phase 1 (in round i + 1), let k be the largest log entry that the leader has assigned to a command.
That is, all log entries after entry k are empty.
These log entries satisfy the preconditions of Phase 1 bypassing, so it is safe for the leader to bypass Phase 1 in round i + 1 for these log entries in the following way.
When a leader receives a command after the Matchmaking phase, it assigns the command a log entry larger than k , skips Phase 1, and executes Phase 2 in round i + 1 with C new immediately.
With this optimization and the round scheme described in
Section 3.6, no state machine commands are delayed.
Commands received during the Matchmaking phase or earlier are chosen in round i by C old in log entries up to and including k .
Commands received during Phase 1, Phase 2, or later are chosen in round i + 1 by C new in log entries k + 1, k + 2, k + 3, and so on.
With this optimization Matchmaker MultiPaxos can be reconﬁgured with minimal performance degradation.
4. Garbage Collection
Recall that the Matchmaker MultiPaxos leader p i in round i uses a single conﬁguration C i for every log entry.
The leader p i can safely issue a G ARBAGE A (cid:104) i (cid:105) command to the matchmakers once it ensures that every log entry satisﬁes one of the three scenarios described in Section 3.5.
Recall from Figurethat at the end of Phase 1 and at the beginning of Phase 2, the log can be divided into three regions.
Each of the three garbage collection scenarios applies to one of the regions.
Scenario 2 applies to Region 3.
These are the log entries for which k
= − 1.
Scenario 1 applies to Region 2, once the leader has successfully chosen commands in all of the log entries in Region 2.
Scenario 3 applies to Region 1 if we make the following adjustments.
First, we deploy 2 f + 1 replicas instead of f + 1.
Second, the leader ensures that the preﬁx of previously chosen log entries is stored on at least f +of the 2 f + 1 replicas.
Third, the leader informs a Phasequorum of C i acceptors that these commands have been stored on the replicas.
Every replica maintains a copy of the log of state machine commands and cannot discard a command after execution.
The log must also be garbage collected over time, for example, by using snapshots [34].
Note that garbage collecting the log is an orthogonal (but also complicated) issue from garbage collecting old conﬁgurations.
It must be done regardless of reconﬁgurations and is outside of the scope of this paper.
In summary, the leader p i of round i executes as follows.
It executes the Matchmaking phase to get the prior conﬁgurations H i .
It executes Phase 1 with the conﬁgurations in H i .
It enters Phase 2 and chooses commands in Region 2.
It informs a Phase 2 quorum of C i acceptors once the commands in Region 1 have been stored on f + 1 replicas.
It issues a G ARBAGE A (cid:104) i (cid:105) command to the matchmakers and awaits f + 1 G ARBAGE B (cid:104) i (cid:105) responses.
At this point, all previous conﬁgurations can be shut down.
Note that the leader can begin processing state machine commands from clients as soon as it enters Phase 2.
It does not have to stall commands during garbage collection.
Note also that during normal operation, old conﬁgurations are garbage collected very quickly.
In Section 7, we show that H i almost always contains a single conﬁguration (i.e. C i − 1 ).
Reconﬁguring Matchmakers
We’ve discussed how Matchmaker MultiPaxos allows us to reconﬁgure the set of acceptors.
In this section, we discuss how to reconﬁgure proposers, replicas, and matchmakers (themselves).
Reconﬁguring proposers and replicas is straightforward.
In fact, Matchmaker MultiPaxos reconﬁgures proposers and replicas in exactly the same way as MultiPaxos [40], so we do not discuss it at length.
In short, a proposer can be safely added or removed at any time.
Replicas can also be safely added or removed at any time so long as we ensure that commands replicated on f + 1 replicas remain replicated on f +replicas.
This is a difﬁcult, yet orthogonal challenge.
Existing approaches can be adopted by Matchmaker MultiPaxos [35].
For performance, a newly introduced proposer should contact an existing proposer or replica to learn about the preﬁx of already chosen commands, and a newly introduced replica should copy the log from an existing replica.
Reconﬁguring matchmakers is a bit more involved, but still relatively straightforward.
Recall that proposers perform the
Matchmaking phase only during a change in round.
Thus, for the vast majority of the time—speciﬁcally, when there is a single, stable leader—the matchmakers are completely idle.
This means that the way we reconﬁgure the matchmakers has to be safe, but it doesn’t have to be efﬁcient.
The matchmakers can be reconﬁgured at any time between round changes without any impact on the performance.
Thus, we use the simplest approach to reconﬁguration: we shut down the old matchmakers and replace them with new ones, making sure that the new matchmakers’ initial state is the same as the old matchmakers’ ﬁnal state.
More concretely, we reconﬁgure from a set M old of matchmakers to a new set M new as follows.
First, a proposer (or any other node) sends a S TOP A (cid:104)(cid:105) message to the matchmakers in M old .
When a matchmaker m i receives a S TOP A (cid:104)(cid:105) message, it stops processing messages (except for other S TOP A (cid:104)(cid:105) messages) and replies with S TOP B (cid:104)
L i , w i (cid:105) where L i is m i ’s log and w i is its garbage collection watermark.
When the proposer receives
S TOP B messages from f + 1 matchmakers
, it knows that the matchmakers have effectively been shut down.
It computes w as the maximum of every returned w i .
It computes L as the union of the returned logs, and removes all entries of L that appear in a round less than w .
An example of this log merging is illustrated in Figure 7.
The proposer then sends L and w to all of the matchmakers in M new .
Each matchmaker adopts these values as its initial state.
At this point, the matchmakers in M new cannot begin processing commands yet.
Naively, it is possible that two different nodes could simultaneously attempt to reconﬁgure to two disjoint sets of matchmakers, say M new and M (cid:48) new .
To avoid this, every matchmaker in M old doubles as a Paxos acceptor.
A proposer attempting to reconﬁgure to M new acts as a Paxos proposer and gets the value M new chosen by the
C 0 ( w )
C
C
L××
C 2 ( w )
L×
C 1 ( w )
C
L××
C 2 ( w )
C 4 matchmakers (which are acting as Paxos acceptors).
Once
M new is chosen, the proposer notiﬁes the matchmakers in
M new that the reconﬁguration is complete and that they are free to start processing commands.
If a proposer contacts a stale set of matchmakers (e.g.,
M old )
, the matchmakers inform the proposer of their successors (e.g., M new ).
This newer set of matchmakers may also be stale, so the proposer repeatedly polls stale matchmakers until it ﬁnds the active set of matchmakers.
In this way, the matchmakers form a chain, with each set of matchmakers pointing to its successor.
Before a set of matchmakers can be shut down, the identity of its successors must be persisted in some name service (e.g., DNS).
Ideally for performance, the name service would always point to the active set of matchmakers, but this is not required for safety.
Theoretical Insights
MultiPaxos To reconﬁgure from a set of nodes N to a new set of nodes N (cid:48) , a MultiPaxos leader gets the value N (cid:48) chosen in the log at some index i .
All commands in the log starting at position i + α are chosen using the nodes in N (cid:48) instead of the nodes in N , where α is some conﬁgurable parameter.
This protocol is called Horizontal MultiPaxos .
[Equation]
no-op e f ··· α chosen with N chosen with N (cid:48)
Matchmaker MultiPaxos has the following advantages over
Horizontal MultiPaxos.
First, the core idea behind Horizontal
MultiPaxos seems simple, but the protocol has a number of hidden subtleties [27].
For example, a newly elected Horizontal MultiPaxos leader with a stale log may not know the latest conﬁguration of nodes.
It may not even know which conﬁguration of nodes to contact to learn the latest conﬁguration of nodes.
This makes it unclear when it is safe to shut down old conﬁgurations because a newly elected Horizontal MultiPaxos leader can be arbitrarily out of date.
These subtleties and the many others described in [27] makes Horizontal MultiPaxos signiﬁcantly more complicated than it initially seems.
Matchmaker Paxos addresses these subtleties directly.
The matchmakers can always be used to learn the latest conﬁguration, and our garbage collection protocol details exactly when and how to shut down old conﬁgurations safely.
Second, horizontal reconﬁguration is not generally applicable.
It is fundamentally incompatible with replication protocols that do not replicate a log.
Moreover, researchers are ﬁnding that avoiding a log can often be advantageous [2,8,18,31,38,39,41].
For example, protocols like Generalized Paxos [18], EPaxos [31], Atlas [8], and Caesar [2] arrange commands in a partially ordered graph instead of a totally ordered log to exploit commutativity between commands.
CASPaxos [38] maintains a single value, instead of a log or graph, for simplicity.
Databases like TAPIR [41] avoid ordering transactions in a log for improved performance, and databases like Meerkat [39] do the same to improve scalability.
Even some protocols with logs cannot use the ideas behind Horizontal MultiPaxos.
For example, Raft cannot safely perform Horizontal MultiPaxos’ reconﬁguration [34].
Because these protocols do not replicate logs, they cannot use MultiPaxos’ horizontal reconﬁguration protocol.
However, while none of the protocols replicate logs, all of them have rounds.
This means that the protocols can either use
Matchmaker Paxos directly, or at least borrow ideas from
Matchmaker Paxos for reconﬁguration.
For example, we are developing a protocol called BPaxos that is an EPaxos [31] variant which partially orders commands into a graph.
BPaxos is a modular protocol that uses Paxos as a black box subroutine.
Due to this modularity, we can directly replace Paxos with Matchmaker Paxos to support reconﬁguration.
The same idea can also be applied to EPaxos.
CASPaxos [38] is similar to Paxos and can be extended to Matchmaker CASPaxos in the same way we extended Paxos to Matchmaker Paxos.
These are two simple examples, and we don’t claim that extending Matchmaker Paxos to some of the other more complicated protocols is always easy.
But, the universality of rounds makes Matchmaker Paxos an attractive foundation on top of which other non-log based protocols can build their own reconﬁguration protocols.
One could argue that these other protocols are not used as much in industry, so it’s not that important for them to have reconﬁguration protocols, but we think the causation is in the reverse direction!
Without reconﬁguration, these protocols cannot be used in industry.
Third, optimizing Horizontal MultiPaxos is not easy.
A
MultiPaxos leader can process at most α unchosen commands at a time.
This makes α an important parameter to tune.
If we set α too low, then we limit the protocol’s pipeline parallelism and the throughput suffers.
Note that a small α reduces the normal case throughput of Horizontal MultiPaxos, not just the throughput during reconﬁguration.
If we set α too high, then we have to wait a long time for a reconﬁguration to complete.
If we are reconﬁguring because of a failed node, then we might have to endure a long reconﬁguration with reduced throughput.
Matchmaker MultiPaxos has no α parameter to tune.
Note that Horizontal MultiPaxos can be implemented with an optimization in which we select a very large α and then get a sequence of α noops in the log to force a quick reconﬁguration.
This optimization helps avoid the difﬁculties of ﬁnding a good value of α , but the optimization introduces a new set of subtleties into the protocol.
For example, the leader cannot process client requests while it is executing Phasefor the α noops.
The protocol has to implement additional mechanisms to avoid this one round trip stall.
Fourth, Horizontal MultiPaxos requires a Phase 1 and
Phase 2 quorum of acceptors from an old conﬁguration in order to perform a reconﬁguration after a leader failure, but
Matchmaker MultiPaxos only requires a Phase 1 quorum.
Some read optimized MultiPaxos variants perform reads against Phase 1 quorums [5].
These protocols beneﬁt from having very small Phase 1 quorums and very large Phasequorums, requiring Horizontal MultiPaxos to contact far more nodes than Matchmaker MultiPaxos during a reconﬁguration.
Finally, we clarify that if Horizontal MultiPaxos is implemented with all of its subtleties ironed out, is deployed with a good choice of α , and is run with small Phase 2 quorums, then it can perform a reconﬁguration without performance degradation.
In this case, Horizontal MultiPaxos and Matchmaker
MultiPaxos both reconﬁgure, in some sense, “optimally”.
Horizontal MultiPaxos also has some advantages over
Matchmaker MultiPaxos.
For example, reconﬁguring the set of matchmakers is simple, but it is still another reconﬁguration protocol that has to be implemented which adds complexity to the system.
Vertical Paxos Matchmaker MultiPaxos signiﬁcantly improves the practicality of Vertical Paxos [22] in a number of ways.
First, Vertical Paxos is a consensus protocol, not a state machine replication protocol, and it’s not easy to extend
Vertical Paxos’ garbage collection protocol to a state machine replication protocol.
Vertical Paxos garbage collects old conﬁgurations in situations similar to Scenario 1 and Scenario 2 from Section 3.5.
It does not include Scenario 3.
Without this, old conﬁgurations cannot be garbage collected, which means that it is never safe to shut down old conﬁgurations.
Second, Vertical Paxos requires an external master but does not describe how to implement the master in an efﬁcient way.
We could implement the master using another state machine replication protocol like MultiPaxos, but this would be both slow and overly complex.
Plus, we would have to implement a reconﬁguration protocol for the master as well.
Our matchmakers are analogous to the external master but show that such a master does not require a nested invocation of state machine replication.
Third, Vertical Paxos requires that a proposer execute Phase 1 in order to perform a reconﬁguration.
Thus, Vertical Paxos cannot be extended to MultiPaxos without causing performance degradation during reconﬁguration.
This is not the case for matchmakers thanks to Phase 1 bypassing.
Fourth, Vertical Paxos does not describe how proposers learn the conﬁgurations used in previous rounds and instead assumes that conﬁgurations are ﬁxed in advance by an oracle.
Matchmaker Paxos shows that this assumption is not necessary, as the matchmakers store every conﬁguration.
Fast Paxos Fast Paxos [19] is a Paxos variant that shaves off one network delay from Paxos in the best case, but can have higher delays if concurrently proposed commands conﬂict.
While Paxos quorums consist of f + 1 out of 2 f + 1 acceptors,
Fast Paxos requires larger quorums.
Many protocols have reduced Fast Paxos quorum sizes a bit, but to date, Fast Paxos quorum sizes have remained larger than classic Paxos quorum sizes [8, 31].
Using matchmakers, we can implement Fast
Paxos with a ﬁxed set of f + 1 acceptors (and hence with f + 1sized quorums).
Speciﬁcally, we deploy Fast Paxos with f +acceptors, with a single unanimous Phase 2 quorum, and with singleton Phase 1 quorums.
A full description of the protocol and a proof of correctness is given in Section C.
DPaxos DPaxos is a Paxos variant that allows every round to use a different subset of acceptors from some ﬁxed set of acceptors.
Matchmaker Paxos obviates the need for a ﬁxed set of nodes.
DPaxos’ scope is limited to a single instance of consensus, whereas Matchmaker MultiPaxos shows how to efﬁciently reconﬁgure across multiple instances of consensus simultaneously.
We also discovered that DPaxos’ garbage collection algorithm is unsafe.
Matchmaker MultiPaxos ﬁxes the bug.
See Section D for details.
Evaluation
We now evaluate Matchmaker MultiPaxos.
Matchmaker MultiPaxos is implemented in Scala using the Netty networking library.
We deployed Matchmaker MultiPaxos on m5.xlarge
AWS EC2 instances within a single availability zone.
We deploy Matchmaker MultiPaxos with f = 1, f + 1 proposers, 2 f + 1 acceptors, 2 f + 1 matchmakers, and 2 f + 1 replicas.
For simplicity, every node is deployed on its own machine, but in practice, nodes can be physically co-located.
In particular, any two logical roles can be placed on the same machine, so long as the two roles are not the same.
For example, we can co-locate a leader, an acceptor, a replica, and a matchmaker, but we can’t co-locate two acceptors (without reducing the fault tolerance of the system).
For simplicity, we deploy
[Table]
Matchmaker MultiPaxos with a trivial no-op state machine in which every state machine command is a one byte no-op.
All of our results generalize to more complex state machines as well (the choice of state machine is orthogonal to reconﬁguration).
7. Reconﬁguration
Experiment Description.
We run a benchmark with 1, 4, and 8 clients.
Every client executes in a closed loop.
It repeatedly proposes a state machine command, waits to receive a response
[Table]
and then immediately proposes another command.
This model is standard for state machine replication protocols [25,31,36] and aligns with the deﬁnitions surrounding linearizability [11].
Every benchmark runs for 35 seconds.
During the ﬁrst 10 seconds, we perform no reconﬁgurations.
From 10 seconds to 20 seconds, the leader reconﬁgures the set of acceptors once every second.
In practice, we would reconﬁgure much less often.
This is a worst case stress test for Matchmaker MultiPaxos.
For each of the ten reconﬁgurations, the leader selects a random set of 2 f + 1 acceptors from a pool of 2 × ( 2 f + 1 ) acceptors.
At 25 seconds, we fail one of the acceptors.
5 seconds later, the leader performs a reconﬁguration to replace the failed acceptor.
The delay ofseconds is completely arbitrary.
The leader can reconﬁgure sooner if desired.
We also perform this experiment with an implementation of
MultiPaxos with horizontal reconﬁguration.
As with Matchmaker MultiPaxos, we deploy MultiPaxos with f + 1 proposers, 2 f + 1 acceptors, and 2 f + 1 replicas.
We set α to 8.
Because α is equal to the number of clients, MultiPaxos never stalls because of an insufﬁciently large α .
We do not implement the noop optimization.
Results.
The latency and throughput of Matchmaker MultiPaxos are shown in Figure 9.
Throughput and latency are both computed using sliding one second windows.
Median latency
[Figure]
The medians, interquartile ranges (IQR), and standard deviations of the latency and throughput (a) during the ﬁrstseconds and (b) between 10 and 20 seconds are shown in
Table 1.
Figure 12 includes violin plots of the same data.
The white circles show the median values, while the thick black rectangles show the 25th and 75th percentiles.
For latency, reconﬁguration has little to no impact (roughly 2% changes) on the medians, IQRs, or standard deviations.
The one exception is that the 8 client standard deviation is signiﬁcantly larger.
This is due to a small number of outliers.
Reconﬁguration has little impact on median throughput, with all differences being statistically insigniﬁcant.
The IQRs and standard deviations sometimes increase and sometimes decrease.
The IQR is always less than 1% of the median throughput, and the standard deviation is always less than 4%.
For every reconﬁguration, the new acceptors become active within a millisecond.
The old acceptors are garbage collected within ﬁve milliseconds.
This means that only one conﬁguration is ever returned by the matchmakers.
We implement
Matchmaker MultiPaxos with an optimization called thriftiness [31]—where P HASE 2A messages are sent to a randomly selected Phase 2 quorum—so the throughput and latency expectedly degrade after we fail an acceptor.
After we replace the failed acceptor, throughput and latency return to normal within two seconds.
The latency and throughput of MultiPaxos is shown in Fig-
[Figure]
T h r o u g h p u t ( c m d s / s e c o n d ) ure 10.
As with Matchmaker MultiPaxos, MultiPaxos can perform a horizontal reconﬁguration without any performance degradation.
The difference in absolute throughput between the two protocols is due to minor implementation differences, but the variance in throughput (rather than the throughput itself) is what is important for this evaluation.
We include the comparison to MultiPaxos for the sake of having some baseline against which we can compare Matchmaker MultiPaxos, but the comparison is shallow.
For this reason, we do not elaborate on the results much.
While Matchmaker MultiPaxos does provide performance beneﬁts over MultiPaxos’ and Raft’s reconﬁguration protocols, our goal is not to replace these protocols.
Rather, there are dozens of other state machine replication protocols (e.g.,
EPaxos [31], CASPaxos [38], Caesar [2], Atlas [8]) and distributed databases (e.g., TAPIR [41], Janus [32], Ocean
Vista [9]) that do not have any reconﬁguration protocol and cannot use the existing reconﬁguration protocols from MultiPaxos or Raft.
Our hope is that the ideas in Matchmaker
MultiPaxos can be used to implement reconﬁguration protocols for these other systems.
For this reason, it is difﬁcult to compare Matchmaker MultiPaxos against some existing baseline because they simply do not exist.
Summary.
This experiment conﬁrms that Matchmaker
MultiPaxos’s throughput and latency remain steady even during abnormally frequent reconﬁguration.
Moreover, it conﬁrms that Matchmaker MultiPaxos can reconﬁgure to a new set of acceptors and retire the old set of acceptors on the order of milliseconds.
[Figure]
7. Leader Failure
Experiment Description.
We deploy Matchmaker MultiPaxos exactly as before.
Now, each benchmark runs forseconds.
During the ﬁrst 7 seconds, there are no reconﬁgurations and no failures.
At 7 seconds, we fail the leader.seconds later, a new leader is elected and resumes normal operation.
The 5 second delay is arbitrary; a new leader could be elected quicker if desired.
Results.
The latency and throughput of the benchmarks are shown in Figure 13.
During the ﬁrst 7 seconds, throughput and latency are both stable.
When the leader fails, the throughput expectedly drops to zero.
The throughput and latency return to normal within two seconds after a new leader is elected.
The results for the same experiment, repeated with
Horizontal MultiPaxos, are shown in Figure 14.
Summary.
This experiment conﬁrms that the extra latency of the Matchmaker phase during a leader change is negligible.
7. Matchmaker Reconﬁguration
Experiment Description.
We deploy Matchmaker MultiPaxos as above.
We again run three benchmarks with 1, 4, and 8 clients.
Each benchmark runs for 40 seconds.
During the ﬁrst 10 seconds, there are no reconﬁgurations and no failures.
Between 10 and 20 seconds, the leader reconﬁgures the set of matchmakers once every second.
Every reconﬁguration randomly selects 2 f + 1 matchmakers from a set of 2 × ( 2 f + 1 ) matchmakers.
At 25 seconds, we fail a matchmaker.
At 30 we perform a matchmaker reconﬁguration to replace the failed matchmaker.
At 35 seconds, we reconﬁgure the acceptors.
[Figure]
L a t e n c y (
[Figure]
s )
[Figure]
client
[Figure]
clients
[Figure]
T h r o u g h p u t ( c o mm a n d s / s e c o n
[Figure] [Figure]
Results.
The latency and throughput of Matchmaker MultiPaxos are shown in Figure 15.
The latency and throughput of the protocol remain steady through the ﬁrst ten matchmaker reconﬁgurations, through the matchmaker failure and recovery, and through the acceptor reconﬁguration.
This is conﬁrmed by the medians, IQRs, and standard deviations of the latency and throughput during the ﬁrst 10 seconds and between 10 and 20 seconds, which are shown in Table 2.
Summary.
This benchmark conﬁrms that matchmakers are off the critical path.
The latency and throughput of Matchmaker MultiPaxos remains steady during a matchmaker reconﬁguration and matchmaker failure.
Moreover, a matchmaker reconﬁguration does not affect the performance of subsequent acceptor reconﬁgurations.
Related Work
SMART.
SMART [27] is a reconﬁguration protocol that resolves many ambiguities in MultiPaxos’ horizontal approach (e.g., when it is safe to retire old conﬁgurations).
Like MultiPaxos’ horizontal reconﬁguration protocol, SMART can reconﬁgure a protocol with minimal performance degradation.
SMART differs from Matchmaker Paxos in a number of ways.
First, like MultiPaxos’ horizontal reconﬁguration protocol, SMART is fundamentally log based and is therefore incompatible with many sophisticated state machine replication protocols.
Second, SMART assumes that acceptors and replicas are always co-located.
This prevents us from reconﬁguring the acceptors without reconﬁguring the replicas.
This is not ideal since we can reconﬁgure an acceptor without copying any state, but must transfer logs from an old replica
[Figure]
[Table]
second)
[Table]
to a new replica.
SMART’s garbage collection also has higher latency that Matchmaker Paxos’ garbage collection.
For Scenario 3, Matchmaker Paxos proposers wait until a preﬁx of the log is stored on f + 1 replicas.
SMART waits for the preﬁx of the log to be executed and snapshotted by f + 1 replicas.
Cheap Paxos.
Cheap Paxos [24] is a MultiPaxos variant that consists of a ﬁxed set of f + 1 main acceptors and f auxiliary acceptors.
During failure-free execution (the normal case), only the main acceptors are contacted.
The auxiliary acceptors perform MultiPaxos’ horizontal reconﬁguration protocol to replace failed main acceptors.
As with Fast Paxos, we can deploy Matchmaker MultiPaxos with only f + 1 acceptors, f fewer than Cheap Paxos.
Matchmaker Paxos does require 2 f + 1 matchmakers, but matchmakers do not act as acceptors and have to process only a single message (i.e. a
M ATCH A message) to perform a reconﬁguration.
L a t e n c y ( m s )
Time
T h r o u g h p u t ( c m d s / s e c o n d )
Raft.
Raft [35] uses a reconﬁguration protocol called joint consensus.
Like MultiPaxos’ horizontal reconﬁguration, joint consensus is log-based and therefore incompatible with many existing replication protocols.
A simpler reconﬁguration protocol for Raft was proposed in [34] but requires more rounds of communication.
Viewstamped Replication (VR).
VR [26] uses a stopthe-world approach to reconﬁguration.
During a reconﬁguration, the entire protocol stops processing commands.
Thus, while the reconﬁguration is quite simple, it is inefﬁcient.
Stoppable Paxos [21] is similar to MultiPaxos’ horizontal reconﬁguration, but also uses a stop-the-world approach.
VR’s stop-the-world approach to reconﬁguration is also adopted by databases built on VR, including TAPIR [41] and Meerkat [39].
We use a similar approach to reconﬁgure matchmakers, but because matchmakers are off the critical path, the performance overheads are invisible.
Fast Paxos Coordinated Recovery.
Fast Paxos has an optimization called coordinated recovery that is similar to Phase 1 Bypassing.
The main difference is that in coordinated recovery, a leader uses Phase 2 information in round i to skip
Phase 1 in round i + 1, whereas with Phase 1 Bypassing, the leader instead uses Phase 1 information.
Note that coordinated recovery is not useful for Matchmaker MultiPaxos.
It is subsumed by Phase 1 Bypassing.
Coordinated recovery is only needed for Fast Paxos because the leader may not know which values were proposed in a round it owns.
Phase 1 Bypassing cannot be applied to Fast Paxos for pretty much the same reason.
DynaStore.
Vertical Paxos assumes its external master is implemented using state machine replication.
MultiPaxos’ horizontal reconﬁguration also depends on consensus.
Matchmaker Paxos does not require consensus to implement match-
[Figure]
clients makers, but we are not the ﬁrst to notice this.
DynaStore [1] showed that reconﬁguring atomic storage does not require consensus.
ZooKeeper.
ZooKeeper, a distributed coordinated service, which uses ZooKeeper Atomic Broadcast [14] is a protocol similar to MultiPaxos that can also reconﬁgure quickly after leader failures.
Conclusion
We presented Matchmaker Paxos and Matchmaker MultiPaxos to address the lack of research on the increasingly important topic of reconﬁguration.
Our protocols achieve a number of desirable properties, both theoretical and practical: they can reconﬁgure without performance degradation, they provide insights into existing protocols, and they generalize better than existing techniques.
Our implementations of Matchmaker Paxos and Matchmaker MultiPaxos can be found online at https://github.com/mwhittaker/ frankenpaxos .
A Garbage Collection Safety
To prove that the three scenarios from Section 3.5 are safe, we repeat the safety proof from Section 3.
The new bits are shown in blue.
Proof.
We prove, for every round i , the statement P ( i ) : “if a proposer proposes a value v in round i (i.e. sends a P HASE 2A message for value v in round i ), then no value other than v has been or will be chosen in any round less than i .”
At most one value is ever proposed in a given round, so at most one value is ever chosen in a given round.
Thus, P ( i ) sufﬁces to prove that Matchmaker Paxos is safe for the following reason.
Assume for contradiction that Matchmaker Paxos chooses distinct values x and y in rounds j
and i with j < i .
Some proposer must have proposed y in round i , so P ( i ) ensures us that no value other than y could have been chosen in round j .
But, x was chosen, a contradiction.
We prove P ( i ) by strong induction on i .
P ( 0 ) is vacuous because there are no rounds less than 0.
For the general case P ( i ) , we assume P ( 0 ) ,..., P ( i − 1 ) .
We perform a case analysis on the proposer’s pseudocode (Algorithm 3).
Either k is − 1 or it is not (line 11).
First, assume it is not.
In this case, the proposer proposes x , the value proposed in round k (line 12).
We perform a case analysis on round j to show that no value other than x has been or will be chosen in any round j < i .
Case 1: j > k .
We show that no value has been or will be chosen in round j .
Recall that at the end of the Matchmaking phase, the proposer computed the set H i of prior conﬁgurations using responses from a set M i of f + 1 matchmakers.
Either H i contains a conﬁguration C j in round j or it doesn’t.
First, suppose it does.
Then, the proposer sent
P HASE 1A (cid:104) i (cid:105) messages to all of the acceptors in C j .
A Phase 1 quorum of these acceptors, say Q , all received P HASE 1A (cid:104) i (cid:105) messages and replied with P HASE 1B messages.
Thus, every acceptor in Q set its round r to i , and in doing so, promised to never vote in any round less than i .
Moreover, none of the acceptors in Q had voted in any round greater than k .
So, every acceptor in Q has not voted and never will vote in round j .
For a value v (cid:48) to be chosen in round j , it must receive votes from some Phase 2 quorum Q (cid:48) of round j acceptors.
But, Q and Q (cid:48) necessarily intersect, so this is impossible.
Thus, no value has been or will be chosen in round j .
Now suppose that H i does not contain a conﬁguration for round j .
Either a conﬁguration C j was garbage collected from H i
or it wasn’t.
First, assume it wasn’t.
Then, H i is the union of f + 1 M ATCH B messages from the f + 1 matchmakers in M i .
Thus, if H i does not contain a conﬁguration for round j , then none of the M ATCH B messages did either.
This means that for every matchmaker m ∈ M i , when m received M ATCH A (cid:104) i , C i (cid:105) , it did not contain a conﬁguration for round j in its log and never did.
Moreover, by processing the M ATCH A (cid:104) i , C i (cid:105) request and inserting C i in log entry i , the matchmaker is guaranteed to never process a M ATCH A (cid:104) j , C j (cid:105) request in the future.
Thus, every matchmaker in M i has not processed a M ATCH
A request in round j and never will.
For a value to be chosen in round j , the proposer executing round j must ﬁrst receive replies from f + 1 matchmakers, say M j , in round j .
But, M i and M j necessarily intersect, so this is impossible.
Thus, no value has been or will be chosen in round j .
Otherwise, a conﬁguration C j was garbage collected from
H i .
Note that none of the matchmakers in M i had received a G ARBAGE A (cid:104) i (cid:48) (cid:105) command for a round i (cid:48) >
i when they responded with their M ATCH B messages.
If they had, they would have ignored our M ATCH A (cid:104) i , C i (cid:105) message.
Let i (cid:48) be the largest round j < i (cid:48) <
i such that a matchmaker in M i had received a G ARBAGE A (cid:104) i (cid:48) (cid:105) message before responding to our
M ATCH A (cid:104) i , C i (cid:105) message.
If i (cid:48) was garbage collected because of Scenario 1, then k would be at least as large as i (cid:48) since we would have intersected the Phase 2 quorum of C i (cid:48) used in round i (cid:48) to get a value chosen.
But k < j < i (cid:48) , a contradiction.
If i (cid:48) was garbage collected because of Scenario 2, then we know no value has been or will be chosen in round j .
If i (cid:48) was garbage collected because of Scenario 3, then we would have intersected the
Phase 2 quorum of C i (cid:48) that knows a value was already chosen, and we would have not proposed a value in the ﬁrst place.
But, we proposed x , a contradiction.
Case 2: j = k .
In a given round, at most one value is proposed, let alone chosen.
x is the value proposed in round k , so no other value could be chosen in round k .
Case 3: j < k .
By induction, P ( k ) states that no value other than x has been or will be chosen in any round less than k .
This includes round j .
Finally, if k is − 1, then we are in the same situation as in
Case 1.
No value has or will be chosen in a round j < i .
B Matchmaker Reconﬁguration Safety
We repeat the safety proof from Section A.
The new bits are shown in blue.
Proof.
We prove, for every round i , the statement P ( i ) : “if a proposer proposes a value v in round i (i.e. sends a P HASE 2A message for value v in round i ), then no value other than v has been or will be chosen in any round less than i .”
At most one value is ever proposed in a given round, so at most one value is ever chosen in a given round.
Thus, P ( i ) sufﬁces to prove that Matchmaker Paxos is safe for the following reason.
Assume for contradiction that Matchmaker Paxos chooses distinct values x and y in rounds j
and i with j < i .
Some proposer must have proposed y in round i , so P ( i ) ensures us that no value other than y could have been chosen in round j .
But, x was chosen, a contradiction.
We prove P ( i ) by strong induction on i .
P ( 0 ) is vacuous because there are no rounds less than 0.
For the general case P ( i ) , we assume P ( 0 ) ,..., P ( i − 1 ) .
We perform a case analysis on the proposer’s pseudocode (Algorithm 3).
Either k is − 1 or it is not (line 11).
First, assume it is not.
In this case, the proposer proposes x , the value proposed in round k (line 12).
We perform a case analysis on round j to show that no value other than x has been or will be chosen in any round j < i .
Case 1: j > k .
We show that no value has been or will be chosen in round j .
Recall that at the end of the Matchmaking phase, the proposer computed the set H i of prior conﬁgurations using responses from a set M i of f + 1 matchmakers.
Either H i contains a conﬁguration C j in round j or it doesn’t.
First, suppose it does.
Then, the proposer sent
P HASE 1A (cid:104) i (cid:105) messages to all of the acceptors in C j .
A Phase 1 quorum of these acceptors, say Q , all received P HASE 1A (cid:104) i (cid:105) messages and replied with P HASE 1B messages.
Thus, every acceptor in Q set its round r to i , and in doing so, promised to never vote in any round less than i .
Moreover, none of the acceptors in Q had voted in any round greater than k .
So, every acceptor in Q has not voted and never will vote in round j .
For a value v (cid:48) to be chosen in round j , it must receive votes from some Phase 2 quorum Q (cid:48) of round j acceptors.
But, Q and Q (cid:48) necessarily intersect, so this is impossible.
Thus, no value has been or will be chosen in round j .
Now suppose that H i does not contain a conﬁguration for round j .
Either a conﬁguration C j was garbage collected from H i
or it wasn’t.
First, assume it wasn’t.
Then, H i is the union of f + 1 M ATCH B messages from the f + 1 matchmakers in M i .
Thus, if H i does not contain a conﬁguration for round j , then none of the M ATCH B messages did either.
This means that for every matchmaker m ∈ M i , when m received M ATCH A (cid:104) i , C i (cid:105) , it did not contain a conﬁguration for round j in its log and never did.
Moreover, no majority in any previous set of matchmakers contained a conﬁguration in round j .
If any majority did have a conﬁguration in round j , then all subsequent matchmakers would as well since a set of matchmakers is initialized from a majority of the previous matchmakers.
Moreover, by processing the M ATCH A (cid:104) i , C i (cid:105) request and inserting C i in log entry i , the matchmaker is guaranteed to never process a M ATCH A (cid:104) j , C j (cid:105) request in the future.
Moreover, no future set of matchmakers will either.
A majority of matchmakers have a conﬁguration in entry i , so all subsequent conﬁgurations will as well.
Therefore, they will all reject a conﬁguration in round j .
Thus, every matchmaker in M i has not processed a M ATCH
A request in round j and never will.
For a value to be chosen in round j , the proposer executing round j must ﬁrst receive replies from f + 1 matchmakers, say M j , in round j .
But, M i and M j necessarily intersect, so this is impossible.
This argument holds for every set of matchmakers.
Thus, no value has been or will be chosen in round j .
Otherwise, a conﬁguration C j was garbage collected from
H i .
Note that none of the matchmakers in M i had received a G ARBAGE A (cid:104) i (cid:48) (cid:105) command for a round i (cid:48) >
i when they responded with their M ATCH B messages.
If they had, they would have ignored our M ATCH A (cid:104) i , C i (cid:105) message.
Similarly, none of the matchmakers in M i were initialized with a garbage collection watermark w > i .
Let i (cid:48) be the largest round j < i (cid:48) <
i that a matchmaker in M i garbage collected before responding to our M ATCH A (cid:104) i , C i (cid:105) message.
If i (cid:48) was garbage collected because of Scenario 1, then k would be at least as large as i (cid:48) since we would have intersected the Phase 2 quorum of C i (cid:48) used in round i (cid:48) to get a value chosen.
But k < j < i (cid:48) , a contradiction.
If i (cid:48) was garbage collected because of Scenario 2, then we know no value has been or will be chosen in round j .
If i (cid:48) was garbage collected because of Scenario 3, then we would have intersected the
Phase 2 quorum of C i (cid:48) that knows a value was already chosen, and we would have not proposed a value in the ﬁrst place.
But, we proposed x , a contradiction.
Case 2: j = k .
In a given round, at most one value is proposed, let alone chosen.
x is the value proposed in round k , so no other value could be chosen in round k .
Case 3: j < k .
We can apply the inductive hypothesis to get P ( k ) which states that no value other than x has been or will be chosen in any round less than k .
This includes round j , which is exactly what we’re trying to prove.
Finally, if k is − 1, then we are in the same situation as in
Case 1.
i , H 1 i (cid:105) ,..., M ATCH B (cid:104) i , H f + 1 i (cid:105) from f + 1 matchmakers do 5:
H i ← (cid:83) f
+ 1 j = 1 H ji 6: send P HASE 1A (cid:104) i (cid:105) to every acceptor in H i 7: upon receiving P HASE 1B (cid:104) i , − , −(cid:105) from a Phase 1 quorum from every conﬁguration in H i do 8: k ← the largest vr in any P HASE 1B (cid:104) i , vr , vv (cid:105) 9:
V ← the corresponding vv ’s in round k 10: if k = − 1 then 11: send P HASE 2A (cid:104) i , any (cid:105) to every acceptor in C i 12: else if V = { v } then 13: send P HASE 2A (cid:104) i , v (cid:105) to every acceptor in C i 14: else 15: send P HASE 2A (cid:104) i , any (cid:105) to every acceptor in C
i
C Fast Paxos
Fast Paxos proposer pseudocode is given in Algorithm 5.
We do not modify the Fast Paxos acceptor or the matchmakers.
For simplicity, we assume that we deploy Fast Paxos with f + 1 acceptors, with a single unanimous Phase 2 quorum, and with singleton Phase 1 quorums.
Generalizing to arbitrary conﬁgurations that satisfy Fast Paxos’ quorum intersection requirements is straightforward.
Note that Fast Paxos cannot leverage Phase 1 Bypassing.
Also note while both MultiPaxos and our Fast Paxos variant both have quorums of size f + 1, our Fast Paxos variant has a ﬁxed set of f + 1 acceptors, while
MultiPaxos can choose any set of f + 1 acceptors from all 2 f + 1 acceptors.
This has some disadvantages in terms of tail latency and fault tolerance.
We now prove that our modiﬁcations to Fast Paxos are safe.
For simplicity, we ignore garbage collection and matchmaker reconﬁguration.
Introducing those two features and proving them correct is pretty much identical to what we did with
Matchmaker Paxos.
Proof.
We prove, for every round
i , the statement P ( i ) which states that if an acceptor votes for a value v in round
i (i.e. sends a P HASE 2B message for value v in round i ), then no value other than v has been or will be chosen in any round less than i .
P ( i ) sufﬁces to prove that Matchmaker Paxos is safe.
Why?
Well, assume for contradiction that
Matchmaker
Paxos chooses distinct values x and y in rounds i and j with i < j .
Some acceptor must have voted for y in round j , so
P ( j ) ensures us that no value other than y could have been chosen in round i .
But, x was chosen, a contradiction.
We prove P ( i ) by strong induction on i .
P ( 0 ) is vacuous because there are no rounds less than 0.
For the general case P ( i ) , we assume P ( 0 ) ,..., P ( i − 1 ) .
We perform a case analysis on the proposer’s pseudocode.
Either k is − 1 or it is not (line 8).
First, assume it is not.
We perform a case analysis on rounds j < i .
Case 1: j > k .
Recall that at the end of the Matchmaking phase, the proposer computed the set H i of prior conﬁgurations using responses from a set M of f + 1 matchmakers.
Either H i contains a conﬁguration C j in round j or it doesn’t.
First, suppose it does.
Then, the proposer sent
P HASE 1A (cid:104) i (cid:105) messages to all of the acceptors in C j .
A Phase 1 quorum of these acceptors, say Q , all received P HASE 1A (cid:104) i (cid:105) messages and replied with P HASE 1B messages.
Thus, every acceptor in Q set its round r to i , and in doing so, promised to never vote in any round less than i .
Moreover, none of the acceptors in Q had voted in any round greater than k .
So, every acceptor in Q has not voted and never will vote in round j .
For a value v (cid:48) to be chosen in round j , it must receive votes from some Phase 2 quorum Q (cid:48) of round j acceptors.
But, Q and Q (cid:48) necessarily intersect, so this is impossible.
Thus, no value has been or will be chosen in round j .
Now suppose that H i does not contain a conﬁguration for round j .
H i is the union of f + 1 M ATCH B messages from the f + 1 matchmakers in M .
Thus, if H i does not contain a conﬁguration for round j , then none of the M ATCH B messages did either.
This means that for every matchmaker m ∈ M , when m received M ATCH A (cid:104) i , C i (cid:105) , it did not contain a conﬁguration for round j in its log.
Moreover, by processing the M ATCH A (cid:104) i , C i (cid:105) request and inserting C i in log entry i , the matchmaker is guaranteed to never process a M ATCH A (cid:104) j , C j (cid:105) request in the future.
Thus, every matchmaker in M has not processed a M ATCH
A request in round j and never will.
For a value to be chosen in round j , the proposer executing round j must ﬁrst receive replies from f + 1 matchmakers, say M (cid:48) , in round j .
But, M and M (cid:48) necessarily intersect, so this is impossible.
Thus, no value has been or will be chosen in round j .
Case 2: j = k .
If V = { v } , then the proposer proposes v .
We must prove that no value other than v has been or will be chosen in round k .
For a value to be chosen in round k , every acceptor must vote for it in round k .
Some acceptor voted for v in round k , so it is the only value with the possibility of receiving a unanimous vote.
Otherwise V contains multiple distinct elements, and the proposer proposes any .
We must prove that no value has been or will be chosen in round k .
This is immediate since no value can receive a unanimous vote in round k , if two different values have received votes in round k .
Case 3: j < k .
If V = { v } , then the proposer proposes v , and we must prove that no value other than v has been or will be chosen in any round less than k .
This is immediate from
P ( k ) .
Otherwise, V = { v 1 , v 2 ,... } , and the proposer proposes any .
We must prove that no value has been or will be chosen in any round less than k .
P ( k ) tells us that no value other than v 1 has been or will be chosen in any round less than k .
P ( k ) also tells us that no value other than v 2 has been or will be chosen in any round less than k .
Thus, no value has been or will be chosen in any round less than k .
Finally, if k is − 1, then we are in the same situation as in
Case 1.
No value has been or will be chosen in any round less than i .
D DPaxos Bug
Consider a DPaxos deployment with f d = 1, f z = 0, three zones, three nodes per zone, and delegate quorums.
Thus, a replication quorum consists of two nodes in one zone, and a leader election quorum consists of two nodes in two zones.
We name the nodes A through I .
Beside each node, we display its ballot, vote ballot, vote value, and intents [20].
Zone
A
B
C
Zone
D
E
F
Zone
G
H
I − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , /
Proposer 1 initiates the leader election phase in ballotfor value x .
It selects { A , B , D , E } as its leader election quorum and { B , C } as its intent.
It sends prepare messages to the leader election quorum, and the leader election quorum replies.
Proposer 1 doesn’t receive any intents, so it does not expand its leader election quorum.
It also learns that no value has been chosen yet, so it proposes value x to B and C .
Both accept the value.
Zone
A
B
C
Zone
D
E
F
Zone
G
H
I 0 , − 1 , ⊥ , { 0 : { B , C }} 0 , 0 , x , { 0 : { B , C }} 0 , 0 , x , / 0 , − 1 , ⊥ , { 0 : { B , C }} 0 , − 1 , ⊥ , { 0 : { B , C }} − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , / − 1 , − 1 , ⊥ , /
Next, proposer 2 initiates the leader election phase in ballot 1 for value y .
It selects { E , F , H , I } as its leader election quorum and { G , H } as its intent.
It sends prepare messages to the leader election quorum, and the leader election quorum replies.
Proposer 2 receives the intent { B , C } in ballot 0 from
E , so it expands its leader election quorum and sends a prepare message to C .
Proposer 2 learns that value x was chosen in ballot 0, so it ditches y and proposes x to G and H .
G accepts, but the propose message to H is dropped.
Zone
[Table] [Table]
Next, garbage collection is run.
The garbage collector contacts G and sees that it has accepted a value in ballot 1.
It informs all the nodes to discard intents in ballots less than 1.
[Table]
Next, proposer 3 initiates the leader election phase in ballot 2 for value
z It selects { D , E , H , I } as its leader election quorum and { E , F } as its intent.
It sends prepare messages to the leader election quorum, and the leader election quorum replies.
Proposer 3 receives intent { G , H } in ballot 1, but has already included H in its leader election quorum, so it does not send any additional prepares.
It learns that no value has been chosen (this is a bug, x was chosen), so it proposes value z to E and G .
Both accept the value, and z is chosen.
This is a bug since x was already chosen.
[Table]
D
[Table]