Matrix Multiplicative Weights Updates in Quantum Zero-Sum Games: Conservation Laws & Recurrence
Abstract
Recent advances in quantum computing and in particular, the introduction of quantum GANs, have led to increased interest in quantum zero-sum game theory, extending the scope of learning algorithms for classical games into the quantum realm.
In this paper, we focus on learning in quantum zero-sum games under Matrix
Multiplicative Weights Update (a generalization of the multiplicative weights update method) and its continuous analogue, Quantum Replicator Dynamics .
When each player selects their state according to quantum replicator dynamics, we show that the system exhibits conservation laws in a quantum-information theoretic sense.
Moreover, we show that the system exhibits Poincaré recurrence, meaning that almost all orbits return arbitrarily close to their initial conditions inﬁnitely often.
Our analysis generalizes previous results in the case of classical games [48, 42].
Introduction
The Nash equilibrium has always been a central concept in non-cooperative game theory.
While the existence of Nash equilibria is well known in ﬁnite games where mixed strategies are allowed [44], it is less clear how to efﬁciently compute these equilibria themselves.
Indeed, much work revolves around ﬁnding methods to solve for Nash equilibria in different contexts, as well as analyzing the complexity of solving for the Nash [33, 15].
Despite the dominance of equilibrium computation in classical theory, recent works have begun to move towards attempting to understand the nature of learning in games [23, 51].
The family of regret-minimizing algorithms known as Follow-The-Regularized-Leader (FTRL) have been studied extensively, both for discrete time [7, 13, 45, 49] and continuous time [42, 21, 48, 47] dynamics.
These algorithms have found many applications in the domain of machine learning - one such example is that of Generative Adversarial Networks (GANs) [25], which have been successfully modeled using zero-sum game theory.
This rich connection has led to a stronger understanding of adversarial learning and improved practical performance of GANs in various settings [21, 32, 36].
On the other side of the coin, quantum computing is a ﬁeld which has garnered much interest in the machine learning community.
Many quantum machine learning algorithms have been proposed, extending standard classical algorithms.
One such example is quantum GANs [14, 39, 12].
While such works describe the architecture of quantum GANs, we still do not fully understand the behavior of learning algorithms in the quantum setting.
In this paper, we provide an initial analysis of learning in zero-sum quantum games.
We do so by studying the learning behaviour of the ubiquitous
Multiplicative Weights Update (MWU) update rule [37, 22, 5], a speciﬁc instance of the FTRL framework, when applied to such games.
There have been several important results studying MWU and its continuous counterpart, replicator dynamics .
[7] showed that classical MWU does not converge in day-to-day behaviour to the mixed
Nash equilibrium when applied to two player zero-sum games.
[42] showed that the orbits of players in zero-sum games exhibit Poincaré recurrence when using replicator dynamics.
However, quantum systems can oftentimes behave in radically different ways to their classical counterparts.
Indeed, allowing for quantum strategies can create situations wherein players can gain greater payoffs than if they were playing using only classical strategies (see e.g. the Mermin–Peres magic square game [40, 46]).
As such, our extension of the classical results to the quantum realm has to be treated with care.
We focus on a matrix generalization of MWU known as Matrix Multiplicative Weights Update (MMWU), which changes the paradigm of standard MWU from cost vectors to cost matrices, and from probability vectors to density matrices.
This generalization has been independently discovered and studied by [54] as Matrix Exponentiated Gradient Updates and [5] as the Matrix Multiplicative
Weights algorithm.
Applications of the MMWU algorithm include solving semi-deﬁnite programs (SDPs) [6] and obtaining bounds on the sample complexity for learning problems in quantum computing [30].
From a game theoretic standpoint, [28] analyzed the MMWU algorithm for quantum zero-sum games, proving time-average convergence to an approximate Nash equilibrium.
We extend this result by studying the learning behaviour of MMWU in quantum zero-sum games, which could lay the groundwork for developing novel algorithms that achieve convergence to the Nash in day-to-day behaviour in quantum zero-sum games, similar to the case of their classical analogues [16, 41].
Contributions.
In this paper, we ﬁrst study the dynamics of MMWU in quantum zero-sum games, utilizing tools from information theory and classical game theory (Section 3).
We provide bounds on the rate of change of the total quantum relative entropy between a fully mixed Nash equilibrium and the evolving state of the system.
We then study the continuous counterpart of MMWU, which we call quantum replicator dynamics (Section 4).
We show that the aforementioned total quantum relative entropy is a constant of motion.
Furthermore, the dynamics do not converge to equilibrium (in the day-to-day sense) but rather exhibit a weaker form of approximate periodicity known as Poincaré recurrence .
Our proof of this result departs from the standard classical method, representing our main technical novelty.
Finally, we present several simulations which corroborate our theoretical results (Section 5).
Preliminaries and Deﬁnitions 2.1 Quantum Theory
Basic concepts.
Similarly to [28] and in order to make the presentation self-contained, we will start by introducing some basic concepts of quantum information and game theory.
First, we refer to a quantum register as a collection of qubits representing a message that is transferred from one party to another.
We associate a vector space H =
C n with any quantum register.
This intuitively represents the maximum number of distinct classical states that can be stored in the register without error.
The state of a quantum register is represented by a density matrix , which is an n × n positive semi-deﬁnite matrix with trace 1 .
We will use D ( H ) to denote the set of all density matrices associated with a register that is described by H .
One can naturally view such density matrices as linear operators acting on H .
When two registers with associated spaces A = C n and B = C m are considered as a joint register, the associated space is the tensor product
A ⊗ B = C nm .
If the two registers are independently prepared in states described by ρ and σ , then the joint state is described by the nm × nm density matrix ρ ⊗ σ .
Next, for a given vector space H =
C n , we deﬁne L ( H ) as the set of all n × n complex matrices.
Furthermore, we denote the subset of L ( H ) given by Hermitian matrices as Herm ( H ) .
A Hermitian matrix A satisﬁes the equality A =
A † , where A † denotes the adjoint (or conjugate transpose) of matrix A .
Subsequently, we deﬁne Pos ( H ) as the subset of Herm ( H ) which consists of all positive semi-deﬁnite n × n matrices.
Finally, the Hilbert-Schmidt inner product on L ( H ) is deﬁned as (cid:104) A, B (cid:105) =
Tr( A † B ) for all A, B ∈
L ( H ) .
Note that (cid:104) A, B (cid:105) is a real number for any Hermitian matrices A and B , and is also additionally non-negative if A and B are positive semi-deﬁnite.
Measurements and observables.
In the context of game theory, we are also interested in the concepts of quantum measurements and observables .
An observable is simply a property of the quantum system which is measurable.
The measurement of a register having associated vector space H =
C n is a collection of linear operators { P i : 1 ≤ i ≤ k } ⊂ Pos ( H ) which satisﬁes (cid:80)
ki
=1 P i = 1 H , where 1 H is the identity matrix on H .
If the register corresponding to H is in a state deﬁned by density matrix ρ and the measurement described by P i is performed, each outcome i will be observed with probability (cid:104) P i , ρ (cid:105) .
An important note is that two mixed states with the same density matrix are indistinguishable from each other by any measurement.
Additional notation and deﬁnitions.
A linear mapping of the form Φ : L( B ) → L( A ) is called a super-operator.
The adjoint super-operator to Φ is Φ † : L( A ) →
L( B ) and is uniquely determined by the condition:
[Equation]
A super-operator Φ : L( B ) → L( A ) is said to be positive if Φ( P ) is positive semi-deﬁnite for every choice of positive semi-deﬁnite operator P ∈ Pos( B ) .
In addition, Φ † is positive if and only if Φ is positive.
There is a one-to-one and linear correspondence between the collection of operators of the form R ∈ L( A ⊗ B ) and the collection of super-operators Φ : L( B ) → L( A ) deﬁned above.
Speciﬁcally, for each super-operator Φ we can deﬁne an operator R as follows:
[Equation]
where E i,j is the matrix with a 1 in entry ( i, j ) and 0 elsewhere.
Conversely, given an operator
R ∈ L( A ⊗ B ) , we can deﬁne:
[Equation]
This correspondence is linear and one can translate back and forth between the two as needed for an given application.
R is assumed to be positive semi-deﬁnite, so the corresponding super-operator Φ is also positive.
2.2 Quantum Game Theory
A quantum system which can be manipulated by any number of agents, and where the utility of the moves is well deﬁned, quantiﬁed and ordered can be conceived as a quantum game.
In particular, a 2-player zero-sum quantum game sees players Alice and Bob each sending a quantum state to a referee, who then performs a measurement on these two states to determine their payoffs.
Let
A = C n and B = C
m be the vector spaces that correspond to the state that Alice and Bob send to the referee.
In order to determine the payoffs of the players’ actions, the referee performs a joint measurement where Alice’s and Bob’s states are viewed as a single register.
Thus, the referee’s measurement can be described by a collection
[Equation]
which satisﬁes the condition (cid:80) ka =1 R a = 1 A⊗B .
We associate each possible measurement outcome a with a payoff for each player.
Since we are only considering zero-sum games, if the payoff that Alice receives from the referee is v ( a ) , then Bob’s corresponding payoff will be − v ( a ) .
Henceforth, we refer to the states sent by Alice and Bob to the referee as ρ and σ respectively.
For a given choice of ρ and σ , Alice’s expected payoff is:
[Equation]
where R = (cid:80) ka
=1 v ( a ) R a .
Likewise, Bob’s corresponding expected payoff is − u ( ρ, σ )
= −(cid:104) R, ρ ⊗ σ (cid:105) .
R is referred to throughout the rest of the paper as a payoff observable .
A necessary and sufﬁcient condition for matrix R acting on A ⊗ B to be obtained from some real valued payoff function v is that R is Hermitian.
From Equation 2, we can equivalently deﬁne the expected payoff of Alice as u ( ρ, σ ) =
(cid:104) ρ, Φ( σ ) (cid:105) .
We will use the latter formulation for the remainder of the paper.
A key notion of equilibrium in classical game theory is the Nash equilibrium.
In the quantum setting, we deﬁne the pair of quantum states ( ρ ∗ , σ ∗ ) as a Nash equilibrium of R if u ( ρ ∗ , σ ∗ ) ≥ u ( ρ, σ ∗ ) and u ( ρ ∗ , σ ∗ ) ≥ u ( ρ ∗ , σ ) (5) for all ρ, σ .
That is, neither Alice nor Bob would prefer to unilaterally deviate from playing ρ ∗ and σ ∗ respectively.
On another note, since the set of available strategies (density matrices) for both agents is convex and compact and the utility function of Alice is bilinear, standard extensions of the of von
Neumann’s Min-Max Theorem apply [20].
Finally, we deﬁne the notion of a ‘fully mixed’ Nash equilibrium.
An equilibrium ( ρ ∗ , σ ∗ ) is fully mixed if ρ ∗ , σ ∗ are full rank.
2.3 Information Theory
We also introduce several information theoretic concepts which will be referenced throughout the paper.
Shannon entropy.
The Shannon entropy of a random variable X where each strategy x is obtained with probability p ( x ) is given by H ( X )
= − (cid:80) x p ( x ) log p ( x ) .
This intuitively is a measure of randomness or uncertainty in the system.
A natural generalization of the Shannon entropy to a quantum context is the von Neumann entropy.
For a quantum mechanical system deﬁned by density matrix ρ , the von Neumann entropy is given by S ( ρ )
= − Tr( ρ log ρ ) .
Bregman divergence.
We are also interested in the notion of Bregman divergence, which measures the distance between two points.
Let x ∗ ∈
X be a Nash equilibrium and let x ∈
X be an arbitrary strategy proﬁle.
Also, let F be a continuously-differentiable, strictly convex function.
The Bregman divergence from x ∗ to x is given by D F ( x ∗ (cid:107) x ) =
F ( x ∗ ) − F ( x ) − (cid:104)∇ F ( x ) , x ∗ − x (cid:105) .
In the context of quantum games, the Bregman divergence is deﬁned using matrix notation.
We deﬁne the quantum relative entropy between two quantum states using the von Neumann entropy
S ( ρ ) .
In particular, the quantum relative entropy between two quantum states ρ and σ is deﬁned as
S ( ρ (cid:107) σ ) =
Tr ( ρ (log ρ − log σ )) .
2.4 Dynamical Systems
Flows.
Consider a differential equation ˙ p = f ( p ) on a topological space P .
The existence and uniqueness theorem for ordinary differential equations guarantees that we can write the unique solution to the differential equation as a continuous map φ : P × H → P .
This is referred to as the ﬂow of the differential equation such that for any point p ∈ P , φ ( p, − ) deﬁnes a function of time corresponding to the trajectory of p .
Conversely, ﬁxing a time t provides a map φ t ≡ φ ( − , t ) :
P → P .
In Section 4, we introduce the notion of quantum replicator dynamics, which are Lipschitz continuous differential equations.
Hence, a unique ﬂow φ of these replicator dynamics exists.
Liouville’s theorem.
Liouville’s theorem can be applied to any system of ordinary differential equations with a continuously differentiable vector ﬁeld ξ on an open domain Y ∈ R d .
The divergence of ξ at y ∈ Y is the trace of the Jacobian at y : div ξ ( y ) =
(cid:80) di
=1 ∂ξ
i ∂y i ( y ) .
Because the divergence is continuous, it is integrable on Lebesgue measurable subsets of Y .
Given any such set C, deﬁne the image of C under ﬂow φ at time t as C ( t ) = { φ ( c, t ) : c ∈ C } .
C ( t ) is measurable and of volume vol [ C ( t )]
= (cid:82) C ( t ) dµ .
Liouville’s formula states that the time derivative of the volume vol [ C ( t )] exists and links it to the divergence of ξ :
[Equation]
If div ξ ( y ) is null at any y ∈ Y , then the volume is conserved.
Since div ξ is continuous, the converse statement is also true - if the volume is conserved on any open set, div ξ ( y ) has to be null at any point y ∈ Y .
Let µ = (cid:15)/ 8 and let N = (cid:100) 64 log ( nm ) /(cid:15) 2 (cid:101) .
Initialize : A 0 = 1 A , ρ
=
A 0 / Tr( A 0 ) , B 0 = 1 B , and σ 0 = B 0 /
Tr( B 0 ) .
for j = 1 . . .
N do
[Equation]
end for
Diffeomorphisms of ﬂows.
A function f between two topological spaces is called a diffeomorphism if i) f is a bijection, ii) f is continuously differentiable, iii) f has a continuously differentiable inverse.
Two ﬂows Φ t : A → A and Ψ t : B → B are diffeomorphic if there exists a diffeomorphism f : A → B such that for each x ∈ A and t ∈ R , f (Φ t ( p )) =
Ψ t ( f ( p )) .
For the purpose of our analysis, the replicator dynamics deﬁned in Equations 17 are translated via a diffeomorphism from the interior of P to a space C
=
Π
i ∈ V R n − 1 , which allows us to show certain desirable properties.
Poincaré recurrence.
The concept of Poincaré recurrence arises from Henri Poincaré’s celebrated 1890 work regarding the three body problem [50].
He proved that if a dynamical system preserves volume and always remains bounded in its orbits, almost all trajectories return arbitrarily close to their initial position, and do so inﬁnitely often.
Theorem 2.1 (Poincaré recurrence) .
If a ﬂow preserves volume and has only bounded orbits then for each open set there exist orbits that intersect the set inﬁnitely often.
MMWU in Quantum Zero-Sum Games
In [28], the MMWU algorithm for zero-sum games is shown to exhibit time-average convergence to an approximate Nash equilibrium in two-player quantum zero-sum games.
The MMWU algorithm is shown in Algorithm 1.
Note that here we focus speciﬁcally on two-player games, and utilize the expected payoffs deﬁned via super-operators Φ : L( A )
→ L( B ) as seen in Equation 2.
Moreover, µ is the step-size in the quantum algorithm.
In this section, we examine closely the update steps for each player in the MMWU
algorithm
(Algorithm 1) and analyze the limiting behaviour of the total quantum relative entropy in the system.
First, we introduce two useful facts which will aid in the analysis.
Fact 3.1 (Golden-Thompson inequality [24, 53]) .
Let A, B be Hermitian matrices.
Then
[Equation]
Fact 3.2 .
Let 0 ≤ A ≤ 1 be a PSD matrix and δ be a real number.
Then,
[Equation]
Next, we put forward two corollaries which will help us prove Theorem 3.5.
We ﬁrst deﬁne
[Equation]
Corollary 3.3.
The change in the sum of quantum relative entropies in a quantum zero-sum game between a fully-mixed Nash equilibrium and the players’ strategies is given by:
[Equation]
Corollary 3.4.
The following trace inequalities hold for PSD matrices A and B updated with
MMWU:
[Equation]
The proof of Corollary 3.4 relies on Facts 3.1 and 3.2, as well as the equality shown in Corollary 3.3.
In many practical scenarios, one would use decreasing step-sizes when running MMWU.
As such, we utilize Corollary 3.4 and take the limit as step-size µ goes to 0 in order to show the following theorem:
Theorem 3.5.
The change in the sum of quantum relative entropies between a fully mixed Nash equilibrium and the player’s strategies and in a two-player zero-sum quantum game tends to zero when step-size µ → 0 .
Speciﬁcally,
[Equation]
The proof of Theorem 3.5, along with Corollaries 3.3 and 3.4 can be found in Appendix C.
Theorem 3.5 further motivates an investigation into the continuous time variant of MMWU, which we call quantum/matrix replicator dynamics.
In particular, we show that in the continuous case, the sum of quantum relative entropies is invariant.
We introduce and study quantum replicator dynamics in detail in Section 4.
Replicator Dynamics in Quantum Zero-Sum Games
We have seen that in the case of discrete dynamics (MMWU), as the step-size becomes inﬁnitesimal, the total quantum relative entropy between the Nash equilibirum and the system state tends to stabilize.
A natural question would then be: does the same result hold in continuous time?
In order to explore this question in greater detail, we ﬁrst need to deﬁne the continuous analogue of MMWU, the quantum replicator dynamics , which has well known classical analogues [51].
We start by rewriting the MMWU update steps, but now deﬁned over a continuous time interval [0 , t ] : [Equation] [Equation]
[Equation]
Note here that we shift the exponential terms from the deﬁnition of A ( t ) and B ( t ) to the corresponding ρ ( t ) and σ ( t ) terms.
This will help simplify some of the proof techniques later on in the paper.
Furthermore, in the rest of the paper we will typically drop from the notation the explicit dependence on t to ease with the notational burden.
It is important to note the following observation, which will be helpful in our later analysis.
Observation 4.1 .
The discrete-time trajectories ρ j and σ j deﬁned in Algorithm 1 are a standard Euler discretization (with step µ ) of the continuous-time trajectories ρ ( t ) and σ ( t ) deﬁned in Equationsand 16.
We now deﬁne the quantum replicator dynamics as:
[Equation]
It is worth noting that in the classical/commuting setting, one can write the replicator equations in a form that describes the relative utility that one agent obtains as compared to the average utility overall.
However, in the quantum case this is not possible in general, since it relies on the assumption that (cid:82) t 0 Φ( σ ( τ )) dτ and Φ( σ ( t )) , and respectively (cid:82) t
Φ † ( ρ ( τ )) dτ and Φ † ( ρ ( t )) commute.
As a consequence of Observation 4.1, we can also conclude that the dynamical system described by the replicator dynamics deﬁned in Equations 17 is a limit case of the dynamical system described by the MMWU algorithm as µ → 0 .
We are now able to state the main theorem for quantum relative entropy in quantum replicator dynamics.
Theorem 4.2.
When applying matrix/quantum replicator dynamics in a quantum zero-sum game with a fully-mixed Nash equilibrium ( ρ ∗ , σ ∗ ) , the sum of quantum relative entropies between the fully-mixed Nash equilibrium and the state of the system ( ρ ( t ) , σ ( t )) is invariant on every system trajectory, i.e.:
[Equation]
4.1 Poincaré Recurrence in Quantum Zero-Sum Games
Now that we have described analytical results surrounding the day-to-day behaviour of quantum replicator dynamics, we seek to understand the dynamics of the trajectories.
After all, invariance of quantum relative entropy does not fully describe how the system moves over time.
We show that for any two-player zero-sum quantum game updated with replicator dynamics, the system exhibits
Poincaré recurrence , insofar as the game is zero-sum and has a fully-mixed Nash equilibria.
As introduced in Section 2, the notion of Poincaré recurrence is a weaker version of periodicity.
To be precise, for almost all initial conditions ρ 0 ∈ P , the replicator dynamics return arbitrarily close to ρinﬁnitely often.
Theorem 4.3.
The quantum replicator dynamics given in Equations 17 are Poincaré recurrent in any two player zero-sum game which has a fully-mixed Nash equilibrium.
The proof of this main theorem involves carefully piecing together several auxiliary results, which we will describe in the rest of the section.
Furthermore, we stress that due to the non-commutative nature of quantum systems, the standard (classical) approach of differentiating the discrete-time dynamics in the primal space of probability distributions does not apply directly unless we have the highly unlikely situation where (cid:82) t
Φ( σ ( τ )) dτ and Φ( σ ( t ))
(resp.
(cid:82) t
Φ † ( ρ ( τ )) dτ and Φ † ( ρ ( t )) ) commute.
This problem with carrying over the standard approach is explicitly discussed in Appendix D.
For the proof in the quantum setting, we ﬁrst deﬁne a canonical transformation on the space of the matrices A ( t ) and B ( t ) , which will be crucial in proving the theorem.
Deﬁnition 4.4 (Canonical transformation) .
We deﬁne the canonical transformation of A (cid:48) ( t ) and
B (cid:48) ( t ) to be a mapping of A ( t ) and B ( t ) as deﬁned by Equations 13 and 15.
In particular, we deﬁne
[Equation]
where v is a ﬁxed vector deﬁned as v = [1 , 0 . . .
0] (cid:62) , such that the values of v † A ( t ) v and v † B ( t ) v are real numbers corresponding to the (1 , 1) -th element of matrices A ( t ) and B ( t ) for all t .
Notice that this creates matrices A (cid:48) ( t ) and B (cid:48) ( t ) which have 0 as the (1 , 1) -th entry.
Under the transformation in Deﬁnition 4.4, the vector ﬁelds ˙ A (cid:48) ( t ) =
F ( A (cid:48) ) and ˙ B (cid:48) ( t ) =
F ( B (cid:48) ) are given by:
[Equation]
(20) where ddt (cid:0) v † A ( t ) v (cid:1) is given by v † dA ( t ) dt v .
Moreover, the values of ρ (cid:48) ( t ) and σ (cid:48) ( t ) are deﬁned as:
[Equation]
Proposition 4.5.
The dynamics of ρ ( t ) and σ ( t ) remain the same after undergoing the canonical transformation.
Equivalently, A (cid:48) ( t ) and A ( t ) ( resp.
B (cid:48) ( t ) and B ( t )) admit the same strategy ρ ( t ) ( resp.
σ ( t )) .
Proposition 4.6.
The mappings A (cid:48) ( t ) and ρ ( t ) ( resp.
B (cid:48) ( t ) and σ ( t )) are diffeomorphic to one another.
Proposition 4.6 will be of crucial importance to our proof technique, since we ﬁrst prove recurrence for the system described by A (cid:48) ( t ) and B (cid:48) ( t ) , then recover recurrence in ρ ( t ) and σ ( t ) .
To show Poincaré recurrence of Equations 17, we ﬁrst prove two key properties: (i) the ﬂow of ˙
A (cid:48) is volume preserving, meaning that the trace of the Jacobian of the respective vector ﬁelds ˙
A (cid:48) ( t ) =
F ( A (cid:48) ) and ˙ B (cid:48) ( t ) =
F ( B (cid:48) ) are zero, and (ii) A (cid:48) and B (cid:48) have bounded orbits from any interior initial condition.
Then, Poincaré recurrence of A (cid:48) and B (cid:48) follows from Poincaré’s recurrence theorem.
Volume Conservation.
We introduce a lemma which shows that in two-player zero-sum quantum replicator dynamics, the canonical transformation produces a dynamical system which preserves volume.
Lemma 4.7.
For two-player zero-sum quantum replicator dynamics, the vector ﬁelds that arise as a result of the canonical transformation in Deﬁnition 4.4 are volume preserving.
The proof of Lemma 4.7 follows from considering the ﬂows of A (cid:48)
( t ) and B (cid:48) ( t ) and showing that the divergences of the vector ﬁelds deﬁned in Equations 20 are equal to zero.
A straightforward application of Liouville’s theorem completes the proof.
Bounded Orbits.
We now show that the transformed dynamical system always has bounded orbits when initialized on the interior of the space of probability density matrices.
Lemma 4.8.
For any ﬁnite initial points A (0) and B (0) , the dynamics mapped to A (cid:48) ( t ) and B (cid:48) ( t ) via the transformation in Deﬁnition 4.4 have bounded orbits.
The proof of Lemma 4.8 relies on Theorem 4.2, and leverages the hermicity of the matrices involved.
Moreover, we use the fact that the canonically transformed matrices A (cid:48) ( t ) and B (cid:48) ( t ) have zero as the (1 , 1) -th element to bound the eigenvalues of A (cid:48) ( t ) and B (cid:48) ( t ) away from inﬁnity.
Now we are ready to prove Theorem 4.3 using Lemmas 4.7 and 4.8.
Proof of Theorem 4.3.
By Lemmas 4.7 and 4.8, as well as the Poincaré recurrence theorem introduced in Section 2, we immediately see that the system of replicator equations given by dA (cid:48) /dt and dB (cid:48)
/dt are Poincaré recurrent since they are volume preserving and have bounded orbits.
Since the ﬂows of
A (cid:48) ( t ) and ρ ( t ) are diffeomorphic to one another (likewise for B (cid:48)
( t ) and σ ( t ) ), dρ/dt and dσ/dt are also Poincaré recurrent.
This concludes the proof.
All proofs of the results in this section are provided in Appendix D.
Experimental Results
[Equation] To corroborate the theoretical results presented in prior sections, we performed relevant simulations of quantum games using both discrete MMWU and replicator dynamics.
In the rest of this section, we standardize the use of quantum game matrices obtained via basis transform (described in more detail in Appendix E).
This effectively allows us to transform classical games to the matrix setting.
First, we show the trajectories of the ﬁrst eigenvalue of each player in a quantum Matching Pennies game, obtained using the discrete MMWU algorithm.
We see that in accordance to Theorem 3.5, the rate of divergence of the trajectories from the uniform Nash goes to zero for cases with rapidly decreasing learning rate µ .
In the case of replicator dynamics, we present Bloch sphere representations of the trajectories in a quantum Matching Pennies game.
The Bloch sphere is a unit 2-sphere representation of a qubit, and we utilize it to visualize the orbits of the replicator dynamics.
In particular, the density matrix representing the strategy of each player at each time-step is given as a point within the sphere, and we plot the movement of these orbits over time.
According to Theorems 4.2 and 4.3, we expect the trajectories of the replicator dynamics to stay on the interior of the Bloch sphere, since the surface of the sphere corresponds to the pure states of the system.
We see from Figure 2 that over time, the system never reaches the boundary of the sphere, which experimentally agrees with our theory.
[Equation]
Finally, in the experiments above we have only studied single qubit systems.
We provide further experiments which show that our results hold beyond the single qubit setting in Appendix E.2.
Conclusion
In this paper, we studied the properties of Matrix Multiplicative Weights Update and its continuous analogue, quantum replicator dynamics, in the context of two-player zero-sum quantum games.
First, we provide a formulation of quantum replicator dynamics which arises from MMWU.
Then, we show that such systems exhibit quantum information-theoretic constant of motions.
Finally, we show that in quantum replicator systems with interior Nash equilibria, the dynamics exhibit Poincaré recurrence.
This work constitutes an initial step towards analyzing learning behaviour in games with quantum information.
In the classical world, showing that conservation laws and recurrence holds has led to a better understanding of game dynamics in increasingly complex settings [52, 43, 47].
Similar work is now potentially possible in the quantum setting.
Moreover, one key implication which can be derived from our recurrence results is to encourage novel discretization methods that preserve the connections to volume preservation and more generally, conservative dynamical systems (e.g. Hamiltonians).
This line of research has received increased interest in recent years [57, 56, 17].
It is a natural question to explore whether such techniques can lead to similar advantages in the quantum setting as well.
Finally, some other interesting directions for future work include:
• extending our results to multi-agent network generalizations of quantum zero-sum games, • understanding the potential behavior of quantum GANs using our results about learning in quantum zero-sum games, and • understanding the quantum setting for different classes of games, e.g., potential games.
Acknowledgements
This research/project is supported by the National Research Foundation, Singapore and DSO National
Laboratories under the AI Singapore Programme (AISG Award
No: AISG2-RP-2020-016), NRF2019NRFANR095 ALIAS grant, grant PIE-SGP-AI-2020-01, NRF 2018 Fellowship NRF-NRFF2018-  and AME Programmatic Fund (Grant No. A20H6b0151) from the Agency for Science, Technology and Research (A*STAR).
Rahul Jain’s research is supported by the National Research Foundation,
Singapore, also through the grant NRF2021-QEP2-02-P05, and the Ministry of Education, Singapore under the Research Centers of Excellence program.
Ryann Sim gratefully acknowledges support from the SUTD President’s Graduate Fellowship (SUTD-PGF).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] The contributions and scope are clearly stated and derived directly from our results.
(b) Did you describe the limitations of your work?
[Yes] The limitations of our work are described in the introduction and in the main text.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] See Section 6. 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] All assumptions are stated either in the main body or the appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are provided in the main text or appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code is compiled in several formats and provided in the supplementary material.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[No] Our simulations do not require training.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Our simulations are deterministic.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Details of the resources and packages used are described in the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the code packages used.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A]
Our work does not utilize human-derived data.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
Our work does not include crowdsourcing or research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]