Policy Optimization for Markov Games: Unified Framework and Faster Convergence
Abstract
This paper studies policy optimization algorithms for multi-agent reinforcement learning.
We begin by proposing an algorithm framework for two-player zero-sum
Markov Games in the full-information setting, where each iteration consists of a policy update step at each state using a certain matrix game algorithm, and a value update step with a certain learning rate.
This framework unifies many existing and new policy optimization algorithms.
We show that the state-wise average policy of this algorithm converges to an approximate Nash equilibrium (NE) of the game, as long as the matrix game algorithms achieve low weighted regret at each state, with respect to weights determined by the speed of the value updates.
Next, we show that this framework instantiated with the Optimistic Follow-The-Regularized-Leader (OFTRL)
algorithm at each state (and smooth value updates) can find an (cid:101)
O
( T − 5 / 6 ) approximate NE in T iterations, and a similar algorithm with slightly modified value update rule achieves a faster (cid:101)
O ( T − 1 ) convergence rate.
These improve over the current best (cid:101)
O ( T − 1 / 2 ) rate of symmetric policy optimization type algorithms.
We also extend this algorithm to multi-player general-sum Markov Games and show an (cid:101)
O ( T − 3 / 4 ) convergence rate to Coarse Correlated Equilibria (CCE).
Finally, we provide a numerical example to verify our theory and investigate the importance of smooth value updates, and find that using “eager” value updates instead (equivalent to the independent natural policy gradient algorithm) may significantly slow down the convergence, even on a simple game with H = 2 layers.
Introduction
Policy optimization, i.e. algorithms that learn to make sequential decisions by local search on the agent’s policy directly, is a widely used class of algorithms in reinforcement learning [40, 44, 45].
Policy optimization algorithms are particularly advantageous in the multi-agent reinforcement learning (MARL) setting (e.g. compared with value-based counterparts), due to their typically lower representational cost and better scalability in both training and execution.
A variety of policy optimization algorithms such as Independent PPO [14], MAPPO [56], QMix [42] have been proposed to solve real-world MARL problems [4, 39, 43].
These algorithms share a same high-level structure with iterative value updates (for certain value estimates) and policy updates (often independently with each agent) using information from the value estimates and/or true rewards.
While policy optimization for MARL has been studied theoretically in a growing body of work, there are still gaps between algorithms used in practice and provably-efficient algorithms studied in theory—Algorithms in practice generally follow two natural design principles: symmetric updates among all agents, and simultaneous learning of values and policies [59, 56].
By contrast, policy optimization algorithms studied in theory often diverge from these principles and incorporate some tweaks, such as (i) asymmetric updates, where one agent takes a much smaller learning rate than the others (two time-scale) [11] or waits until the other agents learn an approximate best response [62]; and (ii) batch-like learning, where policies are optimized to sufficient precision with respect to the current value estimate before the next value update [8].
There is so far a lacking of systematic studies on the performance of the more vanilla policy optimization algorithms following the above two principles, even under the setting where full-information feedback from the game is available.
Towards bridging these gaps, this paper studies policy optimization algorithms for Markov games, with a focus on algorithms with symmetric updates and simultaneous learning of values and policies.
Our contributions can be summarized as follows:
• We propose an algorithm framework for two-player zero-sum Markov games in the fullinformation setting (Section 3).
This framework unifies many existing and new policy optimization algorithms such as Nash V-Learning, Gradient Descent/Ascent, as well as seemingly disparate algorithms such as Nash Q-Learning (Section 3.2).
We prove that the state-wise average policy outputted by the above algorithm is an approximate Nash Equilibrium (NE), so long as suitable per-state weighted regrets are bounded (Section 3.1).
This generic result can be instantiated in a modular fashion to derive convergence guarantees for the many examples above.
• We instantiate our framework to show that a new algorithm based on Optimistic Follow-TheRegularized-Leader (OFTRL) and smooth value updates finds an (cid:101)
O ( T − 5 / 6 ) approximate NE in T iterations (Section 4).
This improves over the current best rate of (cid:101)
O
( T − 1 / 2 ) achieved by symmetric policy optimization type algorithms.
In addition, we also propose a slightly modified OFTRL algorithm that further improves the rate to (cid:101)
O ( T − 1 ) , which matches with the known best rate for all policy optimization type algorithm.
• We additionally extend the above OFTRL algorithm to multi-player general-sum Markov games and show an (cid:101)
O
( T − 3 / 4 ) convergence rate to Coarse Correlated Equilibria (CCE), which is also the first rate faster than (cid:101) O
( T − 1 / 2 ) for policy optimization in general-sum Markov games (Section 4.1).
• We perform simulations on a carefully constructed zero-sum Markov game with H = 2 layers to verify our convergence guarantees.
The numerical tests further suggest the importance of smooth value updates : the Independent Natural Policy Gradient algorithm (as one instantiation of our algorithm framework with “eager” value updates) appears to converge much slower (Section 5)
1.1 Related work
Two-player zero-sum MGs Markov games (MGs) [30] (also known as Stochastic Games [47]) is a widely studied model for multi-agent reinforcement learning.
In the most basic setting of twoplayer zero-sum MGs, algorithms for computing the NE have been extensively studied in both the full-information setting [31, 20, 19] and the sample-based/online setting [6, 52, 22, 48, 58, 2, 55, 3, 33, 25, 21, 57, 10, 34].
Our algorithm framework incorporates (the full-information version of) several algorithms in this line of work.
Policy optimization for zero-sum MGs Policy optimization for single-agent Markov Decision
Processes has been extensively in a recent line of work, e.g. [1, 5, 32, 46, 37, 35, 7, 15, 54] and the many references therein.
For two-player zero-sum MGs, the Nash V-Learning algorithm of Bai et al.
[3] (originally proposed for the sample-based online setting) can be viewed as an independent policy optimization algorithm, and can be adapted to the full-information setting with (cid:101)
O ( T − 1 / 2 ) convergence rate.
Daskalakis et al.
[11] prove that the independent policy gradient algorithm with an asymmetric two time-scale learning rate can learn the NE (for one player only) with polynomial iteration/sample complexity.
Zhao et al.
[62] show that another asymmetric algorithm that simulates a policy gradient/best response dynamics converges to NE with (cid:101) O (1 /T ) rate.
Cen et al.
[8] use a symmetric optimistic (extragradient) subroutine for matrix games to learn zero-sum MGs in a layer-wise fashion (the matrix games at each state are learned to sufficient precision before the backup, more like a Value Iteration type algorithm), and also derive an (cid:101)
O (1 /T ) convergence rate.
The closest to our work is Wei et al.
[53] which proves that Optimistic Gradient Descent/Ascent (OGDA), combined with smooth value updates at all layers simultaneously, converges to an NE with (cid:101)
O ( T − 1 / 2 ) rate for both the average duality gap and the last iterate.
The (cid:101)
O
( T − 5 / 6 ) rate of our OFTRL algorithm improves over [53] and is the first such faster rate for symmetric, policy optimization type algorithms.
The (cid:101) O
(1 /T ) rate of our modified OFTRL algorithm matches with rates in [62, 7], while still maintaining symmetric update and simultaneous learning of values and policies.
Multi-player general-sum MGs A recent line of work shows that a generalization of the Vlearning algorithm to the multi-player general-sum setting can learn Coarse Correlated Equilibria (CCE) [49, 24, 36] and Correlated Equilibria (CE) [49, 24].
The algorithmic designs in these works are specially tailored to the sample-based setting, where the best possible rate is (cid:101) O
( T − 1 / 2 ) .
In contrast, this paper considers the full-information setting and proposes new algorithms achieving the faster (cid:101)
O
( T − 3 / 4 ) for learning CCE in general-sum MGs.
The complexity of learning a structured
CCE (such as Markovian or stationary Markovian) has been studed in [13, 26].
Another recent line of work considers learning NE in Markov Potential Games [60, 29, 49, 16, 61], which can be seen as a cooperative-type subclass of general-sum MGs.
Optimistic algorithms in normal-form games Technically, our accelerated rates build on the recent line of work on faster rates for optimistic no-regret algorithms in normal-form games [50, 41, 9, 12].
Specifically, our ˜ O ( T − 5 / 6 ) rate for two-player zero-sum MGs builds upon a first-order smoothness analysis of [9], our improved (cid:101) O
( T − 1 ) rate in the same setting (achieved by the modified
OFTRL algorithm)
leverages analysis of [41, 50] on bounding the summed regret over the two players, and our ˜ O ( T − 3 / 4 ) rate for multi-player general-sum MGs follows from the RVU-property [50,
Definition 3].
Our incorporation of these techniques involves non-trivial new components such as weighted first-order smoothness bounds and handling changing game rewards.
Preliminaries
We consider the tabular episodic (finite-horizon) two-player-zero-sum Markov games (MGs), which can be denoted as M ( H, S , A , B , P , r ) , where H is the horizon length; S is the state space with |S| =
S ; A , B are the action space of the max-player and min-player respectively, with |A| =
A, |B| =
B ; P = { P h } Hh =1 is the transition probabilities, where each P h ( s ′ | s, a, b ) gives the probability of transition to state s ′ from state-action ( s, a, b ) ; r = { r h } Hh =1 are the reward functions, such that r h ( s, a, b ) is reward 3 of the max-player and − r h ( s, a, b ) is the reward of the min-player at time step h and state-action ( s, a, b ) .
In each episode, the MG starts with a deterministic initial state s 1 .
Then at each time step 1 ≤ h ≤ H , both players observes the state s h , the max-player takes an action a h ∈ A , and the min-player takes an action b h ∈ B .
Then, both players receive their rewards r h
( s h , a h , b h ) and − r h ( s h , a h , b h ) , respectively, and the system transits to the next state s t +1 ∼ P h ( ·| s h , a h , b h ) .
Policies & value functions
A (Markov) policy µ of the max-player is a collection of policies µ = { µ h : S → ∆ A } Hh =1 , where each µ h ( ·| s h ) ∈ ∆
A specifies the probability of taking action a h at ( h, s h ) .
Similarly, a (Markov) policy ν of the min-player is defined as ν
= {
ν h : S → ∆ B } .
For any policy ( µ, ν ) (not necessarily Markov), we use V µ,νh : S → R and Q µ,νh : S ×
A × B → R to denote the value function and Q-function at time step h , respectively, i.e.
[Equation]
For notational simplicity, we use the following abbreviation: [ P h V ]( s, a, b ) :
=
E s ′ ∼ P h ( ·| s,a,b ) V ( s ′ ) for any value function V .
By definition of the value functions and Q-functions, we have the following
Bellman equations
[Equation]
[Equation]
[Equation]
The goal for the max-player is to maximize the value function, whereas the goal for the min-player is to minimize the value function.
Best response & Nash equilibrium For any Markov policy µ of a max-player, there exists a best response for the min-player, which can be taken as a Markov policy ν † ( µ ) such that V µ,ν † ( µ ) h ( s )
=
inf ν V µ,νh ( s ) for all ( s, h ) ∈ S
×
[ H ] .
For simplicity we define V µ, † h
[Equation]
By symmetry, we can also define µ † ( ν ) and V † ,νh .
It is known (e.g. [17]) that there exist Markov policies ( µ ⋆ , ν ⋆ ) that perform optimally against best responses.
These policies are also equivalent to Nash Equilibria (NEs) of the game, where no player can gain by switching to a different policy unilaterally.
It can also be shown that any NE ( µ ⋆ , ν ⋆ ) satisfies the following minimax equation
[Equation]
Thus, while the NE policy ( µ ⋆ , ν ⋆ ) may not be unique, all of them share the same value functions, which we denote as V ⋆h :=
V µ ⋆ ,ν ⋆ h .
The Q-function Q ⋆h can be defined similarly.
In this paper, our main goal is to find an approximate NE, which is formally defined below.
Definition 1 ( ε -approximate Nash Equilibrium) .
For any ε ≥ 0 , a policy ( µ, ν ) is an ε -approximate
Nash Equilibrium ( ε -NE) if NEGap( µ, ν ) :
= V † ,ν 1 ( s 1 ) −
[Equation]
µ, † 1 ( s 1 ) ≤ ε.
Full-information setting We consider finding an approximate NE under the full-information setting, where we can query the exact value of r h
+ P h V h +1 ∈ R
S×A×B for any layer h ∈
[ H ] and
V function V h +1 ∈ R S .
Further, the majority of algorithms are policy optimization algorithms
that only query ( r h + P h
V
h
+1 )
ν h ∈ R S×A
and ( r h + P h V h +1 )
⊤ µ h ∈ R S×B for policies µ h , ν h .
(See
Appendix I.1 for additional discussions.)
Additional notation For any ( h, s ) ∈
[ H ] × S and Q function Q h : S ×
A × B → R , we define shorthand [( µ h )
⊤ Q h ν h ]( s ) :
= ⟨ Q h ( s, · , · ) , µ h ( ·| s ) ×
ν h ( ·| s ) ⟩ for any policy ( µ, ν ) .
Similarly, we let [ Q h ν h ]( s, · ) :=
E b ∼ ν h ( ·| s )
[ Q h ( s, · , b )] ∈ R A , and [ Q ⊤ h µ h ]( s, · ) :=
E a ∼ µ h ( ·| s )
[ Q h ( s, a, · )] ∈ R B .
We use A ∨ B :=
max { A, B } .
An algorithm framework for zero-sum Markov games
We begin by presenting an algorithm framework that unifies many existing and new algorithms for two-player zero-sum Markov Games, and its performance guarantee that could be specialized to yield concrete convergence results for many specific algorithms.
Our algorithm framework, described in Algorithm 1, consists of two main components: the policy update step computing policies ( µ t , ν t ) , and the value update step computing the Q estimate Q th ’s.
Policy update via matrix game algorithms
In the policy update step (Line 5), for each ( h, s ) , the two players update policies ( µ th ( ·| s ) , ν th ( ·| s )) at ( h, s ) using some matrix game algorithm
MatrixGameAlg which takes as input all past Q matrices and all past policies of both players.
The MatrixGameAlg offers a flexible interface that allows many choices such as the matrix NE subroutine over the most recent Q matrix MatrixNE ( Q t − 1 h ( s, · , · )) , or any independent no-regret algorithm (for both players), such as Follow-The-Regularized-Leader (FTRL) (10) or projected
Gradient Descent-Ascent (11) considered in the examples later.
Value update with learning rate { β t } For any ( h, s, a, b ) , the value update step (Line 6) updates
Q th ( s, a, b ) by the newest value function r h + P h
[( µ th +1 )
⊤ Q
th +1 ν th +1 ] propagated from layer h
+ 1 , using a sequence of learning rates { β t } t ≥ 1 which we assume to be within [0 , 1] (with β 1 := 1 ).
{ β t } controls the speed of the value update, with two important special cases:
(1) Eager value updates, where we set β t = 1 so that Q th performs policy evaluation of the current policy ( µ t , ν t ) , that is, Q th = Q µ t ,ν t h .
(2) Smooth (incremental) value updates, where we choose β t → 0 as t → ∞ .
In this case, the Q th moves slower (resembling a critic in Actor-Critic like algorithms), and becomes a weighted average of all past updates.
A standard choice that is frequently used is from [23] (and many subsequent work),
[Equation]
1: Require: Learning do 4: do 5:
Policy update: Update policies all s
: ( µ th ( ·| s ) , ν th ( ·| s )) ← MatrixGameAlg (cid:16)(cid:8) Q ih ( s, · , · ) (cid:9) t − 1 i =1 , (cid:8) µ ih ( ·| s ) (cid:9) t − 1 i =1 , (cid:8) ν ih ( ·| s ) (cid:9) t − 1 i =1 (cid:17) .
Value update:
Q b
← (1 − β t ) Q t − 1 h ( s, a b ) + β t (cid:0) r h +
[( µ th +1 )
⊤ Q
th +1 ν th +1 ] (cid:1) s a, b ) .
(4) 7: Output
: (cid:98) µ Th ( ·| s ) ← (cid:80) Tt =1 β tT µ th ( ·| s ) , (cid:98) ν
Th ( ·| s ) ← (cid:80) Tt =1 β tT ν th ( ·| s ) for all ( h, s ) ∈
[ H ] × S .
For any { β t } , the update (4) implies that
[Equation]
where β it ’s are a group of weights summing to one ( (cid:80) ti =
1 β it = 1 ) defined as
[Equation]
Note that with smooth value updates ( β t < 1 ), Q th is not necessarily the Q-function of any policy.
Upon finishing, the algorithm outputs the state-wise average policy ( (cid:98) µ T , (cid:98) ν T ) defined in (5),
where each (cid:98) µ Th ( ·| s ) is the weighted average of µ th ( ·| s ) using weights { β tT } Tt =1 (and similarly for (cid:98) ν T ), which we remark can be implemented efficiently using moving averages (cf.
Appendix C.1).
Symmetric & simultaneous learning, (de)centralization We remark that Algorithm 1 by definition performs simultaneous learning (of policies and values) at all layers, and also yields symmetric (policy) updates if MatrixGameAlg is a symmetric algorithm with respect to µ and ν .
Also, although
Algorithm 1 appears to be a centralized algorithm as it maintains Q values in (4), this does not preclude possibilities that the algorithm can be executed in a decentralized fashion.
This can happen e.g. when the Q-update (4) can be rewritten as an equivalent V-update (cf.
Example 1 & 2).
3.1 Theoretical guarantee
We are now ready to state the main theoretical guarantee of Algorithm 1, which states that the ( (cid:98) µ T , (cid:98) ν T ) is an approximate NE, as long as the algorithm achieves low per-state weighted regrets w.r.t.
weights { β it } ti =1 , defined as
[Equation]
Theorem 2 (Main guarantee of Algorithm 1) .
Suppose that the per-state regrets can be upperbounded as reg th ≤ reg th for all ( h, t ) ∈
[ H ] × [ T ] , where reg th is non-increasing in t : reg th ≥ reg t +1 h for all t ≥ 1 .
Then, the output policy ( (cid:98) µ T , (cid:98) ν T ) of Algorithm 1 satisfies
[Equation]
for all T ≥ 2 and some absolute constant C > 0 , where c β is a constant depending on { β t } t ≥ 1 :
[Equation]
Specifically, c β = (cid:0) 1 + 1 H (cid:1) if β t = α t =
H +1 H + t , and c β = 1 if β t = 1 .
Bound (8) is typically dominated by the second term on the right hand side, suggesting that
the
NEGap can be bounded by the average weighted regret (cid:101) O (cid:16) 1 T (cid:80)
Tt =1 max h reg th (cid:17) , if c Hβ = O (1) .
Theorem 2 serves as a modular tool for analyzing a broad class of algorithms: As long as this average regret is sublinear in T (including—but not limited to—choosing MatrixGameAlg as uncoupled no-regret algorithms), the output policy will be an approximate NE.
We emphasize though that this result is not yet end-to-end, as each reg th is a weighted regret w.r.t.
the particular set of weights (cid:8) β it (cid:9) t i
=1 , minimizing which may require careful algorithm designs and/or case-by-case analyses.
We provide some concrete examples in Section 3.2 to demonstrate the usefulness of Theorem 2.
We remark that the state-wise average policy considered in Theorem 2 is an average policy that is also Markovian by definition, which is different from existing work which considers either the (Markovian) last iterate [53] or non-Markovian average policies (e.g. [3]).
However, this guarantee relies on full-information feedback (so that per-state regret bounds are available), and it remains an open question how such guarantees could be generalized to sample-based settings.
Proof overview The proof of Theorem 2 follows by (1) bounding NEGap( (cid:98) µ T , (cid:98) ν T ) in terms of perstate regrets w.r.t.
the Nash value functions Q ⋆h ’s by performance difference arguments (Lemma C.1); (2) recursively bounding the value estimation error δ th :=
∥
Q th − Q ⋆h ∥ ∞ (Lemma C.2) which yields the constant c β ; and (3) combining the above to translate the regret from Q ⋆h ’s to Q th ’s (which we assume to be bounded by reg th ) and obtain the theorem.
The full proof can be found in Appendix C.
3.2 Examples
We now demonstrate the generality of Algorithm 1 and Theorem 2 by showing that they subsume many existing algorithms (and yield new algorithms) for two-player-zero-sum Markov games, and provide new guarantees with the particular output policy (5).
Example 1 (Nash V-Learning [3], full-information version) :
The full algorithm (Algorithm 5) can be found in Appendix D.1.
The algorithm is a special case of Algorithm 1 with β t = α t
=
( H + 1) /
( H + t ) , and MatrixGameAlg chosen as the weighted FTRL algorithm
[Equation]
where w t := α tt /α 1 t .
Combining Theorem 2 with the standard regret bound of weighted FTRL, this algorithm achieves NEGap
[Equation]
η
[Equation] [Equation]
T (Proposition D.2).
Additionally, although the original Nash V-learning algorithm [3] updates the V values (which makes the algorithm implementable in a decentralized fashion) instead of the Q values used in Algorithm 1, these two forms are actually equivalent in the full-information setting (Proposition D.1).
♢
Compared with the (cid:101) O
( (cid:112) H 5 S max { A, B } /T ) guarantee of (the non-Markovian output policy of)
Nash V-Learning in the sample-based online setting [3, 51, 24], our rate achieves better (logarithmic)
S, A, B dependence due to our full-information setting, and worse H dependence which happens as our output policy is the (Markovian) state-wise average policies, whose guarantee (Theorem 2) follows from a different analysis.
Example 2 (GDA-Critic) :
This algorithm is a special case of Algorithm 1 with β t = α t
=
( H + 1) /
( H + t ) , and MatrixGameAlg as projected gradient descent/ascent (GDA), i.e.,
[Equation]
Similar as Nash V-Learning, GDA-Critic also admits an equivalent form with V value updates (full description in Algorithm 6).
As GDA achieves weighted regret bounds with any monotone weights including (cid:8) α it (cid:9) t i =
(Lemma B.1), we can invoke Theorem 2 to show that this algorithm achieves
NEGap( (cid:98)
[Equation]
, (cid:98)
[Equation]
T ) ≤ (cid:101)
[Equation]
( H
[Equation] [Equation]
( A ∨ B ) 1 /
[Equation]
/ √ T ) if we choose η ≍ 1 / √ T (Proposition D.4).
The GDA-critic algorithm is also similar to the OGDA-MG algorithm of Wei et al.
[53], except that we use the (non-optimistic) vanilla version of GDA.
To our best knowledge, the above algorithm and guarantee are not known.
We remark that even ignoring difference between GDA and OGDA, the above guarantee cannot be obtained by direct adaptation of the results of [53] which focus on either the average duality gap and/or last-iterate convergence.
♢
Besides the above examples, Algorithm 1 also incorporates the following algorithms which are typically not categorized as policy optimization algorithms (see Appendix I.2 for a discussion of the categorization).
Example 3 (Nash Q-Learning [20, 3], full-information version) :
This algorithm is a special case of
Algorithm 1 with β t = α t =
( H + 1) /
( H + t ) and MatrixGameAlg as the matrix Nash subroutine
[Equation]
(Full description in Algorithm 7.)
Although MatrixNE ( Q t − 1 h ( s, · , · )) is not by default a no-regret algorithm, using the fact that (cid:13)(cid:13)
Q th − Q t − 1 h (cid:13)(cid:13) ∞ is small (due to the small α t ) we can show that it is close to a (hypothetical) “Be-The-Leader” style algorithm that computes the matrix NE of the current
Q matrix Q th which achieves ≤ 0 regret (Lemma D.3).
Combining this with Theorem 2 shows that this algorithm achieves NEGap( (cid:98) µ T , (cid:98) ν T ) ≤ (cid:101)
O
( H 4 /T )
(Proposition D.5).
♢
Example 4 (Nash Policy Iteration (Nash-PI)) :
This classical algorithm (Algorithm 8) performs iterative policy evaluation and policy improvement (also similar to Nash Value Iteration [47, 2, 33]):
[Equation]
[Equation]
) ← MatrixNE ( Q
[Equation]
h ( s
[Equation]
This is also a special case of Algorithm 1 with β t = 1 and MatrixGameAlg set as MatrixNE .
It is a standard result that this algorithm converges exactly (achieving zero NE gap) in H steps, and this fact can be obtained using our framework as well (Proposition D.6).
♢
Fast convergence of optimistic FTRL
In this section, we instantiate Algorithm 1 by choosing MatrixGameAlg as the Optimistic FollowThe-Regularized-Leader (OFTRL) algorithm.
OFTRL is also an uncoupled no-regret algorithm that is known to enjoy faster convergence than standard FTRL under additional loss smoothness assumptions [41, 50, 9, 12].
We show that, using OFTRL, Algorithm 1 enjoys faster convergence than the (cid:101)
O (1 / √ T ) rate of using FTRL or GDA (cf.
Example 1 & 2).
Concretely, we use the following weighted OFTRL algorithm at each ( h, s, t ) :
[Equation]
(13) where w t is the same weights as defined in Example 1, and we choose β t = α t
= ( H + 1) /
( H + t ) .
Theorem 3 (Fast convergence of OFTRL in zero-sum Markov Games) .
Suppose Algorithm 1 is instantiated with β t = α t
=
( H + 1) /
( H + t ) and MatrixGameAlg to be the OFTRL algorithm (13) with any η ≤ 1 /H (full description in Algorithm 9).
Then the per-state regret can be bounded as follows for some absolute constant C > 0 :
[Equation] [Equation]
Further, choosing η = poly( H, log( A ∨ B ) , log T ) · T − 1 / 6 , the output (state-wise average) policy ( (cid:98) µ T , (cid:98) ν T ) achieves approximate NE guarantee
[Equation] [Equation]
To our best knowledge, the (cid:101) O
( T − 5 / 6 ) rate asserted in Theorem 3 is the first rate faster than the standard (cid:101) O
(1 / √ T ) for symmetric, policy optimization type algorithms in two-player zero-sum
Markov games.
The closest existing result to this is of Wei et al.
[53], who analyze the OGDA algorithm with smooth value updates and show a (cid:101)
O (1 / √ T ) convergence of both the average NEGap and the NEGap of the last-iterate.
However, these only imply at most a (cid:101) O
(1 / √ T ) rate for the average policies, and not our faster rate 4 .
Cen et al.
[8], Zhao et al.
[62] show (cid:101) O (1 /T ) convergence of algorithms with optimistic gradient-based policy updates, which are however very different styles of algorithms that either performs layer-wise learning (the matrix games at each state are learned to sufficient precision before the backup) similar as Value Iteration (the matrix games at each state are learned to sufficient precision before the backup) [8], or uses strongly asymmetric updates that simulate a policy gradient-best response dynamics [62].
By contrast, our Algorithm 9 (as well as its modified version in Algorithm 10 with (cid:101) O ( T − 1 ) rate) runs symmetric no-regret dynamics for both players, simultaneously at all layers.
Proof overview The proof of Theorem 3 (deferred to Appendix E) builds upon the recent line of work on fast convergence of optimistic algorithms [41, 50, 9], in particular the work of Chen and
Peng [9] which shows an (cid:101) O
( T − 5 / 6 ) convergence rate of OFTRL for two-player normal-form games.
Our regret bound (14) generalizes this result non-trivially by additionally handling (1) The weighted regret, which requires bounding the weighted stability of the OFTRL iterates by a new analysis of the potential functions (Lemma B.4), and (2)
The errors induced by changing game matrices , as
Q th ( s, · , · ) changes over t .
Plugging (14) into Theorem 2 yields the policy guarantee (15).
Modified OFTRL algorithm with (cid:101) O
( T − 1 ) rate We further slightly modify Algorithm 9 to design a new OFTRL style algorithm with (cid:101)
O
( T − 1 ) convergence rate (Algorithm 10 and Theorem F.1), which improves over the (cid:101) O
( T − 5 / 6 ) of Theorem 3 and matches the known best convergence rate for policy optimization type algorithms in two-player zero-sum Markov games.
Algorithm 10 still uses
OFTRL in its policy update step, and the main difference from Algorithm 9 is in its value update step:
Rather than maintaining a single Q th , the two players now each maintain their own value estimate
Q th , Q th which are still updated in an incremental fashion similar to (though not strictly speaking an instantiation of) the update rule (4) in our main algorithm framework.
Details of the algorithm as well as the proofs are deferred to Appendix F.
4.1 Extension to multi-player general-sum Markov games
Our fast convergence result can be extended to the more general setting of multi-player general-sum
Markov games.
Concretely, we consider general-sum Markov games with m ≥ 2 players, S states,
H steps, where the i -th player has action space
A i with A max := max i ∈ [ m ] |A
i | and her own reward function.
The goal is to find a correlated policy over all players that is an approximate Coarse
Correlated Equilibrium (CCE) of the game (see Appendix G.1 for the detailed setup).
We show that the OFTRL algorithm works for general-sum Markov games as well, with a fast (cid:101)
O ( T − 3 / 4 ) convergence to CCE.
The formal statement and proof is in Theorem G.1 & Appendix G.4.
Theorem 4 (Fast convergence of OFTRL in general-sum Markov Games; Informal version of Theorem G.1) .
For m -player general-sum Markov Games, running the OFTRL algorithm (Algorithm 12) for T rounds, the output (correlated) policy (cid:98) π is an ε -approximate CCE, where
[Equation]
A baseline result for this problem would be (cid:101) O
( T − 1 / 2 ) , which may be obtained directly by adapting existing proofs of the V-Learning algorithm [49, 24] to the full-information setting.
Our Theoremshows that a faster (cid:101) O
( T − 3 / 4 ) rate is available by using the OFTRL algorithm, which to our best knowledge is the first such result for policy optimization in general-sum Markov games.
We also remark that the output policy (cid:98) π above is not a state-wise average policy as in the zero-sum setting, but rather a mixture policy that is in general non-Markov (cf.
Algorithm 13), which is similar as (and slightly simpler than) the “certified policies” used in existing work [3, 49, 24].
The proof of
Theorem 4 builds upon the RVU property of OFTRL [50] and additionally handles changing game rewards, similar as in Theorem 3.
A proof sketch and comparison with the (cid:101)
O ( T − 5 / 6 ) analysis of the zero-sum case can be found in Appendix G.2.
Simulations
We perform numerical studies on the various policy optimization algorithms.
Our goal is two-fold: (1) Verify the convergence guarantees in our theorems and examples; (2) Test some other important special cases of Algorithm 1 that may not yet admit a provable guarantee
To this end we consider three algorithms covered by the framework in Algorithm 1:
FTRL (Nash V-Learning) with smooth value updates β t = α t (Example 1 & Algorithm 5).
Here the output policy ( (cid:98) µ T , (cid:98) ν T ) are the state-wise averages with weights (cid:8)
α iT (cid:9) T i =1 , and achieves NEGap( (cid:98) µ T , (cid:98) ν T )
≲ T − 1 / 2 if we choose η ≍ T − 1 / 2 (Proposition D.2).
OFTRL with smooth value updates β t = α t (Algorithm 9).
Here the output policy ( (cid:98) µ T , (cid:98) ν T ) are the state-wise averages with weights (cid:8)
α iT (cid:9) T i =1 , and achieves NEGap( (cid:98) µ T , (cid:98) ν T )
≲ T − 5 / 6 if we choose η ≍ T − 1 / 6 (Theorem 3).
We also consider the more aggressive choice η = 1 . 3.
INPG (Independent Natural Policy Gradients).
This algorithm is an instantiation of Algorithm 1 (cf.
Appendix H.3 for formal justifications) with eager value updates ( β t = 1 ), and MatrixGameAlg chosen as standard unweighted FTRL (a.k.a.
Hedge) for all ( h, s, t ) : µ th
[Table]
s
[Table]
µ
[Table]
h
[Table] [Table]
⊤ µ t − 1 h (cid:105) ( s ) (cid:17) .
For this algorithm, we choose two standard learning rates:
η = 1 , and η = T − 1 / 2 , and use the vanilla (state-wise) average as the output policies (since the last-iterate is known to be cyclic): (cid:98) µ Th ( ·| s ) =
1 T (cid:80)
Tt =1 µ th ( ·| s ) , (cid:98) ν
Th ( ·| s ) =
1 T (cid:80)
Tt
=1 ν th ( ·| s ) for all ( h, s ) ∈
[ H ] × S .
The main motivation for considering INPG is that it is a natural generalization of both the widelystudied NPG algorithm for single-agent RL, and the standard Hedge algorithm for zero-sum matrix games.
In both cases the algorithm admits favorable convergence guarantees: NPG converges with rate O
( T − 1 ) [1, 27, 38, 7] (in both last iterate and averaging) using η = O (1) ; Hedge converges with rate O
( T − 1 / 2 ) in zero-sum matrix games (e.g. [41]) using η ≍ T − 1 / 2 .
However, to our best knowledge, the convergence of INPG for zero-sum Markov games is unclear, and it is commented by Wei et al.
[53, Section 5] that eager value updates ( β t = 1 ) could cause the value function of the ( h + 1) th layer to oscillate, which make learning unstable or even biased within the h -th layer.
A two-layer numerical example We design a simple zero-sum Markov game with two layers and small state/action spaces ( H = 2 , S = 4 , A = 2 ; see Appendix H.1 for the detailed description).
The main feature of this game is that the reward in the first layer is much lower magnitude than that of the second layer (the scale is roughly | r 1 ( s, · , · ) | ≈ 0 .
1 | r 2 ( s, · , · ) | ), which may exaggerate the aforementioned unstable effect.
We also choose a careful initialization ( µ 1 , ν 1 ) which is non-uniform (and modify the FTRL / OFTRL algorithms to start at this initialization, cf.
Appendix H.1)
but with all entries bounded in [0 . 15 , 0 . 85] .
We test all three algorithms above on this game, with this initialization, T ∈ { 10 3 , 3 × 10 3 , 10 4 , . . .
, 10 7 } , and η chosen correspondingly as described above.
Results Figure 1a plots the NEGap of the final output policies, one for each {algorithm, ( T, η ) }.
Observe that FTRL converges with rate roughly T − .
570 ≲ T − 1 / 2 , and OFTRL with η = T − 1 /converges with rate T − .
835 ≈ T − 5 / 6 , both corroborating our theory.
Further, OFTRL with η =appears to converge with rate T − 1 ; showing this may be an interesting open theoretical question.
On the other hand, the INPG algorithm appears to be much slower: The η = 1 version does not seem to converge, whereas the convergence of η = T − 1 / 2 version is not clear but at least substantially slower than T − 1 / 2 ( T − .
308 given by the linear fit) .
To further understand the behavior of INPG, we visualize its layer-wise NEGap ’s for h ∈ { 1 , 2 } (on our example), defined as the NEGap of the h -th layer’s policies with respect to Q ⋆h :
[Equation]
Note that NEGap-Layer-1 is a lower bound of NEGap( µ, ν ) (cf.
Appendix H.3)
and thus needs to be minimized by any convergent algorithm.
By contrast, on our example, NEGap-Layer-2 is concerned with the last layer only, and can be minimized by any algorithm that works on matrix games.
[Figure]
with T = 10and
η = T − 1 / 2 .
As expected, the NEGap-Layer-2 converges nicely for both algorithms with similar rates (Figure 1b)
albeit the oscillation of INPG, whereas their behavior on NEGap-Layer-1 is drastically different: FTRL still converges, whereas INPG seems to be oscillating around a non-zero bias (Figure 1c).
This suggests that INPG may indeed be suffer from a non-vanishing bias in the first layer caused by the second layer’s learning dynamics.
(See Appendix H.2 for additional illustrations.)
It would be an interesting open question to investigate the convergence of INPG theoretically.
Conclusion
This paper provides a unified framework for analyzing a large class of policy optimization algorithms for two-player zero-sum Markov games.
Using our framework, we prove new fast convergence rates for the OFTRL algorithm with smooth value updates: (cid:101)
O
( T − 5 / 6 ) for learning Nash Equilibria two-player zero-sum Markov games, which can be further accelerated to ( T − 1 ) by slightly modifying the framework; and (cid:101) O
( T − 3 / 4 ) for learning Coarse Correlated Equilibria in multi-player general-sum
Markov games.
We further demonstrate the importance of smooth value updates on a simple numerical example.
We believe our work opens up many other interesting directions, such as whether improved rates (e.g. (cid:101) O ( T − 1 ) ) are available for the unmodified OFTRL algorithm, or further investigation of policy optimization algorithms with eager value updates (such as Independent Natural Policy
Gradients).
Finally, a limitation of this work is its focus on the full-information setting, and it is an important open question how to generalize our analyses to the sample-based setting.
Acknowledgment
The authors would like to thank Chi Jin, Yuanhao Wang, Tiancheng Yu, Shicong Cen, and Song Mei for the valuable discussions.
Runyu Zhang is supported by NSF AI institute: 2112085, ONR YIP:
N00014-19-1-2217, NSF CNS: 2003111 and NSF CPS: 2038603.
A Technical tools A.1 Properties of α it
Throughout this section, the sequence (cid:8) β it (cid:9) i ∈ [ t ] is defined through sequence { β t } t ≥ 1 as in (6), and α it is its special case with β t = α t , where { α t } t ≥ 1 is defined in (3).
We present some basic algebraic properties of α it that will be used in later proofs.
Lemma A.1.
Given a sequence { ∆ th } h,t defined by
[Equation]
where { β t } is non-increasing w.r.t.
t .
Then ∆ t +1 h ≤
[Equation]
th for all ( t, h
[Equation]
∈ N ×
[ H + 1] .
Proof.
We prove by doing backward induction on h .
For the base case of induction, notice that the claim is true for H + 1 .
Assume the claim is true for h
+ 1 .
At step h , we have
[Equation]
where the inequality follows from the inductive hypothesis and β t +1 ≤ β t .
The following lemma is taken from [23].
Lemma A.2.
The sequence α it satisfies the following:
(a) (cid:80) ∞
t = i α it
= 1 + 1 /H for all
i ≥ 1 .
Lemma A.3 (Convolution of β tT with decaying sequences) .
From general { β t } t ≥ 1 sequence, we have that
X T
[Equation]
=
[Equation]
(cid:88)
[Equation] [Equation]
T , for T ≥ 2 .
Specifically if β t = α t , the following holds for all T ≥ 1 :
(a) A T := (cid:80) T t =1 α tT · 1 t 2 ≤ 4 T .
(b) B T := (cid:80)
Tt =
α tT α t ≤ ( H +1) 2 H ( H + T ) (c) C T :=
(cid:80) Tt =
1 α tT · α 2 t ≤ 4 HT .
Proof.
We first prove for the inequality on X T for general { β t } t ≥ 1 sequence.
We start with showing that X t +1 ≤ X t , ∀ t ≥ 1 .
[Equation]
Since
[Equation]
we have that X t +1 − X t ≤ 0 .Thus
[Equation]
Now we prove for the specific case where β t = α t : (a) Note that A 1 =
and we have the recursive relationship
[Equation]
by definition of the sequence α tT .
In particular this implies A T +1 ≤ A T , since A T is a weighted average of 1 /t 2 ≥ 1 / ( T + 1) 2 .
Therefore we have
[Equation]
Above, the step (cid:80) Tt =
s α st ≤ (cid:80) ∞
t = s
α st = 1 + 1 /H follows from Lemma A.2.
(b) From the definition of B T we have that
[Equation]
Since B 1 = α 21 = 1 ≤ ( H +1) 2 H ( H +1) , we have that B T ≤ ( H +1) 2 H ( H + T ) via proof by induction.
(c) Since α t ≤ 1 , we have that C T ≤ B T , thus by part (b)
C T ≤ B T
[Equation]
( H +
[Equation] [Equation]
Consider the sequence { w t } t ≥ 1 defined by (cf.
also Example 1)
[Equation]
Note that we also have w t = α t T /α 1 T for any T ≥ t .
Lemma A.4 (Properties of w t ) .
The following holds for all t ≥ 2 :
(a) w t /w t − 1 =
( H + t − 1) /
( t − 1) .
(b) ( 1 w t − 1 − 1 w t ) (cid:80) t − 1 i =1 w i =
H/
( H + 1) .
Proof.
(a) We have
[Equation]
(b) We have
[Equation]
Above, (i) used part (a).
A.2 Other technical lemmas
Lemma A.5 (Smoothness of Exponential Weights) .
Let g 1 , g 2 ∈ R n and
[Equation]
where H ( x ) := (cid:80) ni
=1 x i log x
i is the standard entropy functional.
Then ∥ x 1 − x 2 ∥ 1 ≤ 2 ∥ g 1 − g 2 ∥ ∞ .
Proof.
Since H is 1 -strongly convex in ∥ · ∥ 1 (Pinsker’s inequality), we have that
[Equation]
H ( x 1 )) − ⟨ x 2 , g 1 ⟩ − ⟨∇ H ( x 2 ) , x 1 − x 2 ⟩ = ⟨ x 1 − x 2 , g 1 ⟩ − ⟨∇ H ( x 2 ) , x 1 − x 2 ⟩ = ⟨ x 1 − x 2 , g 1 − g 2 ⟩ + ⟨ g 2 − ∇ H ( x 2 ) , x 1 − x 2 ⟩ ≤ ⟨ x 1 − x 2 , g 1 − g 2 ⟩ (From the optimality of x 2 ) ≤ ∥ x 1 − x 2 ∥ 1 ∥
g 1 − g 2 ∥ ∞ , which completes the proof.
B Bound for regret minimization algorithms B.1 Projected gradient descent
Require:
Learning rate η > 0 .
The following weighted regret bound for projected gradient descent is standard.
For completeness we provide a proof here.
For simplicity of notation, denote the diameter of X ⊂ R d by R and
G = max t ∈ [ T ] ∥
g t ∥ 2 .
Lemma B.1 (Weighted regret bound for projected gradient descent) .
For any weights { w t } t ≥ 1 ∈ R >with w t ≤ w t
+1 for all t ≥ 1 , Algorithm 2 achieves max
z ∈X
T (cid:88)
[Equation]
Proof.
By following the standard GD analysis, we first have
[Equation]
where the inequality follows from z ∈ X and x t
+1 =
P X ( y t +1 ) .
By multiplying both sides with w t and taking summation over t ∈ [ T ]
[Equation]
we have
T
[Equation]
Above, (i) follows as w t +1 ≥ w t .
This completes the proof.
B.2 Follow-The-Regularized Leader (FTRL)
In this subsection
[Equation]
we consider the following weighted FTRL algorithm over the probability simplex ∆ [ A ] with the standard (negative) entropy regularizer Φ( x ) :
= (cid:80)
a ∈ [ A ] x ( a ) log x ( a ) .
Below the notation x t ( a ) denotes the a -th entry of x t .
Weights w
} t ≥ 1 ⊂ R >
[Equation]
Note that (18) has a closed-form solution via exponential weights:
[Equation]
Lemma B.2 (Regret bound of weighted FTRL) .
Suppose the weights are non-decreasing: w t +1 ≥ w t for all t ≥ 1 , and max t ∥ g t ∥ ∞ ≤ G .
Then Algorithm 3 achieves weighted regret bound
[Equation]
Proof.
Applying standard anytime FTRL analysis (see, e.g., Excercise 28.12 in [28]) with loss sequence { w t g t } t ≥ 1 and learning rate { η/w t } t ≥ 1 , we have that
T (cid:88) t =
[Equation]
−
[Equation]
Φ( x
[Equation]
) η
[Equation]
≤ w T log A η +
T (cid:88) t = (cid:18) w t ⟨ x t − x t +1 , g t ⟩ − w t ∥ x t − x t +1 ∥ 21 2 η (cid:19) (Pinsker’s inequality) ≤ w T log A η +
T (cid:88) t = w t η 2 ∥ g t ∥ 2 ∞ ≤ w T log A η + ηG
T (cid:88) t = w t , which completes the proof.
B.3 Optimistic Follow-The-Regularized-Leader (OFTRL)
We consider the following OFTRL algorithm on the probability simplex ∆ [ A ] with standard (negative) entropy regularizer Φ( x ) = (cid:80) a ∈
[ A ] x ( a ) log x ( a ) .
Require:
Learning rate { η t } t ≥ 1 .
[Equation]
(cid:42)
x
[Equation]
t (cid:88)
[Equation]
g
[Equation]
(cid:43)
[Equation]
Note that (20) has a closed-form solution via exponential weights:
[Equation]
The following regret bound for OFTRL follows similarly as standard OFTRL analysis, see, e.g. [41,
Lemma 1].
For completeness, we provide a proof here.
Lemma B.3 (Regret bound for OFTRL) .
Suppose the learning rates are non-increasing: η t ≥ η t + for all t ≥ 1 .
Then Algorithm 4 achieves the following bound for all x ∈ X :
[Equation]
Proof.
Consider a fixed T ≥ 1 .
Note that Algorithm 4 is equivalent to Algorithm (25) with regularizer
R t ( · ) :=
(Φ( · ) + log A ) /η
t +1 ≥ 0 for t ≥ 0 , and R T ( · ) :=
R T − 1 ( · ) (Note that the shifting by log A does not affect the algorithm.)
We first decompose the regret into the following three terms
T (cid:88)
[Equation]
where { q t } t ≥ 1 is defined in (26).
By Lemma B.5, we can upper bound the first two terms by
R T ( x ) − min x ′ R 0 ( x ′ )
+ S T ≤ R T ( x ) + S T and obtain
T (cid:88) t = ⟨ x t − x, g t ⟩
≤ R T ( x ) +
T (cid:88) t
= ( R t
[Equation] [Equation]
[Equation] [Equation]
T (cid:88)
[Equation]
≤ R T ( x ) +
T (cid:88)
[Equation]
(cid:19)
[Equation]
where the second inequality uses R t − 1 ≤ R t and R t − 1 is 1 /η t strongly-convex w.r.t.
∥ · ∥ 1 .
Finally, we conclude the proof by applying Cauchy-Schwarz inequality:
[Equation]
and triangle inequality
[Equation]
and the bound R T ( x ) ≤ log A/η T
+1 = log
A/η T for any x ∈ ∆ [ A ] .
The following lemma bounds the total variation of the iterates in terms of the smoothness of loss vectors and prediction vectors.
This can be seen as a generalization of [9, Lemma 3.2] to the case with changing learning rate and arbitrary prediction vectors.
Lemma B.4 (Bounding stability by the smoothness of loss) .
Suppose the learning rates are nonincreasing: η t ≥ η t +1 for all t ≥ 1 .
Then the OFTRL algorithm (20) satisfies (understanding
M 1 := 0 )
T (cid:88) t =
2 η
[Equation]
In particular
[Equation]
choosing the prediction vector M t =
g t − 1 with g 0 :=
[Equation]
and assume ∥ g t − g t − 1 ∥ ∞ ≤
G t for all t ≥
[Equation]
we have
[Equation]
Proof.
We first prove (22).
For any t ≥ 2 , the optimality condition of (20) for x t gives
[Equation]
for all x ′ ∈ ∆ [ A ] .
In particular
[Equation]
this holds for x ′ =
x t − 1 , from which we get
[Equation]
where (i) is by Pinsker’s inequality, and (ii) is since the KL divergence is the Bregman divergence of Φ .
Summing the above over t = 2 , . .
. , T yields
T (cid:88)
[Equation] [Equation]
This proves (22).
Then, (23) follows by plugging in the regret bound given by Lemma B.3:
[Equation]
Finally, (24) is a direct consequence of (23) by plugging in M t =
g t − 1 and ∥ g t − g t − 1 ∥ ∞ ≤ G t for all t ≥ 1 .
B.3.1 Auxiliary lemma for OFTRL with general regularizers
Consider an OFTRL algorithm with loss function { g t } t ≥
⊂ R d , parameter space
X ⊂ R d , and convex regularizers R t :
X → R for t ≥ 0 :
[Equation]
Define auxiliary sequence
[Equation]
Recall the Bregman divergence associated with any convex regularizer R : X → R is given by
[Equation]
Lemma B.5 (Auxiliary lemma for OFTRL with general regularizers) .
Algorithm (25) achieves the following for any T ≥ 1 and x ∈ X :
[Equation]
where
S T :=
[Equation]
Proof.
We prove the lemma by induction.
The above relation holds trivially for T = 0 .
Assume the relation holds for τ = T − 1 .
For τ = T , we have
T
[Equation]
+ S T − 1 + ⟨ q T +1 , g T ⟩ + ⟨ x T − q T +1 , M T ⟩ (definition of q T ) ≤
T − 1 (cid:88) t = ⟨ x T , g t ⟩ + R T − 1 ( x T ) − D R T − 1 ( x T , q T )
− min x ′ ∈X R 0 ( x ′ )
+ S T − 1 + ⟨ q T +1 , g T ⟩ + ⟨ x T − q T +1 , M T ⟩ (optimality of q T ) =
min x ∈X (cid:34) ⟨ x,
T − 1 (cid:88) t =
g t + M T ⟩ + R T − 1 ( x ) (cid:35) − D R T − 1 ( x T , q T ) − min x ′ ∈X R 0 ( x ′ )
+ S T − 1 + ⟨ q T +1 , g T − M T ⟩ (definition of x T ) ≤⟨
q T +1 ,
T − 1 (cid:88) t =
g t + M T ⟩ + R T − 1 ( q T +1 ) − D R T − 1 ( q T +1 , x T )
− D R T − 1 ( x T , q T ) − min x ′ ∈X R 0 ( x ′ )
+ S T − 1 + ⟨ q T +1 , g T − M T ⟩ (optimality of x T ) =
min x ∈X ⟨ x,
T (cid:88) t =
g t ⟩ + R T ( x ) + ( R T − 1 ( q T +1 )
− R T ( q T +1 ))
− D R T − 1 ( q T +1 , x T )
− D R T − 1 ( x T , q T ) − min x ′ ∈X R 0 ( x ′ )
+ S T −(definition of q T +1 ) , which completes the induction.
C Proofs for Section 3.
In this section we prove Theorem 2.
The proof relies on the following two lemmas.
Lemma C.1 (Performance difference for Markov policies) .
In two-player zero-sum Markov games, suppose a Markov policy ( µ, ν ) satisfies the following for all h ∈
[ H + 1] :
[Equation]
Then we have for all h ∈
[ H ] that
[Equation]
Proof.
We prove by backward induction over h .
The claim is trivial for h = H + 1 .
Suppose the claim holds for step h
+ 1 .
At step h ,
[Equation]
≤
[Equation]
Notice that ∥
[Equation]
inductive hypothesis)
Combining the two inequalities we get
[Equation]
This proves the claim for ∥ V † ,νh − V ⋆h ∥ ∞ .
The same argument also holds for ∥ V µ, † h − V ⋆h ∥ ∞ , which completes the proof.
Throughout the rest of this section, we define the following shorthand for the value estimation error:
[Equation]
where Q th is the estimated value in Algorithm 1.
Lemma C.2 (Recursion of value estimation) .
Algorithm 1 guarantees that for all ( t, h ) ∈
[ T ] ×
[ H ] ,
[Equation]
Further, suppose that reg th ≤ reg th for all ( h, t ) ∈
[ H ] × [ T ] , where reg th is non-increasing in t : reg th ≥ reg t +1 h for all t ≥ 1 .
Then we have
[Equation]
where c β is defined in (9) .
Proof.
Fix ( h, s, a, b ) ∈
[ H ] × S ×
A × B .
From the definition of Q ⋆h we have that
[Equation]
Above, the last equality is derived from the update rule (3), which implies that
[Equation]
Therefore we have
[Equation]
Apply similar analysis to the min-player, we get
Q t h
[Equation]
a, b ) − Q ⋆
[Equation]
) ≤ t (cid:88) i = β
[Equation]
t δ i h
+1 + reg t
h +1 , ∀ s, a
[Equation]
b.
Thus we get δ th ≤ t (cid:88)
i =
β it δ
ih +
+ reg th +1 , which completes the proof of the first inequality in the Lemma.
Now consider an auxiliary sequence { ∆ th } h,t defined by
[Equation]
all
[Equation]
Where reg th is the upperbound of reg th defined in Theorem 2.
Observe that { ∆ th } h,t satisfies the following properties
[Equation]
Therefore
[Equation]
to control δ th
[Equation]
it suffices to bound ∆ th ≤ 1 t (cid:80) ti =1 ∆ ih
[Equation]
which follows from the standard argument in [23]:
[Equation]
β
[Equation]
Above, the last step used the fact that c β ≥ 1 .
This completes the proof of the second inequality in the Lemma.
We are now ready to prove the main theorem.
Proof of Theorem 2.
Fix any ( h, s ) ∈
[ H ] × S .
We first give a bound for max
[Equation]
Q
[Equation]
ν † (cid:105)
[Equation]
s )
, i
[Equation]
e. the per-state duality gap of ( (cid:98) µ T , (cid:98) ν T ) with respect to Q ⋆ h .
We have
[Equation]
′ (Lemma C.2)
[Table]
t =
[Table]
t β tT (cid:33)(cid:32)
[Table]
(cid:88) i =
[Table]
max h ′ reg ih ′ (cid:33) .
Apply Lemma A.3 into the above inequality, we get that
[Equation]
Since
[Equation]
by applying Lemma C
[Equation]
1 and the preceding per-state duality gap bound
[Equation]
we have
[Equation]
This completes the proof.
C.1 Implementation of state-wise average policy
Here we explain how to implement the state-wise average policy (5) efficiently via moving average.
Recall that the weights { β tT } Tt =1 are defined via { β t } t ≥ 1 through (6).
Therefore, for each ( h, s ) , we can maintain a set of moving averages
[Equation]
for all t ≥ 1 during the execution of Algorithm 1.
At time T , we output the moving average µ Th ( ·| s ) , which is exactly the desired state-wise average policy in (5):
[Equation]
D Algorithm details and proofs for Section 3. D.1 Nash V-Learning (full-information version)
The full description of Nash V-Learning (Example 1) using V updates is presented in Algorithm 5.
Proposition D.1 (Equivalence between V update and Q update) .
Nash V-learning (full-information version) in Algorithm 5 is equivalent to Algorithm 1 with the MatrixGameAlg as weighted FTRL (10) .
Proof.
It suffices to show that, for the Q value defined in (4) and the V value defined in (30), the following holds for all ( h, s, a, b ) and all t ∈
[ T ] :
[Equation]
Since α 1 = 1 , it is not hard to verify that
[Equation]
( s, a, b ) ,
We now prove by induction on both t and h .
Given that Q 1 h ( s, a, b ) = (cid:2)
r h
+ P h V 1 h +1 (cid:3)
( s, a, b ) , it is not hard to verify that Q tH ( s, a, b ) =
r H ( s, a, b ) , ∀ k ≥ 0 .
We assume that (31) holds for ( t − 1 , h ) and ( t, h + 1) , then for ( t, h ) , from (3)
Q th
[Equation]
Require:
Learning rate { α t } t ≥ 1 in (3) and corresponding { w t } t ≥ 1 (cf.
Example 1); η > 0 .
Initialize: Set V 0 h ( s )
=
H − h + 1 for all ( h, s ) ∈
[ H ] × S .
for t = 1 , . . .
, T do for h = H, . . .
, 1 do
Update policy for all s ∈ S (understanding w 0 :
= 1 ):
[Equation] [Equation]
Update V value for all s ∈ S :
[Equation]
30) ≤
[Equation]
1 − α t ) (cid:2) r
[Equation]
P h
[Equation]
( s
[Equation] [Equation]
r h + P h
[Equation]
( µ th +
[Equation]
r h +
[Equation]
P h +
[Equation]
V h +2 )
ν th +1 (cid:3)(cid:1) ( s, a, b ) (inductive hypothesis)
= (cid:2) r h + P h (cid:0) (1 − α
[Equation]
from (
[Equation]
which completes the proof by induction.
Lemma D.1 (Per-state regret bound for Nash V-learning) .
Algorithm 5 achieves the following per-state regret bound:
[Equation]
Proof.
Fix any ( h, s ) .
Note that the update of { µ th ( ·| s ) } t ≥ 1 in (29) is equivalent to the weighted
FTRL algorithm (Algorithm 3) with loss vectors
g i = −
[Equation]
Q ih ν ih (cid:3) ( s ) .
Thus by Lemma B.2 we get for any t ≥ 1 that
[Equation]
Further, recalling α it = w i · α 1 t for 1 ≤ i ≤ t , we have
[Equation]
The similar bound also holds for reg th,ν , and thus we have that
[Equation]
Proposition D.2 (Guarantee of Nash V-Learning) .
Algorithm 5 achieves
[Equation]
Specifically, choosing η = 4 √ HT , we have
[Equation]
Proof.
From Lemma D.
[Equation]
we can take reg th as reg th =
( H +1)log( A ∨ B ) ηt + ηH 2 2 .
Then from Theorem
[Equation]
we have
[Equation]
Thus, choosing η = 4 √ HT , we get
[Equation]
D.2 GDA-Critic
The full description of GDA-Critic (Example 2) using V updates is presented in Algorithm
[Equation]
Algorithm 6 GDA-Critic
Require:
Learning rate { α t } t ≥ 1 (defined in (3)), and η > 0 .
Initialize: set V 0 h
( s ) =
H − h + 1 and µ 0 ( ·| s ) , ν 0 ( ·| s ) to be uniform for all
( h, s ) ∈
[ H ] × S .
for t = 1 , . . .
, T do for h = H, . . .
, 1 do
Update policy for all s ∈ S :
[Equation]
Update V value for all s ∈ S :
[Equation]
Similar as Proposition D.1 (with the same proof), the following equivalence between Q updates and
V updates also holds for GDA-Critic.
Proposition D.
(Equivalence between Q updates and V updates for GDA-Critic) .
Algorithm 6 is equivalent to our algorithm framework (Algorithm 1) with the MatrixGameAlg instantiated as (11) .
Lemma D.2 (Per-state regret bound for GDA-Critic) .
Algorithm 6 achieves the following per-state regret bound:
[Equation]
Proof.
Fix any ( h, s ) and t ≥ 1 .
We apply Lemma B.1 to the projected gradient descent (or ascent) update (32), with weights w i = α it and loss vectors
g i ’s − (cid:2) Q ih ν ih
[Equation]
( s ) or (cid:104)(cid:0) Q ih (cid:1) ⊤ µ ih (cid:105) ( s ) respectively.
For the gradient ascent update for µ th ( ·| s ) , we get
[Equation]
Proposition D.4 (Guarantee of GDA-Critic) .
Algorithm 6 achieves
NEGap
[Equation]
Specifically, picking η = 4 √ ( A ∨ B ) HT yields
[Equation]
Proof.
From Lemma D.2, we can take reg th as 2( H +1) ηt + η
( A ∨ B ) H 2, then from Theorem
[Equation]
max h reg th ≤ 4 H (cid:18) 2( H + 1) ηT
[Equation]
( A
[Equation]
H (cid:19)
[Equation]
24 H
[Equation]
log( T ) · 1 T
T (cid:88) t = (cid:18)
[Equation]
( H + 1) ηt +
[Equation]
( A ∨ B ) H (cid:19) ≤ 14 η ( A ∨ B ) H 4 log( T ) + 48 H 2 log( T ) · 1 T
T (cid:88) t
= ( H + 1) ηt + 16 HηT ≤ 14 η ( A ∨ B ) H 4 log( T ) + 208 log( T ) 2 H 3 ηT .
Thus, pick η = 4 ( A ∨ B ) √ HT , we get
NEGap
[Equation]
T .
D.3 Nash Q-Learning (full-information version)
The Nash Q-Learning algorithm (Example 3) is described in Algorithm 7.
Lemma D.3 (Per-state regret bound for Nash Q-Learning) .
Algorithm 7 achieves the following per-state regret bound:
[Equation]
Proof.
We have
[Equation]
Require:
Learning rate (For Nash Q-learning) { β
[Equation]
= α t } ;
Initialize: Q 0 h ( s, a, b ) ← H − h + 1 for all ( h, s, a, b ) .
for k = 1 , . . .
, K do for h = H, . . .
, 1 do
Update policy for all s ∈ S :
[Equation]
Update Q value for all (
[Equation]
, a, b
[Equation]
∈ S
[Equation]
A × B :
[Equation]
The same bound also holds for reg th,ν ( s ) , thus reg th ≤ t (cid:88) i =
[Equation]
i − 1 h ∥ ∞
[Equation]
Since
[Equation]
substituting this into the above equations we have that
[Equation]
(From Lemma A.3 (b)) , which completes the proof
[Equation]
Proposition D.5 (Guarantee for Nash Q-Learning) .
Algorithm 7 achieves
[Equation]
Proof.
From Theorem 2 and Lemma D.3 we have that
[Equation]
Initialize: Q 0 h ( s, a, b ) ← H − h + 1 for all ( h, s, a, b ) .
for t = 1 , . . .
, T do for h = H, . . .
, 1 do
Update policy for all s ∈ S :
[Equation]
Update Q value for all ( s, a, b ) ∈ S
×
A × B :
[Equation]
D.4 Nash Policy Iteration
The full description of Nash Policy Iteration (Nash-PI, Example 4) is presented in Algorithm 8.
Note that from (34), we have that Q kh equals to Q µ k × ν k h .
Based on this observation, we have the following lemma.
Lemma D.4 (Exact learning of Q functions) .
For Algorithm 8, we have for any h ∈
[ H ] and t ≥
H − h
+
that
[Equation]
Proof.
We prove this by backward induction over h .
For h = H , we have that
[Equation]
Assume that for h + 1 , the condition holds, then for time horizon h and iteration step t ≥
H − h + 1 , we have that
[Equation]
Additionally, from the inductive hypothesis
[Equation]
we have that
[Equation]
Thus
[Equation]
which completes the proof.
Proposition D.6 (Guarantee for Nash-PI) .
Algorithm 8 achieves NEGap( (cid:98) µ T , (cid:98) ν T ) = 0 for T ≥ H .
Proof.
For this proposition we will not proof by calling Theorem 2, but instead directly apply Lemma
C.1, which is an auxiliary lemma for proving Theorem 2.
Note that Nash-PI corresponds is equivalent to using β t = 1 in Algorithm 1, so that β it
= 1 { i = t } .
From Lemma D.4 we have that
[Equation]
Thus for t ≥ H , max
[Equation]
1: Initialize: Q 0 h ( s, a, b ) ← H − h + 1 for all ( h, s, a, b ) .
2: for t = 1 , . . .
, T do 3: for h = H, . . .
, 1 do 4:
Update policies for all s ∈ S by OFTRL:
[Equation]
)( s,
[Equation]
(cid:105)(cid:17) .
Update Q-value for all ( s, a, b ) ∈ S ×
A × B :
[Equation] [Equation]
6: Output state-wise average policy:
[Equation]
Then applying Lemma C.1, we obtain
[Equation]
This is the desired result.
E Proof of Theorem
In this section we prove Theorem 3.
The full algorithm box of OFTRL for Markov Games is provided in Algorithm 9.
We aim to show that
[Equation]
Bounding per-state regret We first bound reg tν,h ( s ) , i.e. the per-state regret for the min-player, for any fixed ( h, s, t ) ∈
[ H ] × [ S ] ×
[ T ] .
(The bound for reg tµ,h ( s ) follows similarly.)
This is the main part of this proof.
Throughout this part, we will fix ( h, s ) and omit these subscripts within the policies and Q functions, so that ν t h ( ·| s ) will be abbreviated as ν t (and similarly for µ t and Q t ).
We will also overload T ≥to be any positive integer (instead of the fixed total number of iterations).
Observe that the above update for ν t is equivalent to the OFTRL algorithm (Algorithm 4) with loss vectors g t = w t ( Q t ) ⊤ µ t (understanding g 0 = 0 and Q 0 h = 0 ), prediction vector M t =
g t − 1 = w t − 1 ( Q t − 1 ) ⊤ µ t − 1 , and learning rate η t = η/w t .
Therefore we can apply the regret bound for
OFTRL in Lemma B.3 and obtain for any T ≥ 1 that
[Equation]
We now relate the terms above to the stability of { µ t } t ≥ 1 (the other player’s policies).
Let
[Equation]
for all t ≥ 1 for shorthand, where ∥·∥ ∞ for a matrix denotes its infinity norm (i
[Equation]
e. entry-wise max absolute value).
Then we have
[Equation]
By symmetry
[Equation]
the similar bound also holds for { ν t }
[Equation]
from which we obtain for any t ≥ 2 that
[Equation]
Plugging the above two bounds into (37), we get
[Equation]
Above, (i) rearranges terms and used the fact that w t − 1 ≤ w t .
The following lemma (proof deferred to Section E.1) bounds term ERR T .
Lemma E.1 (Bound on ERR T ) .
Suppose η ≤ 1 /H .
Then for any T ≥ 1 , we have
[Equation]
To bound term STAB T , note that it is exactly the total distance (in squared L 1 norm) of the sequence { µ t } t ≥ 1 , which itself follows an OFTRL algorithm with loss sequence g ′ t := − w t Q t ν t , M ′ t :
= g ′ t − 1 , and η t = η/w t .
Therefore we can apply the stability bound (24) in Lemma B.4 to obtain that
STAB T = 4 η 2 H 2 ·
T (cid:88)
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where here we take G ′ t = w t H ≥ (cid:13)(cid:13)
g ′ t − g ′
[Equation]
− 1 (cid:13)(cid:13) ∞ , (i) holds whenever η ≤ 1 /H , and (ii) follows as w T − 1 H ≤ w
[Equation] [Equation]
≤ 2 w T log A/η .
Plugging the above bounds into (38) yields that for any T ≥ 1 , reg Tν,h ( s )
=
max ν † ∈ ∆ B
T (cid:88) t = α tT (cid:124)(cid:123)(cid:122)(cid:125)
α 1 T · w t (cid:10) ν t − ν † , ( Q t ) ⊤ µ t (cid:11) =
α 1 T max ν † ∈ ∆ B
T (cid:88) t = w t (cid:10) ν t − ν † , ( Q t ) ⊤ µ t (cid:11) ≤ log B · ( α 1 T w T )
η + α 1 T ERR T + α 1 T (cid:34)
STAB T −
T (cid:88) t =  16 ηH 2 w t − 1 (cid:13)(cid:13) w t Q t ν t − w t − 1 Q t − 1 ν t − 1 (cid:13)(cid:13) 2 ∞ (cid:35) ≤ log B · α TT η + 192 HηT + α 1 T (cid:20) 16 ηH 2 w T log A + 8 η 2 H
T − 1 (cid:88) t
=
(cid:13)(cid:13)
w t Q t ν t − w t − 1 Q t − 1 ν t − 1 (cid:13)(cid:13) ∞ −
T (cid:88) t =  16 ηH 2 w t − 1 (cid:13)(cid:13) w t Q t ν t − w t − 1 Q t − 1 ν t − 1 (cid:13)(cid:13) 2 ∞ (cid:21) ( i ) ≤ log B · α TT η + 192 HηT + α 1 T (cid:20) 32 ηH(cid:124)(cid:123)(cid:122)(cid:125) ≤ 1 /η w T log
A
+
T − 1 (cid:88) t
= (cid:18) 8 η 2 H 2 (cid:13)(cid:13) w t Q t ν t − w t −
[Equation]
Above, (i) used the fact that 8 η 2 H 2 (cid:13)(cid:13)
w 1 Q 1 ν 1 (cid:13)(cid:13) ∞ ≤ 8 η 2 H 3 w 1 ≤ 8 η 2 H 3 w T ≤ 16 ηH 2 w T log A , (ii) used the fact that 8 η 2 H 2 z − z 2 / (16 ηH 2 w t − 1 ) ≤ 256 η 5 H 6 w t − 1 by the AM-GM inequality, and (iii) used the fact that α TT = α T
=
( H + 1) /
( H + T ) ≤ 2 H/T , where C ≤ 256 is an absolute constant.
By symmetry, the same regret bound also holds for reg Tµ,h ( s ) , which gives that for any t ≥
[Equation]
Note that reg th is decreasing in t .
This is the desired regret bound.
Performance of output policy As our algorithm chooses β t = α t =
( H + 1) /
( H + t ) , we can invoke Theorem 2 with c β = 1 + 1 /H ≥ (cid:80) ∞ t =
j α jt (by Lemma A.2)
so that c Hβ =
(1 + 1 /H )
H ≤ e ≤ 3 .
Further, by the above regret bound,
[Equation]
Plugging this into Theorem 2 yields that the output policy ( (cid:98) µ T , (cid:98) ν T ) satisfies
[Equation]
(cid:18)
[Equation]
Choosing η = (log T log( A ∨ B ) /H
4 T ) 1 / 6 ∧ (1 /H ) , we get
NEGap( (cid:98) µ T , (cid:98) ν T ) ≤
O (cid:16) H 14 / 3 (log( A ∨ B )) 5 / 6 (log T ) 11 / 6 · T −
[Equation]
6 + H 5 log( A ∨ B )(log T ) 2 /T (cid:17) .
This proves (36) and thus Theorem 3.
E.1 Proof of Lemma E.
Recall our notation Q t :
=
Q th ( s, · , · ) ∈ [0 , H ] A × B for
some fixed
[Equation]
For t = 1 , we have (cid:13)(cid:13) w t Q t − w t − 1 Q t − 1 (cid:13)(cid:13) 2 ∞ ≤ w 21 H 2 =
H
Substituting this into the expression of
T gives
[Equation]
Above, (i) used w t − 1 ≤ w t and α 1 T w t = α tT ; (ii) used the fact that 2 ηH 2 ≤ 2 /η (as η ≤ 1 /H ) and thus 2 ηH 2 + 1 / (8 η ) ≤ (2 + 1 / 8) /η ≤ 3 /η ; (iii) used the fact that 2 ηH 2 ≤ 48 H 2 /η which also follows from η ≤ 1 /H ≤ 1 ; (iv) used Lemma A.3(a).
This is the desired result.
In this section we show that a slightly modified OFTRL algorithm (described Algorithm 10) achieves (cid:101)
O ( T − 1 ) convergence rate for finding NE in two-player zero-sum Markov Games, improving over the (cid:101)
O
( T − 5 / 6 ) of Algorithm 9.
[Table]
[Equation]
Algorithm 10 keeps track of a series of Q th , Q th ’s that are upper-bounds and lower-bounds of Q ⋆h respectively.
The policy update is similar to the update as the OFTRL algorithm (Algorithm 4), but here µ is performing OFTRL with respect to Q th ’s while ν with respect to Q th ’s. The value updates (39) are slightly different from the value update in our unified framework, however, we remark that it is still an incremental update because the terms inside the inner product (cid:80) ti
=
α it Q ih +1 ν
ih +1 , (cid:80) ti =
α it (cid:16) Q ih +1 (cid:17) ⊤ µ ih +1 are incremental updates, which leads to that fact that
Q th , Q th ’s are also updating incrementally..
Further, the algorithm can be performed in a decentralized manner, which is stated in Algorithm 11.
The convergence result is stated in Theorem F.1.
Theorem F.1 (Convergence rate of modified OFTRL) .
Algorithm 10 with η = 1 16 H guarantees that
[Equation]
where C is some absolute constant.
F.1 Proof of Theorem F.
In this section, we consider the following definitions of regret, which is slightly different from the definition in (7):
[Equation]
∈ ∆
A (cid:80) ti =
α it (cid:68) µ † − µ ih ( ·| s ) , (cid:104) Q ih ν ih (cid:105) ( s, · ) (cid:69) , reg th,ν ( s ) :
= max ν † ∈ ∆ B (cid:80) ti =
α it (cid:68) ν ih ( ·| s ) − ν † , (cid:104) ( Q ih ) ⊤ µ ih (cid:105) ( s, · ) (cid:69) , reg th,µ + ν :=
max s ∈S reg th,µ ( s ) + reg th,ν ( s ) .
We first prove that Q th and Q th upper and lower bounds Q ⋆h respectively.
Lemma F.1.
[Equation]
[Table] [Table]
[Table] [Table]
[Equation]
Proof.
We prove by induction on ( h, t ) .
Given the initialization, for t =
the condition holds.
Since
Q tH +1 , Q tH +1 = 0 , we have that for
h =
H + 1 the condition holds.
Assume that the condition hold for ( i, h + 1) , i ≤ t , then
[Equation]
Using similar strategy, we can also show that Q th ( s, a, b ) ≤ Q ⋆h ( s, a, b ) , which implies that the condition hold for ( t, h ) , and thus finishes the proof by induction.
Throughout the rest of this section, we define the following shorthand for the gap between Q th , Q th defined in (39):
[Equation]
Lemma F.2 (Recursion of δ th ) .
Algorithm 10 guarantees that for all ( t, h ) ∈
[ T ] ×
[ H ] , δ th ≤ t (cid:88) i = α it δ ih +
+ reg th +1 ,µ + ν .
Further, suppose that reg t h,µ + ν ≤ reg t h,µ + ν for all ( h, t ) ∈
[ H ] × [ T ] , where reg t h,µ + ν is nonincreasing in t : reg th,µ + ν ≥ reg t +1 h,µ
+ ν for all t ≥ 1 .
Then we have
[Equation]
Proof.
The proof structure resembles Lemma C.2.
From the definition of Q th , Q th , we have that
[Equation]
Then using the same argument as Lemma C
[Equation] [Equation]
we can consider an auxiliary sequence
[Equation]
Observe that { ∆ th } h,t satisfies the following properties
[Equation]
Therefore
[Equation]
to control δ th
[Equation]
it suffices to bound ∆ th ≤ 1 t (cid:80) ti =1 ∆ ih , which follows from the standard argument in [23]:
[Equation]
which completes the proof.
Lemma F.3 (Bound the NEGap by reg h,µ + ν ) .
Suppose that the per-state regrets (summing over the two agents) can be upper-bounded as reg t h,µ + ν ≤ reg t h,µ + ν for all
( h, t ) ∈
[ H ] × [ T ] where reg th,µ + ν is non-increasing in t : reg th,µ + ν ≥ reg t +1 h,µ + ν for all t ≥ 1 .
Then, the output policy ( (cid:98) µ T , (cid:98) ν T ) of Algorithm 10 satisfies
[Equation]
Proof.
From Lemma C.1 we have that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Lemma F.4 (Bound reg th,µ + ν ) .
Running Algorithm 10 with η = 1 16 H can guarantee that
[Equation]
Proof.
From Lemma B.3, substituting g t = w t Q th ν th ( s ) , M t = w t Q t − 1 h ν t − 1 h ( s ) , η t = ηw t , we can get that
T (cid:88) t = w t (cid:104)(cid:68) µ † , Q th ν th (cid:69) − (cid:68) µ th , Q th ν th (cid:69)(cid:105) ≤ w T log A η + η
T (cid:88) t = w t ∥
Q th ν th ( s ) −
[Equation]
Further we have that
[Equation]
From the definition of Q th we have that
[Equation]
Substitute this inequality to the regret bound we have reg
[Equation]
=
[Equation]
α
[Equation]
−
[Equation]
(Lemma A.3) ≤
[Equation]
s
[Equation]
Similar bound holds for reg
Th,ν : reg
[Equation]
∥
[Equation]
Summing reg Th,µ ( s ) , reg Th,ν ( s ) together we get reg
Th
[Equation]
Since α t − 1 T α tT ≥ 1 H for t ≥ 2 , by setting η = 1 16 H we can guarantee that 2 ηα tT − α t − 1 T 8 η ≤ 0 , thus reg Th,µ + ν ( s ) ≤ 2 α T log( A ∨ B )
η
[Equation] [Equation]
[Equation] [Equation]
A ∨
[Equation]
Given Lemma F.3 and F.4, we are now ready to prove Theorem F.1.
Proof Theorem F.1.
From Lemma F.3 and F.4 we have that:
[Equation]
which completes the proof
[Equation]
G Optimistic policy optimization for general-sum Markov Games G.1 Preliminaries
Here we formally present the preliminaries for multi-player general-sum Markov games, parallel to the zero-sum setting considered in Section 2.
Multi-player general-sum Markov games We consider tabular episodic (finite-horizon) m -player general-sum Markov games (MGs), which can be denoted as M ( H, S , {A i } mi =1 , P , { r i } mi =1 ) , where
H is the horizon length; S is the state space with |S| = S ;
A i is the action space of the i -th player, with |A
i | =
A i .
We use a = ( a 1 , . .
. , a m ) ∈ (cid:81) i ∈
[ m ] A i
=:
A to denote a joint action taken by all players; P = { P h } Hh =1 is the transition probabilities, where each P h ( s ′ | s, a ) gives the probability of transition to state s ′ from state-action ( s, a ) ; r i = { r i,h } Hh =1 are the reward functions, where each r i,h ( s, a ) is the deterministic reward function of the i -th player at time step h and state-action ( s, a ) .
In each episode, the MG starts with a deterministic initial state s 1 .
Then at each time step 1 ≤ h ≤ H , all players observes the state s h , each player takes an action a i,h ∈
A i .
Then, each player receive their rewards r
i,h ( s h , a h ) , and the game transitions to the next state s t +1 ∼ P h ( ·| s h , a h ) .
Policies & value functions
A (Markov) policy π i of the
i -th player is a collection of policies π i =
{ π i,h : S → ∆ A i } Hh =1 , where each π i,h ( ·| s h ) ∈ ∆
A i specifies the probability of taking action a i,h at ( h, s h ) .
We use π = { π i } i ∈ [ m ] to denote a product policy of all players.
For any joint policy π (not necessarily a product policy), we use V πi,h : S → R and Q πi,h : S ×
A → R to denote the ( i -th player’s) value function and Q-function at time step h , respectively, i.e.
[Equation]
For notational simplicity, we use the following abbreviation: [ P h V ]( s, a ) :
=
E s ′ ∼ P h ( ·| s, a ) V ( s ′ ) for any value function V .
By definition of the value functions and Q-functions, we have the following
Bellman equations for all Markov product policy π and all ( i, h, s, a ) :
[Equation]
The goal for the i -th player is to maximize their own value function.
Correlated policy & best response A (general) correlated policy π is any policy for which players may take actions in a history-dependent and correlated fashion.
More precisely, a correlated policy π is a mapping (cid:8) π h : Ω × ( S × A ) h − 1 × S → ∆ A (cid:9) , and executes as follows.
At the beginning of an episode, a random seed w ∈ Ω is sampled from some distribution (also denoted as Ω with slight abuse of notation).
Then, at each step h and state s h , suppose the history so far is ( s 1 , a 1 , . . .
, s h − 1 , a h − 1 ) .
Then, π samples a joint action a h ∼ π h ( ·| ω, ( s 1 , a 1 , . . .
, s h − 1 , a h − 1 ); s h ) .
This formulation allows each π h ( ·| ω, · , · ) to be a Markov product policy for any fixed ω while still making π to be a correlated policy, due to the correlation introduced by ω .
For any correlated policy π , let π − i denote the (marginal) policy of all but the i -th player.
Then, the ( i -th) player’s best-response value function is
[Equation]
where the max is over all (potentially history-dependent) policy π † i for the i -th player.
Coarse Correlated Equilibrium (CCE) For general-sum MGs, we consider learning an approximate Coarse Correlated Equilibrium [33, 49] defined as follows.
Definition G.1 ( ε -approximate Coarse Correlated Equilibrium) .
For any ε ≥ 0 , a correlated policy π is an ε -approximate Coarse Correlated Equilibrium ( ε -CCE) if
[Equation]
[Equation] [Equation]
[Equation]
Additional notation For any Q function Q i,h ( s, · ) : S × ( (cid:81) mi =1 A i )
→ R and joint policy π h ( ·| s ) , we use [ Q i,h π h ]( s ) :
= ⟨ Q i,h ( s, · ) , π h ( ·| s ) ⟩ for shorthand.
Similarly, for any joint policy π − i,h ( ·| s ) over all
but the i -th player, [ Q i,h π − i,h ]( s, a i ) :
= ⟨ Q i,h ( s, a i , · ) , π − i,h ( ·| s ) ⟩ .
G.2 Algorithm and formal statement of result
Require:
Product policies π t ′ h ′
( ·| s ′ )
= (cid:81) mi
=1 π t ′
i,h ′ ( ·| s ′
) for
all
( h ′
t ′
, s
′ ) ∈
[ H ] ×
[ T ] ∈ S .
1: Sample j ∈ [ t ] with probability P ( j = i ) =
α it .
2: Play policy π jh at the h -th step of the game.
3: Play policy (cid:98) π jh +1 for step h
+ 1 onward.
Theorem G.1 (Formal version of Theorem 4) .
Suppose Algorithm 12 is run for T rounds.
Then the per-state regret can be bounded as follows for some absolute constant C > 0 :
[Equation]
Further, choosing η = (log A max log T/ ( H 3 T )) 1 / 4 ( m − 1) − 1 / 2 , the output policy (cid:98) π T achieves
[Equation]
Proof overview and remarks The proof of Theorem G.1 also follows by relating the performance of the output policy by per-state regrets via performance difference (Lemma G.2, similar as Theorem 2), and bounding per-state regrets as reg th ≤ reg th := (cid:101)
O (1 / ( ηt ) + η 3 ( m − 1) 2 ) which gives the theorem.
The latter builds upon the fast convergence analysis of OFTRL in multi-player normal-form games [50] as well as additional handling of the changing game rewards, similar as in Theorem 3.
Note that the (cid:101) O
( T − 3 / 4 ) rate here is worse than (cid:101)
O ( T − 5 / 6 ) for the zero-sum setting in Theorem 3.
This happens as the fine-grained analysis of OFTRL [9] used there relies critically on the game having two players (for translating between the iterate stabilities and loss stabilities between each other), and becomes infeasible when there are more than 2 players.
We first present some lemmas in Section G.3.
The proof of Theorem G.1 is then provided in
Section G.4.
G.3 Useful lemmas
We additionally define the V-values maintained by Algorithm 12 as
[Equation]
for all ( i, h, t, s ) ∈ [ m ] × [ H ] × [ T ] × S , where Q ti,h and π th are the Q-functions and joint policies maintained within Algorithm 12.
Note that by (46)
[Equation]
we immediately have
[Equation]
We also define the value functions of (cid:98) π th and of its best response for any ( i, h, t, s ) as (see e
[Equation]
g. [49,
Definition C.4 & Eq.(8)]):
[Equation]
Lemma G.1 (Equivalence of value functions) .
For Algorithm 12, we have for all i ∈ [ m ] and all ( h, s, t ) ∈
[ H + 1] × S ×
[ T ] that
[Equation]
Proof.
We prove this by backward induction over h ∈ [ H + 1] .
The claim trivially holds for h = H + 1 .
Suppose the claim holds for steps h + 1 onward and all ( s, t ) ∈ S
× [ T ] .
For step h and any fixed ( s, t ) ∈ S × [ T ] , note that
[Equation]
Above, (i) follows by (48); (ii) uses the inductive hypothesis; (iii) uses the definition of the output policy (cid:98) π th (cf. Algorithm 13), which samples j ∈ [ t ] with probability α j t , plays π j h ( ·| s ) , and plays (cid:98) π jh +1 for the rest of the game.
This proves the case for step h and thus the lemma.
Define the weighted per-state regrets as
[Equation]
The following lemma bounds the difference between the values of the certified policy π th (Algorithm 13) and its best-response.
Lemma G.2 (Recursion of best-response values) .
For the policy (cid:98) π th defined in Algorithm 13, we have for all ( i, h, t ) ∈ [ m ] ×
[ H ] ×
[ T ] that
[Equation]
Proof.
Fix ( i, h, t ) ∈ [ m ] ×
[ H ] ×
[ T ] .
We have for any s ∈ S that
[Equation]
[Table]
[Equation]
Above, (i) follows by substituting V † , (cid:98) π j − i,
h +
i,h + with V (cid:98)
π
jh +
i,h +1 and paying the additive error; (ii) follows from Lemma G.1.
This proves the desired result.
Lemma G.3 (Guarantee of Algorithm 12 via per-state regrets) .
Suppose that the per-state regrets (50) can be upper-bounded as reg th ≤ reg th for all ( h, t ) ∈
[ H ] × [ T ] , where reg th is non-increasing in t : reg th ≥ reg t +1 h for all t ≥ 1 .
Then running Algorithm 12 will guarantee that
[Equation]
for all T ≥ 2 , where C > 0 is an absolute constant.
Proof.
For any ( h, t ) ∈ [ H + 1] ×
[ T ] , define
[Equation]
Then Lemma G.2 implies the recursive relationship
[Equation]
th
[Equation]
th
[Equation]
jt δ jh +
[Equation]
(With δ tH +1 ≡ 0 for all t ∈
[ T ] .)
Therefore we can imitate the proof of Lemma C.2 and obtain that, for any reg th such that reg th ≤ reg th and reg th ≥ reg t +1 h ,
[Equation]
where c β = 1 + 1 /H =
sup j ≥ 1 (cid:80) ∞ t =
j α jt by Lemma A.2(a).
Further, by definition of the output policy (cid:98) π T = (cid:98) π T 1 (cf.
Algorithm 13), we have
[Equation]
where C ≤ 3 as c H − 1 β ≤ (1 + 1 /H )
H ≤ e ≤ 3 .
This is the desired result.
G.4 Proof of Theorem G.
Bounding per-state regret We first bound reg ti,h ( s ) (cf.
definition in (49)), i.e. the per-state regret for the i -th player, for any fixed ( i, h, s, t ) ∈ [ m ] ×
[ H ] × [ S ] ×
[ T ] .
This is the main part of this proof.
Throughout this part, we will fix ( i, h, s, t ) , and omit the subscript ( h, s ) within the policies and Q functions, so that π ti,h ( ·| s ) will be abbreviated as π ti , and Q ti,h ( s, · ) will be abbreviated as Q ti .
We will also reload T ≥ 1 to be any positive integer (instead of the fixed total number of iterations).
We first observe that the update (45) for π ti,h ( ·| s ) is exactly equivalent to the OFTRL algorithm (Algorithm 4) with loss vectors g t = w t ( Q ti )
⊤ π t − i (understanding g 0 = 0 and Q 0 i = 0 ), prediction vector M t = w t ( Q t − 1 i )
⊤ π t − 1 − i , and learning rate η t = η/w t .
Therefore we can apply the regret bound for OFTRL in Lemma B.3 and obtain for any T ≥ 1 that
[Equation]
II .
[Equation]
Above, the last inequality uses the fact that (cid:13)(cid:13)
( Q ti − Q t − 1 i )
π t − i (cid:13)(cid:13) ∞ ≤ (cid:13)(cid:13)
Q ti − Q t − 1 i (cid:13)(cid:13) ∞ for t ≥ 1 , and (cid:13)(cid:13) Q t − 1 i ( π
[Equation]
(cid:13)(cid:13) 1 for t ≥ 2 .
For term I , noticing that (cid:13)(cid:13) Q ti − Q t − 1 i (cid:13)(cid:13) ∞ ≤ α t H by (46), we have
I ≤
[Equation]
(cid:88)
[Equation]
=
[Equation]
Bounding term II requires the following lemma on the stability of the iterates.
The proof can be found in Section G.5.
Lemma G.4 (Stability of iterates) .
We have for any i ∈ [ m ] and any t ≥ 2 that (recall the subscripts ( h, s ) are omitted below): (cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)
[Equation]
≤
[Equation]
Consequently, (cid:13)(cid:13)
π t − i − π t
[Equation]
[Equation]
(cid:13)(cid:13)
[Equation]
[Equation] [Equation]
Using Lemma G.
[Equation]
we have
[Equation]
Plugging the preceding bounds into (52) and using α 1 T · w t = α tT yields that
[Equation]
[Equation]
Above, (i) used the fact that α TT = α T
=
( H + 1) /
( H + T ) ≤ 2 H/T , and (cid:80) Tt =
α tT α 2 t ≤ 4 H/T by
Lemma A.3(c), and C ≤ 32 is an absolute constant.
This proves the per-state regret bounds claimed in Theorem G.1.
Overall policy guarantee Plugging the above per-state regret bounds into Lemma G.3 yields that, the output policy (cid:98) π T of Algorithm 12 achieves
[Equation]
Choosing η = (log A max log T/ ( H 3 T )) 1 / 4 ( m − 1) − 1 / 2 , the above can be upper bounded as
O (cid:16) H
[Equation] [Equation]
A
[Equation] [Equation]
+
[Equation] [Equation]
[Equation]
A
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
T
[Equation] [Equation]
(cid:17) , which is the desired result.
G.5 Proof of Lemma G.
We first prove (53).
By the OFTRL update (45) and the smoothness of exponential weights (Lemma A.5), we have for any t ≥ 2 that
[Equation]
where G t , G t − 1 are the (weighted) total losses in (45):
[Equation]
Therefore we have
[Equation]
Above, (i) uses the fact that Q ji π j − i ∈ [0 , H ] entry-wise for all j ≥ 1 , and (ii) uses Lemma A.4(b).
This proves (53).
The above directly implies (54) by the following bound on the TV distance (or L 1 norm) of product distributions [50]:
[Equation]
This completes the proof.
H Experimental details and additional studies H.1 Experimental details for Section
Details about the game The simulations in Section 5 is performed on the following two-playerzero-sum Markov game with H = 2 .
The state space at h = 1 only consists of a single state
S 1 = { s 0 } .
The state space at h = 2 consists of four different states S
= { s 11 , s 12 , s 21 , s 22 } .
The action spaces are the same for every state, namely A
= { a 1 , a 2 } , B = {
b 1 , b 2 } , i.e. each player has two actions.
The transition from S 1 × A × B → S 2 is deterministic, which takes the following form:
[Equation]
The instantaneous reward r h depends only on the action (and not the state), i.e., r h : A × B → [0 , 1] , which takes values as (scaled) identity matrices:
[Equation]
Direct calculation yields that the Nash values and policies for this game is given by
V ⋆
[Equation]
( s ) = 0 .
5 for s ∈ { s 11 , s 12 , s 21 , s 22 } , Q
[Equation]
=
[Equation]
(cid:21)
[Equation]
( ·| s 0 )
= (cid:20) 0 .0 .(cid:21) , V ⋆ 1 ( s 0 )
= 0 .
Initialization All algorithms in Figure 1 use the following initialization ( µ 0 , ν 0 ) :
At h = 1 : µ 01 ( a 1 | s 0 )
= 0 .
3 , µ 01 ( a 2 | s 0 ) = 0 .
7 , ν 01 ( b 1 | s 0 )
= 0 .
7 , ν 01 ( b 2 | s 0 )
= 0 .
At h = 2 : µ 02 ( a 1 | s 11 ) = 0 .
248 , µ 02 ( a 2 | s 11 ) = 0 .
752 , ν 02 ( b 1 | s 11 )
= 0 .
248 , ν 02 ( b 2 | s 11 )
= 0 .
752; µ 02 ( a 1 | s 12 ) = 0 .
500 , µ 02 ( a 2 | s 12 ) = 0 .
500 , ν 02 ( b 1 | s 12 ) = 0 .
168 , ν 02 ( b 2 | s 12 ) = 0 . 832; µ 02 ( a 1 | s 21 ) = 0 .
500 , µ 02 ( a 2 | s 21 ) = 0 .
500 , ν 02 ( b 1 | s 21 ) = 0 .
168 , ν 02 ( b 2 | s 21 ) = 0 . 832; µ 02 ( a 1 | s 22 ) = 0 .
752 , µ 02 ( a 2 | s 22 ) = 0 .
248 , ν 02 ( b 1 | s 22 )
= 0 .
248 , ν 02 ( b 2 | s 22 )
= 0 .
Standard FTRL (Algorithm 3) and OFTRL (Algorithm 4) by default uses the uniform distribution as the initialization, as it minimizes the (neg)entropy Φ( · ) .
To make them initialized at µ 0 , we change the regularizers for µ h ( ·| s ) to be KL ( ·∥ µ 0 h ( ·| s )) for the max-player.
(And similarly KL ( ·∥ ν 0 h ( ·| s )) for the min-player.)
Note that our actual initialization above satisfies the property that all its values are bounded within the interval [0 . 15 , 0 . 85] .
In particular, KL ( µ ′ ∥ µ 0 h ( ·| s )) for any other µ ′ ∈
∆ A is bounded by O (log(1 / 0 . 15))
=
O (1) , and thus all the convergence theorems will still hold with this modified regularizer, with at most a larger (multiplicative) constant than with the Φ( · ) regularizer.
Remark on runtime Running all our experiments takes approximately 6.46 hours CPU running time (Intel(R) Core(TM) i5-8250U CPU).
H.2 Additional visualizations for the INPG algorithm
Figure 1 shows that the INPG algorithm (with η = 1 / √ T ) appears to converge much slower than O ( T − 1 / 2 ) (which is the rate for FTRL with η = 1 / √ T ).
Here we present some further understandings of this phenomenon by visualizing the optimization trajectories of the INPG algorithm.
[Figure]
Figure 2a shows the evolution of the value functions at h = 2 over iteration step t , for the last 5 × 10 4 steps.
For all four states, the policy optimization is equivalent to Hedge on the matrix game with identity reward matrix and thus exhibits an expected cyclic behavior and leads to the sinusoidal-like curves shown in Figure 2a.
However, due to the choice of our specific initialization ( µ 0 , ν 0 ) , the four curves behave like the same periodic curves with different “phases” .
Figure 2b shows the evolution of the policy at h = 1 (specifically, µ t 1 ( a 1 | s 0 ) which is the probability of the max-player taking action a 1 ) over t , for the last 4 × 10 5 steps.
(The result for the min-player is similar.)
The curve also behaves periodically, and appears to be a superposition of two waves, one main waive with larger magnitude and period, and another oscillation with smaller magnitude and period.
Qualitatively, the main wave is caused by the intrinsic cyclic behavior of learning with respect to the (fixed) reward at the h = 1 , while the oscillation is caused by the changing reward that is backed-up from h = 2 .
Further, as the reward in the second layer has much higher magnitude than the first layer in this game, the oscillation has a non-negligible magnitude.
The horizontal line in Figure 2b plots the final output policy (cid:98) µ T 1 ( a 1 | s 0 )
≈ 0 .
52 , which we recall is the average of µ t 1 ( a 1 | s 0 ) over the entire run t ∈ [ T ] (cf.
Section 5).
Note that the unique Nash equilibrium satisfies µ ⋆ 1 ( a 1 | s 0 ) = 0 .
5 (56), and the error (cid:98) µ T 1 ( a 1 | s 0 ) − µ ⋆ 1 ( a 1 | s 0 ) ≈ 0 .
We suspect that this may be an intrinsic bias caused by the aforementioned correlation between the two layers’ learning processes (in particular, the different “phases” of the second-layer’s learning over the four states), and may also be the cause of the slow convergence for INPG shown in Figure 1.
H.3 Additional theoretical justifications
INPG as an instantiation of Algorithm
Here we show why the instantiation of Algorithmwith β t = 1 and
[Equation]
considered in Section 5 is equivalent to the Independent Natural Policy Gradient (INPG) algorithm.
Indeed, choosing β t = 1 in Algorithm 1 ensures that Q th = Q µ t ,ν t h (the true value function of ( µ t , ν t ) ).
Therefore, the above update is equivalent to
[Equation]
This is exactly an independent two-player version of the Natural Policy Gradient algorithm
(e.g. [1]), where each player plays an NPG algorithm as if they are facing their own Markov Decision Process, with the opponent fixed.
NEGap-Layer-1 lower bounds NEGap
Here we show NEGap-Layer-1 ( µ, ν ) ≤ NEGap( µ, ν ) for any ( µ, ν ) .
From the definition of V ⋆h we have that
[Equation] [Equation]
Thus
[Equation]
Thus for our example
[Equation]
I Additional details I.1 Learning setting
In this paper we consider the full-information setting for learning Markov Games formally defined via the following oracle:
Given any h ∈
[ H ] , policies µ h
+1 = { µ h +1 ( ·| s
) ∈ ∆ A } s ∈S , ν h
+1 =
{ ν h +1 ( ·| s
) ∈ ∆ B } s ∈S
[Equation]
and V function V h +1 ∈ R S , we can query the exact value of
[Equation]
[Equation]
Note that our algorithm framework (Algorithm 1) and all its subsequent instantiations can execute under the above oracle: Each iteration of Algorithm 1 makes H queries to this oracle, one for each h ∈
[ H ] with value function V h
+1 =
( µ th +1 )
⊤ Q th +1 ν th +1 (cf.
For the purpose of comparing policy optimization type algorithms, we also consider the following weaker oracle:
Given any h ∈
[ H ] , policies µ h
= { µ h ( ·| s
) ∈
∆ A } s ∈S , ν h
= { ν h ( ·| s ) ∈
∆ B } s ∈S and V function V h +1 ∈ R S , we can query the expected value of its one-step backup under each player’s policy:
[Equation]
This oracle is also considered in [53, Section 3].
It is clear that one query to (57) can implement one query to (58), and thus (58) is a weaker oracle.
I.2 Comparison of algorithms
We first remark that all algorithms considered in Section 3.2, 4, & 5 fall into the unified framework of
Algorithm 1 and thus are implementable using the full-information oracle (57).
Second, most algorithms we consider: Nash V-Learning (Example 1), GDA (Example 2), OFTRL (Algorithm 4 & Theorem 3), as well as INPG (Section 5) are policy optimization type algorithms which can be implemented using the weaker oracle (58) (see implementation details in Algorithm 5,6,9).
In this paper, we are interested in comparing the rates of the above algorithms, where OFTRL achieves a faster (cid:101)
O ( T − 5 / 6 ) rate, and Nash V-Learning and GDA achieve the standard (cid:101)
O
( T − 1 / 2 ) rate.
(The convergence rate of INPG in theory is still open.)
By contrast, Nash Q-Learning (Example 3) and Nash Policy Iteration (Example 4) are not typically considered as policy optimization algorithms, and they cannot be implemented using (58).
Our unified framework (Algorithm 1) allows using (57) and thus encapsulates these two algorithms, but due to the usage of a stronger oracle, their convergence rates are not comparable with the aforementioned policy optimization algorithms.