Policy Optimization for Markov Games: Unified Framework and Faster Convergence
Anonymous Author(s)
Abstract
This paper studies policy optimization algorithms for multi-agent reinforcementlearning.
We begin by proposing an algorithm framework for two-player zero-sum
Markov Games in the full-information setting, where each iteration consists of apolicy update step at each state using a certain matrix game algorithm, and a valueupdate step with a certain learning rate.
This framework unifies many existing andnew policy optimization algorithms.
We show that the state-wise average policy ofthis algorithm converges to an approximate Nash equilibrium (NE) of the game,as long as the matrix game algorithms achieve low weighted regret at each state,with respect to weights determined by the speed of the value updates.
Next, weshow that this framework instantiated with the Optimistic Follow-The-RegularizedLeader (OFTRL) algorithm at each state (and smooth value updates) can find an(cid:101) O ( T − 5 / 6 ) approximate NE in T iterations, which improves over the current best(cid:101)
O ( T − 1 / 2 ) rate of symmetric policy optimization type algorithms.
We also extendthis algorithm to multi-player general-sum Markov Games and show an (cid:101)
O
( T − 3 / 4 )convergence rate to Coarse Correlated Equilibria (CCE).
Finally, we provide anumerical example to verify our theory and investigate the importance of smoothvalue updates, and find that using “eager" value updates instead (equivalent to theindependent natural policy gradient algorithm) may significantly slow down theconvergence, even on a simple game with H = 2 layers.
Introduction
Policy optimization, i.e. algorithms that learn to make sequential decisions by local search onthe agent’s policy directly, is a widely used class of algorithms in reinforcement learning [38, 42,43].
Policy optimization algorithms are particularly advantageous in the multi-agent reinforcementlearning (MARL) setting (e g. compared with value-based counterparts), due to their typicallylower representational cost and better scalability in both training and execution.
A variety of policyoptimization algorithms such as Independent PPO [13], MAPPO [53], QMix [40] have been proposedto solve real-world MARL problems [4, 37, 41].
These algorithms share a same high-level structurewith iterative value updates (for certain value estimates) and policy updates (often independentlywith each agent) using information from the value estimates and/or true rewards.
While policy optimization for MARL has been studied theoretically in a growing body of work,there are still gaps between algorithms used in practice and provably-efficient algorithms studied intheory—Algorithms in practice generally follow two natural design principles: symmetric updatesamong all agents, and simultaneous learning of values and policies [56, 53].
By contrast, policyoptimization algorithms studied in theory often diverge from these principles and incorporate sometweaks, such as (i) asymmetric updates, where one agent takes a much smaller learning rate than theothers (two time-scale) [11] or waits until the other agents learn an approximate best response [59] and (ii) batch-like learning, where policies are optimized to sufficient precision with respect to thecurrent value estimate before the next value update [8].
There is so far a lacking of systematic studieson the performance of the more vanilla policy optimization algorithms following the above twoprinciples, even under the setting where full-information feedback from the game is available.
Towards bridging these gaps, this paper studies policy optimization algorithms for Markov games,with a focus on algorithms with symmetric updates and simultaneous learning of values and policies.
Our contributions can be summarized as follows:
• We propose an algorithm framework (Section 3) for two-player zero-sum Markov games in thefull-information setting.
This framework unifies many existing and new policy optimizationalgorithms such as Nash V-Learning, Gradient Descent/Ascent, as well as seemingly disparatealgorithms such as Nash Q-Learning (Section 3.2).
We prove that the state-wise average policyoutputted by the above algorithm is an approximate Nash Equilibrium (NE), so long as suitableper-state weighted regrets are bounded (Section 3.1).
This generic result can be instantiated in amodular fashion to derive convergence guarantees for the many examples above.•
We instantiate our framework to show that a new algorithm based on Optimistic Follow-TheRegularized-Leader (OFTRL) and smooth value updates finds an (cid:101)
O
( T − 5 / 6 ) approximate NEin T iterations (Section 4).
This improves over the current best rate of (cid:101)
O
( T − 1 / 2 ) achievedby symmetric policy optimization type algorithms.
We additionally extend the above OFTRLalgorithm to multi-player general-sum Markov games and show an (cid:101)
O
( T − 3 / 4 ) convergence rateto Coarse Correlated Equilibria (CCE), which is also the first rate faster than (cid:101) O
( T − 1 / 2 ) forpolicy optimization in general-sum Markov games (Section 4.1).• We perform simulations on a carefully constructed zero-sum Markov game with H = 2 layers toverify our convergence guarantees.
The numerical tests further suggest the importance of smoothvalue updates : the Independent Natural
Policy Gradient algorithm (as one instantiation of ouralgorithm framework with “eager” value updates) appears to converge much slower (Section 5).
1.1 Related work
Two-player zero-sum MGs Markov games (MGs)
[28] (also known as Stochastic Games [45])is a widely studied model for multi-agent reinforcement learning.
In the most basic setting of twoplayer zero-sum MGs, algorithms for computing the NE have been extensively studied in both thefull-information setting [29, 19, 18] and the sample-based/online setting [6, 49, 21, 46, 55, 2, 52,3, 31, 24, 20, 54, 10, 32].
Our algorithm framework incorporates (the full-information version of)several algorithms in this line of work.
Policy optimization for zero-sum MGs Policy optimization for single-agent Markov Decision
Processes has been extensively in a recent line of work, e.g. [1, 5, 30, 44, 35, 33, 7, 14, 51] andthe many references therein.
For two-player zero-sum MGs, the Nash V-Learning algorithm of Baiet al.
[3] (originally proposed for the sample-based online setting) can be viewed as an independentpolicy optimization algorithm, and can be adapted to the full-information setting with (cid:101)
O ( T − 1 / 2 )convergence rate.
Daskalakis et al.
[11] prove that the independent policy gradient algorithm withan asymmetric two time-scale learning rate can learn the NE (for one player only) with polynomialiteration/sample complexity.
Zhao et al.
[59] show that another asymmetric algorithm that simulatesa policy gradient/best response dynamics converges to NE with (cid:101) O
(1 /T ) rate.
Cen et al.
[8] usea symmetric optimistic (extragradient) subroutine for matrix games to learn zero-sum MGs in alayer-wise fashion (the matrix games at each state are learned to sufficient precision before thebackup, more like a Value Iteration type algorithm), and also derive an (cid:101) O (1 /T ) convergence rate.
The closest to our work is Wei et al.
[50] which proves that Optimistic Gradient Descent/Ascent(OGDA), combined with smooth value updates at all layers simultaneously, converges to an NEwith (cid:101)
O ( T − 1 / 2 ) rate for both the average duality gap and the last iterate.
The fast (cid:101)
O
( T − 5 / 6 ) rateof our OFTRL algorithm improves over [50] and is the first such faster rate for symmetric, policyoptimization type algorithms.
Multi-player general-sum MGs A recent line of work shows that a generalization of the Vlearning algorithm to the multi-player general-sum setting can learn Coarse Correlated Equilibria(CCE) [47, 23, 34] and Correlated Equilibria (CE) [47, 23].
The algorithmic designs in these works are specially tailored to the sample-based setting, where the best possible rate is (cid:101) O
( T − 1 / 2 ) .
1 Incontrast, this paper considers the full-information setting and proposes new algorithms achievingthe faster (cid:101)
O
( T − 3 / 4 ) for learning CCE in general-sum MGs.
Another recent line of work considerslearning NE in Markov Potential Games [57, 27, 47, 15, 58], which can be seen as a cooperative-typesubclass of general-sum MGs.
Optimistic algorithms in normal-form games Technically, our accelerated rates build on therecent line of work on faster rates for optimistic no-regret algorithms in normal-form games [48,39, 9, 12].
Specifically, our ˜ O ( T − 5 / 6 ) rate for two-player zero-sum MGs builds upon a first-ordersmoothness analysis of [9], and our ˜ O ( T − 3 / 4 ) rate for multi-player general-sum MGs follows fromthe RVU-property [48, Definition 3].
Our incorporation of these techniques involves non-trivial newcomponents such as weighted first-order smoothness bounds and handling changing game rewards.
Preliminaries
We consider the tabular episodic (finite-horizon) two-player-zero-sum Markov games (MGs), whichcan be denoted as M ( H, S , A , B , P , r ) , where H is the horizon length; S is the state space with |S| =
S ; A , B are the action space of the max-player and min-player respectively, with |A| =
A, |B| =
B ; 103 P = { P h } Hh =1 is the transition probabilities, where each P h ( s ′ | s, a, b ) gives the probability oftransition to state s ′ from state-action ( s, a, b ) ; r = { r h } Hh =1 are the reward functions, such thatr h ( s, a, b ) is reward 2 of the max-player and − r h ( s, a, b ) is the reward of the min-player at time steph and state-action ( s, a, b ) .
In each episode, the MG starts with a deterministic initial state s 1 .
Then ateach time step 1 ≤ h ≤ H , both players observes the state s h , the max-player takes an action a h ∈ A ,and the min-player takes an action b h ∈ B .
Then, both players receive their rewards r h
( s h , a h , b h )and − r h ( s h , a h , b h ) , respectively, and the system transits to the next state s t +1 ∼ P h ( ·| s h , a h , b h ) .
Policies & value functions
A (Markov) policy µ of the max-player is a collection of policiesµ = { µ h : S → ∆ A } Hh =1 , where each µ h ( ·| s h ) ∈
∆
A specifies the probability of taking action a h at( h, s h ) .
Similarly, a (Markov) policy ν of the min-player is defined as ν
= {
ν h : S → ∆ B } .
For anypolicy ( µ, ν ) (not necessarily Markov), we use V µ,νh : S → R and Q µ,νh : S ×
A × B → R to denotethe value function and Q-function at time step h , respectively, i.e.
[Equation]
For notational simplicity, we use the following abbreviation: [ P h V ]( s, a, b ) :
=
E s ′ ∼ P h ( ·| s,a,b ) V ( s ′ )for any value function V .
By definition of the value functions and Q-functions, we have the following
Bellman equations
[Equation]
The goal for the max-player is to maximize the value function, whereas the goal for the min-player isto minimize the value function.
Best response & Nash equilibrium For any Markov policy µ of a max-player, there exists a bestresponse for the min-player, which can be taken as a Markov policy ν † ( µ ) such that V µ,ν † ( µ ) h ( s )
=
inf ν V µ,νh ( s ) for all ( s, h ) ∈ S
×
[ H ] .
For simplicity we define V µ, † h :=
V µ, [Equation] † ( µ ) h .
By symmetry, wecan also define µ † ( ν ) and V † ,νh .
It is known (e.g. [16]) that there exist Markov policies ( µ ⋆ , ν ⋆ ) thatperform optimally against best responses.
These policies are also equivalent to Nash Equilibria (NEs)of the game, where no player can gain by switching to a different policy unilaterally.
It can also beshown that any NE ( µ ⋆ , ν ⋆ ) satisfies the following minimax equation
[Equation]
[Table]
[Table]
s ))
[Table]
[Table]
(cid:98)
[Equation] [Equation]
(cid:80)
[Equation] [Equation]
, (cid:98)
[Equation]
( ·|
[Equation]
[Table]
=
[Table]
th
·| s [Equation] s )
[Table]
H ] × S .
Thus, while the NE policy ( µ ⋆ , ν ⋆ ) may not be unique, all of them share the same value functions,which we denote as V ⋆h :=
V µ ⋆ ,ν ⋆ h .
The Q-function Q ⋆h can be defined similarly.
In this paper, ourmain goal is to find an approximate NE, which is formally defined below.
Definition 1 ( ε -approximate Nash Equilibrium) .
For any ε ≥ 0 , a policy ( µ, ν ) is an ε -approximate
Nash Equilibrium ( ε -NE) if NEGap( µ, ν ) :
= V † ,ν 1 ( s 1 ) − V µ, † 1 ( s 1 ) ≤ ε.
Additional notation For any ( h, s ) ∈
[ H ] × S and Q function Q h : S ×
A × B → R , we defineshorthand [( µ h )
⊤ Q h ν h ]( s ) :
= ⟨ Q h ( s, · , · ) , µ h ( ·| s ) ×
ν h ( ·| s ) ⟩ for any policy ( µ, ν ) .
Similarly, welet [ Q h ν h ]( s, · ) :=
E b ∼ ν h ( ·| s )
[ Q h ( s, · , b )] ∈ R A , and [ Q ⊤ h µ h ]( s, · ) :=
E a ∼ µ h ( ·| s )
[ Q h ( s, a, · )] ∈ R B .
We use A ∨ B :=
max { A, B } .
An algorithm framework for zero-sum Markov games
We begin by presenting an algorithm framework that unifies many existing and new algorithms fortwo-player zero-sum Markov Games, and its performance guarantee that could be specialized to yieldconcrete convergence results for many specific algorithms.
Our algorithm framework, described in Algorithm 1, consists of two main components: the policyupdate step computing policies ( µ t , ν t ) , and the value update step computing the Q estimate Q th ’s.
Policy update via matrix game algorithms
In the policy update step (Line 5), for each ( h, s ) ,the two players update policies ( µ th ( ·| s ) , ν th ( ·| s )) at ( h, s ) using some matrix game algorithm
MatrixGameAlg which takes as input all past Q matrices and all past policies of both players.
The MatrixGameAlg offers a flexible interface that allows many choices such as the matrix NEsubroutine over the most recent Q matrix MatrixNE ( Q t − 1 h ( s, · , · )) , or any independent no-regret algorithm (for both players), such as Follow-The-Regularized-Leader (FTRL) (9) or projected Gradient
Descent-Ascent (10) considered in the examples later.
Value update with learning rate { β t } For any ( h, s, a, b ) , the value update step (Line 6) updates
Q th ( s, a, b ) by the newest value function r h + P h
[( µ th +1 )
⊤ Q
th +1 ν th +1 ] propagated from layer h
+ 1 ,using a sequence of learning rates { β t } t ≥ 1 which we assume to be within [0 , 1] (with β 1 := 1 ).
{ β t }controls the speed of the value update, with two important special cases:
(1) Eager value updates, where we set β t = 1 so that Q th performs policy evaluation of the current
policy ( µ t , ν t ) , that is, Q th = Q µ t ,ν t h .
(2) Smooth (incremental) value updates, where we choose β t → 0 as t → ∞ .
In this case, the Q t hmoves slower (resembling a critic in Actor-Critic like algorithms), and becomes a weighted
average of all past updates.
A standard choice that is frequently used is from [22] (and many
subsequent work),
[Equation]
For any { β t } , the update (3) implies that
[Equation]
where β it ’s are a group of weights summing to one ( (cid:80) ti =
1 β it = 1 ) defined as
[Equation]
Note that with smooth value updates ( β t < 1 ), Q th is not necessarily the Q-function of any policy.
Upon finishing, the algorithm outputs the state-wise average policy ( (cid:98) µ T , (cid:98) ν T ) defined in (4), whereeach (cid:98) µ Th ( ·| s ) is the weighted average of µ th ( ·| s ) using weights { β tT } Tt =1 (and similarly for (cid:98) ν T ).
Symmetric & simultaneous learning, (de)centralization We remark that Algorithm 1 by definition performs simultaneous learning (of policies and values) at all layers, and also yields symmetric(policy) updates if MatrixGameAlg is a symmetric algorithm with respect to µ and ν .
Also, although
Algorithm 1 appears to be a centralized algorithm as it maintains Q values in (3), this does notpreclude possibilities that the algorithm can be executed in a decentralized fashion.
This can happene.g.
when the Q-update (3) can be rewritten as an equivalent V-update (cf.
Example 1 & 2).
3.1 Theoretical guarantee
We are now ready to state the main theoretical guarantee of Algorithm 1, which states that the( (cid:98) µ T , (cid:98) ν T ) is an approximate NE, as long as the algorithm achieves low per-state weighted regrets w.r.t.weights { β it } ti =1 , defined as
[Equation]
Theorem 2 (Main guarantee of Algorithm 1) .
Suppose that the per-state regrets can be upperbounded as reg th ≤ reg th for all ( h, t ) ∈
[ H ] × [ T ] , where reg th is non-increasing in t : reg th ≥ reg t +1 hfor all t ≥ 1 .
Then, the output policy ( (cid:98) µ T , (cid:98) ν T ) of Algorithm 1 satisfies
[Equation]
for all T ≥ 2 and some absolute constant C > 0 , where c β is a constant depending on { β t } t ≥ 1 :c β := sup j
[Equation]
(cid:80)
[Equation]
[Equation]
Specifically, c β = (cid:0) 1 + 1 H (cid:1) if β t = α t =
H +1 H + t , and c β = 1 if β t = 1 .
Bound (7) is typically dominated by the second term on the right hand side, suggesting that the
NEGap can be bounded by the average weighted regret (cid:101) O (cid:16) 1 T (cid:80)
Tt =1 max h reg th (cid:17) , if c Hβ = O (1) .
Theorem 2 serves as a modular tool for analyzing a broad class of algorithms: As long as this averageregret is sublinear in T (including—but not limited to—choosing MatrixGameAlg as uncoupledno-regret algorithms), the output policy will be an approximate NE.
We emphasize though that thisresult is not yet end-to-end, as each reg th is a weighted regret w.r.t.
the particular set of weights 185 (cid:8) β it (cid:9) t i
=1 , minimizing which may require careful algorithm designs and/or case-by-case analyses.
Weprovide some concrete examples in Section 3.2 to demonstrate the usefulness of Theorem 2.
We remark that the state-wise average policy considered in Theorem 2 is an average policy thatis also Markovian by definition, which is different from existing work which considers either the(Markovian) last iterate [50] or non-Markovian average policies (e.g. [3]).
However, this guaranteerelies on full-information feedback (so that per-state regret bounds are available), and it remains anopen question how such guarantees could be generalized to sample-based settings.
Proof overview The proof of Theorem 2 follows by (1) bounding NEGap( (cid:98) µ T , (cid:98) ν T ) in terms of perstate regrets w.r.t.
the Nash value functions Q ⋆h ’s by performance difference arguments (Lemma C.1);(2) recursively bounding the value estimation error
δ th := ∥
Q th − Q ⋆h ∥ ∞ (Lemma C.2) which yieldsthe constant c β ; and (3) combining the above to translate the regret from Q ⋆h ’s to Q th ’s (which weassume to be bounded by reg th ) and obtain the theorem.
The full proof can be found in Appendix C.
3.2 Examples
We now demonstrate the generality of Algorithm 1 and Theorem 2 by showing that they subsumemany existing algorithms (and yield new algorithms) for two-player-zero-sum Markov games, andprovide new guarantees with the particular output policy (4).
Example 1 (Nash V-Learning [3], full-information version) :
The full algorithm (Algorithm 5)can be found in Appendix D.1.
The algorithm is a special case of Algorithm 1 with β t = α t =( H + 1) /
( H + t ) , and MatrixGameAlg chosen as the weighted FTRL algorithmµ th ( a | s ) ∝ a exp (cid:32)
η w t −t [Equation] 1 (cid:88) i =
[Equation]
where w t := α t t /α 1 t .
Combining Theorem 2 with the standard regret bound of weighted FTRL, thisalgorithms achieves NEGap( (cid:98) µ T , (cid:98) ν T ) ≤ (cid:101)
O
( H 7 / 2 / √ T ) choosing η ≍ 1 / √ T (Proposition D.2).
Additionally, although the original Nash V-learning algorithm [3] updates the V values (which makesthe algorithm implementable in a decentralized fashion) instead of the Q values used in Algorithm 1,these two forms are actually equivalent in the full-information setting (Proposition D.1).
♢
Example 2 (GDA-Critic) :
This algorithm is a special case of Algorithm 1 with β t = α t =( H + 1) /
( H + t ) , and MatrixGameAlg as projected gradient descent/ascent (GDA), i.e.,
[Equation]
Similar as Nash V-Learning, GDA-Critic also admits an equivalent form with V value updates (fulldescription in Algorithm 6).
As GDA achieves weighted regret bounds with any monotone weightsincluding (cid:8) α it (cid:9) t i =
(Lemma B.1), we can invoke Theorem 2 to show that this algorithm achieves
NEGap( (cid:98) µ T , (cid:98) ν T ) ≤ (cid:101) O
( H 7 / 2 ( A ∨ B ) 1 / 2 / √ T ) if we choose η ≍ 1 / √ T (Proposition D.4).
The GDA-critic algorithm is also similar to the OGDA-MG algorithm of Wei et al.
[50], except thatwe use the (non-optimistic) vanilla version of GDA.
To our best knowledge, the above algorithm andguarantee are not known.
We remark that even ignoring difference between GDA and OGDA, theabove guarantee cannot be obtained by direct adaptation of the results of [50] which focus on eitherthe average duality gap and/or last-iterate convergence.
♢
Besides the above examples, Algorithm 1 also incorporates the following algorithms which aretypically not categorized as policy optimization algorithms.
Example 3 (Nash Q-Learning [19, 3], full-information version) :
This algorithm is a special case of
Algorithm 1 with β t = α t =
( H + 1) /
( H + t ) and MatrixGameAlg as the matrix Nash subroutine
[Equation]
(Full description in Algorithm 7.)
Although MatrixNE ( Q t − 1 h ( s, · , · )) is not by default a no-regretalgorithm, using the fact that (cid:13)(cid:13)
Q th − Q t − 1 h (cid:13)(cid:13) ∞ is small (due to the small α t ) we can show that it isclose to a (hypothetical) “Be-The-Leader” style algorithm that computes the matrix NE of the current
Q matrix Q th which achieves ≤ 0 regret (Lemma D.3).
Combining this with Theorem 2 shows thatthis algorithm achieves NEGap( (cid:98) µ T , (cid:98)
ν T ) ≤ (cid:101) O
( H 4 /T )
(Proposition D.5).
♢
Example 4 (Nash Policy Iteration (Nash-PI)) :
This classical algorithm (Algorithm 8) performsiterative policy evaluation and policy improvement (also similar to Nash Value Iteration [45, 2, 31]):
[Equation]
) ← MatrixNE
[Equation] [Equation]
This is also a special case of Algorithm 1 with β t = 1 and MatrixGameAlg set as MatrixNE .
It is astandard result that this algorithm converges exactly (achieving zero NE gap) in H steps, and thisfact can be obtained using our framework as well (Proposition D.6).
♢
Fast convergence of optimistic FTRL
In this section, we instantiate Algorithm 1 by choosing MatrixGameAlg as the Optimistic FollowThe-Regularized-Leader (OFTRL) algorithm.
OFTRL is also an uncoupled no-regret algorithmthat is known to enjoy faster convergence than standard FTRL under additional loss smoothnessassumptions [39, 48, 9, 12].
We show that, using OFTRL, Algorithm 1 enjoys faster convergencethan the (cid:101)
O
(1 / √ T ) rate of using FTRL or GDA (cf.
Example 1 & 2).
Concretely, we use the following weighted OFTRL algorithm at each ( h, s, t ) :
[Equation]
where w t is the same weights as defined in Example 1, and we choose β t = α t
= ( H + 1) /
( H + t ) .
Theorem 3 (Fast convergence of OFTRL in zero-sum Markov Games) .
Suppose Algorithm 1 isinstantiated with β t = α t
=
( H + 1) /
( H + t ) and MatrixGameAlg to be the OFTRL algorithm (12)with any η ≤ 1 /H (full description in Algorithm 9).
Then the per-state regret can be bounded asfollows for some absolute constant C > 0 :
[Equation]
6 (cid:21) for all ( h, t
[Equation]
∈ [ H ]
[Equation]
[ T ] .
[Equation]
Further, choosing η = poly( H, log( A ∨ B ) , log T ) · T − 1 / 6 , the output (state-wise average) policy( (cid:98) µ T , (cid:98) ν T ) achieves approximate NE guarantee [Equation]
[Equation]
To our best knowledge, the (cid:101) O
( T − 5 / 6 ) rate asserted in Theorem 3 is the first rate faster than thestandard (cid:101) O
(1 / √ T ) for symmetric, policy optimization type algorithms in two-player zero-sum
Markov games.
The closest existing result to this is of Wei et al.
[50], who analyze the OGDAalgorithm with smooth value updates and show a (cid:101) O
(1 / √ T ) convergence of both the average NEGapand the NEGap of the last-iterate.
However, these only imply at most a (cid:101) O
(1 / √ T ) rate for theaverage policies, and not our faster rate 3 .
Cen et al.
[8], Zhao et al.
[59] show (cid:101) O
(1 /T ) convergenceof algorithms with optimistic gradient-based policy updates, which are however very different stylesof algorithms that either performs layer-wise learning (the matrix games at each state are learned tosufficient precision before the backup) similar as Value Iteration [8], or uses strongly asymmetricupdates that simulate a policy gradient-best response dynamics [59].
By contrast, our Algorithm 9runs symmetric no-regret dynamics for both players, simultaneously at all layers.
Proof overview The proof of Theorem 3 (deferred to Appendix E) builds upon the recent line ofwork on fast convergence of optimistic algorithms [39, 48, 9], in particular the work of Chen and
Peng [9] which shows an (cid:101) O
( T − 5 / 6 ) convergence rate of OFTRL for two-player normal-form games.
Our regret bound (13) generalizes this result non-trivially by additionally handling (1) The weightedregret, which requires bounding the weighted stability of the OFTRL iterates by a new analysis ofthe potential functions (Lemma B.4), and (2)
The errors induced by changing game matrices , as
Q th ( s, · , · ) changes over t .
Plugging (13) into Theorem 2 yields the policy guarantee (14).
We also remark that, while OFTRL for zero-sum matrix games admits an improved (cid:101)
O ( T − 1 ) rate [39,48] by an (cid:101)
O ( T − 1 ) bound on the summed regret over the two players , that analysis is not yet applicablein our Markov game setting, as our analysis requires bounded individual regrets for each playerwhich is not implied by bounds on their summation.
4.1 Extension to multi-player general-sum Markov games
Our fast convergence result can be extended to the more general setting of multi-player general-sum
Markov games.
Concretely, we consider general-sum Markov games with m ≥ 2 players, S states,
H steps, where the i -th player has action space
A i with A max := max i ∈ [ m ] |A
i | and her own rewardfunction.
The goal is to find a correlated policy over all players that is an approximate Coarse
Correlated Equilibrium (CCE) of the game (see Appendix F.1 for the detailed setup).
We show that the OFTRL algorithm works for general-sum Markov games as well, with a fast(cid:101)
O ( T − 3 / 4 ) convergence to CCE.
The formal statement and proof is in Theorem F.1 & Appendix F.4.
Theorem 4 (Fast convergence of OFTRL in general-sum Markov Games; Informal version of Theorem F.1) .
For m -player general-sum Markov Games, running the OFTRL algorithm (Algorithm 10)for T rounds, the output (correlated) policy (cid:98) π is an ε -approximate CCE, where
[Equation]
A baseline result for this problem would be (cid:101) O
( T − 1 / 2 ) , which may be obtained directly by adaptingexisting proofs of the V-Learning algorithm [47, 23] to the full-information setting.
Our Theorem 4shows that a faster (cid:101) O
( T − 3 / 4 ) rate is available by using the OFTRL algorithm, which to our bestknowledge is the first such result for policy optimization in general-sum Markov games.
We alsoremark that the output policy (cid:98) π above is not a state-wise average policy as in the zero-sum setting,but rather a mixture policy that is in general non-Markov (cf.
Algorithm 11), which is similar as(and slightly simpler than) the “certified policies” used in existing work [3, 47, 23].
The proof of
Theorem 4 builds upon the RVU property of OFTRL [48] and additionally handles changing gamerewards, similar as in Theorem 3.
A proof sketch and comparison with the (cid:101)
O ( T − 5 / 6 ) analysis of thezero-sum case can be found in Appendix F.2.
Simulations
We perform numerical studies on the various policy optimization algorithms.
Our goal is two-fold:(1)
Verify the convergence guarantees in our theorems and examples; (2) Test some other importantspecial cases of Algorithm 1 that may not yet admit a provable guarantee.
To this end, we consider three algorithms covered by the framework in Algorithm 1:
FTRL (Nash V-Learning) with smooth value updates β t = α t
(Example
Algorithm
Here
the output policy ( (cid:98) µ T , (cid:98) ν T ) are the state-wise averages with weights (cid:8)
α iT (cid:9) T i =1 , and achievesNEGap( (cid:98) µ T , (cid:98) ν T )
≲ T − 1 / 2 if we choose η ≍ T − 1 / 2 (Proposition D.2).
OFTRL with smooth value updates β t = α t (Algorithm 9).
Here the output policy ( (cid:98) µ T , (cid:98) ν T )are the state-wise averages with weights (cid:8)
α iT (cid:9) T i =1 , and achieves NEGap( (cid:98) µ T , (cid:98) ν T )
≲ T − 5 / 6 ifwe
choose
η ≍ T − 1 / 6 (Theorem 3).
We also consider the more aggressive choice
η = 1 .
INPG (Independent Natural Policy Gradients).
This algorithm is an instantiation of Algorithm 1 (cf.
Appendix G.3 for formal justifications) with eager value updates ( β t = 1 ), and
MatrixGameAlg chosen as standard unweighted FTRL (a
k.a. Hedge) for all ( h, s, t )
[Equation]
| s ) exp (cid:16) − η (cid:104)(cid:0) Q t − 1 h (cid:1) ⊤ µ t − 1 h (cid:105) ( s ) (cid:17) .
For this algorithm, we choose two standard learning rates:
η = 1 , and η = T − 1 / 2 , and use thevanilla (state-wise) average as the output policies (since the last-iterate is known to be cyclic):(cid:98) µ Th ( ·| s ) =
1 T (cid:80)
Tt =1 µ th ( ·| s ) , (cid:98) ν
Th ( ·| s ) =
1 T (cid:80)
Tt
=1 ν th ( ·| s ) for all ( h, s ) ∈
[ H ] × S .
The main motivation for considering INPG is that it is a natural generalization of both the widelystudied NPG algorithm for single-agent RL, and the standard Hedge algorithm for zero-sum matrixgames.
In both cases the algorithm admits favorable convergence guarantees: NPG converges withrate O ( T − 1 ) [1, 25, 36, 7] (in both last iterate and averaging) using η = O (1) ; Hedge convergeswith rate O ( T − 1 / 2 ) in zero-sum matrix games (e.g. [39]) using η ≍ T − 1 / 2 .
However, to our bestknowledge, the convergence of INPG for zero-sum Markov games is unclear, and it is commentedby Wei et al.
[50, Section 5] that eager value updates ( β t = 1 ) could cause the value function of the( h + 1) th layer to oscillate, which make learning unstable or even biased within the h -th layer.
[Figure]
A two-layer numerical example We design a simple zero-sum Markov game with two layers andsmall state/action spaces ( H = 2 , S = 4 , A = 2 ; see Appendix G.1 for the detailed description).
The main feature of this game is that the reward in the first layer is much lower magnitude than thatof the second layer (the scale is roughly | r 1 ( s, · , · ) | ≈ 0 .
1 | r 2 ( s, · , · ) | ), which may exaggerate theaforementioned unstable effect.
We also choose a careful initialization ( µ 1 , ν 1 ) which is non-uniform(and modify the FTRL / OFTRL algorithms to start at this initialization, cf.
Appendix G.1)
butwith all entries bounded in [0 . 15 , 0 . 85] .
We test all three algorithms above on this game, with thisinitialization, T ∈ { 10 3 , 3 × 10 3 , 10 4 , . . .
, 10 7 } , and η chosen correspondingly as described above.
Results Figure 1a plots the NEGap of the final output policies, one for each {algorithm, ( T, η ) }.
Observe that FTRL converges with rate roughly T − .
570 ≲ T − 1 / 2 , and OFTRL with η = T − 1 / 6converges with rate T − .
835 ≈ T − 5 / 6 , both corroborating our theory.
Further, OFTRL with η = 1appears to converge with rate T − 1 ; showing this may be an interesting open theoretical question.
On the other hand, the INPG algorithm appears to be much slower: The η = 1 version does not seemto converge, whereas the convergence of η = T − 1 / 2 version is not clear but at least substantiallyslower than T − 1 / 2 ( T − .
308 given by the linear fit) .
To further understand the behavior of INPG, we visualize its layer-wise NEGap ’s for h ∈ { 1 , 2 } (onour example), defined as the NEGap of the h -th layer’s policies with respect to Q ⋆h :
[Equation]
[Table]
Note that NEGap-Layer-1 is a lower bound of NEGap( µ, ν ) (cf.
Appendix G.3)
and thus needs to beminimized by any convergent algorithm.
By contrast, on our example, NEGap-Layer-2 is concernedwith the last layer only, and can be minimized by any algorithm that works on matrix games.
Figure 1b & 1c plot the layer-wise NEGap ’s of INPG against FTRL, on the single run with T = 10 7and η =
T − 1 / 2 .
As expected, the NEGap-Layer-2 converges nicely for both algorithms withsimilar rates (Figure 1b) albeit the oscillation of INPG, whereas their behavior on NEGap-Layer-1 isdrastically different: FTRL still converges, whereas INPG seems to be oscillating around a non-zerobias (Figure 1c).
This suggests that INPG may indeed be suffer from a non-vanishing bias in the firstlayer caused by the second layer’s learning dynamics.
(See Appendix G.2 for additional illustrations.)
It would be an interesting open question to investigate the convergence of INPG theoretically.
Conclusion
This paper provides a unified framework for analyzing a large class of policy optimization algorithmsfor two-player zero-sum Markov games.
Using our framework, we prove new fast convergencerates for the OFTRL algorithm with smooth value updates: (cid:101)
O
( T − 5 / 6 ) for learning Nash Equilibriatwo-player zero-sum Markov games, and (cid:101) O
( T − 3 / 4 ) for learning Coarse Correlated Equilibria inmulti-player general-sum Markov games.
We further demonstrate the importance of smooth valueupdates on a simple numerical example.
We believe our work opens up many interesting directions,such as whether improved rates (e.g. (cid:101) O ( T − 1 ) ) are available for OFTRL algorithms, or furtherinvestigation of policy optimization algorithms with eager value updates (such as Independent Natural
Policy Gradients).
Finally, a limitation of this work is its focus on the full-information setting, and itis an important open question how to generalize our anlayses to the sample-based setting.