Regularized Gradient Descent Ascent for Two-Player Zero-Sum Markov Games
Abstract
We study the problem of ﬁnding the Nash equilibrium in a two-player zero-sum
Markov game.
Due to its formulation as a minimax optimization program, a natural approach to solve the problem is to perform gradient descent/ascent with respect to each player in an alternating fashion.
However, due to the non-convexity/nonconcavity of the underlying objective function, theoretical understandings of this method are limited.
In our paper, we consider solving an entropy-regularized variant of the Markov game.
The regularization introduces structure into the optimization landscape that make the solutions more identiﬁable and allow the problem to be solved more efﬁciently.
Our main contribution is to show that under proper choices of the regularization parameter, the gradient descent ascent algorithm converges to the Nash equilibrium of the original unregularized problem.
We explicitly characterize the ﬁnite-time performance of the last iterate of our algorithm, which vastly improves over the existing convergence bound of the gradient descent ascent algorithm without regularization.
Finally, we complement the analysis with numerical simulations that illustrate the accelerated convergence of the algorithm.
Introduction
The two-player zero-sum Markov game is a special case of competitive multi-agent reinforcement learning where two agents driven by opposite reward functions jointly determine the state transition in an environment.
Usually cast as a non-convex non-concave minimax optimization program, this framework ﬁnds applications in many practical problems including game playing [Lanctot et al., 2019, Vinyals et al., 2019], robotics [Riedmiller and Gabel, 2007, Shalev-Shwartz et al., 2016] and robust policy optimization [Pinto et al., 2017].
A convenient class of methods frequently used to solve multi-agent reinforcement learning problems is the independent learning approach.
Independent learning algorithms proceed iteratively with each player taking turns to optimize its own objective while pretending that the other players’ policies are ﬁxed to their current iterates.
In the context of two-player zero-sum Markov games, the independent learning algorithm performs gradient descent ascent (GDA), which alternates between the gradient updates of the two agents that seek to maximize and minimize the same value function.
Despite the popularity of such algorithms in practice, their theoretical understandings are sparse and do not follow from those in the single-agent case as the environment is not stationary from the eye of any agent.
[Daskalakis et al., 2017] shows that iterates of GDA can possibly diverge or be trapped in limit cycles even in the simplest single-state case when the two players learn with the same rate.
It may be tempting to analyze the two-player zero-sum Markov game by applying the existing theoretical results on minimax optimization.
However, as the objective function in a Markov game is not convex or concave, current analytical tools in minimax optimization that require the objective function to be convex/concave at least on one side are inapplicable.
Fortunately, the Markov game has its own structure: it exhibits a “gradient domination” condition with respect to each player, which essentially guarantees that every stationary point of the value function is globally optimal.
Exploiting this property, Daskalakis et al.
[2020] builds on the theory of Lin et al.
[2020a] and shows that a two-time-scale GDA algorithm converges to the Nash equilibrium of the Markov game with a complexity that depends polynomially on the speciﬁed precision.
However, deriving an explicit ﬁnite-time convergence rate is still an open problem.
In addition, the analysis in Daskalakis et al.
[2020] does not guarantee the convergence of the last iterate; convergence is shown on the average of all past iterates.
In this paper, we show that introducing an entropy regularizer into the value function signiﬁcantly accelerates the convergence of GDA to the Nash equilibrium.
By dynamicially adjusting the regularization weight towards zero, we are able to give a ﬁnite-time last-iterate convergence guarantee to the
Nash equilibrium of the original Markov game.
Main Contributions
• We show that the entropy-regularized Markov game is highly structured; in particular, it obeys a condition similar to the well-known Polyak- Ł ojasiewicz condition, which allows linear convergence of GDA to the (unique) equilibrium point of the regularized game with ﬁxed regularization weight.
We also show that the distance of the equilibrium point of the regularized game to the equilibrium point of the original game can be bounded in terms of the regularizing weight.
• We show that by dynamically driving the regularization weight towards zero, we can solve the original Markov game.
We propose two approaches to reduce the regularization weight and study their ﬁnite-time convergence.
The ﬁrst approach uses a piecewise constant weight that decays geometrically fast, and its analysis follows as a straightforward consequence of our analysis for the case of ﬁxed regularization weight.
To reach a Nash equilibrium of the Markov game up to error ✏ , we ﬁnd that this approach requires at most O ( ✏  3 ) gradient updates, where O only hides structural constants.
The second approach reduces the regularization weight online along with the gradient updates.
Through a multi-time-scale analysis, we optimize the regularization weight sequence along with the step size as polynomial functions of k , where k is the iteration index.
We show that the last iterate of the GDA algorithm converges to the Nash equilibrium of the original Markov game at a rate of O ( k  1 / 3 ) .
Compared with the state-of-the-art analysis of the GDA algorithm without regularization which shows that the convergence rate of the averaged iterates is polynomial in the desired precision and all related parameters, our algorithms enjoy faster last-iterate convergence guarantees.
1.1 Related Work
A Markov game reduces to a standard Markov Decision Process (MDP) with respect to one player if the policy of the other player is ﬁxed.
This is an important observation that allows our work to exploit the recent advances in the analysis of policy gradient methods for MDPs [Nachum et al., 2017, Neu et al., 2017, Agarwal et al., 2020, Mei et al., 2020, Lan, 2022].
Various entropy-based regularizers are introduced in these works that inspire the regularization of this paper.
Our particular regularization is also considered by Cen et al.
[2021], but we discuss and leverage structure in the regularized Markov game that was previously unknown.
As the two-player zero-sum Markov game can be formulated a minimax optimization problem, our work relates to the vast volume of literature in this domain.
Minimax optimization has been extensively studied in the case where the objective function is convex/concave with respect to at least one variable [Lin et al., 2020a,b, Wang and Li, 2020, Ostrovskii et al., 2021].
In the general non-convex non-concave setting, the problem becomes much more challenging as even the notion of stationarity is unclear [Jin et al., 2020].
In Nouiehed et al.
[2019], non-convex non-concave objective functions obeying a one–sided P Ł condition are considered, which the authors use to show the convergence of GDA.
Yang et al.
[2020] analyzes GDA under a two-sided P Ł condition and has a tight connection to our work as the value function of our regularized Markov game also has structure that is similar to, but weaker than, the P Ł condition on two sides.
By exploiting the gradient domination condition of a Markov game with respect to each player,
Daskalakis et al.
[2020] is the ﬁrst to show that the GDA algorithm provably converges to a Nash equilibrium of a Markov game.
A ﬁnite-time complexity is not derived in Daskalakis et al.
[2020], but their analysis and choice of step sizes indicate that the convergence rate is at least worse than
O ( k  1 / 10 . 5 ) .
Additionally, Daskalakis et al.
[2020] does not guarantee the convergence of the last iterate, but rather analyzes the average of all iterates.
In contrast, our work provides a ﬁnite-time convergence analysis on the last iterate of the GDA algorithm.
While our work treats the Markov game purely from the optimization perspective, we would like to point out another related line of works that consider value-based methods [Perolat et al., 2015, Bai and Jin, 2020, Xie et al., 2020, Cen et al., 2021, Sayin et al., 2022].
In particular, Perolat et al.
[2015] is among the ﬁrst works to extend value-based methods from single-agent MDP to two-player Markov games.
Since then, the basic techniques for analyzing value-based methods for Markov games are relatively well-known.
Bai and Jin [2020] considers a value iteration algorithm with conﬁdence bounds.
In Cen et al.
[2021], a nested-loop algorithm is designed where the outer loop employs value iteration and the inner loop runs a gradient-descent-ascent-ﬂavored algorithm to solve a regularized bimatrix game.
In comparison, pure policy optimization algorithms are much less understood for
Markov games, but this is an important subject to study due to their wide use in practice.
In singleagent MDPs, value-based methods and policy optimization methods enjoy comparable convergence guarantees today, and our work aims to narrow the gap between the understanding of these two classes of algorithms in two-player Markov games.
Finally, we note the recent surge of interest in solving two-player games and minimax optimization programs with extragradient or optimistic gradient methods in the cases where vanilla gradient algorithms often cannot be shown to converge [Chavdarova et al., 2019, Mokhtari et al., 2020, Li et al., 2022, Wei et al., 2021, Zhao et al., 2021, Cen et al., 2021, Chen et al., 2021].
These methods typically require multiple gradient evaluations at each iteration and are more complicated to implement.
Most related to our work, Cen et al.
[2021] shows the linear convergence of an extragradient algorithm for solving regularized bilinear matrix games.
They also show that a regularized Markov game can be decomposed into a series of regularized matrix games and present a nested-loop extragradient algorithm which solves these games successively and eventually converges to the Nash equilibrium of the regularized Markov game.
The regularization weight can then be selected based on the desired precision of the unregularized problem.
Although our overall goal of ﬁnding the Nash equilibrium of a general Markov game is the same, the manner in which we decompose and analyze the problem is different.
Our analysis here is based on GDA applied directly to a general regularized Markov game.
We show that for a ﬁxed regularization parameter for a general Markov game, GDA has linear convergence to the modiﬁed equilibrium point.
We also give a scheduling scheme for adjusting the regularization parameter as the GDA iterations proceed, making them converge to the solution to the original problem.
Preliminaries
We consider a two-player Markov game characterized by M = ( S , A , B , P ,  , r ) .
Here, S is the ﬁnite state space,
A and B are the ﬁnite action spaces of the two players,  2 (0 , 1) is the discount factor, and r : S ⇥ A ⇥ B !
[0 , 1] is the reward function.
Let  F denote the probability simplex over a set F , and P : S ⇥
A ⇥ B !
S be the transition probability kernel, with P ( s
| s, a, b ) specifying the probability of the game transitioning from state s to s 0 when the ﬁrst player selects action a 2 A and the second player selects b 2 B .
The policies of the two players are denoted by ⇡ 2  SA and  2  SB , with ⇡ ( a | s ) ,  ( b | s ) denoting the probability of selecting action a , b in state s according to ⇡ ,  .
Given a policy pair ( ⇡ ,  ) , we measure its performance in state s 2 S by the value function
[Equation]
Under a ﬁxed initial distribution ⇢ 2  S , we deﬁne the discounted cumulative reward under ( ⇡ ,  )
[Equation]
where the dependence on ⇢ is dropped for simplicity.
It is known that the Nash equilibrium always exists in two-player zero-sum Markov games [Shapley, 1953], i.e. there exists an optimal policy pair ( ⇡ ? ,  ? )
such that
[Equation]
However, as J is generally non-concave with respect to the policy of the ﬁrst player and non-convex with respect to that of the second player, direct GDA updates may not ﬁnd ( ⇡ ? ,  ? ) and usually exhibit an oscillation behavior, which we illustrate through numerical simulations in Section 5.
Our approach to address this issue is to enhance the structure of the Markov game through regularization.
2.1 Entropy-Regularized Two-Player Zero-Sum Markov Games
In this section we deﬁne the entropy regularization and discuss structure of the regularized objective function and its connection to the original problem.
Let the regularizers be
[Equation]
We deﬁne the regularized value function
[Equation]
b k | s
[Equation]
) ⌘ | s
= s i , where ⌧  0 is a weight parameter.
Again under a ﬁxed initial distribution ⇢ 2  S we denote
J ⌧ ( ⇡ ,  ) , E s ⇠ ⇢ [ V ⇡ ,  ⌧ ( s )] .
The regularized advantage function is
A ⇡
[Equation]
s, a, b ) , r
[Equation]
s, a, b
[Equation]
( s ) , which later helps us to express the policy gradient.
We use d ⇡ ,  ⇢ 2  S to denote the discounted visitation distribution under any policy pair ( ⇡ ,  ) and the initial state distribution ⇢
[Equation]
i For sufﬁcient state visitation, we assume that the initial state distribution is bounded away from zero.
This is a standard assumption in the entropy-regularized MDP literature [Mei et al., 2020, Ying et al., 2022].
Assumption 1.
The initial state distribution ⇢ is strictly positive for any state, and we denote ⇢ min = min s 2 S ⇢ ( s ) > 0 .
When the policy of the ﬁrst player is ﬁxed to ⇡ 2  SA , the Markov game reduces to an MDP for the second player with state transition probability e
P  ( s 0 | s, b ) =
P a 2 A P ( s 0 | s, a, b ) ⇡ ( a | s ) and reward function e r  ( s, b ) =
P a 2 A r ( s, a, b ) ⇡ ( a | s ) .
A similar argument holds for the ﬁrst player if the second player’s policy is ﬁxed.
To denote the operators that map one player’s policy to the best response of the other player and the corresponding value function, we deﬁne
[Equation]
For any ⌧ > 0 , the following lemma bounds the performance difference between optimal and suboptimal policies and establishes the uniqueness of ⇡ ⌧ (  ) and  ⌧ ( ⇡ ) .
When ⌧ = 0 , we use ⇡ 0 (  ) and  0 ( ⇡ ) to denote one of the maximizers and minimizers since they may not be unique.
Lemma 1 (Performance Difference) .
Under Assumption 1 and given ⌧ > 0 , ⇡ ⌧ (  ) is unique for any  2  SB , and  ⌧ ( ⇡ ) is unique for any ⇡ 2  SA .
Given any min player policy  2  SB ,
[Equation]
Given any max player policy ⇡ 2  SA ,
[Equation]
The Nash equilibrium of the regularized problem is sometimes referred to as the quantal response equilibrium [McKelvey and Palfrey, 1995] and is known to exist under any ⌧ .
Leveraging Lemma 1, we formally state the conditions guaranteeing its existence and afﬁrm that it is unique.
Lemma 2 (Minimax Theorem for Entropy-Regularized Markov Game) .
Under Assumption 1, for any regularization weight ⌧ > 0 , there exists a unique Nash equilibrium policy pair ( ⇡ ?⌧ ,  ?⌧ ) such that
[Equation]
We are only interested in the solution of the regularized Markov game if it gives us knowledge of the original problem in (1).
In the following lemma, we show that the distance between the Nash equilibrium of the regularized game and that of the original one is bounded by the regularization weight.
This is an important condition guaranteeing that we can ﬁnd an approximate solution to the original Markov game by solving the regularized problem.
In addition, this lemma also shows that the same policy pair produces value functions with bounded distance under two regularization weights.
Lemma 3.
For any ⌧  ⌧ 0  0 and policy ⇡ ,
[Equation]
2.2 Softmax Parameterization
In this work we use a tabular softmax policy parameterization and maintain two tables ✓ 2 R S ⇥ A , 2 R S ⇥ B that parameterize the policies of the two players according to
[Equation]
The gradients of the regularized value function with respect to the policy parameters admit closedform expressions
[Equation]
and computing them exactly requires knowledge of the dynamics of the environment.
Note that the gradients of value function and the regularizer are Lipschitz with respect to the policy parameters with constants L V = 8 (1   ) 3 and L H = 4+8log |A| (1   ) 3 .
This property is more formally stated and proved in Lemmas 5 and 6 of the appendix.
We next present an important property that we will later exploit to study the convergence of the
GDA updates to the solution of the regularized Markov game.
Under the softmax parameterization, the regularized value function enjoys a gradient domination condition with respect to the policy parameter that resembles the P Ł condition.
Lemma 4 (PL-Type Condition) .
Under Assumption 1, we have for any ✓ 2 R S ⇥ A and
[Equation]
S ⇥
[Equation]
The P Ł condition is a tool commonly used in the optimization community to show the linear convergence of the gradient descent algorithm
[Karimi et al., 2016, Yu and Jin, 2019, Khaled and Richtárik, 2020, Zeng et al., 2021b].
The condition in Lemma 4 is weaker than the common
P Ł condition in two aspects.
First, our P Ł coefﬁcient is a function of the smallest policy entry.
When we seek to bound the gradient of the iterates kr ✓ J ⌧ ( ⇡ ✓ k ,  k ) k 2 and kr J ⌧ ( ⇡ ✓ k ,  k ) k 2 later in the analysis, the P Ł coefﬁcients will depend on min s,a ⇡ ✓ k ( a | s ) and min s,b  k ( b | s ) , which may not be lower bounded by any positive constant.
Second, the coefﬁcients involve ⌧ , which is not a constant but needs to be carefully chosen to control the error between the regularized problem and the original one.
Solving Regularized Markov Games
Leveraging the structure introduced in Section 2, our ﬁrst aim is to establish the ﬁnite-time convergence of the GDA algorithm to the Nash equilibrium of the regularized Markov game under a ﬁxed regularization weight ⌧ > 0 .
The GDA algorithm executes the updates
[Equation]
The convergence bound we will derive reﬂects a trade-off for the regularization weight ⌧ : when ⌧ is large, we get faster convergence to the Nash equilibrium of the regularized problem, but it is farther away from the Nash equilibrium of the original one.
The result in this section will inspire the ⌧ adjustment schemes designed later in the paper to achieve the best possible convergence to the Nash equilibrium of the original unregularized Markov game.
It can be shown that the Nash equilibrium of the regularized Markov game is a pair of completely mixed policies, i.e. 8 ⌧ > 0 there exists c ⌧ > 0 such that min s,a ⇡ ?
⌧ ( a | s )  c ⌧ , and min s,b  ?
⌧ ( b | s )  c ⌧
[Nachum et al., 2017].
In this work, we further assume the existence of a uniform lower bound on the entries of ( ⇡ ?⌧ ,  ?⌧ ) across ⌧ .
We provide more explanation of the assumption in Remark 1.
Assumption 2.
There exists a positive constant c (independent of ⌧ ) such that for any ⌧ >
[Equation]
To measure the convergence of the iterates to the Nash equilibrium of the regularized Markov game, we recall the deﬁnition of g ⌧ in (2) and deﬁne
[Equation]
The convergence metric is asymmetric for two players: the ﬁrst player is quantiﬁed by its performance when the second player takes the most adversarial policy, while the second player is evaluated under the current policy iterate of the ﬁrst player.
We note that  ⇡ k and   k are non-negative, and  ⇡ k =   k =implies that ( ⇡ ✓ k ,  k ) is the Nash equilibrium.
Under this convergence metric, the following theorem states that the GDA updates in (9) solve the regularized Markov game linearly fast.
The proofs of the theoretical results of this paper are presented in Section A of the appendix.
Theorem 1.
We deﬁne L = 3 L H max { ⌧ , 1 } , C 1 = ⇢ min c 2 64log(2) , and C 2 = 2 p |S| p (   ) ⇢ min c , and choose the initial policy parameters to be ✓ 0 = 0 2 R |S| ⇥ |A| and 0 = 0 2 R |S| ⇥ |B| (the initial policies ⇡ ✓ 0 and  0 are uniform).
Let the step sizes of (9) be ↵ k = ↵ ,  k
=  , with ↵ ,  satisfying
[Equation]
If Assumption 1 holds and
[Equation]
then the iterates of (9) satisfy for all k
[Equation]
Theorem 1 establishes the linear convergence of the iterates of (9) to the Nash equilibrium of (5), provided that the initial condition (11) is satisﬁed.
The convergence is faster when ⌧ is large and slower when ⌧ is small.
Choosing ⌧ to be large enough guarantees the initial condition (see Section C of the appendix for more discussion) but causes the Nash equilibrium of the regularized Markov game to be distant from that of the original Markov game.
This motivates us to make the regularization weight a decaying sequence that starts off large enough to meet the initial condition and becomes smaller over time to narrow the gap between the regularized Markov game and the original one.
We discuss two such schemes of reducing the regularization weight in the next section.
Main Results - Solving the Original Markov Game
This section presents two approaches to adjust the regularization weight that allow the GDA algorithm to converge to the Nash equilibrium of the original Markov game.
The ﬁrst approach uses a piecewise constant weight and results in the nested-loop updates stated in Algorithm 1.
In the inner loop the regularization weight and step sizes are ﬁxed, and the two players update their policy iterates towards the Nash equilibrium of the regularized Markov game.
The outer loop iteration reduces the regularization weight to make the regularized Markov game approach the original one.
The regularization weight decays geometrically in the outer loop, i.e. ⌧ t +1 = ⌘⌧ t , where ⌘ 2 (0 , 1) must be carefully balanced.
On the one hand, recalling the deﬁnition of g ⌧ in (2) and deﬁning
[Equation]
we need ⌘ to be large enough that if ✓ t, 0 and t, 0 observe the initial condition 3  ⇡ t, 0 +   t, 0  C 1 ⌧ t , then so do ✓ t +1 , 0 and t +1 , 0 in the worst case.
On the other hand, an ⌘ selected excessively large makes the reduction of ⌧ t too slow to achieve the best possible convergence rate.
Our next theoretical result, as a corollary of Theorem 1, properly chooses ⌘ and K t and establishes the convergence of
Algorithm 1 to the Nash equilibrium of the original original problem.
Initialize: Policy parameters ✓ 0 , 0 = 0 2 R S ⇥
A
and 0 , 0 = 0 2 R S ⇥ B , step size sequences { ↵ t } and {  t } , an initial regularization parameter ⌧ 0 for t = 0 , 1 , · · · , T do for k = 0 , 1 , do
[Equation] [Equation]
end
Set initial policies for next outer loop iteration ✓ t +1 , 0 = ✓ t,K t , t +1 ,
=
t,K t Reduce regularization weight ⌧ t +1 = ⌘⌧ t and properly adjust ↵ t ,  t end
Corollary 1.
Suppose that Assumption 1-2 hold and ⌧ 0 is chosen such that 3  ⇡ 0 , 0 +   0 , 0  C 1 ⌧ 0 1 .
We choose ⌘ = C 1 +2 L  2 C 1 +2 L  , where L  = 4 log |A| + 3 log |B| + log |B| 1   and C 1 is deﬁned in Theorem 1.
Then, under proper choices of ↵ t and  t , the iterates of Algorithm 1 converge to a point such that
[Equation] [Equation]
in at most T =
O (log( ✏  1 )) outer loop iterations.
The total number of gradient updates required is
P
Tt =
K t =
O ( ✏  3 ) .
Corollary 1 guarantees that ( ⇡ ✓ T ,  T ) converge to an ✏ -approximate Nash equilibrium of the original
Markov game in T =
O ( ✏  3 ) gradient steps.
In order to achieve this rate, K t has to be adjusted along with ⌧ t : we need K t =
O ( ⌧  3 t ) when ⌧ t becomes smaller than 1.
The varying number of inner loop iterations may cause inconvenience for practical implementation.
To address this issue, we next propose another scheme of adjusting the regularization weight that is carried out online along with the update of the policy iterates.
Algorithm 2: Policy Gradient Descent Ascent Algorithm with Diminishing Regularization Weight
Initialize: Policy parameters ✓ 0 = 0 2 R S ⇥ A and 0 = 0 2 R S ⇥ B , step size sequences { ↵ k } and {  k } , regularization parameter sequence { ⌧ k } for k = 0 , 1 , · · · , K do 1) Max player update:
[Equation]
Min player update:
[Equation]
end
Presented in Algorithm 2, the second approach is a single-loop algorithm that reduces the regularization weight as a polynomial function of the iteration k .
We deﬁne the auxiliary convergence metrics
[Equation]
which measure the convergence of ( ⇡ ✓ k ,  k ) to the Nash equilibrium of the Markov game regularized with weight ⌧ k .
To judge the performance of the iterates in the original Markov game, we are ultimately interested in bounding J ( ⇡ ? ,  ? )
g 0 ( ⇡ ✓ k ) and J ( ⇡ ✓ k ,  k )  g 0 ( ⇡ ✓ k ) .
Thanks to
Lemma 3, we can quantify how fast  ⇡ k and   k approach these desired quantities as ⌧ k decays to 0.
Under an initial condition on  ⇡ k and   k , we now establish the convergence rate of Algorithm 2 to ( ⇡ ? ,  ? ) of (1) through a multi-time-scale analysis.
Theorem 2.
Let the step sizes and regularization parameter be
[Equation]
with ↵ 0 ,  0 , ⌧ 0 , and h  1 satisfying a system of inequalities discussed in details in the analysis.
Under Assumption 1-2, the iterates of Algorithm 2 satisfy for all k
[Equation]
where the constant C 1 is deﬁned in Theorem 1.
Theorem 2 states that the last iterate of Algorithm 2 converges to an O ( k  1 / 3 )
-approximate
Nash equilibrium of the original Markov game in k iterations.
This translates to the same sample complexity as Algorithm 1 derived in Corollary 1.
Compared with Algorithm 1, reducing ⌧ k online along with the gradient updates in a single loop simpliﬁes the algorithm and makes tracking the regularization weight, step sizes, and policy iterates simpler and more convenient.
We note that the techniques in
Daskalakis et al.
[2020] may be used to analyze the ﬁnite-time performance of GDA for Markov games and lead to a convergence rate at least worse than O ( k  1 / 10 . 5 ) , which we improve over.
Remark 1.
Assumption 2 is a restrictive assumption that does not seem necessary but rather arises as an artifact of the current analysis.
When we apply the weaker PL-type condition (Lemma 4) in the analysis, the entries of the iterates ⇡ ✓ k ,  k need to be uniformly lower bounded, which is difﬁcult to establish using the game structure.
We come up with an innovative induction approach to quantify the connection between min s,a ⇡ ✓ k ( a | s ) , min s,b  k ( b | s ) and the optimal gap  ⇡ k ,   k .
This approach allows us to transform the uniform lower bound requirement on ⇡ ✓ k ,  k to that on the Nash equilibrium, leading to Assumption 2.
It is a future work to remove/relax this assumption.
A Markov game is said to be completely mixed if every Nash equilibrium of the game consists of a pair of completely mixed policies, i.e. min s,a ⇡ ?
( a | s ) > 0 , min s,b  ?
( b | s ) > 0 for any Nash equilibrium ( ⇡ ? ,  ? ) of the Markov game (if more than one exists).
Assumption 2 intuitively seems no stronger than requiring the original Markov game to be completely mixed.
If the original Markov game has at least one completely mixed Nash equilibrium, the Nash equilibrium of the regularized
Markov game should also be completely mixed even when the regularization weight is small, since the regularization encourages the solution to be more uniform.
The reward function that results in completely mixed Markov games is well studied in Raghavan [1978], Kaplansky [1995], Das et al.
Numerical Simulations
In this section, we numerically verify the convergence of Algorithm 2 on small-scale synthetic
Markov games.
Our aim is to conﬁrm that the algorithm indeed converges rather than to visualize the exact convergence rate, as achieving the theoretical rate derived in Theorem 2 requires very careful selection of all involved parameters.
Considering an environment with |S| = 2 and |A| = |B| = 2 , we ﬁrst choose the reward and transition probability kernel such that the Markov game is completely mixed 2 .
We run Algorithm 2 for 50000 iterations with ↵ k = 10  3 ,  k = 10  2 , ⌧ k = ( k + 1)  1 / 3 , and measure the convergence of ⇡ k and  k by metrics considered in (13) and (14) of Theorem 2.
As shown in the ﬁrst plot of Figure 1, the last iterate exhibits an initial oscillation behavior but converge smoothly after 10000 iterations.
In comparison, we visualize the convergence of the last iterate and averaged iterate of the GDA algorithm without any regularization (second and third plots of Figure 1), where the average is computed with equal weights as ¯ ⇡ k =
[Equation]
The existing theoretical results in this case guarantee the convergence of the averaged iterate but not the last iterate [Daskalakis et al., 2020].
According to our simulations, the last iterate indeed does not converge, while the averaged iterate does, but at a slower rate than the convergence of the last iterate of the GDA algorithm under the decaying regularization.
The theoretical results derived in this paper rely on Assumption 2.
To investigate whether this assumption is truly necessary, we also apply Algorithm 2 to a Markov game that has a deterministic
Nash equilibrium and does not observe Assumption 2 3 .
As illustrated in Figure 2, the experiment shows that Algorithm 2 still converges correctly to ( ⇡ ? ,  ? ) of (1).
This observation suggests that Assumption 2 may be an artifact of the current analysis and motivates for us to investigate ways to remove/relax this assumption in the future.
We note that the pure GDA approach without regularization also has a last-iterate convergence and does not exhibit the oscillation behavior observed in Figure 1, since the gradients of both players never change signs regardless of the policy of the opponent in this Markov game.
Conclusion & Future Work
In this paper, we present the ﬁnite-time analysis of two GDA algorithms that provably ﬁnd the Nash equilibrium of a Markov game with the help of a structured entropy regularization.
Future directions of this work include formalizing the link between Assumption 2 and completely mixed Markov games, investigating the possibility of relaxing this assumption, and characterizing the convergence of the stochastic GDA algorithm where the players do not have knowledge of the environment dynamics and can only take samples to estimate the gradients.
Acknowledgement
Sihan Zeng and Justin Romberg were supported in part by ARL
DCIST CRA W911NF-17-2-0181.
The work of Thinh T. Doan was supported in part by the Commonwealth Cyber Initiative.