S AMPLE E FFICIENT S TOCHASTIC P OLICY E XTRAGRADIENT A LGORITHM FOR Z ERO -S UM M ARKOV G AME
Anonymous authors
Paper under double-blind review
A BSTRACT
Two-player zero-sum Markov game is a fundamental problem in reinforcement learning and game theory.
Although many algorithms have been proposed for solving zero-sum Markov games in the existing literature, many of them either require a full knowledge of the environment or are not sample-efﬁcient.
In this paper, we develop a fully decentralized and sample-efﬁcient stochastic policy extragradient algorithm for solving zero-sum Markov games.
In particular, our algorithm utilizes multiple stochastic estimators to accurately estimate the value functions involved in the stochastic updates, and leverages entropy regularization to accelerate the convergence.
Speciﬁcally, with a proper entropy-regularization parameter, we prove that the stochastic policy extragradient algorithm has a sample complexity of the order O (
[Table]
) for ﬁnding a solution that achieves (cid:15) -Nash equilibrium duality gap.
Such a sample complexity result substantially improves the state-of-the-art complexity result.
I NTRODUCTION
Competitive reinforcement learning (RL) is an emerging and popular framework that has broad applications in various areas, including market pricing applications (Kononen and Oja, 2004),
realtime strategy-making
Vinyals et al., 2019), board games (Silver et al., 2017; Moerland et al., 2018)
and inverse RL (Zhang et al., 2019).
In particular, an important and fundamental formulation of competitive RL is the two-player zero-sum Markov game, which involves two competing players that interact with a common environment and receive zero-sum rewards.
Both players aim to learn the optimal policy that achieves the Nash equilibrium of accumulated rewards.
Algorithms for solving Markov games are very different from conventional single-agent RL algorithms.
In particular, both players must learn to improve their policies based on feedback from the opponent and the environment, but usually the opponent will not reveal any sensitive information (e.g., actions or policy) or cooperate with each other.
In the existing literature, numerous algorithms have been developed for solving zero-sum Markov games, including Q-learning (Fan et al., 2020; Zhu and Zhao, 2020), ﬁtted Q iteration (Zhang et al., 2021), policy gradient (Daskalakis et al., 2020; Zhao et al., 2021), policy extragradient (Cen et al., 2021), model-based Monte Carlo estimation (Zhang et al., 2020), optimistic gradient descent ascent (Wei et al., 2021), etc.
However, many of these algorithms require both players to access their opponent’s actions (Wei et al., 2017; Sidford et al., 2020; Huang et al., 2021; Jafarnia-Jahromi et al., 2021), which causes privacy issues.
On the other hand, some algorithms need to know about the environment transition kernel and reward mapping (Cen et al., 2021), which are usually unknown a priori in practice.
Moreover, other algorithms require both players to perform asymmetric policy updates using different numbers of iterations, learning rates or exploration probabilities (Zhao et al., 2021;
Daskalakis et al., 2020), which are generally hard to coordinate in advance between two competing players.
Therefore, it is desired to develop an algorithm for solving Markov games that avoids all the aforementioned issues.
Moreover, from a theoretical perspective, the convergence and sample complexity of these existing algorithms for solving Markov games have not been comprehensively studied.
Speciﬁcally, some studies established the convergence of the algorithms with i.i.d.
samples (Zhao et al., 2021; Guo et al., 2021), which violates the dependent nature of samples collected from the dynamic Markov
[Table]
[Table] Our work
[Table]
decision process.
Also, other algorithms suffer from an extremely high sample complexity to achieve an approximate Nash equilibrium solution (Wei et al., 2021).
Hence, we are motivated to answer the following fundamental question.
• Q: Can we develop a fully decentralized algorithm that is model-free and takes symmetric and private policy updates for solving zero-sum Markov games, with provable convergence guarantee and an improved sample complexity for achieving a Nash equilibrium solution?
In this paper
[Table]
we provide positive and comprehensive answers to this question by developing a fully decentralized stochastic policy extragradient algorithm.
In Table [Table] we compare the key properties and sample complexity of our algorithm with those of the existing algorithms.
We also refer to
Appendix F for more explanations on Table 1.
Our contributions are summarized as follows.
O UR C ONTRIBUTIONS
We consider a standard zero-sum Markov game with discounted reward over inﬁnite horizon.
To solve such a Markov game, we propose a stochastic variant of the policy extragradient algorithm
(Cen et al., 2021) that satisﬁes the following amenable properties
• Our algorithm uses multiple stochastic estimators to estimate the value functions involved in the predictive updates for solving entropy-regularized matrix games, and therefore the algorithm does not rely on any prior knowledge of the environment transition kernel (model-free).
Moreover, the resulting stochastic policy updates of our algorithm for both players are symmetric and do not involve any sensitive information of the opponent (private).
• Compared with the stochastic estimators used in (Wei et al., 2021), our estimators have much smaller variance that helps improve the estimation accuracy.
Speciﬁcally, by developing new techniques (explained in the next bullet), we establish tight high-probability estimation error bounds for our stochastic estimators with Markovian samples.
Then, with a proper entropyregularization parameter, we show that our stochastic policy extragradient algorithm requires a sample complexity of the order O ( t mix A max µ min (cid:15) 5 . 5 (1 − γ ) 13 . 5 ) to achieve an (cid:15) -Nash equilibrium duality gap, which substantially improves the state-of-the-art complexity result of (Wei et al., 2021).
• We develop novel techniques to bound the estimation error of the proposed stochastic estimators, whose numerator and denominator involve sample average approximations.
First, we propose a special estimation error decomposition that avoids divergence of the bound caused by possibly small numerical values of the sample average involved in the denominator of the stochastic estimators.
Second, we leverage this error decomposition and the recursive structure of the stochastic estimators to derive a contraction property of the estimation errors, which eventually leads to tight bounds for the estimation error.
We refer to Section 4 for more elaboration on our technical novelties.
1. O THER RELATED WORK
Other settings of two-player zero-sum Markov games.
In this paper, we focus on a standard setting of two-player zero-sum Markov game with discount and inﬁnite horizon in the discrete time domain.
There are other settings of two-player zero-sum Markov games.
For example, Bai et al.
; Huang et al.
(2021) studied a two-player zero-sum Markov game with ﬁnite horizon and without discount, whereas Daskalakis et al.
(2020) considered ﬁnite random horizon without discount.
Jafarnia-Jahromi et al.
(2021) also considered the setting without discount, and it allows one of the players to constantly adjust its policy based on the entire history of states and actions.
Ghosh et al.
(2021) studied a two-player zero-sum Markov game in the continuous time domain.
Multi-agent general-sum Markov game.
Some works studied multi-agent Markov games, which extend the two-player zero-sum Markov games to multiple players without the zero-sum constraint (Wang and Sandholm, 2002; Hu and Wellman, 2003;
Deng et al., 2021;
Leonardos et al., 2021).
More speciﬁcally, Leonardos et al.
(2021) deﬁned and studied Markov potential game which has a potential function assumption.
Guo et al.
; Elie et al.
Gu et al.
studied mean-ﬁeld games with a large number of players.
Entropy regularization and
value iteration
Our algorithm leverages entropy regularization and value iteration to accelerate convergence.
Entropy regularization is a popular technique that has been widely used in reinforcement learning (Neu et al., 2017; Geist et al., 2019; Mei et al., 2020; Cen et al., 2020) and Markov game (Mertikopoulos and Sandholm, 2016; Savas et al., 2019; Cen et al., 2021) to encourage environment exploration and accelerate algorithm convergence.
Value iteration is also a classical method that is widely used in both single-agent reinforcement learning (Ernst et al., 2005; Tamar et al., 2016; Farahmand and Ghavamzadeh, 2021) and Markov games (Zhu and Zhao, 2020; Cen et al., 2021).
With full knowledge of the environment, it exponentially converges to the ﬁxed point of Bellman operator (Cen et al., 2021).
Compared to another similar classical method called policy iteration, value iteration does not need policy evaluation which involves additional computation (Sutton and Barto, 2018).
B ACKGROUND OF M ARKOV G AME AND E NTROPY R EGULARIZATION 2. T WO - PLAYER ZERO - SUM M ARKOV GAME
In a zero-sum Markov game, two players compete with each other in a common environment.
Throughout, the state space is denoted as S .
The action spaces and policies of both players are denoted as A (1) , π (1) and A (2) , π (2) , respectively.
Here, π (1) ∈ ∆( |A (1) | ) , π (2) ∈ ∆( |A (2) | ) are random policies deﬁned over the corresponding simplex sets.
The reward function is denoted as
R : S × A (1) ×
A (2) → [0 , 1] , and the discount factor is denoted as γ ∈ (0 , 1) .
At any time t , both players observe state s t ∈ S of the environment.
Then, both players respectively select their actions following their own policies, i.e., a (1) t ∼ π (1) ( ·| s t ) and a (2) t ∼ π (2) ( ·| s t ) .
After that, the environment state transfers to a new state s t +1 following the underlying transition kernel
P ( ·| s t , a (1) t , a (2) t ) , and both players receive zero-sum rewards, i.e., R (1) t = − R (2) t =
R t , where
R t := R ( s t , a (1) t , a (2) t ) .
With this Markov decision process, we can deﬁne the following state value function associated with the players’ policies π (1) and π (2) for any environment state s ∈ S .
[Equation]
The goal of both players is to compete via the following minimax game in all states s .
[Equation]
In particular, it has been shown in (Shapley, 1953) that there exists a Nash equilibrium policy pair π (1) ∗ , π (2) ∗ for zero-sum Markov games, i.e., V π (1) ,π (2) ∗ ( s )
≤ V π (1) ∗ ,π (2) ∗ ( s )
≤ V π (1) ∗ ,π (2) ( s ) holds for any other policies π (1) , π (2) and for all states s .
2. E NTROPY - REGULARIZED M ARKOV GAME
Entropy regularization is a popular technique that has been widely used in reinforcement learning (Neu et al., 2017; Geist et al., 2019; Mei et al., 2020; Cen et al., 2020) and Markov game (Mertikopoulos and Sandholm, 2016; Savas et al., 2019; Cen et al., 2021) to encourage environment exploration and accelerate algorithm convergence.
Speciﬁcally, for the zero-sum Markov game, we can deﬁne an entropy-regularized state value function by adding entropy regularization to the state value function in (1) as follows (Cen et al., 2021).
[Equation]
where τ > 0 is called the regularization parameter.
With the above deﬁnition, we further deﬁne the following entropy-regularized state-action value function (also called Q -function)
(Cen et al., 2021).
[Equation]
In particular, V ( τ ) π (1) ,π (2) can be obtained from Q ( τ ) π (1) ,π (2) as follows.
[Equation]
where H ( π ) denotes the entropy of policy π , and we deﬁne this mapping as f τ for convenience.
For the entropy regularized Markov game, it has an equilibrium policy pair that solves the minimax optimization problem min π (2) max π (1) V ( τ ) π (1) ,π (2) ( s ) .
Such a policy pair is called the quantal response equilibrium (QRE).
Our goal is to ﬁnd the equilibrium policy pair of the original Markov game in (2) by solving the entropy-regularized Markov game with a proper regularization parameter τ .
In particular, compared with the equilibrium policy of the Markov game, the QRE tends to have a larger entropy due to the entropy regularization, which encourages the players to explore and obtain a better understanding of the environment.
Another advantage of considering the entropy-regularized
Markov game is that the entropy regularization makes the minimax problem have a better optimization geometry that accelerates the convergence of the optimization process.
S TOCHASTIC P OLICY E XTRAGRADIENT A LGORITHM FOR E NTROPY -R EGULARIZED M ARKOV G AME
In this section, we develop a stochastic policy extragradient (SPE) algorithm for solving entropyregularized Markov games.
First, we recap the policy extragradient (PE) algorithm, which is introduced in (Cen et al., 2021) to solve entropy-regularized Markov games with full knowledge of the environment transition kernel and reward mapping.
Then, we propose the model-free SPE algorithm that solves entropy-regularized Markov games using only stochastic samples.
3. R EVIEW OF POLICY EXTRAGRADIENT ALGORITHM
Value iteration is a classical reinforcement learning algorithm that requires full knowledge of the environment and achieves an exponential convergence rate.
In particular, for the entropy-regularized
Markov game, the k -th value iteration update is deﬁned as follows.
[Equation]
where we deﬁne Q k ( s ) :
= Q k ( s, ·
, · ) ∈ R |A (1) |×|A (2)
| .
This algorithm alternatively updates all the entries of the value functions Q k and V k .
Thanks to the entropy regularization in the function f τ (see (5) for the deﬁnition), the minimax matrix game in (7) is τ -strongly concave in π (1) and τ -strongly convex in π (2) , and therefore it has a unique solution.
To solve the entropy-regularized minimax matrix game in (7), Cen et al.
(2021) proposed a predictive update (PU) algorithm.
Speciﬁcally, with uniform policy initialization, i.e., π ( m ) k, 0 ( s ) = 1 |A ( m ) | , ∀ m ∈
{ 1 , 2 } , ∀ s ∈ S , the PU algorithm performs the following policy updates: for t = 0 , 1 , 2 , ...
[Equation]
where we use the following notations (superscript ( \ m ) denotes the opponent of the m -th player.).
[Equation]
Once we obtain the output policy pair ( π (1) k , π (2) k ) of the PU algorithm, we can obtain an approximation of V k +1 ( s ) as V (cid:48)
k +1 ( s ) =
f τ (cid:0) Q k ( s ); π (1) k ( s ) , π (2) k
( s ) (cid:1) , which will be further used in the next
Q -value function update (6) to replace V k +1 ( s (cid:48) ) .
The updates (6), (7) & (8) are referred to as policy extragradient (PE) algorithm.
In the PE algorithm, the PU update in (8) allows both players to take symmetric updates without revealing their private actions, and has been shown to converge to the unique solution of the entropyregularized matrix game (7) exponentially fast (Cen et al., 2021).
However, the PE algorithm has several limitations.
First, in the PU update, each player m ∈ { 1 , 2 } needs to query the quantities
Q ( m ) k,t ( s, a ( m ) ) , Q ( m ) k,t ( s, a ( m ) ) from its opponent.
To compute these quantities, the opponent needs to multiply the entire Q -table by its own policy vector.
This requires both players to coordinate with each other and share a Q -table.
Second, the update of the Q -table in (6) requires full knowledge of the environment transition kernel P and the reward mapping R , which are unknown a priori in practice.
To overcome these limitations, we develop a fully stochastic PE algorithm in the next subsection.
3. S TOCHASTIC POLICY EXTRAGRADIENT ALGORITHM
The major challenge of the PE algorithm is computing the quantities Q ( m ) k,t , Q ( m ) k,t and V (cid:48)
k +1 , which requires coordinating with the opponent and involves the environment information.
Here, we develop a model-free and fully stochastic variant of PE that estimates these key quantities using Markovian stochastic samples.
We refer to this algorithm as stochastic policy extragradient (SPE) .
Speciﬁcally, we ﬁrst estimate the quantity V (cid:48)
k +1 ( s ) =
f τ (cid:0) Q k ( s ); π (1) k ( s ) , π (2) k
( s ) (cid:1) .
By deﬁnition of f τ in (5) and the update of Q k in (6) (now we use V (cid:48) k
( s (cid:48) ) instead of V k ( s (cid:48) ) )
and using some standard tricks on random variables (see Lemma 2 in Appendix B for a full proof), we can rewrite V (cid:48)
k +1 ( s ) as
V (cid:48)
k
[Equation]
where µ k ( s ) denotes the stationary state distribution associated with the policy pair ( π (1) k , π (2) k ) , and the expectation is taken over (cid:101) s ∼ µ k , (cid:101) a (1) ∼ π (1) k
( s ) , (cid:101) a (2) ∼ π (2) k
( s ) , s (cid:48) ∼ P ( ·| (cid:101) s, (cid:101) a (1) , (cid:101) a (2) ) .
To estimate this quantity, we query a set N k +1 (with cardinality N k +1 ) of samples from the Markov decision process following the pair of policies ( π (1) k , π (2) k ) .
Then, we estimate V (cid:48)
k +1 ( s ) as
[Equation]
Intuitively, we use the sample average of Markovian samples to estimate the expectation terms in (11).
Thanks to the concentration phenomenon of dependent samples (Paulin, 2015), these sample averages converge to the desired expected values provided that the sample size is sufﬁciently large.
Next, we estimate Q ( m ) k,t , m ∈ 1 , 2 .
Leveraging (9) and (6), we obtain the following equivalent characterization for both players m ∈ 1 , 2 (see Lemma 2 in Appendix B for the proof of equivalence).
[Equation]
where 1 {·} denotes the indicator function, µ k,t denotes the stationary state distribution associated with the policy pair ( π (1) k,t , π (2) k,t ) , and the expectation is taken over (cid:101) s ∼ µ k,t , (cid:101) a (1) ∼ π (1) k,t ( s ) , (cid:101) a (2) ∼ π (2) k,t ( s ) , s (cid:48) ∼ P ( ·| (cid:101) s, (cid:101) a (1) , (cid:101) a (2) ) .
To estimate this quantity, we query a set N k,t (with cardinality N k,t ) of samples from the Markov decision process following a pair of smoothed policies π (cid:48) ( m ) k,t ( s ) =
(1 − (cid:15) (cid:48) )
π ( m ) k,t ( s ) + (cid:15) (cid:48) |A
( m ) | 1 , where (cid:15) (cid:48) ∈ [0 , 1] is a small smoothing constant that will be theoretically determined later.
Then, we estimate Q ( m ) k,t
( s, a ( m ) ) as follows.
[Equation]
where we have replaced the expectations with sample averages, and replaced V (cid:48)
k ( s (cid:48) ) with (cid:98)
V k
( s (cid:48) ) .
Here, the Markovian samples are queried following the (cid:15) (cid:48) -smoothed policies ( π (cid:48) (1) k,t , π (cid:48) (2) k,t ) .
On one hand, (cid:15) (cid:48) should not be too small so that it keeps the denominator of the above estimation away from zero.
On the other hand, (cid:15) (cid:48) should not be too large so that it is sufﬁciently close to the original policy.
Similarly, to estimate Q ( m ) k,t , we query another set N k,t (with cardinality N k,t ) of samples from the
Markov decision process following a pair of smoothed policies π (cid:48)
[Equation]
=
[Equation]
+
[Equation]
1 , where (cid:15) (cid:48) ∈ [0 , 1] will be theoretically determined later.
Then, we estimate Q ( m ) k,t as follows.
[Equation]
Remark 1.
The estimators (12) , (14) & (15) essentially use ratio of sample averages to approximate ratio of expectations. Estimators with similar structure have been used in (Ortner and Auer, 2007;
Xia et al., 2016; Wei et al., 2021).
However, these works analyze the estimation error of their estimators with independence samples.
As a comparison, our analysis of the estimation error bounds the additional bias induced by the correlated Markovian samples, and achieves an improved sample complexity in the main Theorem 2.
We summarize our stochastic policy extragradient (SPE) algorithm in Algorithm 1.
Speciﬁcally, in
SPE, we estimate the quantities Q ( m ) k,t , Q ( m ) k,t , V (cid:48)
k using their corresponding stochastic estimators.
As a result, the SPE algorithm is model-free, and the updates for both players are symmetric and private.
F INITE -T IME C ONVERGENCE A NALYSIS OF SPE
Throughout our convergence analysis, we adopt the following two standard assumptions.
Assumption 1.
Denote T π (1) ,π (2) ( s, s (cid:48) ) :
= inf { t ≥ 1 : s t =
s (cid:48) | s
= s } as the ﬁrst-visit time under the policy pair π (1) , π (2) .
We assume that
[Equation]
Assumption 1 is widely used in the reinforcement learning literature (Ortner and Auer, 2007; Ortner, 2020; Wei et al., 2021; Jafarnia-Jahromi et al., 2021).
It ensures that every state will be visited at least once within a ﬁnite duration of time, thus ensuring that all the states will be visited inﬁnitely often.
This guarantees sufﬁcient exploration.
In our analysis, we use the following equivalent statement
Initialize: V (cid:48)
( s ) for all s ∈ S .
for value iterations k = 0 , 1 , . .
, K − 1 do
Initialize π (1) k, 0 , π (2) k, 0 with uniform distribution.
for PU iterations t = 0 , 1 , . . .
, T k − 1 do
Players 1,2 sample N k,t Markovian samples following smoothed policies π (cid:48) (1) k,t , π (cid:48) (2) k,t .
Every player m ∈ { 1 , 2 } computes (cid:98) Q ( m ) k,t ( s, a ( m ) ) for all s, a ( m ) using (14).
Players 1,2 sample N k,t Markovian samples following smoothed policies π (cid:48) (1) k,t , π (cid:48) (2) k,t .
Every player m ∈ { 1 , 2 } computes (cid:98) Q ( m ) k,t ( s, a ( m ) ) for all s, a ( m ) using (15).
Implement the t -th PU iteration for all s, a (1) , a (2) using (8) with estimations (14)&(15).
end
Let π ( m ) k
= π ( m ) k,T k , m = 1 , 2 .
Players sample N k Markovian samples following π (1) k , π (2) k .
Compute (cid:98) V
k
+
( s ) for all s using (12).
end
Output: π (1) K − 1 , π (2) K − 1 . of Assumption 1 for convenience, which means that the stationary state distribution µ π (1) ,π (2) has a uniform lower bound µ min > 0 .
Their equivalence is based on Theorem 5.5.11 of (Durrett, 2019).
[Equation]
Assumption 2.
There exists a mixing time t mix ∈ N such that for any policy pair π (1) , π (2) and its corresponding stationary state distribution µ π (1) ,π (2) , we have
[Equation]
where P π (1) ,π (2) ( s t mix ) denotes the state distribution under the policy pair π (1) , π (2) at time t mix , and d TV denotes the total variation distance between two probability distributions.
Assumption 2 is also widely adopted in the existing literature (Greensmith et al., 2004; Ortner, 2020; Ciosek, 2021).
It ensures that the state distribution is not too far away from its stationary distribution within a ﬁnite time t mix .
We note that this assumption is much weaker than the popular uniformly ergodic Markov chain assumption, which assumes that d TV ( P π (1) ,π (2) ( s t ) , µ π (1) ,π (2) ) decays exponentially fast as t → ∞ (Bhandari et al., 2018; Qiu et al., 2019; Xu and Liang, 2021).
In this subsection, we analyze the ﬁnite-time convergence of Algorithm 1 for solving the entropyregularized Markov game (5).
We focus on the convergence rate of the following Nash equilibrium duality gap, which is a standard optimality metric widely adopted in the existing literature (Xu et al., 2020; Jin and Sidford, 2021; Wei et al., 2021).
[Equation]
In particular, when τ = 0 , D (0) ( π (1) , π (2) ) corresponds to the duality gap of the original Markov game.
Throughout, we deﬁne A max := max {|A (1) | , |A (2) |} , Q max :
= 1+ γτ ln
A max 1 − γ and V max :
= 1+ τ ln
A max 1 − γ .
We also require that the batch sizes of Algorithm 1 satisfy the following conditions.
[Equation]
Then, we obtain the following convergence result of Algorithm 1, where T sum :=
(cid:80) K − 1 k =0 T k .
Theorem 1 (Finite-time convergence rate) .
Apply Algorithm 1 to solve the entropy-regularized
Markov game with τ ∈ (0 , 1] .
Choose learning rate η = [2( τ + Q max )] − 1 , initialization (cid:107) (cid:98) V 0 (cid:107) ∞ ≤
V max and batch sizes N k,t , N k,t , N k +1 that satisfy (18) & (19) .
Then, the Nash equilibrium duality gap converges at the following rate with probability at least 1 − δ .
[Equation]
Remark 2.
In the proof of Theorem 1, we also prove that the convergence rate of the Q-function estimation error (cid:107) Q K − Q ( τ ) ∗ (cid:107) ∞ is (1 − γ ) times the convergence rate in (20) .
Here, Q K corresponds to the Q-function associated with the policy pair ( π (1) K , π (2) K ) produced by Algorithm 1 in the K -th iteration, and Q ( τ ) ∗ corresponds to the optimal Q-function associated with the Nash equilibrium policy pair π (1) ∗ τ , π (2) ∗ τ of the entropy-regularized Markov game.
Theorem 1 characterizes the convergence of duality gap of the SPE algorithm under general hyperparameter scheduling of the batch sizes N k,t , N k,t , N k +1 and number of inner iterations T k .
Speciﬁcally, it can be seen that as the number of outer iterations K and inner iterations T k increase, the duality gap converges to an exponentially weighted average of N − 2 / 3 k,t , N − 2 / 3 k,t +1 and N − 1 k +1 , and the gap can be made arbitrarily small by choosing sufﬁciently large batch sizes. To provide an intuitive understanding, we can set these hyper-parameters as constants, i.e., T k ≡ T , N k,t ≡ N , N k,t ≡ N , N k +1 ≡ N (cid:48) , and then the convergence rate in (20) simpliﬁes to [Equation]
To elaborate, the ﬁrst term characterizes the exponential convergence of the K outer value iterations.
This convergence is faster than the sublinear convergence O ( (cid:112) ln K/K ) established in (Wei et al., 2021).
The second term characterizes the exponential convergence of the T inner PU iterations. Moreover, the last two terms characterize the estimation errors O ( N − 1 ) and O ( N − 2 / 3 )+ O ( N − 2 / 3 ) of the estimators (12) and (14,15), respectively. As a comparison, Wei et al. (2021) established a much larger estimation error
O ( N − 1 / 3 ) .
These improvements, as we show in Theorem 2 later, lead to a substantially improved overall sample complexity over the state-of-the-art result.
In particular, due to the exponentially weighted average structure in (20), the sample complexity can be further optimized by an adaptive scheduling of the batch sizes.
Technical Novelty. We further comment on the proof of Theorem 1.
Note that the analysis of the PE algorithm in (Cen et al., 2021) requires full knowledge of the environment and does not characterize the convergence of duality gap.
As a comparison, to establish the duality gap convergence rate (20) of the model-free SPE, we need to make substantial new developments to tightly bound the estimation errors of the proposed stochastic estimators.
We elaborate our technical contributions below.
• As we explained in Remark 1, the sample averages involved in our estimators are correlated Markovian samples.
To bound the additional bias induced by these correlated samples, we apply the concentration inequalities developed in (Paulin, 2015) for dependent samples to establish tight high-probability estimation error bounds.
As a comparison, Wei et al.
(2021) assumes independence of these samples (although their setting is Markovian sampling).
• We develop a much reﬁned analysis of the state value function estimation error (cid:107) (cid:98)
V k +1 − V (cid:48)
k +1 (cid:107) ∞ , which is the key to develop tight bounds for all the other estimation errors.
Speciﬁcally, we ﬁrst propose the following error decomposition for any state s
[Equation]
where (cid:98) v k +
( s ) , (cid:98) µ k +1 ( s ) are sample average estimators of v k +
( s ) ,
µ k +1 ( s ) , respectively, and we refer to Appendix B for the deﬁnitions of these terms.
The motivation is that the (cid:12)(cid:12)(cid:98) v k +
( s ) (cid:12)(cid:12) in the second term helps cancel out the estimator (cid:98) µ k ( s ) in the denominator, and then all the denominators do not involve any sample average estimators, which may take a small numerical value that causes divergence and a loose concentration bound.
By leveraging this special decomposition and the recursive structure of the stochastic estimator (12), we are able to establish the following key contraction property of the estimation error (see (85) in Appendix B).
[Equation]
By telescoping the above contraction bound, we obtain tight estimation error bounds for all the proposed stochastic estimators.
As a comparison, Wei et al.
(2021) directly applied the AzumaHoeffding inequality with independent samples to bound the entire estimator and obtain a loose error bound, and Liu et al.
(2021) simply assumed a small upper bound for the estimation error.
• We develop a stochastic predictive update (SPU) algorithm with general inexact value function estimations and a ﬁnite-time convergence analysis of its duality gap (see Lemma 1 for the SPU algorithm and its convergence proof).
This generalizes the convergence result of the PU algorithm established in (Cen et al., 2021), which uses exact value functions based on full knowledge of the environment.
Finally, by incorporating our developed tight estimation error bounds into the ﬁnite-time duality gap bound of SPU, we obtain the desired convergence rate in Theorem
Based on Theorem 1, we obtain the following sample complexity of SPE for achieving an (cid:15) -Nash equilibrium duality gap of the original Markov game, i.e., D (0)
( π (1) K − 1 , π (2) K − 1 ) ≤ (cid:15) .
Here, we adopt an adaptive batch size scheduling scheme to optimize the complexity order.
The overall
sample
complexity is given by (cid:80) K − 1 k =0 (cid:2)
N k +1 + 2 (cid:80)
T k − 1 t =
( N k,t + N k,t +1 ) (cid:3) .
Theorem 2 (Sample complexity) .
Implement Algorithm 1 with η = O (cid:0) 1 − γ (cid:1) , τ = O (cid:0) (cid:15) (1 − γ )
ln
A max (cid:1) ,
K = O (cid:104) 11 − γ ln (cid:16) ln
A max (cid:15) (1 − γ ) (cid:17)(cid:105) and T k =[-
-]
[Equation]
Choose the following adaptive batch sizes.
[Equation]
Then, for any (cid:15) ≤ ln A max 1 − γ , the overall sample complexity to achieve D (0) ( π (1) K − 1 , π (2) K − 1 ) ≤ (cid:15) is (cid:101) O (cid:0)
[Equation]
Please refer to (118) in Appendix E for a complete expression.
The above complexity result is obtained by choosing a small τ = O (cid:0) (cid:15) (1 − γ )
ln
A max (cid:1) for the convergence rate result in Theorem 1.
Speciﬁcally, we show in Lemma 6 that the duality gap is Lipschitz continuous with regard to the entropy regularization parameter, i.e., (cid:12)(cid:12) D ( τ ) ( π (1) , π (2) )
− D (0) ( π (1) , π (2) ) (cid:12)(cid:12) ≤ 2 τ ln
A max 1 − γ .
Therefore, by choosing a proper small τ , convergence of the duality gap D ( τ ) of the entropy-regularized Markov game implies the convergence of the duality gap D (0) of the original
Markov game.
Remark on Improvement of Sample Complexity. Our sample complexity (cid:101) O (cid:0) [Table] substantially improves the state-of-the-art complexity (cid:101) O (cid:0) A 3max |S| 10 .
5 (cid:15) 8 µ min (1 − γ ) 29 .
5 (cid:1) established in (Wei et al., 2021). We next elaborate the improvement of sample complexity in two different levels: population level (with known environment) and stochastic level (using stochastic samples). First, in the population level, the original policy extragradient (PE) algorithm in (Cen et al., 2021) already achieves a faster convergence rate than the OGDA-based algorithm proposed in (Wei et al., 2021). Speciﬁcally, to achieve an (cid:15) -Nash equilibrium point of the original Markov game, PE requires O (cid:0) A max (1 − γ ) (cid:15)
ln 2 ( (cid:15) − 1 ) (cid:1) iterations by choosing a proper entropy regularization parameter τ = O ( (cid:15) (1 − γ ) ln A max ) . As a comparison, the OGDA-based algorithm requires (cid:101) O (cid:16)
{+[Equation] iterations by substituting (cid:15) = 
 (no stochastic error) and their choice+} of step size into the Theorem 1 of Wei et al.
This improvement is in the order of (cid:101) O ( (cid:15) − 1 ) and is due to the use of entropy regularization in the PE algorithm, which improves the geometry of the bilinear matrix game. Second, the rest of the improvement of sample complexity (about (cid:101) O ( (cid:15) − 1 . 5 ) ) comes from the stochastic level. Speciﬁcally, our stochastic PE (SPE) allows to use a large constant learning rate η = O (cid:0) 1 − γ (cid:1) , whereas the OGDA-based algorithm in (Wei et al., 2021) needs to use a substantially smaller learning rate
η = O ( (cid:112) (1 − γ ) 5 |S| − 1 ) , which signiﬁcantly slows down its convergence in both the population and stochastic levels.
Moreover, both the PE and our SPE take O (ln (cid:15) − 1 ) inner updates to achieve an accurate solution of the matrix game (7), whereas the OGDA-based algorithm uses only one inner update to solve the matrix game and hence suffers from a larger optimization error.
Finally, as we explained in the previous technical novelty paragraph, we improve the techniques used in bounding the estimation errors.
Speciﬁcally, Wei et al.
(2021) bounds the estimation errors in all the iterations by a small constant (cid:15) using independent samples.
As a comparison, we establish a key contraction property of the estimation error over the iterations with correlated Markovian samples. Such a property allows us to use a growing batch size that bounds the errors loosely in the initial iterations and achieves tight error bounds in the end. All these improvements together lead to establishment of the desired sample complexity of SPE.  C ONCLUSION
In this paper, we developed a model-free, provably convergent, sample efﬁcient, symmetric and private stochastic policy extra gradient algorithm for solving two-player zero-sum Markov games.
Our algorithm leverages entropy regularization to facilitate the algorithm convergence and develops new stochastic estimators to accurately estimate the value functions.
We proved that our SPE algorithm achieved a fast convergence rate in terms of the Nash equilibrium duality gap and moreover, achieves a substantially improved sample complexity over the state-of-the-art result.
We believe our algorithm deepens the understanding of Markov games from a computation complexity perspective.
In the future study, we are excited about exploring other topics along this direction.
For example, it is interesting to extend SPE algorithm to the multi-agent setting for solving general-sum Markov games and competitive games that involve multiple cooperative teams.
A A NALYSIS OF STOCHASTIC PU (SPU) FOR ENTROPY - REGULARIZED MATRIX GAME
We ﬁrst consider the following zero-sum entropy regularized matrix game, which can be considered as a simple special case of Markov game with only one state.
[Equation]
22) where Q ∈ R d 1 × d 2 is ﬁxed.
The solution ( µ ∗ τ , ν ∗ τ ) of the above problem, also known as the quantal response equilibrium (QRE), satisﬁes the following condition (Cen et al., 2021)
[Equation]
Our stochastic PU (SPU) algorithm for the above matrix game is written as follows,
[Equation]
[Equation] [Equation] [Equation]
[Equation]
where δ (1) t is an additive noise to the underlying true quantity Qν t , and the other noises δ (2) t , δ (1) t , δ (2) t are similar.
If all these noises are zero, the above stochastic PU algorithm reduces to the PU algorithm for matrix game deﬁned in Section 2 of (Cen et al., 2021).
The convergence rate of stochastic PU for matrix game is shown below.
The proof logic follows from that of (Cen et al., 2021).
Lemma 1.
Choose learning rate η ≤ (cid:2) 2( τ + (cid:107) Q (cid:107) ∞ ) (cid:3) − 1 and use the uniform policy initialization µ
= 1 /d 1 , ν
= 1 /d 2 .
Then, the SPU algorithm deﬁned by eqs.
(24) - (27) for entropy-regularized matrix game has the following convergence rates.
[Equation] [Equation]
Proof.
Eq.
(26) implies that for some constant c ∈ R ,
[Equation]
where we used (cid:104) µ t +1 − µ ∗ τ , 1 (cid:105) = 0 .
Similarly
[Equation]
eqs.
(27)&(23) respectively imply that
[Equation]
Similarly
[Equation]
eqs.
(24)&(26) imply that
[Equation]
where (i) uses the Pinsker’s inequality and η ≤ (cid:2) 2( τ + (cid:107) Q (cid:107) ∞ ) (cid:3) − 1 .
Similarly to the above derivation
[Equation]
we can infer from eqs.
that
[Equation]
(31)+eq. (32)- ητ [eq. (33)+eq.
(34)] yields that
[Equation]
+ (cid:10) ln
( ν t +1 /ν ∗ τ ) − (1 − ητ ) ln
( ν t /ν ∗ τ ) , ν t +1 − ν ∗ τ (cid:11) =
KL ( µ ∗ τ (cid:107) µ t +1 ) − (1 − ητ ) KL ( µ ∗ τ (cid:107) µ t ) + (cid:10) ln( µ t +1 /µ ∗ τ ) − ln( µ t +1 /µ
t +1 )
− (1 − ητ ) (cid:2) ln( µ t +1 /µ ∗ τ ) − ln( µ t +1 /µ t ) (cid:3) , µ t +1 (cid:11) + KL ( ν ∗ τ (cid:107) ν t +1 ) − (1 − ητ )
KL
( ν ∗ τ (cid:107) ν t ) + (cid:10) ln
( ν t +1 /ν ∗ τ ) − ln( ν t +1 /ν t +1 ) − (1 − ητ ) (cid:2) ln
( ν t +1 /ν ∗ τ ) − ln( ν t +1 /ν t ) (cid:3) , ν t +1 (cid:11) =
KL ( µ ∗ τ (cid:107) µ t +1 ) − (1 − ητ ) KL ( µ ∗ τ (cid:107) µ t ) + ητKL ( µ t +1 (cid:107) µ ∗ τ ) + (1 − ητ ) KL ( µ t +1 (cid:107) µ t )
+ KL ( µ t +1 (cid:107) µ t +1 )
+ KL ( ν ∗ τ (cid:107) ν t +1 ) − (1 − ητ )
KL
( ν ∗ τ (cid:107) ν t ) + ητKL ( ν t +1 (cid:107) ν ∗ τ ) + (1 − ητ )
KL ( ν t +1 (cid:107) ν t ) +
KL
( ν t +1 (cid:107) ν t +1 ) − (cid:10)
[Equation]
where (i) uses eqs.
The left side of the above inequality has the following upper bound.
[Equation]
Substituting eq.
(38) into eq.
(37) and rearranging it yields that
[Equation]
(39) where we use η ≤ (cid:2) [Equation] τ + [Equation]
2 τ )
[Equation]
This further implies that
[Equation]
Consequently, eq. (28) can be proved via iterating the above inequality and using the facts that
[Equation]
(Since µ 0 and ν 0 are uniform probability vectors).
Next
[Equation]
we will prove eq.
[Equation]
Computing eq.
(31) − ητ · eq.
(33) yields that
[Equation]
Hence, we conclude that
[Equation]
[Equation]
[Table]
[Equation]
where (i) uses eq.
(41)&(34), (ii) follows the derivation of eq.
(37), (iii) uses the Pinsker’s inequality, (iv) uses eq.
[Equation]
and (v) uses η ≤ (cid:2) 2( τ + (cid:107) Q (cid:107) ∞ ) (cid:3) − 1 .
In a similar way
[Equation]
we can also prove that
[Equation]
Combining eqs.
(42)&(43) yields that
[Equation]
where (i) uses eq.
This proves eq.
Next, to prove the duality gap (30), we ﬁrst derive an upper bound of KL ( µ (cid:63)τ (cid:107) µ t +1 )
+ KL ( ν (cid:63)τ (cid:107)
ν t +1 ) .
KL
[Equation]
∞
[Equation]
[Equation]
where (i) uses the Pinker’s inequality and η ≤ (cid:2) 2( τ + (cid:107) Q (cid:107) ∞ ) (cid:3) − 1 .
Rearranging the above inequality yields that
[Equation]
Similarly
[Equation]
we can prove that
[Equation]
Summing up eqs.
(45)&(46) yields that
[Equation]
where (i) uses eq.
(39) and η ≤ (cid:2) [Equation] τ + [Equation]
Finally [Equation] we prove the duality gap bound (30).
[Equation]
where (i) uses eq.
[Equation]
( ii [Equation] (49) where (i) uses eq.
(34) and (ii) uses the Pinsker’s inequality.
Then, eq. (49) & eq.
(48) imply that
[Equation]
, (50) where (i) uses eq.
(45), (ii) uses the Pinker’s inequality and η ≤ (cid:2) 2( τ + (cid:107) Q [Equation] ∞ [Equation]
Similarly, it can be proved that
[Equation]
Therefore, the duality gap (30) can be proved as follows.
[Equation]
[Table]
[Equation]
) where (i) adds up eqs.
(50) & (51), (ii) uses eq.
(47), (iii) uses eq.
(39), and (iv) uses eq.
(28) and 4 τ − 1 (cid:107) Q (cid:107) 2 ∞ ≤ 2 /η (since η ≤ (cid:2) 2( τ + (cid:107) Q (cid:107) ∞ ) (cid:3) − 1 , τ ≤ 1 ).
This proves eq.
B E STIMATION ERROR BOUNDS
In this section, we derive error bounds for the estimators (cid:98) V k
{++
 ≈ V (cid:48)+}
k +1 , (cid:98) Q ( m ) k,t ≈ Q ( m ) k,t , (cid:98) Q ( m ) k,t ≈ Q ( m ) k,t used in Algorithm 1.
For convenience, we deﬁne the following additional notations.
[Equation] [Equation]
[Equation] [Equation] [Equation] [Equation]
where eqs. (53)-(59) and (60)-(65) are introduced for the description and error bound proof of the estimations (cid:98)
V k
+
≈ V (cid:48)
k +1 and (cid:98) Q ( m ) k,t ≈ Q ( m ) k,t , respectively.
Speciﬁcally, eq.
(53) estimates the stationary state distribution µ k associated with the policy pair π (1) k , π (2)
k using state frequency (cid:98) µ k ( s ) .
In eqs.
(54)&(55), the expectation is taken over (cid:101) s ∼ µ k , (cid:101) a (1) ∼ π (1) k
( s ) , (cid:101) a (2) ∼ π (2) k
( s ) , s (cid:48) ∼
P ( ·| (cid:101) s, (cid:101) a (1) , (cid:101) a (2) ) .
The deﬁnition of E µ (cid:48) k,t ,π (cid:48) (1) k,t ,π (cid:48) (2) k,t in eqs.
(61)&(62) is similar.
Equations (58) & (64) give two equivalent deﬁnitions of V (cid:48)
k +1 and Q (cid:48)
( m ) k,t ( s, a ( m ) ) , respectively 1 (We will prove the equivalence in Lemma 2 below).
Equation (60) estimates the stationary state distribution µ (cid:48) k,t associated with the smoothed policy pair π (cid:48) (1) k,t , π (cid:48) (2) k,t , and thus approximates the stationary state distribution
µ k,t associated with the current policy pair π (1) k,t , π (2) k,t .
To prove the estimation error bounds, we ﬁrst prove the following two lemmas.
Lemma 2.
Eq.
(13) and the second “=” of eqs. (58) & (64) hold for all s , a (1) , a (2) .
Proof.
The second “=” of eq. (64) under m = 1 can be proved as follows.
[Equation]
where (i) uses eq.
(61) which denotes E µ (cid:48) k,t ,π (cid:48) (1) k,t ,π (cid:48) (2) k,t as expectation over (cid:101) s ∼ µ (cid:48)
k,t , (cid:101) a (1) ∼ π (cid:48) (1) k,t ( (cid:101) s ) , (cid:101) a (2) ∼ π (cid:48) (2) k,t ( (cid:101) s ) , s (cid:48) ∼ P ( ·| (cid:101) s, (cid:101) a (1) , (cid:101) a (2) ) , (ii) uses independence between (cid:101) s ∼ µ (cid:48)
k,t , (cid:101) a (1) ∼ π (cid:48) (1) k,t ( s ) and (cid:101) a (2) ∼ π (cid:48) (2) k,t ( s ) , s (cid:48) ∼ P ( ·| s, a (1) , (cid:101) a (2) ) , (iii) uses eq.
(57), and (iv) uses the ﬁrst deﬁnition of eq.
The second “=” of eq. (64) under m = 2 and eq.
(13) can be proved in a similar way.
The second “=” of eq. (58) can be proved as follows. [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where (i [Table] uses eq.
[Table] ii [Table] uses independence between
[Table] and
[Table] ∼ π [Table] 1) , (cid:101) a (2) ) , (iii) uses eq.
(57), and (iv) uses eq.
Lemma 3.
If (cid:107) (cid:98) V 0 (cid:107) ∞ ≤ V max :
= 1+ τ ln
A max 1 − γ , then for all k ≥ 0 , 0 ≤ t ≤ T k − 1 , we have
[Equation]
where Q max := 1 + γV max = 1+ γτ ln A max 1 − γ with A max := max (cid:0) |A
(1) | , |A (2) | (cid:1) .
Proof.
We will ﬁrst prove (cid:107) (cid:98) V k (cid:107) ∞
≤ V max in eq.
(66) by induction.
Suppose (cid:107) (cid:98) V k (cid:48) (cid:107) ∞
≤ V max holds for a certain k (cid:
) ∈ N .
Then, eq.
(59) implies that for all s ∈ S ,
[Equation]
where (i) uses the inequality that 0 ≤ H ( π ( m ) ) ≤ ln |A
( m ) | ≤ ln A max , ∀ π ( m ) ∈ ∆( |A ( m ) | ) .
Since (cid:107) (cid:98) V 0 (cid:107) ∞ ≤ V max , (cid:107) (cid:98) V k (cid:107) ∞
≤ V max for all k ∈ N .
A similar induction yields that (cid:107) V (cid:48) k (cid:107) ∞ ≤ V max .
Next, we prove (cid:107) V ∗ (cid:107) ∞ ≤ V max , (cid:107) Q ∗ (cid:107) ∞ ≤ Q max in eqs.
(66) & (68) respectively.
Notice that V ∗ and Q ∗ have the following relation.
[Equation]
Hence
[Equation]
Substituting eq.
(70) into eq. (71) yields that
[Equation]
The proof of eq.
(66) is ﬁnished.
Then, substituting (cid:107) V ∗ (cid:107) ∞ ≤ V max into eq.
(70) proves that [Equation] Q ∗ [Equation]
∞ ≤ Q max .
Equation (67) can be proved as follows.
[Equation]
where (i) uses eq.
(58) and (ii) uses V max :
= 1+ τ ln
A max 1 − γ [Equation] A max .
Next, we prove eq.
It can be easily seen from eqs.
that
[Equation]
(cid:12)(cid:12) ≤
[Equation]
Q max , which implies that (cid:107) Q k (cid:107) ∞ ≤ Q max .
Hence,
[Equation]
where (i) uses eq.
This proves eq.
With the above two Lemmas, we can derive the estimation error bounds as follows.
Lemma 4.
Use hyperparameter choices
[Equation]
[Equation] and N
[Equation]
1 ≥
[Equation]
min (cid:17) in Algorithm 1.
Then with probability at least 1 − δ , all the following bounds hold for all 0 ≤ k ≤ K − 1 , 0 ≤ t ≤ T k − 1 , m = 1 , 2 , s ∈ S , a ( m ) ∈ A ( m ) , where
T sum := (cid:80) K − 1 k =0 T k .
{+(cid:13)(cid:13)(cid:98) V k − V (cid:48) k (cid:13)(cid:13) ∞ ≤ 2 V max γ k + 
 V max 1 − γ (cid:115) t mix N k +1 µ min ln (cid:16) 4 K |S| δ √ µ min (cid:17) (72) (cid:13)(cid:13) (cid:98) Q ( m ) k,t − Q ( m ) k,t (cid:13)(cid:13) ∞+}
≤ Q max (cid:34) 640 t mix A max
N k,t µ min (cid:15) (cid:48)
ln (cid:16) 20 T sum |S| A max δ √ µ min (cid:17) +(cid:115) t mix A max N k,t µ min (cid:15) (cid:48) ln (cid:16) 20 T sum |S| A max δ √ µ min (cid:17) + (cid:15) (cid:48) (cid:35) + 2 (cid:13)(cid:13)(cid:98) V k − V (cid:48) k (cid:13)(cid:13) ∞ (73) (cid:13)(cid:13)(cid:98) Q ( m ) k,t − Q ( m ) k,t
(cid:13)(cid:13) ∞ ≤ Q max (cid:34) 640 t mix A max
N k,t µ min (cid:15) (cid:48)
ln (cid:16) 20 T sum |S| A max δ √ µ min (cid:17) +(cid:115) t mix A max
N k,t µ min (cid:15) (cid:48)
ln (cid:16) 20 T sum |S| A max δ √ µ min
[Equation] More speciﬁcally [Equation] the upper bounds (73) & (74) can be simpliﬁed respectively as follows when (cid:15) (cid:48)
= (cid:104) 650 t mix A max N k,t [Equation] ln (cid:16) [Equation]
|
[Equation]
| A
[Equation]
and (cid:15)
[Equation]
(cid:13)(cid:13)(cid:98)
[Equation]
Proof.
If the initial state distribution of the minibatch N k,t equals µ (cid:48) k,t , i.e., s k,t, 0 ∼ µ (cid:48) k,t , then N k,t (cid:98) q ( m ) k,t ( s, a ( m ) ) :
= (cid:80) i ∈N k,t g ( s i , a (1) i , a (2) i , s i +1 )
( g ( s i , a (1) i , a (2) i , s i +1 ) :
= (cid:2) R i + γ (cid:98) V k
( s i +1 ) (cid:3) 1 { s i =
s, a ( m ) i = a ( m ) } ) and its expectated value N k,t q (cid:48) ( m ) k,t
( s, a ( m ) ) satisfy the following concentration bound.
[Equation]
where P µ (cid:48) k,t is under the initial state distribution µ (cid:48) k,t , (i) uses Theorem 3.4 of (Paulin, 2015) and the inequalities that (cid:12)(cid:12)
g
( s i , a (1) i , a (2) i , s i + [Equation] − E s i ∼ µ
[Equation] k,t g ( s i , a (1) i , a (2) i , s i +1 ) (cid:12)(cid:12) ≤ [Equation] 1 + γV max )
[Equation] max (Based on Lemma 3) and that var s i ∼ µ
[Equation]
Q
[Equation]
) | s ) , (ii) uses Proposition 3.4 of (Paulin, 2015) which states that the pseudo spectral gap γ ps has a lower bound 1 / (2 t mix ) for any uniformly ergodic Markov chain (This condition holds for our MDP with ﬁnitely many states and actions).
Then, for any initial state distribution s t,k, 0 ∼ ξ , Proposition 3.10 of (Paulin, 2015) implies that
[Equation]
where (i) uses eq.
(77) and the inequality that E s ∼ ξ (cid:104) ξ ( s ) µ
[Equation]
k
[Equation]
t
[Equation]
s
[Equation]
≤
[Equation]
In a similar way
[Equation]
the following concentration bound can be proved for N k [Equation] t (cid:98) µ k [Equation]
t ( s ) : = (cid:80) i ∈N k,t 1 { s i = s } and N k,t µ (cid:48) k,t ( s ) =
[Equation] N k [Equation] µ k
[Equation]
t ( s ) (cid:3) .
[Equation]
where we use the inequalities that var µ (cid:48)
k,t 1 { s i = s } ≤ µ (cid:48) k,t ( s ) and that (cid:12)(cid:12)
{ s i = s } − E µ (cid:48)
k,t 1 { s i = s } (cid:12)(cid:12) ≤ 1 .
Letting the right hand sides of eqs.
(78)&(79) be upper bounded by δ/ 4 and applying the union bound yields that, with probability at least 1 − δ/ 2 , the following two inequalities simultaneously hold. [Equation] [Equation] min
[Equation]
t
[Equation]
s
[Equation]
(81) where (i) holds since N k,t ≥ 650 t mix A max µ min ln (cid:16) 20 T sum |S| A max δ √ µ min (cid:17) ≥ 650 t mix and µ (cid:48) k,t ( s ) ≥ µ min .
Similarly
[Equation]
it can be proved that the following two inequalities holds with probability at least 1 − δ/ 2 .
[Equation]
Hence, eqs.
(80)-(83) hold with probability at least 1 − δ .
In this case, we have that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
) where (i) uses eqs.
[Equation]
[Table]
s
[Table]
(cid:34)
[Table]
(cid:17)
+
[Table]
(cid:112)
[Table]
[Table]
[Equation]
[Equation]
(cid:17) , (85) where (i) uses eqs.
(58)&(59), (ii) uses eqs.
(67), (83)&(84), (iii) uses Q max ≤ V max , (iv) uses µ k ( s ) ≥ µ min and N k +1 ≥ 40 t mix µ min ln (cid:16)
[Equation]
min (cid:17) ≥ 40 t mix , and (v) uses N k +1 ≥ 40 t mix µ min ln (cid:16) 4 δ √ µ min (cid:17) .
Applying the union bound to the above inequality over all 0 ≤ k ≤ K − 1 , s ∈ S and taking maximum over s ∈ S , we obtain that with probability at least 1 − δ ,
[Equation]
Iterating the above inequality yields that with probability at least 1 − δ ,
[Equation]
86) where (i) uses the fact that (cid:107) V (cid:48) 0 (cid:107) ∞ , (cid:107) (cid:98) V 0 (cid:107) ∞ ≤ V max .
Hence, with probability at least 1 − 2 δ , eqs.
(80), (81) & (86) hold simultaneously.
In this case,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
where (i) uses eqs.
(64)&(65), (ii) uses eq.
(64), (iii) uses eq.
(68) and the inequality that (cid:98) µ k,t ( s ) ≥ µ (cid:48) k,t ( s ) / 2 implied by (i) of eq.
(81), (iv) uses eqs.
(80)&(81) and the following inequality based on eqs.
(61)&(62), and (v) uses µ (cid:48) k,t ( s ) ≥ µ min (based on Assumption 1), π (cid:48)
( m ) k,t ( a ( m ) | s ) =
(1 − (cid:15)
[Equation]
) π ( m ) k,t ( a ( m ) | s ) + (cid:15) (cid:48) / |A
( m ) | ≥ (cid:15) (cid:48) /A
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Notice that the following inequality always holds
[Equation] [Equation]
where (i) uses eq.
Hence, combining eqs.
[Equation]
it can be seen that with probability at least 1 − 2 δ , the following inequality holds
[Equation]
Similarly
[Equation]
it can be proved that with probability at least 1 − 2 δ ,
[Equation]
Finally
[Equation]
eqs.
(72)-(74) are proved by applying a union bound to eq.
(86) and to eqs.
(89)&(90) over all 0 ≤ k ≤ T − 1 , 0 ≤ t ≤ T k − 1 , m = 1 , 2 , s ∈ S , a ( m ) ∈ A ( m ) .
Now we consider the hyperparameter choice (cid:15)
[Equation]
=
[Equation] [Equation]
N
[Equation]
min
ln
[Equation]
20 T
[Equation]
S| A max δ √ µ min (cid:17)(cid:105) 1 / 3 .
First, this is a valid choice, since N k,t ≥
[Equation]
(cid:17) implies that (cid:15) (cid:48) ∈
Then, for this (cid:15) (cid:48) , the upper bound (73) is simpliﬁed as follows.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where (i) uses t mix A max
N
[Equation]
min
ln
[Equation] [Equation]
sum |S| A max
[Equation]
µ min (cid:17) =
(cid:15) (cid:48) 3 / 650 , and (ii) uses (cid:15) (cid:48) ∈ [0 , 1] .
This proves eq.
The proof of eq.
(76) is similar.
C P ROPERTIES OF THE D UALITY G AP
In this section, we prove some useful properties of the duality gap
[Equation]
Lemma 5.
For any policy pair π (1) , π (2) , it holds that
[Equation]
1) ( s ) , π (2) ( s ) (cid:3) − f τ
[Equation]
Q ( τ ) ∗ ( s
[Equation]
, π (1) ( s ) , π (cid:48) (2) ( s ) (cid:3)(cid:105) .
This Lemma generalized the Lemma 32 of (Wei et al., 2021) to entropy-regularized Markov game.
Proof.
Throughout this proof, we denote the policy pair ( π (1) ∗ τ , π (2) ∗ τ ) as the Nash equilibrium and their associated V-function and Q-function are respectively denoted as V ( τ ) ∗ and Q ( τ ) ∗ .
Note that
[Equation]
[Table]
[Equation]
where (i) uses f τ
[Equation]
min
[Equation]
Applying max s
[Equation]
π (cid:48) (1) to both sides of the above inequality and rearranging it yields that max
[Equation]
Similarly, we can obtain that [Equation] Therefore
[Equation]
[Table]
where (i) uses eqs.
Lemma 6.
D ( τ ) ( π (1) , π (2) ) is 2ln
A max 1 − γ -Lipschitz continuous with regard to τ .
Proof.
The deﬁnition of the state value function V ( τ ) π (1) ,π (2) in (3) can be rewritten as follows.
[Equation]
Hence, for any τ, τ (cid:48) ≥ 0 ,
[Equation]
where (i) uses 0 ≤ H ( π ( m ) ( s t )) ≤
ln |A
( m ) | ≤ ln A max .
Hence, this Lemma can be proved as follows.
[Equation]
[Table]
[Equation]
[Table]
where (i) uses eq.
[Table]
D P ROOF OF T HEOREM
Theorem 1 (Finite-time convergence rate) .
Apply Algorithm 1 to solve the entropy-regularized
Markov game with τ ∈ (0 , 1] .
Choose learning rate η = [2( τ + Q max )] − 1 , initialization (cid:107) (cid:98) V 0 (cid:107) ∞ ≤
V max and batch sizes N k,t , N k,t , N k +1 that satisfy (18) & (19) .
Then, the Nash equilibrium duality gap converges at the following rate with probability at least 1 − δ .
[Equation]
Proof.
First, consider the SPU iterations that are deﬁned by replacing Q ( m ) k,t , Q ( m ) k,t +1 in (8) with (cid:98) Q ( m ) k,t , (cid:98) Q ( m ) k,t +1 , respectively, for m = 1 , 2 .
We can apply Lemma 1 with the quantities being speciﬁed as follows.
• Q :=
Q k ( s ) , h τ ( µ, ν ) :
= f τ (cid:0) Q k ( s ); µ, ν (cid:1) , • µ t := π (1) k,t ( s ) ,
ν t := π (2) k,t ( s ) , µ t +1 :
=
π (1) k,t +1 ( s ) , ν t +1 :
= π (2) k,t +1 ( s ) , µ ∗ τ := π ∗ (1) k ( s ) ,
ν ∗ τ := π ∗ (2) k ( s ) ,
[Equation]
where π (1) ∗ k ( s )
[Equation]
π (2) ∗ k ( s ) is the solution policy pair to the minimax optimization problem min
[Equation]
Consequently, eqs.
(29) & (30) with t =
T k − 1 imply the following two inequalities, respectively
[Equation]
Then
[Equation]
deﬁne the soft Bellman operator B τ as follows.
[Equation]
where Q : S × A (1) × A (2) → R and Q ( s (cid:48) )
∈ R |A (1) |×|A (2) | with each entry given by [ Q ( s (cid:48) )] a (1) ,a (2) :
= Q ( s, a (1) , a (2) ) .
It can be proved that B τ is a contraction operator and has a unique ﬁxed point Q ( τ ) ∗ (Cen et al., 2021), that is,
[Equation]
As a result
[Equation]
we obtain that
[Equation]
where (i) uses eq.
(99) and (ii) uses eq.
Throughout the proof, suppose that eqs.
(72), (75) & (76) hold simultaneously which occurs with probability at least 1 − δ .
In this case, we have
[Equation]
where (i) uses eq.
(75) and the inequality that ( a + b ) 2 ≤ 2 a
+ 2 b 2 for any a, b ≥ 0 , (ii) uses eq.
(72) and also ( a + b ) 2 ≤ 2 a
+ 2 b 2 for any a, b ≥ 0 .
Similarly
[Equation]
we obtain that
[Equation]
Then [Equation] iterating eq.
(100) yields that
[Equation]
[Equation]
where (i) uses eqs.
(68)&(95), (ii) uses eqs.
(101)&(102), (iii) uses η = [2( τ + Q max )] − 1 ≤ 1 / (2 τ ) , (cid:80) T k − 1 t =0 (1 − ητ ) T k − 2 − t ≤
[Equation]
(1 −
[Equation]
) ≤ 2 ητ and (cid:80) K {+− 2 k =
+}
γ K + k − 1 ≤ γ K − 1 1 − γ , and (iv) uses Q max =
O ( V max ) for γ ≈ 1 and η = [2( τ + Q max )] − 1 =
O ( Q − 1max ) =
O ( V − 1 max ) (Since Q max ≥ γτ =
O ( τ ) ).
Using Lemma 5, the convergence rate of the duality gap in eq.
(20) can be proved as follows.
[Equation]
(cid:17)(cid:105)(cid:33)
[Equation]
[Table]
[Equation]
where (i) uses Lemma 5, (ii) uses eq.
(96) and the deﬁnition of the function f τ , (iii) uses eqs.
(101)&(102), (iv) uses eq.
(103) and η = [2( τ + Q max )] − 1 ≤ 1 / (2 τ ) , (cid:80) T k − 1 t =0 (1 − ητ )
T k − 2 − t ≤ητ (1 − ητ ) ≤ 2 ητ , and (v) uses Q max
=
O ( V max ) for γ ≈ 1 and η = [2( τ + Q max )] − 1 =
O ( Q − 1max ) =
O ( V − 1 max ) (Since Q max ≥ γτ
= O ( τ ) ).
E P ROOF OF T HEOREM
Theorem 2 (Sample complexity) .
Implement Algorithm 1 with η = O (cid:0) 1 − γ (cid:1) , τ = O [Equation] [Equation]
γ )
ln
[Equation] max
[Equation] K = O (cid:104) [Equation]
γ ln (cid:16) ln A max (cid:15) (1 − γ ) (cid:17)(cid:105) and T k = {+
+}
+ k ln γ − 1 ln(1 − ητ ) − 1 .
Choose the following adaptive batch sizes.
[Equation]
Then, for any (cid:15) ≤ ln A max 1 − γ , the overall sample complexity to achieve D (0) ( π (1) K − 1 , π (2) K − 1 ) ≤ (cid:15) is (cid:101) O (cid:0)
[Equation]
5 (cid:1) .
Please refer to (118) in Appendix E for a complete expression.
Proof.
Since τ = O (cid:0) (cid:15) (1 − γ )
ln
A max (cid:1) , we have
[Equation] [Equation]
Since T k = 1 + k ln γ − 1 ln(1 − ητ ) − 1 , we have
[Equation]
max
[Equation]
max
[Equation]
1 − γ ) (cid:17)(cid:105) .
Hence, ln (cid:16) T sum |S| A max δµ min (cid:17)
[Equation]
(cid:16) |S| A max
[Equation]
A max δµ
[Equation]
(cid:15) (1 − γ )(cid:17)
[Equation] ln (cid:16) [Equation] ln (cid:16) ln
A max (cid:15) (1 − γ )
(cid:17)(cid:17)(cid:105) =
O (cid:104) ln (cid:16) |S| A max (cid:15)δµ [Equation] (1 − γ ) (cid:17)(cid:105) .
Similarly, ln (cid:16) [Equation] | [Equation]
min (cid:17) [Equation] O (cid:104) [Equation]
(cid:16)(cid:16) | [Equation] |
[Equation] [Equation] ) (cid:17)
[Equation]
(cid:16) ln A max (cid:15) (1 − γ ) (cid:17)(cid:17)(cid:105)
[Equation]
where (i) uses (cid:15) ≤ ln A max 1 − γ .
With the above equalities
[Equation]
we will prove below that the hyperparameter choices in Theorem 2 are valid and satisfy the conditions of Theorem 1 with proper constants hidden in O ( · ) [Equation]
[Equation]
≥(112)
T k =1 + k ln γ −ln(1 − ητ )
{+− 1 ≥(113) N k +
 = O (cid:104) t mix (ln 2 A max ) γ − k/(cid:15) 2 µ min (1 − γ )ln+} (cid:16) |S| ln( (cid:15) − 1 ln A max ) δµ min (1 − γ ) (cid:17)(cid:105) ≥ 650 t mix µ min ln (cid:16) δ √ µ min (cid:17) =
O (cid:104) t mix µ min ln (cid:16) δµ min (cid:17)(cid:105) (114)
N k,t = O (cid:104) t mix A max (1 − ητ ) − 3( t +1)
/µ min (1 − γ )ln (cid:16) |S| A max (cid:15)δµ min (1 − γ ) (cid:17)(cid:105) ≥ 650 t mix A max µ min ln (cid:16) 20 T sum |S| A max δ √ µ min (cid:17)
= O (cid:104) t mix A max µ min ln (cid:16) |S| A max (cid:15)δµ min (1 − γ ) (cid:17)(cid:105) (115)
N k,t = O (cid:104) t mix A max (ln 3 / 2 A max )(1 − ητ ) − 3( t +1)
/µ min (cid:15) 3 / 2 (1 − γ )ln (cid:16) |S| A max (cid:15)δµ min (1 − γ ) (cid:17)(cid:105) [Equation]
With these hyperparameters and eqs.
[Equation]
the duality gap bound (20) can be simpliﬁed as follows,
[Equation]
|S| A max (cid:15)δµ min (1 − γ ) (cid:17)(cid:105) 2 / 3 K − 1 (cid:88) k
= γ K − k − 1 T k − 1 (cid:88) t
= (1 − ητ )
T k − 2 − t (cid:32)
O (cid:104) t mix A max µ min (1 − γ ) 3 ln (cid:16) |S| A max (cid:15)δµ min (1 − γ ) (cid:17)(cid:105) − 2 / 3 (1 − ητ ) 2( t +1) /+
O (cid:16) ln A max (cid:15) (1 − γ )(cid:17) O (cid:104) t mix A max (ln 3 / 2 A max ) µ min (cid:15) 3 / 2 (1 − γ )ln (cid:16) |S| A max (cid:15)δµ min (1 − γ ) (cid:17)(cid:105) − 2 / 3 (1 − ητ ) 2( t +1) / 5 (cid:33) + γ K (ln 2 A max ) (cid:15) 2 (1 − γ )+ t mix (ln 2 A max ) (cid:15) 2 µ min (1 − γ ) 8 ln (cid:16) |S| ln( (cid:15) − 1 ln A max ) δµ min (1 − γ ) (cid:17)
K − 1 (cid:88) k
= γ K − k − 1 O (cid:104) t mix (ln 2 A max ) γ − k/(cid:15) 2 µ min (1 − γ )ln (cid:16) |S| ln( (cid:15) − 1 ln A max ) δµ min (1 − γ ) (cid:17)(cid:105) − 1 (cid:35) ≤ O (cid:34) ln
{+A max (1 − γ ) 
+}
Kγ K +
K − 1 (cid:88) k
= γ K − k − 1 T k − 1 (cid:88) t
=
(1 − ητ )
T k − 1 − 3( t +1) /+
γ K (ln 2 A max ) (cid:15) 2 (1 − γ )
+
K − 1 (cid:88) k
= γ K − k/ 2 −(cid:35) ( ii ) ≤ O (cid:34) ln
A max (1 − γ ) {+
+}
Kγ K +
K − 1 (cid:88) k
= γ K − 1 (1 − ητ ) − 3 / 5 [(1 − ητ ) − 3 T k / 5 − 1] (1 − ητ )
− 3 / 5 −+ γ K
(ln 2 A max ) (cid:15) 2 (1 − γ )
+ γ K − 1 γ − K/ 2 −γ − 1 / 2 −(cid:35)
( iii ) ≤ O (cid:34) ln
A max (1 − γ ) {+
+}
Kγ K +
K − 1 (cid:88) k
= γ K − 1 − 3 k/1 − (1 − ητ ) 3 / 5 + γ K
(ln 2 A max ) (cid:15) 2 (1 − γ )
+ γ ( K {+− 1) /
+}
− γ 1 /(cid:35)
( iv ) ≤ O (cid:34) ln
A max (1 − γ ) {+
+}
{+Kγ K + γ K − 1 ( γ − 3 K/ 5 − 1) ητ ( γ − 3 / 5 − 1) + γ K (ln 2 A max ) (cid:15) 2 (1 − γ ) + γ K/1 − γ (cid:35) ( v ) ≤ O (cid:34) γ K ln A max (1 − γ ) 3 ln (cid:16) ln A max (cid:15) (1 − γ ) (cid:17) + γ 2 K/ 
+}
ln
[Equation]
where (i), (ii) and (iii) use (1 − ητ )
T k − 1 = γ k based on eq.
(113), (i) also uses eqs.
(105), (108)-(111) & (114)-(116), (iv) uses 1 − (1 − ητ ) 3 /
= O ( ητ ) , 1 − γ 1 / 2 =
O (1 − γ ) , (v) uses eqs.
(110)- (112), (vi) uses O (cid:104)
[Equation]
[Equation] [Equation]
and γ K =
[Equation]
ln
[Equation] [Equation]
max implied by eq.
(112), and the numeric constant c > 0 in (vii) exists.
By replacing (cid:15) with (cid:15)/c , we obtain that
D τ (cid:0) π (1) K − 1 , π (2) K − 1 (cid:1) ≤ (cid:15) which means ( π (1) K − 1 , π (2) K − 1 ) is an (cid:15) -Nash equilibrium policy pair, and the orders of all the hyperparameter choices remain the same.
In this case, the overall sample complexity is given by
[Equation]
[Table]
ln A max (cid:15) (1 − γ )(cid:105) K − 1 (cid:88) k
= γ − 3 k/≤ O (cid:104) t mix (ln 2 A max ) (cid:15) 2 µ min (1 − γ ) 8 ln (cid:16) |S| ln( (cid:15) − 1 ln A max ) δµ min (1 − γ ) (cid:17)(cid:105) γ − K/+ O (cid:104) t mix A max (ln 5 / 2 A max ) µ min (cid:15) 5 / 2 (1 − γ )ln (cid:16) |S| A max (cid:15)δµ min (1 − γ ) (cid:17)(cid:105) γ − 3 K/γ − 3 / 5 −( iii ) ≤ O (cid:104) t mix (ln 2 A max ) (cid:15)
2 µ min (1 − γ ) 8 ln (cid:16) |S| ln( (cid:15) − 1 ln A max ) δµ min (1 − γ ) (cid:17)(cid:105) {+ln 5 / 4 A max (cid:15) 5 / 2 (1 − γ ) 
+}
.+ O (cid:104) t mix A max (ln 5 / 2 A max ) µ min (cid:15) 5 / 2 (1 − γ )ln (cid:16) |S| A max (cid:15)δµ min (1 − γ ) (cid:17)(cid:105) ln
3 / 2 A max (cid:15) 3 (1 − γ ) 5 .
( iv ) =
O (cid:104) t mix A max (ln 4 A max ) µ min (cid:15) 5 .
(1 − γ ) 13 .
5 ln (cid:16) |S| A max (cid:15)δµ min (1 − γ ) (cid:17)(cid:105) (118) =
(cid:101) O (cid:0) t mix A max µ min (cid:15) 5 . 5 (1 − γ ) 13 .(cid:1) where (i) uses N k,t ≤ N k,t and eqs.
(114)&(116), (ii) uses γ − 1 / 2 −
= O (1 − γ ) , 1 − (1 − ητ ) 3 / 5 =
O ( ητ ) =
O (cid:0) (cid:15) (1 − γ )
ln A max (cid:1) and (1 − ητ )
T k − 1 = γ k , (iii) uses γ − 3 / 5 −
= O (1 − γ ) and γ K = (cid:15) 5 (1 − γ ) 7 .
5 ln 5 / 2 A max implied by the hyperparameter choice (112), and (iv) uses the fact that the second term of (iii) is larger than the ﬁrst term of (iii).
F E XPLANATION OF T ABLE
In this section, we explain Table (1) with more details.
In the column “Model-free”, a (cid:88) means that an algorithm does not need any prior knowledge of the environment, including reward mapping and transition kernel.
In the column “Private update”, a (cid:88) means that the algorithm updates do not involve the opponent’s sensitive information, including action and policy.
In the column “Symmetric update”, a (cid:88) means that both players perform symmetric updates.
In the column “Data type”, “Markovian” means that the algorithm uses samples queried from the dependent Markov decision process.
The “–” sign means no stochastic samples are used.
The sample complexity is deﬁned as the total number of samples required to achieve an (cid:15) -duality gap for the Markov game.
A “–” sign in this column means that such type of sample complexity is unavailable in that paper or its setting is different from ours.
We use (cid:101) O to hide all the logarithm factors.
We next explain how the sample complexity of (Wei et al., 2021) is calculated.
Corollary 4 of (Wei et al., 2021) shows that the iterate-average duality gap is smaller than (cid:15) with the hyper-parameter choices: number of iterations T
= (cid:101) O (cid:104)
[Equation]
(cid:105) and number of samples queried per iteration
[Equation]
(we replace ξ with (cid:15) ).
Hence, the overall sample complexity is
[Equation]
where (i) uses the choice of learning rate η ≤ O (cid:104)(cid:113) (
[Equation]
γ ) 5 |S| (cid:105) required by their Algorithm 1.
[Equation]
In this section, we will prove that the policy extragradient (PE) algorithm Cen et al.
(2021) is rational.
The following analysis can be directly extended to our stochastic policy extragradient (SPE) algorithm by applying the estimation error bounds in Appendix B.
An algorithm is called rational if, whenever the player 2 adopts an arbitrary stationary policy π (2) , the policy of the player 1 converges to the best response to π (2) , i.e., (cid:101) π (1) ∗ τ ( s ) =
arg max π (1) ( s ) V ( τ ) π (1) ,π (2) ( s ) for all s ∈ S .
We add tilde in (cid:101) π (1) ∗ τ ( s ) to differentiate it from the Nash policy pair ( π (1) ∗ τ ( s ) , π (2) ∗ τ ( s )) and some consequent notations are similar.
Denote (cid:101) Q ( τ ) ∗ := Q
[Equation]
(cid:101) V ( τ
[Equation]
∗ :=
[Equation]
(cid:101) π (1) ∗ τ ,π (2) as the optimal value functions and deﬁne the soft Bellman operator as follows
[Equation]
[Equation]
Similar to eqs.
(98)&(99), it can be proved that the soft Bellman operator (cid:101) B τ is also a contraction operator and has a unique ﬁxed point (cid:101) Q ( τ ) ∗ , i.e., (cid:107)
[Equation]
{+To solve modiﬁed Markov game with ﬁxed policy π (2) , we develop the single-player-perspective version of the PE algorithm in Algorithm 2, which basically approximates the value iteration Q k +
+}
= (cid:101) B
τ ( Q k ) .
The main modiﬁcation from the original PE algorithm is that the new policy update (126) only keeps the third expression of the PU steps (8) since only that affects the ﬁnal policy of player 1 with ﬁxed π (2) .
This new policy update (126) aims to ﬁnd π (1) that maximizes the following single-player Markov game problem with ﬁxed Q k ( s ) and π (2) ( s ) for all s ∈ S , while the original
PU algorithm seeks the Nash equilibrium ( π (1) , π (2) ) of the following function.
[Equation]
[Equation]
Similar to eq.
(23), it can be derived that the solution (cid:101) π ∗ (1) k
( s ) to the above optimization problem satisﬁes the following condition
Cen et al.
[Equation]
[Equation]
Initialize: V 0 ( s ) for all s ∈ S .
for value iterations k = 0 , 1 , . .
, K − 1 do
Compute the following Q functions for all ( s, a (1) , a (2) ) .
[Equation]
Initialize π (1) k, 0 with uniform distribution.
for PU iterations t = 0 , 1 , . . .
, T − 1 do
Player 1 implements the t -th policy update for all s, a (1) as follows.
[Equation]
end
Let π (1) k
= π (1) k,T , and perform the following value iteration for all s
[Equation]
127) end
Output: π (
[Equation]
We ﬁrst prove that the value functions V k and Q k in Algorithm 2 are bounded as follows.
Lemma 7.
If (cid:107) V 0 (cid:107) ∞
≤ V max :
= 1+ τ ln
A max 1 − γ in Algorithm 2, then for all k ≥ 0 , 0 ≤ t ≤ T k − 1 , we have
[Equation]
where Q max := 1 + γV max = 1+ γτ ln A max 1 − γ with A max := max (cid:0) |A
(1) | , |A (2) | (cid:1) .
Proof.
The proof is similar to that of Lemma 3.
Similar to Lemma 1, we can prove the following lemma about the convergence rates of the modiﬁed
PU steps deﬁned by eq.
Lemma 8.
Apply Algorithm 2 to solve the entropy regularized game with τ > 0 .
Choose learning rate η ≤ τ − 1 and initialization (cid:107) V 0 (cid:107) ∞ ≤ V max .
Then, the policy update deﬁned by eq.
(126) for maximizing the function (122) has the following convergence rates.
[Equation]
Proof.
Taking logarithm of eq.
(126) yields that there exists c k,t ∈ R such that
[Equation]
Iterating the above equality over t = 0 , 1 , . .
. , t yields that there exists c ∈ R such that
[Equation]
where we use the uniform policy initialization π (1) k, 0 ( s ) = 1 / |A (1) | .
Taking logarithm of eq.
(123) yields that there exists c ∗ ∈ R such that
[Equation]
Therefore, eq.
(130) can be proved as follows.
[Equation]
) T where (i) uses eqs.
(132)&(133), (ii) uses (cid:104) (cid:101) π ∗ (1) k
[Equation]
{+s ) − π (1) k,T ( s ) , 1 (cid:105) = 
+}
and (iii) uses (cid:107) Q k ( s ) (cid:107) ∞ ≤
Q max , (cid:107) (cid:101) π ∗ (1) k − π (1) k,T (cid:107) 1 ≤ 2 , (cid:107) π 2 ( s ) (cid:107) 1 = 1 and 1 − ητ > 0 .
[Equation]
where (i) uses eq.
(133) and (cid:10)(cid:101) π ∗ (1) k
{+( s ) − π (1) k,T ( s ) , 1 (cid:11) = 
+}
and (ii) uses eq.
Finally
[Equation]
we prove the convergence rate of Algorithm 2 as follows.
Theorem 3.
Apply Algorithm 2 to solve the entropy regularized game with τ > 0 .
Choose learning rate η ≤ τ − 1 and initialization (cid:107) V 0 (cid:107) ∞ ≤ V max .
Then, the Q function estimation error (cid:107) Q K − 1 − (cid:101) Q ( τ ) ∗ (cid:107) ∞ and the function value gap max s (
cid:2)(cid:101) V ( τ ) ∗ ( s ) − V ( τ ) π (1) K − 1 ,π
[Equation]
( s ) (cid:3) converge at the following rates
[Equation]
Proof.
[Equation]
[Table]
[Equation]
where (i) uses eq.
(121), (ii) uses eq.
(120), (iii) uses eq.
Iterating the above inequality yields that
[Equation]
which proves eq.
Finally
[Equation]
eq.
(135) can be proved as follows
[Equation]
Applying max s to both sides of the above inequality and rearranging it yields that
[Equation]
[Table]
[Equation]
where (i) uses eq.
(131) and the following inequality that holds for all s, π (1) ( s ) , π (2) ( s ) , (ii) uses eq.
(134) and V K − 1 ( s ) =
f τ (cid:0) Q ( τ ) K − 1 ( s ); π (1) K − 1 ( s ) , π (2) ( s ) (cid:1) .
[Equation]
T .
In Theorem 3, the convergence rates of both the Q function estimation error (cid:107) Q K − 1 − (cid:101) Q ( τ ) ∗ (cid:107) ∞ and the function value gap max s (cid:2)(cid:101) V ( τ ) ∗ ( s ) − V ( τ ) π (1) K − 1 ,π (2) ( s ) (cid:3) consist of two terms O ( γ K ) and O (1 − ητ ) T , which characterize the exponential convergence rates of the K outer value iterations and the T inner policy updates respectively.
Such a convergence result indicates that Algorithm 2 converges to the optimal solution to the regularized Markov game, and thus can be arbitrarily close to the optimal solution of the unregularized Markov game with sufﬁciently small τ .
As a result, the PU algorithm Cen et al.
(2020) is rational.
Similar result can be directly extended to our stochastic policy extragradient (SPE) algorithm by applying the estimation error bounds in Appendix B.