C AN W E F IND N ASH E QUILIBRIA AT A L INEAR R ATE IN M ARKOV G AMES ?
A BSTRACT
We study decentralized learning in two-player zero-sum discounted Markov games where the goal is to design a policy optimization algorithm for either agent satisfying two properties.
First, the player does not need to know the policy of the opponent to update its policy.
Second, when both players adopt the algorithm, their joint policy converges to a Nash equilibrium of the game.
To this end, we construct a meta algorithm, dubbed as Homotopy-PO , which provably finds a
Nash equilibrium at a global linear rate .
In particular, Homotopy-PO interweaves two base algorithms Local-Fast and Global-Slow via homotopy continuation.
Local-Fast is an algorithm that enjoys local linear convergence while Global-Slow is an algorithm that converges globally but at a slower sublinear rate.
By switching between these two base algorithms, Global-Slow essentially serves as a “guide” which identifies a benign neighborhood where
Local-Fast enjoys fast convergence.
However, since the exact size of such a neighborhood is unknown, we apply a doubling trick to switch between these two base algorithms.
The switching scheme is delicately designed so that the aggregated performance of the algorithm is driven by Local-Fast .
Furthermore, we prove that Local-Fast and Global-Slow can both be instantiated by variants of optimistic gradient descent/ascent (OGDA) method, which is of independent interest.
I NTRODUCTION
Multi-agent reinforcement learning (MARL) which studies how a group of agents interact with each other and make decisions in a shared environment (Zhang et al., 2021a) has received much attention in recent years due to its wide applications in games (Lanctot et al., 2019; Silver et al., 2017; Vinyals et al., 2019), robust reinforcement learning (Pinto et al., 2017; Tessler et al., 2019;
Zhang et al., 2021b), robotics (Shalev-Shwartz et al., 2016; Matignon et al., 2012), among many others.
Problems in MARL are frequently formulated as Markov Games (Littman, 1994; Shapley, 1953).
In this paper, we focus on one important class of Markov games: two-player zero-sum
Markov games.
In such a game, the two players compete against each other in an environment where state transition and reward depend on both players’ actions.
Our goal is to design efficient policy optimization methods to find Nash equilibria in zero-sum
Markov games.
This task is usually formulated as a nonconvex-nonconcave minimax optimization problem.
There have been works showing that Nash equilibria in matrix games, which are a special kind of zero-sum Markov games with convex-concave structures, can be found at a linear rate (Gilpin et al., 2012; Wei et al., 2020).
However, due to the nonconvexity-nonconcavity, theoretical understanding of zero-sum Markov games is sparser.
Existing methods have either sublinear rates for finding Nash equilibria, or linear rates for finding regularized Nash equiliria such as quantal response equilibria which are approximations for Nash equilibria (Alacaoglu et al., 2022; Cen et al., 2021;
Daskalakis et al., 2020; Pattathil et al., 2022; Perolat et al., 2015; Wei et al., 2021; Yang &
Ma, 2022; Zeng et al., 2022; Zhang et al., 2022; Zhao et al., 2022).
A natural question is:
Q1:
Can we find Nash equilibria for two-player zero-sum Markov games at a linear rate?
Furthermore, in Markov games, it is desirable to design decentralized algorithms.
That is, when a player updates its policy, it does not need to know the policy of other agents, as such information is usually unavailable especially when the game is competitive in nature.
Meanwhile, other desiderata in MARL include symmetric updates and rationality .
Here symmetry means that the algorithm employed by each player is the same/symmetric, and their updates differ only through using the different local information possessed by each player.
Rationality means that if other players adopt stationary policy, the algorithm will converge to the best-response policy (Sayin et al., 2021; Wei et al., 2021).
In other words, the algorithm finds the optimal policy of the player.
In decentralized learning, each player observes dynamic local information due to the changes in other players’ policy, which makes it more challenging to design efficient algorithms (Daskalakis et al., 2020; Hernandez-Leal et al., 2017; Sayin et al., 2021).
Symmetric update also poses challenges for the convergence.
Condon (1990) shows multiple variants of value iteration with symmetric updates can cycle and fail to find NEs.
Gradient descent/ascent (GDA) with symmetric update can cycle even in matrix games (Daskalakis et al., 2018; Mertikopoulos et al., 2018).
Thus, an even more challenging question to pose is:
Q2:
Can we further answer Q1 with a decentralized algorithm that is symmetric and rational?
In this paper, we give the first affirmative answers to Q1 and Q2 .
In specific, we propose a meta algorithm Homotopy-PO which provably converges to a Nash equilibrium (NE) with two base algorithms Local-Fast and Global-Slow .
Homotopy-PO is a homotopy continuation style algorithm that switches between Local-Fast and Global-Slow , where Global-Slow behaves as a “guide” which identifies a benign neighborhood for Local-Fast to enjoy linear convergence.
A novel switching scheme is designed to achieve global linear convergence without knowing the size of such a neighborhood.
Next, we propose the averaging independent optimistic gradient descent/ascent (Averaging OGDA) method and the independent optimistic policy gradient descent/ascent (OGDA) method.
Then, we instantiate Homotopy-PO by proving that Averaging OGDA and OGDA satisfy the conditions of Global-Slow and Local-Fast , respectively.
This yields the first algorithm which provably finds Nash equilibria in zero-sum Markov games at a global linear rate.
In addition, Homotopy-PO is decentralized, symmetric, rational and last-iterate convergent.
Our contribution.
Our contribution is two-fold.
First, we propose a meta algorithm
Homotopy-PO which is shown to converge to Nash equilibria of two-player zero-sum Markov games with global linear convergence, when the two base algorithms satisfy certain benign properties.
Moreover, Homotopy-PO is a decentralized algorithm and enjoys additional desiderata in
MARL including symmetric update, rationality and last-iterate convergence.
Second, we instantiate
Homotopy-PO by designing two base algorithms based on variants of GDA methods, which are proved to satisfy the conditions required by Homotopy-PO .
In particular, we prove that the example base algorithm OGDA enjoys local linear convergence to Nash equilibria, which might be of independent interest.
1. R ELATED WORK
A more comprehensive literature review is moved to Appendix A due to the space limitation.
Of particular relevance are two decentralized algorithms Daskalakis et al.
(2020) and Wei et al.
Daskalakis et al.
(2020) consider an independent policy gradient descent/ascent algorithm which is a natural extension of single-agent policy gradient descent to two-player zero-sum Markov games.
They utilize the two-sided gradient dominance to prove a sub-linear convergence rate of the gradientdescent-ascent (GDA) method.
This is the first non-asymptotic convergence result of GDA for finding Nash equilibria in Markov games.
However, their method is asymmetric, where one-player takes much smaller steps than its opponent.
And their convergence results are base on average policies with no explicit guarantee for last-iterate convergence.
Wei et al.
(2021) propose an actor-critic optimistic policy gradient descent/ascent algorithm that is simultaneous decentralized, symmetric, rational and has O (1 / √ t ) last-iterate convergence rate to the Nash equilibrium set.
They use a critic which averages the approximate value functions from past iterations to tame nonstationarity in approximate Q-functions and get better approximations for policy gradients.
A classical averaging stepsize from Jin et al.
(2018) is utilized by the critic so that the errors accumulate slowly and last- iterate convergence is obtained.
However, the critic using such choice of stepsizes throughout the game also destroys linear convergence since relatively large errors from past iterations cannot decay fast enough and will harm the accuracy of more recent steps.
N OTATIONS AND P RELIMINARIES
For integers n ≤ n ′ , we denote [ n : n ′ ]
= { n, n + 1 , · · · , n ′ } and [ n ] = { 1 , · · · , n } .
We use ∥·∥ to denote the Euclidean norm, and ∥ · ∥ p denotes the ℓ p -norm.
For any vector x ∈ R d and closed convex set C ⊆ R d , let P C ( x ) denote the unique projection point of x onto C .
In addition, the distance between x and
C is denoted by dist ( x , C ) =
∥ x − P C ( x ) ∥ .
Markov game.
A two-player zero-sum discounted Markov game is denoted by a tuple MG =
( S , A , B , P , R , γ ) , where S =
[ S ] is the state space;
A =
[ A ] and B =
[ B ] are the action spaces of the min-player and the max-player respectively; P :
S ×
A × B → ∆ S is the transition kernel, R = { R s } s ∈S ⊆
A × B is the reward function, and γ is the discount factor.
Specifically, at state s , when the min-player takes action a and the max-player takes action b at state s , P ( s ′ | s, a, b ) is probability that the next state becomes s ′ , R s ( a, b ) is the reward received by the max-player, and the min-player receives an loss − R s ( a, b ) .
We assume that the rewards are bounded in [0 , 1] without loss of generality.
Let x = { x s } s ∈S and y =
{ y s } s ∈S denote the policies of the min-player and the max-player,
where x s ∈ ∆ A and y s ∈ ∆ B .
The policy spaces of the min-player and the max-player are denoted by X = (∆ A ) S , Y = (∆ B ) S .
Let Z =
X
× Y denote the product policy space.
The policy x ∈ X ( y ∈ Y ) is treated as an AS -dimensional ( BS -dimensional) vector, and the policy pair z
= ( x , y ) is treated as an ( A + B ) S -dimensional vector where z s =
( x s , y s ) represents an ( A + B ) -dimensional vector by concatenating x s and y s .
The value function under the policy pair ( x , y ) is defined as an S -dimensional vector with its entries representing the expected cumulative rewards: V x , y ( s ) =
E x , y [ (cid:80) + ∞ t =
γ t R s t ( a t , b t ) | s 0 = s ] .
Define V x , † ( V † , y ) as the value functions of x ( y ) with its best response, i.e., V x , † ( s ) = max y ′ ∈Y V x , y ′ ( s ) , V † , y ( s ) =
min x ′ ∈X V x ′ , y ( s ) .
For state s ∈ S , define the Bellman target operator Q s : R S → R A × B such for vector v ∈ R S ,
[Equation]
The Q-function Q x , y = { Q x , y s } s ∈S is defined as a collection of A -by- B matrices with Q x , y s =
Q s [ V x , y ] .
The (state) visitation distribution is defined as d x , y s ( s ′ )
= (cid:80) + ∞ t =
γ t Pr x , y [ s t = s ′
| s 0 = s ] .
For any distribution ρρρ ∈ ∆ S , we abbreviate V x , y ( ρρρ ) =
(cid:80) s ∈S ρρρ ( s ) V x , y ( s ) , d x ,
y ρρρ ( s ) =
(cid:80) s ′ ∈S ρρρ ( s ′ )
d x , y s ′ ( s ) .
From Gilpin et al.
(2012), there is a problem-dependent constant c + >such that for any policy pair z
= ( x
, y ) ∈
Z and s ∈ S ,
[Equation]
Nash equilibrium.
The minimax game value of state s is defined as v ∗ ( s ) =
min x ∈X max y ∈Y V x , y ( s ) = max y ∈Y min x ∈X V x , y ( s ) .
A policy pair ( x , y ) is called a Nash equilibirum (NE) if and only if: for any s ∈ S ,
[Equation]
Define the minimax Q-functions as Q ∗ s = Q s [ v ∗ ] .
Define the sets X ∗ s and Y ∗ s as
[Equation]
Then X ∗ s and Y ∗ s are non-empty, closed and convex.
Denote Z ∗ s =
X ∗ s × Y ∗ s .
Let X ∗ =
(cid:81) s ∈S X ∗ s ,
Y ∗ = (cid:81) s ∈S Y ∗ s , Z ∗ =
(cid:81) s ∈S Z ∗ s .
A policy pair ( x ∗ , y ∗ ) attains Nash equilibrium if and only if ( x ∗ , y ∗ )
∈
Z ∗ , i.e., ( x ∗ s , y ∗ s ) ∈
Z ∗ s for any s ∈ S (Bas¸ar & Olsder, 1998; Filar & Vrieze, 2012).
We denote the closure of the NE set’s neighborhood as B ( Z ∗ , c ) = { z ∈ Z : dist( z , Z ∗ ) ≤ c } .
Interaction protocol.
In each iteration, each player plays a policy and observes the marginal reward function and the marginal transition kernel, i.e., in iteration t , the min-player plays x t ∈ X , while the max-player plays y t ∈ Y .
The min-player receives the marginal reward function r tx : S×A → [0 , 1] with r tx ( s, a ) = (cid:80) b ∈B y ts ( b ) R s ( a, b ) and marginal transition kernel P tx : S ×
A → ∆ S with P tx ( s ′ | s, a ) = (cid:80) b ∈B y ts ( b ) P ( s ′ | s, a, b ) , while the max-player receives r ty and P ty which are defined analogously.
Each player is oblivious to its opponent’s policy.
A H OMOTOPY C ONTINUATION A LGORITHM WITH G LOBAL L INEAR C ONVERGENCE
We propose a decentralized algorithm with global linear convergence by (1) proposing a meta algorithm which can achieve global linear convergence with two base algorithms, (2) providing examples for the base algorithms.
The analysis for the example base algorithms are in Section 4 and Section 5.
3. A HOMOTOPY CONTINUTATION META ALGORITHM
We present a homotopy continuation meta algorithm.
It can achieve global linear convergence by switching between two base algorithms: Global-Slow base algorithm ( Global-Slow ) and LocalFast base algorithm ( Local-Fast ) .
Global-Slow is globally convergent, but only attains a (cid:101) O
( 1 T ) rate.
Local-Fast is not necessarily globally convergent but attains a linear convergence rate in a neighborhood of the Nash equilibrium set.
Global-Slow base algorithm : by calling Global-Slow ([ T 1 : T 2 ] , ˜ z , η ′ )
during time interval [ T 1 : T 2 ] where ˜ z
= (˜ x , ˜ y ) is the initial policy pair, the players play policy pair z t = ( x t , y t ) for each iteration t ∈
[ T 1 : T 2 ] , and compute an average policy pair (cid:98) z [ T 1 : T 2 ]
= ( (cid:98) x
[ T 1 : T 2 ] , (cid:98) y
[ T 1 : T 2 ] ) at the end of iteration T 2 such that z t , (cid:98) z [ T 1 : T 2 ] satisfy the following two properties:
• global convergence : there is a problem-dependent constant C ′
> such that
[Equation]
This property means the average policy produced by Global-Slow converges to the NE set at a sublinear (cid:101) O
(1 /T ) rate.
• geometric boundedness : there exists a problem-dependent constant D 0 > 0 (possibly D 0 > 1 ) such that if η ′ ≤ 1 , then for any t ∈ [ T 1 : T 2 ] ,
[Equation]
This property ensures that the iterate z t at any intermediate time t and the average policy (cid:98) z [ T 1 : T 2 ] do not diverge faster than geometrically from the NE set.
In Global-Slow , { z t } t ∈ [ T 1 : T 2 ] are the policy pairs played during [ T 1 : T 2 ] , while (cid:98) z
[ T 1 : T 2 ] will mainly be used as the initial policy in the next switch to Local-Fast in the meta algorithm Homotopy-PO (Algorithm 1).
Local-Fast base algorithm : by calling Local-Fast ([ T 1 : T 2 ] , (cid:98) z , η ) during time interval [ T 1 : T 2 ] where (cid:98)
z
= ( (cid:98) x , (cid:98) y ) is the initial policy pair, the players play policy pair z t = ( x t , y t ) for each iteration t ∈
[ T 1 : T 2 ] such that z t satisfies the local linear convergence property:
• local linear convergence : there exist problem-dependent constants c 0 ∈ (0 , 1) and δ 0 , Γ 0 >such that if dist 2 ( (cid:98) z , Z ∗ ) <
δ 0 η 4 , then for any t ∈ [ T 1 : T 2 ] dist
[Equation] [Equation]
t
[Equation]
In other words, if initialized a neighborhood of Z ∗ with radius (cid:112) δ
η 4 , Local-Fast converges to
Z ∗ at a linear rate.
With these base algorithms, a naive and impractical approach is to run Global-Slow first until z t reaches B ( Z ∗ , (cid:112) δ 0 η 4 ) , and then, run Local-Fast to achieve linear convergence.
However, the problem is we do not know the value of δ 0 .
That is, when running the algorithm, since
Z ∗ is unknown, it is impossible tell whether the algorithm has reached the benign neighborhood for Local-Fast to enjoy the linear rate .
Thus, we cannot decide when to switch from
Global-Slow to Local-Fast .
Algorithm
Input: iterations: [0 : T ] , initial policy pair: z 0 ∈ Z , stepsizes: η, η ′ >set k = 1 , (cid:101) I 0lf = − 1 , z − 1 = zwhile (cid:101)
I k − 1 lf < T do
I k
gs =
(cid:101)
I k − 1 lf + 1 , (cid:101)
I k
gs =
min
{I k gs + 2 k − 1 , T } , I k lf = (cid:101)
I k
gs
+ 1 , (cid:101)
I k lf =
min {I k lf + 4 k − 1 , T } during time interval [ I k gs : (cid:101)
I k gs ] , run Global-Slow
([ I k gs : (cid:101)
I k gs ] , z (cid:101)
I k − 1 lf , η ′ ) and compute an average policy (cid:98)
z
[ I k gs : (cid:101)
I k gs ] during time interval [ I k lf : (cid:101)
I k lf ] , run Local-Fast
([ I k lf : (cid:101) I k lf ] , (cid:98) z
[ I k gs : (cid:101)
I k gs ] , η ) k ← k + 1 end
To overcome this problem, we propose a homotopy continuation method Homotopy-PO which smartly switches between Global-Slow and Local-Fast .
The pseudocode is in Algorithm 1.
In Homotopy-PO , we split [0 : T ] into the segments:
[Equation]
I k lf ] ∪ · · · where [ I k gs : (cid:101)
I k gs ] is the time interval of the k -th call to Global-Slow and (cid:12)(cid:12)
[ I k gs : (cid:101)
I k gs ] (cid:12)(cid:12) =
2 k ; [ I k lf : (cid:101) I k lf ] is the time interval of the k -th call to Local-Fast and | [ I k lf : (cid:101) I k lf ] | = 4 k .
The switching scheme of Homotopy-PO method can be summarized as below: starting from k = 1 ,
• (Step 1) during time interval [ I k gs : (cid:101)
I k gs ] , run Global-Slow for (cid:12)(cid:12)
[ I k gs : (cid:101)
I k gs ] (cid:12)(cid:12) =
2 k iterations with the initial policy z (cid:101)
I k − 1 lf (for k ≥ 1 , it is the last-iterate policy of the last call to Local-Fast ) • (Step 2) during time interval [ I k lf : (cid:101) I k lf ] , run Local-Fast for (cid:12)(cid:12)
[ I k lf : (cid:101) I k lf ] (cid:12)(cid:12) =
4 k iterations with the initial policy (cid:98) z
[ I k gs : (cid:101)
I k gs
] that is the average policy of the last call to Global-Slow • (Step 3)
k ← k + 1 , goto Step 1.
Now, we elaborate on how Homotopy-PO achieves global linear convergence given a Global-Slow base algorithm and a Local-Fast base algorithm.
Specifically, there are two hidden phases which are oblivious to the players and only used for analysis.
The two phases are split by k ∗ =
max { k ∗ 1 , k ∗ 2 } , where 2 k ∗ 1 = (cid:101)
O (1 / ( √ δ 0 η 2 )) and 2 k ∗
=
O
( 1 c 0 η 2 log( D 0 Γ 0 ))
= (cid:101) O
(1 / ( c 0 η 2 )) .
The value of k ∗ is unknown to the players.
Hidden Phase I.
In the beginning, Global-Slow behaves like a “guide” in the sense that its average policy (cid:98) z
[ I k gs : (cid:101)
I k gs ] is getting closer to the NE set as k goes.
For small k , dist( z t , Z ∗ ) could possibly increase when running Local-Fast .
However, since the average policy (cid:98) z [ I k gs : (cid:101)
I k gs ] is the initial policy of the k -th call to Local-Fast , by the global convergence as in (2), for k ≥ k ∗ 1 , (cid:98)
z
[ I k gs : (cid:101)
I k gs ] will reach B ( Z ∗ , (cid:112)
δ 0 η 4 ) .
Thus, after k ≥ k ∗ 1 , each time when we switch to Local-Fast , it will exhibit linear convergence during time interval [ I k lf : (cid:101)
I k lf ] .
Hidden Phase II.
After k ≥ k ∗ 1 , Local-Fast enjoys fast linear convergence and becomes the main contributor to the convergence (see segments AB , CD in Figure 1).
Thanks to the fast convergence of Local-Fast , in this phase, dist( z t , Z ∗ ) can be much smaller than C ′ /t .
Note that we use z (cid:101)
I k − 1 lf as the initial policy of the k -th call to Global-Slow .
Thus, Global-Slow could possibly cause dist( z t , Z ∗ ) to increase.
However, instead of bounding dist( z t , Z ∗ ) by (2), now (3) can provide a tighter bound for dist( z t , Z ∗ ) when calling Global-Slow during Hidden Phase
II.
(3) implies that dist( z t , Z ∗ ) increases at most geometrically when running Global-Slow (see segments BC , DE in Figure 1).
After 2 k ≥ O ( 1 c 0 η 2 log( D 0 Γ 0 ))
( k ≥ k ∗ 2 ), the possible increase of dist( z t , Z ∗ ) caused by Global-Slow is much less than the decrease caused by
Local-Fast , and thus, can be “omitted”.
More specifically, in AB , dist 2 ( z t , Z ∗ ) converges at rate of 1 − c
η 2 for |
[ I k lf : (cid:101) I
k lf ] | = 4 k iterations, while in BC , dist 2 ( z t , Z ∗ ) diverges at rate of D 0 for |
[ I k +1gs : (cid:101)
I k +1gs ] | = 2 k +1 iterations.
Then, since 4 k / 2 k +1 = 2 k − 1 , if one step increase of Global-Slow is much smaller than 2 k − 1 steps of decrease of Local-Fast , i.e.,
D 0 (1 − c
η 2 / 2) 2 k − 1 ≪ 1 , then, we obtain the global linear convergence (see the line AC in Figure 1).
[Table]
[Figure]
Hidden
Phase
I has at most
O (4 k ∗ ) steps, where O (4 k ∗ ) is polynomial in
C ′ , 1 /c 0 , 1 /δ 0 , 1 /η, 1 /η ′ and only logarithmic in D 0 , Γ 0 .
Then, it enters Hidden Phase
II and linear convergence begins.
This yields the global linear convergence.
The formal proof is deferred to Appendix E.
Theorem 1 Let { z t = ( x t , y t ) } t ∈ [0: T ] be the policy pairs played when running
Homotopy-PO (Algorithm 1).
Then, there exists a problem-dependent constant D ≤ (cid:101) O
(poly( C ′ , 1 /c 0 , 1 /δ 0 , 1 /η, 1 /η ′ ))
such that for any t ∈ [0 : T ] , we have dist 2 ( z t , Z ∗ ) ≤
[Equation]
[Equation]
D , where the value of C ′ , c 0 , δ 0 , Γ 0 can be found in the definitions of Global-Slow and Local-Fast .
As D is independent of t , Theoremguarantees the global linear convergence of
Homotopy-PO .
3. E XAMPLES OF BASE ALGORITHMS
We introduce the averaging independent optimistic gradient descent/ascent (Averaging OGDA) method and the independent optimistic policy gradient descent/ascent (OGDA) method which will serve as examples for Global-Slow and Local-Fast respectively.
Both Averaging OGDA and
OGDA are symmetric, rational and decentralized algorithms.
The pseudocodes are in Appendix G.
Example of
Global-Slow base algorithm (Averaging
OGDA).
By running
Averaging-OGDA ([ T 1 : T 2 ] , ˜ z , η ′ ) with initial policy ˜ z
= (˜ x , ˜ y ) , the min-player initializes (cid:101) x T 1 = x T 1 = ˜ x and V T 1 ( s )
=
V † , ˜ y
( s ) , the max-player initializes (cid:101) y T 1 = y T 1 = ˜
y and
V T 1 ( s ) =
V ˜ x , † ( s ) , and they update for t ∈ [ T 1 + 1 : T 2 ] as follows:
[Equation]
where q js = Q s [ V j ] y js , q js =
Q s [ V j ] ⊤ x js , and Q s [ · ] is the Bellman target operator defined in the introduction part.
The min-player and the max-player compute the average policies
[Equation]
We use the classical averaging stepsizes { α jt } from Jin et al.
(2018): α t = H +1 H + t , α jt = α j (cid:81) tk = j +
(1 − α k ) (1 ≤ j ≤ t − 1) , α tt = α t , with H = 1+ γ 1 − γ .
In Averaging OGDA, x t , y t are the policies played at iteration t ∈
[ T 1 : T 2 ] , and (cid:101) x t , V t , (cid:101)
y t , V t are local auxiliary variables help to generate such sequences of x t , y t .
The global convergence and geometric boundedness of
Averaging OGDA are shown in Section 4.
The RHS of (2) in the definition of Global-Slow can be directly extended to different convergence rates and more algorithms such as Wei et al.
(2021) with a different initialization can serve as the generalized Global-Slow .
More details are in
Appendix H.
Example of Local-Fast base algorithm (OGDA).
By running OGDA ([ T 1 : T 2 ] , (cid:98) z , η ) with initial policy (cid:98) z
= ( (cid:98) x , (cid:98) y ) , the min-player initializes (cid:101) x
T 1 = x T 1 = (cid:98) x , the max-player initializes (cid:101)
y T 1 = y T 1 = (cid:98) y , and they update for t ∈ [ T 1 + 1 : T 2 ] as follows:
[Equation]
where we abbreviate Q ts = Q x t , y t s for t ∈ [ T 1 : T 2 ] .
In OGDA, x t , y t are the policies played at iteration t ∈
[ T 1 : T 2 ] , while (cid:101) x t , (cid:101) y t are local auxiliary variables.
OGDA can be considered as a natural extension of the classical optimistic gradient descent/ascent to Markov games because when there is only one state ( S = 1 ), it reduces to the classical OGDA method for matrix games.
The proof for local linear convergence of OGDA is of independent interest and shown in Section 5.
G LOBAL LINEAR CONVERGENCE
We can instantiate the meta algorithm Homotopy-PO by using OGDA (8) as Local-Fast and
Averaging OGDA (6) as Global-Slow .
This gives the provable global linear convergence for zero-sum discounted Markov games as in Theorem 2.
In practice, Homotopy-PO can also exhibit linear convergence.
Our numerical experiments are in Appendix I due to the space limitation.
Theorem 2 (Global Linear Convergence) Let { z t = ( x t , y t ) } t ∈ [0: T ] be the policy pairs played when running Homotopy-PO (Algorithm 1), where Local-Fast uses OGDA with η ≤ (1 − γ ) 52 32 √ S ( A + B ) , and Global-Slow uses Averaging OGDA with η ′ ≤ 1 − γ 16max { A,B } .
Then, there exist problem-dependent constants c ∈ (0 , 1) and M > 0 such that for any t ∈ [0 : T ] ,
[Equation]
[Equation]
where c = Ω( c 2+ / poly( S, A, B, 1 / (1 − γ ))) and M = poly( S, A, B, 1 / (1 − γ ) , 1 /c + ) .
Decentralized implementation.
Since both OGDA and Averaging OGDA are symmetric, rational and decentralized, our instantiation of Homotopy-PO is naturally a symmetric, rational and decentralized algorithm.
Pseudocodes and more details can be found in Appendix G.
Linear rate comparison with matrix games.
For the convex-concave matrix games, Gilpin et al.
(2012) and Wei et al.
propose centralized/decentralized methods with global linear rates of (1 − O ( φ ( G )))
t and (1 − O ( φ ( G ) 2 )) t respectively, where φ ( G ) is a certain condition measure of matrix G .
Details of φ ( G ) are in Lemma 22.
The constant c + in (1) can be naturally defined as c + = min s ∈S φ ( Q ∗ s ) (see Corollary 2).
Thus, the global linear convergence rate for zero-sum Markov games in Theorem 2 is comparable to solving matrix games up to polynomials in
S, A, B, 1 / (1 − γ ) .
G LOBAL C ONVERGENCE AND G EOMETRIC B OUNDEDNESS OF A VERAGING OGDA
We show that the Averaging OGDA (6) method has O (log T/T ) global convergence rate and geometric boundedness.
Thus, Averaging OGDA can be serve as Global-Slow in Homotopy-PO .
Global convergence.
The proof for global convergence of Averaging OGDA adapts several standard techniques from Markov games (Zhang et al., 2022; Wei et al., 2021).
We attach its proof in
Appendix D.1 for completeness.
Theorem 3 (Global Convergence) Let (cid:98) z
[ T 1 : T 2 ] =
( (cid:98) x
[ T 1 : T 2 ] , (cid:98) y
[ T 1 : T 2 ] ) be the average policy (7) generated by running Averaging-OGDA ([ T 1 : T 2 ] , ˜ z , η ′ ) with η ′ ≤
[Equation]
{ A
[Equation]
B } .
There is a problemdependent constant C ′
=
O ( √ S ( A + B ) c
+ (1 − γ ) 6 ) such that (cid:98) z
[ T 1 : T 2 ] satisfies
[Equation]
This gives the (cid:101) O
(1 /T )
global convergence rate of Global-Slow .
This property guarantees that
Global-Slow can serve as a “guide” in Hidden Phase I as described in Section 3.1.
Geometric boundedness.
The proof of geometric boundedness mainly relies on the stability of projected gradient descent with respect to the NE set (Appendix B).
We will prove that the increase of dist( z t , Z ∗ ) is at most geometric by providing mutual bounds among { dist ( z t , Z ∗ ) } , { dist( (cid:101) z t , Z ∗ ) } , {∥ V t − V t ∥ ∞ } , { max b q ts ( b ) − min a q ts ( a ) } inductively.
The formal proof is in
Appendix D.2.
Theorem 4 (Geometric Boundedness) Let { z t } t ∈
[ T 1 : T 2 ] , (cid:98) z [ T 1 : T 2 ] be the policy pairs played and the average policy pair generated by running Averaging-OGDA ([ T 1 : T 2 ] , ˜ z , η ′ ) with η ′ ≤ 1 , then there is a problem-dependent constant D
=
O ( S ( A + B ) 2 ( −
[Equation]
) 4 ) (possibly D 0 > 1 ) such that for any t ∈ [ T 1 : T 2 ] ,
[Equation] [Equation]
This property is important in our proof for the main theorem (Theorem 2).
It means that when running Global-Slow in Hidden Phase II, though dist( z t , Z ∗ ) can possibly increase due to D 0 > 1 , dist( z t , Z ∗ ) can only increase geometrically (see segments BC , DE in Figure 1).
L OCAL L INEAR C ONVERGENCE OF OGDA
We show that OGDA (8) has local linear convergence.
Thus, OGDA can be used as the base algorithm Local-Fast in Homotopy-PO .
To prove the local linear convergence, we provide a novel analysis for OGDA which starts from the following two observations.
Observation I (Lemma 7) saddle-point metric subregularity (SP-MS) can be generalized to Markov games, i.e., for any policy pair z ∈ Z and s ∈ S ,
[Equation]
Observation I guarantees the progress of projected gradient descent/ascent (PGDA) is substantial.
Observation II (Appendix B, Lemma 10) when running OGDA (8), the change in policy pair becomes smaller when z t , (cid:101) z t are approaching the NE set, i.e.,
[Equation]
Observation II implies the stability of state visitation distribution.
Intuitively, it can help us relate
Markov games to matrix games in a neighborhood of the NE set.
Theorem 5 (Local Linear Convergence) Let { z t } t ∈
[ T 1 : T 2 ] be the policy pairs played when running
OGDA ([ T 1 : T 2 ] , (cid:98) z , η ) with stepsize η ≤
[Equation]
52 32 √ S
[Equation]
A +
[Equation]
Then, there are problem-dependent constants c ∈ (0 , 1) , δ 0 > 0 such that if dist 2 ( (cid:98) z , Z ∗ ) ≤ δ 0 η 4 , then for any t ≥ T 1 ,
[Equation]
[Equation]
where c 0 = Ω( c 2+ / poly( S, A, B, 1 / (1 − γ ))) and δ 0 = Ω( c 4+ / poly( S, A, B, 1 / (1 − γ ))) .
We provide a proof sketch below.
The formal proof is in Appendix C.
Proof sketch of Theorem 5.
We denote the projections by (cid:101) x t ∗ =
P X ∗ ( (cid:101) x t ) , (cid:101) y t ∗ =
P Y ∗ ( (cid:101) y t ) , (cid:101)
z t ∗ =
P Z ∗ ( (cid:101) z t ) .
Let ρρρ 0 be the uniform distribution on S .
Our proof for the local linear convergence of OGDA has the following steps.
Step I: One-step analysis (Appendix C.1).
The main obstacle in adopting standard analysis for normal form games to Markov games lies in the fact that as Markov games are nonconvex-nonconcave, it may happen that (cid:68) x
t +1 s
[Equation]
To overcome this problem, our strategy is to consider a weighted sum of (cid:68) x
t +1 s
[Equation]
and (cid:68)(cid:101)
y t ∗ s − y t +1 s , (cid:0) Q t +1 s (cid:1)
⊤ x t +1 s (cid:69) .
As ( (cid:101) x t ∗ , (cid:101) y t ∗ ) attains a Nash equilibrium, V x t +1 , (cid:101)
y t ∗ ( ρρρ 0 ) −
V (cid:101) x t ∗ , y t +1 ( ρρρ 0 ) ≥ 0 .
Then, by applying performance difference lemma (Lemma 20), we have (cid:88)
[Equation]
where d tx ( s ) =
d (cid:101) x
( t − 1) ∗ , y t ρρρ( s ) and d ty ( s ) =
d x t , (cid:101) y
( t − 1) ∗ ρρρ( s ) for t > T 1 .
Since Q ts is smooth in z t , we can adopt standard regret analysis for optimistic gradient descent in normal form games to bound (cid:68) x t +1 s − (cid:101) x t ∗ s ,
Q t +1 s
y t
+1 s (cid:69) and (cid:68)(cid:101)
y t ∗ s − y t +1 s , (cid:0) Q t +1 s (cid:1)
⊤ x t +1 s (cid:69) .
This yields the following inequality (which is equivalent to Lemma 4)
[Equation]
17) where C Λ =
[Equation]
Step II: Progress of projected gradient descent (Appendix C.2).
We combine (13) from Observation I and standard analysis of projected gradient descent (Lemma 5) to essentially show that there exists a problem-dependent constant c ′ +
= O ( c 2+ η 2 / poly( S, A, B, 1 / (1 − γ ))) such that
[Equation]
Step III: Stability of visitation distribution near the NE set (Appendix C.3).
Using (14) from
Observation II and the non-expansive property of projections onto convex sets, we will show ∥ z t +1 − z t ∥ 2 + ∥ (cid:101)
z t ∗ − (cid:101) z
( t − 1) ∗ ∥ 2 ≤ O (cid:0) Λ t − 1 (cid:1) .
Then, as d tx ( s ) , d ty ( s ) in (16) are continuous in z t and (cid:101)
z t , we can find a problem-dependent constant δ
=
O
( c 4+ η 4 / poly( S, A, B, 1 / (1 − γ )))
such that if Λ t − 1 ≤ δ , then (cid:101) Θ t can be bounded by
[Equation]
Step IV: Induction (Appendix C.4).
By (17), (18), (19) from Steps I, II, III above, intuitively, we can deduce that when Λ t − 1 ≤ δ , the “one-step linear convergence” is achieved
[Equation]
By a coupled induction with Step III, given the initial policy (cid:98) z in the neighborhood B ( Z ∗ , √ δ ) of the NE set, the policy pair z t will always stay in B ( Z ∗ , √ δ ) .
Then, Λ t converges linearly.
This yields the local linear convergence of OGDA as in Theorem 5.
A CKNOWLEDGEMENT
We would like to thank anonymous reviewers for their helpful comments.
A F URTHER RELATED WORK
Sampling-based two-player zero-sum Markov games.
Finding Nash equilibria of zero-sum
Markov games in sampling-based/online setting is receiving extensive studies in recent years (Zhang et al., 2020; Liu et al., 2021;
Bai et al., 2020; Bai & Jin, 2020; Brafman & Tennenholtz, 2002; Sidford et al., 2020; Tian et al., 2021; Wei et al., 2017; Xie et al., 2020; Chen et al., 2022; Li et al., 2022).
In this paper, we are more concerned with known model or perfect recall settings.
Specifically, our focus is on how to design efficient policy optimization methods to solve the minimax optimization problem formulated by zero-sum Markov games.
Therefore, these works are not directly relevant to us.
Minimax optimization.
Zero-sum Markov games are usually studied as minimax optimization problems.
Finding Nash equilibria/saddle points in convex-concave and nonconvex-concave problems have been extensively studied (Lin et al., 2020b; Tseng, 1995; Mokhtari et al., 2020a;b;
Thekumparampil et al., 2019; Lu et al., 2020; Nouiehed et al., 2019; Kong & Monteiro, 2021; Lin et al., 2020a).
Due to the nonconcexity-nonconcavity of zero-sum Markov games, existing tools in convex-concave and nonconvex-concave optimization are hard to be adapted here.
For nonconvex-nonconcave optimization, Nouiehed et al.
; Yang et al.
(2020) study two-timescale/asymmetric gradient descent/ascent methods under the PŁ condition, where two-time-scale/asymmetric refers to that oneplayer chooses a much smaller step than its opponent, or one-player waits until its opponent finds the best response.
Daskalakis et al.
(2020) establish the two-sided gradient dominance condition for zero-sum Markov games, which can be related to the two-sided PŁ condition.
And they utilize this gradient dominance property to study the finite-time performance of two-timescale gradient descent/ascent (GDA) algorithm in zero-sum Markov games and prove its sub-linear convergence rate.
Zhao et al.
(2022) consider function approximation and propose another two-timescale method that finds a NE at (cid:101)
O
(1 /t ) rate.
Matrix games.
Matrix games are a special kind of Markov games with single state.
Since matrix games are naturally convex-concave, global linear convergence has been achieved in finding Nash equilibria in matrix games (Gilpin et al., 2012; Wei et al., 2020).
The linear convergence of their algorithms relies on the following fact: the duality gap of one policy pair can be lower bounded by its distance to the NE set multiplied by a matrix condition measure (see Lemma 22 for more details).
This property is called saddle-point metric subregularity (SP-MS) in Wei et al.
Similar techniques have been extended to extensive form games and get linear convergence (Lee et al., 2021;
Piliouras et al., 2022).
Averaging techniques.
Averaging techniques are usually used to tame nonstationarity in approximate Q functions, where the players utilize information from past iterations to obtain better approximations for value functions and policy gradients.
Wei et al.
(2021) propose an actor-critic
OGDA method which uses averaged value functions to get more accurate policy gradients, and then achieve a O (1 / √ t ) last-iterate convergence rate to the NE set.
Zhang et al.
(2022) propose a modified OFTRL method, where the min-player and the max-players employ a lower and upper bound for value functions separately.
The lower and upper bounds are computed from approximate Qfunctions in past iterations.
Their method has (cid:101) O
(1 /t ) convergence rate for the average policy.
Yang & Ma (2022) show that the average policy of an OFTRL method whose approximate Q-functions are also averaged from past estimates can find Nash equilibria at the rate of O (1 /t ) with no logarithmic factors.
Regularized Markov games.
Adding regularizer can greatly refine the structures of matrix games and Markov games and is considered a powerful tool to tackle nonconvexity-nonconcavity of zero-sum Markov games.
Cen et al.
(2021) study entropy-regularized matrix games and achieve dimension-free last-iterate linear convergence to the quantal response equilibrium which is an approximation for the Nash equilibrium.
They further connect value iteration with matrix games and use the contraction property of the Bellman operator to prove the linear convergence to the quantal response equilibrium of the Markov games.
By choosing small regularization weights, their method can find an ϵ -Nash equilibrium in (cid:101) O
(1 /ϵ ) iterations.
Zeng et al.
(2022) also consider adding entropy regularization to help find Nash equilibria in zero-sum Markov games.
They prove the O ( t − 1 / 3 ) convergence rate of a variant of GDA by driving regularization weights dynamically to zero.
How- ever, to obtain Nash equilibria, the regularization weights have to be reduced to zero in the learning process.
The time complexities of existing regularized methods are usually inversely proportional to the regularization weights.
Reducing such weights to zero could possibly lead to sub-linear rates.
B S TABILITY OF PROJECTED GRADIENT DESCENT / ASCENT WITH RESPECT TO THE N ASH EQUILIBRIUM SET
In this section, we show the stability of the distance to the Nash equilibrium set after one step of projected gradient descent/ascent.
The results in this section are important in our proofs for the local linear convergence of OGDA and the geometric boundedness of Averaging OGDA.
The following lemma shows that projected gradient descent/ascent is very “stable” on the NE set.
More specifically, if the players have attained a Nash equilibirum, then, their policies will remain invariant by doing projected gradient descent/ascent.
Lemma 1 For any Nash equilibrium z
= ( x , y ) ∈
Z ∗ , let x + , y + be the variables after one step of projected policy gradient descent/ascent, i.e., for s ∈ S
[Equation]
Let z + = ( x + , y + ) , then, z + = z .
Proof of Lemma 1.
Let u ∗ s = Q ∗ s y s .
By Lemma 19, x s ∈ arg min x ′ s ∈ ∆
A ⟨ x ′ s , Q ∗ s y s ⟩ .
Equivalently, supp ( x s ) ⊆
arg min a u ∗ s ( a ) , where supp ( x s ) is the index set of the nonzero entries in x s .
Next, we will show x + s = x s .
Since x + s is the projection onto ∆ A and Slater’s condition is naturally satisfied in the simplex constraint, by the KKT conditions,
[Equation]
A ] , λ
[Equation]
x
[Equation]
a ∈ [ A ] x + s ( a ) = 1 .
Then, for a ∈ [ A ] , λ a > 0 only if x + s ( a ) = 0 ; otherwise, x + s ( a ) =
x s ( a ) − η u ∗ s ( a ) + λ 0 .
Thus, x + s ( a ) = max { x s ( a ) − η u ∗ s ( a ) + λ 0 , 0 } .
If λ 0 = min a ∈ [ A ] u ∗ s ( a ) , then by combining with supp ( x s ) ⊆
arg min a u ∗ s ( a ) , we have
[Equation]
, i.e., (cid:80) a max { x s ( a ) − η u ∗ s ( a ) + λ 0 , 0 } = 1 .
Thus, for λ> min a ∈
[ A ] u ∗ s ( a ) or λ < min a ∈
[ A ] u ∗ s ( a ) , we will have (cid:80) a max { x s ( a ) − η u ∗ s ( a ) + λ 0 , 0 } > 1 or (cid:80) a max { x s ( a ) − η u ∗ s ( a ) + λ 0 , 0 } < 1 , respectively.
To meet the condition (cid:80) a ∈ [ A ] x + s ( a ) = 1 , we have to let λ 0 = min a ∈ [ A ] u ∗ s ( a ) .
Now, x
[Equation]
s
[Equation]
a ) , ∀ a ∈ A .
Analogously, y + s = y s .
□
The following lemma is a perturbed version of Lemma 1.
Lemma
For any z = ( x , y ) ∈
Z , (cid:101) z
= ( (cid:101) x , (cid:101) y ) ∈
Z and matrices (cid:110) Q s , (cid:98) Q s (cid:111) s ∈S ⊆
R A × B , let x + , y + be the position after one step of projected policy gradient descent/ascent, i.e., for s ∈ S
[Equation]
Let z + = ( x + , y + ) , then,
[Equation]
Proof of Lemma 2.
Denote x ∗ =
P X ∗ ( x ) , y ∗ =
P Y ∗ ( y ) , z ∗ =
( x ∗ , y ∗ ) ; (cid:101) x ∗ =
P X ∗ ( (cid:101) x ) , (cid:101) y ∗
=
P Y ∗ ( (cid:101) y ) , (cid:101) z ∗ = (cid:0)(cid:101) x ∗ , (cid:101) y ∗ (cid:1) .
Let u s = Q s y s , u ∗ s = Q ∗ s y ∗ s , then ∥
[Equation]
∥ ∞
[Equation]
, Y
[Equation]
) , i
[Equation]
e.,
[Equation] [Equation]
By Lemma 19, ( (cid:101) x ∗ , y ∗ ) is also a Nash equilibrium.
Denote (cid:101) x ∗ + s = P ∆ A (cid:0)(cid:101) x ∗ s − η Q ∗ s y ∗ s (cid:1) .
Then, by Lemma 1, (cid:101) x ∗
+ s = (cid:101) x ∗ s .
By triangle inequality, we have
[Equation]
where the first equality is by (21) and the second inequality comes from the fact
that for any aaa , b ∈ R A , ∥P ∆ A ( aaa ) − P ∆
A ( b ) ∥ ≤ ∥ aaa − b ∥ .
Taking square and summing over s ∈ S and combining with (20) yield that
[Equation]
Analogously,
[Equation]
Then, the result follows by summing up the bound for ∥ x + − (cid:101) x
∥ 2 and ∥ y + − (cid:101) y
∥ 2 .
□
C P ROOF FOR LOCAL LINEAR CONVERGENCE OF OGDA
In this section, we prove the local linear convergence of OGDA (Theorem 5).
For notational simplicity, we assume T 1 = 0 in the analysis below.
Recall the OGDA algorithm ( T 1 = 0 ): the min-player and max-player initialize
[Equation]
and the min-player updates for t ≥ 1 as follows
[Equation]
while the max-player updates for t ≥ 1 as follows
[Equation]
Here, we denote
[Equation]
The policy x t and y t are played by the min-player and the max-player at iteration t .
And (cid:101) x t , (cid:101)
y t are local auxiliary variables to help generate the policies x t and y t .
Since we initialize x 0 = (cid:98) x , y
= (cid:98) y , we drop the notation of (cid:98) x , (cid:98) y below and directly use x 0 , y 0 to denote the initial policies.
The OGDA method is a decentralized algorithm, its decentralized implementation is in Algorithm(for the min-player) and Algorithm 3 (for the max-player) of Appendix G.
To prove the local linear convergence of OGDA, we first introduce some notations and auxiliary variables.
Additional notations and auxiliary variables.
We use 1 , 0 to denote the all-ones and all-zeros vectors or matrices, whose dimensions are determined from the context.
1 i is the i -th standard basis of the Euclidean space, i.e., the i -th entry of 1 i equals one, and the others entries equal zero.
The operators >, ≥ , <, ≤ are overloaded for vectors and matrices in entry-wise sense.
We denote the policy pairs z t = ( x t , y t ) , (cid:101) z t = (cid:16)(cid:101) x t , (cid:101) y t (cid:17) and denote the projections onto the Nash equilibrium sets as (cid:101)
x t ∗ s =
P X ∗ s ( x ts ) , (cid:101)
y t ∗ s =
P Y ∗ s ( y ts ) , (cid:101)
z t ∗ s =
P Z ∗ s ( z ts ) .
Since (cid:101) x t , (cid:101) y t , (cid:101) z t are treated as concatenated vectors, we have from the elementary property of the ℓ 2 -norm that (cid:101)
z t ∗ s =
( (cid:101) x t ∗ s , (cid:101)
y t ∗
[Equation]
) , (cid:101)
[Equation]
=
[Equation]
=
[Equation]
s
[Equation]
= (cid:110)(cid:101) z t
[Equation]
s ∈S , and
[Equation]
Let ρρρ 0 be the uniform distribution on S .
Then, we denote the state visitation distribution under the policy pairs ( (cid:101) x ( t − 1) ∗ , y t ) and ( x t , (cid:101) y
( t − 1) ∗ ) as
[Equation]
It follows by definition that for any s ∈ S ,
[Equation]
Define weighted sums of distances
[Equation]
[Equation] [Equation]
We will show the linear convergence of Λ t given dist 2 (cid:0) z 0 , Z ∗ (cid:1) ≤ δ 0 η 4 for some problemdependent constant δ 0 >
[Equation]
C. O NE - STEP ANALYSIS
Our proof for local linear convergence starts from the following elementary lemma, which is derived by combining a standard analysis of optimistic gradient descent/ascent with the smoothness of Q x , y s with respect to the policy pair ( x , y )
[Equation]
Lemma 3 Let (cid:110) x
t , (cid:101) x t , y t , (cid:101) y t (cid:111) be generated from OGDA (23) , (24) .
Then, for any t ≥
[Equation]
we have
[Equation]
x t +
[Equation]
− (cid:101) x ts (cid:13)(cid:13)(cid:13) +
[Equation]
[Equation]
(cid:13)(cid:13)
z
[Equation]
+1 −
[Equation]
(cid:13)(cid:13)(28) and η (cid:68)(cid:101)
[Equation]
≤
[Equation]
(cid:18)(cid:13)(cid:13)(cid:13)(cid:101) y
[Equation]
(cid:101) y t
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:101)
y t +1 s
[Equation]
(cid:101)
[Equation]
t
[Equation]
(cid:13)(cid:13)(cid:13)(cid:19)
[Equation]
4 (cid:13)(cid:13)(cid:13)(cid:101)
y t +1 s
− y t +1 s (cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13) y t +1 s − (cid:101)
y ts (cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)
[Equation]
Proof of Lemma 3.
We abbreviate (cid:101) x
t ∗ = x ∗ , (cid:101) x t ∗ s = x ∗ s in this proof.
By (23b), since (cid:101) x t +1 s is the projection onto ∆ A , we have
[Equation]
− (cid:101) x ts + η Q t +1 s y t
+1 s (cid:69) ≥ 0 , ∀ t ≥ 0 .
Equivalently, η (cid:68)(cid:101) x
t +1 s − x ∗ s ,
Q t +1 s
y t
+1 s (cid:69) ≤ 1 2 (cid:18)(cid:13)(cid:13)(cid:13)(cid:101) x
ts − x ∗ s (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13)(cid:101) x t +1 s − x ∗ s (cid:13)(cid:13)(cid:13) − (cid:13)(cid:13)(cid:13)(cid:101) x
t +1 s − (cid:101) x
ts (cid:13)(cid:13)(cid:13) 2 (cid:19) .
Similarly, from (23a), (cid:68) (cid:101) x t +1 s − x t +1 s , x t +1 s − (cid:101) x t s + η Q ts y t s (cid:69) ≥ 0 , ∀ t ≥i.e., η (cid:68) x t +1 s − (cid:101) x t +1 s , Q t s y t s (cid:69) ≤ 1 2 (cid:18)(cid:13)(cid:13)(cid:13)(cid:101) x t +1 s − (cid:101) x t s (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13)(cid:101) x t +1 s − x t +1 s (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) x t +1 s − (cid:101) x t s (cid:13)(cid:13)(cid:13)(cid:19) .
Then, we have
[Equation]
By Lemma
[Equation]
we have
[Equation]
Then, (28) follows by combining (30) with (31).
And (29) follows by similar arguments.
□
We consider weighted sum of (28) and (29) using the state visitation distribution d tx ( s ) , d ty ( s ) defined in (25) as the weighting coefficients.
Lemma 4 (One-Step Analysis) Let (cid:110) x
t , (cid:101) x t , y t , (cid:101) y t (cid:111) be generated from OGDA with η ≤ (1 − γ )32 √ S ( A + B ) .
Then, for any t ≥ 0 ,
[Equation]
Proof of Lemma 4.
Recall that ρρρ 0 be the uniform distribution on S .
By Lemma 20,
[Equation]
As (cid:101) x t
[Equation]
∈ X ∗ , (cid:101)
y t ∗ ∈
Y ∗ , by Lemma 19, (cid:16)(cid:101) x t ∗ , (cid:101)
y t ∗ (cid:17) also attains Nash equilibrium.
Thus, we have
[Equation]
Substituting (28), (29) into (33) yields that
[Equation]
By combining with the facts that
[Equation]
x
[Equation]
[Table]
[Equation]
s ∈S d t +1 x ( s )
= 1 and their counterparts for the max-player, we have
[Equation]
≤
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(35) where the last inequality is by our condition on η .
By combining (34) with (35) and rearranging
[Equation]
we have
[Equation]
□
C. P ROGRESS OF PROJECTED GRADIENT DESCENT
The following lemma is a standard step in the analysis of projected gradient descent.
Lemma
If η ≤ 1 − γ max { √ A, √ B } , for any t ≥ 0 , let ρρρ 0 be the uniform distribution on S , then
[Equation]
Proof of Lemma 5.
Since (cid:101) x t +1 s is a projection onto ∆ A , for any x ′ s ∈ ∆ A ,
[Equation]
x t
[Equation]
i.e.,
[Equation]
Then, by combining with the condition on η ,
[Equation]
For any s 0 ∈ S and x ′ ∈ X , by Lemma 20 and the fact that (cid:80) s ∈S d x ′ , y
[Equation]
[Table]
[Equation]
Similarly,
[Equation]
By (93) and (94), we have
[Equation] [Equation]
Then, by combining (36)
[Equation]
(37), (38) and the condition on η , we have
[Equation]
[Table]
≤
[Table]
The result follows by taking sum over s 0 ∈ S .
□
Next, we extend Lemma 4 of Gilpin et al.
and Theorem 5 of Wei et al.
(2020) from matrix games to Markov games.
Firstly, we prove the following auxiliary lemma, which is used in the proof of Lemma 7.
This lemma is straightforward from the contraction and monotonicity of the Bellman operator, we attach its proof for completeness.
Lemma 6 For policies x ∈ X and y ∈ Y , if there is a vector v ∈ R S such that for any s ∈ S ⟨ x s , Q s [ v ] y s ⟩ ≥ v ( s ) , then, we have that for any s ∈ S ,
V
[Equation]
Proof of Lemma 6.
For any vector u ∈ R S , define the mapping Φ : R S → R S with
[Equation]
Then, for any u 1 , u 2 ∈ R S , by definition,
[Equation]
Thus, we have
[Equation]
i.e., Φ is a contraction mapping.
Define v 1 =
Φ[ v ] and v k +1 =
Φ[ v k ] , ...
Then, by (39), we have
[Equation]
Then, the limit of v k exists and we denote the limit v ∗ =
lim k →∞ v k .
Obviously, v ∗ is a fixed point of Φ because
[Equation]
As V x , y ( s ) = ⟨ x s , Q s [ V x , y ]
y s ⟩ , we have Φ[ V x , y ]
=
V x , y , i.
[Equation]
., V x , y is a fixed point of Φ .
By the contraction property of Φ as in (39), its fixed point is unique.
Thus,
[Equation]
By definition, for any u 1 , u 2 ∈ R S , if u 1 ≥ u 2 in entry-wise sense, then Φ[ u 1 ] ≥ Φ[ u 2 ] in entrywise sense.
Since the condition ⟨ x s , Q s [ v ] y s ⟩ ≥ v ( s ) for any s ∈ S is equivalent to v 1 ≥ v in entry-wise sense.
By induction, we have v k ( s ) is non-decreasing in k .
Combining with the fact that v ∗ =
lim k →∞ v k , we have that for any s ∈ S ,
[Equation]
□
The following lemma is an extension of Lemma 4 of Gilpin et al.
and Theorem 5 of Wei et al.
(2020) for matrix games to Markov games, it plays an important role in lower bounding the progress of gradient descent/ascent.
Lemma 7 There exists a problem-dependent constant c + > 0 such that for any z =
( x , y ) ∈
Z and s ∈ S ,
[Equation]
Proof of Lemma 7.
Recall that v ∗ ( s ) is the minimax game value at state s and Q ∗ s =
Q s [ v ∗ ] .
For any s ∈ S , choose
[Equation]
Then, by Shapley’s theorem (Lemma 19), X ∗ s × Y ∗ s is the NE set for the matrix game min x ′ max y ′
x ′⊤ Q ∗ s y ′ .
[Equation]
Then, by Lemma 22, for any s ∈ S , there exists a constant φ ( Q ∗ s ) >
depending only on the matrix Q ∗ s such that
[Equation]
Define the policies (cid:98) x = { (cid:98) x s } s ∈S and (cid:98)
y
= { (cid:98) y s } s ∈S .
Combining (40) with Lemma 6 yields that for any s
[Equation]
Then, by definition, in entry-wise sense,
[Equation]
By combining the above equations, we have for any s ∈ S ,
[Equation]
where the second last inequality is by (40), the last inequality is by (41).
Then, the proof is completed by (1).
□
By combining Lemma 5 and Lemma 7, we provide lower bound for the progress of projected gradient descent (PGD).
Lemma 8 (Progress of PGD) Let (cid:110)
z t , (cid:101)
z t (cid:111) t ≥be generated from OGDA with η ≤ 1 − γ , then for any t ≥ 0 , we have
[Equation]
Proof of Lemma 8.
By Lemma 5 and Lemma 7, we have
[Equation]
where the last inequality above comes from the fact that d tx ( s ) ≤ 1 , d ty ( s ) ≤ 1 for any s ∈ S .
□
C. S TABILITY OF STATE VISITATION DISTRIBUTION NEAR THE N ASH EQUILIBRIUM SET
The main motivation behind the proofs in this section is Lemma 1, which shows that projected gradient descent is very “stable” on the NE set.
The following lemma is a perturbed version of Lemma 1.
It is extensively used in the proof of
Lemma 10.
Its proof follows by Lemma 2 and Lemma 21 with a simplification of coefficients.
Lemma
For any z = ( x , y ) ∈
Z and (cid:101)
z =
( (cid:101) x , (cid:101) y
) ∈
Z , let x + , y
+ be the policy after one step of projected policy gradient descent/ascent, i.e., for s ∈ S
[Equation]
Let z + = ( x + , y + ) , then,
[Equation]
Proof of Lemma 9.
Denote x ∗ =
P X ∗ ( x ) , y ∗ =
P Y ∗ ( y ) and z ∗ =
( x ∗ , y ∗ ) .
By Lemma 19, ( x ∗ , y ∗ ) attains Nash equilibrium and Q x ∗ , y ∗ s = Q ∗ s .
By (91), we have
[Equation]
Then, by combining with Lemma 2, we have
[Equation]
□
The following lemma uses Lemma 9 to show that when Λ t is close to 0 , (cid:13)(cid:13)
z t +1 − z t (cid:13)(cid:13) , (cid:13)(cid:13)(cid:13)(cid:101) z t +1 − (cid:101)
z t (cid:13)(cid:13)(cid:13) will be small, which implies the difference between (cid:101) Θ t and Θ t will also be small.
Lemma 10 Consider the sequence (cid:110)
z t , (cid:101)
z t (cid:111) generated from OGDA with stepsize η ≤
[Equation]
1 −
[Equation]
2 2 √ 2 S ( A + B ) .
There is a problem-dependent constant δ 1 = O (cid:16) (1 − γ ) 5 S 3 ( A + B ) (cid:17) > 0 such that for any τ > 0 and t ≥ 1 , if Λ t − 1 ≤ τ 2 δ 1 , we have
[Equation]
Proof of Lemma 10.
By the condition on η , we have 8 S ( A + B ) 2 η 2 ( − γ )≤ 1 .
Denote c ′ =
S 1 − γ , c ′′ = 8 S 1 − γ and define the problem-dependent constant
[Equation]
We also denote δ =
τ
δ 1 below.
The positive constants c 1 , c 2 , · · · , c 7 below are all polynomials in S, A, B, 1 / (1 − γ ) , the definition for each of them follows from the line it first occurs.
Since d tx ( s ) , d ty ( s ) ≥ 1 − γS = c ′ , the condition Λ t − 1 ≤ τ
[Equation]
=
[Equation]
implies that
[Equation]
Then,
[Equation]
By applying Lemma 9 with (cid:101) z := (cid:101)
z t − 1 , z :=
z t − 1 , we have
[Equation]
Thus, dist
[Equation]
By setting (cid:101) z := (cid:101)
z t − 1 , z :=
z t in Lemma 9, we have
[Equation]
δ def
[Equation]
δ.
Therefore, dist 2 (cid:16)(cid:101) z t , Z
[Equation]
≤ 2dist
[Equation]
Again, utilize Lemma 9 with (cid:101) z := (cid:101) z t , z :=
z t , we have
[Equation]
Thus,
[Equation]
Now we can bound
[Equation]
−
[Equation]
Since X ∗ s is a convex set, the projection onto it is non-expansive,
i
[Table]
e., (cid:13)(cid:13)(cid:13)(cid:101) x t
[Table]
s −
[Table]
(cid:13)(cid:13)(cid:13)
[Table]
(cid:13)(cid:13)(cid:13) .
Then,
[Table]
[Equation]
Analogously,
[Equation]
By Lemma 21 and (25), for any s ∈ S ,
[Equation]
Similarly, we also have for any s ∈ S ,
[Equation]
What remains is to bound the term √
( A + B ) c 7 δ− γ on the RHS of (50) and (51).
Using (43)-(49), we have
• (by (43))
c
1 = 10 c ′ + 2 c ′′ • (by (44))
c 2 = 22 c
′ + 4 c ′′ • (by (45))
c 3 = 30 c ′ + 4 c ′′ • (by (46))
c 4 = 62 c ′ + 8 c ′′ • (by (47))
c 5 = 518 c ′ + 68 c ′′ • (by (48))
c 6 = 1674 c ′ + 222 c ′′ • (by (
)) c 7 = 1704 c ′ + 226 c ′′
By the definition of δ 1 in (42) and our notation δ = τ 2 δ 1 , we have
[Equation]
Then, by combining with (50), we have (cid:12)(cid:12)
d tx ( s ) − d t +1 x ( s ) (cid:12)(cid:12) ≤ τ (1 − γ ) S .
By combining with the fact that d tx ( s ) ≥ 1 − γS , we have
[Equation]
τ
[Equation]
s
[Equation]
Analogously, for any s ∈ S , (cid:12)(cid:12)
d ty ( s ) − d t +1 y
( s
) (cid:12)(cid:12) ≤ τ d ty ( s ) .
Then, the result follows by the definition of Θ t and (cid:101) Θ t in (26).
□
C. P ROOF OF T HEOREM
By Lemma 8, we have
[Equation]
where
[Equation]
Then, combining (52) with Lemma 4 and the definitions of Λ t , Θ t , (cid:101) Θ t in (27), (26) yield that for any t ≥ 1 ,
[Equation]
Define
[Equation]
Recall the problem-dependent constant δ 1 > 0 defined in Lemma 10, we define δ
= c
20 δ 1 > 0 .
Now, we prove Λ t +1 ≤ (1 − c 0 )
t Λ t by induction.
For the case t = 0 , firstly, by the definitions of Λ 0 , (cid:101) Θ 0 , Θ 0 in (27), (26) and the fact that 1 − γS ≤ d tx ( s ) ≤ 1 , 1 − γS ≤ d ty ( s ) ≤ 1 , we have
[Equation]
Λ 0 .
Then, by combining with Lemma 4, (52) and the fact that (cid:101) z
= z 0 , we have
[Equation]
If we have shown Λ j +1 ≤ (1 − c 0 ) j Λ j for j = 0 , · · · , t − 1 , we next prove it for t .
By induction hypothesis,
[Equation]
By Lemma 10, (cid:101)
[Equation]
Then, by combining with (54) and the fact that c 0 ≤ c 8 / 2 , c 0 ≤ 1 / 2 from the definition of c 0 in (55), we have
[Equation]
By induction, we have for any t ≥ 0 ,
[Equation]
Using the fact that 1 − γS ≤ d tx ( s ) ≤ 1 , 1 − γS ≤ d ty ( s ) ≤ 1 and the definition of Λ t in (27), we have dist
[Equation]
z t , Z ∗
[Equation]
where (cid:98) z
= ( (cid:98) x , (cid:98) y ) is the initial policy pair (22).
This completes the proof for local linear convergence of OGDA.
As for the order of c 0 and δ 0 , by (53) and (55), c
= O (cid:18)
[Equation]
S(cid:19) .
By Lemma 10, δ 1 = O (cid:16) (1 − γ ) 5 S 3 ( A + B ) (cid:17) .
Then, by (57) and (56),
[Equation]
Since we need η ≤ O ( (1 − γ ) 52 √ S ( A + B ) ) in Theorem 5, by setting η = O ( (1 − γ ) 52 √ S ( A + B ) ) , we have the linear convergence rate
[Equation]
and to have linear convergence, dist( z T 1 , Z ∗ ) needs to satisfy
[Equation]
D P ROOFS FOR GLOBAL CONVERGENCE AND GEOMETRIC BOUNDEDNESS OF A VERAGING OGDA
In this section, we prove that the Averaging OGDA method introduced in (6) of Section 3.2 can serve as Global-Slow in the meta algorithm Homotopy-PO .
The proof of global convergence (Theorem 3) is in Appendix D.1.
The proof of geometric boundedness (Theorem 4) is in Appendix D.2.
To begin with, let us recall the Averaging OGDA method: The min-player initializes
[Equation]
while the max-player initializes
[Equation]
The min-player updates for t > T 1 as follows:
[Equation]
where
[Equation] [Equation]
and Q s [ · ] is the Bellman target operator defined in the introduction.
Meanwhile, the max-player updates for t > T 1 as follows:
[Equation]
where
[Equation]
At the end iteration T 2 , the min-player and the max-player compute the following average policies respectively
[Equation]
The min-player plays policy x t and the max-player plays policy y t at iteration t .
The variables (cid:101) x t ,
V t and (cid:101)
y t , V t are all local auxiliary variables to help generate the policies x t and y t .
Averaging OGDA is a decentralized algorithm whose decentralized implementation is in Algorithm 4 (for the min-player) and Algorithm 5 (for the max-player) of Appendix G.
D. G LOBAL CONVERGENCE RATE OF A VERAGING OGDA
Our task in this section is to prove the global convergence of Averaging OGDA (Theorem 3).
To this end, we need to bound dist 2 (cid:16)(cid:98)
z
[ T 1 : T 2 ] , Z ∗ (cid:17) by O (log( T 2 − T 1 ) /
( T 2 − T 1 )) .
Our roadmap can be depicted as follows:
[Equation]
The regrets above are defined as
[Equation]
More specifically, we bound the distance dist( (cid:98) z
[ T 1 : T 2 ] , Z ∗ ) in the following steps:
(Lemma 11) bounding dist( (cid:98) z
[ T 1 : T 2 ] , Z ∗ ) by O ( (cid:13) (cid:13)(cid:13) V T 2 +1 − V T
+
(cid:13) (cid:13)(cid:13) ∞ ) : dist (cid:16)(cid:98) z
[ T 1 : T 2 ] , Z ∗ (cid:17) ≤ O (cid:16)(cid:13)(cid:13)(cid:13) V T 2 +1 − V T 2 +1 (cid:13)(cid:13)(cid:13) ∞ (cid:17) 2.
(Lemma 12) bounding O ( (cid:13)(cid:13)(cid:13) V T 2 +1 − V T
+1 (cid:13)(cid:13)(cid:13) ∞ ) by regrets: (cid:13)(cid:13)(cid:13) V T 2 + − V T 2 +1 (cid:13)(cid:13)(cid:13) ∞ ≤ Reg T 1 : T 2 + O (
[Equation]
−
[Equation]
T 2 (cid:88) t =
T
Reg T 1 : t +
[Equation]
(Lemma 14) bounding the regrets:
[Equation]
The following fact about α jt can be found in Section 4 of Jin et al.
It will be used extensively in our proofs below.
Fact
The stepsize α jt satisfy:
(i) (cid:80) ∞
t =
j α jt = 1 + 1 H , ∀ t ≥ 1 .
(ii) (cid:80) tj =1 α jt = 1 , ∀ t ≥ 1 .
(iii) α jt ≤ α t and α jt +1 ≤ α jt , ∀ t ≥ 1 , 1 ≤ j ≤ t .
Firstly, we show that the local auxiliary variables V t ( s ) , V t ( s ) are lower and upper bounds for v ∗ ( s ) .
Then, to bound (cid:13)(cid:13)(cid:13) V t − v ∗ (cid:13)(cid:13)(cid:13) ∞ and (cid:13)(cid:13) V t − v ∗ (cid:13)(cid:13) ∞ , it suffices to bound (cid:13)(cid:13)(cid:13) V t − V t (cid:13)(cid:13)(cid:13) ∞ .
Fact 2 For any t ∈
[ T 1 : T 2 ] and s ∈ S ,
[Equation]
Proof of Fact 2.
By (61a), we have
[Equation]
By the definition of q ts in (62),
[Equation]
Recall that by Lemma 19, v ∗ ( s ) =
min x s max y s ⟨ x s , Q ∗ s y s ⟩ and Q ∗ s
= Q s [ v ∗ ] .
Suppose V j ( s ) ≤ v ∗ ( s ) for any s ∈ S and j ∈ [ T 1 : t ] , then we have
[Equation]
which leads to V t +1 ( s ) ≤ v ∗ ( s ) for any s ∈ S .
Then, it follows by induction that V t ( s ) ≤ v ∗ ( s ) , min a ∈A q ts ( a ) ≤ v ∗ ( s ) for any t ∈
[ T 1 : T 2 ] and s ∈ S .
Analogously, V t ( s ) ≥ v ∗ ( s ) , max b ∈B q ts ( a ) ≥ v ∗ ( s ) for any t ∈
[ T 1 : T 2 ] and s ∈ S .
It also follows by induction directly that the value of V t ( s ) , V t ( s ) , min a ∈A q ts ( a ) , max b ∈B q ts ( a ) stays in [0 , 11 − γ ] .
□
The following lemma shows that to bound dist 2 (cid:16)(cid:98) z
[ T 1 : T 2 ] , Z ∗ (cid:17) , it suffices to bound (cid:13)(cid:13)(cid:13) V t − V t (cid:13)(cid:13)(cid:13) ∞ .
Lemma 11 There is a problem-dependent constant (cid:98) C = √ S c + > 0 such that the average policy (cid:98) z
[ T 1 : T 2 ] = (cid:16)(cid:98) x
[ T 1 : T 2 ] , (cid:98) y
[ T 1 : T 2 ] (cid:17) satisfies dist
[Equation] [Equation]
≤
[Equation] [Equation]
Proof of Lemma 11.
Recall that Q ∗ s =
Q s [ v ∗ ] .
By (62) and Fact 2,
[Equation]
Analogously,
[Equation]
Thus
[Equation] [Equation]
By (1)
[Equation]
The positiveness of c + is guaranteed by Lemma 22.
Let (cid:98) C = √ S c
+ , then,
[Equation]
□
The following lemma mainly uses Fact 1 (i) and induction to show that (cid:13)(cid:13)(cid:13) V T 2 +
− V T
[Equation]
+
[Equation]
(cid:13)(cid:13)(cid:13) ∞ can be bounded by weighted sum of the regrets.
Lemma 12 The value functions V T 2 +1 , V T 2 +1 satisfies
[Equation]
Proof of Lemma 12.
By Fact 2 and the definition of the operator Q s [ · ]
[Equation]
we have
[Equation]
The following relation follows by definitions of V t in (61a) and q js in (62),
[Equation]
Analogously,
[Equation]
Summing up the above two equations yields that
[Equation]
where the last inequality is by (64).
Thus,
[Equation] [Equation]
[Equation]
Taking sum on both sides of the above equation and combining with Fact 1 (i) yield that
[Equation]
≤
[Equation]
≤
[Equation]
≤
[Equation]
where the last inequality is from the fact that H = 1+ γ 1 − γ .
After rearranging, we have
[Equation]
Since α j − T 1 +1 T 2 − T 1 +1 ≤ α T 2 − T 1 +1 ≤
H +1 T 2 − T 1 +1 for any j ∈ [ T 1 : t ] , by setting t := T 2 in (65) and substituting (66), we have
[Equation]
The next lemma is used to derive Lemma 14.
Lemma
For any t ∈ [ T 1 : T 2 − 1] and s ∈ S ,
[Equation]
Proof of Lemma 13.
By (62) and Fact 2, we have
[Equation]
By Fact 2, (cid:13)(cid:13)(cid:13) q ts (cid:13)(cid:13)(cid:13) ∞ ≤ 11 − γ .
Then, by the definition of V t in (61a), for any s ∈ S ,
[Equation]
where the third inequality uses the facts that (cid:80) tj ′
=
1 α j ′ t =
and α jt +1 ≤ α jt , α jt ≤ α t for any 1 ≤ j ≤ t .
Thus,
[Equation]
By substituting (68) into (67), we have
[Equation]
The bound for (cid:13)(cid:13) q ts − q t +1 s
(cid:13)(cid:13) 2 follows analogously.
□
We bound the regrets in the following lemma.
Its proof is mainly from combining standard analysis in RVU property (see for instance Rakhlin & Sridharan (2013); Syrgkanis et al.
and with
Lemma 13.
Lemma
For any t ∈ [ T 1 : T 2 ] , if η ≤
[Equation]
2max { A,B } , we have
[Equation]
Proof of Lemma 14.
Choose an arbitrary point x ∗ s from ∆ A .
Since (cid:101) x t +1 s is the projection onto ∆ A , we have
[Equation]
Then, we have
[Equation]
Analogously,
[Equation]
Then, by combining the above two equations
[Equation]
we have
[Equation]
where
[Equation]
By taking sum on both sides of the above equation
[Equation]
we have
[Equation]
Analogously, for any y ∗ s ∈ ∆ B ,
[Equation]
where
[Equation]
Since H ≥ 1 , we have α t − T 1 +2 T 2 − T 1 +1 /α
t − T 1 +1 T 2 − T 1 +1 ≤ 2 .
Then, by combining with the condition on η and the fact that (cid:13)(cid:13) x
t +1 − x t (cid:13)(cid:13) 2 ≤ 2 (cid:13)(cid:13)(cid:13) x
t +1 − (cid:101) x t (cid:13)(cid:13)(cid:13) 2 + 2 (cid:13)(cid:13)(cid:13)(cid:101) x
[Equation]
− x t (cid:13)(cid:13)(cid:13) 2 , we have
[Equation]
Then, by combining the definitions of ∆ t +1 x and ∆ t +1 y with Lemma
[Equation]
we have
T 2 −
[Equation] [Equation]
where the first inequality also uses (71) and the max-player’s counterpart of (71), the second inequality is by the condition on η and Fact 1.
By combining (69), (70), (72),
[Equation]
The bound of Reg T 1 : t for t ∈
[ T 1 : T 2 ] follows by similar arguments.
□
Now, we can prove the global convergence of Averaging OGDA (Theorem 3) by combining
Lemma 11, Lemma 12 and Lemma 14.
Proof of Theorem 3.
By Lemma 12, Lemma 14, we have
[Equation] [Equation]
∞
[Equation]
Since (cid:80) T 2 t = T 1 α t
[Equation]
T 1 +1 ≤ (
[Equation]
[Equation] [Equation]
T
[Equation]
1 + , we have
[Equation]
By Lemma 11, we have
[Equation]
, where
[Equation]
□
Remark 1 The initialization V T 1 = V † , y T 1 and V T 1 = V x T 1 , † is only used to show the geometric boundedness in Theorem 4.
When Averaging OGDA is used independently rather than called in
Homotopy-PO (Algorithm 1), we can simply choose V T 1 ( s ) = 0 and V T 1 =− γ for any s ∈ S .
The global convergence rate in Theorem 3 still holds.
D. G EOMETRIC BOUNDEDNESS OF A VERAGING OGDA
In this section, we prove the geometric boundedness of Averaging OGDA (Theorem 4).
The geometric boundedness of averaging OGDA essentially relies on the stability of projected gradient descent/ascent characterized in Lemma 2.
Intuitively, when (cid:8) z j (cid:9)
j ∈
[ T 1 : t ] are close to the Nash equilibrium set, (cid:110)
V j ( s ) , V j ( s ) (cid:111)
j ∈
[ T 1 : t ] will be close to v ∗ ( s ) .
Thus, min a q ts ( a ) , max b q ts ( b ) will also be close to v ∗ ( s ) .
Then, by Lemma 2, z t +1 will not be far away from the Nash equilibrium set.
Our proofs in this section can be summarized as: providing mutual bounds among { dist ( z t , Z ∗ ) } , (cid:110)
[Equation]
induction.
The following fact shows that (cid:13)(cid:13) (cid:13) V T 1 − V T 1 (cid:13)(cid:13) (cid:13) ∞ can be bounded by dist (cid:0) z T 1 , Z ∗ (cid:1) .
Lemma
The approximate value functions V T 1 , V T 1 satisfy
[Equation]
Proof of Lemma 15.
By Fact 2, V T 1 ( s ) ≤ v ∗ ( s ) ≤ V T 1 ( s ) .
By Lemma 19, V † , y t ∗ ( s ) =
v ∗ ( s ) .
Since the min-player initializes V T 1 ( s )
=
V † , y T 1 ( s ) , by combining with Lemma 21, we have
[Equation]
Analogously,
[Equation]
The result follows by summing the above two equations and combining with the fact that dist
[Equation]
□
The following lemma follows directly by the definition of V t , V t in (61a), (63a) and the fact that (cid:80) tj =1 α jt = 1 .
Lemma
For any t ∈ [ T 1 : T 2 − 1] and s ∈ S
[Equation]
The following lemma bound the expansion of dist ( z t , Z ∗ ) .
Its proof mainly uses Lemma 2.
Lemma 17 For any t ∈
[ T 1 + 1 : T 2 − 1] , we have
[Equation]
, Z ∗ (cid:1) .
In addition, dist
[Equation]
Proof of Lemma 17.
By Fact 2, we have (cid:13)(cid:13)
V t − v ∗ (cid:13)(cid:13)∞ + (cid:13)(cid:13)(cid:13) V t − v ∗ (cid:13)(cid:13)(cid:13)∞ ≤ (cid:13)(cid:13)(cid:13) V t − V t (cid:13)(cid:13)(cid:13)∞ .
Then,
B max ( a,b )
[Equation]
(cid:12)(cid:12) Q
[Equation] [Equation]
]( a, b )
[Equation]
( a, b ) (cid:12)(cid:12)
+ A max ( a,b ) ∈A×B (cid:12)(cid:12)(cid:12) Q s [ V t ]( a, b ) − Q ∗ s ( a, b ) (cid:12)(cid:12)(cid:12) ≤ γ 2 max {
[Equation]
B } (cid:18)(cid:13)(cid:13) V t − v ∗ (cid:13)(cid:13) 2 ∞ + (cid:13)(cid:13)(cid:13) V t − v ∗ (cid:13)(cid:13)(cid:13)∞ (cid:19) ≤ max { A, B } (cid:13)(cid:13)(cid:13) V t − V t (cid:13)(cid:13)(cid:13)∞ .
Then, by Lemma 2, we have
[Equation]
[Table]
z t −
Z ∗
+
(cid:101)
z
− (cid:101) z t −
(cid:13)(cid:13)(cid:13) of
( z t
Z by the fact that dist 2 (cid:0) z t
, Z
≤
2dist
[Equation]
(cid:16)(cid:101)
[Equation]
z t +1 − (cid:101)
z t (cid:13)(cid:13)(cid:13) 2 .
The bound of dist 2 ( z T 1 +1 , Z ∗ ) follows by combining with
Lemma 15.
□
The following lemma is straightforward from the definitions of q ts and q ts .
Lemma
For any t ∈ [ T 1 : T 2 ] and s ∈ S ,
[Equation]
Proof of Lemma 18.
For any s ∈ S , we have
[Equation]
Analogously,
[Equation]
Then, the proof is completed by combining the above two equations with the facts that
[Equation]
□
Now, we can prove the geometric boundedness of Averaging OGDA (Theorem 4) by combining
Lemma 16, Lemma 17, Lemma 18 inductively.
Proof of Theorem 4.
By Lemma
[Equation]
where
[Equation]
By Lemma 18,
[Equation]
where
[Equation]
By Lemma 17 and the fact that √ A 1 + A
[Equation]
+
[Equation]
[Table]
where
[Equation]
Define
[Equation]
Next, we prove (78) by induction max
[Equation]
The case of j = T 1 follows by (73), (74), (75).
Now, suppose that we have shown (78) for j ∈ [ T 1 : t ] .
Then, by Lemma 16 and the induction hypothesis (78),
[Equation]
By combining the above equation with (74) and the induction hypothesis (78),
[Equation]
By combining the above two equations with (76), (77) and the induction hypothesis (78),
[Equation]
By the definition of (cid:101) D , we have proved (78) for t +1 .
By induction, (78) holds for any t ∈
[ T 1 : T 2 ] .
The following relation is implied by (78) directly
[Equation]
where ˜ z =
(˜ x , ˜ y ) is the initial policy pair (59), (60).
Then, (11) follows by setting D
= (cid:101) D 2 .
By definition, we have D
= O ( S ( A + B ) 2 / (1 − γ ) 4 ) under the condition η ≤ 1 .
By Shapley’s theorem (Lemma 19), Z ∗ s =
X ∗ s × Y ∗ s is the set of Nash equilibria of a matrix game.
Thus,
Z ∗ s is convex, then, Z ∗ is also convex.
Thus, we have
[Equation]
As D 0 ≥ 1 in our definition, we have
[Equation]
This gives (12).
□
E P ROOFS FOR GLOBAL LINEAR CONVERGENCE
Proof of Theorem 1.
Recall the constants c 0 , δ 0 defined in the local linear convergence of
Local-Fast , D 0 defined in the geometric boundedness of Global-Slow , C ′ defined in the global convergence of Global-Slow in Section 3.1.
Define
[Equation]
Let M ∗ = max (cid:110)
( M
[Equation]
1 ) 2 , M
[Equation]
2 , ( M
[Equation]
3 ) 2 (cid:111) .
Then, the order of M ∗
[Equation]
For simplicity we denote
[Equation]
Note that (cid:98) z
k = (cid:98) z
[ I k gs : (cid:101)
I k gs ] is the initial policy pair of the k -th call to Local-Fast .
Define k ∗ as
[Equation]
Then,
[Equation]
Firstly, we provide bounds for (cid:98)
z k after k ≥ k ∗ .
For any k ≥ k ∗ , since (cid:101) I k
gs −
I k gs + 1 = 2 k ≥ 2 k ∗ ≥ M ∗ 1 , by (2) and the definition of M ∗ 1 , the policy pair (cid:98)
z k satisfies
[Equation]
Since (cid:98) z k is the initial policy pair of Local-Fast in time interval [ I k lf : (cid:101) I k lf ] , by (5), for t ∈
[ I k lf : (cid:101) I k lf ] , dist
[Equation]
(cid:17) .
Since 4 k ≥ 4 k ∗ ≥ M ∗ 2 , Γ 0 · (cid:18) 1 − c 0 η (cid:19) 4 k −≤ 1 .
Since 2 k ≥ 2 k ∗ ≥ M ∗ 3 , we have (cid:18) 1 − c 0 η (cid:19) 4 k −≤ (cid:18) 1 − c 0 η
[Equation]
(cid:19)
k +1 · ( 2 k − 1 − 1 ) ≤ max { D 0 , 1 } 2 k
+1 .
Then, by combining the above three equations,
[Equation]
By (4) and the fact that z (cid:101) I k lf is the initial policy pair of the ( k + 1) -th call to Global-Slow ,
[Equation]
Then, by combining (82) and (83)
[Equation]
we have
[Equation]
Next, we give a rough bound of dist 2 ( z t , Z ∗ ) for t ∈ [ I k lf : (cid:101) I k +1gs ] .
For t ∈ [ I k lf : (cid:101)
I k lf ] , by (5),
[Equation]
For t ∈ [ I k +1gs : (cid:101) I k +1gs ] , since z (cid:101)
I k lf is the initial policy pair of the ( k + 1) -th call to Global-Slow , it follows by (3)
that
[Equation]
where the first inequality is from (3); the second inequality is from the fact that (cid:12)(cid:12)(cid:12)
[Equation]
2 k +1 ; the third inequality is by (82).
Thus, for any t ∈ [ I k lf : (cid:101) I k +1gs ] ,
[Equation]
Now, we are ready to bound dist( z t , Z ∗ ) for each t ∈ [0 : T ] .
Firstly, we fix a k ′ ≥ k ∗ + 1 and a t ′ ∈
[ I k ′ lf : (cid:101) I k ′
+
gs ] .
Then, the time interval [0 : t ′ ] can be divided into:
[Equation]
By (84), we have dist
[Equation]
By combining with (85), we have
[Equation]
z
[Equation] [Equation]
By
[Equation] [Equation]
Thus,
[Equation]
Then, for the time t ′ we have fixed, dist
[Equation]
z t
[Equation]
Since the above arguments can be applied to any k ′ ≥ k ∗ + 1 and t ∈ [ I k ′ lf : (cid:101) I k ′
+
gs ] , we have that (87) holds for any t ≥ I k ∗ +1 lf .
By similar arguments to (86), we have (cid:101)
I k ∗ +1 gs ≤ 128 M ∗ / 3 .
Then, for any t ∈ [0 : (cid:101)
I k ∗ +
gs ] , dist( z t , Z ∗ ) ≤
[Equation]
S
[Equation]
S max { Γ 0 , 1 } · (cid:18)
[Equation]
Then, by combining with (87), for any t ∈ [0 : T ] ,
[Equation] [Equation]
(88) □
Proof of Theorem 2.
By Theorem 3 and Theorem 4, Averaging OGDA can serve as the base algorithm Global-Slow in the meta algorithm Homotopy-PO .
By Theorem 5, OGDA can serve as the base algorithm Local-Fast in the meta algorithm Homotopy-PO .
Then, by Theorem 1, we have the global linear convergence of the instantiation of Homotopy-PO with OGDA and Averaging OGDA.
More specifically, by Theorem 1 and (57), the constant c in (9) satisfies c >
and it is of order
[Equation]
By combining (80) with Theorem 3, Theorem 4, (57), (58), the constant M in (9) is of order
[Equation]
This completes the proof for global linear convergence of our instantiation for Homotopy-PO .
□
Remark 2 Theorem 2 requires η ≤ O ( (1 − γ )
52 √ S
A + B ) ) for OGDA and η ′ ≤ O
( 1 − γ A + B ) for Averaging
OGDA.
If we set η = O ( (1 − γ ) 52 √ S ( A + B ) ) , then the linear convergence rate is
[Equation]
If we set η = O ( (1 − γ ) 52 √ S ( A + B ) ) for OGDA and η ′
=
O ( 1 − γ A + B ) for Averaging OGDA, then the length of
Hidden Phase I is of order
[Equation]
F A UXILIARY LEMMAS
The following lemma gives a characterization of Nash equilibrium.
Its proof can be found in Section 3.9 of Filar & Vrieze (2012).
Lemma 19 Consider Markov game G = ( S , A , B , r, P , γ ) .
Given the minimax game value v ∗ ( s ) =
min x ∈X max y ∈Y V x , y ( s ) .
A policy pair ( x ∗ , y ∗ ) ∈
X × Y is a Nash equilibrium if and only if it holds for any s ∈ S that ( x ∗ s , y ∗ s ) is a Nash equilibrium of the matrix game
[Equation]
(89) where Q ∗ s is an A -by- B matrix with Q ∗ s ( a, b )
=
R s ( a, b )+ γ (cid:80) s ′ ∈S v ∗ ( s ′ )
P ( s ′ | s, a, b ) .
In addition, the minimax game value and the Nash equilibrium set of the matrix game (89) are v ∗ ( s ) and Z ∗ s =
X ∗ s ×
Y ∗ s , respectively.
Then, the Nash equilibrum set of Markov game G is Z ∗ =
(cid:81) s ∈S Z ∗ s .
The following lemma is known as “performance difference lemma” (Kakade & Langford, 2002).
It is used extensively throughout this paper.
Lemma 20 (Performance Difference Lemma)
For any policies x , x ′ ∈ X , y ∈ Y and state s 0 ∈ S , we have
[Equation]
The following lemma is standard.
We provide its proof for completeness.
Lemma
For any policies x , x ′ ∈ X , y , y ′ ∈ Y and state s ∈ S , state distribution ρ ∈ ∆ S , action pair ( a, b ) ∈
A × B .
Let z
=
( x , y ) and z ′ =
( x ′ , y ′ ) , then
[Equation]
Proof of Lemma 21.
By performance difference lemma (Lemma 20),
[Equation]
Similarly,
[Equation]
Then, by triangle inequality and the fact that (cid:80) s ′ ∈S
d x
[Equation]
, y s
[Equation]
, we have
[Equation]
Then, (91) follows by combining (90) with the definition Q x , y s
= Q s [ V x
[Equation]
To bound the difference of state visitation distribution
[Equation]
we fix s, s ′ ∈ S .
Let P ∈ R S × S be the transition matrix of policy pair ( x , y ) , i.e.,
[Equation]
Similarly, define P ′ as the transition matrix of ( x ′ , y ′ ) .
Then, d x , y s ( s 1 ) is the ( s, s 1 ) -th entry of (1 − γ )
( I − P ) − 1 ; d x ′ ,
y ′ s
( s 1 ) is the ( s, s 1 ) -th entry of (cid:0) I − P
′ (cid:1) − 1 .
By definition, for any s, s 1 ∈ S , (cid:88)
[Equation]
Thus, we have (cid:13)(cid:13) P − P ′
(cid:13)(cid:13) ∞ ≤ max ˜ s ∈S ∥
z ˜ s − z ′ ˜ s
∥ 1 .
By combining with the fact that (cid:13)(cid:13) ( I − P ) − 1 (cid:13)(cid:13) ∞ ≤ (cid:80) ∞ i =
γ i (cid:13)(cid:13) P i (cid:13)(cid:13) ∞ ≤ 11 − γ , we have
[Equation]
1 (cid:0) P − P ′ (cid:1) (cid:0)
I − P ′ (cid:1) − 1 1 s 1 (cid:69)(cid:12)(cid:12) (cid:12) ≤ (1 − γ ) (cid:13)(cid:13)(cid:13) ( I − P ) − 1 (cid:13)(cid:13)(cid:13) ∞ (cid:13)(cid:13)
P − P ′ (cid:13)(cid:13) ∞ (cid:13)(cid:13)(cid:13)(cid:0)
I − P
′ (cid:1) − 1 (cid:13)(cid:13)(cid:13) ∞ ≤ √
[Equation]
+
[Equation]
max s ′ ∈S
∥ z ′ s
[Equation]
z ′ s ′ ∥
− γ ≤ √ A +
[Equation]
z − z ′ ∥ 1 − γ .
Then, (cid:12)(cid:12)(cid:12) d x , y ρρρ ( s ) − d x ′ ,
y ′ ρρρ ( s ) (cid:12)(cid:12)(cid:12) ≤ (cid:88) s 0 ∈S ρ ( s 0 )
(cid:12)(cid:12)(cid:12) d x , y s 0 ( s ) − d x ′ , y ′ s( s ) (cid:12)(cid:12)(cid:12) ≤ √ A
+ B ∥
z − z ′ ∥− γ .
To show (93), we choose (cid:98) y ∈ arg max y V x , y ( s ) , then, by performance difference lemma (Lemma 20),
V x , (cid:98) y ( s ) − V x ′ , (cid:98) y
( s ) ≤ 1 − γ (cid:88)
s ′ ∈S
d x , (cid:98) y s ( s ′ )
∥ x s ′ − x ′ s ′
∥ 1 (cid:13)(cid:13)(cid:13) Q x ′ , (cid:98)
y s ′ (cid:98)
y s (cid:13)(cid:13)(cid:13)
[Equation]
Analogously, V x ′ , † ( s ) − V x , † ( s ) ≤ √ A ∥ x − x ′ ∥ (1 − γ ) 2 .
Thus, (cid:12)(cid:12)(cid:12) V x , † ( s ) − V x ′ , † ( s ) (cid:12)(cid:12)(cid:12) ≤ √ A ∥ x − x ′ ∥ (1 − γ ) 2 .
The inequality (94) follows similarly.
□
As a direct corollary of (93), (94), we can bound the Nash gap max s ∈S V x , † ( s ) − V † , y ( s ) by dist( z , Z ∗ ) .
Corollary 1 For any
[Equation]
Proof of Corollary 1.
Denote P X ∗ ( x ) = x ∗ , P Y ∗ ( y ) =
y ∗ , then z ∗ =
( x ∗ , y ∗ ) =
P Z ∗ ( z ) .
By the definition of Nash equilibria,
[Equation]
Then, by combining with (93), (94), for any s ∈ S ,
[Equation]
This completes the proof.
□
The following lemma is paraphrased from Lemma 4 of Gilpin et al.
and is also known as saddle-point metric subregularity of matrix games as in Theorem 5 of Wei et al.
It essentially shows that in matrix game min x max y x ⊤ Gy , the suboptimality of any policy pair can be lower bounded by a certain condition measure φ ( G ) of the matrix G multiplied by the policy pair’s distance to the Nash equilibrium set of the matrix game.
Lemma 22 (Lemma 4 of Gilpin et al.
Theorem 5 of Wei et al.
For any matrix G
[Equation]
, let X
[Equation]
G
[Equation]
=
[Equation]
and Y
[Equation]
Then, it holds that for any x ∈ ∆ A and y ∈ ∆ B ,
[Equation]
where φ ( Q ) > 0 is a certain condition measure of the matrix Q .
As a direct corollary of Lemma 22, we can instantiate the value of c + in (1).
Corollary 2 Let c + = min s ∈S φ ( Q ∗ s ) , then, for any policy pair z
= ( x
, y ) ∈
Z and s ∈ S ,
[Equation]
G D ECENTRALIZED IMPLEMENTATION OF THE ALGORITHMS
Recall that in our interaction protocol, the min-player only has access to its marginal reward function r tx and marginal transition kernel P tx , while the max-player only has access to its marginal reward function r ty and marginal transition kernel P ty .
Equivalently, in each iteration, the min-player receives full information of the Markov Decision
Process (MDP) M tx =
( S , A , P tx , r tx , γ ) , the max-player receives M ty = (cid:0) S , B , P ty , r ty , γ (cid:1) .
The marginal rewards and transition kernels are defined as
[Equation]
The value function of the policy x in the MDP M tx is defined as an S -dimensional vector containing the expected cumulative rewards of each state, i.e.,
[Equation]
The q-function q x , M tx = { q x , M
[Equation]
s } s ∈S is defined as a collection of A -dimensional vector with
[Equation]
The counterparts V M ty , y ( s ) , q M ty , y s for the max-player are defined similarly.
In the pseudocodes below, for any set C , I C denotes its indicator.
The decentralized implementation of OGDA (8) is in Algorithm 2 (min-player’s perspective) and
Algorithm 3 (max-player’s perspective).
The decentralized implementation of Averaging OGDA (6) is in Algorithm 4 (min-player’s perspective) and Algorithm 5 (max-player’s perspective).
Then, our instantiation of the meta algorithm Homotopy-PO which uses Averaging OGDA as
Global-Slow and OGDA as Local-Fast is naturally a decentralized algorithm.
The pseudocodes are presented in Algorithm 6 (min-player’s perspective) and Algorithm 7 (max-player’s perspective).
• Equivalence between OGDA (8) and Algorithm 2,
To prove the equivalence between OGDA (8) and Algorithm 2, 3, it suffices to show that q x t , M tx s =
Q ts y ts .
Actually, both q x t , M tx s and Q ts y ts equals the marginal q-function of the local MDP M tx =
{S , A , P tx , r tx , γ } observed by the min-player at iteration t .
By definition, we have for any s ∈ S , V x t , y t ( s ) =
V x t , M tx ( s ) =
V M ty , y t ( s ) .
Then, we have q x t , M Tx
[Equation]
a ) = (cid:88)
[Equation]
Thus, q x t , M Tx s = Q
ts y ts .
Analogously, q y t , M Ty s
[Equation]
Q ts
[Equation]
⊤ x ts .
This gives the equivalence between
OGDA (8) and Algorithm 2, 3.
Algorithm 2: x
Input: time interval: [ T 1 : T 2 ] , initial policy: (cid:98) x ∈ X , stepsize: η Initialize x T 1 = (cid:98) x for t = T 1 , · · · , T 2 do play policy x t receive r tx and P tx compute the q-function (cid:110) q x t , M tx s (cid:111) s ∈S in the MDP M tx =
( S , A , P tx , r tx , γ ) optimistic gradient descent (cid:101)
[Equation]
[Equation]
(cid:16)(cid:101) x
[Equation]
x t +1 s =
P ∆
A (cid:16)(cid:101) x ts − η q x t , M tx s (cid:17) end
Input: time interval: [ T 1 : T 2 ] , initial policy: (cid:98) y ∈ Y , stepsize: η , Initialize y T 1 = (cid:98) y for t
= T 1 , · · · , T 2 do play policy y t receive r ty and P ty compute the q-function (cid:26) q M ty , y t s (cid:27) s ∈S in the MDP M ty = (cid:0) S , B , P ty , r ty , γ (cid:1) optimistic gradient ascent (cid:101)
[Equation]
B (cid:18) (cid:101)
[Equation]
(cid:19)
[Equation]
(cid:18) (cid:101)
[Equation]
ts
[Equation]
(cid:19) end
Input: time interval: [ T 1 : T 2 ] , initial policy ˜ x ∈ X , stepsize: η
Initialize x T 1 = ˜ x for t = T 1 , · · · , T 2 do play policy x t receive r tx and P tx if t =
=
T 1 then solve the MDP M T 1 x = (cid:0) S , A , P T 1 x , r T 1 x , γ (cid:1) to compute
V T 1 ( s ) =
min x ′ ∈X V x ′ , M T 1 x ( s ) for any s ∈ S end compute for ( s, a ) ∈ S × A , q ts ( a ) =
r tx ( s, a ) + γ (cid:80) s ′ ∈S P tx ( s ′ | s, a ) V t ( s ′ ) optimistic gradient descent (cid:101) x
ts =
I { t = T 1 } · x T 1 s
+ I { t>T 1 }
· P ∆ A (cid:16)(cid:101) x t − 1 s −
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
update value function
[Table]
end
Compute the average policy (cid:98) x [
[Table]
=
[Table]
α t
[Table]
1 +
[Table] [Table]
1 +1 x
[Table]
Input: time interval: [ T 1 : T 2 ] , initial policy ˜ y ∈ Y , stepsize: η
Initialize y T 1 = ˜
y for t =
T 1 , · · · , T 2 do play policy y t receive r ty and P ty if t =
=
T 1 then solve the MDP M T 1 y
=
(cid:0) S , B , P T 1 y , r T 1 y , γ (cid:1) to compute
V T 1 ( s ) = max y ′ ∈Y V M T 1 y , y ′
( s ) for any s ∈ S end compute for ( s, b ) ∈ S × B , q ts ( b ) =
r ty ( s, b ) + γ (cid:80) s ′ ∈S P ty ( s ′ | s, b ) V t ( s ′ )
optimistic gradient ascent (cid:101) y
ts =
I { t = T 1 } · y T 1 s
+ I { t>T 1 } · P ∆ B (cid:16)(cid:101)
y t − 1 s + η q ts (cid:17)
y t
+1 s = P ∆ B (cid:16)(cid:101)
y ts + η q ts (cid:17)
update value function V t +1 ( s )
= max b ∈B (cid:80) tj
=
T 1 α j − T
+1 t − T 1 +1 q js ( b ) end
Compute the average policy (cid:98) y
[ T 1 : T 2 ] = (cid:80) T 2 t =
T 1 α t − T 1 +1 T 2 − T
+1 y t
• Equivalence between Averaging OGDA (6) and Algorithm 4,
Firstly, it follows by definition that
[Equation] [Equation]
Thus, the initiation steps in Averaging OGDA (6) and Algorithm 4, 5 are equivalent.
Thus, V T 1 in
Averaging OGDA (6) equals that in Algorithm 4.
Consider the variable q ts ( a ) defined in Algorithm 4,
[Equation]
By substituting (95) into (97) and combining the definition of the Bellman target operator in the introduction, we have
[Equation]
The RHS of (98) is exactly our definition for q ts in Averaging OGDA (6) in Section 3.2.
Analogously, the definition for q ts equals in (6) and Algorithm 5.
Then, by induction, (cid:110) q ts , q ts , V t ( s ) , V t ( s ) (cid:111) t ∈ [ T 1 : T 2 ] ,s ∈S has the same value in Averaging OGDA (6) and Algorithm 4, 5.
This gives the equivalence of Averaging OGDA (6) and Algorithm 4, 5.
• Decentralized implementation of Homotopy-PO
Recall that we have shown in Section 4 and Section 5 that Averaging OGDA (6) and OGDA (8) can serve as the base algorithm Global-Slow and Local-Fast in the meta algorithm
Homotopy-PO , respectively.
Thus, we can interpolate Averaging OGDA as Global-Slow and
OGDA as Local-Fast in the meta algorithm Homotopy-PO to obtain a globally linearly convergence algorithm.
We have shown that Averaging OGDA can be implemented in a decentralized manner (Algorithm 4, 5), and OGDA is also a decentralized algorithm (Algorithm 2, 3).
In addition, the inputs of
Algorithm 2, 3 and Algorithm 4, 5 only need local information (the min-player only needs x T 1 , the max-player only needs y T 1 ) with no requirement for knowledge of its opponent’s policies.
Thus, the algorithm constructed by interpolating Averaging OGDA and OGDA into Homotopy-PO is naturally a decentralized algorithm.
The pseudocodes in the min-player’s and the max-player’s perspectives are illustrated in Algorithm 6 and Algorithm 7, respectively.
We make final remarks that our instantiation for Homotopy-PO is symmetric and rational.
Since the min-player and the max-player use equal stepsize η for OGDA and equal stepsize η ′ for Averaging OGDA, the players have symmetric roles in our algorithms.
Rationality means one player can converge to the best response set when its opponent chooses a stationary policy.
This property is naturally possessed by decentralized and symmetric algorithms.
Similar arguments for rationality can also be found in some existing decentralized algorithms, see for instance Sayin et al.
; Wei et al. (2021).
We attach the proof for rationality here for completeness.
In addition, since our instantiation of Homotopy-PO has linear convergence, it is not only rational but also able to guarantee the linear convergence to the best response set.
Theorem 6 (Rationality)
If the max-player chooses a stationary policy (cid:98) y
= { (cid:98) y s } s ∈S ∈ Y and the min-player runs the instantiation of Homotopy-PO (Algorithm 6), then x t will converge to the best response set (cid:8) x
∈ X : V x , (cid:98) y ( s ) =
V † , (cid:98) y
( s ) , ∀ s ∈ S (cid:9) at a linear rate.
Analogously, if the min-player chooses a stationary policy (cid:98) x = { (cid:98) x s } s ∈S ∈ X and the max-player runs the instantiation of Homotopy-PO (Algorithm 7), then y t will converge to the best response set (cid:8)
[Equation]
at a linear rate.
Proof of Theorem 6.
Since the min-player and the max-player are symmetric, without loss of generality, we let the max-player chooses a stationary policy (cid:98)
y
= { (cid:98) y s } s ∈S ∈ Y .
Then, we define a new Markov game MG ′
= ( S , A , (cid:98) B , (cid:98) P , (cid:98) R , γ ) , where S , A , γ have the same meaning as in the original Markov game.
Now, the action set of the max-player only has one action (cid:98) B
= { 1 } .
(cid:98) P ( s ′ | s, a, 1) =
(cid:80) b ∈B P ( s ′ | s, a, b ) (cid:98) y s ( b ) represents the transition probability to state s ′ when the min-player takes action
a
and the max-player plays the stationary policy (cid:98) y .
Similarly, define (cid:98) R s ( a, 1)
= (cid:80) b ∈B R s ( a, b ) (cid:98) y s ( b ) as the marginal reward function that the min-player will receive when its opponent chooses the stationary policy (cid:98) y .
Denote the one-sided NE set of the min-player in the new Markov game MG ′ by X ∗ ( MG ′ ) .
By definition, the minimax game values (cid:98) v ∗ of MG ′ are (cid:98) v ∗ ( s ) =
V † , (cid:98) y
( s ) .
Then, for any x ∗ ∈ X ∗ ( MG ′ ) ,
V x ∗ , (cid:98) y
( s ) =
V † , (cid:98) y
( s ) for any s ∈ S .
Equivalently, X ∗ ( MG ′ ) is the best response set of (cid:98) y .
By applying Theorem 2 to the new Makov game MG ′ , we have that the policy x t played by the minplayer will converge at a global linear rate to X ∗ ( MG ′ )
that is the best response set of (cid:98) y .
Similar arguments also hold for the max-player.
This gives the rationality.
Input: iterations: [0 : T ] , initial policy: x 0 ∈ X , stepsizes: η, η ′ >set k = 1 , (cid:101) I 0lf = − 1 , x − 1 = xwhile (cid:101)
I k − 1 lf < T do
I k
gs =
(cid:101)
I k − 1 lf + 1 , (cid:101)
I k
gs =
min
{I k gs + 2 k − 1 , T } , I k lf = (cid:101)
I k
gs
+ 1 , (cid:101)
I k lf =
min {I k lf + 4 k − 1 , T } during time interval [ I k gs : (cid:101)
I k gs ] , run x-Averaging-OGDA ([ I k gs : (cid:101)
I k gs ] , x (cid:101)
I k − 1 lf , η ′ ) and compute an average policy (cid:98)
x [ I k gs : (cid:101)
I k gs ]
(Algorithm 4) during time interval [ I k lf : (cid:101) I k lf ] , run x-OGDA ([ I k lf : (cid:101) I k lf ] , (cid:98) x
[ I k gs : (cid:101)
I k gs ] , η ) (Algorithm 2)
k ← k + 1 end
Input: iterations: [0 : T ] , initial policy: y 0 ∈ Y , stepsizes: η, η ′ >set k = 1 , (cid:101) I 0lf = − 1 , y − 1 = ywhile (cid:101)
I k − 1 lf < T do
I k
gs =
(cid:101)
I k − 1 lf + 1 , (cid:101)
I k
gs =
min
{I k gs + 2 k − 1 , T } , I k lf = (cid:101)
I k
gs
+ 1 , (cid:101)
I k lf =
min {I k lf + 4 k − 1 , T } during time interval [ I k gs : (cid:101)
I k gs ] , run y-Averaging-OGDA ([ I k
gs : (cid:101)
I k gs ] , y (cid:101)
I k − 1 lf , η ′ ) and compute an average policy (cid:98)
y
[ I k gs : (cid:101)
I k gs ]
(Algorithm 5) during time interval [ I k lf : (cid:101)
I k lf ] , run y-OGDA ([ I k lf : (cid:101)
I k lf ] , (cid:98) y
[ I k gs : (cid:101)
I k gs ] , η ) (Algorithm 3)
k ← k + 1 end
H N ATURAL GENERALIZATION OF G LOBAL -S LOW WITH MORE EXAMPLE
In this section, we mainly (1) show the convergence results of Homotopy-PO when Global-Slow base algorithm has different rates on the RHS of (2); (2) provide another example of Global-Slow base algorithm with generalized global convergence rates by proving the geometric boundedness of
Algorithm 1 in Wei et al.
(2021) with a slightly modified initialization.
H.
C ONVERGENCE RESULT OF H OMOTOPY -PO WHEN G LOBAL -S LOW HAS DIFFERENT CONVERGENCE RATES
To avoid abuse of notations, we call the Global-Slow algorithm with more general global convergence rates by Generalized Global-Slow base algorithm .
Generalized Global-Slow base algorithm: by calling Gen-Global-Slow
([ T 1 : T 2 ] , ˜ z , η ′ ) during time interval [ T 1 : T 2 ] where ˜ z
= (˜ x , ˜ y ) is the initial policy pair, the players play policy pair z t = ( x t , y t ) for each iteration t ∈
[ T 1 : T 2 ] , and compute a policy pair (cid:98) z [ T 1 : T 2 ]
= ( (cid:98) x
[ T 1 : T 2 ] , (cid:98) y
[ T 1 : T 2 ] ) at the end of iteration T 2 such that z t , (cid:98) z [ T 1 : T 2 ] satisfy the following two properties:
• global convergence : there is a problem-dependent constant (cid:98) C ′ > 0 and real numbers p 1 > 0 and p 2 , p 3 ≥ 0 such that
[Equation]
• geometric boundedness : there exists a problem-dependent constant (cid:98) D 0 > 0 (possibly (cid:98) D 0 > 1 ) such that if η ′ ≤ 1 , then for any t ∈ [ T 1 : T 2 ] ,
[Equation]
The main difference between Gen-Global-Slow and Global-Slow is that (1) the RHS of (99) in the definition of Gen-Global-Slow add more flexibility in the power numbers then the condition (2) in the definition of Global-Slow ; (2) (cid:98)
z [ T 1 : T 2 ] need not to be an average policy.
In the example (104) below, we can simply set (cid:98) z
[ T 1 : T 2 ] =
z T 2 .
By similar arguments with Theorem 1, we have the following convergence rates for Homotopy-PO with generalized Global-Slow .
Theorem 7 Let { z t = ( x t , y t ) } t ∈ [0: T ] be the policy pairs played when running Homotopy-PO (Algorithm 1) where Global-Slow is replaced by Gen-Global-Slow .
Then, for any t ∈ [0 :
T ] , we have
[Equation]
where the value of (cid:98) C ′ , c 0 , δ 0 , Γ 0 can be found in the definitions of Gen-Global-Slow and
Local-Fast and
[Equation]
Proof of Theorem 7.
Let c 0 , δ 0 be defined in the local linear convergence of Local-Fast , D 0 defined in the geometric boundedness of Gen-Global-Slow , (cid:98) C ′ defined in the global convergence of Gen-Global-Slow .
Define
[Equation]
M ∗ 2 and M ∗ 3 are defined the same as in the proof of Theorem 1 in Appendix E.
Analogous to the proof of Theorem 1, we also let (cid:99) M ∗ = max (cid:110)
( M ∗ 1 ) 2 , M ∗ 2 , ( M ∗ 3 ) 2 (cid:111) .
This gives the order of (cid:99) M ∗ in (101).
Notice that the global linear rate only depends on the local linear rate of Local-Fast and the geometric boundedness of Global-Slow .
The global convergence rate of Gen-Global-Slow is only relevant to the length of Hidden Phase I, i.e., (cid:99) M ∗ 1 will only affect the length of Hidden
Phase I.
Then the rest of this proof follows from Theorem 1 directly.
Analogously to (88), we also have
[Equation]
This gives the convergence result of Homotopy-PO when equipped with Gen-Global-Slow and Local-Fast .
□
H. A NOTHER EXAMPLE OF G LOBAL -S LOW BASE ALGORITHM
Next, we show that the algorithm in Wei et al.
(2021) with a slightly modified initialization can serve as an example of Gen-Global-Slow .
It is shown in Theorem 2 of Wei et al.
that
Algorithm 1 therein has a sub-linear last-iterate global convergence rate which satisfies the RHS of (99) with p 1 = 12 , p 2 = 2 , p 3 = 0 .
To instantiate that Algorithm 1 in Wei et al.
(2021) can be an example of Gen-Global-Slow , it suffices to prove its geometric boundedness.
We remark that geometric boundedness may not hold for the original Algorithm 1 in Wei et al.
(2021) since its initialization V 0 ( s ) =
may cause the policy gradients in the first step to deviate largely.
However, this problem can be fixed simply by changing the initialization to V 0 ( s ) =
V x 1 , y 1 ( s ) .
When running Algorithm 1 of Wei et al.
(2021) in the full-information setting (with the different initialization discussed above) during the time interval [ T 1 : T 2 ] , the min-player and the max-player initialize (cid:101) x T
= x T 1 = ˜ x , (cid:101) y T 1 = y T 1 = ˜
y and
[Equation] [Equation]
104e) where (cid:98) Q t s =
Q
s
[ V t − 1 ] and β t =
H
+1 H
+ t with H
= (cid:108) 21 − γ (cid:109) .
Recall that Q s [ · ] is the Bellman target operator defined in the introduction.
When using the algorithm (104) with initialization (103), the output policy can be set as
[Equation]
We also denote z
[Equation]
=
[Equation]
∗ =
P Y ∗ ( y t ) , z t ∗ =
P Z ∗ ( z ) =
( x t ∗ , y t ∗ ) in the analysis below.
Next, we proceed to show the geometric boundedness of the algorithm of Wei et al.
(2021) with the slightly modified initialization in a similar way with Appendix D.2.
We first provide mutual bounds among { dist( z t , Z ∗ ) } and {∥ V t ( s ) − v ∗ ( s ) ∥ ∞ } in Lemma 23 and Lemma 24 below.
Lemma 23 Let { z t , V t } be generated from (104) with initialization (103) .
For any t ≥ T 1 ,
[Equation]
Proof of Lemma 23.
Firstly, define β jt = β
j Π tk = j +1 (1 − β k ) for 0 ≤ j ≤ t − 1 and β tt = β t .
Since β 0 t = 0 , by (104e), for any t ≥ T
[Equation]
(cid:88)
[Equation]
By the definition of (cid:98) Q j s , we have
[Equation]
By Lemma
[Equation]
v ∗ ( s ) =
(cid:10) x
j ∗ s , Q s [ v ∗ ] y j ∗ s (cid:11) .
Thus, for any t ≥ T 1 and s ∈ S , by combining the above equations
[Equation]
we have
[Equation]
β j − T 1 +1 t − T 1 +1 (cid:13)(cid:13)
x js − x j ∗ s (cid:13)(cid:13) 1 (cid:13)(cid:13)(cid:13) (cid:98) Q j s y js (cid:13)(cid:13)(cid:13) ∞
≤ t (cid:88)
j
=
Tβ j − T 1 +1 t − T
+ (cid:32)(cid:13)(cid:13) z js − z
j ∗ s (cid:13)(cid:13)1 − γ + (cid:13)(cid:13)
V j − 1 − v ∗ (cid:13)(cid:13) ∞ (cid:33) .
Then, the proof is completed by the fact that
[Equation]
s
[Equation]
□
Lemma 24 Let { z t , V t } be generated from (104) with initialization (103) .
Then, for any t ≥ T 1 , dist 2 (
[Equation]
Proof of Lemma 24.
By applying Lemma 2 to (104) and substituting (105), we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
≤
[Equation]
[Table]
[Equation]
The bound of dist 2 ( (cid:101) z t +1 , Z ∗ ) follows by the fact that dist 2 ( (cid:101)
z t +1 , Z ∗ ) ≤ 2dist 2 ( (cid:101)
z t , Z ∗ ) + 2 (cid:13)(cid:13)(cid:13)(cid:101)
z t +1 − (cid:101)
z t (cid:13)(cid:13)(cid:13) 2 .
The bound of dist 2 ( z t +1 , Z ∗ )
follows by the fact that dist 2 (cid:0) z t +1 , Z ∗ (cid:1) ≤ 2dist 2 (cid:16)(cid:101) z t +1 , Z ∗ (cid:17) + 2 (cid:13)(cid:13)(cid:13)
z t +1 − (cid:101) z t +1 (cid:13)(cid:13)(cid:13) 2 and substituting the bound for dist 2 ( (cid:101)
z t +1 , Z ∗ ) .
□
Next, we show the geometric boundedness of Algorithm 1 in Wei et al.
(2021) with the initialization (103).
Theorem 8 (Geometric Boundedness of Algorithm 1 in Wei et al.
Let { z t } t ∈
[ T 1 : T 2 ] be the policy pairs played by running the algorithm (104) with initialization (103) .
If η ≤ 1 , then there is a problem-dependent constant (cid:98) D
= O ( S ( A + B )
[Equation] [Equation]
4 ) (possibly (cid:98) D 0 > 1 ) such that for any t ∈ [ T 1 : T 2 ] ,
[Equation]
Proof of Theorem 8.
We will show (106) by proving (112) inductively.
Firstly, we define some constants which are used in the definition of (cid:98) D 0 .
By Lemma 23, for t ≥ T 1 ,
[Equation]
where
[Equation]
By Lemma 24 and the condition η ≤ 1 , for t ≥ T 1 ,
[Equation] [Equation]
where
[Equation]
For the initialization (103), by (91),
[Equation]
where
[Equation]
Define
[Equation]
By definition, (cid:98) D
[Equation]
O
[Equation]
Now, we proceed to prove (112) by induction.
[Equation]
The case of j = T 1 follows by (110) and the initialization z T 1 = (cid:101)
z T 1 = ˜ z .
Suppose we have shown (112) for j ∈ [ T 1 : t ] .
Then, by (107), the fact (cid:98) D 0 ≥ 1 and induction hypothesis,
[Equation]
By (108) and induction hypothesis
[Equation]
Analogously, by (109) and induction hypothesis
[Equation]
Thus, we have shown (112) for j = t + 1 .
By induction, (112) holds for any j ∈ [ T 1 : T 2 ] , which implies (106) directly.
This completes the proof for the geometric boundedness of the algorithm (104) with the initialization (103).
□
Remark
When the meta algorithm Homotopy-PO switches between Algorithm 1 of Wei et al.
(2021) (with the slightly modified initialization) and OGDA (8) , then by Theorem 7 and (57) ,
[Equation]
Then, if η = O ( (1 − γ ) 52 √ S ( A + B ) ) for OGDA, the linear rate is
[Equation]
As in Algorithm 1 of Wei et al.
(2021), the stepsize η therein has to satisfy η ≤ (1 − γ ) 52 10 4 √ S .
By combining
Theorem 1 of Wei et al. (2021) with (100) , (101) , (57) , (58) , if η = O ( (1 − γ ) 52 √ S ( A + B ) ) for OGDA and η ′ =
O ( (1 − γ ) 52 √ S ) for Algorithm 1 in Wei et al.
(2021), then the length of Hidden Phase I is of order
[Equation]
I N UMERICAL EXPERIMENTS
In this section, we evaluate the numerical performance of Homotopy-PO where Local-Fast and Global-Slow are instantiated with OGDA and Averaging OGDA respectively.
Markov game model.
We generate a sequence of zero-sum Markov games randomly and independently in the way described below and test the performance of Homotopy-PO on each of the games.
In each Markov game generated below, the number of states is S = 10 , the min-player and max-player have A = B = 10 actions respectively, and the discount factor γ = 0 . 99 .
The reward functions { R s ( a, b ) } s ∈S ,a ∈A ,b ∈B are generated from uniform distribution on [0 , 1] independently.
To generate the transition kernel, for each ( s, a, b ) , we first choose an integer i s,a,b uniformly at random from [ S ] .
Then, we choose a random subset M s,a,b ⊆ S with | M s,a,b | = i s,a,b .
Then for each s ′ ∈
M s,a,b , we set (cid:98) P ( s ′ | s, a, b ) from uniform distribution on [0 , 1] independently, and for s ′ ∈
S\ M s,a,b , we set (cid:98) P ( s ′ | s, a, b ) = 0 .
Finally, we normalize
P ( s ′ | s, a, b ) = (cid:98) P
( s ′ | s, a, b ) / (cid:80) s ′′ ∈S (cid:98) P ( s ′′ | s, a, b ) for each ( s, a, b ) to get the transition kernel.
For the initial policies, we first generate { u s } s ∈S with u s ( a ) chosen from uniform distribution on [0 , 1] for each s ∈ S , a ∈ A .
Then, we normalize x 0 s = u s / ∥ u s ∥ 1 for each s ∈ S .
The initial policy (cid:8) y 0 s (cid:9) s ∈S of the max-player is generated independently in the same way.
Algorithm implementation.
In all the experiments below, we set the stepsizes η = 0 .
1 in OGDA and also η ′ = 0 .
1 in Averaging OGDA.
We find our algorithm has linear convergence in all the experiments with these stepsizes.
Performance metric.
We measure the closeness of z t to the Nash equilibria set by the Nash gap max s ∈S V x t , † ( s ) − V † , y t ( s ) .
By combining Lemma 7 and Corollary 1 with the fact that dist( z , Z ∗ ) ≤ √ S max s ∈S dist( z s , Z ∗ s ) , we have the following relation between the Nash gap max s ∈S V x , † ( s ) − V † , y ( s ) and the distance to the NE set dist( z , Z ∗ ) : for any z = ( x , y ) ∈ Z ,
[Equation]
[Equation]
Thus, the linear convergence of dist( z t , Z ∗ ) is equivalent to the linear convergence of the Nash gap max s ∈S V x t , † ( s ) − V † , y t ( s ) up to problem-dependent constants.
In the figures below, y -axis represents the logarithmic of the Nash gap log (cid:0) max s ∈S V x t , † ( s ) − V † , y t ( s ) (cid:1) , x -axis represents the iteration number.
Remark 4 As we can see, there are discontinuities when switching from Averaging OGDA to OGDA in the figures below.
This is because Averaging OGDA is an averaging style method.
Recall that the y -axis represents log (cid:16) max s ∈S V x t , † ( s ) − V † , y t ( s ) (cid:17) .
However, the initial policy pair of the k -th call of OGDA is the average policy (cid:98)
z
[ I k gs : (cid:101)
I k gs ]
= (cid:80) (cid:101)
I k
gs t =
I k
gs α t
−I k
gs +
2 k z t .
Since it is quite possible that (cid:98)
z
[ I k gs : (cid:101)
I k gs ]
̸ = z (cid:101)
I k gs , there can be some discontinuities in the figures below when switching from
Averaging OGDA to OGDA.
On the other hand, our theoretical bound in Figure 1 is continuous because by setting t =
T 2 in (3) , theoretically dist 2 ( z T 2 , Z ∗ ) ≤ D T 2 − T 1· dist 2 (˜ z , Z ∗ ) whose bound equals the bound for dist
( (cid:98) z
[ T 1 : T 2 ] , Z ∗ ) on the RHS of (4) .
We remark that in practice, it is predictable that z (cid:101)
I k
gs ̸ = (cid:98) z
[ I k gs : (cid:101)
I k gs ] in most cases.
Numerical performance.
We validate the linear convergence of our instantiation of
Homotopy-PO , where Global-Slow and Local-Fast are instantiated by Averaging OGDA and OGDA respectively.
Figure 2 shows the performance when the min-player and max-player run Algorithm 6 and Algorithm 7, respectively.
We do 10 random and independent trials and the algorithm exhibits linear convergence in every trial.
The plot shows the average trajectory and standard deviation of therandom and independent trials.
The vertical dotted line is at the end of 7-th call to OGDA (iteration t = 22098 ).
As we can see, on the RHS of the dotted line (after t > 22098 ), the algorithm converges linearly and the Nash gap is less than 10 − 5 after 2 × 10 5 iterations.
The standard deviation of the random trials is illustrated by the shadow area.
Since the switching pattern is 2 k iterations of Averaging OGDA followed by 4 k iterations of OGDA, Averaging OGDA is only run for 1022 iterations in the total 2 × 10 5 iterations.
Thus, Averaging OGDA is hardly seen in Figure 2.
We magnify the trajectory of the 9-th call to Averaging OGDA as a subfigure in Figure 2.
We can find that Averaging OGDA increases in its 9-th call.
This has been predicted in our theoretical bounds (see segment
BC in Figure 1).
The 8-th call to OGDA has 4 8 iterations, while the 9-th call to Averaging OGDA only has 2 9 iterations.
We have 4 8 / 2 9 = 128 , i.e., the iterations of OGDA are hundreds of times more than those in the successive call to Averaging OGDA.
Then the increase caused by Averaging
OGDA can be naturally “omitted” compared with the decrease from OGDA.
This aligns with our theoretical bounds in Figure 1 (see the relation between the segments AB and BC in Figure 1).
[Figure]
dotted line (equivalently, after t > 22098 )
, the algorithm exhibits fast linear convergence.
In our switching pattern, 2 k ≪ 4 k when k is large.
Thus, Averaging OGDA is almost “invisible”.
We magnify the 9-th call to Averaging OGDA as a subfigure.
Though Averaging OGDA can increase, its increase is negligible by the decrease from hundreds of times more steps of OGDA.
This aligns with our theoretical guarantees (see the relation between segments AB and BC in Figure 1).
To avoid the problem that the iterations of Averaging OGDA is too few to be “visible”, we do another group of trials by generalizing the switching pattern slightly.
Recall that in Algorithm 1, the k -th call to Global-Slow has 2 k iterations while the k -th call to Local-Fast has 4 k iterations.
It is worth noting that the choices of 2 k and 4 k in Algorithm 1 is only for simplicity.
The proofs for linear convergence of Homotopy-PO can be directly generalized to the case when the k -th call to Global-Slow and Local-Fast has (cid:6) u k (cid:7) and (cid:6) v k (cid:7) iterations respectively whenever u , v are real numbers satisfying v > u > 1 .
Then to see how Homotopy-PO switches between Averaging OGDA and OGDA and see the performance difference between Averaging OGDA and OGDA separately, we test the performance of Homotopy-PO where the k -th call to Global-Slow and
Local-Fast has 2 k and (cid:6) 2 .
1 k (cid:7) iterations respectively.
We do another 10 random and independent trials in this switching pattern.
The average trajectory and standard deviation are illustrated in
[Figure]
l oga r i t h
m o f t
he
N a s h gap
Averaging OGDA
OGDA illustrated
Figure 3, where the iterations of Averaging OGDA are drawn in red while those of OGDA are drawn in blue.
We show the trajectories of the first 15 calls of Averaging OGDA and OGDA (iterations 1 ≤ t ≤ 195592 ) in Figure 3.
The discontinuity of the trajectory is because Averaging OGDA is an averaging style method and OGDA uses the average policy (cid:98)
z
[ I k gs : (cid:101)
I k gs ] rather than z (cid:101)
I k gs as the initial policy (see Remark 4).
We draw a vertical dotted line at the end of the 12-th call to OGDA (iteration t = 22237 ).
It can be observed that on the RHS of the dotted line (after t > 22237 ), the algorithm exhibits linear convergence.
On the RHS of the dotted line, the performance of Averaging OGDA is generally inferior to OGDA.
Averaging OGDA can even increase in some iterations.
This coincides with our theoretical bounds (see the segment BC in Figure 1).
Thanks to the fast and efficient linear convergence of OGDA together with the fact that the iterations of Averaging OGDA take up less and less proportion in the total iterations, the algorithm can exhibit linear convergence on the RHS of the vertical dotted line.
This also aligns with our theoretical bounds illustrated in Figure 1.
To see the switches between Averaging OGDA and OGDA clearly in each trial, in Figure 4 and
Figure 5 below, we present the 10 random trials of the changed switching pattern ( 2 k iterations of
Averaging OGDA followed by (cid:6) 2 .
1 k (cid:7) iterations of OGDA).
We illustrate the trajectories of the first 15 calls of Averaging OGDA and OGDA (iterations 1 ≤ t ≤ 195592 ) in Figure 4 and Figure 5.
In each subplots, we draw a vertical dotted line at the end of the 12-th call to OGDA (iteration t = 22237 ).
It can be observed that on the RHS of the dotted line (after t > 22237 ), the algorithm has linear convergence in each trial.
In some of the trials, Averaging OGDA can increase in some iterations.
This is predicted (see segment BC in Figure 1).
Since OGDA converges linearly and
Averaging OGDA takes less and less proportion in the total iterations, the algorithm can still exhibit linear convergence on the RHS of the dotted line ( t > 22237 ).
This aligns with our theoretical bounds (see the relation between segments AB and BC in Figure 1).
Even in the worst case (the 8-th trial), the Nash gap is less than 10 − 3 after 2 × 10 5 iterations.
And in some fast cases such as the 3-rd, 4-th, 5-th, 9-th, 10-th trials, the Nash gap can be less than 10 − 6 or even 10 − 8 in about 2 × 10iterations.
[Figure]
described for is an averaging style method (
Remark bounds in Figure
We also compare our algorithm with Alg. 1 in Wei et al.
We choose both stepsize of our
Homotopy-PO and Alg. 1 in Wei et al.
(2021) to be 0 .
We choose the discount factor γ = 0 .
5 , and the rest settings are the same with those in the experiments above.
The switching scheme is chosen to be the same with that in Figure 3 above.
The comparison between Homotopy-PO and Alg.
1 in Wei et al.
(2021) is illustrated in Figure 6, where the curves are drawn by taking average over
[Figure]
5 random trajectories and connecting the points at the time points when Homotopy-PO switches between Averaging OGDA and OGDA.
As we can see in Figure 6, Homotopy-PO is faster than Alg. 1 in Wei et al.
iteration number 101010 - 1010 - 10l oga r
i t h m o f t
he
N a s h gap
Homotopy-PO
Alg. 1 in
J D ISCUSSIONS J. P OSSIBLE TRANSLATION TO LEARNING ALGORITHMS
We remark that it is possible to translate our algorithm into learning algorithms.
Here, we tentatively discuss the analogues of local linear convergence of OGDA under the following two cases and give an intuitive analysis for each case.
The analogues of global convergence and geometric boundedness of Averaging OGDA can be discussed similarly.
• Case 1: Assuming access to a simulator (generative model).
If there is a simulator (generative model) and the players can draw lots of samples in one iteration, then it is possible to get linear convergence against the iteration number.
More specifically, at iteration t , for each s , N t samples are drawn from the distributions P ( ·| s, a t,j , b t,j ) , where 1 ≤ j ≤ N t and a t,j ∼ x t s , b t,j ∼ y t s .
We define a very small variable δ t =
O ( c 0 η 2 (1 − c 0 η 2 ) t ) .
Define the truncated simplex
∆ t A = { x ∈ R
A : x ( a ) ≥ δ t , (cid:80) x
( a ) = 1 } .
∆ t B is defined analogously.
At iteration t , we replace the projection operator P
∆ A ( · ) and P ∆ B ( · ) with P ∆ t A ( · ) and P ∆ t B ( · ) .
This guarantees that each action is taken with probability at least δ t .
Then by Hoeffding’s inequality, each action a is taken by the min-player for at least O ( N t δ t ) times with high probability (w.h.p).
Then, the empirical marginal reward (cid:98) r tx and marginal transition kernel (cid:98) P tx observed by the min-player satisfy the following relation w.h.p.,
[Equation]
In this remark, (cid:101) O ( · ) suppresses logarithmic terms and problem parameters such as S, A, B, 1 / (1 − γ ) for simplicity.
Thus, we have ∥ (cid:98) V x t ,y t − V x t ,y t ∥
[Equation]
1 N t δ
[Equation]
) , ∥
[Equation] [Equation]
∥
[Equation]
(cid:113)
[Equation]
, | (cid:98)
[Equation]
( a,
[Equation]
−
[Equation]
( a, b ) | ≤ (cid:101) O
( (cid:113) 1 N t δ t )
for any ( s, a, b ) .
Here, we use (cid:98) · overhead to indicate the empirical quantities.
And the replacement of P ∆ A ( · ) by P ∆ t A ( · ) will add an error term whose ℓ ∞ -norm is at most δ t .
Thus in each iteration t , new error terms of order (cid:101) O ( (cid:113) 1 N t δ t + δ t ) are added.
At iteration t , let ¯
x
t
+1 be the ideal variable computed from { x j } j ≤ t with exact value of the marginal information r tx and P tx .
Let x t +1 be the real variable computed in the learning process.
¯ y t +1 and y t +1 are defined similarly.
Then, by Cauchy-Schwartz inequality, dist 2 ( z t +1 , Z ∗ ) ≤ (1 + c 0 η 2 / 2) dist 2 (¯ z t +1 , Z ∗ )
+ (1 + 1 / (2 c 0 η 2 )) (cid:101) O
( 1 N t δ t + δ 2 t ) .
After adding these error terms to the proof of Theorem 5, the bounds for the potential functions Λ t defined in (27) will be
[Equation]
Then by setting N t
= O ( c 30 η
(1 − c 0 η 2 ) 2 t ) , we can show by induction that Λ t
≤ O ((1 − c 0 η 2 / 3) t ) .
This gives the local linear convergence of OGDA when the players can draw lots of samples in one iteration.
• Case 2: Using an ergodic assumption.
When no simulator is available, we consider translating our algorithm into a learning algorithm under an ergodic assumption.
The assumption is that there exists a constant L 0 > 0 such that for any policy pair z
= ( x, y ) , if the min-player and the maxplayer play policy x and y respectively in L 0 successive iterations t ∈ [ T 0 : T 0 + L 0 − 1] , then, for any initial state s T 0 and state s ∈ S , there exists a t ′ ∈
[ T 0 : T 0 + L 0 − 1] such that s t ′ =
s .
Briefly, this assumption requires that when the players choose a stationary policy pair for successive Literations, then every state must be visited at least once in these D 0 iterations.
Under this assumption, our strategy is to regard successive L 0 × N k iterations as a virtual iteration
k .
In this way, we divide [1 : T ] into [
[Equation]
where T 2 k − T 2 k − 1 =
L 0 × N k .
Then, in the time interval [ T 2 k − 1 : T 2 k ] , each state s has been visited for at least N k times.
This is similar to the case when we have a simulator and N k samples are drawn for each state s in iteration k .
In this way, by applying our algorithm and analysis for the simulator case (Case 1 above), we can show the local linear convergence with respect to the virtual iteration number k .
J. D ISCUSSIONS ABOUT THE INTUITIONS BEHIND A VERAGING OGDA AND OGDA
• In Markov games, the main challenge of finding an NE
is
to
estimate
the minimax
game
values { v ∗ ( s ) }
If { v ∗ ( s ) } are already known, the players can use Q s [ v ∗
y ts and Q s
v ∗
⊤ x ts as policy gradients to do optimistic gradient descent
ascent.
Then finding
an NE is reduced to
solving
S
matrix
games
min
[Table]
y
[Table]
x
[Table]
s Q s [ v ∗ ] y s separately.
Approximating { v ∗ ( s ) } s ∈S is
dif-
ficult
because (1) zero-sum Markov games are nonconvex-nonconcave problems
and then, solving { v ∗ ( s ) } is almost as hard as our goal which is finding a NE; (2) the players only know marginal reward and marginal transition kernel, so in each player’s perspective, the marginal MDP observed is dynamic.
More specifically, the errors in approximations for { v ∗ ( s ) } will cause inaccuracy in approximate Q-functions and policy gradients.
The inaccuracy in policy gradients will make z t far from the NE set and then induce errors in approximations for { v ∗ ( s ) } .
This will easily generate a vicious circle and make the errors blow up.
• Averaging OGDA and OGDA employ different ways to approximate { v ∗ ( s ) } s ∈S in each iteration.
OGDA directly uses V x t ,y t ( s ) to approximate v ∗ ( s ) .
Thus, the min-player uses Q ts y ts =
Q s [ V x t ,y t ] y ts as approximate Q-functions to do optimistic gradient descent, while the max-player uses ( Q ts )
⊤ x ts = Q s [ V x t ,y t ] ⊤ x ts as approximate Q-functions to do optimistic gradient ascent.
On the other hand, in Averaging OGDA, the min-player and max-player use V t ( s ) and V t ( s ) respectively to approximate { v ∗ ( s ) } .
Thus, the min-player uses q ts = Q s [ V t ( s )]
y ts as approximate Q-functions to do optimistic gradient descent, while the max-player uses q ts = Q s [ V t ( s )]
⊤ x ts as approximate Q-functions to do optimistic gradient ascent.
V t ( s ) and V t ( s ) are computed through the averaging technique.
Take the min-player as an example, where V t ( s ) = min a ∈A (cid:80) t − 1 j = T 1 α j − T 1 +1 t −
Tq js ( a ) is a step of value iteration on the average of past Q-function approximations.
We will elaborate on the intuition of V t ( s ) below, but before that, we first show why the averaging step is needed to achieve global convergence (equivalently, the disadvantage of using V x t ,y t ( s ) to approximate v ∗ ( s ) ).
• Technical challenge in the analysis of OGDA.
The main difficulty in the analysis of OGDA is the nonconvex-nonconcave essence of zero-sum Markov games.
As discussed in Section 5.
of Daskalakis et al.
(2020), the failure of the Minty Variational Inequality (MVI) property in zero-sum Markov games poses challenges for the last-iterate convergence of extragradient methods/optimistic gradient methods.
More specifically, given the objective function f ( z ) with z = ( x, y ) and F ( z ) =
( ∇ x f ( z ) , −∇ y f ( z )) , the MVI property means that there exists a point z ∗ =
( x ∗ , y ∗ ) such that ⟨ F ( z ) , z − z ∗ ⟩ ≥ 0 for any z .
Proposition 2 of Daskalakis et al.
(2020) proves that when setting f ( x, y ) =
V x , y ( s ) for some state s ∈ S , the MVI property can fail in arbitrarily small neighborhoods of the NE set.
More specifically, for the OGDA method (8), it may happen that there exists some s ∈ S such that
[Equation]
(114) where (cid:101) x t ∗ =
P X ∗ ( (cid:101) x t ) , (cid:101) y t ∗ =
P Y ∗ ( (cid:101) y t ) are projections.
We also denote (cid:101)
z t ∗ =
P Z ∗ ( (cid:101) z t ) .
The troublesome case (114) implies that going in the directions of policy gradients may deviate from rather than get close to the NE set.
A naive bound to evaluate how worse the policy gradients can be is:
[Equation]
Q t +1 s (cid:1)
⊤ x t +1 s (cid:69) ≥ − 2 max ( a,b )
∈A×B (cid:12)(cid:12) Q t +1 s ( a, b ) − Q ∗ s ( a, b ) (cid:12)(cid:12) .
The troublesome error term in the naive bound is of order 2 max ( a,b ) ∈A×B (cid:12) (cid:12) Q t +1 s ( a, b ) −
Q ∗ s ( a, b ) (cid:12)(cid:12) =
O (dist( z t , Z ∗ )) .
On the other hand, as we show in Appendix C.2, projected optimistic gradient descent/ascent can only provide progress of order O (dist 2 ( z t , Z ∗ )) .
When z t is close to the NE set, the error term can be much larger than the progress, i.e., 2 max ( a,b ) ∈A×B (cid:12)(cid:12)
Q t +1 s ( a, b ) −
Q ∗ s ( a, b ) (cid:12)(cid:12) <
< O (dist 2 ( z t , Z ∗ )) .
This prevents us from even showing the local convergence of
OGDA.
• Intuition of Averaging OGDA.
Averaging OGDA tackles
the
problem (114) by using V t , V t
instead of V
x t
,y t to approximate v ∗ .
The corresponding policy gradients Q s [ V t ] y ts , Q s [ V t ] ⊤ x ts are good directions in the sense that we can provide a good lower bound for ( x ts − x t ∗ s )
⊤ Q s [ V t ] y ts + ( y t ∗ s − y ts )
⊤ Q s [ V t ] ⊤ x ts .
More specifically, by Lemma 19 and Fact 2, we have
[Equation]
As in Appendix D.1, the term ∥ V t − V t ∥ ∞ is relatively easy to control.
Thus, Q
s [ V t ] y ts , Q s [ V t
[Equation]
⊤ x ts are “good” directions.
• More discussions on OGDA.
Although Averaging OGDA use the averaging technique to achieve global convergence, the averaging technique will make the errors from past iterations fail to decrease exponentially fast.
This prevents Averaging OGDA from achieving linear convergence.
To get local linear convergence, OGDA is a natural candidate because V x t ,y t depends only on the most recent policies, thus, it avoids errors from the past steps.
The problem (114) mentioned above necessitates novel analysis for OGDA.
Our strategy for the local linear convergence of OGDA starts from considering a weighted sum of
(cid:68)
x t
+ s
[Equation]
x t
[Equation]
s
, Q t +
s
y t
[Equation]
1 s
[Equation]
and
(cid:68) (cid:101)
[Equation]
⊤
[Equation]
Let ρρρ 0 denote the uniform distribution on S .
As ( (cid:101) x t
∗
, (cid:101) y t
∗ )
attains a
Nash
equilibrium
[Equation]
Thus
0 ≤
[Equation]
−
V (cid:101) x t ∗ , y t +1 ( ρρρ 0 ) .
Then by applying
performance difference lemma
Lemma 20),
we
have a variant of the MVI property with time-varying coefficients which is as in (16), which can be regarded as a variant of the MVI property with time-varying coefficients.
In order to
utilize ( to get local linear convergence, we still need to tackle the following two problems:
• whether we can find a neighborhood of the NE set such that the time-varying coefficients d tx ( s ) , d ty ( s ) in (16) are “stable”?
• if the time-varying coefficients d tx ( s ) , d ty ( s ) in (16) can be “stable” in a small neighborhood of the NE set, will the difference between Q ts and Q ∗ s prevent the local linear convergence?
To address the above questions, we mainly use the two geometric observations in Section 5.
Observation I guarantees the progress of projected gradient descent/ascent is substantial.
This means that the difference between Q ts and Q ∗ s will not be troublesome in deriving the local linear convergence.
Observation II implies the stability of state visitation distribution.
Thus, the time-varying coefficients d tx ( s ) , d ty ( s ) will be “stable” when z t , (cid:101)
z t are approaching the NE set.
In other words, we can find a problem-dependent neighborhood where the time-varying coefficients d tx ( s ) , d ty ( s ) will possess some “stability”.
We remark that since d tx ( s ) , d ty ( s ) can vary rapidly when z t is far from the NE set, our analysis for OGDA only hold in a small neighborhood of the NE set.
Whether
OGDA has global convergence is still an open problem.