A S HARP A NALYSIS OF M ODEL - BASED R EINFORCEMENT L EARNING WITH S ELF -P LAY
Anonymous authors
Paper under double-blind review
A BSTRACT
Model-based algorithms—algorithms that explore the environment through building and utilizing an estimated model—are widely used in reinforcement learning practice and theoretically shown to achieve optimal sample efﬁciency for singleagent reinforcement learning in Markov Decision Processes (MDPs).
However, for multi-agent reinforcement learning in Markov games, the current best known sample complexity for model-based algorithms is rather suboptimal and compares unfavorably against recent model-free approaches.
In this paper, we present a sharp analysis of model-based self-play algorithms for multi-agent Markov games.
We design an algorithm Optimistic Nash Value Iteration (Nash-VI) for two-player zero-sum Markov games that is able to output an (cid:15)
-approximate
Nash policy in ˜
O ( H 3 SAB/(cid:15) 2 ) episodes of game playing, where S is the number of states, A, B are the number of actions for the two players respectively, and H is the horizon length.
This signiﬁcantly improves over the best known model-based guarantee of ˜ O
( H 4 S 2 AB/(cid:15) 2 ) , and is the ﬁrst that matches the information-theoretic lower bound Ω( H 3 S ( A + B ) /(cid:15) 2 ) except for a min { A, B } factor.
In addition, our guarantee compares favorably against the best known model-free algorithm if min { A, B } =
o ( H 3 ) , and outputs a single Markov policy while existing sampleefﬁcient model-free algorithms output a nested mixture of Markov policies that is in general non-Markov and rather inconvenient to store and execute.
We further adapt our analysis to designing a provably efﬁcient task-agnostic algorithm for zero-sum Markov games, and designing the ﬁrst line of provably sample-efﬁcient algorithms for multi-player general-sum Markov games.
I NTRODUCTION
This paper is concerned with the problem of multi-agent reinforcement learning (multi-agent RL), in which multiple agents learn to make decisions in an unknown environment in order to maximize their (own) cumulative rewards.
Multi-agent RL has achieved signiﬁcant recent success in traditionally hard AI challenges including large-scale strategy games (such as GO)
(Silver et al., 2016; 2017), real-time video games involving team play such as Starcraft and Dota2 (OpenAI, 2018; Vinyals et al., 2019), as well as behavior learning in complex social scenarios (Baker et al., 2020).
Achieving human-like (or super-human) performance in these games using multi-agent RL typically requires a large number of samples (steps of game playing) due to the necessity of exploration, and how to improve the sample complexity of multi-agent RL has been an important research question.
One prevalent approach towards solving multi-agent RL is model-based methods, that is, to use the existing visitation data to build an estimate of the model (i.e. transition dynamics and rewards), run an ofﬂine planning algorithm on the estimated model to obtain the policy, and play the policy in the environment.
Such a principle underlies some of the earliest single-agent online RL algorithms such as E3 (Kearns & Singh, 2002) and RMax (Brafman & Tennenholtz, 2002), and is conceptually appealing for multi-agent RL too since the multi-agent structure does not add complexity onto the model estimation part and only requires an appropriate multi-agent planning algorithm (such as value iteration for games (Shapley, 1953)) in a black-box fashion.
On the other hand, modelfree methods do not directly build estimates of the model, but instead directly estimate the value functions or action-value (Q) functions of the problem at the optimal/equilibrium policies, and play the greedy policies with respect to the estimated value functions.
Model-free algorithms have also
[Table]
been well developed for multi-agent RL such as friend-or-foe Q-Learning (Littman, 2001) and Nash
Q-Learning (Hu & Wellman, 2003) [Table]
While both model-based and model-free algorithms have been shown to be provably efﬁcient in multi-agent RL in a recent line of work (Bai & Jin, 2020;
Xie et al., 2020; Bai et al., 2020)
[Table]
a more precise understanding of the optimal sample complexities within these two types of algorithms (respectively) is still lacking.
In the speciﬁc setting of two-player zero-sum Markov games, the current best sample complexity for model-based algorithms is achieved by the VI-ULCB (Value
Iteration with Upper/Lower Conﬁdence Bounds) algorithm (Bai & Jin, 2020; Xie et al., 2020):
In a tabular Markov game with S states, { A, B } actions for the two players, and horizon length H ,
VI-ULCB is able to ﬁnd an (cid:15)
-approximate
Nash equilibrium policy in ˜ O
( H 4 S 2 AB/(cid:15) 2 ) episodes of game playing.
However, compared with the information-theoretic lower bound Ω( H 3 S ( A + B ) /(cid:15) 2 ) , this rate has suboptimal dependencies on all of H , S , and A, B .
In contrast, the current best sample complexity for model-free algorithms is achieved by Nash V-Learning (Bai et al., 2020), which ﬁnds an (cid:15) -approximate
Nash policy in ˜ O
( H 6 S ( A + B ) /(cid:15) 2 ) episodes.
Compared with the lower bound, this is tight except for a poly( H ) factor, which may seemingly suggest that model-free algorithms could be superior to model-based ones in multi-agent RL.
However, such a conclusion would be in stark contrast to the single-agent MDP setting, where it is known that model-based algorithms are able to achieve minimax optimal sample complexities (Jaksch et al., 2010; Azar et al., 2017).
It naturally arises whether model-free algorithms are indeed superior in multi-agent settings, or whether the existing analyses of model-based algorithms are not tight.
This motivates us to ask the following research question:
Question : How sample-efﬁcient are model-based algorithms in multi-agent RL?
In this paper, we advance the theoretical understandings of multi-agent RL by presenting a sharp analysis of model-based algorithms on Markov games.
Our core contribution is the design of a new model-based algorithm Optimistic Nash Value Iteration (Nash-VI) that achieves an almost optimal sample complexity for zero-sum Markov games and improves signiﬁcantly over existing modelbased approaches.
We summarize our main contributions as follows.
A comparison between our and prior results can be found in Table 1.
• We design a new model-based algorithm Optimistic Nash Value Iteration (Nash-VI) that provably ﬁnds (cid:15) -approximate
Nash equilibria for Markov games in ˜ O
( H 3 SAB/(cid:15) 2 ) episodes of game playing (Section 3).
This improves over the best existing model-based algorithm by O ( HS ) and is the ﬁrst algorithm that matches the sample complexity lower bound except for a ˜ O (min { A, B } ) factor, showing that model-based algorithms can indeed achieve an almost optimal sample complexity.
Further, unlike state-of-the-art model-free algorithms such as Nash V-Learning (Bai et al., 2020), this algorithm achieves in addition a ˜ O ( √ T ) regret bound, and outputs a simple Markov policy ( instead of a
nested mixture of Markov policies as returned by Nash V-Learning). • We design an alternative algorithm Optimistic Value Iteration with Zero Reward (VI-Zero) that is able to perform task-agnostic (reward-free) learning for multiple Markov games sharing the same transition (Section 4).
For N > 1 games with the same transition and different (known) rewards, VI-Zero can ﬁnd (cid:15) -approximate
Nash policy for all games simultaneously in ˜ O
( H 4 SAB log N/(cid:15) 2 ) episodes of game playing, which scales logarithmically in the number of games.
• We design the ﬁrst line of sample-efﬁcient algorithms for multi-player general-sum Markov games.
In a multi-player game with M players and A i actions per player, we show that an (cid:15) nearoptimal policy can be found in ˜ O
( H 4 S 2 (cid:81) i ∈
[ M ] A i /(cid:15) 2 ) episodes, where the desired optimality can be either one of Nash equilibrium, correlated equilibrium (CE), or coarse correlated equilibrium (CCE).
We achieve this guarantee by either a multi-player version of Nash-VI or a multi-player version of reward-free value iteration (Section 5 & Appendix C).
Due to space limit, we defer a detailed survey of related works to Appendix A.
P RELIMINARIES
In this paper, we consider Markov Games (MGs, Shapley, 1953; Littman, 1994), which are also known as stochastic games in the literature.
Markov games are the generalization of standard
Markov Decision Processes (MDPs) into the multi-player setting, where each player seeks to maximize her own utility.
For simplicity, in this section we describe the important special case of twoplayer zero-sum games , and return to the general formulation in Appendix C.
Formally, we consider the tabular episodic version of two-player zero-sum Markov game, which we denote as MG( H, S , A , B , P , r ) .
Here H is the number of steps in each episode, S is the set of states with |S| ≤ S , ( A , B ) are the sets of actions of the max-player and the min-player respectively with |A| ≤ A and |B| ≤ B , P = { P h } h ∈
[ H ] is a collection of transition matrices, so that P h ( ·| s, a, b ) gives the distribution of the next state if action pair ( a, b ) is taken at state s at step h , and
r = { r h } h ∈ [ H ] is a collection of reward functions, where r h : S ×
A × B → [0 , 1] is the deterministic reward function at step h .
This reward represents both the gain of the max-player and the loss of the min-player, making the problem a zero-sum Markov game.
In each episode of this MG, we start with a ﬁxed initial state s 1 .
At each step h ∈
[ H ] , both players observe state s h ∈ S , and pick their own actions a h ∈ A and b h ∈ B simultaneously.
Then, both players observe the actions of their opponent, receive reward r h ( s h , a h , b h ) , and then the environment transitions to the next state s h +1 ∼ P h ( ·| s h , a h , b h ) .
The episode ends when s H +1 is reached.
Policy, value function.
A (Markov) policy µ of the max-player is a collection of H functions { µ h : S → ∆ A } h ∈ [ H ] , each mapping from a state to a distribution over actions.
(Here ∆ A is the probability simplex over action set A .)
Similarly, a policy ν of the min-player is a collection of H functions { ν h : S → ∆ B } h ∈
[ H ] .
We use the notation µ h ( a | s ) and ν h ( b | s ) to represent the probability of taking action a or b for state s at step h under Markov policy µ or ν respectively.
We use V µ,νh : S → R to denote the value function at step h under policy µ and ν , so that V µ,νh ( s ) gives the expected cumulative rewards received under policy µ and ν , starting from s at step h :
[Equation]
We also deﬁne Q µ,νh : S ×
A × B → R to be the Q -value function at step h so that Q µ,νh ( s, a, b ) gives the cumulative rewards received under policy µ and ν , starting from ( s, a, b ) at step h :
[Equation]
µ
[Equation]
For simplicity, we deﬁne operator P h as [ P h V ]( s, a, b ) :
=
E s (cid:48) ∼ P h ( ·| s,a,b ) V ( s (cid:48) ) for any value function V .
We also use notation [ D π Q ]( s ) :
= E ( a,b ) ∼ π ( · , ·| s ) Q ( s, a, b ) for any action-value function Q .
By deﬁnition of value functions, we have the Bellman equation
[Equation]
for all ( s, a, b, h ) ∈ S ×
A × B × [ H ] , and at the ( H + 1) th step we have V
µ,νH +1 ( s ) =
for all s ∈ S .
Best response and Nash equilibrium.
For any policy of the max-player µ , there exists a best response of the min-player, which is a policy ν † ( µ ) satisfying V µ,ν † ( µ ) h ( s ) =
inf ν V µ,νh ( s ) for any ( s, h ) ∈ S
×
[ H ] .
We denote V µ, † h :=
V µ,ν † ( µ ) h .
By symmetry, we can also deﬁne µ † ( ν ) and V † ,νh .
It is further known (cf.
(Filar & Vrieze, 2012))
that there exist policies µ (cid:63) , ν (cid:63) that are optimal against the best responses of the opponents, in the sense that
[Equation]
for all ( s, h ) .
We call these optimal strategies ( µ (cid:63) , ν (cid:63) ) the Nash equilibrium of the Markov game, which satisﬁes the following minimax equation 2 :
[Equation]
Intuitively, a Nash equilibrium gives a solution in which no player has anything to gain by changing only her own policy.
We further abbreviate the values of Nash equilibrium
[Equation]
and Q (cid:63)h .
We refer readers to Appendix D for Bellman optimality equations for (the value functions of) the best responses and the Nash equilibrium.
Learning Objective.
We measure the suboptimality of any pair of general policies (ˆ µ, ˆ ν ) using the gap between their performance and the performance of the optimal strategy (i.e., Nash equilibrium) when playing against the best responses respectively:
[Equation]
Deﬁnition 1 ( (cid:15) -approximate
Nash equilibrium) .
A pair of general policies (ˆ µ, ˆ ν ) is an (cid:15) - approximate Nash equilibrium , if V † , ˆ ν 1 ( s 1 ) − V ˆ µ, † 1 ( s 1 ) ≤ (cid:15) .
Deﬁnition 2 (Regret) .
Let ( µ k , ν k ) denote the policies deployed by the algorithm in the k th episode.
After a total of K episodes, the regret is deﬁned as
[Equation]
One goal of reinforcement learning is to design algorithms for Markov games that can ﬁnd an (cid:15)
-approximate
Nash equilibrium using a number of episodes that is small in its dependency on
S, A, B, H as well as 1 /(cid:15) (PAC sample complexity bound).
An alternative goal is to design algorithms for Markov games that achieves regret that is sublinear in K , and polynomial in S, A, B, H (regret bound).
We remark that any sublinear regret algorithm can be directly converted to a polynomial-sample PAC algorithm via the standard online-to-batch conversion (see e.g., Jin et al.
O PTIMISTIC N ASH V ALUE I TERATION
In this section, we present our main algorithm—Optimistic Nash Value Iteration (Nash-VI), and provide its theoretical guarantee.
3. A LGORITHM DESCRIPTION
We describe our Nash-VI Algorithm 1.
In each episode, the algorithm can be decomposed into two parts.
• Line 3-13 (Optimistic planning from the estimated model): Performs value iteration with bonus using the empirical estimate of the transition ˆ P , and computes a new (joint) policy π which is “greedy” with respect to the estimated value functions;
• Line 16-19 (Play the policy and update the model estimate): Executes the policy π , collects samples, and updates the estimate of the transition
ˆ P .
At a high-level, this two-phase strategy is standard in the majority of model-based RL algorithms, and also underlies provably efﬁcient model-based algorithms such as UCBVI for single-agent (MDP) setting (Azar et al., 2017) and VI-ULCB for the two-player Markov game setting (Bai &
Jin, 2020).
However, VI-ULCB has two undesirable drawbacks: the sample complexity is not tight in any of H , S , and A, B dependency, and its computational complexity is PPAD-complete (a complexity class conjectured to be computationally hard (Daskalakis, 2013)).
As we elaborate in the following, our Nash-VI algorithm differs from VI-ULCB in a few important technical aspects, which allows it to signiﬁcantly improve the sample complexity over VI-ULCB, and ensures that our algorithm terminates in polynomial time.
Before digging into explanations of techniques, we remark that line 14-15 is only used for computing the output policies.
It chooses policy π out to be the policy in the episode with minimum gap ( V 1 −
V 1 )( s 1 ) .
Our ﬁnal output policies ( µ out , ν out ) are simply the marginal policies of π out .
That is, for all ( s, h ) ∈ S ×
[ H ] , µ out h ( ·| s ) :
= (cid:80) b ∈B
π out h ( · , b | s ) , and ν out h ( ·| s ) :
= (cid:80) a ∈A π out h ( a, ·| s ) .
3.1. O VERVIEW OF TECHNIQUES
Auxiliary bonus γ .
The major improvement over VI-ULCB (Bai & Jin, 2020) comes from the use of a different style of bonus term γ (line 8), in addition to the standard bonus β (line 7), in value iteration steps (line 9-10).
This is also the main technical contribution of our Nash-VI algorithm.
This auxiliary bonus γ is computed by applying the empirical transition matrix ˆ P h to the gap at the next step
V h
+
− V h
+1 ,
This is very different from standard bonus β , which is typically designed according to the concentration inequalities.
The main purpose of these value iteration steps (line 9-10) is to ensure that the estimated values Q h and Q h are with high probability the upper bound and the lower bound of the Q -value of the current policy when facing best responses (see Lemma 20 and 22 for more details) 3 .
To do so, prior work (Bai & Jin, 2020) only adds bonus β , which needs to be as large as ˜Θ( (cid:112) S/t ) .
In contrast, the inclusion of auxiliary bonus γ in our algorithm allows a much smaller choice for bonus β —which scales only as ˜ O
( (cid:112) 1 /t ) —while still maintaining valid conﬁdence bounds.
This technique alone brings down the sample complexity to ˜ O
( H 4 SAB/(cid:15) 2 ) , removing an entire S factor compared to VI-ULCB.
Furthermore, the coefﬁcient in γ is only c/H for some absolute constant c , which ensures that the introduction of error term γ would hurt the overall sample complexity only up to a constant factor.
Bernstein concentration.
Our Nash-VI allows two choices of the bonus function β =
B ONUS ( t, ˆ
[Equation]
Hoeffding
[Equation]
Bernstein
[Equation]
where ˆ σ 2 is the estimated variance, ι is the logarithmic factors and c is absolute constant.
The ˆ V in line 7 is the empirical variance operator deﬁned as (cid:98) V h V
= (cid:98) P
h V 2 − ( (cid:98) P h V ) 2 for any V ∈ [0 , H ] S .
The design of both bonuses stem from the Hoeffding and Bernstein concentration inequalities.
Further, the Bernstein bonus uses a sharper concentration, which saves an H factor in sample complexity compared to the Hoeffding bonus (similar to the single-agent setting (Azar et al., 2017)).
This further reduces the sample complexity to ˜ O
( H 3 SAB/(cid:15) 2 ) which matches the lower bound in all H, S, (cid:15) factors.
Coarse Correlated Equalibirum (CCE).
The prior algorithm VI-ULCB (Bai & Jin, 2020) computes the “greedy” policy with respect to the estimated value functions by directly computing the
Nash equilibrium for the Q -value at each step h .
However, since the algorithm maintains both the upper conﬁdence bound and lower conﬁdence bound of the Q -value, this leads to the requirement
1: Initialize: for any ( s, a, b, h ) , Q h ( s, a, b ) ← H , Q h ( s, a, b ) ← 0 , ∆ ← H , N h ( s, a, b ) ← 0 .
2: for episode k = 1 , . . .
, K do 3: for step
h
= H, H − 1 , . .
, 1 do 4: for ( s, a, b ) ∈ S ×
A × B do 5: t ← N h ( s, a, b ) .
6: if t > 0 then 7: β ← B ONUS ( t, (cid:98) V h
[( V h +1 + V h +1 ) / 2]( s, a, b )) .
8: γ ← ( c/H ) (cid:98) P h ( V h
+1 − V h
+1 )
( s, a, b ) .
Q h ( s, a, b ) ← min { ( r h + (cid:98) P
h V h +1 )
( s, a, b ) + γ + β, H } .
Q h ( s, a, b ) ← max { ( r h + (cid:98) P
h V h +1 )
( s, a, b ) − γ − β, 0 } .
11: for s ∈ S do 12: π h ( · , ·| s ) ← CCE ( Q h ( s, · , · ) , Q h ( s, · , · )) .
V h ( s ) ← ( D π h Q h )( s );
V h ( s ) ← ( D π h Q h )( s ) .
14: if ( V 1 − V 1 )( s 1 ) < ∆ then 15: ∆ ← ( V 1 − V 1 )( s 1 ) and π out ← π .
16: for step h = 1 , . . .
, H do 17: take action ( a h , b h ) ∼ π h ( · , ·| s h ) , observe reward r h and next state s to compute the Nash equilibrium for a two-player general-sum matrix game, which is in general
PPAD-complete (Daskalakis, 2013).
To overcome this computational challenge, we compute a relaxation of the Nash equilibrium—
Coarse Correlated Equalibirum (CCE) —instead, a technique ﬁrst introduced by Xie et al. (2020) to address reinforcement learning problems in Markov Games.
Formally, for any pair of matrices Q, Q ∈ [0 , H ] A × B , CCE ( Q, Q ) returns a distribution π ∈ ∆ A×B such that
[Equation]
Intuitively, in a CCE the players choose their actions in a potentially correlated way such that no one can beneﬁt from unilateral unconditional deviation.
A CCE always exists, since Nash equilibrium is also a CCE and a Nash equilibrium always exists.
Furthermore, a CCE can be computed by linear programming in polynomial time.
We remark that different from Nash equilibrium where the policies of each player are independent, the policies given by CCE are in general correlated for each player.
Therefore, executing such a policy (line 17) requires the cooperation of two players.
3. T HEORETICAL GUARANTEES
Now we are ready to present the theoretical guarantees for Algorithm 1.
We let π k denote the policy computed in line 12 in the k th episode, and µ k , ν k denote the marginal policy of π k for each player.
Theorem 3 (Nash-VI with Hoeffding bonus) .
For any p ∈ (0 , 1] , letting ι = log( SABT/p ) , then with probability at least 1 − p , Algorithm 1 with Hoeffding type bonus (3) (with some absolute c > 0 ) achieves:
• ( V † ,ν out− V µ out , †)( s 1 ) ≤ (cid:15) , if the number of episodes K ≥ Ω( H 4 SABι/(cid:15) 2 + H 3 S 2 ABι 2 /(cid:15) ) .
• Regret( K ) =
(cid:80) K k
=1 ( V † ,ν k− V µ k , †)( s 1 ) ≤ O ( √ H 3 SABTι + H 3 S 2 ABι 2 ) .
Theorem 3 provides both a sample complexity bound and a regret bound for Nash-VI to ﬁnd an (cid:15)
-approximate
Nash equilibrium.
For small (cid:15) ≤ H/ ( Sι ) , the sample complexity scales as ˜ O
( H 4 SAB/(cid:15) 2 ) .
Similarly, for large T ≥ H 3 S 3 ABι 3 , the regret scales as ˜ O ( √ H 3 SABT ) , where
T = KH is the total number of steps played within K episodes.
Theorem 3 is signiﬁcant in that it improves the sample complexity of the model-based algorithm in Markov games from S 2 to S (and the regret from S to √ S ).
This is achieved by adding the new auxiliary bonus γ in value iteration steps as explained in Section 3.1.
The proof of Theorem 3 can be found in Appendix F.1.
Our next theorem states that when using Bernstein bonus instead of Hoeffding bonus as in (3), the sample complexity of Nash-VI algorithm can be further improved by a H factor in the leading order term (and the regret improved by a √ H factor).
Theorem 4 (Nash-VI with the Bernstein bonus) .
For any p ∈ (0 , 1] , letting ι = log( SABT/p ) , then with probability at least 1 − p , Algorithm 1 with Bernstein type bonus (3) (with some absolute c > 0 ) achieves:
• ( V † ,ν out− V µ out , †)( s 1 ) ≤ (cid:15) , if the number of episodes K ≥ Ω( H 3 SABι/(cid:15) 2 + H 3 S 2 ABι 2 /(cid:15) ) .
• Regret( K ) =
(cid:80)
Kk =1 ( V † ,ν k [Equation] µ k , †)( [Equation]
[Equation]
[Equation] Compared with the information-theoretic sample complexity lower bound Ω( H 3 S ( A + B ) ι/ (cid:15) and regret lower bound Ω( (cid:112)
H 2 S ( A + B ) T ) (Bai & Jin, 2020), when (cid:15) is small, Nash-VI with
Bernstein bonus achieves the optimal dependency on all of H, S, (cid:15) up to logarithmic factors in both the sample complexity and the regret, and the only gap that remains open is a AB/ ( A + B ) ≤ min { A, B } factor.
The proof of Theorem 4 can be found in Appendix F.2.
Comparison with model-free approaches.
Different from our model-based approach, a recently proposed model-free algorithm Nash V-Learning (Bai et al., 2020) achieves sample complexity ˜
O ( H 6 S ( A + B ) ι/(cid:15) 2 ) , which has a tight ( A + B ) dependency on A, B .
However, our Nash-VI has the following important advantages over Nash V-Learning: 1.
Our sample complexity has a better dependency on horizon H ; 2.
Our algorithm outputs a single pair of Markov policies ( µ out , ν out ) while their algorithm outputs a generic history-dependent policy that can be only written as a nested mixture of Markov policies; 3.
The model-free algorithms in Bai et al.
(2020) cannot be directly modiﬁed to obtain a √ T -regret (so that the exploration policies can be arbitrarily poor), while our model-based algorithm has the √ T -regret guarantee. We comment that although both Nash-VI and
Nash V-Learning have polynomial running time, the later enjoys a better computational complexity because Nash-VI requires to solve LPs for computing CCEs in each episode.
R EWARD - FREE L EARNING
In this section, we modify our model-based algorithm Nash-VI for the reward-free exploration setting (Jin et al., 2020b), which is also known as the task-agnostic (Zhang et al., 2020b) or reward-agnostic setting.
Reward-free learning has two phases:
In the exploration phase, the agent ﬁrst collects a dataset of transitions D = { ( s k,h , a k,h , b k,h , s k,h +1 ) } ( k,h ) ∈ [ K ] × [ H ] from a Markov game M without the guidance of reward information.
After the exploration, in the planning phase, for each task i ∈ [ N ] , D is augmented with stochastic reward information to become D i = ( s k,h , a k,h , b k,h , s k,h +1 , r k,h ) } ( k,h ) ∈
[ K ] × [ H ] , where r k,h is sampled from an unknown reward distribution with expectation equal to r ih ( s k,h , a k,h , b k,h ) . Here, we use r i to refer to the unknown reward function of the i th task. The goal is to compute nearly-optimal policies for N tasks under M , given the augmented datasets.
There are strong practical motivations for considering the reward-free setting.
First, in applications such as robotics, we face multiple tasks in sequential systems with shared transition dynamics (i.e. the world) but very different rewards.
There, we prefer to learn the underlying transition independent of reward information.
Second, from the algorithm design perspective, decoupling exploration and planning (i.e. performing exploration without reward information) can be valuable for designing new algorithms in more challenging settings (e.g. with function approximation).
Due to space limits, we defer the description of our algorithm Optimistic Value Iteration with Zero
Reward (VI-Zero, Algorithm 2) to Appendix B and only state its theoretical guarantees here.
The following theorem claims that the empirical transition (cid:98) P out outputted by VI-Zero is close to the true transition P , in the sense that any Nash equilibrium of the M ( (cid:98) P , (cid:98) r i ) ( i ∈ [ N ] ) is also an approximate Nash equilibrium of the true underlying Markov game M ( P , r i ) , where (cid:98) r i is the empirical estimate of r i computed using D i .
Theorem 5 (Sample complexity of VI-Zero) .
There exists an absolute constant c , for any p ∈ (0 , 1] , (cid:15) ∈ (0 , H ] , N ∈ N , if we choose bonus β t =
c ( (cid:112) H 2 ι/t
+ H
Sι/t ) with ι = log( NSABT/p ) and K ≥ c ( H 4 SABι/(cid:15) 2 + H 3 S 2 ABι 2 /(cid:15) ) , then with probability at least 1 − p , the output (cid:98) P out of Algorithm 2 has the following property: for any N ﬁxed reward functions r 1 , . . .
, r N , a Nash equilibrium of Markov game M ( (cid:98) P out , (cid:98) r i ) is also an (cid:15) -approximate
Nash equilibrium of the true
Markov game M ( P , r i ) for all i ∈ [ N ] .
Theorem 5 shows that, when (cid:15) is small, VI-Zero only needs ˜ O
( H 4 SAB/(cid:15) 2 ) samples to learn an estimate of the transition (cid:98) P out , which is accurate enough to learn the approximate Nash equilibrium for any N ﬁxed rewards.
The most important advantage of reward-free learning comes from the sample complexity only scaling polylogarithmically with respect to the number of tasks or reward functions N .
This is in sharp contrast to the reward-aware algorithms (e.g. Nash-VI), where the algorithm has to be rerun for each different task, and the total sample complexity must scale linearly in N .
In exchange for this beneﬁt, compared to Nash-VI, VI-Zero loses a factor of H in the leading term of sample complexity since we cannot use Bernstein bonus anymore due to the lack of reward information.
VI-Zero also does not have a regret guarantee, since again without reward information, the exploration policies are naturally sub-optimal.
The proof of Theorem 5 can be found in Appendix G.1.
Connections with reward-free learning in MDPs.
Since MDPs are special cases of Markov games, our algorithm VI-Zero directly applies to the single-agent setting, and yields a sample complexity similar to existing results (Zhang et al., 2020b; Wang et al., 2020).
However, distinct from existing results which require both the exploration algorithm and the planning algorithm to be specially designed to work together, our algorithm allows an arbitrary planning algorithm as long as it computes the Nash equilibrium of a Markov game with known transition and reward.
Therefore, our results completely decouple the exploration and the planning.
Lower bound for reward-free learning.
Finally, we comment that despite the sample complexity in Theorem 5 scaling as AB instead of A + B , our next theorem states that unlike the general rewardaware setting, this AB scaling is unavoidable in the reward-free setting.
This reveals an intrinsic gap between the reward-free and reward-aware learning: An A + B dependency is only achievable via sampling schemes that are reward-aware.
A similar lower bound is also presented in recent work (Zhang et al., 2020a) for the discounted setting with a different hard instance construction.
Theorem 6 (Lower bound for reward-free learning of Markov games) .
There exists an absolute constant c > 0 such that for any (cid:15) ∈ (0 , c ] , there exists a family of Markov games M ( (cid:15) ) satisfying that: for any reward-free algorithm
A using K ≤ cH 2 SAB/(cid:15) 2 episodes, there exists a Markov game M ∈ M ( (cid:15) )
such that if we run A on M and output policies (ˆ µ, ˆ ν ) , then with probability at least 1 / 4 , we have ( V † , ˆ ν− V ˆ µ, † 1 )( s 1 ) ≥ (cid:15) .
This lower bound shows that the sample complexity in Theorem 5 is optimal in S , A, B , and (cid:15) .
The proof of Theorem 6 can be found in Appendix G.3.
M ULTI - PLAYER GENERAL - SUM GAMES
We adapt our analysis to multi-player general-sum games and present the ﬁrst lines of provably efﬁcient algorithms.
Concretely, we design two model-based algorithms Multi-Nash-VI and MultiVI-Zero (Algorithm 3 and Algorithm 4) that can ﬁnd an ( (cid:15) -approximate) { N ASH , CE, CCE } equilibrium for any multi-player general-sum Markov game in ˜ O
( H 4 S 2 (cid:81) mi =1 A i /(cid:15) 2 ) episodes of game playing, where A i is the number of actions for player i ∈ { 1 , . . .
, m } (Theorem 15 and Theorem 16).
Due to space limit, we defer the detailed setups, algorithms and results to Appendix C.
C ONCLUSION
In this paper we provided a sharp analysis of model-based algorithms for Markov games.
Our new algorithm Nash-VI can ﬁnd an (cid:15) -approximate
Nash equilibrium of a zero-sum Markov game in ˜ O
( H 3 SAB/(cid:15) 2 ) episodes of game playing, which almost matches the sample complexity lower bound except for the AB vs. A + B dependency.
We also applied our analysis to derive new efﬁcient algorithms for task-agnostic game playing, as well as the ﬁrst line of multi-player generalsum Markov games.
There are a number of compelling future directions to this work.
For example, can we achieve A + B instead of AB sample complexity for zero-sum games using model-based approaches (thus closing the gap between lower and upper bounds)?
How can we design more efﬁcient algorithms for general-sum games with better sample complexity (e g., O ( S ) instead of O ( S 2 ) )?
We leave these problems as future work
A R ELATED W ORK
Markov games.
Markov games (or stochastic games) are proposed in the early 1950s (Shapley, 1953).
They are widely used to model multi-agent RL.
Learning the Nash equilibria of Markov games has been studied in Littman (1994; 2001);
Hu & Wellman (2003);
Hansen et al.
; Lee et al.
(2020), where the transition matrix and reward are assumed to be known, or in the asymptotic setting where the number of data goes to inﬁnity.
These results do not directly apply to the nonasymptotic setting where the transition and reward are unknown and only a limited amount of data are available for estimating them.
Another line of work assumes certain strong reachability assumptions under which sophisticated exploration strategies are not required.
A prevalent approach is to assume access to simulators (generative models) that enable the agent to directly sample transition and reward information for any state-action pair.
In this setting, Jia et al.
; Sidford et al.
; Zhang et al.
(2020a) provide non-asymptotic bounds on the number of calls to the simulator for ﬁnding an (cid:15) approximate Nash equilibrium.
Wei et al.
(2017) studies Markov games under an alternative assumption that no matter what strategy one agent sticks to, the other agent can always reach all states by playing a certain policy.
Non-asymptotic guarantees without reachability assumptions.
The recent work of Bai & Jin (2020); Xie et al.
(2020) provide the ﬁrst line of non-asymptotic sample complexity guarantees on learning Markov games without these reachability assumptions, in which exploration is essential.
However, both results suffer from highly suboptimal sample complexity.
The results of Xie et al.
(2020) also apply to the linear function approximation setting.
More recently, two model-free algorithms—Nash Q-Learning and Nash V-Learning—are shown to achieve better sample complexity guarantees (Bai et al., 2020).
In particular, the Nash V-learning algorithm achieves the nearoptimal dependence on S , A and B .
However, the dependence on H is worse than our results and the output policy is a nested mixture, which is hard to implement.
We compare our results with existing non-asymptotic guarantees in Table 1.
We remark that the classical R-max algorithm (Brafman & Tennenholtz, 2002) also provides provable guarantees for learning Markov games.
However, Brafman & Tennenholtz (2002) uses a weaker deﬁnition of regret (similar to the online setting in Xie et al.
and consequently their result does not imply any sample complexity result for ﬁnding Nash equilibrium policies.
Adversarial MDPs.
Another way to model the multi-player bahavior is to use adversarial MDPs .
Most work in this line considers the setting with adversarial rewards (Zimin & Neu, 2013; Rosenberg & Mansour, 2019; Jin et al., 2019), where the reward can be manipulated by an adversary arbitrarily and the goal is to compete with the optimal (stationary) policy in hindsight.
Adversarial MDP with changing dynamics is computationally hard even under full-information feedback (Yadkori et al., 2013).
Notice these results do not directly imply provable self-play algorithms in our setting, because the opponent in Markov games can affect both the reward and the transition.
Single-agent RL.
There is a rich literature on reinforcement learning in MDPs (see e.g., Jaksch et al., 2010; Osband et al., 2014; Azar et al., 2017; Dann et al., 2017; Strehl et al., 2006; Jin et al., 2018).
MDP is a special case of Markov games, where only a single agent interacts with a stochastic environment.
For the tabular episodic setting with nonstationary dynamics and no simulators, the best sample complexity is ˜ O
( H 3 SA/(cid:15) 2 ) , achieved by model-based algorithm in Azar et al.
(2017) and model-free algorithms in Zhang et al.
(2020c)
, respectively, where S is the number of states,
A is the number of actions, H is the length of each episode.
Both of them match the lower bound Ω( H 3 SA/(cid:15) 2 ) (Jaksch et al., 2010; Osband & Van Roy, 2016; Jin et al., 2018).
Reward-free and task-agnostic exploration.
Jin et al.
(2020a) proposes a new paradigm of learning an MDP, which they called reward-free exploration.
In this setting, the agent goes through a two-stage process.
In the exploration phase the agent can interacts with the environment without knowing the reward function and in the planning phase the reward function is given and the agent needs output a policy.
The goal is to make the output policy near optimal for any given reward function.
A closely related setting is task-agnostic learning, where the reward function is determined at the very beginning but not revealed until the planning phase.
Notice algorithms for task-agnostic
Require: Bonus β t .
then 7: do 9: π
V
12: ∆ do 14:
15: add
17: Output learning can also be transferred to reward-free exploration by taking union bound w.r.t.
different possible reward function.
In Table 1, VI-explore (Bai & Jin, 2020) and Algorithm 2 can also be applied to this setting.
Jin et al.
(2020a) also proposes an algorithm, which ﬁrst ﬁnds a covering policy to maximize the probability to reach each state separately and then collects data following this policy.
Zhang et al.
(2020b) takes a different approach by ﬁrst runs the optimistic Q-learning algorithm (Jin et al., 2018) with zero reward to explore the environment, and then they utilizes the trajectories collected to compute a policy in an incremental manner.
Wang et al.
(2020) follows a simialr scheme, but studies reward-free exploration in linear-parametrized MDPs.
B O PTIMISTIC V ALUE I TERATION WITH Z ERO R EWARD – VI-Z ERO
We now describe our algorithm for reward-free learning in zero-sum Markov games.
Exploration phase.
In the ﬁrst phase of reward-free learning, we deploy algorithm Optimistic
Value Iteration with Zero Reward (VI-Zero, Algorithm 2).
This algorithm differs from the rewardaware Nash-VI (Algorithm 1) in two important aspects.
First, we use zero reward in the exploration phase (Line 7), and only maintains an upper bound of the (reward-free) value function instead of both upper and lower bounds.
Second, our exploration policy is the maximizing (instead of CCE) policy of the value function (Line 9).
We remark that the (cid:101) Q h ( s, a, b ) maintained in the algorithmis no longer an upper bound for any actual value function (as it has no reward), but rather a measure of uncertainty or suboptimality that the agent may suffer—if she takes action ( a, b ) at state s and step h , and makes decisions by utilizing the empirical estimate (cid:98) P in the remaining steps (see a rigorous version of this statement in Lemma 27).
Finally, the empirical transition (cid:98) P of the episode that minimizes (cid:101) V 1 ( s 1 ) is outputted and passed to the planning phase.
Planning phase.
After obtaining the estimate of tranisiton (cid:98) P , our planning algorithm is rather simple.
For the i th task, let (cid:98) r i be the empirical estimate of r i computed using the i th augmented dataset D i .
Then we compute the Nash equilibrium of the Markov game M ( (cid:98) P , (cid:98) r i ) with estimated transition (cid:98) P and reward (cid:98) r i .
Since both (cid:98) P and (cid:98) r i are known exactly, this is a pure computation problem without any sampling error and can be efﬁciently solved by simple planning algorithms such as the vanilla Nash value iteration without optimism (see Appendix G.2 for more details).
C M ULTIPLAYER G ENERAL - SUM M ARKOV G AMES
In this section, we extend both our model-based algorithms (Algorithm 1 and Algorithm 2) to the setting of multiplayer general-sum Markov games, and present corresponding theoretical guarantees.
C. P ROBLEM FORMULATION
A general-sum
Markov game (general-sum
MG) with m players is a tuple
MG( H, S , {A i } mi =1 , P , { r i } mi =1 ) , where H , S denote the length of each episode and the state space.
Different from the two-player zero-sum setting, we now have m different action spaces, where A i is the action space for the i th player and |A
i | =
A i .
We let a :=
( a 1 , · · · , a m ) denote the (tuple of) joint actions by all m players.
P =
{ P h } h ∈ [ H ] is a collection of transition matrices, so that P h ( ·| s, a ) gives the distribution of the next state if actions a are taken at state s at step h , and r i = { r h,i } h ∈
[ H ] is a collection of reward functions for i th player, so that r h,i ( s, a ) gives the reward received by the i th player if actions a are taken at state s at step h .
In this section, we consider three versions of equlibrium for general-sum MGs: Nash equilibrium (NE), correlated equilibrium (CE), and coarse correlated equilibrium (CCE), all being standard solution notions in games (Nisan et al., 2007).
These three notions coincide on two-player zero-sum games, but are not equivalent to each other on multi-player general-sum games; any one of them could be desired depending on the application at hand.
Below we introduce their deﬁnitions.
(Approximate)
Nash equilibrium in general-sum MG.
The policy of the i th player is denoted as π i :=
(cid:8)
[Equation] [Equation]
: S
[Equation]
(cid:9) h ∈ [ H ] .
We denote the product policy of all the players as π := π 1 × · · · × π M , and denote the policy of the all the players except the i th player as π − i .
We deﬁne
V πh,i ( s ) as the expected cumulative reward that will be received by the i th player if starting at state s at step h and all players follow policy π .
For any strategy π − i , there also exists a best response of the i th player, which is a policy µ † ( π − i ) satisfying V µ † ( π − i ) ,π − i h,
i
[Equation]
i ( s ) for any ( s, h ) ∈ S ×
[ H ] .
We denote V † ,π − i
h,
i
[Equation]
The Q-functions of the best response can be deﬁned similarly.
Our ﬁrst objective is to ﬁnd an approximate Nash equilibrium of Markov games.
Deﬁnition 7 ( (cid:15) -approximate
Nash equilibrium in general-sum MG) .
A product policy π is an (cid:15) - approximate Nash equilibrium if max
[Equation]
The above deﬁnition requires the suboptimality gap ( V † ,π − i 1 ,i − V π 1 ,i )( s 1 ) to be less than (cid:15) for all player i .
This is consistent with the two-player case (Deﬁnition 1) up to a constant of 2, since in the two-player zero-sum setting, we have V π 1 , 1 ( s 1 ) =
− V π 1 , 2 ( s 1 ) for any product policy π = ( µ, ν ) , and therefore ( V † ,ν 1 , 1 − V µ, † 1 , 1 )( s 1 ) ≤ 2 max i ∈ [2] ( V † ,π − i 1 ,i − V π 1 ,i )( s 1 ) ≤ 2( V † ,ν 1 , 1 − V µ, † 1 , 1 )( s 1 ) .We can similarly deﬁne the regret.
Deﬁnition 8 (Nash-regret in general-sum MG) .
Let π k denote the (product) policy deployed by the algorithm in the k th episode.
After a total of K episodes, the regret is deﬁned as
[Equation]
(Approximate)
CCE in general-sum MG.
The coarse correlated equilibrium (CCE) is a relaxed version of Nash equilibrium in which we consider general correlated policies instead of product policies.
Let A =
A 1 × · · · × A m denote the joint action space.
Deﬁnition 9 (CCE in in general-sum MG) .
A (correlated) policy π := { π h ( s ) ∈
∆ A : ( h, s ) ∈ [ H ] × S} is a CCE if max i ∈
[ m ] V † ,π − i h,i ( s
[Equation]
≤ V πh,i ( s ) for all ( s, h ) ∈ S ×
[ H ] .
Compared with a Nash equilibrium, a CEE is not necessarily a product policy, that is, we may not have π h ( s ) ∈
∆ A 1 × · · · × ∆ A m .
Similarly, we also deﬁne (cid:15) -approximate
CCE and CCE-regret below.
Deﬁnition 10 ( (cid:15) -approximate
CCE in general-sum MG) .
A policy π := { π h ( s ) ∈
∆ A : ( h, s ) ∈ [ H ] × S} is an (cid:15) -approximate
CCE if max i ∈
[ m ] ( V † ,π − i 1 ,
[Equation]
Deﬁnition 11 (CCE-regret in general-sum MG) .
Let policy π k denote the (correlated) policy deployed by the algorithm in the k th episode.
After a total of K episodes, the regret is deﬁned as
[Equation]
(Approximate) CE in general-sum MG.
The correlated equilibrium (CE) is another relaxation of the Nash equilibrium.
To deﬁne CE, we ﬁrst introduce the concept of strategy modiﬁcation: A strategy modiﬁcation φ := { φ h,s (
a ) ∈
A i : ( h, s, a ) ∈
[ H ] ×S ×A i } for player i is a set of S
× H injective functions from A i to itself.
Let Φ i denote the set of all possible strategy modiﬁcations for player i .
One can compose a strategy modiﬁcation φ with any Markov policy π and obtain a new policy φ (cid:5) π such that when policy π chooses to play a :=
( a 1 , . .
. , a m ) at state s and step h , policy φ (cid:5) π will play ( a 1 , . . .
, a i − 1 , φ h,s ( a i ) , a i +1 , . .
. , a m ) instead.
Deﬁnition 12 (CE in general-sum MG) .
A policy π := { π h ( s ) ∈
∆ A : ( h, s ) ∈ [ H ] × S} is a CE if
[Equation]
π h,i ( s ) ≤ V πh,i ( s ) holds for all ( s, h ) ∈ S ×
[ H ] .
Similarly, we have an approximate version of CE and CE-regret.
Deﬁnition 13 ( (cid:15) -approximate CE in Markov games) .
A policy π := { π h ( s ) ∈
∆ A : ( h, s ) ∈ [ H ] × S} is an (cid:15) -approximate CE
if max i ∈ [ m ] max φ ∈
Φ i ( V φ (cid:5) π 1 ,i − V π 1 ,i )( s 1 ) ≤ (cid:15) .
Deﬁnition 14 (CE-regret in multiplayer Markov games) .
Let product policy π k denote the policy deployed by the algorithm in the k th episode.
After a total of K episodes, the regret is deﬁned as
[Equation]
Relationship between Nash, CE, and CCE For general-sum MGs, we have { Nash } ⊆ { CE } ⊆ { CCE } , so that they form a nested set of notions of equilibria (Nisan et al., 2007).
Indeed, one can easily verify that if we restrict the choice of strategy modiﬁcation φ to those consisting of only constant functions, i.e., φ h,s ( a ) being independent of a , Deﬁnition 12 will reduce to the deﬁnition of CCE policy.
In addition, any Nash equilibrium is a CE by deﬁnition.
Finally, since a Nash equilibrium always exists, so does CE and CCE.
C. M ULTIPLAYER OPTIMISTIC N ASH VALUE ITERATION
Here we present the Multi-Nash-VI algorithm, which is an extension of Algorithm 1 for multi-player general-sum Markov games.
The E QUILIBRIUM Subroutine.
Our E QUILIBRIUM subroutine in Line 11 could be taken from either one of the { N ASH , CE , CCE } subroutines for one-step games.
When using N ASH , we compute the Nash equilibrium of a one-step multi-player game (see, e.g., Berg & Sandholm (2016) for an overview of the available algorithms); the worst-case computational complexity of such a subroutine will be PPAD-hard (Daskalakis, 2013).
When using CE or CCE, we ﬁnd CEs or CCEs of the one-step games respectively, which can be solved in polynomial time using linear programming.
However, the policies found are not guaranteed to be a product policy.
We remark that in
Algorithm 1 we used the CCE subroutine for ﬁnding Nash in two-player zero-sum games, which seemingly contrasts the principle of using the right subroutine for ﬁnding the right equilibrium, but nevertheless works as the Nash equilibrium and CCE are equivalent in zero-sum games.
Now we are ready to present the theoretical guarantees for Algorithm 3.
We let π k denote the policy computed in line 11 of Algorithm 3 in the k th episode.
Theorem 15 (Multi-Nash-VI) .
There exists an absolute constant c , for any p ∈ (0 , 1] , let ι = log( SABT/p ) , then with probability at least 1 − p , Algorithm 3 with bonus β t =
c (cid:112) SH 2 ι/t and
E QUILIBRIUM being one of { N ASH , CE , CCE } satisﬁes (repsectively):
1: Initialize: for any ( s, a , h, i ) , Q h,i ( s, a ) ← H , Q h,i ( s, a ) ← 0 , ∆ ← H , N h ( s, a ) ← 0 .
2: for episode k = 1 , . . .
, K do 3: for step
h
= H, H − 1 , . .
, 1 do 4: for ( s, a ) ∈ S ×
A 1 × · · · × A m do 5: t ← N h ( s, a ) ; 6: if t > 0 then 7: for player i = 1 , 2 , . . .
, m do 8:
Q h,i ( s, a ) ← min { ( r h,i + (cid:98) P h V h
+1 ,i )( s, a ) + β t , H } .
Q h,i ( s, a ) ← max { ( r h,i + (cid:98) P h V h
+1 ,i )( s, a ) − β t , 0 } .
10: for s ∈ S do 11: π h ( ·| s ) ← E QUILIBRIUM ( Q h, 1 ( s, · ) , Q h, 2 ( s, · ) , · · · , Q h,M ( s, · )) .
12: for player i = 1 , 2 , . . .
, m do 13:
V h,i ( s ) ← ( D π h Q h,i )( s ) ; V h,i ( s ) ← ( D π h Q h,i )( s ) .
14: if max i ∈ [ m ] ( V 1 ,i − V 1 ,i )( s 1 ) <
∆ then 15: ∆ ← max i ∈ [ m ] ( V 1 ,i − V 1 ,i )( s 1 ) and π out ← π .
16: for step h = 1 , . . .
, H do 17: take action a h ∼ π h ( ·| s h ) , observe reward r h and next state s h +1 . 18: add 1 to N h ( s h , a h ) and N h ( s h , a h , s h +1 ) .
19: (cid:98) P h ( ·| s h , a h ) ← N h
( s h , a h , · ) /N
h ( s h , a h ) .
20: Output π out .
• π out is an (cid:15) -approximate { N ASH , CE , CCE } , if the number of episodes K ≥ Ω( H 4 S 2 ( (cid:81) mi =1 A i ) ι/(cid:15) 2 ) .
• Regret { Nash , CE , CCE } ( K ) ≤ O ( (cid:112) H 3 S 2 ( (cid:81) mi
=1 A i ) Tι ) .
In the situation where the E QUILIBRIUM subroutine is taken as N ASH , Theorem 15 provides the sample complexity bound of Multi-Nash-VI algorithm to ﬁnd a (cid:15) -approximate
Nash equilibrium and its regret bound.
Compared with our earlier result in two-player zero-sum games (Theorem 3), here the sample complexity scales as S 2 H 4 instead of SH 3 .
This is because the auxiliary bonus and Bernstein concentration technique do not apply here.
Furthermore, the sample complexity is proportional to (cid:81) mi =1 A i , which increases exponentially as the number of players increases.
Runtime of Algorithm
We remark that while the Nash guarantee is the strongest among the three guarantees presented in Theorem 15, the runtime of Algorithm 3 in the Nash case is not guaranteed to be polynomial and in the worst case PPAD-hard (due to the hardness of the N ASH subroutine).
In contrast, the CE and CCE guarantees are weaker, but the corresponding algorithms are guaranteed to ﬁnish in polynomial time.
C. M ULTIPLAYER REWARD - FREE LEARNING
We can also generalize VI-Zero to the multiplayer setting and obtain Algorithm 4, Multi-VI-Zero, which is almost the same as VI-Zero except that its exploration bonus β t is larger than that of VIZero by a √ S factor.
Similar to Theorem 5, we have the following theoretical guarantee claiming that any { N ASH ,CCE,CE } of the M ( (cid:98) P , (cid:98) r i ) ( i ∈ [ N ] ) is also an approximate { N ASH ,CCE,CE } of the true
Markov game M ( P , r i ) , where (cid:98) P out is the empirical transition outputted by Algorithm 4 and (cid:98) r i is the empirical estimate of r i .
Theorem 16 (Multi-VI-Zero) .
There exists an absolute constant c , for any p ∈ (0 , 1] , (cid:15) ∈ (0 , H ] , N ∈ N , if we choose bonus β t =
c (cid:112) H 2 Sι/t with ι = log( NSABT/p ) and K ≥ c ( H 4 S 2 ( (cid:81) mi =1 A i ) ι/(cid:15) 2 )
, then with probability at least 1 − p , the output (cid:98) P out of Algorithm 4 has the following property: for any N ﬁxed reward functions r 1 , . . .
, r N , any { N ASH , CCE , CE } of
1: Initialize: for any ( s, a , h ) , (cid:101)
V h ( s, a ) ← H , ∆ ← H , N h ( s, a ) ← 0 .
2: for episode k = 1 , . . .
, K do 3: for step
h
= H, H − 1 , . .
, 1 do 4: for ( s, a ) ∈ S ×
A 1 × · · · × A m do 5: t ← N h ( s, a ) .
6: if t > 0 then 7: (cid:101) Q h ( s, a ) ← min { ( (cid:98) P h (cid:101) V h +1 )
( s, a ) + β t , H } .
8: for s ∈ S do 9: π h ( s ) ← arg max a ∈A 1 ×···×A m (cid:101) Q h
( s, a ) .
10: (cid:101)
V h
( s ) ← ( D π h (cid:101) Q h )( s ) .
11: if (cid:101) V 1 ( s 1 ) <
∆ then 12: ∆ ← (cid:101) V 1 ( s 1 ) and (cid:98) P out ← (cid:98) P .
13: for step h = 1 , . . .
, H do 14: take action a h ∼ π h ( · , ·| s h ) , observe next state s h +1 .
15: add 1 to N h
( s h , a h ) and N h ( s h , a h , s h +1 ) .
16: (cid:98) P h ( ·| s h , a h ) ← N h
( s h , a h , · ) /N
h ( s h , a h ) .
17: Output (cid:98) P out .
Markov game M ( (cid:98) P out , (cid:98) r i ) is also an (cid:15) -approximate { N ASH , CCE , CE } of the true Markov game
M ( P , r i ) for all i ∈ [ N ] .
The proof of Theorem 16 can be found in Appendix H.2.
It is worth mentioning that the empirical
Markov game M ( (cid:98) P out , (cid:98) r i ) may have multiple { Nash equilibria,CCEs,CEs } and Theorem 16 ensures that all of them are (cid:15)
-approximate
{ Nash equilibria,CCEs,CEs } of the true Markov game.
Also, note that the sample complexity here is quadratic in the number of states because we are using the exploration bonus β t = (cid:112) H 2 Sι/t that is larger than usual by a √ S factor.
D B ELLMAN E QUATIONS FOR M ARKOV G AMES
In this section, we present the Bellman equations for different types of values in Markov games.
Fixed policies.
For any pair of Markov policy ( µ, ν ) , by deﬁnition of their values in (1) (2), we have the following Bellman equations:
[Equation]
for all ( s, a, b, h ) ∈ S ×
A × B × [ H ] , where V µ,νH +1 ( s ) = 0 for all s ∈ S .
Best responses.
For any Markov policy µ of the max-player, by deﬁnition, we have the following
Bellman equations for values of its best response:
[Equation]
for all ( s, a, b, h ) ∈ S ×
A × B × [ H ] , where V µ, † H +
( s )
=
for all s ∈ S .
Similarly, for any Markov policy ν of the min-player, we also have the following symmetric version of Bellman equations for values of its best response:
[Equation]
for all ( s, a, b, h ) ∈ S ×
A × B × [ H ] , where V † ,νH +1 ( s ) = 0 for all s ∈ S .
Nash equilibria.
Finally, by deﬁnition of Nash equilibria in Markov games, we have the following
Bellman optimality equations:
[Equation]
for all ( s, a, b, h ) ∈ S ×
A × B × [ H ] , where V (cid:63)H +1 ( s ) = 0 for all s ∈ S .
E P ROPERTIES OF C OARSE C ORRELATED E QUILIBRIUM
Recall the deﬁnition for CCE in our main paper (4), we restate it here after rescaling.
For any pair of matrices P, Q ∈ [0 , 1] n × m , the subroutine CCE ( P, Q ) returns a distribution π ∈ ∆ n × m that satisﬁes:
[Equation]
We make three remarks on CCE.
First, a CCE always exists since a Nash equilibrium for a generalsum game with payoff matrices ( P, Q ) is also a CCE deﬁned by ( P, Q ) , and a Nash equilibrium always exists.
Second, a CCE can be efﬁciently computed, since above constraints (5) for CCE can be rewritten as n + m linear constraints on π ∈ ∆ n × m , which can be efﬁciently resolved by standard linear programming algorithm.
Third, a CCE in general-sum games needs not to be a Nash equilibrium.
However, a CCE in zero-sum games is guaranteed to be a Nash equalibrium.
Proposition 17.
Let π = CCE ( Q, Q ) , and ( µ, ν ) be the marginal distribution over both players’ actions induced by π .
Then ( µ, ν ) is a Nash equilibrium for payoff matrix Q .
Proof of Proposition 17.
Let N (cid:63) be the value of Nash equilibrium for Q .
Since π = CCE ( Q, Q ) , by deﬁnition, we have:
[Equation]
This gives:
[Equation]
which ﬁnishes the proof.
Intuitively, a CCE procedure can be used in Nash Q-learning for ﬁnding an approximate Nash equilibrium, because the values of upper conﬁdence and lower conﬁdence ( Q and Q ) will be eventually very close, so that the preconditions of Proposition 17 becomes approximately satisﬁed.
F P ROOF FOR S ECTION 3 – O PTIMISTIC N ASH V ALUE I TERATION F. P ROOF OF T HEOREM
We denote V k , Q k , π k , µ k and ν k 4 for values and policies at the beginning of the k -th episode.
In particular, N kh ( s, a, b ) is the number we have visited the state-action tuple ( s, a, b ) at the h -th step before the k -th episode.
N kh ( s, a, b, s (cid:48) ) is deﬁned by the same token.
Using this notation, we can further deﬁne the empirical transition by (cid:98) P kh ( s (cid:48) | s, a, b ) :
= N kh ( s, a, b, s (cid:48) ) /N
kh ( s, a, b ) .
If
N kh ( s, a, b ) = 0 , we set (cid:98) P kh ( s (cid:48) | s, a, b )
= 1 /S .
As a result, the bonus terms can be written as
[Equation]
[Equation]
for some large absolute constant C > 0 .
Lemma 18. Let c 1 be some large absolute constant. Deﬁne event E 0 to be: for all h, s, a, b, s (cid:48) and k
[Equation]
[ K ] ,
[Equation]
We have P ( E 1 ) ≥ 1 − p .
Proof.
The proof is standard and folklore: apply standard concentration inequalities and then take a union bound.
For completeness, we provide the proof of the second one here.
Consider a ﬁxed ( s, a, b, h ) tuple.
Let’s consider the following equivalent random process: (a) before the agent starts, the environment samples { s (1) , s (2) , . . .
, s ( K ) } independently from P h ( · | s, a, b ) ; (b) during the interaction between the agent and environment, the i th time the agent reaches ( s, a, b, h ) , the environment will make the agent transit to s ( i ) .
Note that the randomness induced by this interaction procedure is exactly the same as the original one, which means the probability of any event in this context is the same as in the original problem.
Therefore, it sufﬁces to prove the target concentration inequality in this ’easy’ context.
Denote by (cid:98) P ( t ) h ( · | s, a, b ) the empirical estimate of P h ( · | s, a, b ) calculated using { s (1) , s (2) , . . .
, s ( t ) } .
For a ﬁxed t and s (cid:48) , by applying the Bernstein inequality and its empirical version, we have with probability at least 1 − p/S 2 ABT ,
[Equation]
Now we can take a union bound over all s, a, b, h, s (cid:48) and t ∈
[ K ] , and obtain that with probability at least 1 − p , for all s, a, b, h, s (cid:48) and t ∈
[ K ] ,
[Equation]
Note that the agent can reach each ( s, a, b, h ) for at most K times, this directly implies that the third inequality also holds with probability at least 1 − p .
We begin with an auxiliary lemma bounding the lower-order term.
Lemma 19.
Suppose event E 0 holds, then there exists absolute constant c 2 such that: if function g ( s ) satisﬁes | g | ( s ) ≤ ( V k h
+1 − V kh +1 )
( s ) for all s , then
[Equation]
≤ c
[Equation]
[Equation]
Proof.
By triangle inequality,
[Equation]
where ( i ) is by the second inequality in event E 0 and ( ii ) is by AM-GM inequality.
This proves the empirical version.
Similarly [Equation] we can show [Equation]
Combining the two bounds completes the proof.
Now we can prove the upper and lower bounds are indeed upper and lower bounds of the best reponses.
Lemma 20.
Suppose event E 0 holds. Then for all h, s, a, b and k ∈ [ K ] , we have
[Equation]
Proof.
The proof is by backward induction.
Suppose the bounds hold for the Q -values in the ( h + 1) th step, we now establish the bounds for the V -values in the ( h + 1) th step and Q -values in the h th -step.
For any state s :
[Equation]
Similarly, we can show V kh +1 ( s )
≤ V µ k , † h
+1 ( s ) .
Therefore, we have: for all s ,
[Equation]
Now consider an arbitrary triple ( s, a, b ) in the h th step.
We have
[Equation]
Invoking Lemma 19 with g = V † ,ν k h
+1 − V (cid:63)h +1 , [Equation] H
[Equation]
(cid:33) . By the ﬁrst inequality in event E 0 , | ( B ) | ≤ O (cid:32)(cid:115) H 2 ι
[Equation]
kh ( s, a, b ) , 1 } (cid:33) . Plugging the two inequalities above back into (10) and recalling the deﬁnition of β kh and γ kh , we obtain Q kh ( s, a, b ) ≥ Q † ,ν k h ( s, a, b ) .
Similarly, we can show Q kh ( s, a, b ) ≤ Q µ k , † h ( s, a, b ) .
Finally we come to the proof of Theorem 3.
Proof of Theorem 3.
Suppose event E 0 holds.
We ﬁrst upper bound the regret.
By Lemma 20, the regret can be upper bounded by
[Equation]
For brevity’s sake, we deﬁne the following notations:
[Equation]
Let F kh be the σ -ﬁeld generated by the following random variables: { ( s ji
[Equation]
a ji , b ji , r ji ) } ( i
[Equation]
j ) ∈
[ H
[Equation]
× [ k
[Equation]
[Equation]
(cid:91)
[Equation]
{ s kh } .
It’s easy to check ζ kh and ξ kh are martingale differences with respect to F kh .
With a slight abuse of notation, we use β kh to refer to β kh ( s kh , a kh , b kh ) and N kh to refer to N kh ( s kh , a kh , b kh ) in the following proof.
We have
[Equation]
where ( i ) and ( ii ) follow from Lemma 19.
Deﬁne c 3 := 1 + 2 c 2 C and κ := 1 + c 3 /H .
Recursing this argument for h ∈ [ H ] and summing over k ,
[Equation]
By Azuma-Hoeffding inequality, with probability at least 1 − p ,
[Equation]
By pigeon-hole argument,
[Equation]
HSABι [Equation] Put everything together, with probability at least 1 − 2 p (one p comes from P ( E 0 ) ≥ 1 − p and the other is for equation (12)), K
[Equation]
For the PAC guarantee, recall that we choose π out = π k (cid:63) such that k (cid:63) =
argmin k (cid:16) V k 1 − V k 1 (cid:17) ( s 1 ) .
As a result,
[Equation]
which concludes the proof.
F. P ROOF OF T HEOREM
We use the same notation as in Appendix F.1 except the form of bonus.
Besides, we deﬁne the empirical variance operator
[Equation]
and the true (population) variance operator
V h V ( s, a, b ) :=
Var s
[Equation]
[Equation]
[Equation]
[Equation] s [Equation] ) for any function V ∈ ∆ S .
If N kh ( s, a, b ) = 0 , we simply set (cid:98) V kh V ( s, a, b ) :
= H 2 regardless of the choice of V .
As a result, the bonus terms can be written as
[Equation]
[Equation]
Lemma 21.
Let c 1 be some large absolute constant.
Deﬁne event
E 1 to be: for all h, s, a, b, s (cid:48) and k ∈ [ K ] ,
[Equation]
We have P ( E 1 ) ≥ 1 − p .
The proof of Lemma 21 is highly similar to that of Lemma 18.
Speciﬁcally, the ﬁrst two can be proved by following basically the same argument in Lemma 18; the third one is standard (e.g., equation (12) in Azar et al.
We omit the proof here.
Since the proof of Lemma 19 does not depend on the form of the bonus, it can also be applied in this section.
As in Appendix F.1, we will prove the upper and lower bounds are indeed upper and lower bounds of the best reponses.
Lemma 22.
Suppose event E 1 holds. Then for all h, s, a, b and k ∈ [ K ] , we have
[Equation]
Proof.
The proof is by backward induction and very similar to that of Lemma 20.
Suppose the bounds hold for the Q -values in the ( h + 1) th step, we now establish the bounds for the V -values in the ( h + 1) th step and Q -values in the h th -step.
The proof for the V -values is the same as (9).
For the Q -values, the decomposition (10) still holds and ( A ) is bounded using Lemma 19 as before.
The only difference is that we need to bound ( B ) more carefully.
First, by the ﬁrst inequality in event E 1 ,
[Equation]
By the relation of V -values in the ( h + 1) th step, | [ [Equation] a [Equation]
) ≤
[Equation]
+
[Equation] [Equation] which implies
[Equation]
where ( i ) is by AM-GM inequality. Plugging the above inequalities back into (10) and recalling the deﬁnition of β kh and γ kh completes the proof.
We need one more lemma to control the error of the empirical variance estimator:
Lemma 23.
Suppose event E 1 holds. Then for all h, s, a, b and k ∈ [ K ] , we have | (cid:98) V kh
[Equation]
V [Equation] (cid:18) [Equation]
H
[Equation]
max { N kh ( s, a, b ) , 1 } (cid:19) .
Proof.
By Lemma 22, we have V kh ( s ) ≥ V π k h ( s ) ≥ V kh
( s ) .
As a result, | (cid:98) V kh [( V kh +1 + V kh +1 ) / − V h V π k h
{++
+}
|
( s, a, b ) =
{+| [ (cid:98) P kh ( V kh +
+}
+ V kh +1 ) 2 / 4 − P h
( V π k h +1 ) 2 ]( s, a, b ) − [( (cid:98) P kh
( V kh +1 + V kh +1 ))
{+2 / 4 − ( P h V π k h +1 ) 2 ]( s, a, b ) | ≤ [ (cid:98) P kh ( V kh +1 ) 2 − P h ( V kh +1 ) 2 − ( (cid:98) P kh V kh +1 ) 
+}
+ ( P h V kh +1 ) 2 ]( s, a, b ) ≤ [ | ( (cid:98) P kh − P h )( V kh +1 ) 2 | + | P h
[( V kh +1 ) 2 − ( V kh +1 ) 2 ] | + | ( (cid:98) P kh V k h +1 ) 2 − ( P h V k h +1 ) 2 | + | ( P h V k h +1 ) 2 − ( P h V k h +1 ) 2 | ]( s, a, b )
These terms can be bounded separately by using event E 1 :
[Equation]
) 2 | ( s, a, b ) ≤ H 2 (cid:107) ( (cid:98) P kh [Equation] P h )(
[Equation]
|
[Equation] a [Equation] ) (cid:107)
[Equation]
O
[Equation]
H(cid:115)
Sι
[Equation]
, | P h
{+[( V kh +1 ) 2 − ( V kh +1 ) 2 ] | ( s, a, b ) ≤ 2 H [ P h ( V kh +
 − V kh +1 )] ( s, a, b ) , |+}
( (cid:98) P kh V kh
[Equation]
[Equation]
− ( P h V kh
[Equation]
1 ) 2 | ( s, a [Equation] ) ≤ 2 H [( (cid:98) P [Equation]
P
[Equation]
[Equation]
a
[Equation]
( H(cid:115)
Sι
[Equation]
| ( P
[Equation]
V k h +
[Equation]
[Equation]
− ( P h V k h +
[Equation]
[Equation]
| (
[Equation]
a
[Equation]
) ≤
[Equation]
H [ P h ( V k h
+1 − V k h +1 )]
( s, a, b ) .
Combining with H 2 (cid:113)
Sι max { N
[Equation]
( s,a,b ) , 1 } ≤ 1 +
H 4 Sι max { N kh ( s,a,b ) , 1 } completes the proof.
Finally we come to the proof of Theorem 4.
Proof of Theorem 4.
Suppose event E 1 holds.
We deﬁne ∆ kh , ζ kh abd ξ kh as in the proof of Theorem 3.
As before we have
[Equation]
By Lemma 23,
[Equation]
where c 4 is some absolute constant. Deﬁne c 5 :=
4 c 2 c 4 C + c 3 and κ := 1 + c 5 /H .
Plugging (18) back into (17), we have
[Equation]
Recursing this argument for h ∈ [ H ] and summing over k , K
[Equation]
The remaining steps are the same as that in the proof of Theorem 3 except that we need to bound the sum of variance term.
By Cauchy-Schwarz, [Equation] By the Law of total variation and standard martingale concentration (see Lemma C.5 in Jin et al.
(2018) for a formal proof), with probability at least 1 − p , we have
[Equation]
Putting all relations together, we obtain that with probability at least 1 − 2 p (one p comes from P ( E 1 ) ≥ 1 − p and the other comes from the inequality for bounding the variance term), [Equation] Rescaling p completes the proof.
G P ROOF FOR S ECTION 4 – R EWARD -F REE L EARNING G. P ROOF OF T HEOREM
In this section, we prove Theorem 5 for the single reward function case, i.e., N = 1 .
The proof for multiple reward functions ( N > 1 ) simply follows from taking a union bound, that is, replacing the failure probability p by Np .
Let ( µ k , ν k ) be an arbitrary Nash-equilibrium policy of (cid:99) M k :=
( (cid:98) P k , (cid:98) r k ) , where (cid:98) P k and (cid:98) r k are our empirical estimate of the transition and the reward at the beginning of the k ’th episode in Algorithm 2, respectively.
We use N kh ( s, a, b ) to denote the number we have visited the state-action tuple ( s, a, b ) at the h -th step before the k ’th episode.
And the bonus used in the k ’th episode can be written as
[Equation]
where ι = log( SABT/p ) and C is some large absolute constant.
We use (cid:98) Q k and (cid:98) V k to denote the empirical optimal value functions of (cid:99) M k as following.
[Equation]
[Equation]
Since ( µ k , ν k ) is a Nash-equilibrium policy of (cid:99) M k , we also have
[Equation]
We begin with stating a useful property of matrix game that will be frequently used in our analysis.
Since its proof is quite simple, we omit it here.
Lemma 24.
Let X , Y ,
Z ∈ R
A × B and ∆ d be the d -dimensional simplex.
Suppose | X − Y | ≤ Z , where the inequality is entry-wise.
Then
[Equation]
Lemma 25.
Let c 1 be some large absolute constant such that c 21 + c 1 ≤ C .
Deﬁne event
E 1 to be: for all h, s, a, b, s (cid:48) and k ∈ [ K ] , 
[Equation]
We have P ( E 1 ) ≥ 1 − p .
Proof.
The proof is standard and folklore: apply standard concentration inequalities and then take a union bound.
For completeness, we provide the proof of the third one here.
Consider a ﬁxed ( s, a, b, h ) tuple.
Let’s consider the following equivalent random process: (a) before the agent starts, the environment samples { s (1) , s (2) , . . .
, s ( K ) } independently from P h ( · | s, a, b ) ; (b) during the interaction between the agent and environment, the i th time the agent reaches ( s, a, b, h ) , the environment will make the agent transit to s ( i ) .
Note that the randomness induced by this interaction procedure is exactly the same as the original one, which means the probability of any event in this context is the same as in the original problem.
Therefore, it sufﬁces to prove the target concentration inequality in this ’easy’ context.
Denote by (cid:98) P ( t ) h ( · | s, a, b ) the empirical estimate of P h ( · | s, a, b ) calculated using { s (1) , s (2) , . . .
, s ( t ) } .
For a ﬁxed t and s (cid:48) , by the empirical Bernstein inequality, we have with probability at least 1 − p/S 2 ABT ,
[Equation]
Now we can take a union bound over all s, a, b, h, s (cid:48) and t ∈
[ K ] , and obtain that with probability at least 1 − p , for all s, a, b, h, s (cid:48) and t ∈
[ K ] ,
[Equation]
Note that the agent can reach each ( s, a, b, h ) for at most K times, this directly implies that the third inequality also holds with probability at least 1 − p .
The following lemma states that the empirical optimal value functions are close to the true optimal ones, and their difference is controlled by the exploration value functions calculated in Algorithm 2.
Lemma 26.
Suppose event E 1 (deﬁned in Lemma 25) holds. Then for all h, s, a, b and k ∈ [ K ] , we have,
[Equation]
Proof.
Let’s prove by doing backward induction on h .
The case of h = H + 1 holds trivially.
Assume the conclusion hold for ( h + 1) ’
th step.
For h ’th step,
[Equation] [Equation]
≤ min (cid:110)(cid:12)(cid:12)(cid:12) [( (cid:98) P kh − P h ) V (cid:63)h +1 ]( s, a, b ) (cid:12)(cid:12)(cid:12) + | ( (cid:98) r kh − r h )( s, a, b ) | + (cid:12)(cid:12)(cid:12) [ (cid:98) P kh
( (cid:98) V
kh +1 − V (cid:63)h +1 )]
( s, a, b ) (cid:12)(cid:12)(cid:12) , H (cid:111) (
[Equation]
≤
[Equation]
β
[Equation]
P kh (cid:101) V kh
[Equation]
a
[Equation]
H
[Equation]
(cid:101)
[Equation] s [Equation] b ) , (25) where ( i ) follows from the induction hypothesis and event E 1 , and ( ii ) follows from the deﬁnition of (cid:101) Q kh .
By Lemma 24, we immediately obtain | (cid:98) V kh
[Equation] Now, we are ready to establish the key lemma in our analysis using Lemma 26.
Lemma 27.
Suppose event E 1 (deﬁned in Lemma 25) holds. Then for all h, s, a, b and k ∈ [ K ] , we have
[Equation]
and
[Equation]
where α H +1 =
and α h =
[(1 + 1 H )
α h +
+ 1 H ] ≤ 4 .
Proof.
We only prove the ﬁrst set of inequalities.
The second one follows exactly the same.
Again, the proof is by performing backward induction on h .
It is trivial to see the conclusion holds for ( H + 1) ’
th step with α H +1 = 0 .
Now, assume the conclusion holds for ( h + 1) ’
th step.
For h ’th step,
[Equation]
where the second inequality follows from the deﬁnition of event E 1 .
We can control the term ( T 1 ) by combining Lemma 26 and the induction hypothesis to bound | V † ,ν k h +1 − V (cid:63)h +1 | , and then applying the third inequality in event E 1 : (
[Equation]
The term ( T 2 ) is bounded by directly applying the induction hypothesis
[Equation]
Plugging (29) and (30) into (28), we obtain
[Equation]
where ( i ) follows from the deﬁnition of β kh
[Equation]
and ( ii ) follows from the deﬁnition of (cid:101) Q kh .
Therefore, by (31), choosing α h =
[(1 + 1 H )
α h +
+ 1 H ] sufﬁces for the purpose of induction.
Now
[Equation]
let’s prove the inequality for V functions.
[Equation]
where ( i ) follows from the deﬁnition of (cid:98) V k h and V † ,ν k h , and ( ii ) uses (31) and Lemma 24.
Theorem 28 (Guarantee for UCB-VI from Azar et al.
For any p ∈ (0 , 1] , choose the exploration bonus β t in Algrothm 2 as (20) .
Then, with probability at least 1 − p ,
[Equation]
Proof of Theorem 5.
Recall that out = arg min k ∈
[ K ] (cid:101) V kh
( s ) .
By Lemma 27 and Theorem 28, with probability at least 1 − 2 p ,
[Equation]
[Equation]
Rescaling p completes the proof.
G. V ANILLA N ASH V ALUE I TERATION
Here, we provide one optional algorithm, Vanilla Nash VI, for computing the Nash equilibrium policy for a known model.
Its only difference from the value iteration algorithm for MDPs is that the maximum operator is replaced by the minimax operator in Line 7.
We remark that the Nash equilibrium for a two-player zero-sum game can be computed in polynomial time.
do do
]( s, a, b ) + (cid:98) r h ( s, a, b ) .
6: for s ∈ S do 7: (ˆ µ h ( · | s ) , ˆ ν h ( · | s ))
← NASH-ZERO-SUM ( Q h ( s, · , · )) .
V h ( s ) ← ˆ µ h ( · | s ) (cid:62) Q h ( s, · , · )ˆ ν h ( · | s ) . 9: Output (ˆ µ, ˆ ν ) ← { (ˆ µ h ( · | s ) , ˆ ν h ( · | s )) } ( h,s ) ∈ [ H ] ×S .
By recalling the deﬁnition of best responses in Appendix D, one can directly see that the output policy (ˆ µ, ˆ ν ) is a Nash equilibrium for (cid:99) M .
G. P ROOF OF T HEOREM
In this section, we ﬁrst prove a Θ( AB/(cid:15) 2 ) lower bound for reward-free matrix games, i.e., S = H = 1 , and then generalize it to Θ( SABH 2 /(cid:15) 2 ) for the Markov games setting. G.3. REWARD - FREE MATRIX GAMES
In the matrix game, let the max-player pick row and the min-player pick column.
We consider the following family of Bernoulli matrix games:
[Equation]
(34) where in matrix game M , the reward is sampled from Bernoulli( M ab ) if the max-player picks the a ’th row and the min-player picks the b ’th column. [Table]
player [Table]
[Table] action . . . b (cid:63) − 1 b (cid:63) b (cid:63) + 1 . . . B + . . .
+ − + . . . + ... a (cid:63) − 1 + . . . + − + . . . + a (cid:63) + . . .
+ + + . . . + a (cid:63) + 1 + . . . + − + . . . + ...
A + . . . + − + . . . + (35) Above, we visualize the hard instance by using + and − to represent 1 / 2+ (cid:15) and 1 / 2 − (cid:15) , respectively.
It is direct to see that the optimal policy for the max-player is always picking the a (cid:63) ’th row and the optimal policy for the min-player is always picking the b (cid:63) ’th column.
If the max-player picks the a (cid:63) ’th row with probability smaller than 2 / 3 , it is at least (cid:15)/ 10 suboptimal.
Lemma 29.
For any ﬁxed matrix game M from M ( (cid:15) ) and N ∈ N , if an algorithm A can output a policy that is at most (cid:15)/ 10 suboptimal with probability at least p using at most N samples, then there exists an algorithm ˆ A that can identify the best row in M with probability at least p using at most N samples.
Proof.
We simply deﬁne ˆ A as running algorithm A and choosing the most played row by its outputted policy as the guess for the best row.
By simple calculation, one can show ˆ A will output the best row in M with probability at least p .
Lemma 29 directly implies that in order to prove the desired lower bound for matrix games:
Claim 30.
for any algorithm A using at most N = AB/ (10 3 (cid:15) 2 ) samples, there exists a matrix game M in M ( (cid:15) ) such that when running A on M , it will output a policy that is at least (cid:15)/ 10 suboptimal for the max-player with probability at least 1 / 4 , it sufﬁces to prove the following claim:
Claim 31.
for any algorithm ˆ A using at most N = AB/ (10 3 (cid:15) 2 ) samples, there exists a matrix game M in M ( (cid:15) ) such that when running ˆ A on M , it will fail to identify the optimal row with probability at least 1 / 4 .
Proof of Claim 31.
WLOG, we assume ˆ A is deterministic.
Since this is the reward-free setting, being deterministic means that algorithm ˆ A will always pull each arm ( a, b ) for some ﬁxed n ( a, b ) times and then output a guess for a (cid:63) which is a function of the reward revealed.
Denote by L the reward revealed after algorithm ˆ A ’s pulling.
Denote by P (cid:63) the probability induced by picking M uniformly at random from M ( (cid:15) ) and running ˆ A on M .
Denote by P a,b the probability induced by running ˆ A on M , whose indices of the special row and the special column are a and b , respectively.
Denote by P 0 ,b the probability induced by running A on M , whose b ’th column are all 1 / 2 − (cid:15) and other columns are all 1 / 2 + (cid:15) .
We want to mention that the M we use to deﬁne P 0 ,b does not belong to M ( (cid:15) ) .
We have
[Equation]
Choosing N = AB/ (10 3 (cid:15) 2 ) concludes the proof.
G.3. REWARD - FREE M ARKOV GAMES
Now let’s generalize the Θ( AB/(cid:15) 2 ) lower bound to Θ( SABH 2 /(cid:15) 2 ) for reward-free Markov games.
We deﬁne the following family of MDPs: [Equation]
where MDP J ( a (cid:63) , b (cid:63) ) is deﬁned as below:
• States and actions: J ( a (cid:63) , b (cid:63) ) is a ﬁnite-horizon MDP with S + 1 states of length H + 1 .
There is a ﬁxed initial state s 0 in the ﬁrst step, S states { s 1 , . . . , s S } for the remaining steps.
The two players have A and B actions, respectively.
• Rewards: there is no reward in the ﬁrst step. For the remaining steps h ∈ { 2 , . . . , H + 1 } , if the agent takes action ( a, b ) at state s i in the h th step, it will receive a binary reward sampled from Bernoulli
[Equation] • Transitions: Regardless of the current state, actions and index of steps, the agent will always transit to one of s 1 , . . .
, s S uniformly at random. It is direct to see that J ( a (cid:63) , b (cid:63) ) is a collection of SH independent matrix games from M ( (cid:15)/H ) .
Therefore, the optimal policy for the max player is to always pick action a (cid:63)h − 1 ,i whenever it reaches state s i in the h th step ( h ≥ 2 ).
In other words, a (cid:63)h − 1 ,i is the unique optimal action for the step-state pair ( h, i ) . At a high level, in order to ﬁnd an (cid:15) -optimal policy for the above Markov game, we need to identify at least half of the entries of a (cid:63) .
Therefore, the number of episodes should be at least
[Equation]
Below we provide a formal proof of this argument, which is almost the same as that for the setting of reward-free matrix games.
We start by proving an analogue of Lemma 29.
Lemma 32.
For any ﬁxed matrix game J ( a (cid:63) , b (cid:63) ) from J ( (cid:15) ) and N ∈ N , if an algorithm A can output a policy that is at most (cid:15)/ 10 3 suboptimal with probability at least p using at most N samples, then there exists an algorithm ˆ A that can correctly identify at least SH − (cid:98) SH/ 500 (cid:99) entries of a (cid:63) with probability at least p using at most N samples.
Proof.
Denote by π the output policy for the max player.
Denote by Z the collection of ( h, i ) ’s in [ H ] ×
[ S ] such that π h +1 ( a (cid:63)h,i | s i ) ≤ 2 / 3 .
Observe that each time the max player picks a suboptimal action, it will incur an 2 (cid:15)/H suboptimality in expectation.
As a result, if π is at most (cid:15)/ 10 3 -suboptimal, we must have
[Equation]
which implies | Z | ≤ SH/ 500 , that is, for at most (cid:98) SH/ 500 (cid:99) different i ’s, π ( a (cid:63)i | s i ) ≤ 2 / 3 .
Therefore, we can simply pick argmax a π h +1 ( a | s i ) as the guess for a (cid:63)h,i .
Since policy π is at most (cid:15)/ 10 3 suboptimal with probability at least p , we can correctly identify the optimal actions for at least SH − (cid:98) SH/ 500 (cid:99) different ( s, h ) pairs also with probability no smaller than p .
Lemma 32 directly implies that in order to prove the desired lower bound for reward-free Markov games:
Claim 33.
for any algorithm A interacting with the environment for at most K = ABSH 2 / (10 4 (cid:15) 2 ) episodes, there exists J ∈ J ( (cid:15) )
such that when running A on J , it will output a policy that is at least (cid:15)/ 10 3 suboptimal for the max-player with probability at least 1 / 4 , it sufﬁces to prove the following claim:
Claim 34.
for any algorithm ˆ
A interacting with the environment for at most K = ABSH 2 / (10 4 (cid:15) 2 ) episodes, there exists J ∈ J ( (cid:15) )
such that when running ˆ A on J , it will fail to identify the optimal actions for at least (cid:98) SH/ 500 (cid:99) + 1 different ( s, h ) pairs with probability at least 1 / 4 .
Proof of Claim 34.
Denote by P (cid:63) ( E (cid:63) )
the probability (expectation) induced by picking J ( a (cid:63) , b (cid:63) ) uniformly at random from J ( (cid:15) ) and running ˆ A on J .
Denote by n wrong the number of ( s, h ) pairs for which ˆ
A fails to identify the optimal actions.
Denote by error h,i the indicator function of the event that ˆ A fails to identify the optimal action for ( h + 1 , i ) .
We prove by contradiction.
Suppose for any J ∈ J ( (cid:15) ) , ˆ A can identify the optimal actions for at least SH − (cid:98) SH/ 500 (cid:99) different ( s, h ) pairs with probability larger than 3 / 4 .
Then we have
[Equation]
Since (cid:80) ( h,i ) ∈
[ H ] ×
[ S ] E (cid:63) [error h,i ] = E (cid:63) [ n wrong ] , there must exists ( h (cid:48) , i (cid:48) )
∈
[ H ] × [ S ] such that
E (cid:63) [error h (cid:48) ,i (cid:48) ] ≤ 101 / 400 .
However, in the following, we show that for every ( h, i ) ∈
[ H ] × [ S ] , ˆ A fails to identify the optimal action for the step-state pair ( h +1 , i ) with probability at least 1 / 3 , which directly implies E (cid:63) [error h,i ] ≥ 1 / 3 for all
( h, i ) ∈
[ H ] × [ S ] .
As a result, we obtain a contraction and Claim 34 holds.
Now, let us prove that for every ( h, i ) ∈
[ H ] × [ S ] , E (cid:63) [error h,i ] ≥ 1 / 6 .
WLOG, we assume ˆ A is deterministic and it runs for exactly K = ABSH 2 / (10 3 (cid:15) 2 ) episodes.
In the following, we consider a ﬁxed ( h (cid:48) , i (cid:48) ) pair.
For technical reason, we deﬁne MDP J − ( h (cid:48) ,i (cid:48) )
( a (cid:63) , b (cid:63) ) as below:
• States, actions and transitions: same as J ( a (cid:63) , b (cid:63) ) .
• Rewards: there is no reward in the ﬁrst step.
For the remaining steps h ∈ { 2 , . . .
, H + 1 } , if the agent takes action ( a, b ) at state s i in the h th step such that ( h − 1 , i ) (cid:54) =
( h (cid:48) , i (cid:48) ) , it will receive a binary reward sampled from Bernoulli
[Equation]
otherwise it will receive a binary reward sampled from
[Equation]
Intuitively, J − ( h (cid:48) ,i (cid:48) )
( a (cid:63) , b (cid:63) ) is the same as J ( a (cid:63) , b (cid:63) ) except that for the max player, all its actions at state s i (cid:48) in the h (cid:48)
th step are equivalent.
In other words, J − ( h (cid:48) ,i (cid:48) ) ( a (cid:63) , b (cid:63) ) is independent of a (cid:63)h (cid:48) ,i (cid:48) .
To proceed, we need to deﬁne the following notations: denote by n ( a, b ) the number of times ˆ A picks action ( a, b ) at state s i (cid:48) in the ( h (cid:48)
+ 1) th step; denote by P ( · | J ( a (cid:63) , b (cid:63) ))
( E [ · | J ( a (cid:63) , b (cid:63) )] ) the probability (expectation) induced by running algorithm ˆ A on J ( a (cid:63) , b (cid:63) ) ; similarly, we deﬁne P ( · |
J − ( h (cid:48) ,i (cid:48) )
( a (cid:63) , b (cid:63) ))
( E [ · | J − ( h (cid:48) ,i (cid:48) )
( a (cid:63) , b (cid:63) )] ); also recall that we denote by P (cid:63) ( E (cid:63) )
the probability (expectation) induced by picking J ( a (cid:63) , b (cid:63) ) uniformly at random from J ( (cid:15) ) and running ˆ A on J ; denote by L the whole trajectory of states, actions and rewards produced by algorithm ˆ A in N episodes; with slight abuse of notation, denote by ˆ A ( L ) the guess of ˆ A for a (cid:63)h (cid:48) ,i (cid:48) based on L .
First, note that for any ( a, b ) ∈
[ A ] × [ B ] , E [ n ( s, a ) | J − ( h (cid:48) ,i (cid:48) )
( a (cid:63) , b (cid:63) )] is independent of ( h (cid:48) , i (cid:48) , a (cid:63) , b (cid:63) ) because the agent cannot observe any reward when interacting with the environment and the transition dynamics of different J − ( h (cid:48) ,i (cid:48) )
( a (cid:63) , b (cid:63) ) ’s are the same.
For simplicity of notation, we denote this expectation by m ( a, b ) .
Note that (cid:80) a,b m ( a, b )
=
K/S because the agent always reach state s i (cid:48) in the ( h (cid:48)
+ 1) th step with probability 1 /S regardless of the actions taken.
We have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
[Table]
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(cid:13)(cid:13)
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Plugging in K = SABH 2 / (10 4 (cid:15) 2 ) completes the proof.
H P ROOF FOR A PPENDIX C – M ULTI - PLAYER G ENERAL - SUM M ARKOV G AMES H. P ROOF OF T HEOREMH.1. NE V ERSION
In this section, we prove Theorem 15 (NE version).
As before, we begin with proving the optimistic estimations are indeed upper bounds of corresponding value and Q-value functions.
Lemma 35.
With probability 1 − p , for any ( s, a , h, k, i ) :
[Equation]
Proof.
For each ﬁxed k , we prove this by induction from h = H + 1 to h = 1 .
For base case, we know at the ( H + 1) -th step,
V kH +1 ,i ( s )
=
V † ,π k − i H
+1 ,i ( s )
= 0 .
Now, assume the inequality (40) holds for the ( h + 1) -th step, for the h -th step, by deﬁnition of the Q functions,
[Equation]
By induction hypothesis, for any s (cid:48) , (cid:18) V kh +1 ,i − V † ,π k − i
h
+1 ,i (cid:19) ( s (cid:48) ) ≥ 0 , and thus ( A ) ≥ 0 .
By uniform concentration (e.g., Lemma 12 in Bai & Jin (2020)), ( B ) ≤ C (cid:113) SH 2 ι/N kh
( s, a ) =
β t .
Putting everything together we have Q kh,i ( s, a ) − Q † ,π k − i h,i ( s, a ) ≥ 0 .
The second inequality can be proved similarly.
Now assume inequality (39) holds for the h -th step, by deﬁnition of value functions and Nash equilibrium,
[Equation]
By Bellman equation,
[Equation]
Since by induction hypothesis, for any ( s, a ) , Q kh,i ( s, a ) ≥ Q † ,π k − i h [Equation] i ( s, a ) .
As a result, we also have
V kh,i ( s ) ≥ V † ,π k − i h,i ( s ) , which is exactly inequality (40) for the h -th step.
The second inequality can be proved similarly.
Proof of Theorem 15.
Let us focus on the i -th player and ignore the subscript when there is no confusion.
To bound
[Equation]
we notice the following propogation:
[Equation]
We can deﬁne (cid:101) Q kh and (cid:101)
V kh recursively by (cid:101) V kH
+1 =
and
[Equation]
Then we can prove inductively that for any k
[Equation]
h
[Equation]
s and a we have
[Equation]
[Equation] Thus we only need to bound (cid:80) Kk =1 (cid:101)
V k 1 ( s ) .
Deﬁne the shorthand notation
[Equation]
We can check ζ kh and ξ kh are martingale difference sequences.
As a result,
[Equation]
Recursing this argument for h ∈ [ H ] and taking the sum,
[Equation]
H.1. CCE V ERSION
The proof is very similar to the NE version.
Speciﬁcally, the only part that uses the properties of NE there is Lemma 35.
We prove a counterpart here.
Lemma 36.
With probability 1 − p , for any ( s, a , h, k, i ) :
[Equation]
Proof.
For each ﬁxed k , we prove this by induction from h = H + 1 to h = 1 .
For base case, we know at the ( H + 1) -th step,
V kH +1 ,i ( s )
=
V † ,π k − i H
+1 ,i ( s )
= 0 .
Now, assume the inequality (40) holds for the ( h + 1) -th step, for the h -th step, by deﬁnition of the Q functions,
[Equation]
By induction hypothesis, for any s (cid:48) , (cid:18) V kh +1 ,i − V † ,π k − i
h
+1 ,i (cid:19) ( s (cid:48) ) ≥ 0 , and thus ( A ) ≥ 0 .
By uniform concentration, ( B ) ≤ C (cid:113) SH 2 ι/N kh ( s, a ) = β t .
Putting everything together we have Q kh,i ( s, a ) −
Q † ,π k − i h,i ( s, a ) ≥ 0 .
The second inequality can be proved similarly.
Now assume inequality (45) holds for the h -th step, by deﬁnition of value functions and CCE,
[Equation]
By Bellman equation,
[Equation]
Since by induction hypothesis, for any ( s, a ) , Q kh,i ( s, a ) ≥ Q † ,π k − i h,i ( s, a ) .
As a result, we also have
V kh,i ( s ) ≥ V † [Equation] π k − i h,i ( s ) , which is exactly inequality (40) for the h -th step.
The second inequality can be proved similarly.
H.1. CE V ERSION
The proof is very similar to the NE version.
Speciﬁcally, the only part that uses the properties of NE there is Lemma 35.
We prove a counterpart here.
Lemma 37.
With probability 1 − p , for any ( s, a , h, k, i ) :
[Equation]
Proof.
For each ﬁxed k , we prove this by induction from h = H + 1 to h = 1 .
For base case, we know at the ( H + 1) -th step,
V kH +1 ,i ( s )
= max φ V φ (cid:5) π k H
+1 ,i ( s )
= 0 .
Now, assume the inequality (40) holds for the ( h + 1) -th step, for the h -th step, by deﬁnition of the Q functions,
[Equation]
By induction hypothesis, for any s (cid:48) , (cid:18) V kh +1 ,i − max φ V φ (cid:5) π k h +
[Equation]
(cid:19) ( s (cid:48) ) ≥ 0 , and thus ( A ) ≥ 0 .
By uniform concentration, ( B ) ≤ C (cid:113) SH 2 ι/N kh ( s, a ) = β t .
Putting everything together we have
Q [Equation] ( s, a ) [Equation]
The second inequality can be proved similarly.
Now assume inequality (47) holds for the h -th step, by deﬁnition of value functions and CE,
[Equation]
By Bellman equation,
[Equation]
Since by induction hypothesis, for any ( s, a ) , Q kh,i ( s, a ) ≥ max φ Q φ (cid:5) π k h,i ( s, a ) .
As a result, we also have V kh,i ( s ) ≥ max φ V φ (cid:5) π k h,i ( s ) , which is exactly inequality (40) for the h -th step.
The second inequality can be proved similarly.
H. P ROOF OF T HEOREM
In this section, we prove each theorem for the single reward function case, i.e., N = 1 .
The proof for the case of multiple reward functions ( N > 1 ) simply follows from taking a union bound, that is, replacing the failure probability p by Np .
H.2. NE VERSION
Let ( µ k , ν k ) be an arbitrary Nash-equilibrium policy of (cid:99) M k :=
( (cid:98) P k , (cid:98) r k ) , where (cid:98) P k and (cid:98) r k are our empirical estimate of the transition and the reward at the beginning of the k ’th episode in Algorithm 4.
Given an arbitrary Nash equilibrium π k of (cid:99) M k , we use (cid:98) Q kh,i and (cid:98) V kh,i to denote its value functions of the i ’th player at the h ’th step in (cid:99) M k .
We prove the following two lemmas, which together imply the conclusion about Nash equilibriums in Theorem 16 as in the proof of Theorem 5.
Lemma 38.
With probability 1 − p , for any ( h, s, a , i, k ) , we have
[Equation]
Proof.
For each ﬁxed k , we prove this by induction from h = H + 1 to h = 1 .
For base case, we know at the ( H + 1) -th step, (cid:98) V kH
+1 ,i =
V π k H
+1 ,i = (cid:98) Q kH
+1 ,i = Q π k H
+1 ,i = 0 .
Now, assume the conclusion holds for the ( h + 1) ’
th step, for the h ’th step, by deﬁnition of the Q functions,
[Equation]
By the induction hypothesis,
[Equation]
By uniform concentration (e.g., Lemma 12 in Bai & Jin (2020)), ( B ) ≤ (cid:113) SH 2 ι/N kh ( s, a ) =
β t .
Putting everything together we have
[Equation]
which proves the ﬁrst inequality in (49).
The inequality for V functions follows directly by noting that the value functions are computed using the same policy π k .
Lemma 39.
With probability 1 − p , for any ( h, s, a , i, k ) , we have [Equation]
[Equation]
Proof.
For each ﬁxed k , we prove this by induction from h = H + 1 to h = 1 .
For base case, we know at the ( H + 1) -th step, (cid:98) V kH
+1 ,i =
V π k − i , † H
+1 ,i = (cid:98) Q kH
+1 ,i =
Q π k − i , † H
+1 ,i = 0 .
Now, assume the conclusion holds for the ( h + 1) ’
th step, for the h ’th step, by deﬁnition of the Q functions,
[Equation]
By the induction hypothesis,
[Equation]
By uniform concentration, ( B ) ≤ (cid:113) SH 2 ι/N kh
( s, a ) =
β t .
Putting everything together we have
[Equation]
which proves the ﬁrst inequality in (50).
It remains to show the inequality for V functions also hold in h ’th step.
Since π k is a Nash-equilibrium policy, we have
[Equation]
By Bellman equation,
[Equation]
Combining the two equations above, and utilizing the bound we just proved for Q functions, we obtain
[Equation]
which completes the whole proof.
H.2. CCE VERSION
The proof is almost the same as that for Nash equilibriums.
We will reuse Lemma 38 and prove an analogue of Lemma 39.
The conclusion for CCEs will follow directly by combining the two lemmas as in the proof of Theorem
[Equation]
Lemma 40.
With probability 1 − p , for any ( h, s, a , i, k ) , we have
[Equation]
Proof.
For each ﬁxed k , we prove this by induction from h = H + 1 to h = 1 .
For base case, we know at the ( H + 1) -th step, (cid:98) V k H
+1 ,i = V
[Equation]
H +1 ,i = 0 .
Now, assume the
[Equation]
[Equation]
By the induction hypothesis, ( A ) ≤ ( (cid:98) P kh (cid:101)
V kh +1 )
( s, a ) .
By uniform concentration, ( B ) ≤ (cid:113) SH 2 ι/N kh
( s, a ) =
β t .
Putting everything together we have
[Equation]
which proves the ﬁrst inequality in (51).
It remains to show the inequality for V functions also hold in h ’th step.
Since π k is a CCE, we have
[Equation]
Observe that V π k − i , † h,i obeys the Bellman optimality equation, so we have
[Equation]
Combining the two equations above, and utilizing the bound we just proved for Q functions, we obtain
[Equation]
which completes the whole proof.
H.2. CE VERSION
The proof is almost the same as that for Nash equilibriums.
We will reuse Lemma 38 and prove an analogue of Lemma 39.
The conclusion for CEs will follow directly by combining the two lemmas as in the proof of Theorem 5.
Lemma 41.
With probability 1 − p , for any ( h, s, a , i, k ) and strategy modiﬁcation φ for player i , we have [Equation]
[Equation]
Proof.
For each ﬁxed k , we prove this by induction from h = H + 1 to h = 1 .
For base case, we know at the ( H + 1) -th step, (cid:98) V kH
+1 ,
i =
V φ (cid:5) π k H +1 ,i = (cid:98) Q kH
+1 ,i =
Q φ (cid:5) π k H +1 ,i = 0 .
Now, assume the conclusion holds for the ( h + 1) ’
th step, for the h ’th step, following exactly the same argument as
Lemma 40, we can show
[Equation]
which proves the ﬁrst inequality in (52).
It remains to show the inequality for V functions also hold in h ’th step.
Since π k is a CE, we have
[Equation]
where the maximum is take over all possible injective functions from A i to itself.
Observe that V φ (cid:5) π k h,i obeys the Bellman optimality equation, so we have
[Equation]
Combining the two equations above, and utilizing the bound we just proved for Q functions, we obtain
[Equation]
which completes the whole proof.