W HEN ARE O FFLINE T WO -P LAYER Z ERO -S UM M ARKOV G AMES S OLVABLE ?
A BSTRACT
We study what dataset assumption permits solving offline two-player zero-sum
Markov games.
In stark contrast to the offline single-agent Markov decision process, we show that the single strategy concentration assumption is insufficient for learning the Nash equilibrium (NE) strategy in offline two-player zero-sum
Markov games.
On the other hand, we propose a new assumption named unilateral concentration and design a pessimism-type algorithm that is provably efficient under this assumption.
In addition, we show that the unilateral concentration assumption is necessary for learning an NE strategy.
Furthermore, our algorithm can achieve minimax sample complexity without any modification for two widely studied settings: dataset with uniform concentration assumption and turn-based
Markov games.
Our work serves as an important initial step towards understanding offline multi-agent reinforcement learning.
I NTRODUCTION
Promising empirical advances have been achieved in reinforcement learning (RL), including mastering the games of Go (Silver et al., 2016), Poker (Brown et al., 2017), real-time strategy games (Vinyals et al., 2019) and robotic control (Kober et al., 2013).
Notably, many of these successes lie in the domain of multi-agent reinforcement learning (MARL).
MARL is about multiple agents interacting in a shared environment and each of them aims to maximize its own long-term reward.
During the learning process, each agent not only needs to identify the environment dynamic but also needs to compete/cooperate with other agents.
One important subarea of MARL is offline MARL.
In many practical scenarios, we only have access to the offline data or it is too expensive to frequently change the policy (Zhang et al., 2021a).
While there are plenty of empirical works on offline MARL (Pan et al., 2021; Jiang & Lu, 2021), the theoretical understanding about offline MARL is still very limited.
In this work, we take an initial step towards understanding when offline MARL is provably solvable.
We consider the two-player zero-sum Markov games, where two players sequentially select actions in a Markovian environment and the first player aims to maximize the total reward while the second player aims to minimize it.
In the offline setting, we have access to a fixed dataset collected by a (possibly unknown) exploration policy and the target is to find a (near-)Nash equilibrium (NE) strategy of the underlying two-agent zero-sum Markov games.
One of the main difficulties in offline RL is distribution shift, i.e. the dataset distribution is different from the distribution induced by the optimal policy.
It is important to understand what is the minimal dataset distribution assumption that permits offline RL.
For single-agent offline RL, it is shown that the pessimism principle allows policy optimization with single policy concentration , i.e. the dataset only covers the optimal policy (Jin et al., 2021b; Zanette et al., 2021; Yin & Wang, 2021;
Rashidinejad et al., 2021).
This assumption is necessary, as it is impossible to learn the optimal policy if it is not covered by the dataset.
However, the dataset coverage assumption for MARL is far from clear.
In this work, we want to answer the following question:
What is the minimal dataset coverage assumption that permits learning an NE strategy in offline two-player zero-sum Markov games?
Generally speaking, MARL is much more difficult than single-agent RL due to the following two reasons.
First, MARL is known to suffer from the non-stationary property, i.e. agents will affect the others during the learning process (Zhang et al., 2021a).
Specifically, the performance may decline if each agent simultaneously tries to improve their own policy depends on others’ current policies.
In addition, multiple agents incur complicated statistical dependence that makes the theoretical analysis difficult.
A line of works studies Markov games with online sampling oracle (Bai et al., 2020; Bai &
Jin, 2020; Liu et al., 2021) or generative model oracle (Sidford et al., 2020; Zhang et al., 2020; Cui & Yang, 2020), where specialized techniques are developed to tackle the above difficulties.
In this paper, we give the first analysis on Markov games in the offline setting.
1. M AIN C ONTRIBUTIONS
First, as a warm up, we consider the most natural extension of the single policy concentration assumption in single-agent RL: the dataset covers the NE strategy in offline two-player zero-sum games.
However, in Section 3.1, we prove that it is impossible to learn the NE strategy under such assumption.
We construct a pair of hard Markov games such that there exists a dataset that covers the NE strategies in both Markov games but no algorithm can distinguish between these two
Markov games and their NE strategies.
This hardness result refutes the conjecture that single strategy concentration works for Markov games.
Second, we propose an assumption named unilateral concentration , which posits that for all strategy µ , ν , strategy pairs ( µ ∗ , ν ) and ( µ, ν ∗ ) are covered by the dataset, where µ is the strategy for the first (max) player, ν is the strategy for the second (min) player, and ( µ ∗ , ν ∗ ) is the NE strategy.
In
Section 3.2, we prove that NE strategy is not learnable even if this assumption is only slightly violated.
The intuition behind these hardness results is that to identify an NE strategy, the algorithm has to compare it with the strategy that one player uses any other strategies as a reference.
Third, we provide positive results showing that NE strategy is PAC learnable under the unilateral concentration assumption.
Combined with the hardness results above, we can conclude that unilateral concentration assumption is the necessary and sufficient dataset coverage assumption for solving offline Markov games .
Our algorithm is based on the pessimism principle that we maintain both a pessimistic and optimistic estimate for each player, respectively.
We show that our algorithm can achieve (cid:101) O
( (cid:112) C ∗ SABH 3 /n ) performance gap under unilateral concentration assumption, where C ∗ quantifies the coverage of the dataset, S is the number of states, A is the number of max player’s action, B is the number of min player’s action, H is the horizon and n is the number of samples.
Fourth, we show that our algorithm is minimax optimal when the dataset satisfies a stronger assumption, uniform concentration, or the Markov game is turn-based.
These are two widely studied settings in the RL community.
Uniform concentration assumes that all state-action pairs is covered by the dataset and turn-based Markov game is a variant of zero-sum Markov game where two players select actions in turns instead of simultaneously.
Although uniform concentration is about the dataset structure and turn-based Markov game is about the environment structure, our algorithm can adapt to both of them without any modification and achieves minimax sample complexity.
Our algorithm is motivated by the Bernstein-type bonus and reference advantage function techniques in Xie et al.
(2021) while we make novel adaptations, monotonic update and self-bounding technique, to realize them in Markov games.
Monotonic update allows a sandwich-type argument that bounds the reference function and further bounds the variance term.
Self-bounding technique is utilized to bound the performance gap by itself and then solve the inequality to derive the final bound on performance gap.
To summarize, (1) we identify the minimal dataset coverage assumption that allows learning the NE strategy in Markov games; (2) we propose a pessimism-based algorithm that achieves polynomial sample complexity based on novel Markov game techniques; and (3) we further show the algorithm is minimax optimal under the uniform concentration assumption or in turn
P RELIMINARIES 2. T WO -P LAYER Z ERO - SUM M ARKOV G AME
Zero-sum Markov game (MG) generalizes single-agent MDP to two-agent case where one agent aims to maximize the total reward while the other one aims to minimize it.
A finite-horizon timeinhomogeneous zero-sum Markov game is described by the tuple G =
( S , A , B , P, r, H ) , where S is the state space,
A is the action space of the first (max) player, B is the action space of the second (min) player, P =
( P 1 , P 2 , · · · , P H ) , P h ∈ R |S||A||B|×|S| , ∀ h ∈
[ H ] is the (unknown) transition probability matrix for time step h , r = ( r 1 , r 2 , · · · , r H ) , r h ∈ R |S||A||B| , ∀ h ∈
[ H ] is the (unknown) deterministic reward vector and H is the horizon length.
* At each timestep h and state s h , if the max player chooses an action a h and the min player chooses an action b h , then the next state at timestep h + 1 follows the distribution s h +1 ∼ P h ( ·| s h , a h , b h ) and both players receive a reward r h ( s h , a h , b h ) .
Both players sequentially choose H actions and at each timestep, the action is chosen simultaneously and then it is revealed to both players.
We assume that we have a fixed initial state sand it is straightforward to generalize our result to the case where the initial state is sampled from a fixed distribution.
Turn-based Markov game is an important subclass of (simultaneous-move)
Markov game, where the max player takes the action first and the min player can take the action after observing the opponent’s action.
It is a widely studied setting (Sidford et al., 2020; Cui & Yang, 2020; Bai & Jin, 2020) and we will provide minimax sample complexity result for this setting in Section 4.3.
We denote a strategy pair as π = ( µ, ν ) , where µ = ( µ 1 , µ 2 , · · · , µ H ) , µ h : S → ∆ A , ∀ h ∈
[ H ] is the strategy of the first player and ν = ( ν 1 , ν 2 , · · · , ν H ) , ν h : S → ∆ B , ∀ h ∈
[ H ] is the strategy of the second player, where ∆ X is the probability simplex on the finite set X .
A deterministic strategy is a strategy that maps state to a single point distribution.
We define the state value function and state-action value function for a strategy pair π similarly as in single-agent MDP:
[Equation]
If the second player’s strategy ν is fixed, then the MG degenerates to an MDP and we call the optimal policy in this MDP as the best response strategy br 1 ( ν ) .
Similarly we can define the br 2 ( µ ) as the best response for the second player.
We will ignore the subscript in br 1 and br 2 when it is clear in the context.
For all h ∈ [ H ] , s h ∈ S , we define
[Equation]
It is well known that Nash equilibrium (NE) strategy π ∗ = ( µ ∗ , ν ∗ ) , i.e. a strategy pair such that no player can benefit from switching its own strategy, exists for zero-sum Markov games with a unique value function (Shapley, 1953).
In other words, µ ∗ and ν ∗ are the best responses to each other.
We define V ∗ h :
= V µ ∗ ,ν ∗ h for all h ∈
[ H ] .
The following weak duality property holds for all strategy pair ( µ, ν ) in MG:
[Equation]
For a strategy pair π = ( µ, ν ) , we can then define the corresponding duality gap as
[Equation]
The duality gap is always non-negative and the NE strategy has zero duality gap Gap( π ∗ ) = 0 .
Duality gap measures how well a strategy pair approximates the NE.
We say a strategy pair π is an ϵ -approximate NE if Gap( π ) ≤ ϵ .
2. O FFLINE T WO -P LAYER Z ERO -S UM G AME
In offline RL, we are given an offline dataset D = { ( s τh , a τh , b τh , r τh , s τh +1 ) } h ∈
[ H ] τ ∈ [ n ] and we cannot do any further sampling (Kakade, 2003).
We assume that the dataset is sampled from some exploration policy ρ = ( ρ 1 , ρ 2 , · · · , ρ H ) , ρ h : S → ∆ A×B , ∀ h ∈
[ H ] .
The target of offline MG is to find an approximate NE with a small duality gap by utilizing the given dataset D .
We use d πh ( s, a, b ) to denote the probability of s, a, b appears at timestep h in the trajectory generated by strategy π for all h ∈
[ H ] .
The dataset distribution d ρh ( s, a, b ) is defined similarly.
A state-action pair ( s, a, b ) at timestep h is covered by strategy π if and only if d πh ( s, a, b ) > 0 .
Strategy π is covered by dataset generated by exploration strategy ρ
if and only if for all ( s, a, b ) covered by π , it is covered by ρ .
In other words, we have
[Equation]
The sample complexity guarantee will depend on this ratio.
Dataset Coverage Assumptions Below we list three different dataset coverage assumptions for
Markov games.
Assumption 2.1.
(Single strategy concentration)
The NE strategy ( µ ∗ , ν ∗ ) is covered by the dataset.
Assumption 2.2.
(Unilateral concentration)
For all strategy µ and ν , ( µ, ν ∗ ) and ( µ ∗ , ν ) are covered by the dataset, where ( µ ∗ , ν ∗ ) is the NE strategy.
Assumption 2.3.
(Uniform concentration)
For all h ∈ [ H ] and ( s, a, b ) ∈ S ×
A × B , ( s, a, b ) at timestep h is covered by the dataset.
Assumption 2.1 is the weakest assumption and is the most straightforward extension of the single policy concentration in single-agent RL (Rashidinejad et al., 2021).
Assumption 2.3 generalizes the uniform policy concentration in single-agent RL (Yin et al., 2020).
Assumption 2.2 is sandwiched by
Assumption 2.1 and Assumption 2.3 as Assumption 2.2 implies Assumption 2.1 and Assumption 2.3 implies Assumption 2.2.
In this work, we will show that Assumption 2.2 is the minimal dataset coverage assumption that allows NE learning and we provide sample complexity bounds that depends on the density ratio (1).
Notations.
We use Var P ( s,a,b ) ( V ) to denote the variance of the random variable V ( s ′ ) where s ′ ∼
P ( ·| s, a, b ) and Var P ( V ) ∈ R SAB to denote a vector whose ( s, a, b ) component is Var P ( s,a,b ) ( V ) .
We define a ∨ b := max { a, b } and a ∧ b := min { a, b } .
In addition, if a is a vector and b is a scalar, the operation is taken on each element of a : [ a ∨ b ] i = a i ∨ b .
For two vector a ∈ R n , b ∈ R n , we use ab ∈ R n to denote the element-wise division: (cid:2) ab (cid:3) i = a i b i .
In addition, if a is scalar, we still use ab ∈ R n to denote the element-wise division: (cid:2) ab (cid:3) i
= ab i .
I MPOSSIBILITY R ESULTS
In this section, we will provide two hard instances.
To begin with, we show that single strategy concentration is not enough for NE strategy learning.
In addition, we show that no assumption weaker than unilateral concentration allows NE strategy learning.
3. W ARM U P : S INGLE S TRATEGY C OVERAGE IS I NSUFFICIENT FOR NE I DENTIFICATION
To illustrate the hardness of offline MARL, below we construct an example showing that single strategy concentration is insufficient for NE identification.
We consider bandits game, i.e. Markov game with horizon H = 1 .
The result can be easily generalized to arbitrary horizon by setting the reward to be 0 in horizons other than h = 1 .
We consider a class of bandit game and dataset such that the NE strategy is well covered and we show that no algorithm can identify the NE strategy for all bandits game and dataset in this class.
Theorem 3.1.
Define a class X of bandits game M and exploration strategy ρ that consists of all M and ρ pairs which satisfy for all s ∈ S and a, b ∈ support( π ∗ ( · , ·| s )) , [Equation]
[Equation]
where π ∗ is the NE strategy in M .
For any algorithm ALG , there exists ( M, ρ ) ∈
X such that the output of the algorithm ALG is at most a 1 / 2 -approximate NE strategy no matter how many data are collected.
Theorem 3.1 denies the possibility that single strategy concentration allows NE strategy learning.
The reason that single policy concentration works in single-agent RL instead of MARL is that we can use the data of two actions to decide which one is better in MDP but we cannot use the data of two action pairs to decide which pair is closer to the NE strategy in Markov games as NE identification needs other action pairs to be the references.
3. N O A SSUMPTION W EAKER T HAN A SSUMPTION 2.2 I S S UFFICIENT FOR NE L EARNING
In this section, we will show that it is impossible to learn the NE strategy even if Assumption 2. is slightly violated.
To begin with, we consider the following deterministic unilateral concentration assumption.
Assumption 3.2.
(Deterministic unilateral concentration)
For all deterministic strategy µ and ν , ( µ, ν ∗ ) and ( µ ∗ , ν ) are covered by the dataset, where ( µ ∗ , ν ∗ ) is the NE strategy.
Immediately we can tell that Assumption 3.2 is satisfied under Assumption 2.2.
These two assumptions are equivalent, which is shown by Proposition 3.3.
The intuition is that any stochastic strategy can be viewed as a combination of several deterministic strategies so it is covered by the dataset if all the deterministic strategies are covered.
Proposition 3.3.
If for all deterministic strategy µ and ν , ( µ, ν ∗ ) and ( µ ∗ , ν ) are covered by the dataset, then we have for all (possibly stochastic) strategy µ ′ and ν ′ , ( µ ′ , ν ∗ ) and ( µ ∗ , ν ′ ) are covered by the dataset.
Theorem 3.4.
Define a class X of bandits game M and exploration strategy ρ that consists of all
M and ρ pairs satisfying that there exists at most one deterministic strategy µ or one deterministic strategy ν
such that ( µ, ν ∗ ) or ( µ ∗ , ν ) is not covered and for all other deterministic strategy µ ′ , ν ′ , the density ratio is bounded
[Equation]
for all h ∈ [ H ] .
For any algorithm ALG , there exists ( M, ρ ) ∈
X such that the output of the algorithm ALG is at most a 0 .
25 -approximate NE strategy no matter how many data is collected.
Remark 3.5 .
We can easily adapt this instance to arbitrary action space by setting all the other rewards to be 0 .
5 and the exploration strategy ρ to be the uniform distribution on ( a i , b j ) such that ( i, j ) ∈ { ( i, j ) : i ∈ { 1 , 2 } or j ∈ { 1 , 2 } , ( i, j ) ̸
= (2 , 1) } .
Remark 3.6 .
It is straightforward to verify that the hard instance in Theorem 3.4 also holds for turn-based Markov games.
As a result, no assumption weaker than Assumption 2.2 is sufficient for
NE learning in turn-based Markov games.
Theorem 3.4 suggests that no assumption weaker than Assumption 3.2 allows NE strategy learning.
As Assumption 2.2 and Assumption 3.2 are equivalent, no assumption weaker than Assumption 2. allows NE strategy learning.
P ROVABLY E FFICIENT A LGORITHM UNDER U NILATERAL C ONCENTRATION
In this section, we show that it is indeed possible to learn the NE with the unilateral concentration assumption.
We propose a novel algorithm called Pessimistic and Optimistic Value Iteration (POVI), which adapts the pessimism principle in single-agent RL to Markov games.
Our sample complexity result depends on the following quantity name unilateral concentrability :
Definition 4.1.
(Unilateral concentrability)
[Equation]
By definition, C ∗ is finite if Assumption 2.2 is satisfied.
Note that there could be multiple different NE strategies, which correspond to different C ∗ .
Our guarantee holds all of them without the knowledge of NE or C ∗ .
4. H OEFFDING - TYPE A LGORITHM WITH D ATA S PLITTING
To illustrate our main algorithm design ideas, we first propose an algorithm with Hoeffding-type bonus and random data splitting.
Given a dataset D = (cid:8) ( s kh , a kh , b kh , r kh , s kh +1 ) (cid:9) n,H k,h =1 , we denote n h ( s, a, b ) =
(cid:80) nk =1 1 (cid:0) ( s kh , a kh , b kh )
= ( s, a, b ) (cid:1) to be the number of times that ( s, a, b ) is visited at timestep h .
We set the empirical reward as
[Equation]
) and the empirical transition kernel as
[Equation]
if n h ( s, a, b ) ≥ 1 and (cid:98) r h ( s, a, b ) = 0 , (cid:98) P h ( s ′ | s, a, b ) = [-
-]1 /S otherwise.
In addition, we use n h ∈ R SAB to denote a vector such that [ n h ] s,a,b = n h ( s, a, b ) .
Now we explain Algorithm 1 in details.
First, we split the dataset D into H small datasets {D h } Hh = with the same size.
Then we use D h to estimate the reward and the transition matrix at timestep h .
The data splitting scheme is to remove the dependence between each timestep.
Then the value function is estimated via a value-iteration-type algorithm.
At each timestep, we maintain both optimistic and pessimistic estimates by adding/minusing a Hoeffding-type bonus.
We use the following Hoeffdingtype bonus:
[Equation]
where ι = log( HSAB/δ )
Then we can compute the pessimistic estimate Q and optimistic estimate
Q :
[Equation] [Equation]
The pessimistic estimate is for the max player, which mimics the pessimism in single-agent RL.
The optimistic estimate is for the min player, which is also a kind of pessimism as the min player’s target is to minimize the reward.
We compute the NE strategy of the matrix game Q ( s, · , · ) and Q ( s, · , · ) respectively and use the NE value to be the state value V ( s ) and V ( s ) .
Note that we only solve a zero-sum matrix game, which is computationally efficient (Chen & Deng, 2006).
Theorem 4.2.
Suppose Assumption 2.2 holds.
For any 0 < δ < 1 and policy µ, ν , with probability 1 − δ , the pessimistic value V h and optimistic value V h of Algorithm 1 satisfies the following arguments for all h ∈
[ H ]
[Equation]
where s h is sampled from the trajectory following the strategy in the expectation.
Theorem 4.2 provides polynomial bounds on the error of the value estimates in Algorithm 1.
It can directly imply the following performance gap bound.
In addition, it provides guarantees for the reference function that will be utilized in the next section.
Corollary 4.3.
Suppose Assumption 2.2 holds.
For any 0 < δ < 1 and n ≥ (cid:101) O ( √ C ∗ SABH 5 ) , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 1 satisfies
[Equation]
Theorem 4.3 shows that the output strategy of Algorithm 1 is an (cid:101)
O (cid:16)(cid:112) C ∗ SABH 5 /n (cid:17) -approximate
NE.
The parameter C ∗ measures how the exploration strategy ρ covers the strategy space ( µ ∗ , ν ) and ( µ, ν ∗ ) for all µ and ν .
4. B ERNSTEIN - TYPE A LGORITHM WITH R EFERENCE A DVANTAGE F UNCTION D ECOMPOSITION
In this section, we will derive an improved performance gap bound (cid:101)
O (cid:16)(cid:112) C ∗ SABH 3 /n (cid:17) .
The extra H 2 is shaved by using Bernstein-type bonus and reference advantage decomposition technique, which is motivated from Xie et al.
However, we want to emphasize that zero-sum Markov game is substantially different from MDP and requires novel adaptation, which we will describe later.
Algorithm 2 is different from Algorithm 1 in two aspects.
First, we use the reference advantage decomposition to remove an H factor.
The dataset is split into three subset with equal size D ref ,
D 0 , D 1 , and D 1 is further split into H subset with equal size {D h, 1 } Hh =1 .
We run algorithm 1 on dataset D ref and we can obtain pessimistic value estimate V ref and optimistic value estimate V ref with guarantees by Theorem 4.2.
Then we use dataset D 0 to estimate P h V ref h +1 and dataset D h, 1 to estimate P h
( V h
+1 − V ref h +1 ) .
Second, we use a Bernstein-type bonus to remove another H factor.
The our updating formulas of Q h and Q h are
[Equation]
where we truncate by the reference function to ensure monotonic update so that Q h and Q h are more accurate pessimistic/optimistic estimate compared with the reference function Q h and Q h .
The bonus functions are defined as
[Equation]
where c is some universal constant and Var (cid:98) P h, 0 ( V ) , Var (cid:98) P h, 1 ( V ) , n h, 0 , n h, 1 are all SAB -dimension vectors and the operations are element-wise.
Theorem 4.4.
Suppose Assumption 2.2 holds.
For any 0 < δ < 1 and n ≥ (cid:101) O ( √ C ∗ SABH 4 ) , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 2 satisfies
[Equation]
As MDP is degenerated Markov game with one player having a fixed action, Markov game inherits the lower bounds of MDP.
Comparing with the lower bound (cid:101) Ω (cid:16)(cid:112) C ∗ SH 3 /n (cid:17) (Xie et al., 2021), our bound is already tight in C ∗ , S , H .
The extra AB factor is from the Cauchy-Schwarz inequality and the fact that the NE of zero-sum Markov game can be mixed strategy.
This makes the bound different from MDP where deterministic optimal policy always exists.
It is unknown whether the AB factor is removable and we leave it to future work.
4. M INIMAX O PTIMAL S AMPLE C OMPLEXITY B OUNDS
In this section, we show that Algorithm 2 directly adapts to two popular settings, i.e. Assumption 2.3 (uniform concentration assumption) and turn-based Markov game.
In addition, minimax sample complexity can be achieved under both settings.
The proof is deferred to Appendix F.
Theorem 4.5.
Suppose
Assumption 2. holds and d m = min { d ρh ( s, a, b ) :
h ∈ [ H ] , ( s, a, b ) ∈ S ×
A × B} .
For any 0 < δ < 1 and n ≥ (cid:101) O (cid:16)(cid:112) H 4 /d m (cid:17) , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 2 satisfies
[Equation]
) (cid:17) . This bound has no explicit dependence on AB because the Cauchy-Schwarz inequality can be applied on d µ ∗ ,ν h instead of (cid:113) d µ ∗ ,
ν h
(See the proof of Theorem F.1).
As the lower bound (cid:101) Ω (cid:16)(cid:112) H 3 / ( nd m ) (cid:17) for MDP (Yin & Wang, 2021) is the lower bound for Markov games, Algorithm 2 achieves minimax sample complexity under assumption 2.3.
Theorem 4.6.
Suppose Assumption 2.2 holds for a turn-based Markov game.
For any 0 < δ < 1 and n ≥ (cid:101) O ( √ C ∗ SH 4 ) , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 2 satisfies
[Equation]
As the lower bound is (cid:101) Ω (cid:16)(cid:112) C ∗ SH 3 /n (cid:17) (Xie et al., 2021), Algorithm 2 can achieve the minimax sample complexity for turn-based Markov games under assumption 2.2.
The difference between turn-based Markov game and simultaneous-move Markov game is that the former one has pure NE strategy, which saves the AB factor when Cauchy-Schwarz inequality is applied (See the proof of
Theorem F.6).
C ONCLUSION
In this work, we study the minimal dataset coverage assumption for NE learning in two-player zero-sum Markov games.
We show that single strategy concentration is not enough for NE learning.
Instead, we find a minimal coverage assumption for NE learning and design an algorithm with sample complexity tight in C ∗ , S , H under such assumption based on novel techniques.
In addition, the algorithm can achieve minimax sample complexity in certain settings.
We believe this work can shed new light on offline MARL.
Here we list several open problems for future work.
One direction is to find the minimax sample complexity of offline Markov games under unilateral concentration.
Importantly, it is unclear whether
AB factor can be reduced to A + B as in the online setting Bai et al.
Another direction is to design decentralized algorithm for offline MARL.
The answer to this question is especially crucial if we want to further study offline MARL with a large number of agents and we do not want the sample complexity scales exponentially with the number of agents.
Lastly, in this paper we only focus on the most fundamental tabular setting.
It is natural to ask how to generalize our findings, especially the unilateral concentration assumption, to the function approximation setting.
A A LGORITHMS
Input: Offline dataset D = (cid:8) ( s kh , a kh , b kh , r kh , s kh +1 ) (cid:9) n,H k,h =1 .
Failure Probability δ .
Initialization: Set V H +1 ( · ) =
V H +1 ( · ) = 0 .
Randomly split the dataset D into {D h } Hh =1 with |D h | = n/H .
Set (cid:98) P h , (cid:98) r h , b h and b h as (2), (3) and (4) using the dataset D h for all h ∈
[ H ] .
for time h = H, H − 1 , . .
. , 1 do
Set Q h ( · , · , · ) as (5).
Compute the NE of Q h ( · , · , · ) as ( µ h ( · ) , ν h ( · )) .
Compute V h ( · ) =
E a ∼ µ h ,b ∼ ν h Q h ( · , a, b ) .
Set Q h ( · , · , · ) as (6).
Compute the NE of Q h ( · , · , · ) as ( µ h ( · ) , ν h ( · )) .
Compute V h ( · ) =
E a ∼ µ h ,b ∼ ν h Q h ( · , a, b ) .
end for
Output µ = ( µ 1 , µ 2 , · · · , µ H ) , ν = ( ν 1 , ν 2 , · · · , ν H ) , { V h } H h
=1 , { V h } H h =1 .
Input: Dataset D = (cid:8) ( s kh , a kh , b kh , r kh , s kh +1 ) (cid:9) n,H k,h =1 .
Failure Probability δ .
Initialization: Randomly split the dataset D into D ref , D 0 , {D h, 1 } Hh =1 with |D ref | = n/ 3 , |D 0 | = n/ 3 |D h | = n/ (3 H ) .
Set V H +
= V H +
= 0 .
Learn the reference value function V ref , V ref ← POVI( D ref ) .
Set (cid:98) P h, 0 and (cid:98) r h, 0 as (2) and (3) using the dataset D 0 for all h ∈
[ H ] .
Set (cid:98) P h, 1 and (cid:98) r h, 1 as (2) and (3) using the dataset D h, 1 for all h ∈
[ H ] .
Set b h, 0 and b h, 0 as (9) using the dataset D 0 for all h ∈
[ H ] .
for time h = H, H − 1 , . .
. , 1 do
Set b h, 1 and b h, 1 as (10) using the dataset D h, 1 for all h ∈
[ H ] .
Set Q h ( · , · , · ) as (7).
Compute the NE of Q h ( · , · , · ) as ( µ h ( · ) , ν h ( · )) .
Compute V h ( · ) =
E a ∼ µ h ,b ∼ ν h Q h ( · , a, b ) .
Set Q h ( · , · , · ) as (8).
Compute the NE of Q h ( · , · , · ) as ( µ h ( · ) , ν h ( · )) .
Compute V h ( · ) =
E a ∼ µ h ,b ∼ ν h Q h ( · , a, b ) .
end for
Output: µ = ( µ 1 , µ 2 , · · · , µ H ) , ν = ( ν 1 , ν 2 , · · · , ν H ) .
B R ELATED W ORK
Here we focus on the theoretical work on two-player zero-sum Markov games and offline RL.
Two-player zero-sum Markov game.
Zero-sum Markov game has been widely studied since the seminal work (Shapley, 1953).
When the transition kernel is unknown, different sampling oracles are utilized to acquire samples, including online sampling (Bai & Jin, 2020;
Xie et al., 2020a;
Liu et al., 2021; Bai et al., 2020; Jin et al., 2021a; Song et al., 2021), generative model sampling (Sidford et al., 2020; Cui & Yang, 2020; Zhang et al., 2020; Jia et al., 2019).
For offline sampling oracle, Zhang et al.
(2021b) provides finite sample bound for a decentralized algorithm with network communication under uniform concentration assumption and Abe & Kaneko (2020) considers offline policy evaluation, again under the uniform concentration assumption.
None of these works considers the minimal dataset assumption that allows NE learning in zero-sum Markov games.
Offline single-agent RL.
Theoretical analysis of offline RL can be traced back to Szepesv´ari &
Munos (2005), under the uniform concentration assumption (analogue to Assumption 2.3).
This assumption has been extensively investigated (Xie & Jiang, 2021;
Xie et al., 2020b; Yin et al., 2020; 2021; Ren et al., 2021).
Recently, a line of works showed that the pessimism principle allows offline policy optimization under a much weaker assumption, single policy concentration, both in tabular case and with function approximation (Rashidinejad et al., 2021;
Yin & Wang, 2021; Xie et al., 2021; Jin et al., 2021b; Uehara & Sun, 2021; Uehara et al., 2021; Zanette et al., 2021).
One closely related work is Xie et al.
(2021), which utilizes the reference advantage function technique and bernstein-type bonus to show a minimax sample complexity (cid:101)
O ( SC ∗ H 3 /n )
in finite-horizon MDP.
We show that the counterpart of single policy concentration in zero-sum Markov game is insufficient for NE strategy learning and use the pessimism principle to design algorithm that works under the unilateral concentration assumption.
C P ROOF IN S ECTION
Proof of Theorem 3.1.
We consider a bandits game with two actions for each player here.
The action set is A = { a 1 , a 2 } for the first (max) player and B = { b 1 , b 2 } for the second (min) player.
We construct the following two bandit games with deterministic rewards.
Then the unique NE of the
[Equation]
Bandits Game
[Equation]
Bandits Gamefirst bandits game is ( a 1 , b 1 ) and the unique NE of the second bandits game is ( a 2 , b 2 ) .
Now we set the exploration strategy ρ to be
[Equation]
It is straightforward to verify that both bandits games with exploration strategy ρ is in the class X .
However, as the dataset only provides information about ( a 1 , b 1 ) and ( a 2 , b 2 ) , which have the same reward, it is impossible for any algorithm to distinguish between these two bandits games.
Suppose the output of ALG is π = ( µ, ν ) with µ ( a 1 )
= p , µ ( a 2 ) =
1 − p and ν ( b 1 ) =
q , ν ( b 2 ) = 1 − q , then we have that π is an 0 .
5(2 − p − q ) -NE for the first bandits game and 0 .
5( p + q ) -NE for the second bandits game.
As a result, there exists a bandits game such that ALG only outputs an at most 0 .
5 -approximate NE strategy.
Proof of Theorem 3.4.
Similar to the proof of Theorem 3.1, we construct the following two bandits games with deterministic rewards.
Then the (unique) NE of the first bandits game is ( a 1 , b 1 ) and the r ( a 1 , b 1 ) = 0 .r
( a 1 , b 2 )
= 0 .r
( a 2 , b 1 ) =
r ( a 2 , b 2 ) = 0 .
Bandits Gamer ( a 1 , b 1 ) = 0 .r
( a 1 , b 2 )
= 0 .r
( a 2 , b 1 ) =
r ( a 2 , b 2 ) = 0 .
Bandits Game
(unique) NE of the second bandits game is ( a 2 , b 2 ) .
Now we set the exploration strategy ρ to be ρ ( a, b ) = (cid:26)( a, b ) =
( a 2 , b 1 ) , 1 / 3 otherwise .
It is straightforward to verify that both bandits game with exploration strategy ρ is in the class defined in Theorem 3.1.
The NE for the first bandits game is ( a 1 , b 1 ) and the NE for the second bandits game is ( a 2 , b 2 ) .
The dataset contains data on ( a 1 , b 1 ) , ( a 1 , b 2 ) , ( a 2 , b 2 ) and no data on ( a 2 , b 1 ) .
It is impossible for an algorithm to distinguish between these two bandits games as they are consistent on the given dataset and they all satisfy the dataset coverage assumption that only one action is not covered.
The rest of the proof follows the proof of Theorem 3.1 and with some computation we can show that the output of ALG must be a 0.25-approximate NE for one of the bandits games.
D P ROOF IN S ECTION 4.
Lemma D.1.
(Concentration)
With probability 1 − δ , we have
[Equation]
holds for all h ∈
[ H ] , s ∈ S , a ∈ A and b ∈ B .
We define this as the good event G .
Proof.
We provide the proof for the first argument and the proof for the second argument holds similarly.
For all s, a, b, h , we have
[Equation]
as whenever n h ( s, a, b ) ≥ 1 , (cid:98) r h ( s, a, b )
[Equation]
h ( s, a, b ) .
For the concentration on (cid:68) (cid:98) P ( ·| s, a, b ) , V h +1 ( · ) (cid:69) , note that V h +1 only depends on the dataset {D t }
Ht = h
+1 while (cid:98) P h ( ·| s, a, b ) only depends on the dataset D h , which means they are independent and then Hoeffding’s inequality can be applied:
[Equation]
The second argument holds similarly.
For the third argument, the proof is from Lemma B.1 in Xie et al.
Lemma D.2.
(Pessimism)
Under the good event G , we have that V h ( s ) ≤ V µ, ∗ h ( s ) and V h ( s ) ≥
V ∗ ,ν h ( s ) hold for all h ∈
[ H ] and s ∈ S .
Proof.
We prove this lemma by induction.
The inequalities trivially hold for h
= H + 1 .
If the inequalities hold for timestep h + 1 , now we consider timestep h .
By the definition of Q h ( s, a, b ) , we have
[Equation]
where the inequality is from Lemma D.1.
With the pessimism on the state-action value function, we can prove the pessimism on the state value function.
[Equation]
where the first inequality is from the definition of NE and the second inequality is from the pessimism of the state-action value function.
The arguments for V h hold similarly.
Then by mathematical induction we can prove the lemma.
Lemma D.3.
Under the good event G , for all h ∈ [ H ] and s h ∈ S , we have
[Equation]
Proof.
We prove the first argument and the second argument can be proven similarly.
By the definition of NE, we have V ∗ h ≤ V µ ∗ ,ν h .
Combined with Lemma D.2, we have the first inequality.
For the second inequality, we have
[Equation]
=
[Equation]
=
[Equation]
[Equation]
[Equation]
(cid:68) (cid:98)
[Equation]
[Equation]
s
[Equation]
a
[Equation]
h )
[Equation]
V h +1 ( · ) (cid:69) + b h ( s h
[Equation]
a h
[Equation]
b h ) (cid:105)
[Equation]
(Lemma D.1)
[Equation]
[Equation]
(cid:34)
[Equation]
(cid:88)
[Equation]
b
[Equation]
a
[Equation]
b
[Equation]
[Equation]
s
[Equation]
(cid:35) .
Theorem D.4.
Suppose Assumption 2.2 holds.
For any 0 < δ < 1 , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 1 satisfies
[Equation]
n .
As a result, we have
[Equation]
Proof.
By Lemma D.3, with probability at least 1 − δ , we have
[Equation]
D
[Equation]
[Equation]
Cauchy-Schwarz Inequality)
[Equation]
n
[Equation]
Similarly we have
[Equation]
n
[Equation]
As a result, we have
[Equation]
Theorem D.5.
Suppose Assumption 2.2 holds.
For any 0 < δ < 1 and strategy µ, ν , with probability 1 − δ , the pessimistic value V h and optimistic estimate V h of Algorithm 1 satisfies
[Equation]
where s h is sampled from the trajectory following the strategy in the expectation at timestep h .
Proof.
We prove the first argument and the second argument can be proven similarly.
By Lemma
D.3, under good event G for all state s we have
[Equation]
We define ν ′ =
( ν 1 , · · · , ν h − 1 , ν h , · · · , ν H ) .
Then we have
[Equation]
=
H (cid:88) t = h
E µ ∗ ,ν ′ [ b h ( s t , a t , b t )] .
Then following the proof of Theorem D.4, we can prove the argument.
E P ROOF IN S ECTION 4.
For simplicity, we only provide the guarantee for the max player and the guarantee for the min player can be proven in a similar manner.
Lemma E.1.
(Concentration)
There exists some absolute constant c > 0 such that the concentration event G ′ holds with probability at least 1 − δ , where
[Equation]
  (cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
n
[Equation]
+
Hι n
[Equation]
  ,
[Equation]
 
[Equation]
n
[Equation]
+
Hι n
[Equation]
Proof.
The proof is a direct application of Lemma C.1 in Xie et al.
(2021) with s, a replaced by s, a, b .
Lemma E.2.
For all h ∈ [ H ] and s ∈ S , we have V h ( s ) ≥ V ref h ( s ) .
Proof.
By the update rule (7), we have Q h ( s, a, b ) ≥ Q ref h ( s, a, b ) for h ∈
[ H ] and s, a, b ∈ S×A×B .
Then by the definition of NE, we have
[Equation]
Lemma E.3.
(Pessimism)
Under the good event G ′ , we have that V h ( s ) ≤ V µ, ∗ h ( s ) holds for all h ∈
[ H ] and s ∈ S .
Proof.
We prove this lemma by induction.
The inequalities trivially hold for h
= H + 1 .
If the inequalities hold for h + 1 , now we consider h .
[Equation]
∨ Q ref h ( s, a, b ) ≤ max (cid:110) r h ( s, a, b ) + ( P h · V ref h +1 )
( s, a, b ) + (cid:16) P h · (cid:16) V h
+1 − V ref h +1 (cid:17)(cid:17)
( s, a, b ) , Q ref h ( s, a, b ) (cid:111) = max (cid:110) r h
( s, a, b )
+ ( P h · V h +1 )
( s, a, b ) , Q ref h ( s, a, b ) (cid:111) ≤ max (cid:110) r h ( s, a, b )
+ ( P h · V h +1 )
( s, a, b ) , r h ( s, a, b )
+ ( P h · V ref h +1 )
( s, a, b ) (cid:111) (Lemma D.2)
≤ r h ( s, a, b )
+ ( P h · V h +1 )
( s, a, b ) (Lemma E.2) ≤ r h ( s, a, b )
+ ( P h · V µ, ∗ h +1 )
( s, a, b ) (Induction hypothesis) =
Q µ, ∗ h ( s, a, b ) .
Then by the definition of NE
[Equation]
we have
[Equation]
With mathematical induction we can prove the lemma
[Equation]
Lemma E
[Equation]
Under the good event G ′ , we have
[Equation]
Proof.
[Equation]
where the last inequality is from telescoping the timestep H
[Equation]
Lemma E.5.
For any strategy ν
[Equation]
we have
[Equation]
Proof.
[Equation]
, a h , b h (cid:3)(cid:3) =
H (cid:88) h
=
E µ ∗ ,ν (cid:104) E (cid:104)(cid:0) V ∗ h
+
( s h +1 )
+ r h
( s h , a h , b h ) − V ∗ h ( s h ) (cid:1) 2 |
s h , a h , b h (cid:105)(cid:105) =
H (cid:88) h
=
E µ ∗ ,ν (cid:104)(cid:0) V ∗ h
+
( s h +1 )
+ r h
( s h , a h , b h ) − V ∗ h ( s h ) (cid:1) 2 (cid:105) =
E µ ∗ ,ν   (cid:32) H (cid:88)
h
= (cid:0) V ∗ h
+
( s h +1 )
+ r h
( s h , a h , b h ) − V ∗ h ( s h ) (cid:1)(cid:33) 2  
[Equation]
Lemma E.6.
The output strategy π = ( µ, ν ) and the pessimistic estimate V of Algorithm 1 satisfy
[Equation]
Proof.
We prove the argument for h = 2 first
[Equation]
We can prove the lemma for arbitrary h by telescoping the argument to timestep h .
Lemma E.7.
For n ≥ (cid:101) O ( √ C ∗ SABH 3 ) , we have
E µ ∗ ,ν
H (cid:88)
[Equation]
O
[Equation]
n
[Equation]
Proof.
E µ ∗ ,ν
H (cid:88) h
= b {+h, 
+}
( s h , a h , b h ) =
c E µ ∗ ,ν
H (cid:88) h =   (cid:118) (cid:117)(cid:117)(cid:116)
Var (cid:98) P h, 0 ( s,a,b ) ( V ref h +1 )
ι n h, 0 ( s, a, b ) ∨+
Hι n h, 0 ( s, a, b ) ∨  ≤ c E µ ∗ ,ν
H (cid:88) h
=   (cid:115)
c
Var P h ( s,a,b ) ( V ref h +1 ) ι nd ρh
[Equation]
s [Equation] + cHι nd
[Equation]
cHι nd
[Equation]
  = c
[Equation]
n
[Table]
[Equation]
n (Lemma E.6 and Theorem D.5)
[Equation]
n .
Lemma E.8.
For n ≥ (cid:101) O ( √ C ∗ SABH 4 ) , we have
[Equation]
n
[Equation]
Proof.
E µ ∗ ,ν
H (cid:88) h
= b h, 1 ( s h , a h , b h ) =
c E µ ∗ ,ν
H (cid:88) h =   (cid:118)(cid:117)(cid:117)(cid:116)
Var (cid:98) [Equation] ( s [Equation]
a,b ) ( V h
[Equation]
1 − V ref h +1 )
ι n
[Equation] b ) [Equation] Hι n
[Equation] b ) [Equation]   ≤ c E µ ∗ ,ν
H (cid:88) h
=   (cid:115) cH
Var P h ( s,a,b ) ( V h
+1 − V ref h +1 )
ι nd
[Equation]
cH [Equation] nd [Equation]

 ≤ c 2 E µ ∗ ,ν
H (cid:88) h =   (cid:118)(cid:117)(cid:117) (cid:116) H (cid:104)
[Equation]
(cid:105) [Equation] ι [Equation]
=
[Equation] H (cid:88) h
[Equation]
(cid:88) (
[Equation] [Equation]
[Equation]
  (cid:118)(cid:117)(cid:117)(cid:116) H (cid:104)
[Equation]
h
[Equation]
(cid:105)
[Equation]

[Table]
[Equation]
Theorem E.9.
Suppose Assumption 2.2 holds.
For any 0 < δ < 1 and n ≥ (cid:101) O ( √ C ∗ SABH 4 ) , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 1 satisfies
[Equation]
n
[Equation]
As a result, we have
[Equation]
O
[Equation]
n [Equation]
Proof.
V µ ∗ ,ν
[Equation]
E [Equation] [Equation] O
[Equation]
n
[Equation]
n
[Equation]
(Lemma E.7 and Lemma E.8) ≤ [Equation] n [Equation]
n
[Equation]
(Lemma G.5)
[Equation]
n
[Equation]
By the definition of NE, we have
[Equation]
The second argument can be proven in a similar manner.
Combining these two argument and we can prove that
[Equation]
F P ROOFS IN S ECTION 4. F. U NIFORM C OVERAGE
Theorem
F.1.
Suppose
Assumption 3. holds and d m = min { d ρh ( s, a, b ) :
h ∈ [ H ] , ( s, a, b ) ∈ S ×
A × B} .
For any 0 < δ < 1 , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 1 satisfies
[Equation]
As a result, we have
[Equation]

Proof.
By Lemma D.3, with probability 1 − δ we have
[Equation]
n
[Equation]
(cid:35) ≤
H (cid:88) h
=
E µ ∗ ,ν (cid:34) (cid:115)
H 3 ιnd ρh ( s, a, b ) (cid:35) (Lemma D.1) =
H (cid:88) h
= (cid:88) ( s,a,b ) d µ ∗ ,ν h ( s, a, b ) (cid:34) (cid:115)
H 3 ιnd ρh ( s, a, b ) (cid:35) ≤
H (cid:88) h
= (cid:88) ( s,a,b ) d µ ∗ ,ν h ( s, a, b )   (cid:115)
H 3 ιnd m 
 ≤(cid:118) (cid:117) (cid:117) (cid:116)
H (cid:88) h
= (cid:88) ( s,a,b ) d µ ∗ ,ν h ( s, a, b ) · (cid:115)(cid:80) H h
=1 (cid:80) ( s,a,b ) d µ ∗ ,
ν h ( s, a, b ) C ∗ H 3 ιnd m (Cauchy-Schwarz Inequality)
= √ H · (cid:115) H 4 ιnd m
[Equation]
Theorem
F.2.
Suppose
Assumption 3. holds and d m = min { d ρh ( s, a, b ) :
h ∈ [ H ] , ( s, a, b ) ∈ S ×
A × B} .
For any 0 < δ < 1 and strategy µ, ν , with probability 1 − δ , the pessimistic value V h and optimistic estimate V h of Algorithm 1 satisfies
[Equation]
where s h is sampled from the trajectory following the strategy in the expectation at timestep h .
Proof.
By Lemma D.3, under good event G for all state s we have
[Equation]
We define ν ′ =
( ν 1 , · · · , ν h − 1 , ν h , · · · , ν H ) .
Then we have
E
[Equation]
[Equation]
∗
[Equation]
[Equation]
[Equation]
Then following the proof of Theorem F.1, we can prove the argument.
Lemma
F.3.
Suppose
Assumption 3. holds and
[Equation]
min { d ρh ( s, a, b ) : h ∈ [ H ] , ( s, a, b ) ∈ S ×
A × B} .
For n ≥ (cid:101) O
( (cid:112) H 3 /d m ) , we have
E µ ∗ ,ν
H (cid:88) h
= b h,
( s h , a h , b h ) ≤ (cid:101) O   (cid:115)
Hnd m (cid:113) V µ ∗ ,ν( s 1 ) − V 1 ( s 1 ) 
 + (cid:101)
O   (cid:115)
Hnd m 
 .
Proof.
E µ ∗ ,ν
H (cid:88) h
= b h,
( s h , a h , b h ) =
c E µ ∗ ,ν
H (cid:88) h =   (cid:118)(cid:117)(cid:117)(cid:116)
Var (cid:98) P h, 0 ( s,a,b ) ( V ref h +1 )
ι n h, 0 ( s, a, b ) ∨+
Hι n h, 0 ( s, a, b ) ∨  ≤ c E µ ∗ ,ν
H (cid:88) h
=   (cid:115)
c
Var P h ( s,a,b ) ( V ref h +1 )
ι nd ρh ( s, a, b ) + cHι nd ρh ( s, a, b ) + cHι nd ρh ( s, a, b ) 
 ≤ c
H (cid:88) h
= (cid:88) ( s,a,b ) d µ ∗ ,ν h ( s, a, b )   (cid:115)
Var P h ( s,a,b ) ( V ref h +1 )
ι nd m + Hι nd m 

[Table]
[Equation]
−
[Equation]
ref
[Equation]
nd
[Equation]
Lemma E.5)
=
c
[Equation]
√
[Equation]
[Equation]
≤
[Equation]
(Lemma E.6 and Theorem F.2)
[Equation]
Lemma F.4.
For n ≥ (cid:101) O
( (cid:112) H 4 /d m ) , we have E µ
[Equation]
H (cid:88) h
= b h
[Equation]
[Equation]
s h
[Equation]
a
[Equation]
b h ) ≤
[Equation]
O  
[Equation]
nd
[Equation]

[Equation]
Proof.
E µ ∗ ,ν
H (cid:88) h
= b h, 1 ( s h , a h , b h ) =
c E µ ∗ ,ν
H (cid:88) h =   (cid:118)(cid:117)(cid:117)(cid:116)
Var (cid:98) P h, 0 ( s,a,b ) ( V h
+1 − V ref h +1 )
ι n h, 1 ( s, a, b ) ∨+
Hι n h, 1 ( s, a, b ) ∨  ≤ c E µ ∗ ,ν
H (cid:88) h
=   (cid:115) cH
Var P h ( s,a,b ) ( V h
+1 − V ref h +1 )
ι nd ρh ( s, a, b ) + cH 2 ι nd ρh ( s, a, b ) + cH 2 ι nd ρh ( s, a, b ) 
 ≤ c 2 E µ ∗ ,ν
H (cid:88) h =   (cid:118)(cid:117)(cid:117)(cid:116) H (cid:104) P h ( V h
+1 − V ref h +1 ) 2 (cid:105) ( s, a, b ) ι nd ρ h ( s, a, b ) +
H 2 ι nd ρ h ( s, a, b ) 
 ≤ c
H (cid:88) h
= (cid:88) ( s,a,b ) d µ ∗ ,ν h ( s, a, b )   (cid:118)(cid:117)(cid:117)(cid:116)
H (cid:104) P h
( V h
+1 − V ref h +1 ) 2 (cid:105) ( s, a, b ) ι nd m + H 2 ι nd m  
[Table]
[Equation] [Equation]
Theorem
F.5.
Suppose
Assumption 3. holds and d m = min { d ρh ( s, a, b ) :
h ∈ [ H ] , ( s, a, b ) ∈ S ×
A × B} .
For any 0 < δ < 1 and n ≥ (cid:101) O
( (cid:112) H 4 /d m ) , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 2 satisfies
[Equation]
As a result, we have
[Equation]


[Equation]
Proof.
V
[Equation]
≤
[Equation]
O  
[Equation]
nd m (cid:113)
[Equation]
µ
[Equation]
( s [Equation] [Equation] V
[Equation]
s
[Equation]
) 
[Equation]
(cid:101)
O   (cid:115)
Hnd m 
 (Lemma F.3 and Lemma F.4) ≤ (cid:101)
O   (cid:115)
Hnd m 
 + (cid:101) O (cid:18)
Hnd m (cid:19) (Lemma G.5)
=
(cid:101)
O   (cid:115)
Hnd m 
 .
By the definition of NE, we have
[Equation]
The second argument can be proven in a similar manner.
Combining two arguments together and we can derive that
[Equation]
For turn-based Markov game, there always exists a pure (deterministic) NE equilibrium strategy.
As a result, we can have that µ ∗ , ν ∗ , µ , ν , µ , ν are all pure strategy.
Theorem F.6.
Suppose Assumption 2.2 holds.
For any 0 < δ < 1 , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 1 satisfies
[Equation]
As a result
[Equation]
we have
[Equation]
Proof.
By Lemma D.3, with probability 1 − δ we have
[Equation]
[Equation]
Lemma D.1) =
[Equation] [Equation]
[Equation]
are deterministic strategy.)
[Equation] [Equation]
Cauchy-Schwarz Inequality)
=
[Equation]
n .
Theorem F.7.
Suppose Assumption 2.2 holds.
For any 0 < δ < 1 and policy µ, ν , with probability 1 − δ , the pessimistic value V h of Algorithm 1 satisfies
[Equation]
where s h is sampled from the trajectory following the strategy in the expectation at timestep h .
Proof.
By Lemma D [Equation] [Equation] under good event G for all state s we have
[Equation]
We define ν ′ =
( ν 1 , · · · , ν h − 1 , ν h , · · · , ν H ) .
Then we have
[Equation]
Then following the proof of Theorem F.
[Equation]
we can prove the argument.
Lemma F.8.
For n ≥ (cid:101) O ( √ C ∗ SH 3 ) , we have
[Equation]
( s 1 ) − V 1 ( s 1 ) (cid:33) + (cid:101) O (cid:32)(cid:114) C ∗ SHn (cid:33) .
Proof.
E µ ∗ ,ν
H (cid:88) h
= b h,
( s h , a h , b h ) =
c E µ ∗ ,ν
H (cid:88) h =   (cid:118) (cid:117)(cid:117) (cid:116)
Var (cid:98) P h, 0 ( s,a,b ) ( V ref h +1 )
ι n h, 0 ( s, a, b ) ∨+
Hι n h, 0 ( s, a, b ) ∨  ≤ c E µ ∗ ,ν
H (cid:88) h
=   (cid:115)
c
Var P h ( s,a,b ) ( V ref h +1 )
ι nd ρh ( s, a, b ) + cHι nd ρh ( s, a, b ) + cHι nd ρh ( s, a, b ) 
 =
c
H (cid:88) h
= (cid:88) s ∈S d µ ∗ ,
ν h ( s, µ ∗ ( s ) , ν ( s ))   (cid:115)
Var P h ( s,µ ∗ ( s ) ,ν ( s ))
( V ref h +1 )
ι nd ρh ( s, µ ∗ ( s ) , ν ( s ))
+
Hι nd ρh ( s, µ ∗ ( s ) , ν ( s )) 
 ≤ c
H (cid:88) h
= (cid:88) s ∈S   (cid:115) C ∗ d µ ∗ ,
ν h ( s, µ ∗ ( s ) , ν ( s ))
Var P h ( s,µ ∗ ( s ) ,ν ( s ))
( V ref h +1 )
ι n + C ∗ Hι n 
 ( µ ∗ , ν are deterministic strategies.)
[Equation]
n
[Equation]
n
[Table]
[Equation]
n (Lemma E.6 and Theorem F.7)
[Equation]
Lemma F.9.
For n ≥ (cid:101) O ( √ C ∗ SABH 4 ) , we have
[Equation]
n
[Equation]
Proof.
E µ ∗ ,ν
H (cid:88) h
= b h, 1 ( s h , a h , b h ) =
c E µ ∗ ,ν
H (cid:88) h =   (cid:118)(cid:117)(cid:117)(cid:116)
Var (cid:98) P h, 0 ( s,a,b ) ( V h
+1 − V ref h +1 )
ι n h, 1 ( s, a, b ) ∨+
Hι n h, 1 ( s, a, b ) ∨  ≤ c E µ ∗ ,ν
H (cid:88) h
=   (cid:115) cH
Var P h ( s,a,b ) ( V h
+1 − V ref h +1 )
ι
[Equation]
cH
[Equation]
nd
[Equation]

 ≤ c 2 E µ ∗ ,ν
H (cid:88) h =   (cid:118)(cid:117)(cid:117)(cid:116)
[Equation]

[Equation]
 
[Equation]
-Schwarz Inequality)
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Theorem F.
[Table]
Suppose Assumption 2.2 holds for a turn-based Markov game.
For any 0 < δ < 1 , with probability 1 − δ , the output policy π = ( µ, ν ) of Algorithm 1 satisfies
[Equation]
s 1 ) − V µ [Equation]
∗
[Equation]
( s
[Equation]
O (cid:32)(cid:114)
[Equation]
SHn
[Equation]
V
[Equation]
O (cid:32)(cid:114)
[Equation]
n
[Equation]
As a result, we have
Gap( µ, ν [Equation]
O (cid:32)(cid:114)
[Equation]
SHn (cid:33) .
Proof.
V
[Equation]
ν
[Equation]
h
[Equation]
b
[Equation]
) (Lemma E.4)
[Equation]
O (cid:32)(cid:114)
[Equation]
n
[Equation]
(cid:33)
[Equation]
(cid:101) O (cid:32)(cid:114) C
[Equation]
n (cid:33) (Lemma F.8 and Lemma F.9) ≤ (cid:101) O (cid:32)(cid:114)
[Equation]
n [Equation] +
[Equation]
O (cid:18) C
[Equation]
n (cid:19) (Lemma G.5)
= (cid:101) O (cid:32)(cid:114) C
[Equation]
SHn (cid:33) .
By the definition of NE, we have
[Equation]
n (cid:33) .
The second argument can be proven in a similar manner.
Combining these two arguments and we can derive that
[Equation]
G A UXILIARY L EMMAS
Lemma G.1.
(Multiplicative Chernoff bound).
Let X be a binomial random variable with parameter p , n .
For any 1 ≥ θ > 0 , we have that
[Equation]
Lemma G.2.
For all (s h , a h , b h ) ∈ K h and any ∥ V ∥ ∞ ≤ H , with probability 1 − δ we have
[Equation]
Var
[Equation]
Proof.
The is a direct application of Lemma G.3 with a union bound.
Lemma G.3.
(Empirical Berstein Inequality (Maurer & Pontil, 2009)) Let n ≥ 2 and V ∈ R S be any functions with ∥ V ∥ ∞ ≤ H , P be any S -dimensional distribution and (cid:98) P be its empirical version using n samples.
Then with probability 1 − δ ,
[Equation]
Lemma G.4.
For 0 ≤ V ≤ V ′ ≤ H , we have
[Equation]
Proof.
[Equation]
[Equation]
s
[Equation]
a
[Equation]
Lemma G.5.
If x ≤ a √ x + b for a, b > 0 , then we have x ≤ 2 a
+ 2 b.
Proof.
We have
[Equation]
If √ x < a 2 , the argument holds directly.
Otherwise we have
[Equation]
So we have √ x ≤ √ b + a , which implies x ≤ 2( a 2 + b ) .