When are Ofﬂine Two-Player Zero-Sum Markov Games Solvable?
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study what dataset assumption permits solving ofﬂine two-player zero-sum
Markov games.
In stark contrast to the ofﬂine single-agent Markov decision process, we show that the single strategy concentration assumption is insufﬁcientfor learning the Nash equilibrium (NE) strategy in ofﬂine two-player zero-sum
Markov games.
On the other hand, we propose a new assumption named unilateral concentration and design a pessimism-type algorithm that is provably efﬁcientunder this assumption.
In addition, we show that the unilateral concentration assumption is necessary for learning an NE strategy.
Furthermore, our algorithmcan achieve minimax sample complexity without any modiﬁcation for two widelystudied settings: dataset with uniform concentration assumption and turn-based
Markov games.
Our work serves as an important initial step towards understanding ofﬂine multi-agent reinforcement learning.
Introduction
Promising empirical advances have been achieved in reinforcement learning (RL), including mastering the game of Go
[Silver et al., 2016],
Poker [Brown et al., 2017], real-time strategy games[Vinyals et al., 2019] and robotic control [Kober et al., 2013].
Notably, many of these successeslie in the domain of multi-agent reinforcement learning (MARL).
MARL is about multiple agentsinteracting in a shared environment and each of them aims to maximize its own long-term reward.
During the learning process, each agent not only needs to identify the environment dynamic but alsoneeds to compete/cooperate with other agents.
One important subarea of MARL is ofﬂine MARL.
Inmany practical scenarios, we only have access to the ofﬂine data or it is too expensive to frequentlychange the policy [Zhang et al., 2021a].
While there are plenty of empirical works on ofﬂine MARL[Pan et al., 2021, Jiang and Lu, 2021], the theoretical understanding is still very limited.
In thiswork, we take an initial step towards understanding when ofﬂine MARL is provably solvable.
We consider two-player zero-sum Markov games, where two players sequentially select actions ina Markovian environment and the ﬁrst player aims to maximize the total reward while the secondplayer aims to minimize it.
In the ofﬂine setting, we have access to a ﬁxed dataset collected bya (possibly unknown) exploration policy and the target is to ﬁnd a (near-)Nash equilibrium (NE)strategy of the underlying two-player zero-sum Markov game.
One of the main difﬁculties in ofﬂine RL is distribution shift, i.e., the dataset distribution is differentfrom the distribution induced by the optimal policy.
It is important to understand what is the minimaldataset distribution assumption that permits ofﬂine RL.
For single-agent ofﬂine RL, it is shownthat the pessimism principle allows policy optimization with single policy concentration , i.e. thedataset only covers the optimal policy [Jin et al., 2021b, Zanette et al., 2021, Yin and Wang, 2021, al., 2021].
This assumption is necessary as it is impossible to learn the optimal policy if it is not covered by the dataset.
However, the dataset coverage assumption for MARL isstill far from clear.
In this work, we want to answer the following question:
What is the minimal dataset coverage assumption that permits learning an NE strategy in ofﬂinetwo-player zero-sum Markov games?
Generally speaking, MARL is much more difﬁcult than single-agent RL due to the following tworeasons.
First, MARL is known to suffer from the non-stationary property, i.e. agents will affect theothers during the learning process [Zhang et al., 2021a].
Speciﬁcally, the performance may declineif each agent simultaneously tries to improve its own policy depending on others’ current policies.
Inaddition, multiple agents incur complicated statistical dependence that makes the theoretical analysisdifﬁcult.
A line of works study Markov games with online sampling oracle [Bai et al., 2020, Bai and
Jin, 2020, Liu et al., 2021] or generative model oracle [Sidford et al., 2020, Zhang et al., 2020, Cuiand Yang, 2020], where specialized techniques are developed to tackle the above difﬁculties.
In thispaper, we give the ﬁrst analysis on ofﬂine Markov games in the fundamental tabular setting.
1.1 Main Contributions
• First, we propose an assumption named unilateral concentration , which posits that for all strategiesµ , ⌫ , strategy pairs ( µ ⇤ , ⌫ ) and ( µ, ⌫ ⇤ ) are covered by the dataset, where µ is the strategy for the
ﬁrst (max) player, ⌫ is the strategy for the second (min) player, and ( µ ⇤ , ⌫ ⇤ ) is an NE strategy.
In Section 3, we prove that NE strategy is not learnable even if this assumption is only slightlyviolated.
The intuition behind the hardness result is that to identify an NE strategy, the algorithmhas to compare it with strategy pairs that one player uses any other strategies as a reference.
Thisresult also implies that the single strategy concentration, which is sufﬁcient for ofﬂine single-agent
RL, is not sufﬁcient for ofﬂine MARL
• Second, we provide positive results showing that NE strategy is PAC learnable under the unilateralconcentration assumption.
Combined with the hardness results above, we conclude that unilateral concentration assumption is the necessary and sufﬁcient dataset coverage assumption for solving of- ﬂine Markov games .
Our algorithm is based on the pessimism principle that we maintain pessimisticestimates for both players, respectively.
We show that our algorithm achieves
e O ( p
C ⇤ SABH 3 /n )performance gap under unilateral concentration assumption, where C ⇤ quantiﬁes the coverage ofthe dataset, S is the number of states, A is the number of the max player’s actions, B is the numberof the min player’s actions, H is the horizon and n is the number of samples.• Third, we show that our algorithm is minimax optimal when the dataset satisﬁes a stronger as-
sumption, uniform concentration or the Markov game is turn-based.
These are two widely studied settings in the RL community.
Uniform concentration assumes that all state-action pairs are coveredby the dataset and turn-based Markov game is a variant of zero-sum Markov games where two players select actions in turns instead of simultaneously.
Although uniform concentration is about thedataset structure and turn-based Markov games are about the environment structure, our algorithmcan adapt to both of them without any modiﬁcation and achieves minimax sample complexity.
Main Techniques.
Our algorithm is motivated by the Bernstein-type bonus and reference advantagefunction techniques in Xie et al.
[2021] while we make novel adaptations, namely monotonic updateand a self-bounding technique, to realize them in Markov games.
The Monotonic update allows asandwich-type argument that bounds the reference function and further bounds the variance term.
The self-bounding technique is utilized to bound the performance gap by itself and then solve theinequality to derive the ﬁnal bound on performance gap.
To summarize, (1) we identify the minimal dataset coverage assumption that allows learning the NEstrategy in Markov games; (2) we propose a pessimism-based algorithm that achieves polynomialsample complexity based on novel Markov game techniques; and (3) we further show the algorithmis minimax optimal under the uniform concentration assumption or in turn-based Markov games.
1.2 Related Work
Here we focus on the theoretical works on two-player zero-sum Markov games and ofﬂine RL.
Two-player zero-sum Markov games.
Zero-sum Markov games have been widely studied sincethe seminal work [Shapley, 1953].
When the transition kernel is unknown, different sampling oraclesare utilized to acquire samples, including online sampling [Bai and Jin, 2020, Xie et al., 2020a, Liuet al., 2021, Bai et al., 2020, Jin et al., 2021a, Song et al., 2021], generative model sampling [Sidfordet al., 2020, Cui and Yang, 2020, Zhang et al., 2020, Jia et al., 2019].
For ofﬂine sampling oracle,
Zhang et al.
[2021b] and Abe and Kaneko [2020] consider decentralized algorithm with networkcommunication and ofﬂine policy evaluation, both under the uniform concentration assumption.
One concurrent work [Zhong et al., 2022] considers zero-sum Markov games with linear functionapproximation.
They also show the single policy coverage is not sufﬁcient and propose a similarunilateral concentration assumption under which they give a provably efﬁcient algorithm.
On theother hand, under the unilateral concentration assumption, their sample complexity is worse thanours when specialized to tabular setting because they did not use Bernstein bonus.
They show it isimpossible to learn in all instances without unilateral concentration.
However, they do not show thatany assumption weaker than unilateral concentration makes learning impossible, which is a negativeresult proven in our paper.
Lastly, our algorithm is minimax optimal for uniform concentrationsetting and turn-based Markov games while their algorithms are not.
Ofﬂine single-agent RL.
Theoretical analysis of ofﬂine RL can be traced back to Szepesv´ari and
Munos [2005], under the uniform concentration assumption (analogue to Assumption 2.3).
Thisassumption has been extensively investigated [Xie and Jiang, 2021, Xie et al., 2020b, Yin et al.
,2020, 2021, Ren et al., 2021].
Recently, a line of works showed that the pessimism principle allowsofﬂine policy optimization under a much weaker assumption, single policy concentration, both intabular case and with function approximation [Rashidinejad et al., 2021, Yin and Wang, 2021, Xieet al., 2021, Jin et al., 2021b, Uehara and Sun, 2021, Uehara et al., 2021, Zanette et al., 2021].
Oneclosely related work is Xie et al.
[2021], which utilizes the reference advantage function techniqueand Bernstein-type bonus to show a minimax sample complexity e
O ( SC ⇤ H 3 /n ) in ﬁnite-horizon
MDP.
We show that the counterpart of single policy concentration in zero-sum Markov games isinsufﬁcient for NE strategy learning and use the pessimism principle to design algorithm that worksunder the unilateral concentration assumption.
Preliminaries
2.1 Two-Player Zero-sum Markov Games
Zero-sum Markov games (MG) generalize single-agent MDP to two-agent case where one agentaims to maximize the total reward while the other one aims to minimize it.
A tabular ﬁnite-horizonzero-sum Markov game is described by the tuple G =
( S , A , B , P, r, H ) , where S is the state space,
A is the action space of the ﬁrst (max) player, B is the action space of the second (min) player,
P =
( P 1 , P 2 , · · · , P H ) , P h 2 R |S||A||B| ⇥ |S| , 8 h
[ H ] is the (unknown) transition probabilitymatrix for time step h , r = ( r 1 , r 2 , · · · , r H ) , r h 2 [0 , 1] |S||A||B| , 8 h 2 [ H ] is the (unknown)
de-
120 terministic reward vector and H is the horizon length.
* This paper focuses on the tabular settingwhere |S| , |A| ,and |B| are ﬁnite.
At each timestep h and state s h , if the max player chooses action 122 a h and the min player chooses action b h , then the next state at timestep h +1 follows the distributions h +1 ⇠ P h ( ·| s h , a h , b h ) and both players receive a reward r h ( s h , a h , b h ) .
Both players sequentiallychoose H actions and at each timestep, the action is chosen simultaneously and then it is revealed toboth players.
We assume that we have a ﬁxed initial state s
and it is straightforward to generalizeour results to the case where the initial state is sampled from a ﬁxed distribution.
Turn-based Markov games are an important subclass of (simultaneous-move) Markov games, wherethe max player takes action ﬁrst and the min player can take action after observing the opponent’saction.
It is a widely studied setting [Sidford et al., 2020, Cui and Yang, 2020, Bai and Jin, 2020]and we will provide minimax sample complexity result for this setting in Section 4.3.
We denote a strategy pair as ⇡ = ( µ, ⌫ ) , where µ = ( µ 1 , µ 2 , · · · , µ H ) , µ h : S !
A , 8 h
[ H ] is the strategy of the ﬁrst player and ⌫ = ( ⌫ 1 , ⌫ 2 , · · · , ⌫ H ) , ⌫ h : S !
B , 8 h
[ H ] is the strategy of the second player, where  X is the probability simplex on the ﬁnite set X .
A deterministic strategy is a strategy that maps state to a single point distribution.
We deﬁne the state value function and state-action value function for a strategy pair ⇡ similarly as in single-agent MDP:
[Equation]
If the second player’s strategy ⌫ is ﬁxed, then the MG degenerates to an MDP and we call the optimal policy in this MDP as the best response strategy br 1 ( ⌫ ) .
Similarly, we can deﬁne the br 2 ( µ ) as the best response for the second player.
We will ignore the subscript in br 1 and br 2 when it is clear in the context.
While the best response may not be unique, the best response value is always unique.
For all h 2 [ H ] , s h 2 S , we deﬁne
[Equation]
⇤ ,
[Equation]
It is well known that Nash equilibrium (NE) strategy ⇡ ⇤ =
( µ ⇤ , ⌫ ⇤ ) , i.e., a strategy pair such that no player can beneﬁt from switching its own strategy, exists for zero-sum Markov games with a unique value function [Shapley, 1953].
In other words, µ ⇤ and ⌫ ⇤ are the best responses to each other.
We deﬁne V
⇤ h
=
V µ ⇤ , ⌫ ⇤ h for all h
[ H ] .
The following weak duality property holds for all strategy pairs ( µ, ⌫ ) in MG:
[Equation]
For a strategy pair ⇡ = ( µ, ⌫ ) , we can then deﬁne the corresponding duality gap as
[Equation]
The duality gap is always non-negative and the NE strategy has zero duality gap Gap( ⇡ ⇤ )
= 0 .
Duality gap measures how well a strategy pair approximates the NE.
We say a strategy pair ⇡ is an✏ -approximate NE if Gap( ⇡ )  ✏ .
2.2 Ofﬂine Two-Player Zero-Sum Game
In ofﬂine RL, we are given an ofﬂine dataset D = { ( s ⌧ h , a ⌧ h , b ⌧ h , r ⌧ h , s ⌧ h +1 ) } h 2 [ H ] ⌧ 2 [ n ] and we cannot doany further sampling [Kakade, 2003].
We assume that the dataset is sampled from some explorationpolicy ⇢ = ( ⇢ 1 , ⇢ 2 , · · · , ⇢ H ) , ⇢ h : S !
A ⇥ B , 8 h 2 [ H ] .
The target of ofﬂine MG is to ﬁnd an 138 approximate NE with a small duality gap by utilizing the given dataset D .
We use d ⇡ h ( s, a, b ) todenote the probability of s, a, b appears at timestep h in the trajectory generated by strategy ⇡ forall
h
[ H ] .
The dataset distribution d ⇢ h ( s, a, b ) is deﬁned similarly.
A state-action pair ( s, a, b ) attimestep h is covered by strategy ⇡
if and only if d ⇡ h ( s, a, b ) > 0 .
Strategy ⇡ is covered by datasetgenerated by exploration strategy ⇢ if and only if for all ( s, a, b ) covered by ⇡ , it is covered by ⇢ .
Inother words, we have
[Equation]
The sample complexity guarantee will depend on this ratio.
Dataset Coverage Assumptions.
Below we list three different dataset coverage assumptions for
Markov games.
Assumption 2.1.
(Single strategy concentration)
One NE strategy ( µ ⇤ , ⌫ ⇤ ) is covered by the dataset.
Assumption 2.2.
(Unilateral concentration)
For all strategies µ and ⌫ , ( µ, ⌫ ⇤ ) and ( µ ⇤ , ⌫ ) are covered by the dataset, where ( µ ⇤ , ⌫ ⇤ ) is one NE strategy.
Assumption 2.3.
(Uniform concentration)
For all h 2 [ H ] and ( s, a, b ) 2 S ⇥ A ⇥ B , ( s, a, b ) attimestep h is covered by the dataset.
Assumption 2.1 is the weakest assumption and is the most straightforward extension of the singlepolicy concentration in single-agent RL [Rashidinejad et al., 2021].
Assumption 2.3 generalizes theuniform policy concentration in single-agent RL [Yin et al., 2020].
Assumption 2.2 is sandwiched by Assumption 2.1 and Assumption 2.3 as Assumption 2.2 implies Assumption 2.1 and Assumption 2.3 implies Assumption 2.2.
In this work, we will show that Assumption 2.2 is the minimaldataset coverage assumption that allows NE learning and we provide sample complexity boundsthat depends on the density ratio (1).
Notations.
We use Var P ( s,a,b ) ( V ) to denote the variance of the random variable V ( s 0 ) where s
⇠
P ( ·| s, a, b ) and Var P ( V ) 2 R SAB to denote a vector whose ( s, a, b ) component is Var P ( s,a,b ) ( V ) .
We deﬁne a _ b := max { a, b } and a ^ b := min { a, b } .
In addition, if a is a vector and b is a scalar,the operation is taken on each element of a : [ a _ b ] i = a i _ b .
For two vector a 2 R n , b 2 R n , weuse ab 2 R n to denote the element-wise division: ⇥ ab ⇤ i = a i b i .
In addition, if a is scalar, we still useab 2 R n to denote the element-wise division: ⇥ ab ⇤ i =
ab i .
We use S, A, B to denote |S| , |A| , |B| .
Impossibility Results
In this section, we show that no assumption weaker than the unilateral concentration assumption(Assumption 2.2), which includes single strategy concentration (Assumption 2.1), allows learningthe NE strategy.
To begin with, we consider the deterministic unilateral concentration assumption.
Assumption 3.1.
(Deterministic unilateral concentration)
For all deterministic strategy µ and ⌫ ,( µ, ⌫ ⇤ ) and ( µ ⇤ , ⌫ ) are covered by the dataset, where ( µ ⇤ , ⌫ ⇤ ) is one NE strategy.
Immediately we can tell that Assumption 3.1 is satisﬁed under Assumption 2.2.
These two assumptions are equivalent, which is shown by Proposition 3.2, because any stochastic strategy can beviewed as a combination of several deterministic strategies.
Proposition 3.2.
If for all deterministic strategy µ and ⌫ , ( µ, ⌫ ⇤ ) and ( µ ⇤ , ⌫ ) are covered by thedataset, then we have for all (possibly stochastic) strategy µ 0 and ⌫ 0 , ( µ 0 , ⌫ ⇤ ) and ( µ ⇤ , ⌫ 0 ) arecovered by the dataset.
For the hardness examples, we consider bandit games, i.e., Markov games with horizon H = 1 .
Theresult can be generalized to arbitrary horizon by setting the reward to be 0 in horizons other thanh = 1 .
We consider a class of bandit games and datasets such that Assumption 3.1 is almost satisﬁedwhile no algorithm can identify the NE strategy for all bandit games and datasets in this class.
As
Assumption 2.2 and Assumption 3.1 are equivalent, no assumption weaker than Assumption 2.2allows NE strategy learning.
A direct corollary is that uniform concentration (Assumption 2.1) isnot sufﬁcient for NE learning.
Theorem 3.3.
Deﬁne a class X of bandit game M and exploration strategy ⇢ that consists of all
M and ⇢ pairs satisfying that there exists at most one deterministic strategy µ or one deterministic strategy ⌫ such that ( µ, ⌫ ⇤ ) or ( µ ⇤ , ⌫ ) is not covered and for all other deterministic strategies µ 0 , ⌫ 0 , the density ratio is bounded
[Equation]
for all h 2 [ H ] .
For any algorithm ALG , there exists ( M, ⇢ ) 2 X such that the output of thealgorithm ALG is at most a 0 .
25 -approximate NE strategy no matter how many data are collected.
Proof.
We consider bandit games with two actions for each player here.
The action set is A ={ a 1 , a 2 } for the ﬁrst (max) player and B = { b 1 , b 2 } for the second (min) player.
We construct the 188 following two bandit games with deterministic rewards.
[Equation]
Bandit Game
Then the (unique) NE of the ﬁrst bandit game is ( a 1 , b 1 ) and the (unique) NE of the secondbandit game is ( a 2 , b 2 ) .
Now we set the exploration strategy ⇢ to be uniform distribution on{ ( a 1 , b 1 ) , ( a 1 , b 2 ) , ( a 2 , b 2 ) } .
We can verify that both bandit games with exploration strategy ⇢ is r ( a 1 , b 1 )
= 0 .r
( a 1 , b 2 )
= 0 .r
( a 2 , b 1 ) =
r ( a 2 , b 2 ) = 0 .
in the class deﬁned in Theorem 3.3.
Note that the dataset contains data on ( a 1 , b 1 ) , ( a 1 , b 2 ) , ( a 2 , b 2 )and no data on ( a 2 , b 1 ) .
It is impossible for an algorithm to distinguish between these two banditgames as they are consistent on the given dataset and they all satisfy the dataset coverage assumption that only one action pair is not covered.
With some calculations, we can show that the output of
ALG is at most a 0.25-approximate NE for one of the instances, which proves the theorem.
Remark 3.4 .
We can easily extend this instance to arbitrary action space by setting ( a i , b j ) =
0for all i / 2 { 1 , 2 } , j 2 { 1 , 2 } , and ( a i , b j )
=
for all j / 2 { 1 , 2 } , i 2 { 1 , 2 } , and the exploration 199 strategy ⇢ to be the uniform distribution on ( a i , b j ) such that ( i, j ) 2 { ( i, j ) :
i 2 { 1 , 2 } or j 2{ 1 , 2 } , ( i, j ) 6 = (2 , 1) } .
Remark 3.5 .
It is straightforward to verify that the hard instance in Theorem 3.3 also holds for turnbased Markov games.
As a result, no assumption weaker than Assumption 2.2 is sufﬁcient for NElearning in turn-based Markov games.
Provably Efﬁcient Algorithm under Unilateral Concentration
In this section, we show that it is indeed possible to learn the NE with the unilateral concentrationassumption.
We propose a novel algorithm called Pessimistic Nash Value Iteration (PNVI), whichadapts the pessimism principle in single-agent RL to Markov games.
Our sample complexity resultdepends on the following quantity named unilateral concentrability :
Deﬁnition 4.1.
(Unilateral concentrability)
For Nash equilibrium ⇡ ⇤ , we deﬁne
[Equation]
By deﬁnition, C ⇤ is ﬁnite if Assumption 2.2 is satisﬁed.
For the rest of the paper, ⇡ ⇤ denotes the
Nash equilibrium that achieves the minimum here.
Note that C ⇤ is not provided to the algorithm.
4.1 Hoeffding-type Algorithm with Data Splitting
To illustrate our main algorithm design ideas, we ﬁrst propose an algorithm with Hoeffding-typebonus and random data splitting.
Given a dataset D =  ( s kh , a kh , b kh , r kh , s kh +1 ) n,H k,h =1 , we denoten h ( s, a, b ) =
P nk =1
( s kh , a kh , b kh ) =
( s, a, b )  to be the number of times that ( s, a, b ) is visited attimestep h .
We set the empirical reward and the empirical transition kernel asb r h ( s, a, b ) =
r h ( s, a, b ) , b
P h ( s 0 | s, a, b ) =
P nk =1
( s kh , a kh , b kh , s kh +1 )
= ( s, a, b, s 0 )
P nk =1
( s k h , a k h , b k h ) =
( s, a, b )  , (2) if n h ( s, a, b )  1 , and b
[Equation]
b
[Equation]
, b ) = 1 /S otherwise.
In addition, we use n h
R SAB to denote a vector such that [ n h ] s,a,b = n h ( s, a, b ) .
Now we explain Algorithm 1 in detail.
First, we split the dataset D into H small datasets {D h } Hh =1 219 with the same size.
Then we use
D h to estimate the reward and the transition matrix at timestep 220 h .
The data splitting scheme is to remove the dependence between each timestep.
Then the valuefunction is estimated via a value-iteration-type algorithm.
At each timestep, we maintain both optimistic and pessimistic estimates by adding/minusing a Hoeffding-type bonus.
We use the following
Hoeffding-type bonus:
[Equation]
where ◆ = log( HSAB/  ) .
Then we compute the pessimistic estimate Q and Q :
[Equation]
Algorithm
Input: Ofﬂine dataset D =  ( s kh , a kh , b kh , r kh , s kh +1 ) n,H k,h =1 .
Failure Probability  .
Initialization: Set V H +1 ( · ) =
V H +1 ( · ) = 0 .
Randomly split the dataset D into {D h } Hh =1 with |D h | = n/H .
Set b r h , b
P h , b h and b h as (2) and (3) using the dataset D h for all h 2 [ H ] .
for time h = H, H  1 , . .
. , 1 do
Set Q h ( · , · , · ) and Q h ( · , · , · ) as (4).
Compute the NE of Q h ( · , · , · ) as ( µ h ( · ) , ⌫ h ( · )) .
Compute V h ( · ) =
E a ⇠ µ h ,b ⇠ ⌫ h Q h ( · , a, b ) .
Compute the NE of Q h ( · , · , · ) as ( µ h ( · ) , ⌫ h ( · )) .
Compute V h ( · ) =
E a ⇠ µ h ,b ⇠ ⌫ h Q h ( · , a, b ) .
end for
Output µ = ( µ 1 , µ 2 , · · · , µ H ) , ⌫ = ( ⌫ 1 , ⌫ 2 , · · · , ⌫ H ) , { V h } Hh =1 , { V h } Hh =1 .
Pessimistic estimate Q h is for the max player, which mimics the pessimism in single-agent RL.
Q husing a positive bonus is for the min player, which is also a kind of pessimism as the min player’starget is to minimize the reward.
We compute the NE strategy of the matrix game Q ( s, · , · ) and
Q ( s, · , · ) respectively and use the NE value to be the state value V ( s ) and V ( s ) .
Note that we onlysolve a zero-sum matrix game, which is computationally efﬁcient [Chen and Deng, 2006].
Remark 4.2 .
If we compute an ✏ NE /H -approximate NE of the matrix game Q ( s, · , · ) and Q ( s, · , · )at each timestep, then the performance gap will only be enlarged by e
O ( ✏ NE ) .
Theorem 4.3.
Suppose Assumption 2.2 holds.
For any 0 <  < 1 and strategy µ, ⌫ , with probability 1   , the pessimistic values V h
and V h of Algorithm 1 satisfy
[Equation]
for all h 2 [ H ] , where s h is sampled from the trajectory following the strategy in the expectation.
Proof Sketch.
For simplicity, we only show the guarantee for the strategy µ of the max player.
First, we show that under good concentration event, the pessimistic value V h is always smaller than the best response value of µ , i.e.
[Equation]
Second, we show that the performance gap of µ is bounded by the expected sum of bonus under thestrategy µ ⇤ , ⌫ , i.e.
[Equation]
Finally, we deﬁne a concatenated strategy ⌫ 0 :
= ( ⌫ 1 , · · · , ⌫ h  1 , ⌫ h , · · · , ⌫ H ) and then we have
[Equation]
As Assumption 2.2 suggests that ( µ ⇤ , ⌫ 0 ) is well covered by the exploration strategy ⇢ , the expectedsum of bonus can be bounded.
See Appendix B for details.
Theorem 4.3 provides polynomial bounds on the error of the value estimates in Algorithm 1.
Itcan directly imply the following performance gap bound.
In addition, it provides guarantees for thereference function that will be utilized in the next section.
Corollary 4.4.
Suppose Assumption 2.2 holds.
For any 0 <  < 1 , with probability 1   , the outputpolicy ⇡ = ( µ, ⌫ ) of Algorithm 1 satisﬁes Gap( ⇡ ) 
[Equation]
Theorem 4.4 shows that the output strategy of Algorithm 1 is an
[Equation]
-approximate
NE.
The parameter C ⇤ measures how the exploration strategy ⇢ covers the unilateral strategies( µ ⇤ , ⌫ ) and ( µ, ⌫ ⇤ ) for all µ and ⌫ .
4.2 Bernstein-type Algorithm with Reference Advantage Function Decomposition
In this section, we will derive an improved performance gap bound e
O ⇣p
C ⇤ SABH 3 /n ⌘ .
Theextra H 2 is shaved by using Bernstein-type bonus and reference advantage decomposition techniquemotivated from Xie et al.
However, we want to emphasize that zero-sum Markov games aresubstantially different from MDP and require novel adaptation, which we will describe later.
Due to the space constraint, we put Algorithm 2 in Appendix A. Algorithm 2 is different from
Algorithm 1 in two aspects.
First, we use the reference advantage decomposition to remove an Hfactor.
The dataset is split into three subset with equal size D ref , D 0 , D 1 , and D 1 is further split 253 into H subset with equal size {D h, 1 } Hh =1 .
We run algorithm 1 on dataset D ref and we can obtainpessimistic value estimate V ref and V ref with guarantees by Theorem 4.3.
Then we use dataset D 0 toestimate P h V ref h +1 and dataset D h, 1 to estimate P h
( V h +1  V ref h +1 ) .
Second, we use a Bernstein-typebonus to remove another H factor.
Our updating formulas of Q h and Q h are
[Equation]
[Equation] [Equation]
where we truncate by the reference function to ensure monotonic update so that Q h and Q h aremore accurate pessimistic/optimistic estimate compared with the reference function Q h and Q h .
The bonus functions are deﬁned as
[Equation]
s
Var b {+P h, 0 ( V ref h +1 ) ◆ n h, 0 _+ H ◆ n h, 0 _ 
 A , b h, 0 = c+}
B@ vuut
Var b
P h, 0 ( V ref h +1 ) ◆ n h, 0 _+ H ◆ n h, {+0 _ 
 CA , (7) b h,+} 1 = c@ s Var b
P h, 1 ( V h +1  V ref h +1 ) ◆ n h, 1 _+
{+H ◆ n h, 1 _ 
+}
A , b h, 1 = c
B@ vuut
Var b
P h, 1 ( V h +1  V ref h +1 ) ◆ n h, 1 _+
{+H ◆ n h, 1 _ 
+}
CA , (8) where c is some universal constant and Var b
P h, 0 ( V ) , Var b
P h, 1 ( V ) , n h, 0 , n h, 1 are all SAB -dimensionvectors and the operations are element-wise.
Theorem 4.5.
Suppose Assumption 2.2 holds.
For any 0 <  < 1 and n  e O ( p C ⇤ SABH 4 ) ,with probability 1   , the output policy ⇡ = ( µ, ⌫ ) of Algorithm 2 satisﬁes Gap( ⇡ ) e
O ⇣p
C ⇤ SABH 3 /n ⌘ .
Remark 4.6 .
n  e
O ( p C ⇤ SABH 4 ) serves as the burn-in cost, which is standard in the literature.
See a more detailed discussion in Li et al.
Proof of Sketch.
For simplicity we only show the guarantee for the strategy µ of the max player.
First we show that under good concentration event, the pessimistic value V h is always sandwiched by the reference value V ref h and the best response value of µ , i.e.,
[Equation]
Second, we show that the performance gap of µ is bounded by the expected sum of bonus under thestrategy µ ⇤ , ⌫ , i.e.,
[Equation]
Then we bound the ﬁrst term by
[Equation]
⇤ , ⌫
[Equation]
where q
[Equation]
s 1 )  V 1 ( s 1 ) is the square root of the term we want to bound.
The second term canbe bounded similarly.
Finally solving the self-bounding inequality for V µ ⇤ , ⌫
[Equation]
s 1 )  V 1 ( s 1 ) and wehave
[Equation]
We utilizes Theorem 4.3 to provide guarantee for the error of the reference function and V ref h ( s ) 
V h ( s )  V µ, ⇤ h ( s ) to bound the variance of the estimation error.
See Appendix C for details.
As MDP are degenerated Markov games with one player having a ﬁxed action, Markov games inheritthe lower bounds of MDP.
Comparing with the lower bound e
[Equation]
[Xie et al., 2021],our bound is already tight in C ⇤ , S , H .
The extra AB factor is from the Cauchy-Schwarz inequalityand the fact that the NE of zero-sum Markov games can be a mixed strategy while deterministicoptimal policy always exists for MDP.
It is unknown whether the AB factor is removable and weleave it to future work.
4.3 Minimax Optimal Sample Complexity Bounds
In this section, we show that Algorithm 2 directly adapts to two popular settings, i.e. Assumption2.3 (uniform concentration assumption) and turn-based Markov games.
In addition, minimax samplecomplexity can be achieved under both settings.
The proof is deferred to Appendix D.
Theorem 4.7.
Set d m = min { d ⇢ h
( s, a, b ) :
h 2 [ H ] , ( s, a, b ) 2 S ⇥ A ⇥ B} .
Suppose Assumption2.3 holds.
For any 0 <  < 1 and n  e O ⇣p H 4 /d m ⌘ , with probability 1   , the output policy⇡ = ( µ, ⌫ ) of Algorithm 2 satisﬁes Gap( ⇡ )  e
O ⇣p H 3 / ( nd m ) ⌘ .
This bound has no explicit dependence on AB because the Cauchy-Schwarz inequality can beapplied on d µ ⇤ , ⌫ h instead of q d µ ⇤ , ⌫ h
(See the proof of Theorem D.1).
As the lower bounde ⌦ ⇣p
H 3 / ( nd m ) ⌘ for MDP
[Yin and Wang, 2021] is the lower bound for Markov games, Algorithm 2 achieves minimax sample complexity under assumption 2.3.
Theorem 4.8.
Suppose Assumption 2.2 holds for a turn-based Markov games.
For any 0 <  < 1and n  e O ( p C ⇤ SH 4 ) , with probability 1   , the output policy ⇡ = ( µ, ⌫ ) of Algorithm 2 satisﬁes
Gap( ⇡ )  e
O ⇣p
C ⇤ SH 3 /n ⌘ .
As the lower bound is
[Equation]
[Xie et al., 2021], Algorithm 2 can achieve the minimaxsample complexity for turn-based Markov games under assumption 2.2.
The difference is due toturn-based Markov games always have pure NE strategies (See the proof of Theorem D.6).
Conclusion
In this work, we study the minimal dataset coverage assumption for NE learning in two-player zerosum Markov games.
We show that single strategy concentration is not enough for NE learning.
Instead, we ﬁnd a minimal coverage assumption for NE learning and design an algorithm withsample complexity tight in C ⇤ , S , H under such assumption based on novel techniques.
In addition,the algorithm can achieve minimax sample complexity in certain settings.
We believe this work canshed new light on ofﬂine MARL.
Here we list several open problems for future work.
One direction is to ﬁnd the minimax samplecomplexity of ofﬂine Markov games under the unilateral concentration.
Importantly, it is unclearwhether AB factor can be reduced [Bai et al., 2020].
Another direction is to design efﬁcient algorithms for ofﬂine MARL with a large number of agents without sample complexity scales exponentially with the number of agents.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]