Minimax-Optimal Multi-Agent RL in Zero-Sum Markov Games With a Generative Model
Abstract
This paper is concerned with two-player zero-sum Markov games — arguably the most basic setting in multi-agent reinforcement learning — with the goal of learning a Nash equilibrium (NE) sample-optimally.
All prior results suffer from at least one of the two obstacles: the curse of horizon and that of multiple agents, regardless of the sampling protocol in use.
We take a ﬁrst step towards settling this problem, assuming access to the most ﬂexible sample mechanism: the generative model.
Focusing on non-stationary ﬁnite-horizon Markov games, we develop a learning algorithm and an adaptive sampling scheme that leverage the optimism principle in adversarial learning (particularly the Follow-the-Regularized-Leader (FTRL) method), with a delicate design of bonus terms that ensure certain decomposability under the FTRL dynamics.
Our algorithm learns an ε -approximate Markov NE policy with
[Equation]
samples, where S is the number of states, H is the horizon, and A (resp.
B ) isthe number of actions for the max-player (resp.
min-player).
This is nearly unimprovable in a minimax sense.
Along the way, we derive a reﬁned regret boundfor FTRL that makes explicit the role of variance-type quantities.
Introduction
The thriving ﬁeld of multi-agent reinforcement learning (MARL) studies how a group of interactingagents makes decisions autonomously in a shared dynamic environment [65].
The recent developments in game playing [57, 8], self-driving vehicles [50] and multi-robot control [40] are primeexamples of MARL in action.
In practice, there is no shortage of situations where the agents involved have conﬂict of interest and they have to act competitively in order to promote their ownbeneﬁts (possibly at the expense of one another).
Scenarios of this kind are frequently modeled via
Markov games (MGs) [51, 37], a framework that has been a fruitful playground to formalize andstimulate the studies of competitive MARL.
In view of the irreconcilable competition between individual players, solutions of competitive
MARL normally take the form of certain equilibrium states, which are perhaps best epitomizedby the concept of Nash equilibrium (NE).
In a Nash equilibrium, no gain can be realized througha unilateral change, and hence no player has incentives to deviate from her current strategy/ policy(assuming no coordination between players).
A myriad of research has been conducted surrounding NEs, which spans various aspects like existence, learnability, computational hardness, and algorithmdesign, among others [51, 17, 10, 45, 44, 19, 37, 24, 42, 29, 6].
Sample efﬁciency in zero-sum Markov games.
One critical challenge impacting MARL appli-
cations is data efﬁciency.
The players involved often have minimal knowledge about how the envi- ronment responds to their actions, and have to learn the preferable actions by probing the unknown environment.
For MARL to expand into applications with enormous dimensionality and long plan- ning horizon, the learning algorithms must manage to make efﬁcient use of the collected data.
Nevertheless, how to learn NEs with optimal sample complexity remains by and large unsettled,even when it comes to the most basic setting — two-player zero-sum Markov games.
In whatfollows, let us review two representative algorithms developed on this front under two drasticallydifferent sampling protocols, and discuss the shortfalls of these cutting-edge results.
To facilitateprecise comparisons, our discussion concentrates
on zero-sum Markov games involving two players where S is the number of states, H indicates the horizon or effective horizon, and A and B denote
respectively the number of actions for the max-player and the min-player.
• Model-based methods under either a generative model or online exploration.
Assuming access to a generative model (so that one can sample arbitrary state-action combinations), Zhang et al.
investigated a natural model-based algorithm, which performs planning (e
g., value iteration) onan empirical MG derived from samples produced by the generative model.
Focusing on stationary discounted inﬁnite-horizon MGs, their algorithm ﬁnds an ε -approximate NE with no more than
[Equation]
In parallel
, Liu et al
studied non-stationary ﬁnite-horizon MGs with online exploration
andobtained similar sample complexity bounds,
i
e.,
[Equation]
for learning ε -NEs.
While these bounds achieve minimax-optimal dependency on the horizon H ,a major drawback emerges as the curse of multiple agents; namely, both results scale with the
number AB of joint actions , a quantity that blows up exponentially with the number of players. • V-learning for online exploration settings.
Focusing on online exploration settings, Bai et al. [4], Jin et al.
proposed an algorithm called V-learning that leverages the advances in online adversarial learning (e
g., adversarial bandits) to circumvent the curse of multiple agents.
This algorithm provably yields an ε -approximate NE in non-stationary ﬁnite-horizon MGs using
[Equation]
which effectively brings down the sample size scaling (2) from AB (i
e., the number of jointactions) to A + B (i
e., the sum of individual actions).
Note, however, that this theory appearssub-optimal in terms of the horizon dependency, as it is a factor of H 2 above the minimax limit.
The above results represent the most advanced sample complexity theory thus far for learning NEs intwo-player zero-sum MGs.
In summary, these state-of-the-art results — irrespective of the samplingmechanism in use — fall short of overcoming at least one of the two major hurdles: (i) the curse ofmultiple agents , and (ii) the curse of horizon .
A natural question to pose is:
Question: can we learn Nash equilibria in a two-player zero-sum Markov gamein a sample-optimal and computation-efﬁcient fashion?
To settle this question favorably, both of the above hurdles need to be crossed simultaneously.
Main contributions.
Recognizing that the above question remains open regardless of the samplingscheme in use, this paper takes a ﬁrst step towards solving it assuming access to the most ﬂexiblesampler: the generative model (or simulator).
In sharp contrast to the single-agent case whereuniform sampling of all state-action pairs sufﬁces [2, 34], the multi-agent scenario requires one totake samples intelligently and adaptively, a crucial step to avoid inefﬁcient use of data (otherwise onecannot hope to break the curse of multiple agents).
With the aim of computing an ε -approximate NE in a non-stationary ﬁnite-horizon two-player zero-sum MG, we come up with a learning algorithm(accompanied by an adaptive sampling strategy) that accomplishes this goal with no more than
[Equation]
calls to the generative model.
Encouragingly, this sample complexity bound matches the minimaxlower limit (up to a logarithmic factor).
Our sample complexity theory is valid for the full ε -range(i.e. , any ε ∈ (0 , H ] ); this unveils that no burn-in cost is needed for our algorithm to achieve sampleoptimality, which lends itself well for sample-hungry applications.
Our algorithm is inspired by two key algorithmic ideas in RL/bandit literature: (1) optimism in theface of uncertainty (by leveraging upper conﬁdence bounds in value estimation), and (2) adversarial learning (particularly the Follow-the-Regularized-Leader (FTRL) algorithm).
In particular, thedesign of data-driven Bernstein-style bonus terms is substantially more challenging than the singleagent case; this requires ﬁne-grained adaptation in response to the policy changes of one another,and needs to ensure decomposability of bonus over steps in a way that is compatible with the FTRLdynamics.
Additionally, the policy returned by our algorithm is Markovian (depending only onthe current state and step number), and the algorithm can be carried out in a decentralized mannerwithout the need of directly observing the opponent’s actions. Along the way, we develop a reﬁnedregret bound for FTRL tailored to weighted linear optimization; our regret bound makes explicit therole of variance-style quantities, which is crucial in optimizing the horizon dependency in Markovgames; see the supplementary material for this reﬁned FTRL regret bound. Other related works.
Recent years have seen an explosion of interest in learning equilibriumsolutions in MARL. While the Nash equilibrium is arguably the most compelling solution concept in Markov games, the ﬁnite-sample/ﬁnite-time studies of NEs largely concentrate on two-player zerosum MGs (e.g., [3, 14, 39, 61, 56, 15, 67, 25]), mainly because computing NEs becomes, for the mostpart, computationally infeasible (i.e., PPAD-complete) when going beyond two-player zero-sum
MGs [17, 18].
Roughly speaking, previous NE-ﬁnding algorithms for two-player zero-sum Markovgames can be categorized into model-based algorithms [44, 64, 38], value-based algorithms [3, 4,63, 46, 27, 13], and policy-based algorithms [9, 19, 62, 66, 12]. In particular, Bai et al.
[4], Jin et al.[27] developed the ﬁrst algorithms to beat the curse of multiple agents in two-player zero-sum MGs,while Jin et al. [27], Daskalakis et al.
[20], Mao and Ba¸sar [39], Song et al. [55] further demonstratedhow to accomplish the same goal when learning other computationally tractable solution concepts(e.g. , coarse correlated equilibria) in general-sum multi-player Markov games.
Additionally, while there are multiple sampling mechanisms (e.g., online exploratory sampling, ofﬂine data) that bear practical relevance, the generative model (or simulator) serves as an idealisticsampling protocol that has received much recent attention in reinforcement learning theory, covering the design of various model-based, model-free and policy-based algorithms [31, 60, 1, 2, 34, 28,52, 58, 59, 35, 30, 53, 43, 32, 11, 22, 62, 21]. When it comes to multi-agent RL, sample-efﬁcientsolutions with a generative model have been proposed in the recent works [54, 16, 64].
Notation.
Before proceeding, let us gather several notation that will be used throughout.
For anypositive integer n , we set [ n ] := { 1 , · · · , n } .
We shall abuse notation and let 1 and 0 denote theall-one vector and all-zero vector, respectively.
For a sequence { α k } k ≥ 1 ⊆ (0 , 1] , we deﬁneα ki : = (cid:40) α i (cid:81) kj = i + [Equation]
[Equation]
− α
[Equation]
) , if
[Equation] α k , if i = [Equation]
(5) for any 1 ≤ i ≤ k .
Next, consider any two vectors a = [ a i ] 1 ≤ i ≤ n and b =
[ b i ] 1 ≤ i ≤ n .
We usea ≤ b (resp.
a ≥ b ) to indicate a i ≥ b i (resp.
a i ≤ b i ) for all i ; we allow scalar functions totake vector-valued arguments in order to denote entrywise operations (e.g., a 2 =
[ a 2 i ] 1 ≤ i ≤ n anda 4 =
[ a 4 i ] 1 ≤ i ≤ n ); and we denote by a ◦ b =
[ a i b i ] 1 ≤ i ≤ n the Hadamard product.
For a ﬁnite set A = { 1 , · · · , A } , we let ∆( A ) = { x ∈ R
A | (cid:80) i x i = 1; x ≥ 0 } be the probability simplex over A .
Background and models
In this section, we introduce the basics for two-player zero-sum Markov games and the solutionconcept of Nash equilibrum.
Two-player Markov games.
A non-stationary ﬁnite-horizon two-player Markov game, denotedby MG = (cid:8) S , A , B , H, P, r (cid:9) , involves two players competing against each other — with the ﬁrstone called a “max-player” and the second one called a “min-player” — and consists of several keyelements to be formalized below.
To begin with, we denote by S
= { 1 , · · · , S } a shared statespace comprising S different states, and let A = { 1 , · · · , A } (resp.
B = { 1 , · · · , B } ) representthe action space of the max-player (resp.
min-player) containing A (resp.
B ) different actions.
Thehorizon length of this ﬁnite-horizon Markov game is denoted by H .
The probability transitionkernel of MG is denoted by P
= { P h } 1 ≤ h ≤ H with P h ∈ R SAB × S ; namely, for any ( s, a, b, h, s (cid:48) ) ∈
S ×A×B× [ H ] ×S , we let P h ( s (cid:48) | s, a, b ) indicate the probability of MG transitioning from state sto state s (cid:48) at step h when the max-player takes action a and min-player takes action b .
The functionr = { r h } 1 ≤ h ≤ H with r h ∈ R SAB represents the reward function; namely, for any ( s, a, b, h ) ∈
S ×
A × B × [ H ] , r h ( s, a, b ) stands for the immediate reward gained in state s at step h if themax-player (resp.
min-player) executes action a (resp.
b ).
Given that our focal point is the family ofzero-sum Markov games, it sufﬁces to introduce a single reward function r .
We shall also assumenormalized rewards throughout this paper in the sense that r h ( s, a, b ) ∈ [0 , 1] for any ( s, a, b, h ) .
Value functions under product policies.
Let µ = { µ h } 1 ≤ h ≤ H (resp.
ν
= { ν h } 1 ≤ h ≤ H ) denotethe policy of the max-player (resp.
min-player), where µ h ( · | s ) ∈ ∆( A ) and ν h ( · | s ) ∈ ∆( B )for any s ∈ S .
More speciﬁcally, µ h ( a | s ) indicates the probability of the max-player selectingaction a in state s at step h ; and ν h ( b | s ) is deﬁned analogously.
Consider a Markovian trajectory{ ( s t , a t , b t , r t ) } 1 ≤ t ≤ H , where s t is the state at time t , a t (resp.
b t ) is the action of the max-player(resp.
min-player) at time t , and r t is the immediate reward observed at time t .
For any givenpolicy µ (resp.
ν ) for the max-player (resp.
min-player) and any step h , we deﬁne the value function
V µ,νh : S → R under the product policy µ × ν as follows:
[Equation]
where the expectation is over the Markovian trajectory { ( s t , a t , b t , r t ) } with the max-player and themin-player executing policies µ and ν , respectively, in an independent fashion; that is, conditionalon s t , we draw a t ∼ µ t ( · | s t ) and b t ∼ ν t ( · | s t ) independently , and then s t +1 ∼ P t ( · | s t , a t , b t ) .
In addition, conditional on the min-player executing policy ν , the optimal value function V (cid:63),ν ={ V (cid:63),νh } 1 ≤ h ≤ H (with V (cid:63),νh : S → R ) of the max-player is deﬁned as
[Equation]
similarly, when the max-player chooses to execute policy µ , the optimal value function V µ,(cid:63) =
{ V µ,(cid:63)h } 1 ≤ h ≤ H (with V µ,(cid:63)h : S → R ) of the min-player is deﬁned as
[Equation]
Furthermore, if we freeze the policy of the min-player to ν , then the Bellman optimality conditionfor the max-player can be expressed as [7]
[Equation]
Solution concept: Nash equilibrium.
In the zero-sum two-player Markov game, the max-playerwishes to maximize the value function, while the min-player aims to minimize the value function instead.
Due to the competing objectives, ﬁnding some sort of equilibrium — particularly the Nash equilibrium [41] — becomes a central topic in Markov games.
More precisely, a policy pair( µ (cid:63) , ν (cid:63) ) ∈
∆( A ) SH × ∆( B ) SH is said to be a (mixed-strategy)
Nash equilibrium of MG if theresulting product policy µ (cid:63) × ν (cid:63) obeys [Equation] and
[Equation]
for all s
[Equation]
S
[Equation]
In other words, conditional on the opponent’s current policy and the assumption that two players actindependently , no player can harvest any gain by unilaterally deviating from its current policy.
In practice, it might be challenging to compute an “exact” Nash equilibrium, and instead one wouldseek to ﬁnd approximate solutions.
Towards this end, let us ﬁrst deﬁne the sub-optimality gap of apolicy pair ( µ, ν ) as follows (measured in an (cid:96) ∞ -based manner)
[Equation]
where
[Equation]
With this measure in place, a policy pair ( µ, ν ) is said to be an ε -approximate Nash-equilibrium —or more concisely, ε -Nash — if the resultant sub-optimality gap of the product policy µ × ν obeys NE - gap ( µ, ν ) ≤ ε .
Generative model / simulator.
In reality, we often do not have access to perfect descriptions (e.g.,accurate knowledge of the transition kernel P ) of the Markov game under consideration; instead, onehas to learn the true model on the basis of a set of data samples.
When it comes to the data generatingmechanism, this paper assumes access to a generative model (also called a simulator)
[31, 30]: ineach call to the generative model, the learner can choose an arbitrary triple ( a, b, h ) ∈
A × B × [ H ]and obtain an independent sample generated based on the true transition kernel:
[Equation]
In words, a generative model facilitates query of arbitrary state-action-step combinations, whichhelps alleviate the sampling constraints arising in online episodic settings.
The goal of the currentpaper is to compute an ε -approximate Nash equilibrium of MG with as few samples as possible,i.e., using a minimal number of calls to the generative model.
Algorithm and main theory
In this section, we put forward an algorithm that seeks to ﬁnd an ε -Nash policy pair with the assistance of a generative model, and demonstrate its sample optimality for the full ε -range.
3.1 Algorithm
We now describe the proposed algorithm, which is inspired by the optimism principle and the FTRLalgorithm for adversarial learning.
Following the dynamic programming approach [7], our algorithmemploys backward recursion from step h
= H back to h = 1 ; in fact, we shall ﬁnish estimating thevalues and policies for step h before moving backward to step h − 1 .
For each h , the max-player(resp.
min-player) calls the generative model for K rounds, with each round drawing SA (resp.
SB )independent samples; as a result, the total sample size is given by KS ( A + B ) H .
In what follows,let us ﬁrst introduce some convenient notation that facilitates our presentation of the algorithm.
Notation.
Consider any step h ∈
[ H ] , and any data collection round k ∈
[ K ] .
The algorithmmaintains the following iterates, whose notation is gathered here.
• V h ∈ R S (resp.
V h ∈ R S ) represents the ﬁnal estimate of the value function at step h by the
max-player (resp.
min-player)
; in particular, we set V H
+1 =
V H +
= 0 .
• Q kh ∈ R SA (resp.
Q kh ∈ R SB ) represents the Q-function estimate of the max-player (resp. minplayer) at step h after the k -th round of data collection. • q kh ∈ R SA (resp.
q kh ∈ R SB ) represents a certain “one-step-look-ahead” Q-function estimate ofthe max-player (resp.
min-player) at step h using samples collected in the k -th round.
• r kh ∈ R SA (resp.
r kh ∈ R SB ) denotes the sample reward vector for step h received by the maxplayer (resp. min-player) in the k -th round.• P kh ∈ R SA × S (resp. P kh ∈ R SB × S ) denotes the empirical probability transition matrix for step hconstructed using the samples collected by the max-player (resp. min-player) in the k -th round.• β kh,Q ∈ R SA (resp.
β kh,Q ∈ R SB ) denotes the bonus (resp. penalty) vector for step h chosen bythe max-player (resp. min-player) in the k -th round for Q-estimation.
• β h,V ∈ R S (resp.
β h,V ∈ R S ) denotes the bonus (resp.
penalty) vector chosen by the max-player(resp.
min-player) for ﬁnal value estimation.
In addition, for any policies µ ∈ ∆( A ) and ν ∈ ∆( B ) , any vector f ∈ R A , and any function g ( · , · )with domain A × B , we adopt the convenient notation
[Equation]
Main steps of the proposed algorithm.
As mentioned above, our algorithm collects multiplerounds of independent samples for each h .
Let us describe below the proposed procedure for themax-player in the k -th round for step h ; the procedure for the min-player proceeds analogously.1.
Sampling and model estimation.
For each ( s, a ) ∈ S × A , draw an independent sample as follows
[Equation]
and receive the reward r k,h,s,a = r h (cid:0) s, a, b k,h,s,a (cid:1) .
These samples are then employed to constructthe sample reward vector r kh ∈ R SA and empirical transition kernel P kh ∈ R SA × S such thatr k h ( s, a ) =
r k,h,s,a and
[Equation]
0 , else (12b) for all ( s, a, s (cid:48) )
∈ S ×
A × S .
Note that each player only needs to build (12b), which does notrequire direct observation of the opponent’s actions.2.
Q-function estimation.
Following the dynamic programming approach, we ﬁrst compute the “onestep-look-ahead” Q-function estimate as follows
[Equation]
We then adopt the update rule of synchronous Q-learning with certain bonus terms (via upperconﬁdence bounds) incorporated:
[Equation]
14) where 0 < α k < 1 is the learning rate, and β kh,Q is some bonus vector to be speciﬁed shortly.
Applying (14) recursively and using the quantities deﬁned in (5), we arrive at
[Equation]
Policy updates.
Once the Q-estimates are updated, we adopt the exponential weights strategy toupdate the policy iterate of the max-player as follows
[Equation]
where η k ( s ) > 0 denotes another learning rate associated with policy updates.
In fact, this subroutine implements the Follow-the-Regularized-Leader update rule [48] [Equation]
[Equation]
with the regularizer F ( · ) chosen to be the negative entropy function F ( π )
= (cid:80) a π ( a ) log (cid:0) π ( a ) (cid:1) .
After carrying out K rounds of the above procedure, our ﬁnal policy estimate (cid:98) µ ∈ ∆( A ) SH and thevalue estimate V h : S → R for step h are taken to be
[Equation]
with { α Kk } deﬁned in (5), where β h,V ( s ) ≥ 0 is another bonus term to be speciﬁed momentarily.
Inparticular, the ﬁnal policy (18a) is a mixture of policy iterates and is Markovian in nature (dependingonly on the current state s and step h ).
The whole procedure is summarized in Algorithm 1.
Choices of learning rates.
Thus far, we have not yet speciﬁed the learning rates, which we describe now.
The learning rates associated with Q-updates are set to be rescaled linear, namely,
[Equation]
for some constant c α ≥ 24 .
In addition, the learning rates associated with policy updates are chosenin a data-driven manner:
[Equation]
where the notation Var π is introduced in (11a).
As will be made clear in the analysis, this choice iscarefully tuned based on our reﬁned understanding of FTRL as well as the other learning rates (19).
Choices of bonus terms.
It remains to specify the bonus terms, which is selected based on fairlyintricate analysis and constitutes a key component of our algorithm design.
Speciﬁcally, we take
[Equation]
for any ( s, a, h, k ) ∈ S ×
A × [ H ] × [ K ] , where c b > 0 is some sufﬁciently large constant.
Asin previous works, the bonus term, which is chosen carefully in a data-driven fashion, needs todominate the uncertainty incurred during the estimation process.
There are some important pointsthat we would like to highlight (which will be made clear in the analysis).• A critical design principle is to ensure that the bonus term is, in some sense, decomposable oversteps, which is crucial in sharpening the horizon dependency.•
The standard choice of the bonus terms is based on Bernstein-type concentration bounds, whichtypically entails some standard deviation term.
The standard deviation term, however, mightbe harder to decompose compared to the variance term.
To circumvent this difﬁculty by replacing standard deviation with variance, our design takes advantage of the elementary bound [Equation] + c , which remains tight if c is taken to be on the typical size of (cid:112) Var ( X ) . 3.2 Main results
As it turns out, the proposed algorithm is provably sample-efﬁcient,
whose sample complexity ischaracterized by the following theorem.
Theorem 1.
Consider any ε ∈ (0 , H ] and any 0 < δ < 1 .
Suppose that
[Equation]
(24) for some large enough universal constant c k > 0 .
With probability at least 1 − δ , the sub-optimalitygap (cf.
(10) ) of the policies ( (cid:98) µ, (cid:98) ν ) returned by Algorithm 1 obeys
[Equation]
Theorem 1 establishes a sample complexity upper bound for the proposed algorithm, which we takea moment to interpret as follows.
[-V H +
-]
[-V H +
-]
do do // empirical
[Equation]
/* see Algorithm 2.
*/ // update Q-estimates with upper/lower confidence bounds.
Compute q kh =
r kh + P kh V
h
+1 , q kh = r
[Equation] P
[Equation]
h
[Equation]
and update
[Equation]
where β kh,Q is given in (21a) , and β kh,Q is obtained by replacing ( µ kh , q kh ) in (21a) with ( ν kh , q kh ).
// update policy estimates using FTRL.
for ( s, a, b ) ∈ S ×
A × B do
[Equation]
where η k +1 is defined in (20) , and η k +1 can be set by replacing ( µ ih , q ih ) in (20) with ( ν ih , q ih ) .
// output the final policy estimate and value estimate for step h .
[Equation]
µ
[Equation]
Sample complexity.
When a generative model is available, Theorem 1 asserts that the total number of samples KS ( A + B ) H needed for Algorithm 1 to yield ε -Nash policies is at most(sample complexity
[Equation]
This delivers the ﬁrst result — for any sampling protocol — that simultaneously overcomes the curseof horizon and that of multiple agents.
In comparison to Zhang et al. [64] (cf.
(1)), this result revealsthat what ultimately matters is the total number of individual actions (i.e., A + B ) as opposed to thetotal number AB of joint actions.
Additionally, our result exhibits improved horizon dependency(by a factor of H 2 ) compared to Bai et al.
[4], Jin et al.
[27] (see (3)), although we should emphasizethat the online sampling protocol therein is more restrictive than a generative model.
Minimax optimality.
To assess the tightness of our sample complexity bound (26), it is helpfulto look at the minimax lower limit.
Suppose without loss of generality that A ≥ B .
Considerthe extreme scenario where B = 1 , in which case the Nash-equilibrium reduces to the optimal × S . [Equation] set
[Equation] [Equation] policy of a single-agent MDP with S states and A actions. It is well-known that for any accuracylevel ε ∈ (0 , H ] , one can construct a non-stationary MDP with S states and A actions such that noalgorithm can learn an ε -optimal policy with fewer than (cid:101) O (cid:0) H 4 SAε 2 (cid:1) samples [2, 36].
This means thatin general, the minimax lower bound (w.r.t.
ﬁnding an ε -Nash policy pair) scales as(minimax lower bound) H
[Equation]
S [Equation] (27) modulo some logarithmic factor.
Taking this together with (26) conﬁrms the minimax optimality ofour algorithm (up to logarithmic terms).
No burn-in sample size and full ε -range.
It is noteworthy that the validity of our sample complexity bound (26) is guaranteed for the entire range of ε -levels (i.e., any ε ∈ (0 , H ] ).
This feature isparticularly appealing in the data-starved applications, as it implies that there is no burn-in samplesize needed for our algorithm to work optimally.
Miscellaneous properties of our algorithm.
Finally, we would like to remark in passing thatour learning algorithm enjoys several properties that are practically appealing.
For instance, thealgorithm can be carried out in a decentralized fashion, with two players acting in a symmetric andindependent manner; the algorithm is “rational” in the sense that it converges to the best-responsepolicy if one of the players freezes its policy; and the output policies are inherently Markovian,which depend only on the current state and step number.
All this is achieved under minimal samplecomplexity with the aid of the generative model.
Discussion
The primary contribution of this paper has been to develop a sample-optimal strategy that simultaneously overcome the curse of multiple agents and optimize the horizon dependency when solvingtwo-player zero-sum Markov games.
This goal was not accomplished in any of the previous works,regardless of the sampling mechanism in use.
The adoption of the adversarial learning subroutine helps break the curse of multiple agents compared to the prior model-based approach [64, 38],whereas the availability of the generative model in conjunction with the delicate bonus design enables sharpened horizon dependency compared to Bai et al.
[4], Jin et al.
Our work opensfurther questions surrounding sample efﬁciency in solving Markov games.
For instance, how toattain minimax-optimal sample complexity if we only have access to online sampling protocols (asopposed to the ﬂexible generative model); and how can we optimize the horizon dependency whencomputing coarse correlated equilibria in multi-agent scenarios [55, 27, 20] without compromisingthe dependency on the size of the action spaces.
In addition, our reﬁned regret bound for FTRL(based on variance-type quantities) only covers the full-information case; it would be of interest togeneralize it to partial-information setting (where only partial entries of the loss vectors are observable each time).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contri-
butions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] This is a theoretical work that we do not foresee any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs,internal cluster, or cloud provider)?
[N/A]4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable infor- mation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...