Minimax-Optimal Multi-Agent RL in Zero-Sum Markov Games With a Generative Model
Abstract
This paper is concerned with two-player zero-sum Markov games — arguably the most basic setting in multi-agent reinforcement learning — with the goal of learning a Nash equilibrium (NE) sample-optimally.
All prior results suffer from at least one of the two obstacles: the curse of multiple agents and the barrier of long horizon, regardless of the sampling protocol in use.
We take a ﬁrst step towards settling this problem, assuming access to a ﬂexible sampling mechanism: the generative model.
Focusing on non-stationary ﬁnite-horizon Markov games, we develop a learning algorithm and an adaptive sampling scheme that leverage the optimism principle in adversarial learning (particularly the Followthe-Regularized-Leader (FTRL) method), with a delicate design of bonus terms that ensure certain decomposability under the FTRL dynamics.
Our algorithm learns an ε -approximate Markov NE policy using
[Equation]
samples, where S is the number of states, H is the horizon, and A (resp.
B ) isthe number of actions for the max-player (resp.
min-player).
This is nearly unimprovable in a minimax sense.
Along the way, we derive a reﬁned regret boundfor FTRL that makes explicit the role of variance-type quantities.
Introduction
The thriving ﬁeld of multi-agent reinforcement learning (MARL) studies how a group of interactingagents make decisions autonomously in a shared dynamic environment [79].
The recent developments in game playing [65, 9], self-driving vehicles [57] and multi-robot control [46] are primeexamples of MARL in action.
In practice, there is no shortage of situations where the agents involved have conﬂict of interest and they have to act competitively in order to promote their ownbeneﬁts (possibly at the expense of one another).
Scenarios of this kind are frequently modeled via
Markov games (MGs) [58, 42], a framework that has been a fruitful playground to formalize andstimulate the studies of competitive MARL.
In view of the irreconcilable competition between individual players, solutions of competitive
MARL normally take the form of certain equilibrium strategy proﬁles, which are perhaps best epitomized by the concept of Nash equilibrium (NE).
In a Nash equilibrium, no gain can be realizedthrough a unilateral change, and hence no player has incentives to deviate from her current strategy/policy (assuming no coordination between players).
A myriad of research has been conductedsurrounding NEs, which spans various aspects like existence, learnability, computational hardness,and algorithm design, among others [58, 19, 11, 52, 51, 21, 42, 27, 49, 32].
Sample efﬁciency in zero-sum Markov games
One critical challenge impacting modern MARLapplications
is data efﬁciency.
The players involved often have minimal knowledge about how theenvironment responds to their actions, and have to learn the dynamics and preferable actions byprobing the unknown environment.
For MARL to expand into applications with enormous dimensionality and long planning horizon, the learning algorithms must manage to make efﬁcient use ofthe collected data.
Nevertheless, how to learn NEs with optimal sample complexity remains by and large unsettled,even when it comes to the most basic setting — two-player zero-sum Markov games.
In whatfollows, let us review two representative algorithms developed on this front under two drasticallydifferent sampling protocols, and discuss the shortfalls of these cutting-edge results.
To facilitateprecise comparisons, our discussion concentrates on zero-sum Markov games involving two players,where S is the number of states, H indicates the horizon or effective horizon, and A and B denoterespectively the number
of actions for the max-player and the min-player
• Model-based methods under either a generative model or online exploration.
Assuming access toa generative model (so that one can sample arbitrary state-action combinations), Zhang et al.
[78]investigated a natural model-based algorithm, which performs planning (e
g., value iteration) onan empirical MG derived from samples produced by the generative model.
Focusing on stationarydiscounted inﬁnite-horizon MGs, their algorithm ﬁnds an ε -approximate NE with no more than
[Equation]
In parallel
, Liu et al
studied non-stationary ﬁnite-horizon MGs with online exploration
andobtained similar sample complexity bounds,
i
e.,
[Equation]
samples
[Equation]
episodes
[Equation]
for learning ε -approximate NEs.
While these bounds achieve minimax-optimal dependency onthe horizon H , a major drawback emerges commonly referred to as the curse of multiple agents;namely, these results scale proportionally with the total number of joint actions (
i
e., AB ), aquantity that blows up exponentially with the number of players.• V-learning for online exploration settings.
Focusing on online exploration settings, Bai et al.[5], Jin et al.
[30] proposed an algorithm called V-learning that leverages the advances in onlineadversarial learning (e
g., adversarial bandits) to circumvent the curse of multiple agents.
Thisalgorithm provably yields an ε -approximate NE in non-stationary ﬁnite-horizon MGs using
[Equation]
which effectively brings down the sample size scaling (2) from AB (i
e., the number of jointactions) to A + B (i
e., the sum of individual actions).
It is worth pointing out, however, that thistheory appears sub-optimal in terms of the horizon dependency, as it is a factor of H 2 above theminimax lower bound.
The above results represent the most advanced sample complexity theory thus far for learning NEs intwo-
player
zero-sum MGs.
In summary, these state-of-the-art results — irrespective of the samplingmechanism in use — fall short of overcoming at least one of the two major hurdles: (i) the curse ofmultiple agents , and (ii) the barrier of long horizon .
A natural question to pose is:
Question: can we learn Nash equilibria in a two-player zero-sum Markov gamein a sample-optimal and computation-efﬁcient fashion?
To settle this question favorably, both of the above-mentioned hurdles need to be crossed simultaneously.
Main contributions.
Recognizing that the above question remains open regardless of the sampling scheme in use, this paper takes a ﬁrst step towards solving it assuming access to the mostﬂexible sampling protocol: the generative model (or simulator).
In sharp contrast to the single-agent case where uniform sampling of all state-action pairs sufﬁces [2, 37], the multi-agent scenario requires one to take samples intelligently and adaptively, a crucial step to avoid inefﬁcient use of data(otherwise one cannot hope to break the curse of multiple agents).
With the aim of computing anε -approximate NE in a non-stationary ﬁnite-horizon two-player zero-sum MG, we come up with alearning algorithm (accompanied by an adaptive sampling strategy) that accomplishes this goal withno more than
[Equation]
drawn from the generative model.
Encouragingly, this sample complexity bound matches the minimax lower limit (up to a logarithmic factor).
Our sample complexity theory is valid for the fullε -range (i.e., any ε ∈ (0 , H ] ); this unveils that no burn-in cost whatsoever is needed for our algorithm to achieve sample optimality, which lends itself well to sample-hungry applications.
The proposed algorithm is inspired by two key algorithmic ideas in RL and bandit literature: (1)optimism in the face of uncertainty (by leveraging upper conﬁdence bounds (UCBs) in value estimation), and (2) online and adversarial learning (particularly the Follow-the-Regularized-Leader(FTRL) algorithm).
Note that the optimal design of bonus terms — typically based on certaindata-driven variance estimates — is substantially more challenging than the single-agent case, as itrequires intricate adaptation in response to the policy changes of one another as well as compatibilitywith the FTRL dynamics.
Two points are worth emphasizing (which will be made precise later on):
• The efﬁcacy of FTRL in breaking the curse of multiple agents has been proved in Jin et al.
[30],Song et al.
To improve horizon dependency, one needs to exploit connections between theperformance of FTRL and certain variances.
Towards this, we develop a reﬁned regret bound forFTRL that unveils the role of variance-style quantities, which was previously unavailable.
• The bonus terms entail Bernstein-style variance estimates that mimic the variance-style quantitiesappearing in our reﬁned FTRL regret bounds, and are carefully chosen so as to ensure certaindecomposability over steps.
This is crucial in optimizing the horizon dependency.
Additionally, the policy returned by our algorithm is Markovian (i.e., the action selection probabilitydepends only on the current state s and step h ), and the algorithm can be carried out in a decentralizedmanner without the need of directly observing the opponent’s actions.
Notation.
Let us gather several convenient notation that will be used multiple times.
For anypositive integer n , we set [ n ] := { 1 , · · · , n } .
We shall abuse notation and let 1 and 0 denote theall-one vector and the all-zero vector, respectively.
For a sequence { α k } k ≥ 1 ⊆ (0 , 1] , we deﬁne
[Equation]
for any 1 ≤ i ≤ k .
For a given vector x ∈ R SA (resp.
y ∈ R SAB )
, we denote byx ( s, a ) (resp.
y ( s, a, b ) ) the entry of x (resp.
y )
associated with the state-action combination ( s, a )(resp.
( s, a, b ) ), as long as it is clear from the context.
Next, consider any two vectors a = [ a i ] 1 ≤ i ≤ n and b =
[ b i ] 1 ≤ i ≤ n .
We use a ≤ b (resp.
a ≥ b ) toindicate that a i ≥ b i (resp.
a i ≤ b i ) holds for all i ; we allow scalar functions to take vector-valuedarguments in order to denote entrywise operations (e.g., a 2 =
[ a 2 i ] 1 ≤ i ≤ n and a 4 =
[ a 4 i ] 1 ≤ i ≤ n ); andwe denote by a ◦ b =
[ a i b i ] 1 ≤ i ≤ n the Hadamard product.
For a ﬁnite set A = { 1 , · · · , A } , wedenote by ∆( A ) = { x ∈ R
A | (cid:80) i x i = 1; x ≥ 0 } the probability simplex over A .
For any functionf with domain A (or B ), we adopt the convenient notation
[Equation]
and
[Equation]
Background and models
In this section, we introduce the basics for two-player zero-sum Markov games and the solutionconcept of Nash equilibrum.
Two-player Markov games.
A non-stationary ﬁnite-horizon two-player Markov game, denotedby MG = (cid:8) S , A , B , H, P, r (cid:9) , involves two players competing against each other — with the ﬁrstone called a “max-player” and the second one called a “min-player” — and consists of several keyelements to be formalized below.
To begin with, we denote by S
= { 1 , · · · , S } a shared statespace comprising S different states, and let A = { 1 , · · · , A } (resp.
B = { 1 , · · · , B } ) representthe action space of the max-player (resp.
min-player) containing A (resp.
B ) different actions.
Thehorizon length of this ﬁnite-horizon Markov game is denoted by H .
The probability transition kernelof MG is denoted by P
= { P h } 1 ≤ h ≤ H with P h ∈ R SAB × S ; namely, for any ( s, a, b, h, s (cid:48) ) ∈
S ×
A × B × [ H ] × S , we let P h ( s (cid:48) | s, a, b ) indicate the probability of MG transitioning fromstate s to state s (cid:48) at step h when the max-player takes action a and the min-player takes action b .
Additionally, r = { r h } 1 ≤ h ≤ H with r h ∈ R SAB represents the reward function; namely, for any( s, a, b, h ) ∈ S
×
A × B × [ H ] , r h ( s, a, b ) stands for the immediate reward gained in state s atstep h if the max-player (resp.
min-player) executes action a (resp.
b ).
Given that our focal point isthe family of zero-sum Markov games, it sufﬁces to introduce a single reward function r .
We shallalso assume normalized rewards throughout this paper in the sense that r h ( s, a, b ) ∈ [0 , 1] for any( s, a, b, h ) ∈ S ×
A × B × [ H ] .
Value functions under product policies.
Let µ = { µ h } 1 ≤ h ≤ H (resp.
ν
= { ν h } 1 ≤ h ≤ H ) denotethe policy of the max-player (resp.
min-player), where µ h ( · | s ) ∈ ∆( A ) and ν h ( · | s ) ∈ ∆( B )for any s ∈ S .
More speciﬁcally, µ h ( a | s ) indicates the probability of the max-player selectingaction a in state s at step h ; and ν h ( b | s ) is deﬁned analogously.
Consider a Markovian trajectory{ ( s t , a t , b t , r t ) } 1 ≤ t ≤ H , where s t is the state at time t , a t (resp.
b t ) is the action of the max-player(resp.
min-player) at time t , and r t is the immediate reward observed at time t .
For any givenpolicy µ (resp.
ν ) of the max-player (resp.
min-player) and any step h , we deﬁne the value function
V µ,νh : S → R under the product policy µ × ν as follows:
[Equation]
where the expectation is taken over the Markovian trajectory { ( s t , a t , b t , r t ) } with the max-playerand the min-player executing policies µ and ν , respectively, in an independent fashion; that is,conditional on s t , we draw a t ∼ µ t ( · | s t ) and b t ∼ ν t ( · | s t ) independently , and then s t +1 ∼
P t ( · | s t , a t , b t ) .
In addition, conditional on the min-player executing policy ν , the optimal valuefunction V (cid:63),ν = { V (cid:63),νh } 1 ≤ h ≤ H (with V (cid:63),νh : S → R ) of the max-player is deﬁned as
[Equation]
8a
[Equation]
similarly, when the max-player chooses to execute policy µ , the optimal value function V µ,(cid:63) =
{ V µ,(cid:63)h } 1 ≤ h ≤ H (with V µ,(cid:63)h : S → R ) of the min-player is deﬁned as
[Equation]
Furthermore, if we freeze the policy of the min-player to ν , then the Bellman optimality conditionfor the max-player can be expressed as [8]
[Equation]
Solution concept: Nash equilibrium.
In the zero-sum two-player Markov game, the max-playerwishes to maximize the value function, while the min-player aims to minimize the value functioninstead.
Due to the competing objectives, ﬁnding some sort of equilibrium — particularly the Nashequilibrium [48] — becomes a central topic in the studies of Markov games.
More precisely, apolicy pair ( µ (cid:63) , ν (cid:63) )
∈ ∆( A ) SH × ∆( B ) SH is said to be a (mixed-strategy)
Nash equilibrium of
MG if the resulting product
[Equation]
obeys
[Equation]
and
[Equation]
In other words, conditional on the opponent’s current policy and the assumption that the two playerstake actions independently , no player can harvest any gain by unilaterally deviating from its currentpolicy.
In practice, it might be challenging to compute an “exact” Nash equilibrium, and instead one wouldseek to ﬁnd approximate solutions.
Towards this end, we ﬁnd it helpful to deﬁne the sub-optimalitygap of a policy pair ( µ, ν ) as follows (measured in an (cid:96) ∞ -based manner)
[Equation]
where
[Equation]
11b)
With this sub-optimality measure in place, a policy pair ( µ, ν ) is said to be an ε -approximate Nashequilibrium — or more concisely, ε -Nash — if the resultant sub-optimality gap of the product policyµ × ν obeys NE - gap ( µ, ν ) ≤ ε .
Generative model / simulator.
In reality, we oftentimes do not have access to perfect descriptions(e.g., accurate knowledge of the transition kernel P ) of the Markov game under consideration; instead, one has to learn the true model on the basis of a set of data samples.
When it comes to thedata generating mechanism, this paper assumes access to a generative model (also called a simulator)
[34, 33]: in each call to the generative model, the learner can choose an arbitrary quadruple( s, a, b, h ) ∈ S ×
A × B × [ H ] and obtain an independent sample generated based on the truetransition kernel:
[Equation]
, b ) .
In words, a generative model facilitates query of arbitrary state-action-step combinations, whichhelps alleviate the sampling constraints arising in online episodic settings for exploration.
The goalof the current paper is to compute an ε -approximate Nash equilibrium of MG with as few samplesas possible, i.e., using a minimal number of calls to the generative model.
Algorithm and main theory
In this section, we put forward an algorithm aimed at ﬁnding an ε -Nash policy pair with the assistance of a generative model, and demonstrate its sample optimality for the full ε -range.
3.1 Algorithm
We now describe the proposed algorithm, which is inspired by the optimism principle and the FTRLalgorithm for online/adversarial learning.
Following the dynamic programming approach [8], ouralgorithm employs backward recursion from step h
= H back to h = 1 ; in fact, we shall ﬁnishestimating the values and policies for step h before moving backward to step h − 1 .
For each h , themax-player (resp.
min-player) calls the generative model for K rounds, with each round drawing
SA (resp.
SB ) independent samples; as a result, the total sample size is given by KS ( A + B ) H .
Inwhat follows, let us ﬁrst introduce some convenient notation that facilitates our presentation of thealgorithm.
Notation.
Consider any step h ∈
[ H ] , and any data collection round k ∈
[ K ] .
The algorithmmaintains the following iterates, whose notation is gathered here.
• V h ∈ R S (resp.
V h ∈ R S ) represents the ﬁnal estimate of the value function at step h by the
max-player (resp.
min-player)
; in particular, we set V H
+1 =
V H +
= 0 .
• Q kh ∈ R SA (resp.
Q kh ∈ R SB ) represents the Q-function estimate of the max-player (resp. minplayer) at step h after the k -th round of data collection.• q kh ∈ R SA (resp.
q kh ∈ R SB ) represents a certain “one-step-look-ahead” Q-function estimate of
the max-player (resp.
min-player) at step h using samples collected in the k -th round.
• r kh ∈ R SA (resp.
r kh ∈ R SB ) denotes the sample reward vector for step h received by the max-
player
• β h,V ∈ R S (resp.
β h,V ∈ R S ) denotes the bonus (resp.
penalty) vector chosen by the max-player(resp.
min-player) during ﬁnal value estimation.
• µ kh : S → ∆( A ) (resp.
ν
kh : S → ∆( B ) ) denotes the policy iterate of the max-player (resp. minplayer) at step h before the beginning of the k -th round of data collection; in particular, we set bothµ 1 h and ν 1 h to be uniform, namely, µ 1 h ( a | s )
= 1 /A for any ( s, a ) ∈ S × A and ν 1 h ( b | s )
= 1 /Bfor
any ( s, b ) ∈ S × B .
Crucially, the above objects are all constructed from the perspective of a single player (either themax-player or the min-player), and are all quantities needed to operate a “single-agent” MDP (asopposed to MARL).
As such, the complexity of storing/updating the above objects only scales withthe aggregate size of the individual action space, rather than the size of the product action space.
This is a key point that allows one to overcome the curse of multiple agents.
Main steps of the proposed algorithm.
As mentioned above, our algorithm collects multiplerounds of independent samples for each h .
Let us describe below the proposed procedure for themax-player in the k -th round for step h ; the procedure for the min-player proceeds analogously.1.
Sampling and model estimation.
For each ( s, a ) ∈ S × A , draw an independent sample as follows [Equation] 12a
[Equation]
and receive the reward r k,h,s,a = r h (cid:0) s, a, b k,h,s,a (cid:1) .
These samples are then employed to constructthe sample reward vector r kh ∈ R SA and empirical probability transition kernel P kh ∈ R SA × Ssuch thatr kh
( s, a ) =
r k,h,s,a and
P kh ( s (cid:48) | s, a ) =
(cid:26) 1 , if s (cid:48)
=
s (cid:48)
k,h,s,a 0 , else (12b) for all ( s, a, s (cid:48) )
∈ S ×
A × S .
Note that the max-player only needs to compute (12b), without theneed of directly observing the opponent’s actions (i.e., { b k,h,s,a } ).2.
Q-function estimation.
Following the dynamic programming approach, we ﬁrst compute the “onestep-look-ahead” Q-function estimate as follows
[Equation]
We then adopt the update rule of Q-learning:
[Equation]
where 0 < α k < 1 is the learning rate.
Applying (14) recursively and using the quantities deﬁnedin (5), we easily arrive at the following expansion:
[Equation]
Policy updates.
Once the Q-estimates are updated, we adopt the exponential weights strategy toupdate the policy iterate of the max-player as follows [Equation]
[Equation]
where η k +1 > 0 denotes another learning rate associated with policy updates for the max-player(to be speciﬁed shortly).
In fact, this subroutine implements the Follow-the-Regularized-Leaderupdate rule [55]
[Equation]
with the regularizer F ( · ) chosen to be the negative entropy function F ( π ) :
= (cid:80) a π ( a ) log (cid:0) π ( a ) (cid:1) .
After carrying out K rounds of the above procedure, our ﬁnal policy estimate (cid:98) µ ∈ ∆( A ) SH and thevalue estimate V h : S → R for step h are taken respectively to be
[Equation]
and
[Equation]
18a
[Equation]
with { α Kk } deﬁned in (5), where β h,V ( s ) ≥ 0 is some bonus term (taking the form of some datadriven upper conﬁdence bound) to be speciﬁed momentarily.
In particular, the ﬁnal policy (18a) is amixture of the policy iterates { µ kh } and is Markovian in nature (i.e., the action selection rule dependsonly on the current state s and step h ).
The whole procedure, including the algorithm for the min-player, is summarized in Algorithm 1.
Choices of learning rates.
Thus far, we have not yet speciﬁed the two sequences of learning rates,which we describe now.
The learning rates associated with Q-function updates are set to be rescaledlinear, namely,
[Equation]
for some absolute constant c α ≥ 24 .
In addition, the learning rates associated with policy updatesare chosen to be:
[Equation]
Choices of bonus terms.
It remains to specify the bonus terms, which are selected based on fairlyintricate upper conﬁdence bounds and constitutes a key (and perhaps most challenging) componentof our algorithm design.
Speciﬁcally, we take
[Equation]
for any ( s, h ) ∈ S ×
[ H ] , where c b > 0 is some sufﬁciently large constant; see also (6) for thedeﬁnition of the variance-style term.
As in previous works, the bonus term, which is chosen carefullyin a data-driven fashion, needs to compensate for the uncertainty incurred during the estimationprocess.
3.2 Main results
As it turns out, the proposed algorithm is provably sample-efﬁcient,
whose sample complexity ischaracterized by the following theorem.
Theorem 1.
Consider any ε ∈ (0 , H ] and any 0 < δ < 1 .
Suppose that
[Equation] [Equation]
for some large enough universal constant c k > 0 .
With probability at least 1 − δ , the sub-optimalitygap (cf.
(11) ) of the policies ( (cid:98) µ, (cid:98) ν ) returned by Algorithm 1 obeys
[Equation]
Theorem 1 establishes a sample complexity upper bound for the proposed algorithm, which we takea moment to interpret as follows.
// set initial value estimates to 0, and initial policies to uniform distributions.
set
V H +
( s ) =
V H +
( s )
=
Q 0 h ( s a ) =
Q 0 h ( s, b ) = 0 , µ 1 h ( a | s )
= 1 /A
and ν 1 h ( b | s )
= 1 /B .
3 for to 1 do for k = 1 to K do // draw independent samples, and construct empirical models.
(cid:0) r kh , r kh , P kh , P kh (cid:1) ← sampling (cid:0) h, µ kh
[Equation]
ν kh (cid:1) .
/* see Algorithm 2.
*/ // update Q-estimates with upper/lower confidence bounds.
Compute q kh =
r kh + P kh V
h
+1 , q kh = r kh
+ P kh V
h +1 , and update
[Equation]
// update policy estimates using FTRL.
for ( s, a, b ) ∈ S ×
A × B doµ k
+1 h
[Equation]
− η
[Equation]
.// output the final policy estimate and value estimate for step h .
Update
[Equation]
(cid:98)
ν h =
K (cid:88)
[Equation]
=
[Equation]
ν
[Equation] [Equation]
= max (cid:40) K (cid:88) k
= α Kk (cid:10) µ
[Equation]
( · | s ) , q kh ( s, · ) (cid:11) − β h,V ( s ) ,(cid:41) , ∀ s ∈ S , where β kh,V is given in (21), and β kh,V is obtained by replacing ( µ kh , q kh ) in (21) with ( ν kh , q kh ).
12 Output: (cid:98) µ = { (cid:98) µ h } 1 ≤ h ≤ H and (cid:98) ν
= { (cid:98) ν h } 1 ≤ h ≤ H .
b h,s,a ∼ ν h ( · | s ) , s
[Equation]
h
[Equation]
s
[Equation]
a ∼ P h ( · | s
[Equation]
a
[Equation]
b h,s,a ) .
|
[Equation]
Sample complexity.
When a generative model is available, Theorem 1 asserts that the total number of samples (i.e., KS ( A + B ) H ) needed for Algorithm 1 to yield ε -Nash policies is at most(sample complexity)
[Equation]
As far as we know, this delivers the ﬁrst result — for any sampling protocol — that simultaneouslyovercomes the long-horizon barrier and the curse of multiple agents.
In comparison to Zhang et al.[78] (cf.
, our result reveals that what ultimately matters is the total number of individual actions(i.e.
, A + B ) as opposed to the total number AB of possible joint actions.
Additionally, our resultexhibits improved horizon dependency (by a factor of H 2 ) compared to Bai et al.
[5], Jin et al.
[30](see (3)), although we should emphasize that the online sampling protocol therein is more restrictivethan a generative model.
Minimax optimality.
In order to assess the tightness of our sample complexity bound (26), it ishelpful to look at the minimax lower limit.
Suppose without loss of generality that A ≥ B .
Considerthe extreme scenario where B = 1 , in which case the Nash-equilibrium reduces to the optimal policyof a single-agent MDP with S states and A actions.
It is well-known that for any given accuracylevel ε ∈ (0 , H ] , one can construct a non-stationary MDP with S states and A actions such that noalgorithm can learn an ε -optimal policy with fewer than the order of H 4 SAεsamples [2, 41].
Thismeans that in general, the minimax sample complexity lower bound (w.r.t.
ﬁnding an ε -Nash policypair) scales as
[Equation]
modulo some logarithmic factor; see Appendix G for a formal statement and its proof.
Taking thistogether with (26) conﬁrms the minimax optimality of our algorithm (up to logarithmic terms).
No burn-in sample size and full ε -range.
It is noteworthy that the validity of our sample complexity bound (26) is guaranteed for the entire range of ε -levels (i.e., any ε ∈ (0 , H ] ).
This feature isparticularly appealing in the data-starved applications, as it implies that there is no burn-in samplesize needed for our algorithm to work optimally.
Miscellaneous properties of our algorithm.
Finally, we remark in passing that our learning algorithm enjoys several properties that might be practically appealing.
For instance, the algorithm canbe carried out in a decentralized fashion, with two players acting in a symmetric and independentmanner (without the need of knowing each other’s individual action); the algorithm is “rational” inthe sense that it converges to the best-response policy if one of the players freezes its policy; and theoutput policies are inherently Markovian, which depend only on the current state and step number.
All this is achieved under minimal sample complexity with the aid of the generative model.
Discussion
The primary contribution of this paper has been to develop a sample-optimal paradigm that simultaneously overcomes the curse of multiple agents and optimizes the horizon dependency when solvingtwo-player zero-sum Markov games.
This goal was not accomplished in any of the previous works,regardless of the sampling mechanism in use.
The adoption of the adversarial learning subroutine helps break the curse of multiple agents compared to the prior model-based approach [78, 43],whereas the availability of the generative model in conjunction with the delicate bonus design enables sharpened horizon dependency compared to Bai et al.
[5], Jin et al.
Our work opensfurther questions surrounding sample efﬁciency in solving Markov games.
For instance, how toattain minimax-optimal sample complexity if we only have access to less idealistic sampling protocol (e.g., local access models [39, 75], and online sampling protocols [3, 29]) as opposed to theﬂexible generative model?
How can we optimize the horizon dependency when computing (coarse)correlated equilibria in multi-agent general-sum scenarios [62, 30, 22] without compromising thedependency on the size of the action spaces.
In addition, our reﬁned regret bound for FTRL (basedon variance-type quantities) only covers the full-information case; it would be of interest to generalize it to the bandit-feedback setting (where only partial entries of the loss vectors are observable)
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contri-
butions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] This is a theoreticalwork that we do not foresee any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were cho-
sen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments mul-
tiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[N/A]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[N/A](b)
Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A](d)
Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A](b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
[N/A]