Minimax-Optimal Multi-Agent RL in Markov Games With a Generative Model
Abstract
This paper studies multi-agent reinforcement learning in Markov games, with the goal of learning Nash equilibria or coarse correlated equilibria (CCE) sampleoptimally.
All prior results suffer from at least one of the two obstacles: the curse of multiple agents and the barrier of long horizon, regardless of the sampling protocol in use.
We take a step towards settling this problem, assuming access to a ﬂexible sampling mechanism: the generative model.
Focusing on non-stationary ﬁnite-horizon Markov games, we develop a fast learning algorithm called QFTRL and an adaptive sampling scheme that leverage the optimism principle in online adversarial learning (particularly the Follow-the-Regularized-Leader (FTRL) method).
Our algorithm learns an ε -approximate CCE in a general-sum
Markov game using (cid:101)
[Equation]
samples, where m is the number of players, S indicates the number of states, H is the horizon, and A i denotes the number of actions for the i -th player.
This is minimax-optimal (up to log factor) when m is ﬁxed.
When applied to twoplayer zero-sum Markov games, our algorithm provably ﬁnds an ε -approximate
Nash equilibrium with a minimal number of samples.
Along the way, we derive a reﬁned regret bound for FTRL that makes explicit the role of variance-type quantities, which might be of independent interest.
Introduction
The thriving ﬁeld of multi-agent reinforcement learning (MARL) studies how a group of interacting agents make decisions autonomously in a shared dynamic environment [80].
The recent developments in game playing [66, 9], self-driving vehicles [58] and multi-robot control [45] are prime examples of MARL in action.
In practice, there is no shortage of situations where the agents involved have conﬂict of interest, and they have to act competitively in order to promote their own beneﬁts (possibly at the expense of one another).
Scenarios of this kind are frequently modeled via
Markov games (MGs) [59, 42], a framework that has been a fruitful playground to formalize and stimulate the studies of competitive MARL.
In view of the irreconcilable competition between individual players, solutions of competitive
MARL normally take the form of certain equilibrium strategy proﬁles, which are perhaps best epitomized by the concept of Nash equilibrium (NE) [49].
In a Nash equilibrium, no gain can be realized through a unilateral change — assuming no coordination between players — and hence no player has incentives to deviate from her current strategy/policy.
A myriad of research has been conducted surrounding NE, which spans various aspects like existence, learnability, computational hardness, and algorithm design, among others [59, 20, 12, 53, 52, 22, 42, 28, 50, 33].
Given that ﬁnding NE is notoriously expensive in general (except for special cases like two-player zero-sum MGs) [20, 21], several more tractable solution concepts have emerged in the studies of game theory and MARL, a 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
prominent example being the coarse correlated equilibirum (CCE) [47].
A key compromise made in the CCE is that it permits the players to act in an coordinated fashion, which contrasts sharply with the absence of coordination in the deﬁnition of NE.
One critical challenge impacting modern MARL applications is data efﬁciency.
The players involved often have minimal knowledge about how the environment responds to their actions, and have to learn the dynamics and preferable actions by probing the unknown environment.
For MARL to expand into applications with enormous dimensionality and long planning horizon, the learning algorithms must manage to make efﬁcient use of the collected data.
Nevertheless, how to learn NE and/or CCE with optimal sample complexity remains by and large unsettled even when it comes to the most basic setting: two-player zero-sum Markov games, as we shall discuss below.
Example: inadequacy in learning two-player zero-sum Markov games.
To facilitate concrete comparisons, let us review two representative algorithms aimed at learning NE in two-player zerosum MGs.
These algorithms have been studied under two drastically different sampling protocols, and we shall discuss the shortfalls of the cutting-edge sample complexity results.
In a two-player zero-sum MG, we denote by S the number of states and H the horizon or effective horizon, whereas
A 1 and A 2 denote respectively the number of actions for the max-player and the min-player.
• Model-based methods under either a generative model or online exploration.
Assuming access
to a generative model (so that one can sample arbitrary state-action tuples), Zhang et al.
[79] investigated a natural model-based algorithm, which performs planning (e.g., value iteration) on an empirical MG derived from samples produced non-adaptively by the generative model.
Focusing on stationary discounted inﬁnite-horizon MGs, their algorithm ﬁnds an ε -approximate NE with
[Equation]
samples
[Equation]
In parallel, Liu et al.
[43] studied non-stationary ﬁnite-horizon MGs with online exploration, and obtained similar sample complexity bounds, i.e., (cid:101)
[Equation]
for learning an ε -approximate NE.
While these bounds achieve minimax-optimal dependency on the horizon H , a major drawback emerges — commonly referred to as the curse of multiple agents; namely, these results scale proportionally with the total number of joint actions (i
e., (cid:81) 1 ≤ i ≤ 2 A i ), a quantity that blows up exponentially with the number of players.
• V-learning for online exploration settings.
Focusing on online exploration settings, Bai et al.
[5], Jin et al.
[31] proposed an algorithm called V-learning that leverages the advances in online adversarial learning (e
g., adversarial bandits) to circumvent the curse of multiple agents.
This algorithm provably yields an ε -approximate NE in non-stationary ﬁnite-horizon MGs using
[Equation]
which effectively brings down the sample size scaling (2) from A 1 A 2 (i
e., the number of joint actions) to A 1 + A 2 (i
e., the sum of individual actions).
It is worth pointing out, however
that this theory appears sub-optimal in terms of the horizon dependency, as it is a factor of H 2 above the minimax lower bound.
Key issues and our main contributions.
While the above summary focuses on two-player zerosum MGs, it unveils a fundamental issue surrounding the sample efﬁciency of learning equilibria; that is, all existing results in this front — irrespective of the sampling mechanism in use — fall short of overcoming at least one of the two major hurdles: (i) the curse of multiple agents , and (ii) the barrier of long horizon .
A natural question to pose is:
Question: can we learn a Nash equilibrium in a two-player zero-sum Markov game in a sample-optimal and computation-efﬁcient fashion?
To settle this favorably, both of the above hurdles need to be crossed simultaneously.
Moving beyond two-player zero-sum MGs, it is not surprising to see that general-sum multi-player MGs have to grapple with the aforementioned two hurdles as well.
Thus, the following question also comes into mind when learning CCE (a compromise due to the general intractability of learning NE):
Question: can we learn a coarse correlated equilibrium in a multi-player general-sum Markov game in a sample-optimal and computation-efﬁcient fashion?
Note that these questions remain open regardless of the sampling scheme in use.
This paper takes a ﬁrst step towards solving the problem by assuming access to the most ﬂexible sampling protocol: the generative model (or simulator).
In stark contrast to the single-agent case where uniform sampling of all state-action pairs sufﬁces [3, 38], the multi-agent scenario requires one to take samples intelligently and adaptively, a crucial step to avoid inefﬁcient use of data (otherwise one cannot hope to break the curse of multiple agents).
With the aim of computing an ε -approximate equilibrium in a non-stationary ﬁnite-horizon MG, we come up with a computationally efﬁcient learning algorithm (accompanied by an adaptive sampling strategy) that accomplishes this goal with no more than
[Equation]
drawn from the generative model.
Encouragingly, this sample complexity bound matches the minimax lower limit (up to a logarithmic factor) as long as the number of players m ≥ 2 is a ﬁxed constant or grows only logarithmically in problem parameters.
Our sample complexity theory is valid for the full ε -range (i.e., any ε ∈ (0 , H ] ); this unveils that no burn-in cost is needed for our algorithm to achieve sample optimality, which lends itself well to sample-hungry applications.
The proposed algorithm is inspired by two key algorithmic ideas in RL and bandit literature: (i) optimism in the face of uncertainty (by leveraging upper conﬁdence bounds (UCBs) in value estimation), and (ii) online and adversarial learning (particularly the Follow-the-Regularized-Leader (FTRL) algorithm).
Note that the optimal design of bonus terms — typically based on certain data-driven variance estimates — is substantially more challenging than the single-agent case, as it requires intricate adaptation in response to the policy changes of one another as well as compatibility with the FTRL dynamics.
Two points are worth emphasizing (which will be made precise later on):
• The efﬁcacy of FTRL in breaking the curse of multiple agents has been
illustrated in
Jin
et al
[31], Song et al.
[63], Mao and Ba¸sar [44].
To improve horizon dependency, one needs to exploit connections between the performance of FTRL and certain variances.
Towards this, we develop a reﬁned regret bound for FTRL that unveils the role of variance-style quantities, which was previously unavailable.
• The bonus terms entail Bernstein-style variance estimates that mimic the variance-style quantities appearing in our reﬁned FTRL regret bounds, and are carefully chosen so as to ensure certain decomposability over steps.
This is crucial in optimizing the horizon dependency.
Additionally, the policy returned by our algorithm is Markovian (i
e., the action selection probability depends only on the current state s and step h ), and the algorithm can be carried out in a decentralized manner without the need of directly observing the opponents’ actions.
Notation.
Let us also gather several convenient notation that shall be used multiple times.
For any positive integer n , we write [ n ] := { 1 , · · · , n } .
We shall abuse notation and let 1 and 0 denote the all-one vector and the all-zero vector, respectively.
For a sequence { α k } k ≥ 1 ⊆ (0 , 1] , we deﬁne
[Equation]
for any 1 ≤ i ≤ k .
For a given vector x ∈ R SA (resp.
y ∈ R SAB ), we denote by x
( s, a ) (resp.
y ( s, a, b ) ) the entry of x (resp.
y )
associated with the state-action combination ( s, a ) (resp.
( s, a, b ) ), as long as it is clear from the context.
Next, consider any two vectors a =
[ a i ] 1 ≤ i ≤ n and b =
[ b i ] 1 ≤ i ≤ n .
We use a ≤ b (resp.
a ≥ b ) to indicate that a i ≥ b i (resp.
a i ≤ b i ) holds for all i ; we allow scalar functions to take vector-valued arguments in order to denote entrywise operations (e.g., a 2 =
[ a 2 i ] 1 ≤ i ≤ n and a 4 =
[ a 4 i ] 1 ≤ i ≤ n ); and we denote by a ◦ b =
[ a i b i ] 1 ≤ i ≤ n the Hadamard product.
For a ﬁnite set A = { 1 , · · · , A } , we denote by ∆( A ) =
{ x ∈ R
A | (cid:80) i x i = 1; x ≥ 0 } the probability simplex over A .
For any function f with domain A (or B ), we adopt the notation
[Equation]
and
[Equation]
Background and models
In this section, we introduce the basics for Markov games, as well as the solution concepts of Nash equilibrium and coarse correlated equilibrium.
Markov games.
A non-stationary ﬁnite-horizon multi-player general-sum Markov game , denoted by MG = (cid:8) S , {A i } 1 ≤ i ≤ m , H, P, r (cid:9) , involves m players competing against each other, and consists of several key elements to be formalized below.
Recall that ∆( S ) represents the probability simplex over the set S .
• S = { 1 , · · · , S } is the state space of the shared environment, which comprises S different states.
• For each 1 ≤ i ≤ m , let A i
= { 1 , · · · , A i } represent the action space of the i -th player, which contains A i different actions.
Here and below, we denote
[Equation]
and
[Equation]
Throughout the paper, we shall often use the boldface letter a ∈ A
(resp.
a − i ∈
A − i )
to denote a
joint action proﬁle of all players
resp.
a joint
action proﬁle excluding the i -th player’s action).
• H stands for the horizon length of the Markov game.
• P
=
{ P h } 1 ≤ h ≤ H — with P h : S ×
A → ∆( S ) — denotes the probability transition kernel of MG .
Namely, for any ( s, a , h, s (cid:48) )
∈ S ×A× [ H ] ×S , we let P h ( s (cid:48) | s, a ) indicate the probability of MG transitioning from state s to state s (cid:48) at step h when the joint action proﬁle taken by the players is a . • r
= { r i,h } 1 ≤ h ≤ H, 1 ≤ i ≤ m — with r i,h : S ×
A → [0 , 1] — represents the (deterministic) reward function.
Namely, for any ( s, a , h
) ∈ S ×A× [ H ] , r i,h ( s, a ) stands for the immediate reward the i -th player gains in state s at step h , if the joint action proﬁle is a .
Here and throughout, we assume normalized rewards in the sense that r i,h ( s, a ) ∈ [0 , 1] for any ( s, a , h, i )
∈
S ×
A ×
[ H ] × [ m ] .
As an important special case, a two-player zero-sum Markov game — denoted by MG = (cid:8)
S , {A 1 , A 2 } , H, P, r (cid:9) — satisﬁes r 2 ,h = − r 1 ,h for all h ∈
[ H ] .
Following the convention, we assume that r 1 ,h ≥ 0 for all h ∈ [ H ] , 1 and refer to the ﬁrst (resp.
second) player as the max-player (resp.
the min-player).
Markov policies.
This paper focuses on the class of Markov policies, such that the action selection strategies of the players are determined by the current state s and the step number h , without depending on previously visited states.
To begin with, let π i
= { π i,h } 1 ≤ h ≤ H represent the policy of the i -th player.
Here, π i,h ( · | s ) ∈ ∆( A i ) for any ( s, h ) ∈ S × [ H ] , where π i,h ( a | s ) indicates the probability of the i -th player selecting action a in state s at step h .
The joint Markov policy can be deﬁned analogously: we let π
=
( π 1 , . . .
, π m ) : S ×
[ H ] → ∆( A ) represent a joint Markov policy of all players, where the joint actions of all players in state s and step h are chosen according to the distribution speciﬁed by π h ( · | s )
= ( π 1 ,h , . .
. , π m,h )( · | s ) ∈ ∆( A ) .
For any given joint policy π , we employ π − i to represent the policies of all but the i -th player, and let π − i,h denote the policies of all but the i -th player at step h .
All policies are assumed throughout to be Markovian, except our brief remarks on non-Markovian policies in Section 3.2.
Additionally, a joint policy π is said to be a product policy if π 1 , . . .
, π m are executed in a statistically independent manner (namely, under policy π the players take actions independently), and we shall adopt the notation π = π 1 × · · · × π m to indicate that π is a product policy.
Value functions.
Consider a Markovian trajectory { ( s h , a h ) } 1 ≤ h ≤ H , where s h ∈ S is the state at step h and a h ∈ A is the joint action proﬁle at step h .
For any given joint policy π and any step h ∈ [ H ] , we deﬁne the value function V πi,h : S → R of the
i -th player under policy π as follows:
[Equation]
where the expectation is taken over the Markovian trajectory { ( s h , a h ) } with the m players jointly executing policy π ; that is, conditional on s h , we draw a h ∼ π h ( · | s h ) and then s h +1 ∼
P h ( · | s h , a h ) .
In addition, consider the case where (i) all but the i -th player executes the joint policy π − i and (ii) the i -th player executes policy π (cid:48)
i independently from the other players; we shall denote by V π (cid:48)
i × π − i i,h the resulting value function under this joint policy π (cid:48)
i × π − i .
By optimizing over all π (cid:48)
i , we can further deﬁne
[Equation]
It is known that there exists at least one policy, denoted by π (cid:63)i (cid:0) π − i (cid:1) : S ×
[ H ] → ∆( A i ) and commonly referred to as the best-response policy , that can simultaneously attain V (cid:63),π −
i
i,h ( s ) for all h ∈
[ H ] and all s ∈ S .
It is worth emphasizing that the best-response policy π (cid:63)i (cid:0) π − i (cid:1) is the best among all policies of the i -th player executed independently of π − i .
Furthermore, if we freeze π − i , then the Bellman optimality condition for the i -th player can be expressed as [8]
[Equation]
for all ( s, h, i ) ∈ S ×
[ H ] × [ m ] , where the joint action proﬁle a is composed of a i for the i -th player and a − i for the remaining ones.
Equilibria of Markov games.
In a multi-agent Markov game, each player wishes to maximize its own value function.
Due to the competing objectives, ﬁnding some sorts of equilibria — e.g., the
Nash equilibrium [48] and the coarse correlated equilibrium [47, 2] — becomes a central topic in the studies of Markov games.
Let us introduce these solution concepts below.
• Nash equilibrium.
A product policy π = π 1 × · · · × π m is said to be a (mixed-strategy)
Nash equilibrium of MG if the following holds:
[Equation]
In other words, conditional on the opponents’ current policy and the assumption that all players take actions independently , no player can harvest any gain by unilaterally deviating from its current policy.
• Coarse correlated equilibrium.
A joint policy π is said to be a coarse correlated equilibrium of MG
[Equation]
While a CCE also ensures that no unilateral deviation (performed independently from others) is beneﬁcial, its key distinction from the deﬁnition of NE lies in the fact that it allows the policy to be correlated across the players.
Any NE of MG is, self-evidently, also a CCE.
In practice, it might be challenging to compute an “exact” equilibrium, and instead one would seek to ﬁnd approximate solutions.
Towards this end, we ﬁnd it helpful to deﬁne the sub-optimality gap of a policy π as follows (measured in an (cid:96)
[Equation]
manner)
[Equation]
where
[Equation]
With this sub-optimality measure in place, a product policy π = π 1 × · · · × π m is said to be an ε -approximate NE — or more concisely, ε -Nash — if the resultant sub-optimality gap obeys gap ( π ) ≤ ε .
Similarly, a joint (and possibly correlated) policy π is said to be an ε -approximate CCE — or more concisely, ε -CCE — if gap ( π ) ≤ ε .
Generative model / simulator.
In reality, we oftentimes do not have access to perfect descriptions (e.g., accurate knowledge of the transition kernel P ) of the Markov game under consideration; instead, one has to learn the true model on the basis of data samples.
When it comes to the data generating mechanism, this paper assumes access to a generative model (also called a simulator)
[35, 34]: in each call to the generative model, the learner can choose an arbitrary ( s, a , h ) ∈ S ×
A ×
[ H ] and obtain an independent sample generated based on the true transition kernel: s (cid:48) ∼ P h ( · | s, a ) .
In words, a generative model facilitates query of arbitrary state-action-step tuples, which helps alleviate the sampling constraints arising in online episodic settings for exploration.
The goal of the current paper is to compute an ε -approximate equilibrium (either NE or CCE) of MG with as few samples as possible, i.e., using a minimal number of calls to the generative model.
Sample-efﬁcient learning with a generative model
In this section, we put forward an efﬁcient algorithm aimed at learning an ε -approximate equilibrium with the assistance of a generative model, and demonstrate its sample optimality for the full ε -range.
3.1 Algorithm description
We now describe the proposed algorithm, which is inspired by the optimism principle and the FTRL algorithm for online/adversarial learning.
Following the dynamic programming approach [8], our algorithm employs backward recursion from step h
= H back to h = 1 ; in fact, we shall ﬁnish the sampling and learning processes for step h before moving backward to step h − 1 .
For each h , the i -th player calls the generative model for K rounds, with each round drawing SA i independent samples; as a result, the total sample size is given by KSH (cid:80) mi =1 A i .
In what follows, let us
ﬁrst
introduce some convenient notation that facilitates our exposition of the algorithm.
Notation.
Consider any step h ∈
[ H ] , any player i ∈ [ m ] , and any data collection round k ∈
[ K ] .
The algorithm maintains the following iterates, whose notation is gathered here with their formal deﬁnitions introduced later.
• (cid:98) V i,h ∈ R S represents the ﬁnal estimate of the value function at step h by the i -th player; in particular, we set (cid:98) V i,
H +
= 0 .
• Q ki,h ∈ R SA i represents the Q-function estimate of the i -th player at step h after the k -th round of data collection.
• q ki,h ∈ R SA i stands for a certain “one-step-look-ahead” Q-function estimate of the i -th player at step h using samples collected in the k -th round.
• r ki,h ∈ R SA i denotes the sample reward vector for step h received by the i -th player in the k -th round.
• P ki,h ∈ R SA i × S denotes the empirical probability transition matrix for step h constructed using the samples collected by the i -th player in the k -th round.
• β
i,h ∈ R S denotes the bonus vector chosen by the i -th player at step h during ﬁnal value estimation.
• π ki,h : S → ∆( A i ) denotes the policy iterate of the i -th player at step h before the beginning of the k -th round of data collection; in particular, we set π 1 i,h to be uniform, namely, π 1 i,h ( a i | s )
= 1 /A
i for any ( s, a i ) ∈ S
× A i .
Crucially, the above objects are all constructed from the perspective of a single player, and hence resemble those needed to operate a “single-agent” MDP (as opposed to MARL).
As such, the complexity of storing/updating the above objects only scales with the aggregate size of the individual action space, rather than the size of the product action space.
Main steps of the proposed algorithm.
As mentioned above, our algorithm collects multiple rounds of independent samples for each h .
In what follows, let us describe the proposed procedure for the i -th player in the k -th round for step h .
Sampling and model estimation.
For each ( s, a i ) ∈ S × A i , draw an independent sample as follows
[Equation]
and
[Equation]
(14a) where a ( k, h, s, a i ) =
[ a j ( k, h, s, a i )] 1 ≤ j ≤ m ∈
A consists of independent individual actions drawn from a
[Equation]
( k, h, s, a
i
[Equation]
ind.
∼ π
[Equation] [Equation]
= i ) and a i ( k, h, s, a i ) =
a i .
(14b)
These samples are then employed to construct the sample reward vector r ki,h ∈ R SA i and empirical probability transition kernel P ki,h ∈ R SA i × S such that r ki,h ( s, a i ) =
r k,i,h,s,a i and
P ki,h ( s (cid:48) | s, a i ) =
(cid:26) 1 , if s (cid:48)
=
s (cid:48)
k,h,s,a i 0 , else (14c) for all ( s, a i , s (cid:48) )
∈ S ×
A i × S .
Note that the i -th player only needs to compute (14c), without the need of directly observing the other players’ actions.
Q-function estimation.
Following the dynamic programming approach, we ﬁrst compute the “onestep-look-ahead” Q-function estimate as follows
[Equation]
We then adopt the update rule of Q-learning:
[Equation]
16) where 0 < α k < 1 is the learning rate.
Applying (16) recursively and using the quantities deﬁned in (5), we easily arrive at the following expansion:
[Equation] [Equation]
Policy updates.
Once the Q-estimates are updated, we adopt the exponential weights strategy to update the policy iterate of the i -th player as follows
[Equation]
where η k +1 > 0 is another learning rate associated with policy updates (to be speciﬁed shortly).
In fact, this subroutine implements the Follow-the-Regularized-Leader strategy [56]:
[Equation]
where the regularizer F ( · ) is chosen to be the negative entropy function F ( µ ) :
=
[Equation]
a
[Equation]
i
[Equation]
a
[Equation]
After carrying out K rounds of the above procedure, our ﬁnal policy estimate (cid:98) π : S × [ H ] → ∆( A ) and the value estimate (cid:98) V i,h : S → R for step h are taken respectively to be
[Equation]
and
[Equation]
20a
[Equation]
20b) for any (cid:0) s,
a =
[ a 1 , . .
, a m ] (cid:1) ∈ S×A , where { α Kk } is deﬁned in (5) and β i,h ( s ) ≥ 0 is some bonus term (taking the form of some data-driven upper conﬁdence bound) to be speciﬁed momentarily.
It is worth pointing out that the ﬁnal policy (20b) takes the form of a mixture of product policies .
In the special case of two-player zero-sum MGs, we can alternatively output a product policy ( two-player zero-sum MGs )
[Equation]
(21) where for each i = 1 , 2 , we take (cid:98) π i = { (cid:98)
π
i,h } 1 ≤ h ≤ H with (cid:98) π
i,h = (cid:80) Kk
=
α Kk π ki,h .
π = { (cid:98) π h } 1 ≤ h ≤ H , where (cid:98)
π h
= (cid:80)
Kk
=1 α Kk (cid:0)
π
k 1 ,h × · · · × π km,h (cid:1) .
The whole procedure is summarized in Algorithm 1.
Choices of learning rates.
Thus far, we have not yet speciﬁed the two sequences of learning rates, which we describe now.
The learning rates associated with Q-function updates are set to be rescaled linear, namely,
[Equation]
for some constant c α ≥ 24 .
In addition, the learning rates associated with policy updates are chosen to be:
[Equation]
Choices of bonus terms.
It remains to specify the bonus terms, which are selected based on fairly intricate upper conﬁdence bounds.
This constitutes a key — and perhaps the most challenging — component of our algorithm design.
Speciﬁcally, we take
[Equation]
for any ( i, s, h ) ∈ [ m ] × S ×
[ H ] , where c b > 0 is some sufﬁciently large constant; see also (6) for the deﬁnition of the variance-style quantity.
As in previous works, the bonus terms, which are chosen carefully in a data-driven fashion, need to compensate for the uncertainty incurred during the estimation process.
3.2 Main results
As it turns out, the proposed algorithm is tractable and provably sample-efﬁcient.
We begin by characterizing its sample complexity when learning Nash equilibria in two-player zero-sum MGs,
2 for do
Draw an independent sample from the generative model: s (cid:48) s,a i ∼ P h (cid:0) · | s, a ( s, a i ) (cid:1) , where a ( s, a i ) =
[ a j ( s, a i )] 1 ≤ j ≤ m is composed of independent individual actions drawn from a j ( s, a i ) ind.
∼ π j,h ( · | s ) ( j (cid:54) = i ) and a i ( s, a i ) = a i .
Set r
( s, a i ) =
r i,h (cid:0) s, a ( s, a i ) (cid:1) and P (cid:0) s (cid:48)
s,a i | s, a i (cid:1)
= 1 .
5 Return: (cid:0) r, P (cid:1) .
and then shift attention to learning CCE in multi-player general-sum MGs (given the intractability of learning NEs in general).
Theorem 1 (NE for two-player zero-sum MGs) .
Consider a two-player zero-sum Markov game, and consider any ε ∈ (0 , H ] and any 0 < δ < 1 .
Suppose that
[Equation]
(26) for some large enough universal constant c k > 0 .
With probability at least 1 − δ , the product policy (cid:98) π 1 × (cid:98) π 2 computed by Algorithm 1 is an ε -approximate Nash equilibrium, i.e., its sub-optimality gap (cf.
obeys gap (cid:0)(cid:98)
π 1 × (cid:98)
π 2 (cid:1) ≤ ε .
Theorem 2 (CCE for multi-player general-sum MGs) .
Consider an m -player general-sum Markov game, and consider any ε ∈ (0 , H ] and any 0 < δ < 1 .
Suppose that
[Equation]
for some large enough universal constant c k > 0 .
With probability at least 1 − δ , the joint policy (cid:98) π returned by Algorithm 1 is an ε -approximate CCE, i.e., its sub-optimality gap (cf.
obeys gap (cid:0)(cid:98) π (cid:1) ≤ ε .
Theorems 1-2 establish sample complexity upper bounds for the proposed algorithm, which we take a moment to interpret as follows.
The proofs of these two theorems are postponed to Appendix C in the supplementary material.
Sample complexity.
When a generative model is available, Theorems 1-2 assert that the total number of samples (i.e., KSH (cid:80) i A i ) needed for Algorithm 1 to work is
[Equation]
for learning
[Equation]
-NE
[Equation] [Equation]
learning
[Equation]
As far as we know, our theorems deliver the ﬁrst results that uncover the plausibility of simultaneously overcoming the long-horizon barrier and the curse of multi-agents.
Let us compare (28) with prior art.
• NE in two-player zero-sum MGs.
First, consider learning ε -NE policies in two-player zero-sum MGs.
In comparison to
Zhang et
al.
79] (cf.
, our result reveals that what ultimately matters is the total number of individual actions (i
e., A 1 + A 2 ) as opposed to the total number A 1 A 2 of possible joint actions; additionally, our results exhibit improved horizon dependency (by a factor of H 2 ) compared to Bai et al.
[5], Jin et al.
[31] (see (3)), although we remark that the online sampling protocol therein is clearly more restrictive than a generative model.
• CCE in multi-player general-sum MGs (for a ﬁxed m ).
Similar messages carry over to the task of learning multi-player general-sum MGs when the number of players m is a ﬁxed constant.
Liu et al.
[43] provided the ﬁrst non-asymptotic result on learning CCE in the exploration setting; the model-based algorithm studied therein learns an ε -CCE using
[Equation]
which is sub-optimal in terms of the dependency on both H and S and suffers from the curse of multiple agents.
A more recent strand of works focused on a type of online RL algorithms called Vlearning, which exploited the effectiveness of adversarial learning subroutines in overcoming the curse of multi-agents [44, 63, 31]; along this line, the state-of-the-art sample complexity bound is [31]:
[Equation]
which remains suboptimal in terms of the horizon dependency.
As a drawback of these works, the policy returned by V-learning is non-Markovian, an issue that has been recently addressed by
Daskalakis et al.
[23] at the price of a much higher sample complexity.
It is worth emphasizing that all these works assume the online exploration setting as opposed to the scenario with a generative model.
Minimax optimality.
To assess the tightness of our result (28), it is helpful to look at the information-theoretic limit.
Following the minimax lower bound for single-agent MDPs [3, 41], one can develop a minimax sample complexity lower bound for Markov games (w.r.t.
ﬁnding either an ε -NE or an ε -CCE) that scales as
[Equation]
modulo some logarithmic factor, provided that m is a ﬁxed constant or grows only logarithmically in problem parameters; see Appendix E.3 in the supplementary material for a formal statement and its proof.
Taking this together with (28) conﬁrms the minimax optimality of our algorithm (up to logarithmic terms) when the number m of players is ﬁxed.
No burn-in sample size and full ε -range.
It is noteworthy that the validity of our sample complexity bound (28) is guaranteed for the entire range of ε -levels (i.e., any ε ∈ (0 , H ] ).
This feature is particularly appealing in the data-starved applications, as it implies that there is no burn-in sample size needed for our algorithm to work optimally.
Miscellaneous properties of our algorithm.
Finally, we would like to remark in passing that our learning algorithm enjoys several properties that might be practically appealing.
For instance, the output policies are Markovian in nature, which depend only on the current state s and step number h .
This is enabled thanks to the availability of the generative model, which allows us to settle the sampling and learning process for step h + 1 completely before moving backward to step h ; in contrast, the online sampling protocol studied in Bai et al.
[5], Jin et al.
[31] cannot be implemented in this way without incurring information loss.
In addition, our algorithm can be carried out in a decentralized fashion (except that the ﬁnal estimate (cid:98) π needs to aggregate policy iterates from all players), with each player acting in a symmetric yet independent manner (without the need of knowing each other’s individual action).
Our algorithm is also “rational” in the sense that it converges to the best-response policy of a player if all other players freeze their policies.
All this is achieved under minimal sample complexity with the aid of the generative model.
Discussion
The primary contribution of this paper has been to develop a sample-optimal paradigm that simultaneously overcomes the curse of multiple agents and optimizes the horizon dependency when solving multi-player Markov games.
This goal was not accomplished in any of the previous works, regardless of the sampling mechanism in use.
The adoption of the adversarial learning subroutine helps break the curse of multiple agents compared to the prior model-based approach [79, 43], whereas the availability of the generative model in conjunction with the variance-aware bonus design improves horizon dependency compared to Bai et al.
[5], Jin et al.
Acknowledgements: Y. Chen is supported in part by the Alfred P. Sloan Research Fellowship, the
Google Research Scholar Award, the AFOSR grant FA9550-22-1-0198, the ONR grant N00014-221-2354, and the NSF grants CCF-2221009, CCF-1907661, IIS-2218713 and IIS-2218773.
Y. Wei is supported in part by the the NSF grants CCF-2106778, DMS-2147546/2015447 and CAREER award DMS-2143215.
Y. Chi are supported in part by the grants ONR N00014-19-1-2404, NSF
CCF-2106778 and DMS-2134080, and CAREER award ECCS-1818571.
Part of this work was done while G. Li, Y. Wei and Y. Chen were visiting the Simons Institute for the Theory of Computing.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] This is a theoretical work that we do not foresee any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Other related works
Let us discuss in passing additional prior works on learning equilibrium solutions in MARL, which have attracted an explosion of interest in recent years.
While the Nash equilibrium is arguably the most compelling solution concept in Markov games, the ﬁnite-sample/ﬁnite-time studies of NE learning concentrate primarily on two-player zero-sum MGs (e.g., Bai and Jin [4], Chen et al.
Mao and Ba¸sar [44], Wei et al.
[70], Tian et al.
[64], Cui and Du [17, 18], Zhong et al.
[83], Jia et al.
[29], Yang and Ma [76], Yan et al.
[74], Dou et al.
[24]), mainly because computing NEs becomes, for the most part, computationally infeasible (i.e., PPAD-complete) when going beyond two-player zero-sum MGs [20, 21].
Roughly speaking, previous NE-ﬁnding algorithms for two-player zero-sum
Markov games can be categorized into model-based algorithms [52, 79, 43], value-based algorithms [4, 5, 73, 54, 31, 15], and policy-based algorithms [10, 22, 71, 82, 14, 81, 11].
In particular, Bai et al.
[5], Jin et al.
[31] developed the ﬁrst algorithms to beat the curse of multiple agents in twoplayer zero-sum MGs, while Jin et al.
[31], Daskalakis et al.
[23], Mao and Ba¸sar [44], Song et al.
[63] further demonstrated how to accomplish the same goal when learning other computationally tractable solution concepts (e.g., coarse correlated equilibria) in general-sum multi-player Markov games.
The recent works Cui and Du [17, 18], Yan et al.
[74] studied how to alleviate the sample size scaling with the number of agents in the presence of ofﬂine data, with Cui and Du [18] providing a sample-efﬁcient algorithm that also learns NEs in multi-agent Markov games (despite computational intractability).
We shall also brieﬂy remark on the prior works that concern RL with a generative model.
While there are multiple sampling mechanisms (e.g., online exploratory sampling, ofﬂine data) that bear practical relevance, the generative model (or simulator) serves as an idealistic sampling protocol that has received much recent attention, covering the design of various model-based, model-free and policy-based algorithms [35, 69, 1, 3, 38, 32, 60, 67, 68, 39, 34, 61, 51, 36, 13, 26, 7, 71, 65, 77, 72, 46, 75, 78, 25].
In single-agent RL, the model-based approach has been shown to be minimaxoptimal for the entire ε -range [38, 1, 3].
When it comes to multi-agent RL, sample-efﬁcient solutions with a generative model have been proposed in the recent works [62, 19, 79], although a provably sample-optimal strategy was previously unavailable.
B Regret bounds for FTRL via variance-type quantities
Before embarking on our analysis for Markov games, we take a detour to study the celebrated
Follow-the-Regularized-Leader algorithm for online weighted linear optimization, which plays a central role in the analysis of Markov games.
B.1 Setting: online learning for weighted linear optimization
Let (cid:96) 1 , . . .
, (cid:96) n ∈ R
A represent an arbitrary sequence of non-negative loss vectors.
We focus on the following setting of online learning or adversarial learning [37]: in each round k ,
the learner makes a randomized prediction by choosing a distribution π k ∈ ∆( A ) over the actions in A = { 1 , · · · , A } ; 2.
subsequently, the learner observes the loss vector (cid:96) k , which is permitted to be adversarially chosen.
To evaluate the performance of the
learner
, we resort to a regret metric w.r.t.
a certain weighted linear objective function.
To be precise, consider a non-negative sequence { α k } 1 ≤ k ≤ n with 0 ≤ α k ≤
; for each 1 ≤ k ≤ n , we deﬁne recursively
the following weighted average of the
loss vectors:
[Equation]
=and
[Equation]
which can be easily shown to enjoy the following expression
[Equation]
with α ki deﬁned in (5).
When the sequential predictions made by the learner are { π k } k ≥ 1 , we deﬁne the associated regret w.r.t.
the above weighted sum of loss vectors as follows:
[Equation]
with R n
[Equation]
a
[Equation]
which compares the learner’s performance (i.e., the expected loss of the learner over time if it draws actions based on π k in round k ) against that of the best ﬁxed action in hindsight.
B.2 Reﬁned regret bounds for FTRL
Follow-the-Regularized-Leader.
The FTRL algorithm [57, 55] tailored to the above online optimization setting adopts the following update rule:
[Equation]
where η k +1 > 0 denotes the learning rate, and F ( · ) is some convex regularization function employed to stabilize the learning process [56].
Throughout this section, we restrict our attention to negativeentropy regularization, namely,
[Equation]
which allows one to express the FTRL update rule as the following exponential weights strategy (see, e.g., Lattimore and Szepesvári [37, Section 28.1])
[Equation] [Equation]
This update rule is also intimately connected to online mirror descent [37].
Reﬁned regret bounds via variance-style quantities.
As it turns out, the regret of FTRL can be upper bounded by certain (weighted) variance-type quantities, as asserted by the following theorem.
Theorem 3.
Suppose that 0 < α 1 ≤ 1 and η 1 = η 2 (1 − α 1 ) .
Also, assume that 0 < α k < 1 and 0 < η k +1 (1 − α k ) ≤ η k for all k ≥ 2 .
In addition, deﬁne
[Equation]
Then the regret (cf.
(32) ) of the FTRL algorithm satisﬁes
[Equation]
where for any (cid:96) ∈ R A and any π ∈ ∆( A ) we deﬁne
[Equation]
Remark 1 .
Note that the FTRL algorithm and the data generating process in this section are both described in a completely deterministic manner; no randomness is involved in the above theorem even though we introduce the variance-style quantities.
The proof of Theorem 3 is postponed to Appendix D. Let us take a moment to discuss the key distinction between Theorem 3 and prior theory.
• A key term in the regret bound (36) is a weighted sum of the “variance-style” quantities { Var π k ( (cid:96) k ) } .
In comparison, prior regret bounds typically involve the norm-type quantities (e
g., the inﬁnity norms {(cid:107) (cid:96) k
(cid:107) 2 ∞ } ) as opposed to the “variances”; see, for instance, Lattimore
and
Szepesvári [37, Corollary 28.8] for a representative existing regret bound that takes the form of the sum of {(cid:107) (cid:96) k
(cid:107) 2 ∞ } that takes the form of the sum of {(cid:107) (cid:96) k (cid:107) 2 ∞ } .
2 While Var ( (cid:96) k ) ≤ (cid:107) (cid:96) k (cid:107) 2 ∞ is orderwise tight in the worst-case scenario for a given iteration k , exploiting the problem-speciﬁc variance-type structure across time is crucial in sharpening the horizon dependence in many RL problems (e.g., Azar et al.
[3], Jin et al.
[30], Li et al.
• The careful reader would remark that the ﬁnal term of (36) relies on the inﬁnity norm (cid:107) (cid:96)
k (cid:107) ∞ as well.
Fortunately, when the products of the learning rates (cid:98) η k α k are chosen to be diminishing (which is the case in our analysis for Markov games), the number of iterations obeying (cid:98) η k α k (cid:107) (cid:96) k (cid:107) ∞ > 1 / 3 is reasonably small, thus ensuring that this term does not exert too much of an inﬂuence on the regret bound
C Proof of Theorems 1-
To begin with, we claim that Theorem 1 is a direct consequence of Theorem 2.
Towards this, note that in a two-player zero-sum Markov game, it is self-evident that (cid:98) π − 1 = (cid:98) π 2 and (cid:98) π − 2 = (cid:98) π 1 (see line 12 of Algorithm 1).
Consequently, Theorem 2 (if it is valid) reveals that
[Equation]
for all s ∈ S .
Moreover, recalling that r 1 ,h = − r 2 ,h for all h ∈
[ H ] , one has V π 1 , 1 ( s ) =
− V π 2 , 1 ( s ) for any joint policy proﬁle π , which taken collectively with (37) results in
[Equation]
Analogously, one has
[Equation]
s
[Equation]
2 ε .
Replacing ε with ε/ 2 immediately establishes
Theorem 1.
With the above argument in mind, the remainder of this section is devoted to proving Theorem 2.
C.1 Preliminaries and notation
Let us start with some preliminary facts and notation.
Given that ε ≤ H , the assumption (27) requires
[Equation]
for some large enough constant c k > 0 , which will be a condition assumed throughout the proof.
We also gather below several basic facts about our choices of learning rates { α i } (cf.
and the corresponding quantities { α ki } (cf.
Lemma 1.
For any k ≥ 1 , one has
[Equation]
In addition, if k ≥ c α log K + 1 and c α ≥ 24 , then one has
[Equation]
Proof.
The result (39a) is standard and has been recorded in previous works (e.g., Jin et al.
Appendix B]).
Regarding (39b), we note that for any i ≤ k/ 2 and k ≥ c α log K + 1 , α ki ≤ k (cid:89)
[Equation]
where we have used the fact that α k =
[Equation]
and the assumption c α ≥ 24 .
Additionally, recognizing the deﬁnition in (15) and the upper bound (cid:98) V i,h +1 ( s ) ≤ H − h (cf. (20a)), we make note of the range of the iterates (cid:8) q ki,h (cid:9) as follows.
Lemma 2.
For any i ∈ [ m ] and any ( h, k, s, a i ) ∈ [ H ] ×
[ K ] × S × A i , it holds that
[Equation]
Next, we introduce several additional notation that helps simplify our presentation of the proof.
For any policy µ : S × [ H ] → ∆( A i ) , we adopt the convenient notation
[Equation]
We shall also employ the expectation operator E h,k − 1 [ · ] (resp.
variance operator Var h,k − 1 [ · ] ) to denote the expectation (resp. variance) conditional on what happens before the beginning of the k -th round of data collection for step h (see Section 3.1 about the data collection process).
C.2 Proof outline
With the above preliminaries in place, we are in a position to present our analysis.
Recall that the joint policy (cid:98) π computed by Algorithm 1 takes the form of a mixture of product policies
[Equation]
at step h .
Consequently, the value function under policy (cid:98) π satisﬁes the following Bellman equation:
[Equation]
42b
[Equation]
for all ( i, s, h ) ∈ [ m ] × S ×
[ H ] .
To establish Theorem 2, we seek to prove the following inequality:
[Equation]
where we remind the reader of the deﬁnition of V (cid:63), (cid:98) π − i i,in (9).
Towards this
[Equation]
let us introduce the following best-response policy of the i -th player:
[Equation]
We make note of the following key decomposition:
[Equation]
where we deﬁne the following auxiliary value functions:
[Equation]
with V (cid:101)
[Equation]
(cid:63)i
[Equation]
(cid:98)
[Equation] [Equation]
45a)
[Equation]
with V (cid:63)
[Equation] [Equation]
45b
[Equation]
with V (cid:98) π i,H +
=
(45c)
Here, we have used the elementary fact V (cid:101) π (cid:63)i × (cid:98) π − i i,h ≤ V (cid:63), (cid:98) π − i i,h .
We shall establish bounds for the above terms in (44), which consists of three steps as outlined below.
Step 1: showing that (cid:98) V i,h is an entrywise upper bound on V (cid:63), (cid:98) π − i
i,h .
The following lemma ascertains that the value estimate (cid:98) V i,h of the i -th player returned by Algorithm 1 is an optimistic estimate of the auxiliary value V (cid:63), (cid:98) π − i i,h deﬁned in (45b).
Evidently, this result cannot happen unless the bonus terms are suitably chosen.
Lemma 3.
With probability at least 1 − δ , it holds that
[Equation]
The proof of this lemma is postponed to Appendix E.1.
Armed with Lemma 3, we can further bound (44) as follows
[Equation]
Step 2: establishing a key recursion.
Recall the deﬁnition of π kh in (41).
Let us deﬁne the following auxiliary reward vectors r (cid:98)
πi
[Equation]
, r i,h ∈ R S as well as the auxiliary probability transition matrices P (cid:98) πi
[Equation]
[Equation]
, P i,h ∈ R S × S such that: for any s, s (cid:48) ∈ S , r (cid:98) πi,h ( s ) :=
K (cid:88) k
= α
[Equation]
E a ∼ π kh ( s ) (cid:2) r i,h ( s, a ) (cid:3) , (48a)
P (cid:98) πi,h ( s, s (cid:48) ) :
=
K (cid:88) k
=
[Equation]
Kk
E a
[Equation]
kh ( s ) (cid:2)
[Equation] [Equation]
(cid:48) | s, a ) (cid:3) , (48b) r (cid:101)
[Equation]
(cid:63)i × (cid:98)
[Equation]
− i i,h ( s )
[Equation]
=
[Equation]
(cid:88) k
=
[Equation]
E (
[Equation]
a − i )
[Equation]
(cid:101)
[Equation]
(cid:63)i
[Equation]
s
[Equation]
π
[Equation]
− i
[Equation]
h ( s ) (cid:2)
[Equation]
,h ( s
[Equation]
a ) (cid:3) , (48c)
P (cid:101) π (cid:63)i × (cid:98) π − i i,h ( s, s (cid:48) ) :
=
K (cid:88) k
=
[Equation]
Kk
E ( a i , a − i )
[Equation]
(cid:101)
[Equation]
(cid:63)i,h ( s ) × π k − i,h ( s ) (cid:2)
P h ( s (cid:48) | s
[Equation]
a ) (cid:3) , (48d) r i,h ( s ) :=
K (cid:88) k
=
[Equation]
Kk (cid:88) a i ∈A i π ki,h ( a i | s ) r ki,h ( s, a i ) , (48e)
P i,h ( s, s (cid:48) ) :
=
K (cid:88) k
= α Kk (cid:88)
[Equation]
48f)
As it turns out, V (cid:98) πi,h (resp.
V (cid:101) π (cid:63)i × (cid:98) π − i i,h , (cid:98) V
i,h ) stays reasonably close to the “one-step-look-ahead” expression
[Equation]
, as revealed by the recursive relations stated in the following lemma; the proof of this lemma is deferred to
Appendix E.2.
Lemma 4.
There exists some universal constant c 3 > 0 such that with probability exceeding 1 − δ ,
[Equation]
[Table] [Equation] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
hold for all h ∈
[ H ] .
Remark 2 .
The right-hand side of each of the bounds in (49) contains a variance-style term (e.g., those terms taking the form of P
i,h ( V i,h +1 ◦ V i,h +1 ) − ( P i,h V
i,h +1 ) ◦ ( P i,h V
i,h +1 ) for some prob- ability transition matrix P
i,h and value vector V i,h +1 ).
Such variance-style terms are direct consequences of our Bernstein-style bonus terms, and are crucial in optimizing the horizon dependency.
With the above lemma in place, one can readily show that
[Equation]
[Table]
[Equation]
for some large enough constant c 4 > 0 , where the last line holds due to Condition (38), the basic fact
P (cid:98) πi,h
[Equation]
V
[Equation]
πi,h +1 ◦ V
[Equation]
πi,h +
[Equation]
≥
[Equation]
P (cid:98) πi,h V (cid:98) πi,h +1 (cid:1) ◦ (cid:0) P
[Equation]
πi,h V
[Equation]
πi
[Equation]
h +1 (cid:1) , and the following fact (for large enough c 4 )
[Equation]
In addition
[Equation]
recalling that (cid:107) V (cid:98) πi,h (cid:107) ∞ , (cid:107) V (cid:98) πi,h +1 (cid:107) ∞
≤ H (cf. (20a)) and recognizing that ζ 0 ≥ 0 (see (50))
[Equation]
we can demonstrate that
[Equation]
This further leads to
[Equation]
, which can be rearranged to yield
[Equation]
Substituting it into (49a) and combining terms give
[Equation]
where we take c 5 = c 3 + c 3 c 4 .
An analogous argument (which is omitted here for brevity) also reveals that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Step 3: invoking the key recursion to establish the desired bound.
We ﬁnd it helpful to introduce the following notation (please note the order of the matrix product)
[Equation]
Armed with this notation
[Equation]
we can invoke the relation (52) recursively and use V (cid:98) πi,h
+1 = V (cid:98) πi,h
+1 =
to obtain
[Equation] [Equation]
Here, (i) uses the Bellman equation; (ii) applies the bound (52) recursively; (iii) holds since for any transition matrices { P i,h } and any sequence { V i,h } obeying V i,h +1 = 0 , one can use the telescoping sum to obtain
H
[Equation]
whereas the last inequality in (56) arises from the assumption (27) when c k is large enough.
Similarly, replacing (cid:98) π i with (cid:101) π (cid:63)i in the above argument and recalling (53) directly lead to
[Equation]
In addition
[Equation]
recalling the deﬁnition of V (cid:98) πi,h (cf. (45c)), r i,h and P i,h (see (48))
[Equation]
we can deduce that
[Equation]
[Table]
[Equation]
which resembles (55).
Thus
[Equation]
repeating the above argument for (56) and applying (54) recursively, we reach
[Equation]
To ﬁnish up, combining (56)
[Equation]
(57) and (58) with (47)
[Equation]
we arrive at
[Equation]
This establishes the ﬁrst inequality in (43)
[Equation]
while the second inequality in (43) can be validated via the same argument.
We have thus completed the proof of Theorem 2.
D Proof of Theorem
This section is devoted to presenting the proof of Theorem 3.
Before embarking on the analysis, let us introduce a convenient auxiliary iterate
[Equation]
or equivalently,
[Equation]
for all a
[Equation]
A
[Equation] [Equation]
which differs from (34) only in the learning rates being used (namely, π k +1 uses η k +1 while π − k + adopts (cid:98) η k ).
D.1 Main steps of the proof
The key steps of the proof lie in justifying the following two claims:
[Equation]
) and for all a ∈ A and all k ≥ 1 ,
[Equation]
62) where for any vector (cid:96) ∈
R
A we deﬁne
[Equation]
In words, the ﬁrst claim (61) allows us to replace the action that appears best in hindsight (cf.
(32)) by the time-varying predictions { π − k +1 } without incurring much cost, whereas the second claim (62) controls the proximity of π − k +1 and π k in each round.
Let us assume the validity of these two claims for the moment, and return to prove them shortly.
In view of the upper bound (61), we are in need of controlling (cid:10) π k − π − k +
[Equation]
, (cid:96) k (cid:11) .
We divide into two cases.
• For any k obeying (cid:98)
η k α k (cid:107) (cid:96) k
(cid:107) ∞ > 1 / 3 , invoke (62) and the non-negativity of (cid:96) k to reach (cid:10) π k − π − k +1 , (cid:96) k (cid:11) ≤ (cid:88)
[Equation]
• In contrast, if
[Table]
k α
[Table]
k
[Table]
∞ ≤ 1 / 3 , then it follows from (62) that
[Table]
[Equation]
where we invoke the elementary facts that (cid:80) a
[Equation]
a
[Equation]
a
[Equation]
k
[Equation]
= 0 and (cid:80) a π k ( a )
[Equation]
k ( a )
[Equation]
k (cid:107)
[Equation]
Putting the above two cases together yields
[Equation] [Equation]
65) where the last inequality holds true since n (cid:88) k
=
[Equation]
Substituting (65) into (61), we can readily arrive at
[Equation]
It thus remains to establish the claims (61) and (62), which we shall accomplish next.
D.2 Proof of claim (61)
We claim that it sufﬁces to prove that
[Equation]
[Equation]
In fact, suppose that this inequality (66) is valid, then one can easily obtain α n 1 (cid:104) π −
[Equation]
[Equation]
with e a the a -th standard basis vector in R A , where the last line holds true since the negative entropy obeys F ( e a ) = 0 for any a ∈ A .
In turn, this implies that
[Equation]
where the last inequality invokes the elementary fact − F ( π ) ≤ log A for any π ∈ ∆( A ) .
Additionally, under the assumptions that η k +1 (1 − α k ) ≤ η k ( k ≥ 1 ), we can use the deﬁnition (5) to obtain
[Equation]
for any k ≥ 2 , which together with the basic fact 0 ≤ − F ( π ) ≤ log A yields −
[Equation]
Substitution into (67) leads to
[Equation]
as advertised.
As a consequence, everything boils down to establishing (66).
Towards this end
[Equation]
we would like to proceed with an induction argument, with the induction hypothesis w.r.t.
n given by (66).
Firstly, the base case with n = 1 simpliﬁes to
[Equation]
given that α 1 = α 11 ; this inequality clearly holds since, according to (33) and (59),
[Equation]
Secondly, suppose that (66) holds w
[Equation]
r
[Equation]
t. n , and we intend to justify it w
[Equation]
r
[Equation]
t. n + 1 .
To do so, we observe that
[Equation]
Here, (i) and (iii) invoke the fact α n +1 k
=
(1 − α n +1 )
α nk and α n +1 n +1 =
α n +1 (according to (5)), (ii) relies on the induction hypothesis (66) w.r.t.
n .
To ﬁnish up, invoke (70) and the deﬁnition (5) to arrive at
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
≤
[Equation]
n
[Equation]
=
[Equation]
where the inequality above makes use of (70), and the last identity comes from (33).
This justiﬁes the induction hypothesis w.r.t.
n + 1 .
Applying the induction argument in turn establishes (66) for all n , thereby concluding the proof.
D.3 Proof of claim (62)
We ﬁrst make the observation that (cid:88)
[Equation]
where the second equality follows from (34).
This in turn allows us to demonstrate that π − k
[Equation] [Equation]
=
[Equation]
where the last inequality holds since exp( − x ) ≥ 1 − x and (cid:80) a π k ( a ) exp (cid:0) − (cid:98)
η k α k (cid:96) k
( a ) (cid:1) ≤ (cid:80) a π k ( a )
= 1 .
Next, suppose that (cid:98) η k α k (cid:107) (cid:96)
k
(cid:107) ∞ ≤ 1 / 3 .
In this case, it is self-evident that (cid:98) η k α k | (cid:96) k
( a ) − E π k [ (cid:96) k ] | ≤ 2 / 3 for all a ∈ A .
Recalling that E π k
[ (cid:96) k ] =
(cid:80) a π k ( a ) (cid:96) k
( a ) , one can derive π
[Equation]
here, the ﬁrst inequality arises since exp( − x ) ≥ 1 − x , while the second inequality can be shown via the elementary inequality exp( − x ) ≤ 1 − x + x 2 for any x ≥ − 1 .
5 and therefore (cid:88)
[Equation] [Equation]
Applying the elementary inequality 1 − a 1+ b ≥ (1 − a )(1 − b ) = 1 − a − b + ab ≥ 1 − a − 2 b for any a ∈ [ − 1 , 1] and b > 0 , we can continue to lower bound (71) as follows
[Equation]
thereby completing the proof.
E Proofs of auxiliary lemmas and details E.1 Proof of Lemma
This section aims to prove Lemma 3, which establishes the inequality (cid:98) V i,h ≥ V (cid:63), (cid:98) π − i i,h .
In what follows, we shall proceed with an induction argument.
The base case with step H + 1 is trivially true, given that
[Equation]
holds for any joint policy.
Next, let us assume that the claim (46) is valid for step h + 1 , namely,
[Equation]
and attempt to justify the validity of this result when h + 1 is replaced with h .
This step is mainly accomplished by applying our reﬁned theory (cf.
Theorem 3) for FTRL (see (19)).
More precisely, we claim that
[Equation]
for any s ∈ S , whose proof is deferred to Appendix E.1.1.
Recall the construction (20a) of (cid:98) V i,h .
If (cid:98) V i,h
= H − h + 1 , then the claimed result (cid:98) V
i,h ≥ V (cid:63), (cid:98) π − i i,h holds trivially.
It thus sufﬁces to focus on the case where
[Equation]
In this case, recalling the deﬁnition of V (cid:63), (cid:98) π − i i,h ( s ) in (45b) gives
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
≤
[Equation]
simultaneously for all ( s, h ) ∈ S × [ H ] .
Here, the second line follows from the induction hypothesis (72) and the deﬁnition (17) of Q Ki,h , the third line invokes the claim (73), whereas the last line comes from our choice (24) of β
i,h (provided c b is large enough) and (74).
This concludes the proof, as long as (73) can be justiﬁed.
E.1.1 Proof of claim (73)
Consider any state s ∈ S .
By virtue of the identity Q ki,h
= (cid:80) kj =
α kj q ji,h (see (17)), the policy update rule (18) (or (19)) for π ki,h ( s ) can essentially be viewed as the FTRL algorithm applied to the loss vectors
[Equation]
Moreover, recalling the deﬁnition (23) of η k +1 and the deﬁnition (22) of α k (with c α ≥ 24 ), we have (cid:18) η k
[Equation]
This property (75) permits us to invoke Theorem 3 to obtain
[Equation]
where ξ i
[Equation]
h is deﬁned as
[Equation]
Here, to see why (i) holds, we make use of the facts that
[Equation]
where the ﬁrst line makes use of (38) for large enough c k , and the second line relies on (39a) in
Lemma
[Equation]
To proceed
[Equation]
let us control the terms in (76) separately.
• We start with the ﬁrst term in (76).
The elementary bound (cid:13) (cid:13) q ki,h (cid:13)(cid:13) ∞
≤ H in Lemma 2 taken together with (39b) in Lemma 1 helps us derive
[Equation]
[Equation]
Turning to the third term in (76), we recall the deﬁnition of
η K
+1 (cf.
23)) to obtain
[Equation] [Equation]
where the inequality comes from Lemma 1.
Finally, we move on to the last term in
For any
k ≥
, combine
Lemma 2 with (78) to obtain
[Equation]
Clearly, the right-hand side of (81) is upper bounded by 1 / 3 for all k obeying k ≥ c 9 H log 4 Kδ for some large enough constant c 9 > 0 (see also (38)).
Consequently, one can derive ξ i,
[Equation]
where the second line comes from (78) and the fact that K/ 2 > c 9 H log 4 Kδ (as a consequence of (38))
[Equation]
and the third line holds due to Lemma 2.
Putting the preceding bounds together and substituting them into (76)
[Equation]
we arrive at max a
[Equation]
where the last line is valid under Condition (38).
This completes the proof of Claim (73).
E.2 Proof of Lemma
In this section, we present the proof of Lemma 4.
To begin with, we introduce the auxiliary quantities
[Equation]
It is also helpful to introduce an auxiliary random action a k,s ∈
A i generated in a way that
[Equation]
which is independent from (cid:101) q ki,h conditional on π ki,h .
This allows us to deﬁne another set of random variables
[Equation]
which plays a central role in our analysis.
It is readily seen from the facts V i,h +1 ( s ) ≤ H − h
(cf. (20a)) and r ki,h ( s, a i ) ∈
[0 , 1] that
[Equation]
Letting e ( i
) ∈
R A i denote the i -th standard basis vector
[Equation]
we learn from the law of total variance that
[Equation]
With these preparations in place
[Equation]
we are ready to embark on the proof.
E.2.1 Proof of inequalities (49a) and (49b)
Recall the deﬁnition of V (cid:98) πi,h ( s ) in (45c) that
[Equation]
It is ﬁrst observed that
[Equation]
where the second identity arises from the deﬁnitions (48) of r (cid:98) πi,h and P (cid:98)
πi
[Equation]
h .
It is also seen that
[Equation]
where the ﬁrst line invokes Lemma 1, (85) and the fact (cid:107) π ki,h ( s ) (cid:107) 1 = 1 .
Another observation is that
W 1 =
[Equation] [Equation]
where the second line makes use of Lemma 1 and the inequality (86).
With the deﬁnitions (87) and (88) in mind, invoking Freedman’s inequality (i.e., Theorem 5) with κ 1 = (cid:113)
K
log
Kδ H then leads to
[Equation]
δ
[Equation]
with probability at least 1 − δ
[Equation]
where the last relation holds true under Condition (38)
[Equation]
To continue, we note the ﬁrst term in (90) can be bounded by Cauchy-Schwarz as follows:
[Equation]
Further
[Equation]
we make note of two additional facts:
weighted mean
[Equation]
• Regarding the square of (cid:98) q ki,h ( s ) , one has (see
[Table]
[Equation]
where we have used the fact that (cid:107) V (cid:98) πi,h +1 (cid:107) ∞
≤ H and (cid:107) r ki,h (cid:107) ∞ ≤ 1 ; consequently,
K (cid:88)
[Equation]
Taking (92) and (93) together with (91) yields
K (cid:88)
k
[Equation]
To ﬁnish up
[Equation]
substituting these into (90) and making use of the assumption (38) give
[Equation]
[Table]
K for any s ∈ S , thus concluding the proof of the ﬁrst claim (49a) of Lemma 4.
The second claim (49b) of Lemma 4 can be established using exactly the same argument, and hence we omit the proof here for the sake of brevity.
E.2.2 Proof of inequality (49c)
We then turn to the last advertised inequality (49c).
Given that r i,h ( s ) + P i,h ( s, · ) (cid:98) V i,h +1 ∈ [0 , H − h + 1] for all s ∈ S , we can recall the deﬁnition (20a) of (cid:98) V i,h to obtain (cid:12)(cid:12)(cid:12)(cid:98)
[Table]
i,h ( s ) −
[Table]
i
[Table]
h
[Table]
+ P i
[Table]
[Equation]
for all s ∈ S .
The remaining analysis is dedicated to bounding the right-hand side of (94).
Let us begin with the following identity:
[Equation]
where we recall the deﬁnitions of r i,h ∈ R S and P
i,h ∈ R S × S in (48).
The key step boils down to bounding the bonus term deﬁned in (24), towards which we ﬁrst claim that
[Equation]
holds for all s ∈ S .
Assuming the validity of this claim, we can then demonstrate that β
i,h ( s
[Equation]
=
[Equation]
b
[Equation]
where we have used the identity (cid:80)
Kk =
α Kk = 1 .
Hence, we can readily establish the desired result (49c) by combining (97) with (95) and (94)
[Equation]
provided that c 3 > 0 is sufﬁciently large.
It remains to justify the claim (96).
Towards this end
[Equation]
we make the observation that
Var
[Equation]
, which results from (cid:107) r ki,h (cid:107) ∞ ≤ 1 and the following relation:
[Equation]
This taken together with the fact (cid:80) Kk
=
α Kk = 1 and Jensen’s inequality yields
K (cid:88)
[Equation]
as claimed.
E.3 Minimax lower bound
In this section
[Equation]
we formalize the minimax lower bound claimed in (31).
Theorem 4 (Minimax lower bound) .
Consider any m ≥ 2 and any 0 < ε ≤ c 1 H for some small enough constant c 1 > 0 .
Then one can construct a collection of m -player zero-sum Markov games {MG θ | θ ∈ Θ } with S states, horizon H , and A i actions for the i -th player ( 1 ≤ i ≤ m ) such that
[Equation]
provided that the total sample size obeys
[Equation]
for some sufﬁciently small constant c 2 > 0 .
Here, the inﬁmum is over all (joint) policy estimator (cid:98) π , and P MG θ denotes the probability when the Markov game is MG θ .
Proof.
Suppose without loss of generality that A 1 ≥ max { A 2 , . . .
A m } .
Let us begin by considering the special scenario with A 2 = . . .
=
A m = 1 ; in this case, computing either the NE or the
CCE reduces to ﬁnding the optimal policy of a single-agent MDP with S states and A 1 actions.
It is well-known that for any given accuracy level ε ∈ (0 , H ] , there exists a non-stationary MDP with S states and A 1 actions such that no algorithm can learn an ε -optimal policy with o (cid:0) H 4 SA 1 ε(cid:1)
samples [3, 41].
More precisely, for any given 0 < ε ≤ c 1 H for some small enough constant c 1 > 0 , one can construct a collection of MDPs {M θ | θ ∈ Θ } such that
[Equation]
with the proviso that the total sample size obeys
[Equation]
for some small enough constant c 2 > 0 .
Here, the inﬁmum is over all policy estimate (cid:98) µ in this single-agent scenario, and P M θ denotes the probability when the MDP is M θ .
Next, let us construct a collection of Markov games by augmenting each of the single-agent MDPs
M θ with A i completely identical actions for the i -th player ( 2 ≤ i ≤ m ); that is, to construct MG θ , we take its reward function and probability transition kernel to be r MG θ i,h ( s, a ) =
[Equation]
if
[Equation]
if
[Equation]
else and
[Equation]
for all ( s, h, a = [ a 1 , . . .
, a m ]) ∈ S ×
[ H ] ×A .
Evidently, ﬁnding either an NE or a CCE of MG θ is equivalent to computing the optimal policy of M θ , given the non-distinguishability of the actions of all but the ﬁrst player in MG θ .
This in turn immediately establishes the advertised lower bound.
E.4 Freedman’s inequality
In this section, we record the Freedman inequality for martingales [27] with slight modiﬁcation, which is a crucial concentration bound for our analysis.
Theorem 5.
Suppose that Y n = (cid:80) nk =1 X k ∈ R , where { X k } is a real-valued scalar sequence obeying |
X k | ≤ R and
E
[Equation]
for all k ≥for some quantity R > 0 .
Deﬁne
[Equation]
where E k − 1 stands for the expectation conditional on { X j } j : j<k .
Consider any arbitrary quantity κ > 0 .
With probability at least 1 − δ , one has
[Equation]
Proof.
Suppose that W n ≤ σ 2 holds deterministically for some quantity σ 2 .
As has been demonstrated in Li et al.
[39, Theorem 5], with probability at least 1 − δ we have
[Equation]
for any positive integer K ≥ 1 .
Recognizing the trivial bound W n ≤ nR 2 , one can take σ 2 =
nRand K = log 2 n to obtain | Y n | ≤
[Equation]
where we have used 4 log 2 n ≤ 3 n for any integer n ≥ 1 .
This establishes the ﬁrst inequality in (103).
The second inequality in (103) is then a direct consequence of the elementary inequality 2 ab ≤ a 2 + b 2 .