R EPRESENTATION L EARNING FOR G ENERAL - SUM L OW - RANK M ARKOV G AMES
Anonymous authors
Paper under double-blind review
A BSTRACT
We study multi-agent general-sum Markov games with nonlinear function approximation.
We focus on low-rank Markov games whose transition matrix admits a hidden low-rank structure on top of an unknown non-linear representation.
The goal is to design an algorithm that (1) finds an ε -equilibrium policy sample efficiently without prior knowledge of the environment or the representation, and (2) permits a deep-learning friendly implementation.
We leverage representation learning and present a model-based and a model-free approach to construct an effective representation from collected data.
For both approaches, the algorithm achieves a sample complexity of poly ( H, d, A, 1 /ε ) , where H is the game horizon, d is the dimension of the feature vector, A is the size of the joint action space and ε is the optimality gap.
When the number of players is large, the above sample complexity can scale exponentially with the number of players in the worst case.
To address this challenge, we consider Markov Games with a factorized transition structure and present an algorithm that escapes such exponential scaling.
To our best knowledge, this is the first sample-efficient algorithm for multi-agent general-sum Markov games that incorporates (non-linear) function approximation.
We accompany our theoretical result with a neural network-based implementation of our algorithm and evaluate it against the widely used deep RL baseline, DQN with fictitious play.
I NTRODUCTION
Multi-agent reinforcement learning (MARL) studies the problem where multiple agents learn to make sequential decisions in an unknown environment to maximize their (own) cumulative rewards.
Recently, MARL has achieved remarkable empirical success, such as in traditional games like GO (Silver et al., 2016, 2017) and Poker (Moravˇcík et al., 2017), real-time video games such as Starcraft and Dota 2(Vinyals et al., 2019; Berner et al., 2019), decentralized controls or multi-agent robotics systems (Brambilla et al., 2013) and autonomous driving (Shalev-Shwartz et al., 2016).
On the theoretical front, however, provably sample-efficient algorithms for Markov games have been largely restricted to either two-player zero-sum games (Bai et al., 2020; Xie et al., 2020;
Chen et al., 2021; Jin et al., 2021c) or general-sum games with small and finite state and action spaces (Bai and
Jin, 2020; Liu et al., 2021; Jin et al., 2021b).
These algorithms typically do not permit a scalable implementation applicable to real-world games, due to either (1) they only work for tabular or linear
Markov games which are too restrictive to model real-world games, or (2) the ones that do handle rich non-linear function approximation (Jin et al., 2021c) are not computationally efficient.
This motivates us to ask the following question:
Can we design an efficient algorithm that (1) provably learns multi-player general-sum Markov games with rich nonlinear function approximation and (2) permits scalable implementations?
This paper presents the first positive answer to the above question.
In particular, we make the following contributions:
We design a new centralized self-play meta algorithm for multi-agent low-rank Markov games: Ge neral R epresentation L earning for M ulti-player G eneral-sum M arkov G ame ( G E RL_MG2 ).
We present a model-based and a model-free instantiation of G E RL_MG2 which differ by the way function approximation is used, and a clean and unified analysis for both approaches.
We show that the model-based variant requires access to an MLE oracle and a NE/CE/CCE oracle for matrix games, and enjoys a ˜ O (cid:0) H 6 d 4 A 2 log( | Φ || Ψ | ) /ε 2 (cid:1) sample complexity to learn an ε -NE/CE/CCE equilibrium policy, where d is the dimension of the feature vector, A is the size of the joint action space, H is the game horizon, Φ and Ψ are the function classes for the representation and emission process.
The model-free variant replaces model-learning with solving a minimax optimization problem, and enjoys a sample complexity of ˜ O (cid:
) H 6 d 4 A 3 M log( | Φ | ) /ε 2 (cid:1) for a slightly restricted class of Markov game with latent block structure.
Both of the above algorithms have sample complexities scaling with the joint action space size, which is exponential in the number of players.
This unfavorable scaling is referred to as the curse of multi-agent , and is unavoidable in the worst case under general function approximation.
We consider a spatial factorization structure where the transition of each player’s local state is directly affected only by at most L = O (1) players in its adjacency.
Given this additional structure, we provide an algorithm that achieves ˜ O
( M 4 H 6 d 2( L +1) 2 ˜
A 2( L +1) /ε 2 ) sample complexity, where ˜ A is the size of a single player’s action space, thus escaping the exponential scaling to the number of agents.
Finally, we provide an efficient implementation of our reward-free algorithm, and show that it achieves superior performance against traditional deep RL baselines without principled representation learning.
1. R ELATED W ORKS
Markov games Markov games (Littman, 1994; Shapley, 1953) is an extensively used framework introduced for game playing with sequential decision making.
Previous works (Littman, 1994; Hu and Wellman, 2003; Hansen et al., 2013) studied how to find the Nash equilibrium of a Markov game when the transition matrix and reward function are known.
When the dynamic of the Markov game is unknown, recent works provide a line of finite-sample guarantees for learning Nash equilibrium in two-player zero-sum Markov games (Bai and Jin, 2020; Xie et al., 2020; Bai et al., 2020;
Zhang et al., 2020; Liu et al., 2021; Jin et al., 2021c; Huang et al., 2021) and learning various equilibriums (including NE,CE,CCE, which are standard solution notions in games (Roughgarden, 2010)) in general-sum Markov games (Liu et al., 2021; Bai et al., 2021; Jin et al., 2021b).
Some of the analyses in these works are based on the techniques for learning single-agent Markov Decision Processes (MDPs) (Azar et al., 2017; Jin et al., 2018, 2020).
RL with Function Approximation Function approximation in reinforcement learning has been extensively studied in recent years.
For the single-agent Markov decision process, function approximation is adopted to achieve a better sample complexity that depends on the complexity of function approximators rather than the size of the state-action space.
For example, (Yang and Wang, 2019; Jin et al., 2020; Zanette et al., 2020) considered the linear MDP model, where the transition probability function and reward function are linear in some feature mapping over state-action pairs.
Another line of works (see, e.g., Jiang et al., 2017; Jin et al., 2021a; Du et al., 2021; Foster et al., 2021) studied the
MDPs with general nonlinear function approximations.
When it comes to Markov game, (Chen et al., 2021; Xie et al., 2020; Jia et al., 2019) studied the
Markov games with linear function approximations.
Recently, (Huang et al., 2021) and (Jin et al., 2021c) proposed the first algorithms for two-player zero-sum Markov games with general function approximation, and provided a sample complexity governed by the minimax Eluder dimension.
However, technical difficulties prevent extending these results to multi-player general-sum Markov games with nonlinear function approximation.
The results for linear function approximation assume a known state-action feature, and are unable to solve the Markov games with a more general non-linear approximation where both the feature and function parameters are unknown.
For the general function class works, their approaches rely heavily on the two-player nature, and it’s not clear how to apply their methods to the general multi-player setting.
Representation Learning in RL Our work is closely related to representation learning in singleagent RL, where the study mainly focuses on the low-rank MDPs.
A low-rank MDP is strictly more general than a linear MDP which assumes the representation is known a priori.
Several related works studied low-rank MDPs with provable sample complexities.
(Agarwal et al., 2020b; Ren et al., 2021) and (Uehara et al., 2021) consider the model-based setting, where the algorithm learns the representation with the model class of the transition probability given.
(Modi et al., 2021) provided a representation learning algorithm under the model-free setting and proved its sample efficiency when the MDP satisfies the minimal reachability assumption.
(Zhang et al., 2022) proposed a model-free method for the more restricted MDP class called Block MDP, but does not rely on the reachability assumption, which is also studied in papers including (Du et al., 2019) and (Misra et al., 2020).
A concurrent work (Qiu et al., 2022) studies representation learning in RL with contrastive learning and extends their algorithm to the Markov game setting.
However, their method requires strong data assumption and does not provide any practical implementation in the Markov game setting.
P ROBLEM S ETTINGS
A general-sum Markov game with M players is defined by a tuple ( S , {A i } Mi =1 , P ⋆ , { r i } Mi =1 , H, d 1 ) .
Here S is the state space,
A i is the action space for player
i , H is the time horizon of each episode and d 1 is the initial state distribution.
We let A =
A 1 × . . .
×
A M and use a = ( a 1 , a 2 , . .
. , a M ) to denote the joint actions by all M players.
Denote ˜
A = max i |A
i | and A = |A| .
P ⋆ = { P ⋆h } Hh = is a collection of transition probabilities, so that P ⋆h ( ·| s, a ) gives the distribution of the next state if actions a are taken at state s and step h .
And r i = { r h,i } Hh =1 is a collection of reward functions, so that r h,i ( s, a ) gives the reward received by player i when actions a are taken at state s and step h .
2. S OLUTION C ONCEPTS
The policy of player i is denoted as π i :=
{ π h,i : S → ∆ A i } h ∈
[ H ] .
We denote the product policy of all the players as π := π 1 × . . .
× π M , here “product” means that conditioned on the same state, the action of each player is sampled independently according to their own policy.
We denote the policy of all the players except the i th player as π − i .
We define V πh,i ( s ) as the expected cumulative reward that will be received by the i th player if starting at state s at step h and all players follow policy π .
For any strategy π − i , there exists a best response of the i th player, which is a policy µ † ( π − i ) satisfying
V µ † ( π − i ) ,π − i h,i ( s ) =
max π
i V π
i ,π − i h,i ( s ) for any ( s, h ) ∈ S ×
[ H ] .
We denote V † ,π − i
h,
i :=
V µ † ( π − i ) ,π − i h,i .
Let v † ,π − i
i
[Equation]
Definition 2.1 (NE) .
A product policy π is a Nash equilibrium (NE) if v πi = v † ,π −
i i , ∀ i ∈
[ M ] .
And we call π an ε -approximate NE if max i ∈
[ M ] { v † ,π − i i − v πi } < ε .
The coarse correlated equilibrium (CCE) is a relaxed version of Nash equilibrium in which we consider general correlated policies instead of product policies.
Definition 2.2 (CCE) .
A correlated policy π is a CCE if V † ,π −
i h,i ( s ) ≤ V πh,i ( s ) for all s ∈ S , h ∈
[ H ] , i ∈
[ M ] .
And we call π an ε -approximate CCE if max i ∈
[ M ] { v † ,π − i i − v πi } < ε .
The correlated equilibrium (CE) is another relaxation of the Nash equilibrium.
To define CE, we first introduce the concept of strategy modification:
A strategy modification ω i :=
{ ω h,i } h ∈
[ H ] for player i is a set of H functions from S × A
i to A i .
Let Ω i :=
{ Ω h,i } h ∈
[ H ] denote the set of all possible strategy modifications for player i .
One can compose a strategy modification ω
i with any Markov policy π and obtain a new policy ω i ◦ π such that when policy π chooses to play a := ( a 1 , . .
. , a M ) at state s and step h ,
policy ω i ◦ π will play ( a 1 , . . .
, a i − 1 , ω h,i ( s, a i ) , a i +1 , . .
. , a M ) instead.
Definition 2.3 (CE) .
A correlated policy π is a CE if max i ∈
[ M ] max ω i ∈
Ω i V ω
i ◦ π h,i ( s ) ≤
V πh,i ( s ) for all ( s, h ) ∈ S ×
[ H ] .
And we call π an ε -approximate CE if max i ∈
[ M ] { max ω
i ∈
Ω i v ω
i ◦ π i − v πi } < ε .
Remark 2.1.
For general-sum Markov Games, we have { NE } ⊆ { CE } ⊆ { CCE } , so that they form a nested set of notions of equilibria (Roughgarden, 2010).
While there exist algorithms to approximately compute the Nash equilibrium (Berg and Sandholm, 2017), the computation of NE for general-sum games in the worst case is still PPAD-hard (Daskalakis, 2013).
On the other hand, CCE and CE can be solved in polynomial time using linear programming (Examples include Papadimitriou and Roughgarden (2008); Blum et al.
Therefore, in this paper we study both NE and these weaker equilibrium concepts that permit more computationally efficient solutions.
2. L OW -R ANK M ARKOV G AMES
In this paper, we consider the class of low-rank Markov games.
A Markov game is called a low-rank
Markov game if the transition probability at any time step h has a latent low-rank structure.
Definition 2.4 (Low-Rank Markov Game) .
We call a Markov game a low-rank Markov game if for any s, s ′ ∈ S , a ∈ A ,
h ∈
[ H ] , i ∈
[ M ] , we have P ⋆h ( s ′ | s, a ) =
ϕ ⋆h ( s, a ) ⊤ w ⋆h ( s ′ ) , where ∥ ϕ ⋆h ( s, a ) ∥ 2 ≤ 1 and ∥ w ⋆h ( s ′ )
∥ 2 ≤ √ d for all ( s, a , s ′ ) .
A special case of low-rank Markov Game is the Block Markov game:
Definition 2.5 (Block Markov Game) .
Consider any h ∈
[ H ] .
A Block Markov game has an emission distribution o h ( ·| z ) ∈ ∆ S and a latent state space transition T h ( z ′ | z, a ) , such that for any s ∈
S , o h ( s | z ) > 0 for a unique latent state z ∈ Z , denoted as ψ ⋆h ( s ) .
Denote Z = |Z| .
Together with the ground truth decoder ψ ⋆h , it defines the transitions P ⋆h ( s ′ | s, a ) = (cid:80)
z ′
∈Z o h ( s ′ | z ′ )
T h ( z ′ | ψ ⋆h ( s ) , a ) .
With the definition of the Block Markov game, one can naturally derive a feature vector that in addition takes the one-hot form: we just need to let the ground truth ϕ
⋆h ( s, a ) at step h be a Z · A -dimensional vector e ( ψ ⋆ ( s ) , a ) where e i is the i -th basis vector.
Correspondingly, for any s ∈ S , w ⋆h ( s ) is a Z · A dimensional vector such that the ( z, a ) -th entry is (cid:80) z ′ ∈Z o [Equation]
s [Equation] z [Equation]
T [Equation] ( z [Equation]
| z [Equation] a ) . Then P ⋆h ( s ′ | s, a ) =
ϕ ⋆h ( s, a ) ⊤ w ⋆h ( s ′ ) , so that the Block Markov game is a low-rank Markov game with rank d
= Z · A .
Learning Objective
The goal of multi-agent reinforcement learning is to design algorithms for
Markov games that find an ε -approximate equilibrium (NE, CCE, CE) from a small number of interactions with the environment.
We focus on the low-rank Markov games whose feature vector ϕ ⋆ and transition probability P ⋆ are both unknown , and the goal is to identify a ε -approximate equilibrium policy with a number of interactions scaling polynomially with d, A, H, 1 ε and the log-cardinality of the function class, without depending on the number of raw states which could be infinite.
A LGORITHM D ESCRIPTION
In this section, we present our algorithm G E RL_MG2 (see Alg. 1).
The algorithm mainly consists of two modules: the representation learning module and the planning module.
We develop the representation learning module base on the past works on the single agent MDP (e.g. Agarwal et al.
(2020b)
; Uehara et al.
(2021); Modi et al.
(2021)), and but modify them to work with UCB-style planning module. Here we denote d πP,h as the state distribution under transition probability P and policy π at step h , and U ( A ) as the uniform distribution over the joint action space. 3. R EPRESENTATION L EARNING
In the representation learning module, the main goal is to learn a representation function ˆ ϕ to approximate ϕ
⋆ , using the data collected so far.
In each episode, the algorithm first collects some new data using the policy derived from the previous episode.
Note that in our data collection scheme, for each time step h , we maintain two buffers D ( n ) h and ˜ D ( n ) h of transition tuples ( s, a, s ′ )
(line 7 of
Alg. 1) which draw the state s from slightly different distributions.
Based on the data collected in history, the representation learning module estimates the feature ˆ ϕ ( n ) and transition probability ˆ P ( n ) .
We propose two versions of the representation learning algorithm (model-based, Alg. 2; model-free,
Alg. 3) based on whether we are given the full model class M h of the transition probability, or only the function class of the state-action features Φ h .
Model-based Representation Learning
In the model-based setting, we assume the access to a realizable model class M h
= { ( w h , ϕ h ) :
w h ∈ Ψ h , ϕ h ∈
Φ h } ,
h ∈
[ H ] such that the true model is included in this class,
i.e., w ⋆h ∈
Ψ h , ϕ ⋆h ∈
Φ h , ∀ h ∈
[ H ] .
Following the norm bounds on ϕ ⋆h , w ⋆h , we assume that the same norm bounds hold for our function approximator, i.e., for any ϕ h ∈
Φ h , w h ∈ Ψ h , we have ∥ ϕ h ( s, a ) ∥ 2 ≤ 1 and ∥ w h ( s ′ )
∥ 2 ≤ √ d for all ( s, a , s ′ ) , and (cid:82) ϕ h ( s, a ) ⊤ [Equation] h ( s ′ )
d s ′ = 1 .
Given the dataset D := D ( n ) h ∪ ˜ D ( n ) h , MBR EP L EARN learns the features and transition probability using maximum likelihood estimation (MLE): (cid:16)
[Equation]
[Equation] [Equation]
s
[Equation]
P
[Equation]
˜ s
[Equation]
a
[Equation]
= D (
[Equation]
) h
[Equation]
a s ′ )
[Equation] [Equation]
n
[Equation]
Learn representation via model based model methods:
[Equation]
Compute from equation 5, for each s,
M ] , set
Q ( n )
[Equation]
,i
[Equation]
s, a ) ← r
[Equation]
i
[Equation]
+
[Equation]
a
[Equation]
Compute π ( n ) h from equation 2 or equation 3 or equation 4.
For each s ∈ S , i ∈
[ M ] , set
V (
[Equation]
11: end for 12:
Let ∆ ( n ) = max i ∈
[ M ] (cid:110) v ( n ) i − v ( n ) i (cid:111)
+ 2 H (cid:112) Aζ ( n ) , where v ( n ) i = (cid:82) S V ( n ) 1 ,i ( s )
d 1 ( s )d s , and v ( n ) i
= (cid:82) S V ( n ) 1 ,i ( s )
d 1 ( s )d s .
13: end for 14: Return ˆ π = π ( n ⋆ ) where n ⋆ = arg min n ∈ [ N ] ∆ ( n ) .
Model-free Representation Learning
In the model-free setting, we are only given the function class of the feature vectors, Φ h , which we assume also includes the true feature ϕ ⋆h .
Given the dataset
D := D ( n ) h ∪ ˜ D ( n ) h , MFR EP L EARN aims to learn a feature vector that is able to linearly fit the
Bellman backup of any function f ( s ) in an appropriately chosen discriminator function class F h .
To be precise, we aim to optimize the following objective:
[Equation]
where the first term is the empirical squared loss and the second term is the conditional expectation of f ( s ′ ) given ( s, a ) , subtracted for the purpose of bias reduction.
Once we obtain an estimation ˆ ϕ ( n ) h , we can construct a non-parametric transition model defined as:
[Equation]
We show that doing Least-square Value Iteration (LSVI) is equivalent to doing model-based planning inside ˆ P ( n ) h (line 10 of Alg. 1), and thus the model-free algorithm can be analyzed in the same way as the model-based algorithm.
In practice, for applications where the raw observation states are high-dimensional, e.g. images, estimating the transition is often much harder than estimating the one-directional feature function.
In such cases, we expect the Ψ class to be much larger than the Φ class and the model-free approach to be more efficient.
3. P LANNING
Based on the feature vector and transition probability computed from the representation learning phase, a new policy π ( n +1) is computed using the planning module.
The planning phase is conducted with a Upper-Confidence-Bound (UCB) style approach, and we maintain both an optimistic and a pessimistic estimation of the value functions and the Q-value functions V ( n ) h,i , V ( n ) h,i , Q ( n ) h,i , Q ( n ) h,i , which are computed recursively through the Bellman’s equation with the bonus function ˆ β ( n ) h
(Line 9 andof Alg. 1).
Here the operator D is defined by ( D π f )( s ) :
= E a ∼ π ( s ) [ f ( s, a )] , ∀ f : S ×
A → R , and π ( n ) h is the policy computed from M induced Q-value functions ˜ Q ( n ) h,i .
For the model-based setting, we simply let ˜ Q ( n ) h,i be the optimistic estimator Q ( n ) h,i .
For the model-free setting, for technical reasons, we instead let ˜ Q ( n ) h,i be the nearest neighbor of Q ( n ) h,i in N h with respect to the ∥ · ∥ ∞ metric, where
N h ⊆ R S×A is a properly designed set of functions, whose construction is deferred to the appendix.
Depending on the problem settings, the policy π ( n ) h takes either one of the following formulations:
• For the NE, we compute π ( n ) h =
[Equation]
[Equation]
such that ∀ s ∈ S , i ∈
[Equation]
• For the CCE, we compute π ( n ) h such that ∀ s ∈ S , i ∈
[ M ] ,
[Equation]
π
[Equation]
• For CE, we compute π ( n ) h such that ∀ s ∈ S , i ∈
[ M ] ,
[Equation]
Without loss of generality we assume the solution to the above formulations is unique, if there are multiple solutions, one can always adopt a deterministic selection rule such that it always outputs the same policy given the same inputs.
Note that although the policy is computed using only the optimistic estimations, we still maintain a pessimistic estimator, which is used to estimate the optimality gap ∆ ( n ) of the current policy.
The algorithm’s output policy ˆ π is chosen to be the one with the minimum estimated optimality gap.
The bonus term ˆ β ( n ) h is a linear bandit style bonus computed using the learned feature ˆ ϕ :
[Equation]
5) where ˆΣ ( n ) h := (cid:80) ( s, a ) ∈D ( n ) h
ˆ ϕ ( n ) h
( s, a )ˆ ϕ ( n ) h ( s, a ) ⊤ + λI d is the empirical covariance matrix.
T HEORETICAL R ESULTS
In this section, we provide the theoretical guarantees of the proposed algorithm for both the modelbased and model-free approaches.
We denote |M| :
=
max h ∈ [ H ] |M h | and | Φ | := max h ∈
[ H ] | Φ h | .
The first theorem provides a guarantee of the sample complexity for the model-based method.
Theorem 4.1 (PAC guarantee of Algorithm 1 (model-based)) .
When Alg. 1 is applied with modelbased representation learning algorithm Alg. 2, with parameters λ
= Θ ( d log( NH | Φ | / δ )) , α ( n ) =
Θ (cid:16) Hd (cid:112)
A log( |M| HN / δ ) (cid:17) , ζ ( n ) =
Θ (cid:0) n − 1 log( |M| HN / δ ) (cid:1) , by setting the number of episodes N to be at most
O
[Equation]
, with probability 1 − δ , the output policy ˆ π is an ε -approximate { NE , CCE , CE } .
Theorem 4.1 shows that G E RL_MG2 can find an ε -approximate { NE , CCE , CE } by running the algorithm for at most ˜ O (cid:0)
H 6 d 4 A 2 ε − 2 (cid:1) episodes, which depends polynomially on the parameters
H, d, A, ε − 1 and only has a logarithmic dependency on the cardinality of the model class |M| .
In particular, when reducing the Markov game to the single-agent MDP setting, the sample complexity of the model-based approach matches the result provided in (Uehara et al., 2021), which is known to have the best sample complexity among all oracle efficient algorithms for low-rank MDPs.
For model-free representation learning, we have the following guarantee:
Theorem 4.2 (PAC guarantee of Algorithm 1 (model-free)) .
When Alg.
1 is applied with model-free representation learning algorithm Alg. 3, and λ =
Θ ( d log( NH | Φ | / δ )) , α ( n ) =
Θ (cid:16) HAd (cid:112) M log( dNHAM | Φ | / δ ) (cid:17) , ζ ( n ) =
Θ (cid:0) d 2 An − 1 log( dNHAM | Φ | / δ ) (cid:1) , and the Markov game is a Block Markov game.
When we set the number of episodes N to be at most
[Equation]
for an appropriately designed function class {N h } Hh =1 and discriminator class {F h } Hh =1 , with probability 1 − δ , the output policy ˆ π is an ε -approximate { NE , CCE , CE } .
For the model-free block Markov game setting, the number of episodes required to find an εapproximate { NE , CCE , CE } becomes ˜ O (cid:0) H 6 d 4 A 3 Mε − 2 (cid:1) .
While it has a worse dependency compared with the model-based approach, the advantage of the model-free approach is it doesn’t require the full model class of the transition probability but only the model class of the feature vector, which applies to a wider range of RL problems.
The proofs of Theorem 4.1 and Theorem 4.2 are deferred to Appendix B and C. Theorem 4.1 and
Theorem 4.2 show that G E RL_MG2 learns low-rank Markov games in a statistically efficient and oracle-efficient manner.
We also remark that our modular analysis can be of independent theoretical interest.
Unlike prior works that make heavy distinctions between model-based and model-free approaches, e.g. (Liu et al., 2021), we show that both approaches can be analyzed in a unified manner.
F ACTORED M ARKOV G AMES
The result in Theorem 4.1 is tractable in games with a moderate number of players.
However, in applications with a large number of players, such as the scenario of autonomous traffic control, the total number of players in the game can be so large that the joint action space size A
= ˜
A M dominates all other factors in the sample complexity bound.
This exponential scaling with the number of players is sometimes referred to as the curse of multi-player .
The only known class of algorithms that overcomes this challenge in Markov games is V-learning (see, e.g., Bai et al., 2020; Jin et al., 2021b), a value-based method that fits the V-function rather than the Q-function, thus removing the dependency on the action space size.
However, V-learning only works for tabular Markov games with finite state and action spaces.
Extending V-learning to the function approximation setting is extremely non-trivial, because even in the single agent setting, no known algorithm can achieve sample efficient learning in MDPs while only performing function approximation on the V-function.
In this section we take a different approach that relies on the following observation.
In a setting where the number of agents is large, there is often a spatial correlation among the agents, such that each agent’s local state is only immediately affected by the agent’s own action and the states of agents in its adjacency.
For example, in smart traffic control, a vehicle’s local environment is only immediately affected by the states of the vehicles around it.
On the other hand, it takes time for the course of actions of a vehicle from afar to propagate its influence on the vehicle of reference.
Such spatial structure motivates the definition of a factored Markov Game.
In a factored Markov Game, each agent i has its local state s i , whose transition is affected by agent i ’s action a
i and the state of the agents in its neighborhood Z i .
We remark that the factored Markov
Game structure still allows an agent to be affected by all other agents in the long run, as long as
the directed graph defined by the neighborhood sets
Z i is connected.
In particular, we have
Definition 5.1 (Low-Rank Factored Markov Game) .
We call a Markov game a low-rank factored
Markov game if for any s, s ′ ∈ S , a ∈ A ,
h ∈
[ H ] , i ∈
[ M ] , we have
[Equation]
where Z i ⊆
[ M ] , ϕ ⋆h,i ( s [ Z i ] , a i ) , w ⋆h,i ( s ′ i ) ∈ R d , ∥
ϕ ⋆h,i ( s [ Z i ] , a i ) ∥ 2 ≤ 1 and ∥ w ⋆h,i ( s ′ i ) ∥ 2 ≤ √ d for all ( s [ Z i ] , a i , s ′ i ) .
We assume | Z i | ≤ L, ∀ i ∈
[ M ] .
And we are given a group of model classes
M h,i , h ∈
[ H ] , i ∈
[ M ] such that ( ϕ ⋆h,i , w ⋆h,i ) ∈
M h,i .
We are now ready to present our algorithm and result in the low-rank factored Markov Game setting.
Surprisingly, the same algorithm G E RL_MG2 works in this setting, with the representation learning module Alg.
2 replaced by Alg. 4, and a few changes of variables.
For simplicity, we focus on the model-based version.
Define ¯ ϕ ( n ) h,i ( s, a )
=
(cid:78)
j ∈ Z i ˆ ϕ ( n ) h,
j ( s [ Z j ] , a j ) ∈ R d
| Zi | where ⊗ means the
Kronecker product.
Let ˆ
[Equation]
where ¯Σ ( n ) h,i := (cid:80) ( s, a ) ∈D ( n )
h ¯ ϕ h,i ( s, a )¯ ϕ h,i ( s,
a ) ⊤ + λI d | Zi | .Then, G E RL_MG2 with ¯ ϕ and the newly defined ˆ β ( n ) , ∆ ( n ) achieves the following guarantee:
Theorem 5.1 (PAC guarantee of G E RL_MG2 in Low-Rank Factored Markov Game) .
When
Alg. 1 is applied with model-based representation learning algorithm Alg. 4, with L = O (1) and parameters λ
= Θ (cid:0) Ld L
[Equation]
( NHM
[Equation]
(cid:1) ,
[Equation]
) =
Θ (cid:0) n − 1 log( |M| HNM / δ ) (cid:1) , by setting the number of episodes N to be at most
[Equation]
with probability 1 − δ , the output policy ˆ π is an ε -approximate { NE , CCE , CE } .
Remark 5.1.
This sample complexity only scales with exp( L ) where L is the degree of the connection graph, which is assumed to be O (1) in Definition 5.1 and in general much smaller than the total number of agents in practice.
We remark that the factored structure is also previously studied in single-agent tabular MDPs (examples include Chen et al.
; Kearns and Koller (1999); Guestrin et al.
; Strehl et al.
Chen et al.
(2020) provided a lower-bound showing that the exponential dependency on L is unimprovable in the worst case.
Therefore, our bound here is also nearly tight, upto polynomial factors.
E XPERIMENT
In this section we investigate our algorithm with proof-of-concept empirical studies.
We design our testing bed using rich observation Markov game with arbitrary latent transitions and rewards.
To
[Table]
solve the rich observation Markov game, an algorithm must correctly decode the latent structure (thus learning the dynamics) as well as solve the latent Markov game to find the NE/CE/CCE strategies concurrently.
Below, we first introduce the setup of the experiments and then make comparisons with prior baselines in the two-player zero-sum setting.
We then follow by showing the efficiency of
G E RL_MG2 in the general-sum setting.
All further experiment details can be found in Appendix.
F.
Here we focus on the model-free version of G E RL_MG2.
Specifically, we implement Algorithm.with deep learning libraries (Paszke et al., 2017).
We defer more details to Appendix.
F.2.
Block Markov game Block Markov game is a multi-agent extension of single agent Block MDP, as defined in Def. 2.5.
We design our Block Markov game by first randomly generating a tabular
Markov game with horizon H , 3 states, 2 players each with 3 actions, and random reward matrix
R h ∈ (0 , 1) 3 × 3 2 × H and random transition matrix T h
( s h , a h ) ∈
∆ S h +1 .
We provide more details (e.g., generation of rich observation) in Appendix F.1.
Zero-sum Markov game
In this section we first show the empirical evaluations under the two-player zero-sum Markov game setting.
For an environment with horizon H , the randomly generated matrix
R denotes the reward for player 1 and − R ⊤ denotes the reward for player 2, respectively.
For the zero-sum game setting, we designed two variants of Block Markov games: one with short horizon ( H = 3 ) and one with long horizon ( H = 10 ).
We show in the following that G E RL_MG2 works in both settings where the other baseline could only work in the short horizon setting.
Baseline We adopt one open-sourced implementation of DQN (Silver et al., 2016) with fictitious self-play (Heinrich et al., 2015).
We keep track of the exploitability of the returned strategy to evaluate the practical performances of the baselines.
In the zero-sum setting, we only need to fix one agent (e.g., agent 2), train the other single agent (the exploiter) to maximize its corresponding return until convergence, and report the difference between the returns of the exploiter and the final return of the final policies.
We include the exploitability in Table.
We provide training curves in Appendix.
F.3 for completeness.
We note that compared with the Deep RL baseline, G E RL_MG2 shows a faster and more stable convergence in both environments, where the baseline is unstable during training and has a much larger exploitability.
General-sum Markov game.
In this section we move on to the general-sum setting.
To our best knowledge, our algorithm is the only principled algorithm that can be implemented on scale under the general-sum setting.
For the general sum setting, we can not just compare our returned value to the oracle NE values, because multiple NE/CCE values may exist.
Instead, we keep track of the exploitability of the policy and plot the training curve on the exploitability in Fig.
2 (deferred to
Appendix. F).
Note that in this case we need to test both policies since their reward matrices are independently sampled.
D ISCUSSION AND F UTURE W ORKS
In this paper, we present the first algorithm that solves general-sum Markov games under function approximation.
We provide both a model-based and a model-free variant of the algorithm and present a unified analysis.
Empirically, we show that our algorithm outperforms existing deep RL baselines in a general benchmark with rich observation.
Future work includes evaluating more challenging benchmarks and extending beyond the low-rank Markov game structure.
Reproducibility Statement.
For theory, we provide proof and additional results in the Appendix.
For empirical results, we provide implementation and environment details and hyperparameters in the Appendix.
We also submit anonymous code in the supplemental materials.
A A DDITIONAL N OTATIONS
Given a (possibly not normalized) transition probability P : S ×
A × S × [ H ] → [0 , 1] and a policy π : S ×
[ H ] → ∆ A , we define the the density function of the state-action pair ( s, a ) at step h under transition P and π by
[Equation]
We abuse the notations a bit and denote d πP,h ( s ) as the marginalized state distribution, i.e., d πP,h ( s ) =
(cid:80) a ∈A d πP,h ( s, a ) .
For any n ∈ [ N ] , h ∈ [ H ] , define
[Equation]
When we use the expectation E ( s, a ) ∼ ρ
[ f ( s, a )] (or E s ∼ ρ
[ f ( s )] ) for some (possibly not normalized) distribution ρ and function f , we simply mean (cid:80) s ∈S , a ∈A
[Equation]
( s,
[Equation]
f
[Equation]
s, a )
[Equation]
or (cid:80)
[Equation]
s )
[Equation]
so that the expectation can be naturally extended to the unnormalized distributions.
For an iteration n , a distribution ρ and a feature ϕ , we denote the expected feature covariance as
[Equation]
Meanwhile, define the empirical covariance by
[Equation]
B A NALYSIS OF THE M ODEL -B ASED M ETHOD B. H IGH P ROBABILITY E VENTS
We define the following event
[Equation]
h
[Equation]
E :=
E 1 ∩ E
[Equation]
To prove E holds with a high probability
[Equation]
we first introduce the following MLE guarantee
[Equation]
whose original version can be found in (Agarwal et al., 2020b):
Lemma B.1 (MLE guarantee) .
For a fixed episode n and any step h , with probability 1 − δ ,
[Equation]
ˆ P ( n ) h
[Equation]
·| s, a ) − P ⋆h ( ·| s, a ) (cid:13)(cid:13)(cid:13) 2(cid:21) ≲n
[Equation]
|M| δ .
As a straightforward corollary, with probability 1 − δ , ∀ n ∈ N + , ∀ h ∈ [ H ] ,
E
[Equation]
a
[Equation] [Equation]
δ .
Proof.
See Agarwal et al.(Agarwal et al., 2020b)
(Theorem 21).
Based on Lemma B.1 and Lemma E.1 in Appendix E, we directly get the following guarantee:
Lemma B.2.
When ˆ P ( n ) h is computed using Alg. 2, if we set
[Equation]
then E holds with probability at least 1 − δ .
B. S TATISTICAL G UARANTEES
Lemma B.3 (One-step back inequality for the learned model) .
Suppose the event E holds.
Consider a set of functions { g h } Hh =1 that satisfies g h ∈ S ×
A → R + , s.t.
∥
g h ∥ ∞ ≤ B .
For any given policy π , we have
[Equation]
Recall Σ n,ρ ( n
[Equation] [Equation]
(cid:104)
[Equation]
Proof.
For step h = 1 , we have
[Equation]
For step h = 2 , . . .
, H −
[Equation]
we observe the following one-step-back decomposition:
[Equation]
[Table]

[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
, B      .
where we use the fact that g h is bounded by B .
Then,
[Equation]
S
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(Definition of ˜ ρ ( n ) h )
Combing the above results together, we get
[Equation]
Σ n,ρ ( n ) h − 1 , ˆ ϕ ( n ) h −, B    
[Equation]
  min  
[Equation]
n
[Equation]
(cid:113)
[Equation]
   
[Equation]
which has finished the proof.
Lemma B.4 (One-step back inequality for the true model) .
Consider a set of functions { g h } Hh =1 that satisfies g h ∈ S ×
A → R + , s.t.
∥
g h ∥ ∞ ≤ B .
Then for any given policy π , we have
[Equation]
Recall
[Equation]
Proof.
For step h = 1 , we have
[Equation]
For step h = 2 , . . .
, H − 1 , we observe the following one-step-back decomposition:
[Equation]
[Equation]
Then, (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:90)
[Equation]
[Table]
[Equation]
(Definition of ρ
[Equation]
Combing the above results together, we get
[Equation]
which has finished the proof
[Equation]
Lemma B.5 (Optimism for NE and CCE) .
Consider an episode n ∈ [ N ] and set α ( n ) =
Θ (cid:16) H (cid:112) nAζ ( n ) + dλ (cid:17) .
When the event E holds and the policy π ( n ) is computed by solving NE or
CCE, we have
[Equation]
Aζ ( n ) , ∀ n ∈ [ N ] , i ∈
[ M ] .
Proof.
Define ˜ µ ( n ) h,i ( ·| s ) :
= arg max µ (cid:18) D µ,π ( n )
[Equation] [Equation]
i Q † ,π ( n ) − i h,i (cid:19) ( s ) as the best response policy for player
i at step h , and let ˜ π ( n ) h = ˜ µ ( n ) h,i × π ( n ) h, − i .
Let f ( n ) h ( s, a ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h ( ·| s, a ) − P ⋆h ( ·| s, a ) (cid:13)(cid:13)(cid:13) 1 , then according to the event E , we have
[Equation]
[Equation]
A direct conclusion of the event E is we can find an absolute constant c , such that
[Equation]
Next
[Equation]
we prove by induction that
[Equation]
First, notice that ∀ h ∈
[ H ] ,
[Equation]
where the inequality uses the fact that π ( n ) h is the NE (or CCE) solution for (cid:110) Q ( n ) h,i (cid:111)
M i =1 .
Now we are ready to prove equation 7:
• When h = H , we have
[Equation]
• Suppose the statement is true for step h
+ 1 , then for step h , we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where we use the fact
[Equation]
and the last row uses the induction assumption.
Therefore
[Equation]
we have proved equation 7.
We then apply h = 1 to equation
[Equation]
and get
[Equation]
∼
[Equation]
Next we are going to bound the second term, let g h ( s, a ) =
min { f ( n ) h ( s, a ) , 1 } and apply Lemma
B.3 to g h , we have for h = 1 ,
[Equation]
And ∀ h ≥ 2 , we have
E
[Equation]
≤
[Equation]
˜
[Equation]
∼ d
[Equation]
) ˆ P
[Equation]
  min   (cid:13)(cid:13)(cid:13)
[Table]

[Table]

[Table]
Note that we here use the fact min { f ( n ) h ( s, a ) , 1 } ≤ 1 , E ( s, a ) ∼ ρ ( n ) h (cid:20)(cid:16)
f ( n ) h ( s, a ) (cid:17)(cid:21) ≤ ζ ( n ) and
E ( s, a ) ∼ ˜ ρ ( n ) h (cid:20)(cid:16) f
( n ) h
( s, a ) (cid:17) 2 (cid:21) ≤ ζ ( n ) .
Then according to our choice of α ( n ) , we get
E ( s, a ) ∼ d ˜ π ( n ) ˆ P ( n ) ,h (cid:104) min (cid:110) f
( n ) h
( s, a ) , 1 (cid:111)(cid:105) ≤ E (˜ s, ˜ a )
∼ d ˜ π ( n ) ˆ P ( n
[Table]
,h −  min   cα ( n )
H (cid:13)(cid:13)(cid:13)
[Equation]
n
[Equation]
˜
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
n
[Equation]
ρ ( n ) h
[Equation]
ˆ ϕ ( n
[Equation]

[Equation]
 
[Equation]
Combining all things together,
[Equation] [Equation]
H
[Equation]
≥ − H (cid:112) Aζ ( n
[Equation]
, which proves the inequality.
Lemma B.6 (Optimism for CE) .
Consider an episode n ∈ [ N ] and set α ( n ) =
Θ
[Equation]
n )
+
[Equation]
When the event E holds, we have
[Equation]
∀ n ∈ [ N ]
[Equation]
i ∈ [ M ] .
Proof.
Denote ˜ ω ( n ) h,i =
[Equation]
Ω
[Equation]
i
[Equation]
n
[Equation]
h
[Equation]
i (cid:17) ( s ) and let ˜ π ( n ) h
= ˜ ω
h,i ◦ π ( n ) h .
Let f ( n ) h ( s, a ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h ( ·| s, a ) − P ⋆h ( ·| s, a ) (cid:13)(cid:13)(cid:13) 1 , then according to the event E , we have
E ( s, a ) ∼ ρ ( n ) h (cid:20)(cid:16) f
( n ) h
( s, a ) (cid:17) 2 (cid:21) ≤ ζ ( n
[Equation]
A direct conclusion of the event E is we can find an absolute constant c , such that
[Equation]
Next
[Equation]
we prove by induction that
[Equation]
1 (cid:111)(cid:105)
[Equation]
h ∈ [ H ] .
[Equation] [Equation]
First, notice that ∀ h ∈
[ H ] ,
[Equation]
where the inequality uses the fact that π ( n ) h is the CE solution for (cid:110) Q ( n ) h,i (cid:111)
M i =1 .
Now we are ready to prove equation 8:
• When h = H , we have
[Equation]
[Table]
the
[Table]
true for h
[Table]
, then for step h ,
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where we use the fact
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
and the last row uses the induction assumption
Therefore
[Equation]
we have proved equation 8.
We then apply h = 1 to equation
[Equation]
and get
E s ∼ d
[Equation]
Next we are going to bound the second term, let g h ( s, a ) =
min { f ( n ) h ( s, a ) , 1 } and apply Lemma
B.3 to g h , we have for h = 1 ,
E ( s, a ) ∼ d ˜ π ( n ) ˆ P ( n ) ,(cid:104) min (cid:110) f
( n ) 1 ( s, a ) , 1 (cid:111)(cid:105) ≤ (cid:115)
A E ( s, a ) ∼ ρ ( n )(cid:20)(cid:16) f ( n ) 1 ( s, a ) (cid:17) 2 (cid:21) ≤ (cid:112)
Aζ ( n ) .
And ∀ h ≥ 2 , we have
E ( s, a ) ∼ d ˜ π ( n ) ˆ P ( n ) ,h (cid:104) min (cid:110) f
( n ) h
( s, a ) , 1 (cid:111)(cid:105) ≤ E (˜ s, ˜ a )
∼ d ˜ π ( n ) ˆ P ( n ) ,h −  min   (cid:13)(cid:13)(cid:13) ˆ ϕ ( n )
h − 1 (˜ s, ˜ a ) (cid:13)(cid:13)(cid:13) Σ −n,ρ ( n ) h − 1 , ˆ ϕ ( n ) h −(cid:115) nA
E ( s, a ) ∼ ˜ ρ ( n ) h (cid:20)(cid:16) f
( n ) h
( s, a ) (cid:17) 2 (cid:21) + dλ + nζ ( n ) , 1   

[Equation]

[Equation]
Note that we here use the fact min { f ( n ) h ( s, a ) , 1 } ≤ 1 , E ( s, a ) ∼ ρ ( n ) h (cid:20)(cid:16)
f ( n ) h ( s, a ) (cid:17)(cid:21) ≤ ζ ( n ) and
E ( s, a ) ∼ ˜ ρ ( n ) h (cid:20)(cid:16) f
( n ) h
( s, a ) (cid:17) 2 (cid:21) ≤ ζ ( n ) .
Then according to our choice of α ( n ) , we get
E
[Equation] [Equation]
n
[Equation]

[Equation]
Combining all things together,
[Equation]
≥
[Equation]
 
[Equation]
≥
[Equation]
, which proves the inequality.
Lemma B.7 (Pessimism) .
Consider an episode n ∈ [ N ] and set α ( n ) =
Θ (cid:16) H (cid:112) nAζ ( n ) + dλ (cid:17) .
When the event E holds, we have v ( n ) i ( s ) − v π ( n ) i ( s ) ≤ H (cid:112)
Aζ ( n ) , ∀ n ∈ [ N ] , i ∈
[ M ] .
Proof.
Let f ( n ) h ( s, a ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h ( ·| s, a ) − P ⋆h ( ·| s, a ) (cid:13)(cid:13)(cid:13) 1 , then according to the event E , we have
E ( s, a ) ∼ ρ ( n
[Equation]
A direct conclusion of the event E is we can find an absolute constant c
[Equation]
such that
[Equation]
[ H
[Equation]
Again
[Equation]
we prove the following inequality by induction:
[Equation]
• When h = H , we have
[Equation]
• Suppose the statement is true for h + 1 , then for step h , we have
[Equation]
where we use the fact
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
and the last row uses the induction assumption.
The remaining steps are exactly the same as the proof in Lemma B.5 or Lemma B.6, we may prove
[Equation]
and
[Equation]

[Equation]
h ≥ 2 .
Combining all things together, we get
[Equation]
i
[Equation]
which has finished the proof
[Equation]
Lemma B.8.
For the model-based algorithm, when we pick λ = Θ (cid:16) d log NH | Φ | δ
[Equation]
, ζ ( n ) =
Θ (cid:16) 1 n log |M| HNδ (cid:17) and α ( n ) =
Θ (cid:16) H (cid:112) nAζ ( n ) + dλ (cid:17) , with probability 1 − δ , we have
N (cid:88)
[Equation]
Proof.
With our choice of λ and ζ ( n ) , according to Lemma B.2, we know E holds with probability 1 − δ .
Furthermore, we have
[Equation]
Let f ( n ) h ( s, a ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h ( ·| s, a ) − P ⋆h ( ·| s, a ) (cid:13)(cid:13)(cid:13) 1 .
According to the definition of the event E , we have
E s ∼ ρ ( n ) h
[Equation]
By definition, we have
[Equation]
For each fixed i ∈ [ M ] , h ∈ [ H ] and n ∈ [ N ] , we have
[Equation]
Note that we use the fact V ( n ) h
+1 ,i ( s ) − V ( n ) h
+1 ,i ( s ) is upper bounded by 2 H 2 , which can be proved easily using induction using the fact that ˆ β ( n ) h ( s, a ) ≤ H .
Applying the above formula recursively to E s ∼ d π ( n ) P⋆,h + (cid:104) V ( n ) h
+1 ,i ( s ) − V ( n ) h
+1 ,i ( s ) (cid:105) , one gets the following result (or more formally, one can prove by induction, just like what we did in Lemma B.5, Lemma B.6 and Lemma B.7):
[Equation]
First, we calculate the first term (a) in Inequality equation 11.
Following Lemma B.4 and noting the bonus ˆ β ( n ) h is O
( H ) , we have
[Equation]
[Table] [Table] [Equation] [Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Note that we use the fact that B = H when applying Lemma B.4.
In addition, we have
[Equation]
≤ d.
Then,
H
[Equation]
Second
[Equation]
we calculate the term (b) in inequality equation 11.
Following Lemma B
[Equation]
4 and noting that f ( n ) h ( s, a is upper-bounded by 2 (i
[Equation]
e., B = 2 in Lemma B.4), we have
H
[Equation]
≤
[Equation]
where in the second inequality, we use E ( s, a ) ∼ ρ ( n ) h
[Equation]
ζ
[Equation]
and in the last line, recall (cid:112) nAζ ( n ) + dλ ≲ α ( n ) /H .
Then, by combining the above calculation of the term (a) and term (b) in inequality equation 11, we have:
[Equation]
Taking maximum over i on both sides and using the definition of ∆ ( n )
[Equation]
we get
[Equation] [Equation]
Hereafter, we take the dominating term out.
Note that
N (cid:88)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(Potential function bound, Lemma E.3 noting ∥
ϕ ⋆h ( s, a ) ∥ 2 ≤ 1 for any ( s, a ) .)
Finally,
[Equation]
(Some algebra.
We take the dominating term out.
Note that α ( n ) is increasing in n )
[Equation]
This concludes the proof
[Equation]
Proof of Theorem 4.
Proof.
For any fixed episode n and agent i , by Lemma B.5, Lemma B.6 and Lemma B.7, we have v
[Equation]
Taking maximum over i on both sides
[Equation]
we have
[Equation]
From Lemma B.8, with probability 1 − δ , we can ensure
N (cid:88)
[Equation]
Therefore, according to Lemma E.4, when we pick N to be
[Equation]
we have
[Equation]
On the other hand, from equation 12, we have
[Equation]
which has finished the proof.
C A NALYSIS OF THE M ODEL -F REE M ETHOD
For the model-free method, throughout this section we assume the Markov game is a block Markov game.
C. C ONSTRUCTION OF N h AND F h
Let C h
= { Σ h : Σ h
=
λI
d + (cid:80) lk =
1 ϕ h
( s k , a k ) ϕ
h ( s k , a k ) ⊤ | ϕ h ∈
Φ h , l ∈ [ N ] , s k ∈ S , a k ∈
A , ∀ k ∈ [ l ] } .
Fix a variable L , for each h ∈
[ H ] , define a function class ˜ F h ∈ R S×A by
[Equation]
For a given parameter ˜ ε
[Equation]
let N h be a ˜ ε -net of ˜ F h under the ∥ · ∥ ∞ metric.
Define Π h as the set of all possible policies produced by equation 2 (or equation 3 or equation 4, according to the problem setting).
We then define the discriminator function class F h as followings:
[Equation]
CCE)
F 3 ,h := (cid:26) f
( s ) :
= max ω h
+1 ,i ∈
Ω h
+1 ,i E a ∼ ( ω h +
[Equation]
i ◦ π h +1 )
( s ) (cid:20)
[Equation] [Equation]
a )
H
[Equation] [Equation]
a
[Equation]
(cid:21)(cid:12)(cid:12)(cid:12)(cid:12) i ∈
[ M ] , π h +1 ∈
Π h
+1 , ϕ h +1 ∈
Φ h +1 , ∥ θ ∥ 2 ≤ √ d (cid:27) , (For CE)
F 4 ,h := (cid:26) f
( s ) :=
E a ∼ π h +1 ( s )
  min (cid:110) c
∥ ϕ h
[Equation]
1 ( s
[Equation]
a ) ∥ Σ − 1 h
[Equation] [Equation]
(cid:111)
H
[Equation] [Equation]
θ   (cid:12)(cid:12)(cid:12) (cid:12) (cid:12) (cid:12) c ∈ [0 , L ] , π
h +1 ∈
Π h
+1 , Σ h +1 ∈ C
h
+1 , ϕ h +1 ∈
Φ h
+
[Equation]
∥ θ ∥ 2 ≤ √ d (cid:27) ,
G :=
{ f : S → [0 , 1] } ,
F h :=
( F 1 ,h ∪ F 2 ,h ∪ F 3 ,h ∪ F 4 ,h ) ∩ G .
C. H IGH P ROBABILITY E VENTS
We define the following event
[Equation]
Similar to the procedure of the model-based case, we first prove a few lemmas which lead to the conclusion that E holds with a high probability.
Lemma C.1.
For any n ∈ [ N ] , h ∈ [ H ] , we have ˆ P ( n ) h ( s ′ | s, a ) =
ˆ ϕ ( n ) h ( s, a )
⊤ ˆ w ( n ) h
( s ′ )
for some ˆ w ( n ) h : S → R d .
For any function f : S → [0 , 1] and n ∈ [ N ] , h ∈
[ H ] , we have (cid:13)(cid:13)(cid:13)(cid:82) S
ˆ w ( n ) h ( s ′ )
f ( s ′ )
d s ′ (cid:13)(cid:13)(cid:13) 2 ≤ √ d , and there exist θ, ˜ θ ∈ R d such that ( P ⋆h f ) ( s, a ) =
ϕ ⋆h ( s, a )
⊤ θ, (cid:16) ˆ P ( n ) h f (cid:17) ( s, a )
=
ˆ ϕ ( n ) h ( s, a ) ⊤ ˜ θ and max {
∥ θ ∥ 2 , ∥ ˜ θ ∥ 2 } ≤ √ d .
Furthermore, we have ∥ ˜ θ ∥ ∞ ≤ 1 .
Proof.
By definition, we have
[Equation]
where θ = (cid:82) S w ⋆h ( s ′ )
f ( s ′ )
d s ′ .
Furthermore, note that ∥ f ∥ ∞ ≤ 1 , according to the assumption on w ⋆h , we have
[Equation]
which implies ∥ θ ∥ 2 ≤ √ d .
For (cid:16) ˆ P ( n ) h f (cid:17) ( s, a ) , let ˆ w ( n ) h
( s ′ ) :
= 
[Equation]
 (cid:88)
[Equation]
Since ϕ ( n ) h ( s, a ) is an one-hot vector, one has (cid:13)(cid:13)(cid:13) ˆ w ( n ) h
( s ′ )
(cid:13)(cid:13)(cid:13) ∞ ≤ 1 , ∀ s ′ ∈ S .
It follows that (cid:13)(cid:13)(cid:13)(cid:82) S
ˆ w ( n ) h ( s ′ )
f
[Equation]
s ′ )
d s ′ (cid:13)(cid:13)(cid:13) ∞ ≤ 1 , and therefore, (cid:13)(cid:13)(cid:13)(cid:82) S
ˆ w ( n ) h ( s ′ )
f ( s ′ )
d s ′ (cid:13)(cid:13)(cid:13) 2 ≤ √ d .
By definition, we have
[Equation]
where ˜ θ = (cid:82) S ˆ w ( n ) h
( s ′ )
f ( s ′ )
d s ′ .
Due to the property we just derived for ˆ w ( n ) h , similar to the proof of the true model, we also have ∥ θ ∥ 2 ≤ √ d .
Meanwhile, one can easily see that ∥ θ ∥ ∞ ≤ 1 , using the fact (cid:13)(cid:13)(cid:13)(cid:82) S ˆ w ( n ) h ( s ′ )
f
( s ′ )
d s ′ (cid:13)(cid:13)(cid:13) ∞ ≤ 1 .
Lemma C.2 (Covering Number of ˜ F h ) .
When Φ h is the set of one-hot vectors and λ ≥ 1 , it’s possible to construct the ˜ ε -net N h such that
|N h
| ≤ M (cid:16) 12 H 2 L 2 d ˜ ε (cid:17) 3 d | Φ | , ∀ h ∈ [ H ] .
Furthermore, we have | Π h | ≤ |N h | M ≤ M M (cid:16) 12 H 2 L 2 d ˜ ε (cid:17)
Md | Φ | M .
Proof.
Recall that
[Equation]
Note that when Φ h is the set of one-hot vectors, Σ h will be a diagonal matrix.
In this case, ˜ F h is the subset of the following function class:
[Equation]
Let Θ be an ℓ 2 -cover of the set { θ ∈ R d : ∥ θ ∥ 2 ≤ 2 H 2 √ d } at scale ˜ ε .
Then we know | Θ | ≤ (cid:16) 4 H 2 √ d ˜ ε (cid:17) d .
Let W be an ℓ ∞ -cover of the set [0 , L ] at scale ˜ ε
=
˜ ε 2 H 2 √ d , we have |W| ≤ 2 H 2 L √ d ˜ ε .
Define the covering set by ¯
[Equation]
ϕ
[Equation]
h
[Equation]
˜ c ∈ W
[Equation]
˜
[Equation]
˜
[Equation]
∈ Θ (cid:111) .
Then, for any f ∈ ˜ F h , by definition, suppose f takes the following form:
[Equation]
Then we can find ˜ θ, ˜ θ ′ ∈ Θ , ˜ c ∈ W
such that ∥ θ − ˜ θ ∥ 2 ≤ ˜ ε, ∥ θ ′ − ˜ θ ′ ∥
≤ ˜ ε and | c − ˜ c | ≤ ˜ ε ′ .
Let ¯ f ( s, a ) :
= r h,i ( s, a ) + min { ˜ cϕ h ( s, a ) ⊤ ˜ θ ′ , H } + ϕ h ( s, a )
⊤ ˜
θ, then we have
[Equation]
√ d ˜ ε ′ + L ˜ ε ≤ 3 L ˜ ε, which implies ¯ F h is a 3 L ˜ ε -covering of ˜ F ′ h (therefore, is a 3 L ˜ ε -covering of ˜ F h ), and we have (cid:12)(cid:12)
¯ F h (cid:12)(cid:12) ≤ M (cid:18)
4 H
Ld ˜ ε (cid:19)
d | Φ | .
Replacing ˜ ε by ˜ ε 3 L , we get an ˜ ε -covering of ˜ F h whose size is no larger than M (cid:16) 12 H 2 L 2 d ˜ ε (cid:17)
3 d | Φ | .
For Π h , since each policy is determined by M members from N h , we have | Π h | ≤ |N h | M , which has finished the proof.
Lemma C.3 (Covering Number of F h ) .
When Φ h is the set of one-hot vectors and λ ≥ 1 .
The γ -covering number of F h is at most 4 M | Π h +1 | (cid:16)
L 2 d γ (cid:17) 3 d | Φ | 2 .
Proof.
We cover F 1 ,h , F 2 ,h , F 3 ,h , F 4 ,h separately.
For F 1 ,h , let Θ be an ℓ 2 -cover of the set { θ ∈
R d : ∥ θ ∥ 2 ≤ √ d } at scale γ .
Then we know | Θ | ≤ (cid:16) 2 √ dγ (cid:17) d .
Define the covering set of F 1 ,h as
[Equation]
For any f ∈ F 1 ,h , suppose
[Equation]
Then we can find ˜ θ, ˜ θ ′ ∈ Θ such that ∥ θ − ˜ θ ∥ 2 ≤ γ, ∥ θ ′ − ˜ θ ′ ∥ 2 ≤ γ .
Let
[Equation]
Then we have
[Equation]
which implies ˜ F 1 ,h is a 2 γ covering of F 1 ,h .
Furthermore, we have
[Equation]
|
[Equation]
For F 2 ,h and F 3 ,h , we construct ˜ F
[Equation]
Similar to the proof of F 1 ,h , we may verify ˜ F 2 ,h is a γ -covering of F 2 ,h , and | ˜ F 2 ,h | ≤ M
[Equation]
Π h +
[Equation]
d γ
[Equation]
d | Φ | .
For F 3 ,h , we only prove the case of NE or CCE, the case of CE can be proved in a similar manner.
We construct
[Equation]
For any f ∈ F 3 ,h , suppose
[Equation]
Then we can find ˜ θ ∈ Θ such that ∥ θ − ˜ θ ∥ 2 ≤ γ .
Let
[Equation] [Equation]
we have
[Equation]
≤ γ, and
[Equation]
which implies
[Equation]
Therefore
[Equation]
we conclude ˜ F 3 ,h is a γ -covering of F 3 ,h , and
[Equation]
Φ | .
For F 4 ,h , note that when Φ h is the set of one-hot vectors, Σ h will be a diagonal matrix.
In this case,
F 4 ,h is the subset of the following function class:
[Equation]
In this case, let W be an ℓ ∞ cover of the set [0 , L ] at scale ˜ γ := γ √ d , we have |W| ≤ L √ d γ .
Let
[Equation]
Then, for any f ∈ F 4 ,h , suppose
[Equation]
Then we can find ˜ θ, ˜ θ ′ ∈ Θ , ˜ c ∈ W such that ∥ θ − ˜ θ ∥ 2 ≤ γ, ∥ θ
′ − ˜ θ ′ ∥ 2 ≤ γ and | c − ˜ c | ≤ ˜ γ .
Let
˜ f
( s ) :=
E a ∼ π h
+1 ( s ) (cid:34) min { ˜ cϕ h
+
( s, a )
⊤ ˜
θ ′ , H }
H+ ϕ h
+
( s, a ) ⊤ ˜
θ (cid:35) , then we have | f ( s ) − ˜ f
( s ) | ≤ E a ∼ π h
+1 ( s ) (cid:104) ∥
ϕ h
+
( s, a ) ∥ 2 (cid:13)(cid:13)(cid:13)
θ − ˜ θ (cid:13)(cid:13)(cid:13) 2 (cid:105) + 1 H
[Equation]
E a ∼ π h +1 ( s ) (cid:104) ∥
ϕ h
+
( s, a ) ∥ 2 (cid:13)(cid:13)(cid:13)
˜ c ˜ θ ′ − cθ ′ (cid:13)(cid:13)(cid:13) 2 (cid:105)
[Equation]
≤
[Equation]
which implies ˜ F 4 ,h is a 3 Lγ -covering of F 4 ,h , and we have
[Equation]
In summary, we know ˜ F h :=
˜ F 1 ,h ∪ ˜ F 2 ,h ∪ ˜ F 3 ,h ∪ ˜ F 4 ,h is a 3 Lγ -covering of F h .
And
[Equation]
Replacing γ by γ 3 L , we get an γ -covering of F h whose size is no larger than
[Equation]
L
[Equation]
d γ
[Equation]
| Φ | 2 , which has finished the proof.
Below we omit the superscript n and subscript h when clear from the context.
Denote
[Equation]
Lemma C.4 (Uniform Convergence for Square Loss) .
Let there be a dataset D := { ( s i , a i , s ′ i ) }
ni = collected in n episodes.
Denote that the data generating distribution in iteration i by d i , and ρ = 1 n (cid:80) ni =1 d i .
Note that d i can depend on the randomness in episodes 1 , . .
, i − 1 .
For a finite feature class Φ and a discriminator class F : S → [0 , 1] with γ -covering number ∥F∥ γ , we will show that, with probability at least 1 − δ :
[Equation]
for all ϕ ∈ Φ , ∥ θ ∥ ∞ ≤ 1 and f ∈ F , where recall that ϕ
⋆ is the true feature and θ ⋆f is defined as
E s ′ ∼ P
⋆
( s, a )
[ f ( s ′ )]
=
⟨ ϕ ⋆ ( s, a ) , θ ⋆f ⟩ .
Proof.
To start, we focus on a given f ∈ F .
We first give a high probability bound on the following deviation term:
[Equation]
Denote g ( s i , a i ) = ϕ
( s i , a i )
⊤ θ and g ⋆
( s i , a i ) =
ϕ
⋆ ( s i , a i ) ⊤ θ ⋆f .
At episode i , let F i − 1 be the σ -field generated by all the random variables over the first i − 1 episodes, for the random variable
Y i :=
( g ( s i , a i ) − f ( s ′ i ))
2 − ( g ⋆ ( s i , a i ) − f ( s ′ i )) 2 , we have
[Equation]
Here the conditional expectation is taken according to the distribution d i |F i − 1 .
The last equality is due to the fact that
[Equation] [Equation]
Next, for the conditional variance of the random variable
[Equation]
we have:
[Equation]
Noticing Y i ∈ [ −
[Equation]
Applying Lemma 1 in (Foster and Rakhlin
[Equation] [Equation]
we get with probability at least 1 − δ ′
[Equation]
we can bound the deviation term above as:
[Equation]
Further, consider a finite point-wise cover of the function class G :=
{ g ( s, a ) =
ϕ
( s, a ) ⊤ θ : ϕ ∈ Φ , ∥ θ ∥ ∞ ≤ 1 } .
Note that, with a ℓ ∞ -cover W of W = {∥ θ ∥ ∞ ≤ 1 } at scale γ , we have for all ( s, a ) and ϕ ∈ Φ , there exists ¯ θ ∈ W , |⟨ ϕ ( s, a ) , θ − ¯ θ ⟩| ≤ γ , and we have |W| =
(cid:16) 2 γ (cid:17) d .
Let ˜ F be a γ -covering set of F .
For any f ∈ F , there exists ¯ f ∈ ˜ F such that ∥ f − ¯ f ∥ ∞ ≤ γ .
Then, applying a union bound over elements in Φ × W × ˜ F , with probability 1 − | Φ ||W|| ˜ F| δ ′ , for all θ ∈ W , f ∈ F , we have:
[Equation]
(setting γ = 1 /n ) where ¯ Y i :
[Equation]
Finally, setting δ = δ ′ /
[Equation]
| Φ ||W|| ˜ F| (cid:17) , we get log 2 δ ′ ≤ log 2(4 n ) d | Φ || ˜
F| δ .
This completes the proof.
Lemma C.5 (Deviation Bounds for Alg. 3) .
Let ε ′ =
128log
( 2(4 n ) d ·| Φ |·∥F∥ 1 / 2 n δ ) n .
If Alg. 3 is called with a dataset D of size n , then with probability at least 1 − δ , for any f ∈ F ⊂ [0 , 1] S , we have
[Equation]
Proof.
We begin by using the result in Lemma C.4 such that, with probability at least 1 − δ , for all ∥ θ ∥ ∞ ≤ 1 , ϕ ∈ Φ and f ∈ F , we have
[Equation]
⋆f , f ) (cid:3) + ε ′ / 2 .
Thus, with probability at least 1 − δ we have:
[Equation]
(since E s ′ ∼ P
⋆
( s, a )
[ f ( s ′ )]
= ϕ
⋆ ( s, a ) ⊤ θ ⋆f )
[Equation]
ˆ ϕ, ˆ
[Equation] [Equation]
( ϕ ⋆ , θ ⋆f , f ) (cid:17) + ε ′ (Lemma C.4, and ∥ ˆ θ f ∥ ∞ ≤ 1 according to the proof in Lemma C.1)
[Equation]
(by the optimality of ˆ ϕ, ˆ θ f under L λ, D ( · , · , f ) ) which means the inequality in the lemma statement holds.
Here, we use ∥ θ ⋆f ∥ 22 ≤ d .
Lemma C.6.
When ˆ P ( n ) h is computed using Alg. 3 and the Markov games is a block Markov game, if we set
[Equation]
then E holds with probability at least 1 − δ .
Proof.
Combining Lemma C.5 and Lemma C.3, we have that
[Equation]
M log (cid:16) dNHML | Φ | δ ˜ ε (cid:17) n 
 , which shows E 1 holds with a high probability.
Combining this result with Lemma E.1, we have proved Lemma C.6.
C. S TATISTICAL G UARANTEES
To ensure the algorithm is well-defined, we first prove the following lemma which implies the optimistic Q-value estimators always belong to the function class ˜ F h .
Lemma C.7.
When α ( n ) ≤ L , we have Q ( n ) h,i ∈ ˜
F h , ∀ h ∈
[ H ] , i ∈
[ M ] , n ∈ [ N ] .
Proof.
Because ˆ β ( n ) h is upper bounded by H , by induction one can easily get V ( n ) h +1 ,i ≤ 2 H 2 .
Then according to the result of Lemma C.1, we know ( ˆ P ( n ) h V ( n ) h
+1 ,i )( s, a )
= ϕ ( n )
h ( s, a ) ⊤ θ with ∥ θ ∥ 2 ≤ 2 H 2 √ d .
We conclude Q ( n ) h,i ∈ ˜ F h .
We will show later that our choice of α ( n ) and L always satisfies the condition α ( n ) ≤ L .
Lemma C.8.
We have
• For NE and CCE,
[Equation]
• For CE,
[Equation]
Proof.
We only prove the case of NE and CCE, the case of CE can be proved similarly.
Let ˜ Q ( n ) h,i be the nearest neighbour of Q ( n ) h,i in N h , we have
[Equation]
which has finished the proof.
Lemma C.9 (One-step back inequality for the learned model) .
Suppose the event E holds.
Consider a set of functions { g h } Hh =1 that satisfies g h ∈ S ×
A → R + , s.t.
∥
g h ∥ ∞ ≤ B .
For a given policy π , suppose E a ∼ U ( A ) [ g h ( · , a )] ∈ F 1 ,h , then we have
[Equation]
≤     
[Equation]
Recall Σ n,ρ ( n ) h , ˆ ϕ ( n ) h
[Equation]
ρ
[Equation]
h
[Equation]
Proof.
For step h = 1 , we have
[Equation]
For step h = 2 , . . .
, H −
[Equation]
we observe the following one-step-back decomposition:
[Equation]
a
[Equation]
=
[Equation]

[Equation]
min 
[Equation]
, B     .
where we use the fact that g h is bounded by B .
Then, (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:90)
S
[Equation]
a ∈A
[Equation]
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
(Definition of ˜ ρ ( n ) h )
Combing the above results together
[Table]
we get
[Equation]
[Table]
, B   
 ≤
[Equation]
  min  
[Table]
[Equation]
[Table]
    , which has finished the proof.
The following lemma is an exact copy of Lemma B.4, and here we state it again just for completeness.
Lemma C.10 (One-step back inequality for the true model) .
Consider a set of functions { g h } Hh = that satisfies g h ∈ S ×
A → R + , s.t.
∥
g h ∥ ∞ ≤ B .
Then for any given policy π , we have
[Equation]
[ g 2 h ( s, a )] + B 2 λd, h ≥
Recall Σ n,γ ( n ) h ,ϕ ⋆h = n E ( s, a ) ∼ γ ( n ) h (cid:2)
ϕ
⋆h ( s, a ) ϕ
⋆h ( s, a ) ⊤ (cid:3) + λI d .
Lemma C.11 (Optimism for NE and CCE) .
Consider an episode n ∈ [ N ] and set α ( n ) =
Θ (cid:16) H (cid:112) nA 2 ζ ( n ) + dλ (cid:17) .
When the event E holds and the policy π ( n ) is computed by solving NE or
CCE, we have
[Equation]
Proof.
Denote ˜ µ ( n ) h,i ( ·| s ) :
= arg max µ (cid:18) D µ,π ( n ) h, − i Q † ,π ( n ) − i h,i (cid:19) ( s ) and let ˜ π ( n ) h = ˜ µ ( n ) h,i × π ( n ) h, − i .
Let f ( n ) h ( s, a ) =
[Equation]
1 H (cid:16) ˆ P ( n ) h
[Equation]
( s, a ) , note that by definition, we have 1 H V † ,π ( n ) − i
h +1 ,i ( s ) is bounded by 1 , and
[Equation]
where we use the result of Lemma C
[Equation]
1 and get 1 H
[Equation]
) is a linear function in ϕ ⋆h
+ and the 2-norm of the weight is upper bounded by √ d .
Then according to the event E
[Equation]
we have
[Equation]
A direct conclusion of the event E is we can find an absolute constant c , such that
[Equation]
Next
[Equation]
we prove by induction that
[Equation]
First, notice that ∀ h ∈
[ H ] ,
[Equation]
where the inequality uses the result of Lemma C.8.
Now we are ready to prove equation 16,
• When h = H , we have
[Equation]
• Suppose the statement is true for h + 1 , then for step h , we have
[Equation]
−
[Equation]
[Table]
[Equation]
where the last row uses the induction assumption
[Equation]
Therefore
[Equation]
we have proved equation 16.
We then apply h = 1 to equation
[Equation]
and get
[Equation]
, a ) (cid:105) − 2 H ˜ ε.
For the second term, since 1 H ˆ P ( n ) h V † ,π ( n ) − i
h
+1 ,i is linear in ˆ ϕ ( n ) h and
1 H P ⋆h V † ,π ( n ) − i
h
+1 ,i is linear in ϕ ⋆h , and according to the result of Lemma C.1, the 2-norm of their weights are both upper bounded by √ d .
Therefore, we have E a ∼ U ( A ) (cid:104) f ( n ) h ( · , a ) (cid:105) ∈ F 1 ,h .
By Lemma C.9, we have for h = 1 ,
E
[Equation]
n
[Equation] [Equation]
(cid:20)(cid:16)
[Equation]
(cid:17)
[Equation]
(cid:21) ≤ (cid:112) Aζ ( n ) .
And ∀ h ≥ 2 , we have
E
[Equation] [Equation]
[Equation]

[Equation]
min
[Equation]
nA
[Equation]
(cid:20)(cid:16)
[Equation]
( n )
[Equation]
( s
[Equation]
a
[Equation]
(cid:17)
[Equation]
(cid:21) + dλ
[Equation]
nA
[Equation]
( n )
[Equation]
1   
 ≲ E (˜ s, ˜ a ) ∼ d ˜ π ( n ) ˆ P ( n ) ,h −  min   (cid:13)(cid:13)(cid:13)
ˆ ϕ ( n )
h − 1 (˜ s, ˜ a ) (cid:13)(cid:13)(cid:13) Σ −n,ρ ( n ) h − 1 , ˆ ϕ ( n ) h −(cid:112) nA 2 ζ ( n ) + dλ,    .
Note that we here use f ( n ) h ( s, a ) ≤ 1 , E ( s, a ) ∼ ρ ( n ) h (cid:20)(cid:16) f ( n ) h ( s, a ) (cid:17) 2 (cid:21) ≤ ζ ( n ) and
E ( s, a ) ∼ ˜ ρ ( n ) h (cid:20)(cid:16) f
( n ) h
( s, a ) (cid:17) 2 (cid:21) ≤ ζ ( n ) .
Then according to our choice of α ( n ) , we get
E ( s, a ) ∼ d ˜ π ( n ) ˆ P ( n ) ,h (cid:104) f ( n ) h ( s, a ) (cid:105) ≤ E (˜ s, ˜ a ) ∼ d ˜ π ( n ) ˆ P ( n ) ,h −  min  
cα ( n )
H (cid:13)(cid:13)(cid:13) ˆ ϕ ( n ) h − 1 (˜ s, ˜ a ) (cid:13)(cid:13)(cid:13) Σ −n,ρ ( n ) h − 1 , ˆ ϕ ( n ) h −,    .
Combining all things together,
[Equation]
which proves the inequality
[Equation]
Lemma C.12 (Optimism for CE) .
Consider an episode n ∈ [ N ] and set α ( n ) =
[Equation]
nA 2 ζ
[Equation]
n ) + dλ (cid:17) .
When the event E holds, we have
[Equation]
Proof.
Denote ˜ ω ( n
[Equation]
,i = arg max
[Equation]
Q
[Equation]
( n ) h,i (cid:17) ( s ) and let ˜ π ( n ) h =
˜ ω
h,i ◦ π ( n ) h .
Let f ( n ) h ( s, a ) =
(cid:12)(cid:12)(cid:12) 1 H (cid:16) ˆ P ( n )
h −
[Equation]
i V ω ◦
π
[Equation]
n ) h +1 ,i (cid:12)(cid:12)(cid:12) ( s, a ) , note that by definition, we have 1 H max ω ∈ Ω
i V ω ◦ π ( n ) h
+1 ,i ( s ) is bounded by 1 , and
[Equation]
where we use the result of Lemma C.1 and get 1 H (cid:16) P ⋆h +1 max ω ∈ Ω
i V ω ◦ π ( n ) h
+
[Equation]
i (cid:17) ( s, a ) is a linear function in ϕ ⋆h and the 2-norm of the weight is upper bounded by √ d .
Then according to the event E , we have
E
[Equation]
A direct conclusion of the event E is we can find an absolute constant c
[Equation]
such that
[Equation]
Next
[Equation]
we prove by induction that
[Equation]
First, notice that ∀ h ∈
[ H ] ,
E
[Equation] [Equation]
where the inequality uses the result of Lemma C.8.
Now we are ready to prove equation 17,
• When h = H , we have
[Equation]
• Suppose the statement is true for h + 1 , then for step h , we have
[Equation]
where the last row uses the induction assumption
[Equation]
Therefore
[Equation]
we have proved equation 17.
We then apply h = 1 to equation
[Equation]
and get
[Equation]
For the second term, since 1 H ˆ P ( n ) h max ω ∈ Ω i V ω ◦ π ( n ) h
+1 ,i is linear in ˆ ϕ ( n ) h and 1 H P ⋆h max
ω ∈ Ω
i V ω ◦ π ( n ) h
+1 ,i is linear in ϕ ⋆ h , and according to the result of Lemma C.1, the 2-norm of their weights are both upper bounded by √ d .
Therefore, we have E a ∼ U ( A ) (cid:104) f ( n ) h ( · , a ) (cid:105) ∈ F 1 ,h .
By Lemma C.9, we have for h = 1 ,
[Equation]
And ∀ h ≥ 2 , we have
[Equation]
nA
[Equation]
n
[Equation]
    .
Note that we here use f ( n ) h ( s, a ) ≤ 1 , E ( s, a ) ∼ ρ ( n ) h (cid:20) (cid:16) f ( n ) h
( s, a ) (cid:17) 2 (cid:21) ≤ ζ ( n ) and
E ( s, a ) ∼ ˜ ρ ( n ) h (cid:20)(cid:16) f
( n ) h
( s, a ) (cid:17) 2 (cid:21) ≤ ζ ( n ) .
Then according to our choice of α ( n ) , we get
E
[Equation]
  min   cα
[Equation]
H (cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
n,ρ ( n ) h − 1 , ˆ ϕ ( n ) h −,    .
Combining all things together,
[Equation]
ε ≥
H
[Equation]
1 (cid:88) h
=
[Equation]

[Equation]
  cα ( n ) (cid:13)(cid:13)(cid:13)
[Equation]
( n ) h (˜ s
[Equation]
˜ a ) (cid:13)(cid:13)(cid:13) Σ −n,ρ ( n ) h , ˆ ϕ ( n ) h , H   
 − H (cid:112)
Aζ ( n ) − 2 H ˜ ε = − H
cid:112)
Aζ ( n ) − 2 H ˜ ε, which proves the inequality.
Lemma C.13 (pessimism) . Consider an episode n ∈ [ N ] and set α ( n ) =
Θ (cid:16) H (cid:112) nA 2 ζ ( n ) + dλ (cid:17) .
When the event E holds, we have
[Equation]
∀
[Equation]
∈ [ N ]
[Equation]
∈ [ M ] .
Proof.
Let ˜ f ( n ) h ( s, a ) =
(cid:12)(cid:12)(cid:12) 1 H (cid:16) ˆ P ( n )
h −
[Equation]
( n ) h +1 ,i (cid:12)(cid:12)(cid:12) ( s, a ) , note that by definition, we have 1 H V π ( n ) h
+1 ,i ( s ) is bounded by 1 , and
[Equation]
where we use the result of Lemma C.1 and get 1 H (cid:16) P ⋆h +1 V π ( n ) h +2 ,i (cid:17) ( s, a ) is a linear function in ϕ ⋆h
+ and the 2-norm of the weight is upper bounded by √ d .
Then according to the event E , we have
E
[Equation] [Equation]
[Equation]
A direct conclusion of the event E is we can find an absolute constant c , such that
[Equation]
Again
[Equation]
we prove the following inequality by induction:
[Equation]
• When h = H , we have
[Equation]
• Suppose the statement is true for h + 1 , then for step h , we have
[Equation]
where the last row uses the induction assumption.
The remaining steps are exactly the same as the proof in Lemma C.11 or Lemma C.12, we may prove
[Equation]
and
[Equation]

[Equation]
Combining all things together, we get v
[Equation]
n )
i
[Equation]
which has finished the proof.
Lemma C.14.
For the model-free algorithm, suppose N is large enough, when we pick λ
= Θ (cid:16) d log NH | Φ | δ
[Equation]
(cid:17) , L = Θ( NHAMd ) , ˜ ε =
[Equation]
2 HN and α ( n ) =
Θ (cid:16) H (cid:112) nA 2 ζ ( n ) + dλ (cid:17) , with probability 1 − δ , we have
[Equation]
Proof.
With our choice of λ and ζ ( n ) , according to Lemma C.6, we know E holds with probability 1 − δ .
Furthermore, with a proper choice of the absolute constants, we have
[Equation]
Let f
[Equation] [Equation]
We first verify 1 2 H 2 (cid:16) V ( n ) h
+1 ,i − V ( n ) h
+1 ,
i (cid:17) ∈ F 4 ,h .
By definition, we have
[Equation]
The first term is equal to 1 H 2 min
[Equation] [Equation]
n
[Equation]
s,
[Equation]
) , H (cid:33) , which is exactly the same as that in the definition of F 4 ,h (note that we use the property α ( n ) ≤ L, ∀ n ∈ [ N ] ).
For the second term, note that we have 0 ≤ 1 2 H 2 (cid:16) V ( n ) h,i − V ( n ) h,i (cid:17) ≤ 1 , ∀ h .
Therefore, by Lemma C.1, 1 2 H 2 P
⋆h
+
(cid:16) V ( n )
h +2 ,i − V ( n ) h +2 ,i (cid:17) ( s, a ) is a linear function in ϕ ⋆h +1 whose weight’s 2-norm is upper bounded by √ d .
Combing the above arguments, we conclude
[Equation]
According to the definition of the event E , we have
[Equation] [Equation]
By definition, we have
[Equation]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the last inequality is calculated using induction.
In particular,
[Equation]
First
[Equation]
we calculate the first term (a) in Inequality equation 20.
Following Lemma C.10 and noting the bonus ˆ β ( n ) h is O
( H ) , we have
[Equation]
(From equation 19)
[Equation]
Note that we use the fact that B = H when applying Lemma D.3.
In addition, we have n
[Equation]

[Equation]

[Equation]
Tr
[Equation]
≤ d.
Then,
[Equation]
Second, we calculate the term (b) in inequality equation 23.
Following Lemma D.3 and noting (cid:16) f ( n ) h ( s, a ) (cid:17) 2 is upper-bounded by 1 (i.e., B = 1 in Lemma D.3), we have
[Equation]
where in the second inequality, we use E ( s, a ) ∼ ρ ( n ) h
[Equation]
n
[Equation]
a
[Equation]
ζ
[Equation]
n ) , and in the last line, recall (cid:112) nAζ ( n ) + dλ ≲ α ( n ) /H .
Then, by combining the above calculation of the term (a) and term (b) in inequality equation 23, we have:
[Equation]
Taking maximum over i on both sides and use the definition of ∆ ( n )
[Equation]
we get
[Equation]
Hereafter, we take the dominating term out.
Note that
N (cid:88)
[Equation]
CS inequality)
[Equation]
(Potential function bound, Lemma E.3 noting ∥
ϕ ⋆h ( s, a ) ∥ 2 ≤ 1 for any ( s, a ) .)
Finally,
[Equation]
n
[Equation] [Equation]
(Some algebra.
We take the dominating term out.
Note that α ( n ) is increasing in n )
[Equation]
This concludes the proof.
Proof of Theorem
[Equation]
Proof.
For any fixed episode n and agent i , by Lemma C.11, Lemma C.12 and Lemma C.
[Equation]
we have
[Equation]
Taking maximum over i on both sides
[Equation]
we have
[Equation]
From Lemma C.
[Equation]
with probability 1 − δ , we can ensure
N
[Equation]
Therefore, according to Lemma E.4, when we pick N to be
O
[Equation]
log
[Equation]
(cid:18) HdAM | Φ | δε (cid:19)(cid:19) , we have
[Equation]
On the other hand, from equation 21, we have
[Equation]
which has finished the proof.
D A NALYSIS OF THE F ACTORED M ARKOV G AMES D. H IGH P ROBABILITY E VENTS
Define the set ¯Φ h,i = { ¯ ϕ h,i ( s, a ) :
= (cid:78)
j ∈
Z i ϕ h,j
( s [ Z j ] , a j ) | ϕ h,j ∈ Φ h,j } .
Let | Φ | = max h,j | Φ h,j | and | ¯Φ | = max h,i | ¯Φ h,i | .
Clearly, we have | ¯Φ | ≤ | Φ | L .
Define the following event
E 1 : ∀ n ∈ [ N ] , h ∈ [ H ] , i ∈
[ M ] , ρ ∈ (cid:110) ρ ( n ) h , ˜ ρ ( n ) h (cid:111) ,
E ρ (cid:20)(cid:13)(cid:13)(cid:13)
ˆ P ( n ) h,i ( ·| s [ Z i ] , a i ) − P ⋆ h,i ( ·| s [ Z i ] , a i ) (cid:13)(cid:13)(cid:13) 2(cid:21) ≤ ζ ( n ) ,
E 2 : ∀ n ∈ [ N ]
[Equation]
h ∈ [ H ]
[Equation]
i
[Equation]
[ M ]
[Equation]
∥
[Equation] [Equation]
a
[Equation]
(cid:18)
[Equation]
n ) h, ¯ ϕh
[Equation]
i (cid:19) − 1 = Θ (cid:32) ∥
¯ ϕ h,i ( s, a ) ∥ Σ − 1 n,ρ ( n ) h , ¯ ϕh,i (cid:33)
E :=
E 1 ∩ E 2 .
The following lemma shows that the event E holds with a high probability with proper choices of the parameters.
Lemma D.1.
When ˆ P ( n ) h is computed using Alg. 2, if we set
[Equation]
then E holds with probability at least 1 − δ .
The proof of Lemma D.1 is follows a similar procedure as that of Lemma B.2, with minor changes on the notations as well as some modifications on the union bound.
D. S TATISTICAL G UARANTEES
Lemma D.2 (One-step back inequality for the learned model) .
Suppose the event E holds.
Consider a set of functions { g h } Hh =1 that satisfies g h ∈ S [ Z i ] ×
A i → R + , s.t.
∥
g h ∥ ∞ ≤ B .
For a given policy π , we have
[Equation]
≤  
[Equation]
min
[Equation]
B
 
[Equation]
h
[Equation]
where
[Equation]
and
[Equation]
Proof.
For step h = 1 , we have
[Equation]
For h ≥
[Equation]
we observe the following one-step-back decomposition:
[Equation]
[Table]
Then,
[Table]
Σ n,
[Table]
n
[Table] [Table]
n
[Table] [Table]
n
[Table]
   (cid:90)
S
[ Z i ] |A
i | (cid:88)
a i ∈A (cid:89) j ∈
[Table]
i (cid:16)
[Table]
w
[Table] [Table]
−
[Table] [Table]
s
[Table]
n )
[Table] [Table]
[Table] [Table]
a j )
[Table]
g h (
[Table] [Table]
, a i )d s [ Z
[Table]
] 
 2 
 + B
[Table]
λd L (
[Table]
|
[Table]
and
[Table] [Table]
= n E (
[Table] [Table]
a )
[Table]
( n )
[Table]
(cid:20)(cid:16) E s
[Table]
P ( n )
[Table]
− 1 (
[Table]
s,
[Table]
a ) ,
[Table]
i
[Table]
U (
[Table]
i ) [
[Table]
h ( s [ Z i ] , a i )] (cid:17)
[Table]
(cid:21) + B 2 λd L
[Table] [Table]
n
[Table]
E
[Table]
s,
[Table] [Table]
U
[Table] [Table]
[Table] [Table]
[Table]
)] (cid:17)
[Table]
(cid:21) + B
[Table]
+
[Table]
Event
[Table]
, ˜ a )
[Equation]
a i
[Equation]
( A
[Equation]
(cid:2)
[Equation]
] , a
[Equation]
(Jensen)
=
n
[Equation]
(Definition of ˜ ρ ( n ) h )
Combing the above results together, we get
E (˜ s, ˜ a i ) ∼ d π ˆ P ( n ) ,h [ g h ( s [ Z i ] , a i )] ≤ E (˜ s, ˜ a ) ∼ d π ˆ P ( n ) ,h −(cid:34) min (cid:40) ˜
A (cid:13)(cid:13)(cid:13) ¯ ϕ ( n )
h − 1 ,i (˜ s, ˜ a ) (cid:13)(cid:13)(cid:13) Σ −n,ρ ( n )
h − 1 , ¯ ϕ ( n ) h − 1 ,i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:90)
S
[ Z i ] |A
i | (cid:88) a i ∈A i  (cid:79)
j ∈ Z i ˆ w ( n ) h − 1 ,j ( s j ) 
 g h ( s [ Z i ] , a i )d s [ Z i ] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) Σ n,ρ ( n ) h − 1 , ¯ ϕ ( n ) h − 1 ,i , B (cid:41)(cid:35) ≤ E (˜ s, ˜ a ) ∼ d π ˆ P ( n ) ,
h −  min   ˜
A (cid:13)(cid:13)(cid:13) ¯ ϕ ( n )
h − 1 ,i (
[Equation]
a
[Equation]
Σ −n,ρ ( n ) h − 1 , ¯ ϕ ( n ) h − 1 ,i (cid:113) n E ( s
[Equation]
a i ) ∼ ˜ ρ ( n ) h [ g 2 h ( s [ Z i ]
[Equation]
a i )]
+ B 2 λd L + B
[Equation]
nζ ( n )
[Equation]
B   
 , which has finished the proof.
Lemma D.3 (One-step back inequality for the true model) .
Consider a set of functions { g h } Hh =1 that satisfies g h ∈ S [ Z i ] ×
A i → R + , s.t.
∥
g h ∥ ∞ ≤ B .
Then for any policy π , we have
[Equation] [Equation]
where
[Equation]
and
[Equation]
Proof.
For step h = 1 , we have
[Equation]
[Equation] [Equation]
For step h = 2 , . . .
, H − 1 , we observe the following one-step-back decomposition:
E (˜ s, ˜ a i ) ∼ d πP⋆,h [ g h ( s [ Z i ] , a i )]
=
E (˜ s, ˜ a ) ∼ d πP⋆,h − 1 ,s ∼ P ⋆h − 1 (˜ s, ˜ a ) , a i ∼ π h ( s )
[ g h ( s [ Z i ] , a i )]
=
E (˜ s, ˜ a ) ∼ d πP⋆,h −   (cid:79)
j ∈ Z i ϕ
⋆h − 1 ,j (˜ s [ Z j ] , ˜ a j ) 
 ⊤ (cid:90)
S (cid:88) a i ∈A i  (cid:79)
j ∈
Z i w ⋆h − 1 ,j ( s j ) 

π h ( a i | s )
g h ( s [ Z i ] , a i )d s 
 ≤ ˜ A E (˜ s, ˜ a ) ∼ d πP⋆,h −   (cid:79)
j ∈ Z i ϕ
⋆h − 1 ,j (˜ s [ Z j ] , ˜ a j ) 
 ⊤ (cid:90)
S (cid:88) a i ∈A i |A
i |  (cid:79)
j ∈
Z i w ⋆h − 1 ,j ( s j ) 
 g h ( s [ Z i ] , a i )d s [ Z i ] 
 ≤ ˜ A E (˜ s, ˜ a ) ∼ d πP⋆,h − (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:79)
j ∈ Z i ϕ
⋆h − 1 ,j (˜ s [ Z j ] , ˜ a j ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) Σ − 1 n,γ ( n ) h − 1 , ¯ ϕ⋆h − 1 ,
i   · (cid:13)(cid:13) (cid:13) (cid:13)(cid:13)(cid:13)(cid:90) S (cid:88) a i ∈A i |A
i |  (cid:79)
j ∈
Z i w ⋆ h − 1 ,j ( s j ) 
 g h ( s [ Z i ] , a i )d s [ Z i ] (cid:13)(cid:13) (cid:13)(cid:13)
(cid:13)(cid:13) Σ n,γ ( n ) h − 1 , ¯ ϕ⋆h − 1 ,i .
Then, (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:90)
S (cid:88) a i ∈A i |A
i |  (cid:79)
j ∈
Z i w ⋆h − 1 ,j ( s j ) 
 g h ( s [ Z i ] , a i )d s [ Z i ] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) Σ n,γ ( n ) h − 1 , ¯ ϕ⋆h − 1 ,i ≤ n E (˜ s, ˜ a ) ∼ γ ( n ) h −   (cid:90)
S (cid:88) a i ∈A i |A
i |  (cid:79)
j ∈
Z i w ⋆h − 1 ,j ( s j ) 
 ⊤  (cid:79)
j ∈ Z i ϕ
⋆h − 1 ,j (˜ s [ Z j ] , ˜ a j ) 
 g h ( s [ Z i ] , a i )d s [ Z i ] 
 2 
 + B
[Equation]
(Use the
[Equation]
[Table]
[Equation]
[Table]
[Equation]
and
[Table]
[Equation]
[Table]
[Equation]
[Table]
(cid:13)(cid:13)
[Equation]
[Equation]
(Definition of ρ ( n ) h ) which has finished the proof.
Lemma D.4 (One-step back inequality for the true model) .
Consider a set of functions { g h } Hh =1 that satisfies g h ∈ S [ ∪ j ∈ Z i
Z j ] ×
A
[ Z i ] → R , s
[Equation]
t. ∥ g h ∥ ∞ ≤ B .
Then for any policy π , we have
[Equation]
where
[Equation]
and
[Equation]
Proof.
This Lemma can be proved using similar steps as those in the proof of Lemma D.3, noting that in this case the dimension of ˜ ϕ
⋆h,i is at most L 2 .
Lemma D.5 (Optimism for NE and CCE) .
Consider an episode n ∈ [ N ] and set α ( n ) =
Θ (cid:16) H ˜
A (cid:112) nζ ( n ) + d L λ (cid:17) .
When the event E holds and the policy π ( n ) is computed by solving
NE or CCE, we have
[Equation]
Proof.
Denote
[Equation]
(cid:19)
[Equation]
s ) and let ˜ π ( n ) h
= ˜ µ ( n ) h,i × π ( n ) h, − i .
Let f ( n ) h ( s, a ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h ( ·| s, a ) − P ⋆h ( ·| s, a ) (cid:13)(cid:13)(cid:13) 1 and f ( n ) h,i ( s [ Z i ] , a i ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h,i ( ·| s [ Z i ] , a i ) − P ⋆h,i ( ·| s [ Z i ] , a i ) (cid:13)(cid:13)(cid:13) 1 .
Then according to the event E , we have
E
[Equation]
a
[Equation]
∼
[Equation]
n ) h (cid:20)(cid:16) f
[Equation]
n
[Equation]
h,
[Equation]
A direct conclusion of the event E is we can find an absolute constant c
[Equation]
such that
[Equation]
Next, similar to the proof in Lemma B.5, we may prove
E s ∼ d(cid:20)
V
[Equation]
n
[Equation]
i
[Equation]
For the second term, note that we have the relation min { f ( n ) h ( s, a ) , 1 } ≤ (cid:80)
Mi =1 min { f ( n ) h,i ( s [ Z i ] , a i ) , 1 } .
By Lemma D.2, we have for h = 1 ,
E
[Equation] [Equation]
≤ (cid:115)
[Equation]
[Table]
f
[Equation]
n )
[Equation] [Equation]
s [ Z i ] ,
[Equation]
i ) (cid:17)
[Equation]
≤ ζ ( n ) .
Then according to our choice of α ( n )
[Equation]
we get
[Equation]
Combining all things together,
[Equation]
n
[Equation]
i
[Equation]
cα ( n ) (cid:13)(cid:13)(cid:13) ¯ ϕ ( n ) h,j ( s, a ) (cid:13)(cid:13)(cid:13) Σ −ρ ( n ) h , ¯ ϕ ( n ) h,j , H   
 − HM (cid:113) ˜
Aζ ( n ) ≥ − HM (cid:113) ˜
Aζ ( n ) , which proves the inequality.
Lemma D.6 (Optimism for CE) .
Consider an episode n ∈ [ N ] and set α ( n ) =
Θ (cid:16) H ˜
A (cid:112) nζ ( n ) + d L λ (cid:17) .
When the event E holds, we have
[Equation]
Proof.
Denote ˜ ω ( n ) h,i
[Equation]
(cid:17) ( s ) and let ˜ π ( n ) h = ˜ ω
h,i ◦ π ( n ) h .
Let f ( n ) h ( s, a ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h ( ·| s, a ) − P ⋆h ( ·| s, a ) (cid:13)(cid:13)(cid:13) 1 and f ( n ) h,i ( s [ Z i ] , a i ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h,i ( ·| s [ Z i ] , a i ) − P ⋆h,i ( ·| s [ Z i ] , a i ) (cid:13)(cid:13)(cid:13) 1 .
Then according to the event E , we have
E ( s, a ) ∼ ρ ( n ) h (cid:20)(cid:16) f
( n ) h,i ( s [ Z i ] , a i ) (cid:17) 2 (cid:21) ≤ ζ ( n ) ,
E ( s, a ) ∼ ˜ ρ ( n ) h (cid:20)(cid:16) f
( n ) h,i ( s [ Z i ] , a i ) (cid:17) 2 (cid:21) ≤ ζ ( n ) , ∀ n ∈ [ N ] , h ∈
[ H ] , i ∈
[ M ] (cid:13)(cid:13)
¯ ϕ h,i ( s, a ) (cid:13)(cid:13)
[Equation]
ˆΣ ( n ) h, ¯ ϕh,i (cid:19)
[Equation]
n,ρ ( n ) h , ¯ ϕh,i (cid:33) , ∀ n ∈ [ N ] , h ∈
[ H ] , ¯ ϕ h,
i ∈ ¯Φ h,i , i ∈
[ M ] .
A direct conclusion of the event E is we can find an absolute constant c , such that β
[Equation]
min
[Equation] [Equation]
Next, similar to the proof in Lemma B.6, we may prove
E
[Equation]
Note that we can use exactly the same steps in the proof of Lemma D.5 to bound the second term, and we get for h = 1 ,
[Equation]
And ∀ h ≥ 2 ,
[Equation]
Combining all things together,
[Equation]
which proves the inequality.
Lemma D.7 (pessimism) . Consider an episode n ∈ [ N ] and set α ( n ) =
Θ (cid:16) H ˜
A (cid:112) nζ ( n ) + d L λ (cid:17) .
When the event E holds, we have v
[Equation]
n
[Equation]
( s
[Equation]
HM
[Equation]
( n ) , ∀ n ∈ [ N ] , i ∈
[ M ] .
Proof.
Let f ( n )
[Equation]
s, a ) =
[Equation]
P ( n )
[Equation]
·|
[Equation]
, a
[Equation] [Equation]
[Equation]
and f ( n ) h,i ( s [ Z i ] , a i ) =
(cid:13)(cid:13)(cid:13) ˆ P ( n ) h,i ( ·| s [ Z i ] , a i ) − P ⋆h,i ( ·| s [ Z i ] , a i ) (cid:13)(cid:13)(cid:13) 1 .
Then according to the event E , we have
E
[Equation]
E
[Equation] [Equation]
Z
[Equation] [Equation]
, ∀ n ∈ [ N ] , h
[Equation]
] , i ∈
[Equation]
M ]
[Equation]
¯ ϕ
[Equation]
a
[Equation]
(cid:18) ˆΣ
[Equation]
n ) h, ¯ ϕh,i (cid:19) −
[Equation]
=
[Equation] [Equation]
a
[Equation]
(cid:13)(cid:13) Σ −n,ρ ( n ) h , ¯ ϕh,i (cid:33) , ∀ n ∈ [ N ] , h ∈
[ H ] , ¯ ϕ h,
i ∈ ¯Φ h,i , i ∈
[ M ] .
A direct conclusion of the event E is we can find an absolute constant c
[Equation]
such that
[Equation] [Equation]
Next, similar to the proof in Lemma B.
[Equation]
we may prove
[Equation]
and we get for h = 1 ,
[Equation]
And ∀ h ≥ 2 ,
[Equation]
n
[Equation]
  .
Finally, we get v
[Equation]
n )
i
[Equation]
≤ HM
[Equation]
which has finished the proof.
Lemma D.8.
When the event E holds and α ( n ) =
Θ (cid:16) H ˜
A (cid:112) nζ ( n ) + d L λ (cid:17) satisfies α (1) ≤ α (2) ≤ . . .
≤ α ( N ) , we have
[Equation]
α
[Equation]
Proof.
Let f
[Equation]
n
[Equation]
a
[Equation]
a
[Equation]
and f ( n ) h,i ( s [ Z i ] , a i ) =
[Equation]
ˆ P ( n ) h,i ( ·| s [ Z i ] , a i ) − P ⋆h,i ( ·| s [ Z i ] , a i ) (cid:13)(cid:13)(cid:13) 1 .
Then according to the event E , we have
E ( s, a ) ∼ ρ ( n ) h (cid:20)(cid:16) f
( n ) h,i ( s [ Z i ] , a i ) (cid:17) 2 (cid:21) ≤ ζ ( n ) ,
E ( s, a ) ∼ ˜ ρ ( n ) h (cid:20)(cid:16) f
( n ) h,i ( s [ Z i ] , a i ) (cid:17) 2 (cid:21) ≤ ζ ( n ) , ∀ n ∈ [ N ] , h ∈
[ H ] , i ∈
[ M ] (cid:13)(cid:13) ¯ ϕ
[Equation]
,i
[Equation]
s
[Equation]
a
[Equation]
n )
[Equation]
i
[Equation]
n
[Equation]
, ∀ n ∈ [ N ]
[Equation]
h ∈ [ H ] , ¯ ϕ h
[Equation]
i ∈ ¯Φ h,i , i ∈
[ M ] .
By definition, we have
[Equation]
With similar steps as those in the proof of Lemma B.8 (note that V ( n ) h,i ( s ) − V ( n ) h,i ( s ) is upper bounded by 2 H 2 M ), we have
E
[Equation] [Equation]
First, we calculate the first term (a) in Inequality equation 23.
Following Lemma D
[Equation]
4, we have
[Equation]
Note that we use the fact that B = H when applying Lemma D.3.
In addition, we have
[Equation]
=
[Equation]
Tr
[Equation]
≤ d L .
Then,
H
[Equation]
Second, we calculate the term (b) in inequality equation 23.
Following Lemma D.3 and noting f ( n ) h,i ( s [ Z i ] , a i ) is upper-bounded by 2 (i
[Equation]
e., B = 2 in Lemma D.3), we have
[Equation] [Equation]
where in the second inequality, we use E ( s, a ) ∼ ρ ( n ) h
[Equation]
f
[Equation]
i
[Equation]
s
[ Z
[Equation] [Equation]
[Equation]
≤ ζ ( n ) , and in the last line, recall ˜
A (cid:112) nζ ( n ) + d L λ ≲ α ( n ) /H .
Then, by combining the above calculation of the term (a) and term (b) in inequality equation 23, we have:
[Equation]
Taking maximum over i on both sides and use the definition of ∆ ( n )
[Equation]
we get ∆ ( n ) =
max i ∈
[ M ] (cid:110) v ( n )
i
[Equation]
n  
+ H
[Equation]
M M
[Equation]
Hereafter, we take the dominating term out.
Note that
[Equation]
E.2)
[Equation]
(Potential function bound, Lemma E.3 noting ∥
ϕ ⋆h,i ( s [ Z i ] , a i ) ∥ 2 ≤ 1 for any ( s, a )
[Equation]
Similarly, we have
[Equation]
Finally,
[Equation]
n
[Equation] [Equation]
(Some algebra.
We take the dominating term out.
Note that α ( n ) is increasing in n )
This concludes the proof.
D. P ROOF OF THE M AIN T HEOREMS
Lemma D.9.
For the model-based algorithm, when we pick λ = Θ (cid:16) Ld L log NHM | Φ | δ (cid:17) , α ( n ) =
Θ (cid:16) H ˜
A (cid:112) nζ ( n ) + d L λ (cid:17) and ζ ( n ) =
Θ (cid:16) 1 n log |M| HNMδ
[Equation]
, with probability 1 − δ , we have
[Equation]
Proof.
The result of Lemma D.1 implies with our choice of λ and ζ ( n ) , the event E holds with probability at least 1 − δ .
In this case, we have
[Equation]
which is a constant unrelated with n .
Therefore, using the result of Lemma D.8, we get
N (cid:88) n
[Equation]
which has finished the proof.
Proof of Theorem
Proof.
For any fixed episode n and agent i , by Lemma D.5, Lemma D.6 and Lemma D.7, we have v † ,π ( n ) − i
i
[Equation]
Taking maximum over i on both sides, we have
[Equation]
From Lemma B.8, with probability 1 − δ , we can ensure
N (cid:88)
[Equation]
Therefore, according to Lemma E.4, when we pick N to be
[Equation]
we have
N
[Equation]
n
[Equation]
On the other hand, from equation 25, we have
[Equation]
which has finished the proof, noting our assumption that L = O (1) .
E A UXILIARY L EMMAS
Lemma E.1 (Concentration of the bonus term (Zanette et al.
(2021), Lemma 39)) .
Set λ ( n ) ≥ Θ( d log( nH | Φ | /δ )) for any n .
Define Σ n,ρ ( n ) h ,ϕ = n E ( s, a ) ∼ ρ ( n ) h
[ ϕ ( s, a ) ϕ ⊤ ( s, a )] + λ ( n
[Equation]
I d
[Equation]
With probability 1 − δ , we have ∀ n ∈ N + , ∀ h ∈ [ H ] , ∀ ϕ ∈ Φ ,
[Equation]
∥ Σ − 1 ρ ( n ) h
[Equation]
ϕ .
Lemma E.2 (Agarwal et al.
(2020a), Lemma G.2) .
Consider the following process.
For n = 1 , . . .
, N ,
M n = M n − 1 + G n with M 0 = λ
I and G n being a positive semidefinite matrix with eigenvalues upper bounded by 1 .
We have
[Equation]
Lemma E.3 (Potential function lemma) .
Suppose Tr( G n ) ≤ B 2 .
[Equation]
Proof.
Let σ 1 , · · · , σ d be the set of singular values of M N recalling M N is a positive semidefinite matrix.
Then, by the AM-GM inequality,
[Equation]
Since we have (cid:80) i σ i =
Tr( M N ) ≤ dλ 0 + NB 2 , the statement is concluded.
Lemma E.4.
For parameters A, B, ε such that A 2 B ε 2 is larger than some absolute constant, when we pick N =
A
[Equation] [Equation]
O
[Equation] [Equation]
[Equation]
ABε (cid:17) , we have
[Equation]
Proof.
We have
[Equation]
Note that
[Equation]
where the right hand side is always true whenever A 2 B ε 2 is larger than some given constant.
Therefore, we get
[Equation]
F E XPERIMENT D ETAILS F. D ETAILED ENVIRONMENT SETUP
In this section we introduce the details of the environment construction of the Block Markov games.
For completeness we repeat certain details already introduced in the main text.
We design our Block
Markov game by first randomly generating a tabular Markov game with horizon H , 3 states,players each with 3 actions, and random reward matrix R h ∈ (0 , 1) 3 × 3 2 × H and random transition matrix T h ( s h , a h ) ∈ ∆( S h +1 ) .
For the reward generalization, for each r ( s, a, s ′ ) entry in the reward matrix, we assign it with a random number sampled from a uniform distribution from -1 to 1.
For the probability matrix generation, for each conditional distribution T ( ·| s, a ) , we randomly samplenumbers from a uniform distribution from -1 to 1 and form the probability simplex by normalization.
For the generation of rich observation (emission distribution), we follow the experiment design of (Misra et al., 2020): the dimension of the observation is 2 ⌈ log( H + |S| +1)
For an observation o that emitted from state s and time step h , we concatenate the one-hot vector of s and h , adding i.i.d.
Gaussian noise N (0 , 0 . 1) on each entry, pend zero at the end if necessary, and finally multiply with a
Hadamard matrix.
In our setting, we have variants with different horizons H .
F. I MPLEMENTATION DETAILS
For the implementation of G E RL_MG2, we break down the introduction into two parts: the implementation of Alg.
3 and the implementation of game solving algorithm with current features (line.
10 and line.
11 of Algorithm.
For the implementation of Algorithm.
3, we follow the same function approximation as (Zhang et al.
and adapt their open-sourced code at https:// github.com/yudasong/briee .
We include an overview of the function class for completeness: we adopt a two layer neural network with tanh non-linearity as the function class as the discriminator class.
For the decoder, we let ψ ( o ) =
softmax ( A ⊤ o ) , where A ∈ R |O|× 3 , and we let ϕ ( o, a ) =
ψ ( o ) ⊗ a .
Here a denotes the one-hot encoding in the joint action space.
Different from Zhang et al.
(2022), we solve the optimization problem by directly solving the minmax-min problem instead of using an iterative method.
We show the implementation in Algorithm.
We first perform minibatch stochastic gradient descent aggressively on the discriminator selection step (line.
5, on ˆ ϕ and f ) and the feature selection step (line.
6, on ϕ ), where in each step we first compute the linear weight w and ˆ w closed-formly and then perform gradient descent/ascend on the features and discriminators.
Note that here the number of iteration T is very small.
For solving the Markov games, in addition to following Algorithm. 1, to solve line.10 (i.e., solving equation 2 or equation 3 or equation 4), we implement the NE/CCE solvers based on the public repository: https://github.com/quantumiracle/MARS .
Note that the essential difference lies in that (Xie et al., 2020)
assumes that the algorithm has the access to the ground-truth feature but our algorithm needs to utilize the different features we learn for each iteration.
We also adopt the Deep
RL baseline from the same public repository.
F. Z ERO - SUM EXPERIMENT TRAINING CURVES
In this section we provide the training curves of G E RL_MG2 and Deep RL baseline in the zero-sum setting in Figure.
F. G ENERAL - SUM EXPERIMENT DETAILS
In this section we complete the remaining details for the general-sum experiment.
We include the training curve in Fig. 2.
[Figure] [Figure]
F. H YPERPARAMETERS
In this section
[Table]
we include the hyperparameter for G E RL_MG2 in Table.
[Table]
and the hyperparameter for DQN in Table.
3 and Table.
[Table] [Table]
[Table] [Table]
[Table] [Table]
1: Input: Dataset D , step h , regularization λ , iterations T .
2: Denote least squares loss: L λ, D ( ϕ, θ, f ) :=
E D (cid:104)(cid:0) ϕ
( s, a ) ⊤ θ − f ( s ) (cid:1) 2 (cid:105) + λ
∥ θ ∥ 22 .
3: Initialize ϕ 0 ∈
Φ h arbitrarily; 4: for t = 0 , 1 , . .
, T do 5:
Discriminator selection: f t = arg max f ∈F h
[min θ L λ, D ( ϕ t , θ, f ) − min ˜ ϕ ∈ Φ , ˜ θ L λ, D (˜ ϕ, ˜ θ, f )] 6:
Feature selection: ϕ t +1 =
arg min ϕ ∈
Φ h (cid:80) ti =1 min θ i L λ, D ( ϕ, θ i , f i ) , ˆ ϕ ← ϕ t + 7: end for 8: Return ˆ ϕ, ˆ P where ˆ P is calculated from equation 1.
[Table] [Table]
[Table] [Table]
Self-play δ {1.5,2}