T OWARDS G ENERAL F UNCTION A PPROXIMATION IN Z ERO -S UM M ARKOV G AMES
A BSTRACT
This paper considers two-player zero-sum finite-horizon Markov games with simultaneous moves.
The study focuses on the challenging settings where the value function or the model is parameterized by general function classes.
Provably efficient algorithms for both decoupled and coordinated settings are developed.
In the decoupled setting where the agent controls a single player and plays against an arbitrary opponent, we propose a new model-free algorithm.
The sample complexity is governed by the Minimax Eluder dimension—a new dimension of the function class in Markov games.
As a special case, this method improves the state-of-the-art algorithm by a √ d factor in the regret when the reward function and transition kernel are parameterized with d -dimensional linear features.
In the coordinated setting where both players are controlled by the agent, we propose a model-based algorithm and a model-free algorithm.
In the model-based algorithm, we prove that sample complexity can be bounded by a generalization of Witness rank to Markov games.
The model-free algorithm enjoys a √ K -regret upper bound where K is the number of episodes.
I NTRODUCTION
In competitive reinforcement learning, there are two agents competing against each other by taking actions.
Their actions together determine the state evolutions and rewards.
Function approximation, especially deep neural networks (LeCun et al., 2015), contributes to the success of RL in many real world applications, such as Atari (Mnih et al., 2013), Go (Silver et al., 2015), autonomous driving (Shalev-Shwartz et al., 2016), Texas holdem poker (Sandholm, 2017), and Dota (Berner et al., 2019).
The goal of competitive reinforcement learning aims to learn the Nash Equilibrium (NE) policy in a trial-and-error fashion.
In a NE, no agent can be better off by unilaterally deviating from her policy.
Most of the existing sample efficient competitive RL algorithms focus on the tabular or linear function approximation cases (Pérolat et al., 2017; Bai & Jin., 2020; Bai et al., 2021; 2020; Xie et al., 2020;
Zhang et al., 2020).
The huge empirical success of neural network based RL methods has remained largely unexplained.
Thus, there exists a wide gap between theory and practice in competitive RL with general function approximation.
One demanding question to ask is:
Can we establish provably efficient competitive RL algorithms with general function approximation?
In this paper, we make a step towards answering this question by providing structural conditions and complexity measures of Markov Games and function classes that allow for efficient learning.
We focus on episodic zero-sum Markov Game (MG) with simultaneous move, where each episode consists of H time steps and two players act simultaneously at each time step.
The state space is arbitrarily large and can be infinite.
We consider two function approximation settings: 1) use a general function class F to approximate the action-value function (Q function); 2) use a general function class M to approximate the environment.
To this end, we introduce algorithms based on optimistic principles.
Due to subtle game-theoretic issues, naive optimism is no longer working in Markov games where minimax optimization is performed.
To deal with this issue, we use the algorithmic idea called ‘alternate optimism’.
Specifically, we develop algorithms for both coordinated and decoupled setting.
In the decoupled setting, the agent controls one player and plays against an arbitrary and potentially adversarial opponent.
The sample efficiency is measured by the gap between the learned value and value of the Nash Equilibrium.
In the coordinated setting, the agent controls both players and the goal is to find the approximate Nash
Equilibrium, i.e. with small duality gap.
We identify key complexity measures of function classes to examine the effectiveness of elimination.
By doing so, we prove upper bounds on sample complexity and regrets of the presented procedures that are independent of the number of states.
Our contributions are summarized into the following two folds:
• In the decoupled setting, we introduce Minimax Eluder dimension–a new complexity measure for competitive RL problems.
We propose an algorithm that incurs at most (cid:101) O
( H √ d E K log N F ) 1 regret in K episodes where d E denotes the Minimax Eluder dimension N F denotes the covering number of function class, with probability at least 1 − p .
As a special case, this result improves Xie et al.
(2020) by a √ d multiplicative factor in their setting when the reward function and transition kernel are linearly parameterized and d is the dimension of feature mapping.
• In coordinated settings, we propose both model-based and model free algorithms.
In the model-based setting, we generalize the witness rank (Sun et al., 2019a) to competitive RL.
We prove that (cid:101) O
( H 3 W 2 /ϵ 2 ) samples are enough to learn a policy ϵ -close to the Nash Equilibrium, where W is witness rank.
In the model-free setting, we develop algorithm for agnostic learning with a candidate policy class Π .
The algorithm incurs at most (cid:101) O
( H (cid:112)
dK log( N F N Π )) regret.
Here d is a variant of Minimax Eluder dimension and N Π denotes the covering number of policy class.
1. R ELATED WORKS
There is a rich literature studying the learning and decision-making of Markov Games (Littman &
Szepesvari, 1996; Greenwald et al., 2003; Grau-Moya et al., 2018; Pérolat et al., 2018; Srinivasan et al., 2018; Sidford et al., 2020; Wei et al., 2017; Pérolat et al., 2017; Bai & Jin., 2020; Bai et al., 2021; 2020; Zhang et al., 2020; Zhao et al., 2021).
The most related to us are perhaps (Xie et al., 2020;
Chen et al., 2021; Jin et al., 2021b), where the authors address the challenge of explorationexploitation tradeoff in large state spaces.
Due to space constraint, a detailed literature discussion is deferred to Appendix A.
P RELIMINARIES
We consider two-player zero-sum simultaneous-moves episodic Markov game, defined by the tuple ( S , A 1 , A 2 , r, P , H ) , where S is the state space,
A i is a finite set of actions that player i ∈ { 1 , 2 } can take, r is the reward function, P is the transition kernel and H is the number of time steps.
At each time step h ∈
[ H ] , player P1 and P2 take actions a ∈ A 1 and b ∈
A 2 respectively upon observing the state x ∈ S , and then both receive the reward r h ( x, a, b ) .
The system then transitions to a new state x ′ ∼ P h ( ·| x, a, b ) according to the transition kernel P .
Throughout this paper, we assume for simplicity that
A 1 =
A 2 =
A and that the rewards r h ( x, a, b ) are deterministic functions of the tuple ( x, a, b ) taking values in [ − 1 , 1] .
Turn-based games are special cases of simultaneous games in the sense that at each state the reward and transition are independent of one player’s action (Xie et al., 2020).
Generalizations to A 1 ̸ =
A 2 and stochastic reward are also straightforward.
Denote by ∆ :
= ∆( A ) the probability simplex over the action space A .
A stochastic policy of P1 is a length- H sequence of functions π :=
{ π h : S (cid:55)→ ∆ } h ∈
[ H ] .
At each step h ∈
[ H ] and state x ∈ S ,
P1 takes an action sampled from the distribution π h ( x ) over A .
Similarly, a stochastic policy of P is given by the sequence ν :=
{ ν h : S (cid:55)→ ∆ } h ∈
[ H ] .
The learning happens in K episodes.
In each episode k , each player P1 and P2 proposes a policy π k and ν k respectively based on history up to the end of episode k − 1 and then executes the policy to observe the trajectories { x kh , a kh , b kh } Hh =1 .
For a fixed policy pair ( π, ν ) , the value function and Q functions for zero-sum game is defined by
[Equation]
where the expectation is over a h ∼ π h ( x h ) , b h ∼ ν h ( x h ) and x h +1 ∼ P h ( ·| x h , a h , b h ) .
V π,νand
Q π,νare often abbreviated to V π,ν and Q π,ν .
In zero-sum games, PI aims to maximize V π,ν ( x 1 ) and P2 aims to minimize V π,ν ( x 1 ) .
Given policy π of P1, the best response policy of P2 is defined by ν ∗ π := argmin ν V π,ν ( x 1 ) .
Similarly given policy ν of P2, the best response policy of P1 is defined by π ∗ ν := argmax π V π,
ν ( x 1 ) .
Then from definitions,
[Equation]
and the equality holds for the Nash Equilibrium (NE) of the game ( π ∗ , ν ∗ ) .
We abbreviate V π ∗ ,ν ∗ ( x 1 ) as V ∗ and Q π ∗ ,ν ∗ ( x 1 ) as Q ∗ .
V ∗ is often referred to as the value of the game.
As common in Markov games literature (Pérolat et al., 2015), we will use the following Bellman operator
[Equation]
and the following Bellman operator for fixed P1’s policy
[Equation]
Notations For a integer H , [ H ] denotes the set { 1 , 2 , . .
, H } .
For a finite set S , |S| denotes its cardinality.
For a matrix A ∈ R d , A i, ∗ and A ∗ ,j denote the i -th row and j -th column of A respectively.
For a function f : S (cid:55)→ R , ∥ f ∥ ∞ denotes sup s ∈S | f ( s ) | .
We use N ( F , ϵ ) to denote the ϵ -covering number of F under metric d ( f, g ) = max h ∥
f h − g h ∥ ∞ .
D ECOUPLED SETTING
In decoupled setting, the algorithm controls P1 to play against P2.
In k -th episode, P1 chooses a policy π k and P2 chooses a policy ν k based on π k .
Then P1 and P2 interact in the Markov Game by playing π k and ν k respectively, and observe the rewards and states in this episode.
Notice that P can be adversarial and its policy is never revealed, meaning that P1 can only maximize its reward by playing π ∗ .
The goal is to thus minimize the following ‘value regret’
[Equation]
A small value regret indicates that we learn the value of MG.
If an algorithm obtains an upper bound on value regret that scales sublinearly with K for all ν k , then it can not be exploited by opponents.
3. F UNCTION APPROXIMATION
We consider function class F = F 1 × · · · × F H where F h ⊂ { f : S ×
A 1 ×
A 2 (cid:55)→ [0 , 1] } to provide function approximations for Q ∗ — the Q-function of Nash Equilibrium.
For a policy pair ( π, ν ) and function f we abuse the notations and use f ( x, π, ν ) =
E a ∼ π,b ∼ ν
[ f ( x, a, b )] to denote the expected reward of executing ( π, ν ) in function f .
We make the following assumptions about F .
Assumption 3.1 (Realizability) .
We assume Q ∗ h ∈ F h for all h ∈
[ H ] .
Assumption 3.2 (Completeness) .
We assume for all h ∈
[ H ] and f ∈ F h +1 , T h f ∈ F h .
Realizability means the function class contains the target Q function.
Completeness says the function class is closed under the Bellman operator.
These assumptions are common in value function approximation literature (e.g. Jin et al.
; Wang et al.
(2020b)
; Jin et al.
(2021a)).
Note that they are weaker than Xie et al.
(2020), as their setting satisfies T h f ∈ F h for any action-value function f .
For a function f ∈ F , we denote the max-min policy of P1 by π f .
Specifically,
[Equation]
Similarly we denote the min-max policy of P2 by ν f .
Given policy π of P1, the best response policy of P2 in terms of function f is defined by
[Equation]
Similarly given policy ν of P2, the best response policy of P1 in terms of function f is denoted by π fν .
When there is no confusion, we will drop the subscript h for simplicity.
Learnability requires the function class to have bounded complexity.
We introduce Minimax Eluder dimension to capture the structural complexity of function classes in zero-sum games.
To define, we use ϵ -independence of distributions and Distributional Eluder dimension (DE) from Russo & Van Roy (2013); Jin et al.
(2021a).
Definition 3.3 ( ϵ -independence of distributions) .
Let G be a function class defined on X , and ν, µ 1 , . . .
, µ n be probability measures over X .
We say that ν is ϵ -independent of { µ 1 , . . . , µ n } with respect to G if there exists g ∈ G such that (cid:112)(cid:80) ni =
( E µ i [ g ]) 2 ≤ ϵ but | E ν [ g ] | ≥ ϵ .
Definition 3.4 (Distributional Eluder dimension (DE)) .
Let G be a function class defined on X , and
D be a family of probability measures over X .
The distributional Eluder dimension dim DE ( G , D , ϵ ) is the length of the longest sequence {
ρ 1 , . . . , ρ n } ⊂ D such that there exists ϵ ′ ≥ ϵ where ρ i is ϵ ′ -independent of { ρ 1 , . .
, ρ i − 1 } for all i ∈ [ n ] .
Now we introduce Minimax Eluder dimension.
Recall the minimax Bellman operator from Eq (1):
[Equation]
thus Minimax Eluder dimension is the Eluder dimension on the Bellman residues with respect to the above minimax Bellman operator.
Definition 3.5 (Minimax Eluder dimension) .
Let ( I − T h ) F :=
{ f h − T h f h +1 : f ∈ F} be the class of minimax residuals induced by function class F at level h and D ∆ =
{D ∆ ,h } h ∈
[ H ] where D ∆ ,h :=
{ δ ( x, a, b ) : x ∈ S , a ∈
A 1 , b ∈
A 2 } is the set of Dirac measures on state-action pair ( x, a, b ) .
The ϵMinimax Eluder dimension of F is defined by dim ME ( F , ϵ ) :
= max h ∈
[ H ] dim DE (( I −T h ) F , D ∆ , ϵ ) .
3. O PTIMISTIC N ASH E LIMINATION FOR M ARKOV G AMES
Now we introduce Optimistic Nash Elimination for Markov Games (ONEMG), presented in Algorithm 1.
This algorithm maintains a confidence set V k that always contains the Q ∗ , and sequentially eliminates inaccurate hypothesis from it.
In k -th episode, P1 first optimally chooses a value function f k in V k − 1 that maximizes the value of the game.
Intuitively, this step performs optimistic planning in the pessimistic scenarios, i.e. assuming P2 plays the best response.
Next, it plays against P2 and augments the data from this episode into replay buffer B .
The confidence set V k is then updated by keeping the functions f with small minimax Bellman errors f h − T
h
f h
+1 , in Line 12.
To estimate f h − T h f h
+1 , we use E B h ( f h , f h +1 )
− inf g ∈F h E B h ( g, f h +1 ) , a standard variance reduction trick to avoid the double-sample issue.
It can be shown that when the value function class is complete,
E B h ( f h , f h +1 ) − inf g ∈F h E B h ( g, f h +1 ) is an unbiased estimator of f h − T
h f h
+1 (Lemma B.1).
Notice that unlike many previous works that add optimistic bonus on every state-action pairs (Xie et al., 2020; Chen et al., 2021), Algorithm 1 only takes optimism in the initial state.
Instead, the constraint set contains every function that has low Bellman residue on all trajectories in the replay buffer.
This ‘global optimism’ technique trades computational efficiency for better sample complexity, and can be found in many previous work in Markov Decision process (Zanette et al., 2020; Jin et al., 2021a).
As we will see later, it is also useful in the coordinated setting.
1: Input: Function class F 2: Initialize
F 3: Set β ← C log( N F
1 /K ) · HK/p ) for constant C 4: for k
= 1 ,
, K do 5:
Compute f k ← argmax f ∈V k − 1 f 1 ( x 1 , π f , ν f ) , let π k ← π f k 6: for step h = 1 , 2 , . .
, H do 7:
Interact with environment by playing action a kh ∼ π kh ( x kh ) 8:
Observe reward r kh , opponent’s action b kh and move to next state x kh + 9: end for 10:
Update B h ← B h ∪ { ( x kh , a kh , b kh , r kh , x kh +1 ) } , ∀ h ∈
[ H ] 11:
For all ξ, ζ ∈ F and h ∈ [ H ] , let
E B h ( ξ h , ζ h +1 ) =
k (cid:88) τ
=
[ ξ h ( x τh , a τh , b τh ) − r τh − ζ h +1 ( x τh +1 , π ζ h +1 , ν ζ h +1 )]12:
Set
V k ← { f ∈ F : E B h ( f h , f h +1 ) ≤ inf g ∈F h E B h ( g, f h +1 )
+ β,
∀ h ∈
[ H ] } 13: end for
3. T HEORETICAL RESULTS
In this section we present the theoretical guarantee for Algorithm 1.
The proof is deferred to
Appendix B
Theorem 3.6.
Under Assumption 3.1 and Assumption 3.2, the regret Eq (3) of Algorithm 1 is upper bounded by
[Equation]
with probability at least 1 − p .
Here d ME = dim ME ( F , (cid:112) 1 /K ) and ζ = N ( F , 1 /K ) /p .
As this theorem suggests, as long as the function class has finite Minimax Eluder dimension and covering number, P1 can achieve √ K -regret against P2.
Notice that when P2 always plays the best response of P1’s policies, this regret guarantee indicates that algorithm eventually learns the Nash
Equilibrium.
In this case, the algorithm takes (cid:101) O
( H 2 d log( |F| ) /ϵ 2 ) samples to learn a policy π that is ϵ -close to π ∗ .
When the opponent plays dummy actions, the Markov Game can be seen as an MDP and the value of the Markov Game is the maximum cumulative sum of rewards one can achieve in this MDP.
In this case, our result reduces to (cid:101) O
( H · (cid:112) K · d BE log( N ( F , 1 /K ))) where d BE is the Bellman eluder dimension in Jin et al.
(2021a).
In particular, when the function class is finite, the regret becomes O
( H (cid:112)
Kd ME log( HK |F| /p ))
that depends logarithmically on the cardinality of F .
Moreover, when the reward and transition kernel have linear structures, i.e. r h ( x, a, b )
=
ϕ
( x, a, b ) ⊤ θ h and P h ( ·| x, a, b ) =
ϕ
( x, a, b ) ⊤ µ h ( · )
where ϕ ( x, a, b ) ∈ R d , then d ME ( F , (cid:112) 1 /K )
= log N ( F , 1 /K )
= (cid:101) O
( d ) and the regret becomes (cid:101) O (cid:0) Hd √ K (cid:1) 2 .
Thus we improve Xie et al.
(2020) by a √ d factor.
We also provide a simpler algorithm tailored for this setting, see Appendix B.1 for details.
Algorithm 1 solves a non-convex optimization problem in Line 5 with highly non-convex constraint set (Line 12).
In general, it is computationally inefficient.
Even when reduced to linear MDP, i.e. the opponent plays dummy actions and the transition matrices and the rewards have low rank structures, it is not known if the same (cid:101) O
( H √ d 2 K ) regret can be achieved with computationally efficient algorithms (Zanette et al., 2020).
Designing computationally efficient algorithms with near-optimal sample efficiency is an interesting further direction.
C OORDINATED SETTING
In the coordinated setting, the agent can control both P1 and P2.
The goal is to find the ϵ -approximate
Nash equilibrium ( π, ν ) in the sense that
[Equation]
by playing P1 and P2 against each other in the Markov game.
In the following sections, we propose both model-based and model-free methods to deal with this problem.
4. M ODEL - BASED ALGORITHM
In model-based methods, we make use of a model class M of candidate models to approximate the true model M ∗ .
We are also given a test function class G to track model misfit.
Given a model M ∈ M , we use Q M ( x, a, b ) , r M ( x, a, b ) and P M ( ·| x, a, b ) to denote the Q-function, reward function and transition kernel of executing action a and b at state x in model M .
For simplicity, we use ( r, x ) ∼ M to represent r ∼ r M ( x, a, b ) and x ∼ P M ( ·| x, a, b ) .
We denote the NE policy in model M as π M and ν M .
For policy π, ν and model M , we use ν Mπ to denote the best response of π in model M and
π Mν is defined similarly.
4.1. A LTERNATE O PTIMISTIC M ODEL E LIMINATION (AOME)
The model-based method, Alternate Optimistic Model Elimination (AOME), is presented in Algorithm 4.
It maintains a constraint set M k of candidate models.
Throughout K iterations, AOME makes sure that the true model M ∗ always belongs M k for all k ∈
[ K ] , and sequentially eliminates incorrect models from M k .
However, the idea of being optimistic only at initial state (Zanette et al., 2020; Jin et al., 2021a) is not directly applicable to this scenario.
Since the objective to be optimized is the duality gap, the target policies being considered are thus policies pairs ( π ∗ ν , ν ) and ( π, ν ∗ π ) .
However, π ∗ ν and ν ∗ π are not available to the agents.
Therefore, the performances of proposed policies are evaluated on out-of-distribution trajectories, meaning that optimism can not be guaranteed.
This causes the distribution shift issue.
In worst cases, low Bellman errors in the collected trajectories provide no information for the trajectories collected by π ∗ ν , ν and π, ν ∗ π .
To address this issue, the algorithm performs global planning by applying the idea of alternate optimism to model-based methods.
In k -th episode, the algorithm first solves the optimization problem M k 1 = arg max M ∈M Q M ( x 1 , π M , ν M ) and let π k = π M k 1 .
This optimization problem corresponds to finding the optimistic estimate of the value of the game.
Indeed, notice that M ∗ ∈ M k implies Q M k 1 ( x 1 , π k , ν k ) ≥ Q M k 1 ( x 1 , π k , ν M k 1 π k ) ≥ V ∗ ( x 1 ) , by optimality of M k 1 .
Next, it solves the second optimization problem M k
2 = arg min M ∈M Q M ( x 1 , π M k 1 , ν M ) and let ν k ← ν M k 2 π k .
This corresponds to finding the pessimistic estimate of V π k ,ν ∗ πk ( x 1 ) .
In fact, we see that M ∗ ∈ M k implies Q M k 2 ( x 1 , π k , ν k ) ≤ V π k ,ν ∗ πk ( x 1 ) , by optimality of M k 2 .
This approach appears in Wei et al.
(2017) to guarantee convergence to Nash equilibrium, where it is referred to as ‘Maximin-EVI’.
In Line 8, the algorithm checks if the values of model M k 1 and M k 2 are close to the value of the true model M ∗ when executing policy π k and ν k .
If the condition holds, we have
[Equation]
which means that AOME finds policy π k that is ϵ -close the NE.
One can also switch the order of alternate optimism and obtain ν k that is ϵ -close
the NE.
We then terminate the process and output π k and ν k .
If the algorithm does not terminate in Line 8, it applies the witnessed model misfit checking method in Sun et al. (2019a).
It starts by computing the empirical Bellman error of Markov games defined as follows
[Equation]
[Equation] As proved in Appendix D, in this case (cid:98) L ( M k 1 , M k 2 , M kj , h k ) must be greater than ϵ/ (8 H ) for some j ∈ [2] and h k ∈
[ H ] .
Then the algorithm samples additional trajectories at level h k and shrinks the constraint set by eliminating models with large empirical model misfit, which is defined as follow (cid:98) E ( [Equation]
[Equation]
We will show that this constraint set maintains more and more accurate models of the environment.
Due to space constraints, the algorithm and theory of our model-based method are deferred to
Appendix C.
4. M ODEL FREE ALGORITHM
In this section we propose a model-free algorithm.
We consider a more general agnostic learning, where The algorithm is given a policy class Π = Π 1 × · · · Π H with Π h ⊂ { π h : S (cid:55)→ ∆ } , h ∈ [ H ] .
Notice by letting Π to be the class of NE policies induced by the action-value function class F , we recover the original setup.
The goal is to find policy π and ν from Π to minimize the duality gap: max π ′ ∈
Π V π ′ ,ν − min ν
′ ∈
Π V π,ν ′ , by playing only policies from Π .
Since only policies from Π are considered, we overload the notation and define the optimal solution in Π as π ∗ =
arg max π ∈ Π min ν ′ ∈
Π V π,ν ′ and ν ∗ =
arg min ν ∈ Π max π ′ ∈
Π V π ′ ,ν .
When Π contains all possible policies, π ∗ and ν ∗ is then the Nash equilibrium.
Similar to Section 3, we consider general function class F = F 1 × · · · × F H where F h ⊂ { f : S ×
A 1 ×
A 2 (cid:55)→ [0 , 1] } .
We overload the notations in Section 3 and use ν fπ to denote the best response of π from Π , namely ν fπ ( x ) :=
argmin ν ∈ Π f ( x, π, ν ) .
Similarly, we use π f ( x ) :
= argmax π ∈ Π min ν ∈ Π
f ( x, π, ν ) .
We also use ν ∗ π ( x ) := argmin ν ∈ Π V π,ν to denote the best response of π in the true model.
4.2. A LTERNATE O PTIMISTIC V ALUE E LIMINATION (AOVE)
Notice that the duality gap can be decomposed as follows
[Equation]
where the optimal part V π ∗ ν ∗ ,ν ∗ − V π ∗ ,ν ∗ π ∗ is fixed.
Therefore we can consider only the P1 part here and the P2 follows by symmetry.
We are interested in the following policy regret
[Equation]
Policy regret measures the extent that P1 can be exploited by policies in Π .
Our algorithm Alternate
Optimistic Value Elimination (AOVE), presented in Algorithm 2, works on minimizing this regret.
Similar to Algorithm 1, it also uses optimism in the initial state to address exploration and exploitation trade-off and constructs a series of confidence sets of hypotheses with small Bellman errors.
The differences primarily lie in a different choice of optimism and a different series of confidence sets constructed with the Bellman operator in Eq 2.
Since the policy class Π is taken into consideration, the algorithm maintains a constraint set B ⊂ Π × F of policy-function pairs.
We use the Bellman operator with regard to fixed P1’s policy, in Eq (2).
Thus intuitively, this constraint set maintains ( π, f ) pairs such that f h − T πh f h
+1 is small for all h ∈
[ H ] .
We apply the ‘alternate optimistic’ principle, presented in Line 5 and Line 6.
Intuitively, the algorithm finds an optimistic planning of π and pessimistic planning of ν , together corresponding to a max-min procedure.
As such, we form an upper bound f 1 ( x 1 , π, ν fπ ) − g 1 ( x 1 , π k , ν gπ k ) of the duality gap.
Note that it is different from the ‘alternate optimistic’ used in model-based settings Wei et al.
(2017) in that the hypothesis chosen in Line 6 is constrained by the policy chosen in Line 5.
This is because in model-based methods, ‘plausible’ models in the confidence set guarantee small simulation errors (Lemma C.6).
However, this is not true for ‘plausible’ value functions in the confidence set.
The rest of the algorithm aims at minimizing this upper bound sequentially by eliminating bad hypothesis in the constraint set.
The elimination process occurs in Line 13, where the algorithm uses history data to eliminate functions with large Bellman error.
To estimate Bellman error, we use the similar method as in Algorithm 1, i.e. subtracting the variance of T πh f h +1 by g = arg inf g ∈F h E D h ( g, f h +1 , π ) .
Notice that a smaller value of f h −T πh f h +1 indicates that function approximation f is not easily exploited by P2 when P1 plays π .
As shown in Appendix D, ( π ∗ , Q ∗ ) is always in the constraint set, and the Bellman errors of the rest hypothesis keep shrinking.
Thus in the presented algorithm of AOVE, the regret of P1 part
V π ∗ ,ν ∗ π ∗ − V π,ν ∗ π is controlled.
By symmetry, we can perform the same procedures on P2 part and obtain an upper bound on V π ∗ ν ,ν − V π ∗ ν ∗ ,ν ∗ .
Combining these together, we show sublinear regret rate of V π ∗ ν ∗ ,ν ∗ − V π ∗ ,ν ∗ π ∗ .
By regret to PAC conversion, this means that we find the policies that is the best achievable in Π .
Algorithm 1: Input: Function class F , policy class Π 2: Initialize B 0 ← Π × F 3: Set β ← C log[ N ( F , 1 /K ) ·
N (Π , 1 /K ) · HK/p ] for some large constant C 4: for k = 1 , 2 , . .
, K do 5:
Find ( π k , f k ) ← argmax ( π,f ) ∈V k − 1 f 1 ( x 1 , π, ν fπ ) 6:
Let g ← argmin g :( π k ,g ) ∈V k − 1 g 1 ( x 1 , π k , ν gπ k ) and set ν k ← ν gπ k 7: for step h = 1 , 2 , . .
, H do 8:
P1 takes action a kh ∼ π kh ( x kh ) , P2 takes action b kh ∼ ν kh ( x kh ) 9:
Observe reward r kh and move to next state x kh + 10: end for 11:
Update B h ← B h ∪ { ( x kh , a kh , b kh , r kh , x kh +1 ) } , ∀ h ∈
[ H ] 12:
For all ξ, ζ ∈ F , π ∈ Π and h ∈ [ H ] , let
E B h ( ξ h , ζ h +1 , π ) =
k (cid:88) τ
=
[ ξ h ( x τh , a τh , b τh ) − r τh − ζ h +1 ( x τh +1 , π, ν ζ h +1 π )]13:
Update
V k ← { ( π, f ) ∈
Π × F : E B h ( f h , f h
+1 , π ) ≤ inf g ∈F h E B h ( g, f h
+1 , π )
+ β,
∀ h ∈
[ H ] } 14: end for
4.2. T HEORETICAL RESULTS
This section presents our theoretical guarantees of Algorithm 2.
First, we introduce two assumptions that take the policy class Π into consideration.
Variants of these assumptions are also common in batch RL (Jin et al., 2021c; Xie et al., 2021).
Assumption 4.1 ( Π -realizability) .
For all π ∈ Π , Q π,ν ∗ π ∈ F .
Assumption 4.2 ( Π -completeness) .
For all h ∈ [ H ] , π ∈ Π , and f ∈ F h +1 , T πh f ∈ F h holds.
Notice that Assumption 4.2 reduces to Q ∗ realizability in MDP.
Both two assumptions hold for linear-parameterized Markov games studied in Xie et al.
In fact, Xie et al.
(2020) satisfies a stronger property: for any policy pair π,
ν and any h ∈
[ H ] , there exists a vector w ∈ R d such that
[Equation]
Now we present the theory of Algorithm 2.
First, we introduce the complexity measure in coordinated setting, which is a variant of Minimax Eluder dimension by replacing the Bellman operator Eq (1) with Eq (2).
This Minimax Eluder dimension also allows a distribution family induced by function class F which is cheap to evaluate in large state space (Jin et al., 2021a).
Definition 4.3 (Minimax Eluder dimension in Coordinated setting) .
Let ( I − T Π h +1 h ) F :=
{ f h − T πh f h +1 : f ∈ F , π ∈ Π h +1 } be the class of residuals induced by the Bellman operator T πh .
Consider the following distribution families: 1) D ∆ = {D ∆ ,h } h ∈ [ H ] where
D ∆ ,h := { δ ( x, a, b ) : x ∈ S , a ∈
A 1 , b ∈
A 2 } is the set of Dirac measures on state-action pair ( x, a, b ) ; 2) D F = {D F ,h } h ∈
[ H ] which is generated by executing ( π f , ν gπ f ) for f, g ∈ F .
The ϵ -Minimax Eluder dimension (in coordinated setting) of F is defined by dim ME ′ ( F , ϵ ) :
= max h ∈ [ H ] min D∈{D ∆ , D F } dim DE (( I − T Π h +1 h ) F , D , ϵ ) .
Now we present our main theory of Algorithm 2, where we make use of a variant of Minimax Eluder dimension that depends on policy class Π .
We use N (Π , ϵ ) to denote the covering number of Π under metric d ( π, π ′ ) :
= max h ∈ [ H ] max f ∈F h ,x ∈S ,b ∈A | f ( x, π, b ) − f ( x, π ′ , b ) | .
Theorem 4.4.
Suppose Assumption 4.1 and Assumption 4.2 holds.
With probability at least 1 − p we have the regret (Eq.
(7) ) of Algorithm 2 upper bounded by
[Equation]
where d ME ′ = dim ME ′ ( F , (cid:112) 1 /K ) and ζ = N ( F , 1 /K )
N (Π , 1 /K ) /p .
As shown in this theorem, the regret is sub-linear in number of episodes K and Eluder dimension d ME ′ which match the regret bound in Markov decision process, e.g. in Jin et al.
(2021a).
Through regret-to-PAC conversion, with high probability the algorithm can find ϵ -optimal policy with O (cid:0) H 2 Kd ME ′ log[ N ( F , 1 /K )
N (Π , 1 /K )
HK/p ] /ϵ 2 (cid:1) sample complexity.
When the reward and transition kernel have linear structures, we also recover the (cid:101) O ( √ d 3 K ) regret of Xie et al.
Although our upper bound depends on the logarithm of covering number of Π , there are some cases when it is small.
For example, when the value-function class F is finite and Π is the induced policy class of F as defined by Π
= { π f , ν gπ f : f, g ∈ F} , then log N Π ( ϵ ) = log |F| and the regret is (cid:101) O
( H (cid:113)
Kd ME ′ log 2 |F| ) .
Similarly, if the model class M is finite and Π is induced policy class of M as defined by Π
= { π M , ν M ′ π M : M, M ′ ∈ M} , then log N Π ( ϵ )
=
log |M| and the regret is (cid:101)
O
( H (cid:113)
Kd ME ′ log 2 |M| ) .
In agnostic setting which allows Π to have arbitrary structure, the term log N (Π , 1 /K ) reflects the difficulty of learning with complex candidate policy sets and we conjecture that this term is inevitable.
Similar to Algorithm 1, Algorithm 2 solves a non-convex optimization problem in Line 5-6 with highly non-convex constraint set (Line 13).
This step is particularly difficult when the function class has complex structures.
Algorithm 2 is in general computationally inefficient, even in cases of linear function approximations.
Indeed, even when the model is known, solving for the NE is PPAD complete.
Developing decentralized and provably efficient RL algorithm for multi-agent Markov game seems a challenging but interesting future direction.
D ISCUSSION
In this paper we study function approximations in zero-sum simultaneous move Markov games.
We design sample efficient algorithms for both decoupled and coordinated learning settings and propose
Minimax Eluder dimension to characterize the complexity of decision making with value function approximation.
The analysis shows √ T regret bounds and half-order dependence on the Minimax
Eluder dimension, matching the results in MDPs and improving the linear function approximations.
Our algorithms for coordinated setting are based on the idea of the ‘alternate optimism’, which also shows applicability in model-based methods.
A CKNOWLEDGEMENTS
BH is supported by the Elite Undergraduate Training Program of School of Mathematical Sciences at Peking University.
JDL acknowledges support of the ARO under MURI Award W911NF-11-10304, the Sloan Research Fellowship, NSF CCF 2002272, NSF IIS 2107304, and an ONR Young
Investigator Award.
A A DDITIONAL RELATED WORKS
There is a rich literature on sample complexity for single agent RL, mostly focusing on the tabular cases (Strehl et al., 2006; Jaksch et al., 2010; Azar et al., 2017; Jin et al., 2018; Russo, 2019;
Zanette & Brunskill, 2019) and the linear function approximations (Yang & Wang, 2020; Wang et al., 2021; Abbasi-Yadkori et al., 2019; Jin et al., 2020; Du et al., 2019b).
For neural network function approximations, Agarwal et al.
; Liu et al.
; Wang et al.
(2020a)
; Yang et al.
(2020) builds provably efficient algorithms on over-parameterized models.
The issue is complicated for general function approximation, as the Q function is hard to learn due to the double-sample issue (Baird, 1995).
In recent years, a line of work studies the structural properties and corresponding complexity measures that allow generalization in RL.
These work can be roughly grouped with two catagories.
One studies structures with low information gain.
Jiang et al.
(2017) proposes Bellman rank and designs a sample efficient algorithm for a large family of MDPs.
Later, Sun et al.
(2019b) proposes
Witness rank and uses this notion to provide a provably efficient model-based RL algorithm.
Recently,
Du et al.
(2021) proposes Bilinear family and the corresponding Bilinear rank that encapsulates a wide variety of structural properties, spanning from both model-based to model-free function approximations.
The other studies structures with low Eluder dimension (Russo & Van Roy, 2013).
Osband & Van Roy (2014) extends Eluder dimension to reinforcement learning and provides a unified analysis of model based methods where the regret is controlled by the Kolmogorov dimension and
Eluder dimension of model function class.
Wang et al.
(2020b) designs a model-free algorithm and shows that regret can be controlled by the Eluder dimension of the value function class.
Recently, Jin et al.
(2021a) proposes Bellman Eluder dimension and builds a new framework to capture a large amount of structures.
It is noted that although lots of structures, such as linear MDP (Jin et al., 2020), fit in both two viewpoints, they are not equivalent.
Among the above work, Jin et al.
(2021a)
; Zanette et al.
(2020) are more related.
The method of placing uncertainty quantification on only initial states, previously appeared in Zanette et al.
(2020); Jin et al.
(2021a), greatly simplifies the argument and improves sample complexity.
The Minimax Eluder dimension is inspired by the Bellman Eluder dimension in Jin et al.
(2021a).
In competitive RL, the setting with access to a sampling oracle or well explored policies is well studied (Littman & Szepesvari, 1996; Greenwald et al., 2003; Grau-Moya et al., 2018; Pérolat et al., 2018; Srinivasan et al., 2018; Sidford et al., 2020).
Under these assumptions, many works consider function approximations (Lagoudakis & Parr, 2002; Pérolat et al., 2015;
Fan et al., 2020; Jia et al., 2019;
Zhao et al., 2021).
However, without strong sampling model or a well explored policy, the issue of exploration-exploitation tradeoff must be addressed.
Most of these work focus on tabular setting (Wei et al., 2017; Pérolat et al., 2017; Bai & Jin., 2020; Bai et al., 2021; 2020; Zhang et al., 2020; Zhao et al., 2021) or linear function approximation settings (Xie et al., 2020; Chen et al., 2021).
Among them, (Wei et al., 2017; Xie et al., 2020) are more related.
The regret-decomposition method used in the decoupled setting is inspired by Xie et al.
The ‘alternate optimism’ used in Algorithmwas previously used as ‘Maximin-EVI’ in Wei et al. (2017), although the ‘alternate optimism’ used in Algorithm 2 is different.
Moreover, the concurrent work of Jin et al.
(2021b)
studies competitive
RL with general function approximation.
They reach a sublinear (cid:101) O
( √ K ) regret for MGs with low minimax Eluder dimensions in the decoupled setting.
Their results in the coordinated setting are based on different assumptions and complexity measures.
Specifically, they assume finite function class and an additional class of exploiters.
It is noted that by letting the policy class be the class of optimal policies induced by the value function class, we have | Π | = |F| and our algorithm achieves on-par guarantee in their setup in terms of the covering numbers of function classes.
In addition, we propose a first provably sample efficient and model-based algorithm for the coordinated setting with general function approximation.
A. T ECHNICAL CHALLENGES
Previous work (Xie et al., 2020) imposes optimistic bonus on the action-value functions in every stateaction pairs and performs planning by the Coarse Correlated Equilibrium (CCE) on the optimistic value functions.
To achieve improved rates, we leverage the idea of ‘global optimism’ (Zanette et al., 2020; Jin et al., 2021a; Du et al., 2021), which maintains a constraint set of candidate functions that do not deviate much from the empirical estimates and performs optimistic planning on the initial state. However, going beyond MDPs towards MGs, two problems arise. First, the concentration property of functions in constraint set is hard to characterize due to multi-agent interplay. For this, we use the concentration methods in Jin et al.
(2021a)
and extend it from MDPs to MGs.
The second and more prominent issue is the exploration and exploitation tradeoff.
Since ‘global optimism’ only obtains optimism along the trajectories of behaviour policies, it may not guarantee optimism on the trajectories of target policies (i.e. NE).
As a result, directly using CCE to plan will cause the duality gaps to diverge.
To deal with this problem, we apply ‘alternate optimism’ to guide explorations, which was previously used in Wei et al.
(2017) for model-based methods.
The ‘alternate optimism’ used in this work is slightly different for value-based methods.
We prove two regret decomposition lemmata to support this optimism principle.
B P ROOF OF T HEOREM 3.
We prove the Theorem 3.6 in the following five steps.
In the first step, we show that Bellman error is small for function in the constraint set.
In the second step, we show that Q ∗ stays in the constraint set throughout the algorithm.
In the third step, we decompose the regret into a summation of Bellman error and Martingale difference.
We can then bound the summation of Bellman error using Minimax
Eluder dimension in step 4.
Finally we combine the aforementioned steps and complete the proof of
Theorem 3.6.
We define ‘optimistic’ value function in step h and episode k as follows
[Equation]
where (cid:98) ν
kh is the best response of π kh in value function f k , i.e., (cid:98) ν
kh := argmin ν f kh ( x kh , π kh , ν ) .
We use ν k
= { ν kh } Hh =1 to denote the policy adopted by P2 in the k -th episode.
Notice that the agent obtains knowledge from ν only through its actions b kh , h ∈
[ H ] .
Step 1: Low Bellman error in the constraint set We first use a standard concentration procedure (Agarwal et al., 2012; Jin et al., 2021a) to bound the Bellman error in constraint sets.
Lemma B.1 (Concentration on Bellman error) .
Let ρ > 0 be an arbitrary fixed number.
With probability at least 1 − p for all ( k, h ) ∈ [ K ] ×
[ H ] we have
[Equation]
Proof.
Consider a fixed ( k, h, f ) sample, let
[Equation]
and F t,h be the filtration induced by { x i 1 , a i 1 , b
i 1 , r i 1 , . . .
, x iH } t − 1 i =1 ∪ { x t [Equation] , a [Equation]
1 , b
[Equation]
, r
[Equation]
, x th , a th , b th } .
Recall the minimax Bellman operator
[Equation]
We have
E [ U t ( h, f ) | F t,h ] =
[( f h − T h f h +1 )
( x th , a th , b th )] · E [ f h ( x th , a th , b th ) + T h f h +1 ( x th , a th , b th ) − 2 r th − 2 min ν ′ max
π ′ f h
{++
+}
( x th +1 , π ′ , ν ′ )
| F t,h ] =
[( f h − T h f h +1 )
( x th , a th , b th )]and
Var[ U t ( h, f ) | F t,h ] ≤ E [( U t ( h, f )) 2 | F t,h ] =
[( f h − T h f h +1 )
( x th , a th , b th )]· E [ (cid:0) f h ( x th , a th , b th ) + T h f h +1 ( x th , a th , b th ) − 2 r th − 2 min ν ′ max
π ′ f
h
{++
+}
( x th +1 , π ′ , ν ′ ) (cid:1) 2 | F t,h ] ≤ 36[( f h − T h f h
+1 )
( x th , a th , b th )] 2 = 36 E [ U t ( h, f ) | F t,h ] .
By we
[Equation] t
[Equation]
Let N ( F , ρ ) be a ρ -cover of F .
Taking a union bound for all ( k, h, g ) ∈ [ K ] ×
[ H ] × N ( F , ρ ) , then with probability at least 1 − p for all ( k, h, g ) ∈
[ K ] ×
[ H ] × N ( F , ρ )
[Equation]
(8) where ι = log( HK | N ( F , ρ ) | /p ) .
Let g
k = argmin g ∈ N ( F ,ρ ) max h ∈ [ H ] ∥
f
kh − g kh ∥ ∞ .
We immediately have
[Equation]
For all ( h, k ) ∈
[ H ] ×
[ K ] , by the definition of V k and f k ∈ V k
we have k (cid:88) t =
U t ( h, f k ) =
k (cid:88) t =
[Equation]
−
[Equation]
≤
[Equation]
=
[Equation]
max
[Equation]
−
[Equation]
t
[Equation]
min
[Equation]
max
[Equation]
f
[Equation]
ν
[Equation]
[Equation]
It thus follows that,
[Equation]
where the first step is due to g k is an ρ -approximation to f k , the second step comes from Eq (9), the third step is due to g k is an ρ -approximation to f k , and the final step comes from Eq (10).
By definition of π k and (cid:98) ν
k , we have
[Equation]
thus we conclude the proof.
Step 2: Q ∗ is always in constraint set.
Lemma B.2.
With probability at least 1 − p , we have Q ∗ ∈ V k for all k ∈
[ K ] .
Proof.
Consider a fixed ( k, h, f ) sample, let
[Equation]
and F t,h be the filtration induced by { x i 1 , a i 1 , b
i 1 , r i 1 , . . .
, x iH } t − 1 i =1 ∪ { x t 1 , a t 1 , b t 1 , r t 1 , . . .
, x th , a th , b th } .
We have
E [ U t ( h, f ) | F t,h ] =
[( f h − Q ∗ h )( x th , a th , b th )]and
Var[ U t ( h, f ) | F t,h ] ≤ 36[( f kh − T h
f kh +1 )
( x th , a th , b th )] 2 = 36 E [ U t ( h, f ) | F t,h ] .
Let N ( F , ρ ) be a ρ -cover of F .
By Freedman’s inequality, for all ( k, h, f ) ∈
[ K ] ×
[ H ] × N ( F , ρ ) we have with probability at least 1 − p , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) k (cid:88) t =
U t ( h, f ) − k (cid:88) t =
[( f h − Q ∗ h )( x th , a th , b th )]
2 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ O (cid:18)(cid:118)(cid:117)(cid:117)(cid:116) log(1 /p )
k (cid:88) t =
[( f h − Q ∗ h )( x th , a th , b th )]
2 + O ( ι ) (cid:19) where ι = log( HK | N ( F , ρ ) | /p ) .
Since [( f h − Q ∗ h )( x th , a th , b th )] 2 ≥ 0 , this implies − k (cid:88) t =
U t ( h, g ) ≤ O ( ι ) .
Therefore for all ( k, h, g ) ∈ [ K ] × [ H ] × N ( F , ρ ) k (cid:88) t = (cid:0) Q ∗ h ( x th [Equation] a th [Equation]
th ) [Equation] th − min ν ′ max π ′ Q ∗ h [-+
-]+
[Equation]
( x th + [Equation] π ′
[Equation]
ν ′ )
(cid:1)≤ k (cid:88) t = (cid:0)
[Equation]
+ O ( [Equation] + kρ ) . Thus we have proven that Q ∗ ∈ V k , ∀ k ∈
[ K ] with probability at least 1 − p .
Remark B.3.
This step implies that
[Equation]
(11) which ensures optimism from the beginning state (‘global optimism’).
Step 3: Regret decomposition The following result is key to our analysis.
It decomposes the deviation from the value of the game into Bellman errors across time steps and a martingale difference sequence.
Lemma B.4.
Define
[Equation]
Then we have
[Equation]
where ϵ kh is the bellman error defined by
[Equation]
Proof.
By definition of (cid:98) ν kh = argmin
ν f kh ( x kh [Equation] π kh , ν )
[Equation]
we have
[Equation]
It thus follows that
[Equation]
Therefore we complete the proof.
Step 4:
Bounding cumulative Bellman error using Minimax Eluder dimension.
Recall that
Lemma B.1 gives:
[Equation]
Combining this and Lemma F
[Equation]
1 with
[Equation]
and ρ k = δ ( x kn , a kh , b kh )
[Equation]
we have come to the following result.
Lemma B.5.
For any ( k, h ) ∈ [ K ] × [ H ] we have
[Equation]
Thus
[Equation]
Step 5: Putting everything together By Eq (11) and Eq (12) [Equation]
K (cid:88) k
= [ V k 1 ( x 1 ) − V π k ,ν k
( x 1 )] ≤
K (cid:88) k = H (cid:88) h = ( − ζ kh + γ kh − (cid:98) γ kh + ϵ kh ) =
H (cid:88) h = K (cid:88) k = ( − ζ kh + γ kh − (cid:98) γ kh )
+
H (cid:88) h
=
K (cid:88) k
= ϵ kh .
Notice that (cid:80)
Hh =1 (cid:80)
Kk =1 ( − ζ kh + γ kh − (cid:98) γ kh ) is a martingale difference sequence and can be bounded by O ( H (cid:112) KH log( HK/p )) with probability at least 1 − p .
By Eq (13) the term (cid:80) Hh =1 (cid:80)
Kk =1 ϵ kh can be bounded by O (cid:18) H (cid:113) K · dim ME ( F , (cid:112)
[Equation]
( F
[Equation]
with probability at least 1 − p .
It thus follows that regret can be bounded by
[Equation]
B. L INEAR FUNCTION APPROXIMATION
In the linear function approximation setting of Xie et al.
[Equation]
where ϕ ( · , · , · ) ∈ R d is known feature vector and θ h ∈ R d and µ h ( · ) ∈ R d are unknown with ∥ θ ∥ 2 ≤ √ d , ∥ ν h ( · ) ∥ 2 ≤ √ d and ∥ ϕ ( · , · ) ∥ 2 ≤ 1 .
In this case Algorithm 1 reduces to the following
Algorithm 3.
Notice that this algorithm can also be seen as a generalization of Zanette et al.
to
Markov games.
Algorithm 3 ONEMG for linear function approximation.
1: Input: Function class F 2: Set β ← C √ d log( HK/p ) for some large constant C 3: for episode k = 1 , 2 , . .
, K do 4:
Receive initial state x k5:
Set w kH +
=
θ kH
+1 =6:
Find
max w kh ,θ kh ,h ∈
[ H ] V k 1 ( x k 1 ) such that for all h ∈
[ H ] :
[Equation]
7: for step h = 1 , 2 , . . .
, H do 8:
P1 plays action a kh ∼ π kh
[Equation]
P2 takes action b kh 10:
Observe reward r kh and move to next state x kh + 11: end for 12: end for
We will prove the following theoretical result of Algorithm 3.
Notice the dependency on d of
Algorithm 3 is d while the dependency on d of Xie et al.
(2020) is d 3 / 2 , thus this result improves theirs by √ d factor.
Theorem B.6.
Regret (Eq (3) ) in Algorithm 3 is bounded by O ( d (cid:112) H 2 K log( dHK/p )) with probability at least 1 − p .
B.1. P ROOF OF T HEOREM B.
We prove Theorem B.6 in the following five steps similar to the previous section.
Step 1: Low Bellman error in the constraint set We use some standard results from previous work.
Notice that the N ( F , ϵ ) =
O ( d log(1 /ϵ ))
and our inherent Bellman error is zero, thus these lemmata can be directly adapted into our setting.
Lemma B.7 (Concentration, adapted from Lemma 1 of Zanette et al.
With probability at least 1 − p , the following holds for all ( k, h ) ∈
[ K ] × [ H ] ,
[Equation]
where β = Ω( (cid:112) d log( dKH/p )) .
Lemma B.8 (Least square error bound, adapted from Lemma 3 of Xie et al.
On the event of
Lemma B.7, the following holds for all ( x, a, b, h, k ) ∈ S
×
A × A ×
[ H ] ×
[ K ] and any policy pair ( π, ν )
[Equation]
Step 2: Q ∗ always stays in the constraint set
Lemma B.9 (Optimism) .
Let Q ∗ h ( · , · , · ) =
ϕ ( · , · , · ) ⊤ θ ∗ h , then on the event of Lemma B.8 and
Lemma B.7 there exists w ∗ h ,
h ∈
[ H ] such that w ∗ h , θ ∗ h ,
h ∈
[ H ] satisfies the constraints of Line 6 in
Algorithm 3.
Proof.
We prove by backward induction on h .
The claim trivially holds for h
= H + 1 .
Now assume the inductive hypothesis holds at h
+ 1 .
Let
[Equation]
where ϕ τh = ϕ ( x τh , a τh , b τh ) and η τh = V kh +
( x τh +1 ) − E x ∼ P ( ·| x τh ,a τh ,b τh )
V
kh +
( x ) .
Notice that inductive hypothesis implies r τh + E x ∼ P ( ·| x τh
[Equation]
a τh ,b τh )
V
{+kh +
 ( x ) =+}
ϕ
( x τh , a τh , b τh )
⊤ θ ∗ h , therefore
[Equation]
It thus suffices to bound the following
[Equation]
where the penultimate step comes from ∥ θ ∗ h ∥ 2 ≤ √ d and Lemma B.7.
Remark B.10.
This fact implies that
[Equation]
[Equation]
Step 3: Recursive decomposition of regret
Lemma B.11 (Regret decomposition) .
Define
[Equation]
Then we have
[Equation]
where ϵ kh is the width defined by
[Equation]
Proof.
By definition of B 0 = argmin ν f kh ( x kh , π kh , ν )
[Equation]
we have
[Equation]
It thus follows that
[Equation]
where the second step comes from Lemma B.8 by letting ( x, a, b )
=
( x kh , a kh , b kh )
(notice that ⟨ ϕ ( x kh , a kh , b kh ) , θ kh ⟩ = Q kh
( x kh , a kh , b kh ) ).
Step 4:
Bounding width by Elliptical Potential Lemma
This step directly makes use of the following standard result.
Lemma B.12 (Elliptical Potential Lemma, Lemma 10 of Xie et al.
Suppose { ϕ t } t ≥ 0 is a sequence in R d satisfying ∥ ϕ t ∥ ≤ 1 , ∀ t .
Let Λ
∈ R d × d be a positive definite matrix, and Λ t = Λ 0 + (cid:80) i ∈
[ t ] ϕ i ϕ
⊤ i .
If the smallest eigenvalue of Λ 0 is lower bounded by 1 , then
[Equation]
Step 5: Putting everything together By Eq (14) and Eq (15)
[Equation]
(cid:88) h =
K (cid:88) k
= ( − ζ [Equation] + γ kh − (cid:98) γ kh )
+
H (cid:88) h
=
K (cid:88) k
= ϵ kh .
Notice that (cid:80)
Hh =1 (cid:80)
Kk =1 ( − ζ kh + γ kh − (cid:98) γ kh ) is a martingale difference sequence and can be bounded by H (cid:112) KH log( HK/p ) with probability at least 1 − p .
By Lemma B.12 the term (cid:80)
Hh =1 (cid:80)
Kk =1 ϵ kh can be bounded as follows
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
with probability at least 1 − p .
It thus follows that with probability at least 1 − p regret can be upper bounded by
O ( d (cid:112) H 2 K log( dHK/p )) .
C M ODEL - BASED METHOD FOR THE COORDINATED SETTING C. A LTERNATE O PTIMISTIC M ODEL E LIMINATION (AOME) Algorithm
← argmax M ∈M Q M ( x 1 , π M , ν M ) , let π k ← π M k5:
Find M k 2 ← argmin M ∈M Q M ( x 1 , π k , ν Mπ k ) , let ν k ← ν M k 1 π k 6:
Execute π k , ν k and collect n 1 rewards { r ih } h ∈
[ H ] ,i ∈ [ n 1 ] 7:
Let (cid:98) V k ← 1 n 1 (cid:80) n 1 i =1 ( (cid:80)
Hh =1 r ih ) 8: if max {| (cid:98) V k − Q M k 1 ( x 1 , π k , ν k ) | , | (cid:98) V k − Q M k 2 ( x 1 , π k , ν k ) |} ≤ ϵ/ 2 then 9:
Terminate and output π k , ν k .
10: else 11:
Find h k such that max i ∈ [2] | (cid:98) L ( M k 1 , M k 2 , M ki , h k ) | ≥ ϵ/ (4 H ) by Eq (5).
Collect n trajectories { ( x ih , a ih , b ih , r ih )
Hh =1 } ni =1 by executing a h , b h ∼ π kh , ν kh , ∀ h ∈
[ H ] 13:
Update constraint set, where (cid:98) E is given by Eq (6)
M k ← { M ∈ M k − 1 : (cid:98) E ( M k 1 , M k 2 , M, h k ) ≤ ϕ } 14: end if 15: end for
C. T HEORY FOR A LTERNATE O PTIMISTIC M ODEL E LIMINATION (AOME)
In this section we prove that Algorithm 4 terminates in finite rounds.
The technique bulk is then showing that in Step 13 the constraint set can only shrink for finite times.
This is dependent on the complexity of model class M and discriminator function class G .
We first generalize Witness rank (Sun et al., 2019b) to Markov games.
Definition C.1 (Witnessed model misfit in Markov games) .
For discriminator class G : S
×
A 1 ×
A 2 × R ×S (cid:55)→ R , models M 1 , M 2 , M ∈ M and level h ∈
[ H ] , the witnessed model misfit of Markov game at level h is defined as follow
[Equation]
where ( x, a, b ) ∼ P M 2 M 1 denotes x τ +1 ∼ P ( ·| x τ , a τ , b τ ) , a τ ∼ π M 1 and b τ ∼ ν M 2 π M 1 for all τ ∈ [ h ] .
Using g ( x h , a h , b h , r, x ) =
r ( x h , a h , b h ) + Q M ( x, π M 1 , ν M 2 π M 1 ) , we can also give a generalization of
Bellman error (Jiang et al., 2016) in Markov games by the following
[Equation]
The final components in our theory are two assumptions from Sun et al.
(2019b).
Define rank( A, β ) to be the smallest integer k
such that A = UV ⊤ with U, V ∈ R n × k and ∥
U i, ∗ ∥ 2 · ∥ V j, ∗ ∥ 2 ≤ β for any pair of rows U i, ∗ , V j, ∗ .
Assumption C.2 (Realizability) .
Assume M ∗ ∈ M .
Assumption C.3 (Witness misfit domination) .
Assume G is finite, ∥ g ∥ ∞ ≤ 2 , ∀ g ∈ G and ∀ M 1 , M 2 , M ∈ M : E ( M 1 , M 2 , M, h ) ≥ L ( M 1 , M 2 , M, h ) .
Now we introduce the Witness rank for Markov games.
Definition C.4 (Witness rank for Markov games) .
Given a model class M , a test function class
G and κ ∈ (0 , 1] .
For h ∈ [ H ] , we define N κ,h ⊂ R |M|
×|M| as the set of matrices such that any matrix A ∈ N κ,h satisfies
[Equation]
The Witness rank for Markov games is defined by
[Equation]
We are in the position to state the main theorem of this section.
Theorem C.5.
Under Assumption C.2 and Assumption C.3, set ϕ
= κϵ 100 H √ W κ where W κ = max h ∈
[ H ] W ( κ, β, M , G , h ) , let T =
HW κ log( β/ 2 ϕ ) , set n 1 = C · H 2 log( HT/p ) /ϵ 2 and n =
C · H 2 W κ |A| log( T |M||G| /p ) /
( κϵ ) 2 for large enough constant C .
Then for all ϵ, p, κ ∈ (0 , 1] , with probability at least 1 − p Algorithm 4 outputs a policy π such that V ∗ ( x 1 ) − V π,ν ∗ π ( x 1 ) ≤ ϵ within at most O ( H 3 W 2 κ |A| κ 2 ϵlog( T |G||M| /p )) trajectories.
As seen in the above theorem, Algorithm 4 has sample complexity that scales quadratic with Witness rank and logarithmically with cardinality of model class.
This matches the sample complexity results in the Markov decision process setting (Sun et al., 2019b).
When the test function is Q-function, the Witnessed model misfit reduced to Bellman error as seen in Eq (17) and witness rank reduce to
Bellman rank.
C. P ROOFS
We prove Theorem C.5 in the following steps.
In the first step, we show a simulation lemma that generalize Jiang et al.
(2017) to Markov games.
In the second step we show concentration of empirical estimates of Bellman error, model misfit and value functions.
In the third step we show that M ∗ always stays in the constraint set.
We examine the conditions when Algorithm 4 terminates or not in the fourth step.
Using these conditions, we bound the number of rounds in the fifth step.
Finally we combine everything and complete the proof of Theorem C.5.
From Definition C.4 we know that there exists A h ∈ N κ,h such that
[Equation]
and we can factorize
[Equation]
where
[Equation]
Step 1:
Simulation Lemma
The analysis begins with a useful Lemma that decouples the value difference into a sum of Bellman errors across time steps.
Lemma C.6 (Simulation Lemma) .
Fix model M 1 , M 2 , M ∈ M .
Let π = π M 1 , ν =
ν
M 2 π M 1 .
Under
Assumption C.3, we have
[Equation]
Proof.
Let ( x, a, b ) ∼ P M 2 M 1 denote x τ +1 ∼ P ( ·| x τ , a τ , b τ ) , a τ ∼ π M 1 and b τ ∼ ν M 2 π M 1 for all τ ∈ [ h ] .
Using definition of Bellman error in Eq (17), we have
[Equation]
We thus complete the proof.
Step 2: Concentrations
In this part we show some standard concentration results.
Lemma C.7.
With probability at least 1 − p , we have the following event
(cid:12)(cid:12)(cid:12) E ( M k 1 , M k 2 , M, h ) − (cid:98) E ( M k 1 , M k 2 , M, h ) (cid:12)(cid:12)(cid:12) ≤ ϕ for all M ∈ M and h ∈
[ H ] , 2.
(cid:12)(cid:12)(cid:12) (cid:98) L ( M k 1 , M k 2 , M ki , h ) − L ( M k 1 , M k 2 , M ki , h ) (cid:12)(cid:12)(cid:12) ≤ ϵ 8 H for all h ∈
[ H ] and i ∈ [2] , 3.
(cid:12)(cid:12)(cid:12) V π k ,ν k − (cid:98) V k (cid:12)(cid:12)(cid:12) ≤ ϵ/
holds for all k ≤ T rounds.
Proof.
Fix one iteration.
By Hoeffding’s inequality, with probability at least 1 − p/ 3 ,
[Equation]
Thus (2) follows from n 1 = C · H 2 log( HT/p ) /ϵ 2 and union bound on all T iterations.
Similarly we can verify (3).
For (1), fix model M ∈ M and g ∈ G , by Hoeffding’s inequality, with probability at least 1 − p/ 3 ,
[Equation]
Thus (1) follows from n = C · H 2 W κ |A| log( T |M||G| /p ) /
( κϵ ) 2 and union bound on all k ∈
[ T ] ,
M ∈ M and g ∈ G .
We complete the proof.
Step 3: M ∗ stays in the constraint set while it shrinks In this step, we show that the constraint set always contains the true environment.
The result is displayed in the following.
Lemma C.8.
Suppose for each round k , the event in Lemma C.7 holds.
Then M ∗ ∈ M k for all k .
Furthermore, let (cid:99) M k
= { M ∈ (cid:99)
M k − 1 : A h k ( M k 1 , M k 2 , M ) ≤ 2 ϕ } with (cid:99) M 0 =
M .
We have
M k ⊂ (cid:99) M k for all k .
Proof.
Since E ( M k 1 , M k 2 , M ∗ , h ) = 0 , ∀ h ∈ [ H ] , we have (cid:98) E ( M k 1 , M k 2 , M ∗ , h k ) ≤
E ( M k 1 , M k 2 , M ∗ , h k ) + ϕ ≤ ϕ , Thus M ∗ is not eliminated.
We prove the second result by induction.
Firstly (cid:99) M 0 =
M 0 .
Assume M k − 1 ⊂ (cid:99)
M k − 1 .
For all M ∈ M k , we have
A h k ( M k 1 , M k 2 , M ) ≤ E ( M k 1 , M k 2 , M, h k ) ≤ (cid:98) E ( M k 1 , M k 2 , M, h k ) + ϕ ≤ 2 ϕ by Line 13 of Algorithm 4.
Therefore M ∈ (cid:99) M
k
, we complete the proof.
Step 4:
Terminate or rank increase The following Lemma is key to the ‘alternate pessimism principle’.
Intuitively, it indicates that the algorithm either terminates and outputs a pair of policies with small duality gap, or detects a time step that provides useful information (large Bellman error).
Lemma C.9.
Suppose for round k , the event in Lemma C.7 holds and M ∗ is not eliminated.
Then if the algorithm terminate, it outputs a policy pair π k such that V ∗ ( x 1 ) − V π k ,ν ∗ πk ( x 1 ) ≤ ϵ ; otherwise
A h k (( M k 1 , M k 2 ) , M ki ) ≥ κϵ 8 H for some i ∈ [2] .
Proof.
Consider the situation that the algorithm does not terminate.
Then with out loss of generality we can assume | (cid:98) V k − Q M k 1 ( x 1 , π k , ν k ) | ≥ ϵ/ 2 .
By Lemma C.7 and Lemma C.6, we have
[Equation]
H
[Equation]
By pigeonhole principle, these exists h ∈
[ H ] such that |L
( M k 1 , M k 2 , M k 1 , h ) | ≥ 3 ϵ 8 H .
For this h we have | (cid:98) L ( M k 1 , M k 2 , M k 1 , h )
| ≥ 3 ϵ 8 H − ϵ 8 H =
ϵ 4 H , thus the h k in Line 11 is well defined and we have | (cid:98) L ( M k 1 , M k 2 , M k 1 , h k )
| ≥ ϵ/ (4 H ) .
Using Lemma C.7 again, we have
[Equation]
| ≥
[Equation]
H .
If the algorithm terminates, we have
[Equation] [Equation]
V
[Equation]
x [Equation] and Q M k 2 ( x 1 , π k , ν k ) ≤ V π k ,ν ∗ πk ( x 1 ) .
Therefore
[Equation]
We thus complete the proof.
Step 5:
Bounding the number of iterations.
This step uses the technique of Jiang et al.
Sun et al.
(2019a), which uses the volume of minimum volume enclosing ellipsoid as a potential function to bound the number of iterations.
The key result is presented in Lemma C.10.
Lemma C.10.
Suppose for every round k , the event in Lemma C.7 holds, then the number of iterations of Algorithm 4 is at most HW κ log( β 2 ϕ ) / log(5 / 3) for certain i ∈ [2] .
Proof.
If the algorithm does not terminate at round k , then by
Lemma
C.9, ⟨ ζ h k ( M k 1 , M k 2 ) , χ h k ( M ki ) ⟩ =
A h k (( M k 1 , M k 2 ) , M ki ) ≥ κϵ 8 H = 6 √ W κ ϕ .
Let O kh denote the origin-centered minimum volume enclosing ellipsoid of { χ h ( M ) : M ∈ (cid:99) M k } .
Let O k − 1 , + h denote the origin-centered minimum volume enclosing ellipsoid of { v ∈
O k − 1 h : ⟨ ζ h
k
( M k 1 , M k 2 ) , v ⟩ ≤ 2 ϕ } .
By Lemma F.2, we have
[Equation]
Define Φ = sup M 1 ,M 2 ∈M ,h ∈ [ H ] ∥ ζ h ( M 1 , M 2 ) ∥ 2 and Ψ = sup M ∈M ,
h ∈
[ H ] ∥ χ h ( M ) ∥ 2 .
Then we have vol( O 0 h ) ≤ V W κ (1) · Ψ W κ where V W κ ( R ) is the volume of Euclidean ball in R W κ with radius
R .
On the other hand, we have
[Equation]
It thus follows that the number of iterations for each h is at most W κ log( ΦΨ2 ϕ ) / log(5 / 3) .
Using β ≥ ΦΨ , this completes the proof.
Step 6: Putting everything together.
From Lemma C.7, with high probability the events holds.
Therefore for the first T rounds, we can apply Lemma C.10 and know that the algorithm indeed terminates in at most T rounds.
The number of trajectories is thus upper bounded by ( n 1 + n ) T =
O ( H 3 W 2 κ |A| κ 2 ϵlog( T |G||M| /p )) .
D T HEORY FOR A LTERNATE O PTIMISTIC V ALUE E LIMINATION (AOVE)
We prove Theorem 4.4 in the following five steps.
In the first step we show that the functions in the constraint set has low Bellman error.
Notice that different from Lemma B.1, here we consider a different Bellman operator in Eq (2) and consider all policies in policy class Π .
In the second step we show that Q π,ν ∗ π belongs to the constraint set for all π ∈ Π and in particular Q ∗ belongs to the constraint set.
In the third step we perform a regret decomposition.
Notice that here we consider two Bellman errors.
We then bound the summation of these Bellman errors by Minimax Eluder dimension in step 4 and complete the proof in step 5.
Step 1: Low Bellman errors in the constraint set.
Lemma D.1 (Concentration on Bellman error) .
Let ρ > 0 be an arbitrary fixed number.
With probability at least 1 − p for all ( k, h ) ∈ [ K ] ×
[ H ] we have
[Equation]
Proof.
For simplicity we only proof ( a ) , and (b) follows similarly.
Consider a fixed ( k, h, f, π ) tuple, let
U t ( h, f, π ) :=
[Equation]
f h
[Equation]
th , a th ,
[Equation]
th
[Equation]
th
[Equation]
+
[Equation]
th
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
and F t,h be the filtration induced by { x i 1 , a i 1 , b
i 1 , r i 1 , . . .
, x iH } t − 1 i =1 ∪ { x t 1 , a t 1 , b t 1 , r t 1 , . . .
, x t h , a t h , b t h } .
We have
[Equation]
and
Var
[Equation]
By Freedman’s inequality, we have with probability at least 1 − p , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) k
[Equation]
Let N ρ × Y ρ be a ρ -cover of F × Π .
The covering of Π is in-terms of the distance of two policies defined as
[Equation]
Taking a union bound for all ( k, h, f , p ) ∈
[ K ] ×
[ H ] × N ρ × Y ρ , we have that with probability at least 1 − p for all ( k, h, f , p ) ∈
[ K ] ×
[ H ] × N ρ × Y ρ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) k
[Equation]
t =
U t ( h
[Equation]
f
[Equation]
p ) −
[Equation]
t =
[Equation]
where ι = log( HK | N ρ || Y ρ | /p ) .
For an arbitrary ( h, k ) ∈
[ H ] ×
[ K ] pair, by the definition of V k and ( π k ,
f k ) ∈
V k we have
[Equation]
where in the first step we use Assumption 3.2.
Define f k = argmin f ∈ N ρ max h ∈
[ H ] ∥
f kh − f kh ∥ ∞ , p k = argmin p ∈ Y
ρ d ( π k , p ) .
Since N ρ is a ρ -cover of F ,
[Equation]
Therefore combining Eq (23) and Eq (24) [Equation]
[Equation]
Recall Eq (22) indicates
[Equation]
[Equation]
Combining Eq (25) and Eq (26) we have
[Equation]
Since f k is an ρ -approximation to f k and p k is an ρ -approximation to π k , we can conclude the proof of ( a ) with
[Equation]
Step 2: ( π, Q π,ν ∗ π ) , ∀ π ∈ Π is always in constraint set.
Lemma D.2.
With probability at least 1 − p , we have ( π ∗ , Q ∗ ) and ( π, Q π, ∗ ) , ∀ π ∈ Π all belong to
V k for all k ∈
[ K ] .
Proof.
Consider a fixed ( k, h, f, π ) tuple, let
[Equation]
and F t,h be the filtration induced by { x i 1 , a i 1 , b
i 1 , r i 1 , . . .
, x iH } t − 1 i =1 ∪ { x t 1 , a t 1 , b t 1 , r t 1 , . . .
, x th , a th , b th } .
We have
[Equation]
, f, π ) | F t,h ] =
[( f h − Q π,ν ∗ π h )( x th , a th , b th )]and
Var[ U t ( h, f, π ) | F t,h ] ≤ 36[( f h − Q π,ν ∗ π h )( x th , a th , b th )] 2 = 36 E [ U t ( h, f, π ) | F t,h ] .
By Freedman’s inequality, we have with probability at least 1 − p , (cid:12) (cid:12) (cid:12)(cid:12)(cid:12) k
(cid:88) t =
U t ( h, f, π ) − k (cid:88) t =
[( f h − Q π,ν ∗ π h )( x th , a th , b th )](cid:12) (cid:12) (cid:12)(cid:12)(cid:12) ≤ O (cid:18)(cid:118)(cid:117)(cid:117)(cid:116) log(1 /p )
k (cid:88) t =
[( f h − Q π,ν ∗ π h )( x th , a th , b th )] 2 + log(1 /p )
(cid:19) .
Let N ρ × Y ρ be a ρ -cover of F × Π .
Taking a union bound for al
( k, h, f , p ) ∈
[ K ] ×
[ H ] × N ρ × Y ρ , we have that with probability at least 1 − p for all ( k, h, f , p ) ∈
[ K ] ×
[ H ] × N ρ × Y ρ − k (cid:88) t =
U t ( h, f , p ) ≤ O ( ι ) where ι = log( HK | N ρ | /p ) .
This implies for all ( k, h, f , p ) ∈
[ K ] ×
[ H ] × N ρ × Y ρ k (cid:88) t =
[Equation]
h
[Equation]
Since N ρ × Y ρ be a ρ -cover of F × Π , for all p k ∈ Y ρ k (cid:88) t = (cid:0) Q p
[Equation]
p h (
[Equation]
th
[Equation]
th , b th
[Equation]
≤ inf
[Equation]
k (cid:88) t =
[Equation]
(cid:0)
[Equation]
O (
[Equation]
[-) (cid:1) 
-]
+ kρ ) .
Again using N ρ × Y ρ be a ρ -cover of F × Π , we have for all π ∈ Π k (cid:88) t = (cid:0) Q π
[Equation]
Therefore we have ( π, Q π,ν ∗ π ) , ∀ π ∈ Π all belong to V k for all k ∈
[ K ] with probability at least 1 − p .
Remark D.3.
An important implication of this fact is the following
[Equation]
which gives an upper bound of the one-side duality gap.
Step 3:
Bounding the regret by Bellman error and martingale.
Lemma D.4.
Define
[Equation]
Then we have
[Equation]
where ϵ kh , (cid:98)
ϵ
kh are the bellman error defined by
[Equation]
Proof.
By definition of ( ν f k π k ) h = argmin ν ∈ Π h f kh
( x kh , π kh , ν ) , we have
[Equation]
It thus follows that
[Equation]
Thus we complete the proof.
Step 4:
Bounding cumulative Bellman error using Minimax Eluder dimension.
Lemma D.5 (Bounding Bellman error) .
For any ( k, h ) ∈ [ K ] × [ H ] we have
[Equation]
Proof.
Let l
[Equation]
[Equation]
and ρ t = δ ( x th , a th , b th ) .
Similarly, we can also let ρ t be generated by ( π t , ν t ) .
Then Lemma D. indicates
[Equation]
Based on this, we can apply Lemma F.1 to obtain the first inequality.
The second follows similarly.
Step 5: Putting them all together.
We can upper bound the regret as follows:
[Equation]
where the first step comes from Lemma D.2, the second step comes from Lemma D.4 and the last step comes from Fubini’s theorem.
Notice that the first term in Eq (27) is a martingale difference sequence and can be bounded by O ( H (cid:112) KH log( HK/p )) with probability at least 1 − p and the second term can be bounded by Lemma D.5.
We thus complete the proof.
E E XAMPLES
In this section we give several examples of Markov games that possess low Minimax Eluder dimension, including tabular Markov Games, linear function approximation, reproducing kernel Hilbert space (RKHS) function approximation, overparameterized neural networks, and generalized linear models .
We first consider reproducing kernel Hilbert space (RKHS) function approximation, which captures many existing models.
Reproducing kernel Hilbert space (RKHS) function approximation.
In this setting, the function class of interest is given by
[Equation]
Here ϕ ( · , · , · ) : S ×
A × A (cid:55)→ H is a feature map to H , θ ∈ H , and B R ( H ) is the ball centered at zero with radius R in a Hilbert space H .
The effective dimension of a set D ⊂ H is characterized by the critical information gain (cid:101) d ( γ, D ) .
Specifically, for γ > 0 and D ⊂ H , we define d n ( γ, D ) =
max x 1 ,...,x n ∈D log det( I + 1 γ (cid:80) ni
=1 x i x ⊤ i ) .
The critical information gain (Du et al., 2021) is thus defined by (cid:101) d ( γ, D ) =
min k ∈ N : k ≥ d n ( γ, D ) k .
The main result is given in the following theorem.
Theorem E.1.
In kernel function approximation, the Minimax Eluder dimensions dim ME ( F , ϵ ) and dim ME ′ ( F , ϵ ) are upper bounded by max h ∈
[ H ] (cid:101) γ
( ϵ 9 R , Φ h ) , where Φ h = { ϕ h ( x, a, b ) : x ∈ S , a, b ∈
A} .
Proof.
We only show dim ME ′ ( F , ϵ ) ≤
max h ∈
[ H ] (cid:101) γ
( ϵ 9 R , Φ h ) , and the other part is totally similar.
Fix an h ∈
[ H ] .
Due to completeness, for any π ∈ Π and f ∈ F we have ( f h −
T πh f h +1 )
( x, a, b ) = ⟨ ϕ h ( x, a, b ) , θ h − T πh θ h +1 ⟩ where T πh θ h +1 is the unique feature in H such that T πh f h
+1 = ⟨ ϕ h ( x, a, b ) , T πh θ h +1 ⟩ .
Consider the longest sequence ν 1 , . .
ν k ∈ D ∆ such that each one item is ϵ -independent of its precedents, distinguished by a sequence of features θ (1) h , . . .
, θ ( k ) h .
To simplify notations, we define ϕ i
=
E ν i
[ ϕ h ( x, a, b )] and θ i
= θ ( i )
h − T πh θ ( i ) h
+1 for all i ∈ [ k ] .
Let Λ i
=
(cid:80) i − [-1 τ =
-][Equation]
=
1 ϕ τ ϕ ⊤ τ + ϵ 2 9 R 2 I .
Then Definition 3.3 gives
[Equation]
where the first step comes from θ i ∈ B R ( H ) and (cid:80) i − 1 j =1 ⟨ ϕ j , θ j ⟩ 2 ≤ ϵ , the second step comes from
Hölder’s inequality and the final step comes from |⟨ ϕ
i , θ i ⟩| ≥ ϵ .
Thus we have
[Equation]
By the definition of critical information gain, k must be less or equal than (cid:101) γ ( ϵ 9 R , Φ h ) .
This result implies several important cases in general function approximation.
• Linear function approximation.
In this setting r h ( x, a, b ) = ϕ ( x, a, b ) ⊤ θ h and P h ( ·| x, a, b ) =
ϕ
( x, a, b ) ⊤ µ h ( · ) where ϕ ( x, a, b ) ∈ R d .
We have shown that this setting satisfies all the realizability and completeness assumptions in the previous sections.
By standard Ellipsoid potential arguments, the critical information gain is upper bounded by O ( d · log( R/ϵ )) .
Therefore, this setting also has low Minimax Eluder dimensions.
• Tabular Markov Games.
This setting can be seen as a special case of linear function approximation where the feature vectors are chosen to be one-hot representations of stateactions pairs.
In this case, d = |S| · |A| 2 and thus Minimax Eluder dimensions is upper bounded by (cid:101)
O ( |S| · |A| 2 ) .
• Overparameterized neural networks.
In overparameterized neural network function approximations, the functions are given by h ( ϕ ( x, a, b )) where h is the Neural Tangent Random Feature space defined in Cao & Gu (2019).
We rewrite the information gain d n ( γ, D ) as max x 1 ,...,x n ∈D log det( I + 1 γ K n ) where ( K n ) i,j = K ( x i , x j ) and K is the Neural Tangent Kernel.
It has been shown (Jacot et al., 2018;
Du et al., 2019c;a; Allen-Zhu
et al., 2019) that with proper initialization and learning rates, the function approximations approximately lie in the Reproducing kernel Hilbert space given by the neural tangent feature as above.
Therefore the Minimax Eluder dimensions are bounded by (cid:101) γ
( ϵ 9 R , Φ h ) + λ , where (cid:101) γ is the critical information gain defined by the d n ( γ, D ) rewritten above and λ is a term that vanishes as width tends to infinity.
Generalized linear models.
Finally, we consider generalized linear model as an extension of the linear function approximations.
Theorem E.2.
Suppose ( f h − T πh f h +1 )
( x, a, b ) =
g ( ϕ ( x, a, b ) ⊤ θ h ) for any π ∈ Π and f ∈ F , where g is a differentiable and strictly increasing function and ϕ, θ h ∈ R d .
Assume g ′ ∈ ( c 1 , c 2 ) and ∥ ϕ ∥ 2 , ∥ θ h ∥ 2 ≤ R for c 1 , c 2 , R > 0 .
Then dim ME ′ ( F , ϵ )
≤ O ( d ( c 2 /c 1 ) 2 log( c 2 R c 1 ϵ )) .
Proof.
The proof mirrors those in Theorem E.1.
For the longest sequence ν 1 , . . .
, ν k ∈ D ∆ such that each one item is ϵ -independent of its precedents, distinguished by a sequence of features θ (1) h , . . .
, θ ( k ) h , we define ϕ i
=
E ν i
[ ϕ h ( x, a, b )] and θ i
= θ ( i )
h − T πh θ ( i ) h
+1 for all i ∈ [ k ] .
Let Λ i
=
(cid:80) i − 1 τ =
ϕ
τ ϕ ⊤ τ + ϵ 2 9 R 2 c 21 I .
Then Definition 3.3 gives
[Equation]
This means det(Λ k ) ≥ (1 + c 21 c 22 ) k − 1 .
But by Elliptic potential lemma, det(Λ k ) ≤ ( R [Equation] kd [Equation]
2 9 R 2 c 21 ) d . Combining these two inequalities gives k ≤ O ( d ( c 2 /c 1 ) 2 log( c 2 R c 1 ϵ )) .
F T ECHNICAL CLAIMS
This section lists the facts we use from the previous work.
Lemma F.1 (Cumulative error control, Lemma 26 of Jin et al. (2021a)) .
Given a function class G defined on X with | g ( x ) | ≤ C for all ( g, x ) ∈ G × X and a family of probability measures D over X .
Suppose sequence { g k } Kk =1 ⊂ G and { ρ k } Hk =1 ⊂ D such that for all k ∈ [ K ] , (cid:80) k − 1 t =1 ( E ρ t [ g k ]) 2 ≤ β .
Then for all k ∈ [ K ] and ω > 0 , k
[Equation]
Lemma F.2 (Lemma 11 of Jiang et al.
Consider a closed and bounded set V ⊂ R d and a vector p ∈ R d .
Let B be any origin-centered enclosing ellipsoid of V .
Suppose there exists v ∈ V such that p ⊤ v ≥ κ and define B + as the minimum volume enclosing ellipsoid of { v ∈ B : | p ⊤ v | ≤ κ 3 √ d } .
With vol ( · ) denoting the (Lebesgue) volume, we have:
[Equation]