L EARNING T WO -P LAYER M IXTURE M ARKOV G AMES : K ERNEL F UNCTION A PPROXIMATION AND C ORRELATED E QUILIBRIUM
A BSTRACT
We consider learning Nash equilibrium in two-player zero-sum Markov games with nonlinear function approximation, where the action-value function is approximated by a function in the Reproducing Kernel Hilbert Space (RKHS).
The key challenge is how to do exploration in the high-dimensional function space.
We propose novel online learning algorithms to ﬁnd an approximate Nash equilibrium by minimizing the duality gap.
At the core of our algorithms are upper and lower conﬁdence bounds that are derived based on the principle of optimism in the face of uncertainty.
We prove that our algorithm is able to attain an O ( √ T ) regret with polynomial computational complexity, under very mild assumptions on the reward function and the underlying dynamic of the Markov Games.
This work provides the ﬁrst complexity results for learning two-player zero-sum Markov games with nonlinear function approximation in the mixture model settings and its implications for function approximation via deep neural networks
I NTRODUCTION
Multi-agent reinforcement learning (MARL) has been the focus of research across a range of research communities (Shapley, 1953; Littman, 1994).
The case of two-player Markov Games (MG) has been of particular interest.
In this case, two players select their actions based on the current state simultaneously and independently.
One player (the max-player) aims to maximize the return based on the reward provided by the environment, while the other (the min-player) aims to minimize it.
A series of recent results have established polynomial sample complexity/regret guarantees that depend on the cardinality of state/action spaces for two-player MG (Wei et al., 2017; Bai & Jin, 2020; Bai et al., 2020; Liu et al., 2020;
Jia et al., 2019; Sidford et al., 2020; Cui & Yang, 2020; Lagoudakis &
Parr, 2002; Perolat et al., 2015; P´erolat et al., 2016a;b; 2017).
Meanwhile, most of the recent successful applications of MARL deal with large state/action spaces that may be continuous or a ﬁne-grained discretization of a continous space.
Examples include
GO (Silver et al., 2016), autonomous driving (Shalev-Shwartz et al., 2016), TexasHold’em poker (Brown & Sandholm, 2019), and AlphaStar for the game Starcraft (Vinyals et al., 2019).
In order to tackle problems with large state/action spaces, researchers have designed MARL algorithms based on function approximation which approximate the original high-dimensional value function/policy by a function approximator.
For instance, Xie et al.
and
Chen et al.
(2021) studied RL for two-player zero-sum MGs with linear function approximation , where it is assumed that there are a set of linear features that span the transition kernel and reward function spaces.
In contrast to
RL with linear function approximation, RL with nonlinear function approximation (e.g., kernel and neural network approximation) aims to take advantage of the superior representational power of nonlinear function compared to linear parameterizations.
For example, Jin et al.
(2021) studied neural-network-based RL in the setting of MGs with low multi-agent Bellman eluder dimension , obtaining algorithms that have polynomial dependence on the complexity of the underlying function class.
Although this yields a strong theoretical guarantee, the speciﬁc algorithm that they propose is not computationally efﬁcient due to the constructed highly nonconvex conﬁdence sets.
The following question is still open: Can we design a computationally and statistically efﬁcient RL algorithm for learning two-player zero-sum Markov Games with nonlinear function approximation?
In this paper, we give an afﬁrmative answer to this question for a class of episodic Markov Games, dubbed mixture Markov Games , when using nonlinear approximation function in the Reproducing
Kernel Hilbert Space (RKHS).
We propose a novel kernel-based MARL algorithmic framework for general episodic two-player MGs, which provides provable regret guarantees.
We summarize the contributions of our work as follows:
• We propose a speciﬁc KernelCCE-VTR algorithm for two-player zero-sum MGs.
In particular, at each episode, KernelCCE-VTR uses kernel function approximation to approximate the optimal value function and constructs corresponding conﬁdence sets, following the “Optimism-in-Face-ofUncertainty” principle (Abbasi-Yadkori et al., 2011) to select an action based on the current state.
In contrast to algorithms in Jin et al.
(2021), which construct implicit conﬁdence sets that are in general computationally intractable, our algorithm KernelCCE-VTR crafts a computationally efﬁcient exploration bonus based on the gram matrix of the kernel function.
• Under the assumption that the transition dynamics belongs to some RKHS, we show that our algorithm KernelCCE-VTR is able to ﬁnd an approximate Nash equilibrium of the game with a (cid:101) O
( d F H 2 √ T ) upper bound on the regret of the duality gap, where H is the horizon, T is the number of the episodes, and d F represents the complexity of the function class F .
When F reduces to the d -dimensional linear function class, our result reduces to (cid:101) O
( dH 2 √ T ) and nearly matches the complexity result in Chen et al.
up to a √ H factor.
To the best of our knowledge, this is the ﬁrst algorithm for learning two-player Markov Games with nonlinear function function approximation that is efﬁcient in terms of both computational and sample complexities.
• We also study the general case where the transition dynamic belongs to some RKHS up to a misspeciﬁcation error.
We show that our KernelCCE-VTR can achieve a similar regret as in the well-speciﬁed case.
In particular, we study the neural network function approximation case which can be regarded as a special instance of the misspeciﬁed RKHS case and derive the corresponding regret bound.
Notation We use lower case letters to denote scalars, lower and upper case bold letters to denote vectors and matrices.
We use (cid:107) · (cid:107) to indicate Euclidean norm, and for a semi-positive deﬁnite matrix Σ and any vector x , (cid:107) x (cid:107) Σ := (cid:107) Σ 1 / 2 x (cid:107) = √ x (cid:62) Σx .
For real t and interval [ a, b ] , we use Π [ a,b ] [ t ] to indicate the projection of t onto [ a, b ] , i.e. Π [ a,b ] [ t ] = max ( a, min( b, t )) .
For positive integer
N we sometimes deﬁne [ N ] = { 1 , . . .
, N } for compactness.
We also adopt the standard big-
O and big- Ω notations: say a n = O ( b n ) if and only if there exists C > 0 , N > 0 , for any n > N , a n ≤ Cb n ; a n = Ω( b n ) if a n ≥ Cb n .
The notations (cid:101) O and (cid:101) Ω are adopted when the C above hides a polylogarithmic factor.
R ELATED W ORK
Online RL with function approximation MARL with function approximation can be seen as an extension of RL with function approximation on MDPs.
There are several lines of work studying RL with function approximation.
The ﬁrst line of work studies the so-called linear MDP which assumes the reward function and transition dynamics are linear functions of a feature mapping deﬁned on the state and action spaces (Yang & Wang, 2020; Jin et al., 2020; Zanette et al., 2020).
These works proposed model-free algorithms with sublinear regret on the number of episodes T .
The second line of work studies the linear mixture MDP which assumes the transition kernel is a linear combination of several base models (Modi et al., 2020; Jia et al., 2020; Zhou et al., 2020a; 2021).
These studies proposed model-based RL algorithms that estimate the transition kernel with ﬁnite sample complexity or sublinear regret guarantees.
The third line of work studies general function approximation which assumes that either the value function or the transition kernel can be approximated by a general class of functions (Osband & Van Roy, 2014; Jiang et al., 2017; Sun et al., 2019; Wang et al., 2020; Yang et al., 2020).
Algorithms proposed in this line enjoy ﬁnite regret or sample complexity bounds that depend on some general complexity measures such as Eluder dimension (Russo & Van Roy, 2013;
Osband & Van Roy, 2014), Bellman rank (Jiang et al., 2017), witness rank (Sun et al., 2019), and information gain (Yang et al., 2020).
Learning two-player MGs with function approximation There is a large body of literature on
MARL for two-player MGs with function approximation.
These works can be generally categorized into MARL with linear function approximation and MARL with general function approximation .
For example, for linear function approximation, Xie et al.
(2020) studied zero-sum simultaneous-move
MGs where both the reward and transition kernel can be parameterized as linear functions of some feature mappings.
They proposed an OMVI-NI algorithm with an (cid:101) O ( √ d 3 H 4 T ) regret, where d is the number of the feature dimension, H is the episode length and T is the number of episodes.
Chen et al.
(2021) studied the linear mixture MGs and proposed a nearly minimax optimal Nash-UCRL-VTR algorithm with an (cid:101)
O ( d √ H 3 T ) regret and an Ω( d √ H 3 T ) matching lower bound.
In contrast to this work, our KernelCCE-VTR does not assume the underlying transition dynamic or reward function have a linear structure.
For MARL with general function approximation, Jin et al.
(2021) studied the two-player zero-sum MGs with low multi-agent Bellman Eluder dimension and proposed a “Golf with Exploiter” algorithm using a general function class.
They showed their algorithm enjoys an (cid:101)
O ( H √ dT log N ) regret, where d is the multi-agent Bellman eluder dimension.
Huang et al.
(2021) studied two-player MGs with a ﬁnite minimax Eluder dimension and proposed an ONEMG method with an (cid:101)
O ( H √ dT log N ) regret, where d is the minimax Eluder dimension.
To obtain the desired function approximator, both Golf with Exploiter and ONEMG need to solve a constrained optimization problem, which is computationally intractable even in the linear function approximation setting.
In contrast to Jin et al.
; Huang et al.
(2021), our proposed KernelCCE-VTR is computationally efﬁcient.
After the initial submission of this paper, we become aware of a recent independent work (Qiu et al., 2021) which also studied the kernel function approximation for two-player MGs.
Here we highlight the differences between these two works.
First, Qiu et al.
(2021) studied the MGs where the expectation of the value function can be parameterized by a function in some RKHS, while we assume the transition dynamic of the MGs lies in an RKHS.
Second, while the regret result in Qiu et al.
(2021) depends on the covering number of the function space, our regret is independent of the covering number.
P RELIMINARIES
In this section, we present the deﬁnition of a two-player Markov Game.
We deﬁne the action value function corresponding to two players’ policies and introduce the Nash equilibrium solution where both players’ policy act as each other’s best response policy.
As a slightly relaxed version of the
Nash equilibrium (NE), we also introduce Coarse Correlated Equilibrium (CCE), which serves as a computational efﬁcient approximation of the NE.
Lastly, we deﬁne a mixture Markov Game setting in which the transition probability lies in an RKHS.
In our context, as a generalization of a ﬁxed kernel function, we deﬁne a weighted kernel function that depends on a pair of bounded value functions and serves as the basis of the algorithm in Section 4.
3. T WO -P LAYER Z ERO -S UM M ARKOV G AMES
In this subsection, we describe simultaneous-move games in the setting of two-player Markov Games (MG).
In the rest of this paper by “game” we mean “zero-sum game” unless otherwise speciﬁed.
A simpler instance of Markov Games, referred to as turn-based games, can be seen as a special case of simultaneous-move games.
In a two-player simultaneous-move Markov Game, the dynamical structure can be captured by an MG formulated as ( S , A 1 , A 2 , r, P , H ) .
S is the space of available states of the environment.
A 1 is the action space of the ﬁrst player and A 2 is the action space of the second player.
H is the time horizon representing the maximum step of each round of play.
The reward function r : { r h ( x, a, b ) : h ∈
[ H ] } is a sequence of mappings from S ×
A 1 ×
A 2 to [ − 1 , 1] .
In the zero-sum setting, the positive reward for the max-player is the negative reward for the min-player.
And the transition matrix P : { P h ( · | x, a, b ) : h ∈
[ H ] } gives for each state actions triplet ( x, a, b ) and at each time h the stochastic response of the environment to the next x (cid:48) ∈ S .
Here by “simultaneous move” we refer to the setting where at each round of game the two players P 1 and P 2 take actions a ∈
A 1 , b ∈
A 2 simultaneously at a given state x ∈ S , in contrast with the turn-based game where r h and P h are deﬁned for a state-action pair ( x, a ) where the action can be taken by either players.
In the context of this paper, for simplicity of notation we let A
=
A 2 =
A , while the results can be easily generalized to the case when A 1 (cid:54) =
A 2 .
Similar deﬁnitions of a two-player simultaneous-move episodic Markov Games can be found in Wei et al.
; Perolat et al.
; Xie et al.
In the above setting, two players P 1 and P 2 take actions according to their own strategies.
We use π :=
{ π h } h ∈ [ H ] to denote the stochastic policy of P 1 and use ν := { ν h } h ∈ [ H ] to denote the stochastic policy of P 2 .
We note that at time h , π h : S (cid:55)→ ∆
A maps the current state x h to a probability distribution of the actions.
As is the same with ν h .
Given two agents policies π, ν across h steps, the state value function is deﬁned as the expected total reward through H steps when at step h ∈
[ H ] player P 1 follows policy π h ( · | x h ) and player P 2 follows policy ν h ( · | x h ) :
[Equation]
and V π,ν ( x ) :
= V π,ν( x ) .
Note that the expectation is taken over all stochasticity in π h , ν h and P h .
The action value function is deﬁned as
[Equation]
and Q π,ν ( x, a, b ) :
= Q π,ν 1 .
From the deﬁnition, we observe that for ∀ x ∈ S , the state value function given policy pair ( π, ν ) is the expectation of the corresponding action value function
[Equation]
where the expectation is taken over the action distribution induced by the policy pair.
3. E QUILIBRIUM AND D UALITY G AP
In this subsection, we recap the concepts of equilibrium and duality gap that have been widely used in the game theory literature.
Nash Equilibrium and Duality Gap
In a two-player Markov Game, P 1 wants to maximize the expected reward V π,ν ( x ) by properly choosing its policy π .
On the contrary, P 2 wants to minimize
V π,ν ( x ) by properly choosing ν .
For ﬁxed ν , we deﬁne the best response policy with respect to V and ν as br ( ν ) and deﬁne V ∗ ,ν h
= V br ( ν ) ,ν h := max π V π,νh and Q ∗ ,νh =
Q br ( ν ) ,ν h = max
π Q π,νh .
Similarly we deﬁne V π, ∗ h :=
min ν V π,νh and Q π, ∗ h :
=
min ν Q π,νh .
The Nash Equilibrium (NE) is a pair of policies ( π ∗ , ν ∗ ) that are the best response policy of each other, meaning V π ∗ , ∗ ( x ) =
V π ∗ ,ν ∗ ( x ) =
V ∗ ,ν ∗ ( x ) .
For notational simplicity we write V ∗ :
= V π ∗ ,ν ∗ , Q ∗ :=
Q π ∗ ,ν ∗ .
By deﬁnition of the best response policy, we have the following weak duality
V π, ∗ h ( x ) ≤ V ∗ h ( x ) ≤ V ∗ ,ν h ( x ) .
We deﬁne the duality gap as (cid:80)
Tt
=1 V ∗ ,ν t( x t 1 ) − V π t , ∗( x t 1 ) , and call it the Regret in the MG setting.
Coarse Correlated Equilibrium
We introduce the Coarse Correlated Equilibrium (CCE) (Aumann, 1987; Moulin & Vial, 1978)
ﬁrst.
Given payoff matrices Q 1 , Q 2 : S ×
A × A (cid:55)→ R , we deﬁne the CCE of the game as a joint distribution σ on A ×
A satisfying for ∀ a (cid:48) ∈
A , b (cid:48) ∈ B :
E ( a,b ) ∼ σ [ Q 1 ( x, a, b )] ≥
[Equation]
[Equation]
[Equation]
[Equation]
where P 1 σ denotes the marginal of σ on the ﬁrst coordinate (min-player) and P 2 σ denotes the marginal of σ on the second coordinate (max-player)
As suggested by Xie et al.
(2020), the set of
CCE includes the set of Correlated Equilibrium (CE) (Aumann, 1987; Moulin & Vial, 1978; Blum &
Monsour, 2007), which further includes the set of NE.
Even though the set of CCE is known as a convex set (Osborne & Rubinstein, 1994), later we will show that a good approximation of the NE can be derived from the approximate CCE for the estimates of the value function corresponding to Pand P 2 , respectively.
3. N ONLINEAR F UNCTION A PPROXIMATION BY R EPRODUCING K ERNEL H ILBERT S PACES
In this subsection, we provide necessary deﬁnitions and notations in approximating action value function with functions belonging to an reproducing kernel Hilbert space (RKHS) via modeling the transition probability.
For the simplicity of notation, we use z =
( x, a, b ) to denote the state action triplet in Z := S ×
A × A .
An RKHS H with kernel k ( · , · ) : Z × Z (cid:55)→ R is a generalization of the linear function class.
Every
RKHS H consists of functions on Z , where there exists a feature mapping φ : Z (cid:55)→ H , such that
∀ f ∈ H and ∀ z ∈ Z , f ( z ) = (cid:104) f, φ ( z ) (cid:105) H .
The kernel k is thus deﬁned for every x, y ∈
Z × Z as k ( x, y ) =
(cid:104) φ ( x ) , φ ( y ) (cid:105) H .
We call φ the feature mapping induced by the RKHS H with kernel k .
In the following sections, we use f (cid:62) g as a simpliﬁcation of (cid:104) f, g (cid:105) H when f, g ∈ H .
We make no distinction in notations between the vector product and the product (cid:104)· , ·(cid:105) H .
However, one can tell the difference from the two objects of the product.
For every RKHS H , there exists a natural eigenvalue decomposition in L 2 ( Z ) .
RKHS approximation is a generalization of the linear function approximation of ﬁnite dimension d , which can be inﬁnite dimensional.
In the following, we deﬁne the so-called kernel mixture MG , which can be regarded as an extension from the linear mixture
MDP
(Jia et al., 2020; Ayoub et al., 2020;
Zhou et al., 2021) and linear mixture MG (Chen et al., 2021) to their kernel counterpart.
Kernel Mixture MG
In a kernel mixture MG model, we model the transition probability
P h ( s (cid:48)
| z ) : Z (cid:55)→ ∆( S ) as an element in an RKHS H with feature mapping φ ( s (cid:48) | z ) : Z → S × H , such that the following equality holds for an unknown parameter θ ∗ h ∈ H :
[Equation]
At time h , for any estimate of the value function V h ( · ) : S (cid:55)→ R , we note that the expected value function at time h
+ 1 , P h V
h
+1 is an element in the RKHS
[Equation]
where φ V h +1 ( z ) :
= (cid:80) s (cid:48) ∈S φ ( s (cid:48) | z ) V h
+1 ( s (cid:48) ) integrates the product of the feature mapping with the estimated value of s (cid:48) over S .
It is worth noting that the quantity φ V ( · ) plays an important role in previous linear mixture model-based algorithms (Jia et al., 2020; Ayoub et al., 2020;
Zhou et al., 2021; Chen et al., 2021).
In this work, we face a general RKHS H and we cannot access the feature mapping φ directly.
Instead, we assume that we can access the weighted kernel function k V 1 ,V 2 ( · , · ) , which is deﬁned as follows:
Deﬁnition 1.
For any function pairs V 1 , V 2 : S →
[0 , 1] which map states to real numbers, the weighted kernel function k
V 1 ,V 2 ( · , · ) is deﬁned as follows for all z 1 , z 2 ∈ Z .
[Equation]
It is easy to see that
[Equation]
which suggests that the weighted kernel function k V 1 ,V 2 ( · , · ) indeed captures the inner product relation between φ V 1 ( z 1 ) and φ V 2 ( z 2 ) .
In this work, we assume that we can access an integration oracle that can calculate k V 1 ,V 2 ( z 1 , z 2 ) for any function V 1 , V 2 and state-action tuples z 1 , z 2 efﬁciently.
We also assume that for any bounded value function V ( · ) : S (cid:55)→ [ − 1 , 1] and any z ∈ Z , (cid:107) φ V ( z ) (cid:107) H ≤ 1 .
Given that the reward function r h ( z ) is known, we obtain through the Bellman equation that
[Equation]
and similar equation holds for Q π, ∗ h .
With these preliminaries prepared, we now move to our main algorithm and its analysis.
A LGORITHM
In this section, we introduce our value targeted iteration algorithm for the two-player zero-sum
Markov Game setting with RKHS function approximation.
We follow the “value-targeted regression” framework and the conﬁdence set design as in UCRL (Jia et al., 2020; Ayoub et al., 2020), and combine the CCE technique (Xie et al., 2020) to deal with the general sum sub-game brought by upper conﬁdence bound (UCB) and lower conﬁdence bound (LCB) value functions.
These techniques enable us to adapt the results in the linear setting to the nonlinear RKHS regime (Chowdhury &
Gopalan, 2017; Yang et al., 2020;
Zhou et al., 2020b) to get a structure-dependent regret bound that is both computationally simple and statistically efﬁcient.
Aiming at ﬁnding an equilibrium ( π ∗ , ν ∗ ) of the value function V π,ν( x 1 ) , we design an algorithm based on value targeted regression (VTR) and upper/lower conﬁdence bound estimations.
As the minplayer targets the minimization of the value function while the max-player targets the maximization of the value function, we use upper conﬁdence bound estimation to encourage exploration of the max-player and use a lower conﬁdence bound to encourage exploration of the min-player.
Thus we need to deﬁne two value functions for the min/max-players respectively, i.e., Q th , Q th , V th , V th , where we adopt the overline notation for the over-estimation of the max-player and the underline notation for the under-estimation of the min-player.
At each round of the game, we solve the following ridge regression problem for minimizing the Bellman error:
[Equation]
Note that in
Eq.
[Equation]
[Equation] [Equation]
only depend on the previous trajectories
[Equation]
We denote the corresponding σ -algebra as F τ − 1 .
Thus we have V τ h
+1 , V τ h +1 ∈ F τ − 1 .
As each V τ h +1 ( x τ h +1 ) can be seen as a stochastic sample of ( P h V τh +1 )
( z τh ) , the regularized regression problem of the max-player in (4.1) can be seen as solving a linear bandit problem with context φ V τh +1 ( z τh ) , reward function ( P h V τh +1 )
( z τh ) and noise term
V τh +1 ( x τh +1 ) − ( P h V τh +1 )
( z τh ) .
A similar statement holds for the min-player as well.
From the solution to the ridge regression problem (4.1), we need to deﬁne the upper/lower conﬁdence bound of the value functions Q ∗ ,νh , Q π, ∗ h respectively.
For simplicity of notations, we deﬁne
[Equation]
and Ψ th := (cid:16)
[Equation] [Equation]
[Equation] [Equation]
[Equation]
(cid:17) (cid:62) .
Also, we deﬁne the gram matrix K th and vector-valued function k th as
[Equation]
and
[Equation]
separately.
We deﬁne K th and k th in a similar way.
For a positive parameter β t > 0 that will be chosen in later analysis, the conﬁdence region centered at θ th in the RKHS H is deﬁned as
[Equation]
where we use (cid:68) Ψ th , θ − θ th (cid:69) H to denote the following t − 1 dimensional vector (cid:16)(cid:68) φ V τh +1 ( z τh ) , θ − θ th (cid:69) H : τ ∈ [ t − 1] (cid:17) .
We omit the deﬁnition of C th which is an analogue of
Eq.
(4.2) by changing all overline symbols to underline ones.
Based on the conﬁdence regions, we construct an optimistic/pessimistic estimate of Q ∗ ,νh as
[Equation]
where Π [ − H,H ] is the projection operator onto [ − H, H ] , which is by deﬁnition the range of value functions.
In fact, Q th has a closed-form solution as follows:
[Equation]
where y th := (cid:104) V 1 h +1 ( x 1 h ) , . . .
V t − 1 h +
( x t − 1 h ) (cid:105) (cid:62) and
b th ( z ) =
λ − 1 / 2 · (cid:20)
k
V th +1 ,V th +
( z, z ) − k th ( z ) (cid:62) (cid:16) K t h
+ λ · I (cid:17) − 1 k th ( z ) (cid:21) 1 / 2 .
An analogous claim holds for Q th .
Given the estimation of
Q th , Q th , the next step is to estimate the corresponding state value functions
V th , V th .
Due to the computational difﬁculty of calculating the NE of a general sum game (Daskalakis et al., 2009), we instead ﬁnd a CCE of the payoff pair ( Q th ( z ) , Q th ( z )) .
More speciﬁcally, we utilize the FIND CCE algorithm in Xie et al.
The full version of the algorithm is presented formally in Algorithm 1.
1: Input: bonus parameter β > 0 .
2: for episode t = 1 , 2 , . . .
, T do 3:
Receive initial state x t4: for step h
= H, H − 1 , . .
, 1 do 5:
Calculate Q th ( · ) , Q th ( · ) as in Eq.
For each x , let σ th ( x ) = FIND CCE ( Q th ,
Q th , x ) 7:
Let V th ( x ) =
E ( a,b ) ∼ σ th ( x ) Q th ( x, a, b ) and V th ( x ) =
E ( a,b ) ∼ σ th ( x ) Q th ( x, a, b ) 8: end for 9: for step h
= 1 , 2 , . .
, H do 10:
Sample a th ∼ π th ( x th ) :
= P 1 σ th ( x th ) , b th ∼ ν th ( x th ) :=
P 2 σ th ( x th ) .
P 1 takes action a th , P 2 takes action b th 12:
Observe next state x th +1 .
13: end for 14: end for
T HEORETICAL A NALYSIS
In this section, we present the regret bound of our algorithm for the kernel mixture Markov Game.
Recall that for the linear function class, the regret upper bound is characterized by the dimension of the linear function, the horizon of the game, and the number of episodes (Chen et al., 2021).
Our analysis in the RKHS function approximation setting aligns with the linear function approximation setting when k V,V (cid:48)
( z, z (cid:48) ) =
φ V ( z ) (cid:62) φ V (cid:48)
( z (cid:48) ) and yields a regret bound of (cid:101)
O
( dH 2 √ T ) .
When considering the nonlinear function class as an approximator of the value function, we need to develop a new concept analogous to the dimension d that characterizes the intrinsic complexity of the function class F .
Under the framework of our theoretical analysis, we present our regret bound in terms of the maximal information gain Γ K ( T, λ ) (Srinivas et al., 2009), the episode number T , and the time horizon H .
We lay out precise deﬁnitions of these notions immediately afterwards.
We ﬁrst deﬁne the effective dimension of the RKHS H with respect to the mixture MG as follows:
Deﬁnition 2 (Srinivas et al. 2009) .
We deﬁne the effective dimension Γ K ( T, λ ) as follows:
[Equation]
for any 1 ≤ i ≤ T, V i : S →
[ − H, H ] , z i ∈
Z , where V i ’s are functions mapping from S to [ − H, H ] and z i ’s are state-action tuples.
Here, K ( { V i } i , { z i } i ) ∈
R T × T and its ( p, q ) -th entry for any 1 ≤ p, q ≤ T is [ K ( { V i } i , { z i } i )] p,q = k V p ,V q ( z p , z q ) .
By the boundedness of φ V as in Section 3.3, it is easy to verify that both the tabular MG and the linear mixture
MG enjoy a ﬁnite effective dimension.
Speciﬁcally, for ﬁnite RKHS H with rank d , Γ K ( T, λ ) =
O ( d · log T ) approximately depicts the rank of H .
Via a concentration argument, we ﬁrst present our main lemma for bounding the estimation error when choosing β t = β for all t ≥ 1 :
Lemma 3.
Assuming that for any h ∈ [ H ] , || θ ∗ h
|| H ≤ B .
Let λ
= 1 + 1 T and β satisﬁes
[Equation]
H (cid:19).
Then for any δ > 0 , with probability at least 1 − δ the following holds for any ( t, h ) ∈
[ T ] ×
[ H ] and any ( x, a, b ) ∈ S ×
A × A :
[Equation]
We are now ready to present the main theorem as follows:
Theorem 4 (RKHS function approximation) .
Assuming that for any h ∈ [ H ] , || θ ∗ h
|| H ≤ B .
Set λ
= 1 + 1 T in the KernelCCE-VTR Algorithm.
For any δ > 0 and any β satisfying
[Equation]
there exists a universal constant c > 0 such that with probability at least 1 − δ , we have
[Equation]
Remark 5.
Theorem 4 suggests that by treating the norm B as a constant, KernelCCE-VTR achieves an (cid:101)
O (Γ K ( T, λ ) H 2 √ T ) regret bound.
When the RKHS degenerates to the Euclidean case, the regret bound corresponding to the standard linear mixture MG case reduces to (cid:101)
O
( dH 2 √ T ) which matches the (cid:101)
O ( dH 3 / 2 √ T ) regret yielded by Chen et al.
up to a √ H factor.
The (cid:101) O ( dH 2 √ T ) regret is also known to be nearly minimax optimal up to a √ H factor (Chen et al., 2021).
Remark 6.
Theorem 4 suggests that the average of the duality gaps 1 /T ·
(cid:80) Tt =1 ( V ∗ ,ν t− V π t , ∗)
= 1 /T ·
Regret ( T ) → 0 , which further indicates that our KernelCCE-VTR algorithm can indeed ﬁnd a good approximation of the Nash Equilibrium ( π th , ν th ) , as the marginalization of the CCE σ th .
N ONLINEAR F UNCTION A PPROXIMATION WITH M ISSPECIFICATION
In Section 5, we focus on the case where the transition probability P h ( s (cid:48) | z ) can be modeled by functions belonging to an RKHS.
In practice, however, the function class may not be conﬁned to an
RKHS, but the distance to it can be bounded.
For example, when considering the more general Neural
Network (NN) function classes, it is well-known that we can approximate it by the RKHS with the corresponding neural tangent kernel (Jacot et al., 2018; Allen-Zhu et al., 2019).
In this section, we attempt to resolve this in two parts, the RKHS approximation with misspeciﬁcation (Section 6.1), and the NN approximation (Section 6.2) as an application of the misspeciﬁcation results.
6. K ERNEL F UNCTION A PPROXIMATION WITH M ISSPECIFICATION
In this subsection, we discuss the case where there exists a misspeciﬁcation error between the RKHS
H and the true transition probability P h ( s (cid:48) | z ) .
Formally, we make the following assumptions:
Assumption 7.
We assume that there exists an ι mis > 0 , an RKHS H with feature mapping φ : Z (cid:55)→ S × H , and an unknown parameter θ ∗
h ∈ H satisfying (cid:107)
θ ∗ h (cid:107) H ≤ B such that for any h ∈
[ H ] , the distance of the transition probability P h to H can be bounded by ι mis :
[Equation]
Similar to Lemma 3 in Section 5, by choosing a proper size β t of the conﬁdence region, we are able to bound the estimation error:
Lemma 8.
Assuming that for any h ∈ [ H ] , || θ ∗ h
|| H ≤ B .
Let λ
= 1 + 1 T and β t satisﬁes
[Equation]
Then for any δ > 0 , with probability at least 1 − δ the following holds for any ( t, h ) ∈
[ T ] ×
[ H ] and any z ∈ Z :
[Equation]
Compared with the choice of β in Lemma 3, Eq.
(6.1) yields an extra O
( Hι mis √ t ) term brought by misspeciﬁcation error.
Now we are ready to present the regret bound when the misspeciﬁcation occurs.
Theorem 9 (RKHS function approximation with misspeciﬁcation) .
Assuming that for any h ∈ [ H ] , || θ ∗ h
|| H ≤ B .
Set λ
= 1 + 1 T in the KernelCCE-VTR Algorithm.
For any δ > 0 and any β t satisfying
[Equation]
there exists a global constant c > 0 such that with probability at least 1 − δ , we have
[Equation]
In words, Theorem 9 suggests that in the misspeﬁciﬁed case, KernelCCE-VTR can achieve the same regret as that in the well-speciﬁed case up to an O ( H 2 Tι mis ) error.
Such a linear dependence on ι mis matches the result of single agent RL for the ﬁnite dimensional case (Jin et al., 2020; Zanette et al., 2020).
6. N EURAL N ETWORK (NN) F UNCTION A PPROXIMATION
The previous subsection focuses on estimating the transition probability using an RKHS when misspeciﬁcation is present.
In this subsection, we utilize Theorem 9 in deriving a regret bound for neural network function approximations.
We see w :=
( x (cid:48) , x, a, b ) (cid:62) =
( x (cid:48) , z ) as a vector in R d that satisﬁes || w ||
= 1 and represent the parameters of a L -Layer fully connected neural network f
by θ :
[Equation] [Equation]
, vec ( W
[Equation]
) (cid:62) (cid:3) (cid:62) , where W i ∈ R m × m for all i ∈
[ L ] .
The neural network f ( w ; θ ) with parameter set θ can be deﬁned as:
[Equation]
where G ( · ) : R (cid:55)→ R is an activation function, W L ∈ R m × 1 , W l ∈ R m × m , 2 ≤ l < L , W 1 ∈ R m × d .
For 1 ≤ l ≤ L − 1 , W l = ( W , 0 ; 0 , W ) , where each entry of W is generated independently from N (0 , 4 /m ) ; W L = ( w (cid:62) , − w (cid:62) ) where each entry of w is generated independently from
N (0 , 2 /m ) .
Given the initialized parameter θ 0 , we take the feature map φ ( w ) = ∇ θ f ( w ; θ 0 ) / √ m as the gradient of f at θ 0 .
We deﬁne the weighted kernel function k V 1 ,V 2 ( · , · ) in Deﬁnition 1 with φ ( w ) .
Similarily, we deﬁne the effective dimension Γ K ( T, λ ) with respect to the kernel function k V 1 ,V 2 ( · , · ) , in the same fashion of Deﬁnition 2.
We assume that for ∀ h ∈ [ H ] our transition probability P h can be modeled by the neural network with parameter θ ∗ h satisfying (cid:13)(cid:13)
θ ∗ h − θ (0) (cid:13)(cid:13)
[Equation] [Equation]
m :
[Equation]
With these at hand we are ready to present our main theorem for NN approximation:
Theorem 10 (NN approximation) .
Assuming that for any h ∈ [ H ] , || θ ∗ h − θ (0)
||
2 ≤ B/ √ m .
There exists a C > 0 independent of m such that if we λ = C 2 (cid:0) 1 + 1 T (cid:1) in the KernelCCE-VTR
Algorithm.
For any δ > 0 and any β t satisfying
[Equation]
there exists a global constant c > 0 such that with probability at least 1 − δ − m − 2 , we have
[Equation]
Theorem 10 suggests that when we use an overparameterized deep neural network ( m (cid:29) 1 ) to approximate the transition dynamic, KernelCCE-VTR achieves an (cid:101)
O (Γ K ( T, λ ) H 2 √ T ) regret, which is of the same order as its counterparts in the general misspeciﬁcation case.
Remark 11.
To match the setting of the misspeciﬁcation case, we consider solving the NN approximation case by a simpliﬁed version of Algorithm 1 that uses only the ﬁrst-order Taylor expansion compared with the full NN solution in Zhou et al.
(2020b)
; Yang et al.
C ONCLUSIONS
In this work, we studied learning two-player mixture MGs using the kernel function approximation.
We introduced a new kernel mixture MG setting and proposed a new algorithm KernelCCE-VTR that utilizes the kernel function of the MG.
We show that our KernelCCE-VTR is able to achieve a sublinear (cid:101) O
( d F H 2 √ T ) regret, which nearly matches the regret lower bound in Zhou et al.
(2021) for learning linear mixture MGs.
We further extend the analysis of the basic RKHS setting to a more general nonlinear function approximation with misspeciﬁcation errors at present and demonstrate that neural networks can be seen as a special application of misspeciﬁcation.