Learning Two-Player Mixture Markov Games: Kernel Function Approximation and Correlated Equilibrium
Abstract
We consider learning Nash equilibria in two-player zero-sum Markov Games withnonlinear function approximation, where the action-value function is approximatedby a function in a Reproducing Kernel Hilbert Space (RKHS).
The key challengeis how to do exploration in the high-dimensional function space.
We proposea novel online learning algorithm to ﬁnd a Nash equilibrium by minimizing theduality gap.
At the core of our algorithms are upper and lower conﬁdence boundsthat are derived based on the principle of optimism in the face of uncertainty.
We prove that our algorithm is able to attain an O ( √ T ) regret with polynomialcomputational complexity, under very mild assumptions on the reward function andthe underlying dynamic of the Markov Games.
We also propose several extensionsof our algorithm, including an algorithm with Bernstein-type bonus that can achievea tighter regret bound, and another algorithm for model misspeciﬁcation that canbe applied to neural network function approximation.
Introduction
Multi-agent reinforcement learning (MARL) has been the focus of research across a range of researchcommunities (Shapley, 1953; Littman, 1994).
The case of two-player Markov Games (MG) hasbeen of particular interest.
In this case, two players select their actions based on the current statesimultaneously and independently.
One player (the max-player) aims to maximize the return basedon the reward provided by the environment, while the other (the min-player) aims to minimize it.
Aseries of recent results have established polynomial sample complexity/regret guarantees that dependon the cardinality of state/action spaces for two-player zero-sum MGs (Wei et al., 2017; Bai & Jin,2020; Bai et al., 2020; Liu et al., 2020; Jia et al., 2019; Sidford et al., 2020; Cui & Yang, 2020;
Lagoudakis & Parr, 2002; Perolat et al., 2015; P´erolat et al., 2016a,b, 2017; Jin et al., 2021b).
Meanwhile, most of the recent successful applications of MARL deal with large state/action spacesthat may be continuous or a ﬁne-grained discretization of a continous space.
Examples include
GO (Silver et al., 2016), autonomous driving (Shalev-Shwartz et al., 2016), TexasHold’em poker(Brown & Sandholm, 2019), and AlphaStar for the game Starcraft (Vinyals et al., 2019).
In order totackle problems with large state/action spaces, researchers have designed MARL algorithms basedon function approximation which approximate the original high-dimensional value function/policyby a function approximator.
For instance, Xie et al.
and
Chen et al.
(2021) studied RL fortwo-player zero-sum MGs with linear function approximation , where it is assumed that there area set of linear features that span the transition kernel and reward function spaces.
In contrast to
RL with linear function approximation, RL with nonlinear function approximation (e.g., kernel andneural network approximation) aims to take advantage of the superior representational power of nonlinear function compared to linear parameterizations.
For example, Jin et al.
(2021c) studiedneural-network-based RL in the setting of MGs with low multi-agent Bellman eluder dimension ,obtaining algorithms that have polynomial dependence on the complexity of the underlying functionclass.
Although this yields a strong theoretical guarantee, the algorithm that they propose is notcomputationally efﬁcient due to the constructed highly nonconvex conﬁdence sets.
The followingquestion is still open: Can we design a computationally and statistically efﬁcient RL algorithm forlearning two-player Markov Games with nonlinear function approximation?
In this paper, we give an afﬁrmative answer to this question for a class of episodic Markov Games,dubbed mixture Markov Games , when using nonlinear approximation function in a Reproducing
Kernel Hilbert Space (RKHS).
We propose a novel kernel-based MARL algorithmic frameworkfor general episodic two-player zero-sum MGs, which provides provable regret guarantees.
Wesummarize the contributions of our work as follows:
• We propose a KernelCCE-VTR algorithm for two-player zero-sum MGs.
In particular, at eachepisode, KernelCCE-VTR uses kernel function approximation to approximate the optimal valuefunction and constructs corresponding conﬁdence sets, following the “Optimism-in-Face-ofUncertainty” principle (Abbasi-Yadkori et al., 2011) to select an action based on the currentstate.
In contrast to algorithms in Jin et al. (2021c), which construct implicit conﬁdence sets thatare in general computationally intractable, our algorithm KernelCCE-VTR crafts a computationally
efﬁcient exploration bonus based on the gram matrix of the kernel function.
• Under the assumption that the transition dynamics belongs to some RKHS, we show that our algo-
rithm KernelCCE-VTR is able to ﬁnd a Nash equilibrium of the game with a (cid:101) O
( d F H 2 √ T ) regret
bound on the duality gap, where H is the horizon, T is the number of the episodes, and d F represents
the complexity of the function class F .
We also propose an extension of KernelCCE-VTR that utilizes weighted kernel ridge regression and a Bernstein-type bonus to achieve (cid:101) O
( d F H 3 / 2 √ T ) regret.
When F reduces to the d -dimensional linear function class, our regret reduces to (cid:101) O
( dH 3 / 2 √ T ) ,which almost matches the lower bound in Chen et al.
(2021).• We also study the general case where the transition dynamic belongs to some RKHS up to amisspeciﬁcation error.
We show that our KernelCCE-VTR can achieve a similar regret as in thewell-speciﬁed case.
In particular, we study the neural network function approximation case whichcan be regarded as a special instance of the misspeciﬁed RKHS case and derive the correspondingregret bound.
Notation.
We use lower case letters to denote scalars, lower and upper case bold letters to denotevectors and matrices.
We use (cid:107) · (cid:107) to indicate Euclidean norm, and for a semi-positive deﬁnite matrixΣ and any vector x , (cid:107) x (cid:107) Σ := (cid:107) Σ 1 / 2 x (cid:107) = √ x (cid:62) Σx .
For real t and interval [ a, b ] , we use Π [ a,b ] [ t ]to indicate the projection of t onto [ a, b ] , i.e. Π [ a,b ] [ t ] = max ( a, min( b, t )) .
For positive integer
N we sometimes deﬁne [ N ] = { 1 , . . .
, N } for compactness.
We also adopt the standard big-
Oand big- Ω notations: say a n = O ( b n ) if and only if there exists C > 0 , N > 0 , for any n > N ,a n ≤ Cb n ; a n = Ω( b n ) if a n ≥ Cb n .
The notations (cid:101) O and (cid:101) Ω are adopted when the C above hides apolylogarithmic factor.
Related work
Online RL with function approximation.
MARL with function approximation can be seen as anextension of RL with function approximation on MDPs.
There are several lines of work studying RLwith function approximation.
The ﬁrst line of work studies the so-called linear MDP which assumesthe reward function and transition dynamics are linear functions of a feature mapping deﬁned on thestate and action spaces (Yang & Wang, 2020; Jin et al., 2020; Zanette et al., 2020).
These workspropose model-free algorithms with sublinear regret on the number of episodes K .
The second lineof work studies the linear mixture MDP which assumes the transition kernel is a linear combinationof several base models (Modi et al., 2020; Jia et al., 2020; Zhou et al., 2021b,a).
These studiesproposed model-based RL algorithms that estimate the transition kernel with ﬁnite sample complexityor sublinear regret guarantees.
The third line of work studies general function approximation whichassumes that either the value function or the transition kernel can be approximated by a general classof functions (Osband & Van Roy, 2014; Jiang et al., 2017; Sun et al., 2019;
Wang et al., 2020; Yanget al., 2020; Du et al., 2021; Jin et al., 2021a).
Algorithms proposed in this line enjoy ﬁnite regretor sample complexity bounds that depend on some general complexity measures such as Eluder dimension (Russo & Van Roy, 2013; Osband & Van Roy, 2014), Bellman rank (Jiang et al., 2017),witness rank (Sun et al., 2019), information gain (Yang et al., 2020), bilinear class (Du et al.
, 2021)and Bellman eluder dimension (Jin et al., 2021a).
Learning two-player MGs with function approximation.
There is a large body of literature on
MARL for two-player MGs with function approximation.
These works can be generally categorizedinto MARL with linear function approximation and MARL with general function approximation .
Forexample, for linear function approximation, Xie et al.
(2020) studied zero-sum simultaneous-move
MGs where both the reward and transition kernel can be parameterized as linear functions of somefeature mappings.
They proposed an OMVI-NI algorithm with an (cid:101) O ( √ d 3 H 3 T ) regret, where d is thenumber of the feature dimension, H is the episode length and T is the total number of rounds.
Chenet al.
(2021) studied the linear mixture MGs and proposed a nearly minimax optimal Nash-UCRLVTR algorithm with an (cid:101) O ( dH √ T ) regret and an Ω( dH √ T ) matching lower bound.
In contrastto this work, our KernelCCE-VTR does not assume the underlying transition dynamic or rewardfunction have a linear structure.
For MARL with general function approximation, Jin et al.
(2021c)studied the two-player zero-sum MGs with low multi-agent Bellman Eluder dimension and proposeda “Golf with Exploiter” algorithm using a general function class.
They showed their algorithm enjoysan (cid:101)
O ( H √ dK log N ) regret, where d is the multi-agent Bellman eluder dimension, K is the numberof episodes.
Huang et al.
(2021) studied two-player MGs with a ﬁnite minimax Eluder dimensionand proposed an ONEMG method with an (cid:101)
O
( H √ dK log N ) regret, where d is the minimax Eluderdimension.
To obtain the desired function approximator, both Golf with Exploiter and ONEMG needto solve a constrained optimization problem, which is computationally intractable even in the linearfunction approximation setting.
In contrast to Jin et al. (2021c) and Huang et al.
(2021), our proposedalgorithms are computationally efﬁcient and nearly-optimal by using the Bernstein-type bonus.
Qiuet al.
(2021) also studied kernel function approximation for two-player MGs.
However, there aretwo key differences between our work and theirs.
First, Qiu et al.
(2021) studied MGs where theexpectation of the value function is in some RKHS; we, on the other hand, assume that the transitiondynamic of the MG lies in an RKHS.
Second, while the regret result in Qiu et al.
(2021) depends onthe covering number of the function space, our regret is independent of the covering number.
Preliminaries
In this section, we present the necessary deﬁnitions that will be adopted throughout the paper.
Section 3.1 describes simultaneous-move games in the setting of zero-sum two-player Markov Games(MG) and recaps the concepts of equilibrium and duality gap that are employed in the game theoryliterature.
Section 3.2 provides necessary deﬁnitions and notations in approximating action valuefunction with functions belonging to a reproducing kernel Hilbert space (RKHS) via modeling thetransition probability.
3.1 Two-player Markov Games
A simpler instance of Markov Games, referred to as turn-based games, can be seen as a special caseof simultaneous-move games.
In a zero-sum two-player simultaneous-move Markov Game, thedynamical structure can be captured by an MG, denoted ( S , A 1 , A 2 , r, P , H ) , where S is the spaceof available states of the environment, A 1 is the action space of the ﬁrst player and A 2 is the actionspace of the second player.
H is the time horizon representing the maximum step of each round ofplay.
The reward function r : { r h ( x, a, b ) : h ∈
[ H ] } is a sequence of mappings from S ×
A 1 ×
A 2to [ − 1 , 1] .
The transition matrix P : { P h ( ·| x, a, b ) : h ∈
[ H ] } gives for each state actions triplet( x, a, b ) and at each time h the stochastic response of the environment to the next x (cid:48) ∈ S .
Here by“simultaneous move” we refer to the setting where at each round of game the two players P 1 and P 2take actions a ∈
A 1 , b ∈
A 2 simultaneously at a given state x ∈ S , in contrast with the turn-basedgame where r h and P h are deﬁned for a state-action pair ( x, a ) where the action can be taken byeither players.
In the context of this paper, for simplicity of notation we let A
=
A 2 =
A , whilethe results can be easily generalized to the case when A 1 (cid:54) =
A 2 .
Similar deﬁnitions of a zero-sumtwo-player simultaneous-move episodic Markov Games can be found in Wei et al.
Perolatet al.
; Xie et al.
In the above setting, two players P 1 and P 2 take actions according to their own strategies.
Weuse π :=
{ π h } h ∈ [ H ] to denote the stochastic policy of P 1 and use ν := { ν h } h ∈ [ H ] to denote thestochastic policy of P 2 .
We note that at time h , π h : S (cid:55)→ ∆
A maps the current state x h to aprobability distribution of the actions, and similarly for ν h .
Given two agents’ policies, π, ν , across hsteps, the state value function is deﬁned as the expected total reward through H steps where at steph ∈ [ H ] player P 1 follows policy π h ( ·| x h ) and player P 2 follows policy ν h ( ·| x h ) ,
[Equation]
and where V π,ν ( x ) :
= V π,ν( x ) .
Note that the expectation is taken over all stochasticity in π h , ν hand P h .
The action value function is deﬁned as
[Equation]
and Q π,ν ( x, a, b ) :
= Q π,ν 1 .
From the deﬁnition of two value functions, we observe that for any x ∈ S ,the state value function given policy pair ( π, ν ) is the expectation of the corresponding action valuefunction
[Equation]
, where the expectation is taken over the action distribution induced by the policy pair.
During thispaper, we use superscript do denote the number of episodes and subscript to denote the number ofhorizon step.
Nash equilibrium and duality gap.
In a zero-sum two-player Markov Game, P 1 wants to maximizethe expected reward V π,ν ( x ) via the choice of the policy π .
On the contrary, P 2 wants to minimize
V π,ν ( x ) by properly choosing ν .
For ﬁxed ν , we deﬁne the best response policy with respect to
V and ν as br ( ν ) and deﬁne V ∗ ,ν h :=
V br ( ν ) ,ν h and Q ∗ ,νh :=
Q br ( ν ) ,ν h , We deﬁne V π, ∗ h :
= V π, br ( π ) hand Q π, ∗ h :=
Q π, br ( π ) h similarly.
A Nash equilibrium is a pair of policies ( π ∗ , ν ∗ ) that are the bestresponse policy for each other, which we write as V π ∗ , ∗ ( x ) =
V π ∗ ,ν ∗ ( x ) =
V ∗ ,ν ∗ ( x ) .
For notationalsimplicity we write V ∗ :
= V π ∗ ,ν ∗ , Q ∗ :=
Q π ∗ ,ν ∗ .
By deﬁnition of the best response policy, weobtain weak duality:
[Equation]
For any policy pair ( π, ν ) , we deﬁne the duality gap as V ∗ ,ν t( x t 1 ) − V π t , ∗( x t 1 ) .
We call a pair an(cid:15)
-approximate
Nash equilibrium (NE) if V ∗ ,ν t( x t 1 ) − V π t , ∗( x t 1 ) ≤ (cid:15) .
We also deﬁne the Regret inthe MG setting as follows:
[Equation]
Coarse Correlated Equilibrium.
We introduce the Coarse Correlated Equilibrium (CCE) notionwhich will be used in our proposed algorithms.
Given payoff matrices Q 1 , Q 2 : S ×
A × A (cid:55)→ Rand the state x , we deﬁne the CCE of the game as a joint distribution σ on A ×
A satisfying:
[Equation]
where P 1 σ denotes the marginal of σ on the ﬁrst coordinate (min-player) and P 2 σ denotes themarginal of σ on the second coordinate (max-player)
We use FIND CCE ( Q 1 , Q 2 , x ) to denote σ .
When σ can be written as a product of two policies over action space A , it is an Nash equilibrium(Xie et al., 2020).
To compute a CCE given Q 1 , Q 2 , x , please see Appendix I.
3.2 Nonlinear function approximation by reproducing kernel Hilbert spaces
For simplicity of notation, we use z =
( x, a, b ) to denote a state-action triplet in Z := S ×
A × A .
An RKHS H with kernel K ( · , · ) : Z × Z (cid:55)→ R is a general form of linear function class.
Every
RKHS H consists of functions on Z , with a feature mapping φ : Z (cid:55)→ H , such that ∀ f ∈ Hand ∀ z ∈ Z , f ( z )
= (cid:104) f, φ ( z ) (cid:105) H .
The kernel K is thus deﬁned for every x,
y ∈
Z × Z as
K ( x, y ) =
(cid:104) φ ( x ) , φ ( y ) (cid:105) H .
We call φ the feature mapping induced by the RKHS H with kernel K .
In the following sections, we use f (cid:62) g as a simpliﬁcation of (cid:104) f, g (cid:105) H when f, g ∈ H .
We make nodistinction in notation between the vector product and the product (cid:104)· , ·(cid:105) H ; the distinction can be readout from the nature of the two objects in the product.
For every RKHS H , there exists a naturaleigenvalue decomposition in L 2 ( Z ) .
RKHS approximation is a generalization of the linear functionapproximation of ﬁnite dimension d which can be inﬁnite dimensional.
In the following, we deﬁnethe so-called kernel mixture MG , which can be regarded as an extension from the linear mixture MDP(Jia et al., 2020; Ayoub et al., 2020;
Zhou et al., 2021a) and linear mixture MG (Chen et al., 2021)
totheir kernel counterpart.
Kernel mixture MG.
In a kernel mixture MG model, we model the transition probability
P h ( s (cid:48) | z ) :
Z (cid:55)→ ∆( S ) as an element in an RKHS H with feature mapping φ ( s (cid:48) | z ) : Z × S → H , such that foran unknown parameter θ ∗ h ∈ H ,
P h ( s (cid:48)
| z ) = (cid:104) φ ( s (cid:48) | z ) , θ ∗ h (cid:105) H for all s (cid:48) ∈ S and z ∈ Z .
A similar MGstructure called kernel MG has been studied by Qiu et al.
(2021), which assumes that the transitionprobability satisﬁes P h ( s (cid:48)
| z ) = (cid:104) φ ( z ) , µ h ( s (cid:48) )
(cid:105) for some φ ( · ) , µ h ( · ) ∈ H .
The single-agent MDPcounterparts of kernel MGs and kernel Mixture MGs are linear MDPs and linear Mixture MDPs.
Zhou et al.
(2021b) have shown that linear MDPs and linear Mixture MDPs are different classes of
MDPs and one cannot be covered by the other.
Following a similar argument, we can also show thatkernel mixture MGs and kernel MGs are different classes of MGs and cannot imply each other.
At time h , for any estimate of the value function V h ( · ) : S (cid:55)→ R , we note that the expectation ofvalue function at time h
+ 1 , P h V
h
+1 is an element in the RKHS P h V
h +
( z ) =
(cid:10)
φ V h
+
( z ) , θ ∗ h (cid:11) H ,where φ V h
+1 ( z ) :
= (cid:80) s (cid:48) ∈S φ ( s (cid:48) | z ) V h
+1 ( s (cid:48) ) integrates the product of the feature mapping with theestimated value of s (cid:48) over S .
It is worth noting that the quantity φ V ( · ) plays an important role inprevious linear mixture model-based algorithms (Jia et al., 2020; Ayoub et al., 2020;
Zhou et al.
,2021a; Chen et al., 2021).
We assume that for any bounded value function V ( · ) : S (cid:55)→ [ − 1 , 1] andany z ∈ Z , (cid:107) φ V ( z ) (cid:107) H ≤ 1 .
Given that the reward function r h ( z ) is known, we obtain through the
Bellman equation that
[Equation]
Weighted kernel function.
In this work, we consider a general RKHS H and do not assume that wecan access the feature mapping φ directly.
Instead, we assume that we can access the weighted kernelfunction k V 1 ,V 2 ( · , · ) , which is deﬁned as follows:
Deﬁnition 1.
For any function pairs V 1 , V 2 : S →
[0 , 1] which map states to real numbers, theweighted kernel function k V 1 ,V 2 ( · , · ) is deﬁned as follows:
[Equation]
It is easy to see from Deﬁnition 1 that
[Equation]
which suggests that the weighted kernel function k V 1 ,V 2 ( · , · ) indeed captures the inner product relationbetween φ V 1 ( z 1 ) and φ V 2 ( z 2 ) .
We assume that we can access an integration oracle that can calculatek V 1 ,V 2 ( z 1 , z 2 ) for any function V 1 , V 2 and state-action tuples z 1 , z 2 efﬁciently.
Algorithm
In this section, we introduce our value-targeted iteration algorithm for the zero-sum two-player
Markov Game setting with RKHS function approximation.
We follow the value-targeted regressionframework and the conﬁdence set design as in UCRL (Jia et al., 2020; Ayoub et al., 2020), andcombine the CCE technique (Xie et al., 2020) to deal with the zero-sum sub-game brought by upper conﬁdence bound (UCB) and lower conﬁdence bound (LCB) value functions.
These techniquesenable us to adapt the results from the linear setting to the nonlinear RKHS regime (Chowdhury &
Gopalan, 2017; Yang et al., 2020;
Zhou et al., 2020) to get a structure-dependent regret bound that isboth computationally simple and statistically efﬁcient.
Algorithm
1: Input: bonus parameter β > 0 .
2: for episode t = 1 , 2 , . . .
, T do 3: for step h
= H, H − 1 , . .
, 1 do 4:
Calculate Q th ( · , · , · ) , Q th ( · , · , · ) as in (4.3) 5:
Let σ th ( · ) =
FIND CCE ( Q th , Q th , · ) 6:
Let V th ( · ) =
E ( a,b ) ∼ σ th ( · ) Q th ( · , a, b ) and
V th ( · ) =
E ( a,b ) ∼ σ th ( · ) Q th ( · , a, b ) 7: end for 8:
Receive initial state x t9: for step h = 1 , 2 , . . .
, H do 10:
Sample ( a th , b th ) ∼ σ th ( x th ) .
P 1 takes action a th , P 2 takes action b th 12:
Observe next state x th +1 .
13: end for 14: end for
To ﬁnd an equilibrium ( π ∗ , ν ∗ ) of the valuefunction V π,ν( x 1 ) , we design an algorithm using value-targeted regression (VTR) and upper/lower conﬁdence bound-based exploration.
As the min-player aims to minimize the valuefunction while the max-player targets to maximize the value function, we use upper conﬁdence bound to encourage the exploration ofthe max-player and use a lower conﬁdencebound to encourage the exploration of the minplayer.
Thus we need to deﬁne two value functions for the min/max-players respectively, i.e.,
Q th , Q th , V th , V th , where we adopt the overlinenotation for the over-estimation by the maxplayer and the underline notation for the underestimation by the min-player.
In the following,we only describe how to estimate the value functions for the max-player, while the value functions for the min-player can be estimated analogously.
At each round of the game, we solve the 240 following ridge regression problem for minimizing the Bellman error:
[Equation]
Note that in (4.1), V τh +1 only depends on the previous trajectories (cid:110) x ji
[Equation]
ji
[Equation]
ji
[Equation]
We denote the corresponding σ -algebra as F τ − 1 .
Thus we have V τh +1 ∈ F τ − 1 .
As each V τh +1 ( x τh +1 )can be seen as a stochastic sample of ( P h V τh +1 )
( z τh ) , the regularized regression problem of themax-player in (4.1) can be seen as solving a linear bandit problem with context φ V τh +1 ( z τh ) , rewardfunction ( P h V τh +1 )
( z τh ) and noise term V τh +1 ( x τh +1 ) − ( P h V τh +1 )
( z τh ) .
From the solution to theridge regression problem (4.1), we can deﬁne the upper/lower conﬁdence bound of the action-valuefunctions Q ∗ ,νh , Q π,
[Equation]
h respectively.
For the simplicity of notation, we deﬁne the vectors Ψ th :=(cid:16) φ
[Equation]
For a positive parameter β t > 0 that will be chosen in later analysis, the conﬁdence region centeredat θ th in the RKHS H is deﬁned as
[Equation]
We omit the deﬁnition of C th which is an analogue of Eq.
(4.2) by changing all overline symbols tounderline ones.
Based on the conﬁdence regions, we construct an optimistic/pessimistic estimate of
Q ∗ ,νh as
[Equation]
where Π [ − H,H ] is the projection operator onto [ − H, H ] , which is by deﬁnition the range of valuefunctions.
For the convenience of conducting an induction argument we deﬁne V tH +1 =
V tH +1 = 0 ,and also V π,νH +1 ( x ) =
and V ∗ ,
ν t H +
=
V π t , ∗ H +1 = 0 , since there is no more future steps starting fromh =
H +1 .
Given the estimation of Q th , Q th , the next step is to estimate the corresponding state valuefunctions V th , V th .
We utilize the FIND CCE algorithm in Xie et al.
to ﬁnd a coarse-correlatedequilibrium of the payoff pair ( Q th ( z ) , Q th ( z )) .
Computational efﬁciency.
By substituting the closed-form solutions to the maximization/minimization problems in (4.3), we can derive the analytic-form for Q th and Q th .
Take Q thas an example, we have
[Equation]
where the gram matrix K th and vector-valued function k th are deﬁned as
[Equation]
(cid:16)
[Equation]
th (cid:17) (cid:62)
[Equation]
Also, we have y th := (cid:104)
V 1 h +
( x 1 h ) , . . .
V t − 1 h +
( x t − 1 h ) (cid:105) (cid:62) and w th ( z ) =
λ − 1 /
[ k V th +1 ,V th +
( z, z ) −k th ( z ) (cid:62) (cid:16) K th + λ · I (cid:17) − 1 k th ( z )] 1 / 2 .
Therefore, by the assumption that the weighted kernel functionk V 1 ,V 2 can be evaluated efﬁciently, Q th and Q th can also be computed efﬁciently.
Furthermore, given
Q th and Q th , FIND CCE can also be implemented efﬁciently (Xie et al., 2020).
Thus, Algorithm 1 iscomputationally efﬁcient.
Main results
In this section, we present the regret bound of our algorithm for the kernel mixture Markov Game.
Recall that for the linear function class, the regret upper bound is characterized by the dimension ofthe linear function, the horizon of the game, and the number of episodes (Chen et al., 2021).
Ouranalysis in the RKHS function approximation setting aligns with the linear function approximationsetting when K ( z, z (cid:48) )
=
φ ( z ) (cid:62) φ ( z (cid:48) ) .
When considering the nonlinear function class as an approximator of the value function, we need todevelop a new concept analogous to the dimension d that characterizes the intrinsic complexity ofthe function class F .
We do so by making use of the maximal information gain, Γ K ( T, λ ) (Srinivaset al., 2009), where T is the episode number and H is the time horizon.
In particular, we deﬁne theeffective dimension of the RKHS H with respect to the mixture MG as follows:
Deﬁnition 2.
We deﬁne the effective dimension Γ K ( T, λ ) as follows:
[Equation]
for any 1 ≤ i ≤ T, V i : S →
[ − H, H ] , z i ∈
Z , where V i ’s are functions mapping from S to[ − H, H ] and z i ’s are state-action tuples.
Here, K ( { V i } i , { z i } i ) ∈
R T × T and its ( p, q ) -th entry forany 1 ≤ p, q ≤ T is [ K ( { V i } i , { z i } i )] p,q = k V p ,V q ( z p , z q ) .
By the boundedness of φ V as in Section 3.2, it is easy to verify that both the tabular MG and thelinear mixture
MG enjoy a ﬁnite effective dimension.
Speciﬁcally, for ﬁnite RKHS H with rank d ,Γ K ( T, λ ) =
O ( d · log T ) approximately depicts the rank of H .
Via a concentration argument, weﬁrst present our main lemma for bounding the estimation error when choosing β t = β for all t ≥ 1 :
Lemma 3.
Assuming that for any h ∈ [ H ] , (cid:107) θ ∗ h (cid:107) H ≤ B .
Let λ
= 1 + 1 /T and β satisﬁes( β/H ) 2 ≥ 2Γ K
( T, λ ) + 2 + 4 · log (1 /δ )
+ 2 λ ( B/H ) 2 .
Then, for any δ > 0 , with probability atleast 1 − δ the following holds for any ( t, h ) ∈
[ T ] ×
[ H ] and any ( x, a, b ) ∈ S ×
A × A :
[Equation]
We are now ready to present our main theorem.
Theorem 4 (RKHS function approximation) .
Under the same conditions as Lemma 3, with probability at least 1 − δ , KernelCCE-VTR has the following regret
[Equation]
Remark 5.
Theorem 4 suggests that by treating the norm B as a constant, KernelCCE-VTR achievesan (cid:101) O
(Γ K ( T, λ ) H 2 √ T ) regret bound.
When the RKHS degenerates to the Euclidean space, theregret bound reduces to (cid:101)
O
( dH 2 √ T ) , which matches the (cid:101)
O ( dH 3 / 2 √ T ) regret for linear mixture
MGs presented by Chen et al.
up to a √ H factor.
Similar to Xie et al.
(2020), by using a standard online-to-batch conversion technique, we can convertthe regret bound in Theorem 4 to a PAC bound.
For simplicity, let the initial states of each episode bethe same, i.e., x t 1 = x 1 .
After T episodes, we select t 0 ∈
[ T ] satisfying
[Equation]
which yields the following sample complexity guarantee for ﬁnding an (cid:15) -approximate NE policy pair( π t 0 , ν t 0 ) .
Corollary 6 (Sample complexity) .
Under the same condition as Theorem 4, by setting T =
O (cid:0) β 2 H 2 Γ K ( T, λ ) /(cid:15) 2 (cid:1) =
(cid:101) O (cid:0) H 4 Γ 2 K ( T, λ ) /(cid:15) 2 (cid:1) and selecting t 0 as in (5.1), the policy pair ( π t 0 , ν t 0 )is an (cid:15) -approximate NE.
Bernstein-type bonus, misspeciﬁcation, and neural functionapproximation
In this section, we propose several extensions of KernelCCE-VTR .
Section 6.1 introduces
KernelCCE-VTR with Bernstein-type bonus.
Section 6.2 discusses kernel function approximationwith misspeciﬁcation.
We also specialize the kernel function approximation with misspeciﬁcation tothe neural function approximation setting, which is deferred to Appendix C due to space limit.
6.1 KernelCCE-VTR with Bernstein-type bonus
Recall that in KernelCCE-VTR , we need to choose β in order to calculate the optimistic and pessimistic state-action value functions
Q th ( · ) , Q th ( · ) deﬁned in (4.3).
The theoretical value of β isdeﬁned in Lemma 3, which controls the uncertainty of the action-value estimate.
Such choice of β isdue to a Hoeffding-type concentration used in the proof of Lemma 3.
It has been shown in Zhou et al.(2021a)
that by using a Bernstein-type bonus and a sharp analysis based on the total variance lemma,one can obtain an improved algorithm with a tighter regret bound.
Following this idea, we propose a
KernelCCE-VTR+ algorithm, which replaces the Hoeffding-type bonus with a Bernstein-type bonus.
To demonstrate the construction of the Bernstein-type bonus, we take the max player for example.
Inparticular, we solve the following weighted kernel ridge regression problem:
[Equation]
where the input is the normalized feature mapping φ V τh +1 ( z τ h ) /R
τ h , the output is the normalized valuefunction V τh +1 ( x τh +1 ) /R
τh , and the normalization factor R τh is an upper bound on the conditionalvariance of V τh +1 ( x τh +1 ) .
It is straightforward to verify that (6.1) admits a closed-form solution.
Given that solution, we can compute the upper conﬁdence bound of the action-value functions
Q ∗ ,νh .
In detail, we deﬁne Ψ th, 1 :=
(cid:16) φ V 1 h
+1 ( z 1 h ) /R
1 h , . .
. , φ V t − 1 h +1 ( z t − 1 h ) /R
t − 1 h (cid:17) (cid:62) ∈
H t − 1 .
The grammatrix K th, 1 , vector-valued function k th, 1 and the conﬁdence region centered at θ th, 1 in the RKHS Hcan be calculated the same as in Algorithm 1, except that Ψ th is replaced by Ψ th, 1 .
Then the optimisticestimate of the action-value function Q ∗ ,νh has the following form:
[Equation]
where y th, 1 :=
(cid:104) V 1 h
+
( x 1 h ) /R
1 h , . . .
V t − 1 h +
( x t − 1 h ) /R
t − 1 h (cid:105) (cid:62) and
[Equation] [Equation]
[Equation]
Due to space limit, we defer the conditional variance estimator R th to Appendix B. Similarly, wecan construct the pessimistic estimate of the action-value function Q π, ∗ h for the min player.
We havethe following informal result for KernelCCE-VTR+ .
The full algorithm and its formal result can befound in Appendix B.
Theorem 7 (Informal) .
Let d eff =
Γ K ( T, λ ) , with proper choice of R th , R th and β t , with probabilityat least 1 − δ , KernelCCE-VTR+ has the following regret
[Equation]
Remark 8.
When T is sufﬁciently large and Γ K ( T, λ ) is larger than H , the regret bound in Theorem7 is dominated by (cid:101)
O (Γ K ( T, λ ) H 3 / 2 √ T ) , which improves the (cid:101)
O (Γ K ( T, λ ) H 2 √ T ) regret derivedin Theorem 4 by a factor of √ H .
Compared with the (cid:101) Ω( dH 3 / 2 √ T ) lower bound proposed in Chenet al.
(2021), our KernelCCE-VTR+ algorithm is almost optimal when it reduces to the linear mixture
MG.
6.2 Kernel function approximation with misspeciﬁcation
In this subsection, we consider the case where the function class may not be conﬁned to an RKHS, butinstead the distance to it can be bounded.
This can be formulated as kernel function approximationwith misspeciﬁcation.
We assume that there exists a misspeciﬁcation error between the RKHS H andthe true transition probability P h ( s (cid:48) | z ) .
Assumption 9.
There exists an ι mis > 0 , an RKHS H with feature mapping φ : Z (cid:55)→ S × H , andan unknown parameter θ ∗
h ∈ H satisfying (cid:107)
θ ∗ h (cid:107) H ≤ B such that for any h ∈
[ H ] , the distance of thetransition probability P h to H can be bounded by ι mis , which is (cid:13)(cid:13) P h ( ·| z ) − (cid:104) φ ( ·| z ) , θ ∗ h (cid:105) H (cid:13)(cid:13) TV ≤ ι mis .
In order to deal with model misspeciﬁcation, the key idea is to enlarge β t in the deﬁnition of theoptimistic action-value function in (4.3).
More speciﬁcally, we will add an extra O
( Hι mis √ t ) termbrought by misspeciﬁcation error to β speciﬁed in Lemma 3.
We can show that KernelCCE-VTRwith such enlarged β will have a sublinear regret in the presence of misspeciﬁcation.
Theorem 10 (RKHS function approximation with misspeciﬁcation) .
Assuming that for any h ∈ [ H ] ,(cid:107) θ ∗ h (cid:107) H ≤ B .
Set λ
= 1 + 1 /T in the KernelCCE-VTR Algorithm.
For any δ > 0 and any β tsatisfying ( β t /H ) 2 ≥ 2Γ K ( T, λ ) + 3 + 6 · log (1 /δ )
+ 3 λ ( B/H )
+ 3 ι 2mis t , there exists a globalconstant c > 0 such that with probability at least 1 − δ , we have
Regret( T ) ≤ c (cid:16) β T H (cid:112) T · Γ K ( T, λ ) +
+ H 2 Tι mis (cid:17) .
In words, Theorem 10 suggests that in the misspeﬁciﬁed case, KernelCCE-VTR can achieve thesame regret as that in the well-speciﬁed case up to an O ( (cid:112) Γ K ( T, λ )
H 2 Tι mis ) error.
Such a lineardependence on ι mis matches the result of single agent RL for the ﬁnite dimensional case (Jin et al.
,2020; Zanette et al., 2020).
Conclusions
In this work, we studied learning for two-player mixture MGs using a kernel function approximation.
We introduced a new formulation of kernel mixture MGs and proposed an algorithm KernelCCE-VTRthat exploits the kernel function of the MG.
We show that our KernelCCE-VTR is able to achieve asublinear (cid:101) O
( d K H 2 √ T ) regret.
We further improve our algorithm with a Bernstein-type bonus andweighted kernel ridge regression , which enjoys a better (cid:101) O
( d K H 3 / 2 √ T ) regret and nearly matchesthe regret lower bound in Chen et al.
(2021) when reducing to linear mixture MGs.
Finally, weextend our analysis of the basic RKHS setting to a more general nonlinear function approximationsetting with misspeciﬁcation errors and demonstrate that neural networks can be treated as a specialinstance of this misspeciﬁcation framework.
We believe our framework and analysis greatly broadensthe expressiveness of the function classes.
We leave to study learning general-sum MGs by kernelfunction approximation as future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Section[-
-]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]