D ECENTRALIZED O PTIMISTIC H YPERPOLICY M IRROR D ESCENT : P ROVABLY N O -R EGRET L EARNING IN
M ARKOV G AMES
Anonymous authors
Paper under double-blind review
A BSTRACT
We study decentralized policy learning in Markov games where we control a single agent to play with nonstationary and possibly adversarial opponents.
Our goal is to develop a no-regret online learning algorithm that (i) takes actions based on the local information observed by the agent and (ii) is able to ﬁnd the best policy in hindsight.
For such a problem, the nonstationary state transitions due to the varying opponent pose a signiﬁcant challenge.
In light of a recent hardness result [33], we focus on the setting where the opponent’s previous policies are revealed to the agent for decision making.
With such an information structure, we propose a new algorithm, Decentralized Optimistic hypeRpolicy mIrror deScent (DORIS), which achieves √ K -regret in the context of general function approximation where K is the number of episodes.
Moreover, when all the agents adopt DORIS, we prove that their mixture policy constitutes an approximate coarse correlated equilibrium.
In particular,
DORIS maintains a hyperpolicy which is a distribution over the policy space.
The hyperpolicy is updated via mirror descent, where the update direction is obtained by an optimistic variant of least-squares policy evaluation.
Furthermore to illustrate the power of our method, we apply DORIS to constrained and vector
-valued MDPs, which can be formulated as zero-sum Markov games with a ﬁctitious opponent.
I NTRODUCTION
Multi-agent reinforcement learning (MARL) studies how each agent learns to maximize its cumulative rewards by interacting with the environment as well as other agents, where the state transitions and rewards are affected by the actions of all the agents.
Equipped with powerful function approximators such as deep neural networks [31], MARL has achieved signiﬁcant empirical success in various domains including the game of Go [47], StarCraft [50], DOTA2 [5], Atari [38], multi-agent robotics systems [8] and autonomous driving [45].
Compared with the centralized setting where a central controller collects the information of all agents and coordinates their behaviors, decentralized algorithms [19, 42] where each agent autonomously chooses its action based on its own local information are often more desirable in MARL applications.
In speciﬁc, decentralized methods (1) are easier to implement and enjoy better scalability, (2) are more robust to possible adversaries, and (3) require less communication overhead [21, 22, 9, 59, 18].
In this work, we aim to design a provably efﬁcient decentralized reinforcement learning (RL) algorithm in the online setting with function approximation.
In the sequel, for the ease of presentation, we refer to the controllable agent as the player and regard the rest of the agents as a meta-agent, called the opponent , which speciﬁes its policies arbitrarily.
Our goal is to maximize the cumulative rewards of the player in the face of a possibly adversarial opponent, in the online setting where the policies of the player and opponent can be based on adaptively gathered local information.
From a theoretical perspective, arguably the most distinctive challenge of the decentralized setting is nonstationarity .
That is, from the perspective of any agent, the states transitions are affected by the policies of other agents in an unpredictable and potentially adversarial way and are thus nonstationary.
This is in stark contrast to the centralized setting which can be regarded as a standard RL problem for the central controller which decides the actions for all the players.
Furthermore, in the online setting, as the environment is unknown, to achieve sample efﬁciency, the player needs to strike a balance between exploration and exploitation in the context of function approximation and in the presence of an adversarial opponent.
The dual challenges of nonstationarity and efﬁcient exploration are thus intertwined, making it challenging to develop provably efﬁcient decentralized MARL algorithms.
Consequently, there seem only limited theoretical understanding of the decentralized MARL setting with a possibly adversarial opponent.
Most of the existing algorithms [7, 53, 49, 27, 23] can only compete against the Nash value of the Markov game when faced with an arbitrary opponent.
This is a much weaker baseline compared with the results in classic matrix games [17, 1] where the player is required to compete against the best ﬁxed policy in hindsight .
Meanwhile, [33] seems the only work we know that can achieve no-regret learning in MARL against the best hindsight policy, which focuses on the policy revealing setting where the player observes the policies played by the opponent in previous episodes.
However, the algorithm and theory in this work are limited to tabular cases and fail to deal with large or even continuous state and action space.
To this end, we would like to answer the following question:
Can we design a decentralized MARL algorithm that provably achieves no-regret against the best ﬁxed policy in hindsight in the context of function approximation?
In this work, we provide a positive answer to the above question under the policy revealing setting with general function approximation.
In speciﬁc, we propose an actor-critic-type algorithm [29] called DORIS , which maintains a distribution over the policy space, named hyperpolicy , for decisionmaking.
To combat the nonstationarity, DORIS updates the hyperpolicy via mirror descent (or equivalently, Hedge [16]).
Furthermore, to encourage exploration, the descent directions of mirror descent are obtained by solving optimistic variants of policy evaluation subproblems with general function approximation, which only involve the local information of the player.
Under standard regularity assumptions on the underlying function classes, we prove that DORIS achieves a sublinear regret in the presence of an adversarial opponent.
In addition, when the agents all adopt DORIS independently, we prove that their average policy constitutes an approximate coarse correlated equilibrium.
At the core of our analysis is a new complexity measure of function classes that is tailored to the decentralized MARL setting.
Furthermore, to demonstrate the power of DORIS , we adapt it for solving constrained Markov decision process (CMDP) and vector-valued Markov decision process (VMDP), which can both be formulated as a zero-sum Markov game with a ﬁctitious opponent.
Our Contributions.
Our contributions are four-fold.
First, we propose a new decentralized policy optimization algorithm, DORIS , that provably achieves no-regret in the context of general function approximation.
As a result, when all agents adopt DORIS , their average policy converges to a CCE of the Markov game.
Secondly, we propose a new complexity measure named Bellman Evaluation
Eluder dimension, which generalizes Bellman Eluder dimension [25] for single-agent MDP to decentralized learning in Markov games, which might be of independent interest.
Third, we modify
DORIS for solving CMDP with general function approximation, which is shown to achieve sublinear regret and constraint violation.
Finally, we extend DORIS to solving the approchability task [36] in vector-valued Markov decision process (VMDP) and attain a near-optimal solution.
To our best knowledge, DORIS seems the ﬁrst provably efﬁcient decentralized algorithm for achieving no-regret in MARL with general function approximation.
Notations.
In this paper we let [ n ] = { 1 , · · · , n } for any integer n .
We denote the set of probability distributions over any set S by ∆ S or ∆( S ) .
We also let (cid:107) · (cid:107) denote the (cid:96) 2 -norm by default.
Related works.
Our work is related to the bodies of literature on decentralized learning with an adversarial opponent, ﬁnding equilibria in self-play Markov games, CMDPs and VMDPs.
These works either consider centralized setting or do not have function approximation in decentralized online setting.
Due to the page limit, we compare to these works in Appendix B.
P RELIMINARIES 2. M ARKOV G AMES
Let us consider an n -agent general-sum Markov game (MG) M MG
=
( S , {A i } ni =1 , { P h } Hh =1 , { r h,i } H,nh =1 ,i =1 , H ) , where S is the state space,
A i is the action space of the i -th agent, P h : S × (cid:81) ni
=1 A i → ∆( S ) is the transition function at the h -th step, r h,i : S × (cid:81) ni
=1 A i → R + is the reward function of the i -th agent at the h -th step and H is the length of each episode.
We assume each episode starts at a ﬁxed start state s 1 and terminates at s H +1 .
At step h ∈
[ H ] , each agent i observes the state s h and takes action a h,i simultaneously.
After that, agent i receives its own reward r h,i ( s h , a h ) where a h :=
( a h, 1 , · · · , a h,n ) is the joint action and the environment transits to a new state s h +1 ∼ P h ( ·| s h , a h ) .
Policy.
A policy of the i -th agent µ i = { µ h,i : S → ∆ A i } h ∈ [ H ] speciﬁes the action selection probability of agent i in each state at each step.
In the following discussion we will drop the h in µ h,i when it is clear from the context.
We use π to represent the joint policy of all agents and µ − i to denote the joint policy of all agents other than i .
Further, we assume each agent i chooses its policy from a policy class Π i .
Similarly, let Π − i :=
(cid:81) j (cid:54) =
i Π j denote the product of all agents’ policy classes excluding the i -th agent.
Value functions and Bellman operators.
Given any joint policy π , the i -th agent’s value function
V πh,i : S → R and action-value (or Q) function Q πh,i : S × (cid:81) ni
=1 A i → R characterize its expected cumulative rewards given a state or a state-action pair, which are deﬁned as below:
[Equation]
where the expectation is w.r.t.
to the distribution of the trajectory induced by executing the joint policy π in M MG .
Here we suppose the action-value function is bounded:
[Equation]
Notice that when the reward function is bounded in [0 , 1] , V max
= H naturally.
2. D ECENTRALIZED P OLICY L EARNING
In this paper we consider the decentralized learning setting [27, 23, 33] where only one agent is under our control, which we call player , and the other agents can be adversarial.
Without loss of generality, assume that we can only control agent 1 and view the other agents as a meta opponent .
To simplify writing, we use a h , A , r h , µ, Π , V πh , Q πh to denote a h, 1 , A 1 , r h, 1 , µ 1 , Π 1 , V πh, 1 , Q πh, 1 respectively.
We also use b h , B , ν, Π (cid:48) to represent the action, the action space, the policy and the policy class of the meta opponent.
By decentralized we mean during the episode, the player can only observe its own rewards, actions and some information of the opponent speciﬁed by the protocol, i.e., { s th , a th , J th , r th } Hh =1 where {J h } Hh =1 is the information revealed by the opponent in each episode and we will specify it later.
At the beginning of the t -th episode, the player chooses a policy µ t from its policy class Π based only on its local information collected from previous episodes, without any coordination from a centralized controller.
Meanwhile, the opponent selects ν t from Π (cid:48) secretly and probably adversely.
The learning objective is to minimize the regret of the player by comparing its performance against the best ﬁxed policy in hindsight as standard in online learning literature [1, 20]:
Deﬁnition 1 (Regret) .
Suppose ( µ t , ν t ) are the policies played by the player and the opponent in the t -th episode.
Then the regret for K episodes is deﬁned as
[Equation]
where µ × ν denotes the joint policy where the player and the opponent play µ and ν independently.
We also use π t to denote µ t × ν t . Achieving low regrets deﬁned in (1) indicates that if we sample a policy µ uniformly from { µ t } Kt = at random, the resulting mixture policy will be close to the best ﬁxed policy in hindsight.
Relation between Deﬁnition 1 and equilibria.
An inspiration for our deﬁnition of regrets comes from the tight connection between low regrets and equilibria in the matrix game [17, 6, 10].
By viewing each policy in the policy class as a pure strategy in the matrix game, we can generalize the notion of equilibria in matrix games to Markov games naturally.
In particular, a correlated mixed strategy proﬁle π can be deﬁned as a mixture of the joint policy of all agents, i.e., π ∈ ∆( (cid:81) i ∈
[ n ] Π i ) .
Suppose the marginal distribution of π over the policy of agent i is µ i , then we can see that µ i is a mixture of the policies in Π i .
For a correlated proﬁle, the agents might not play their mixed policies µ i independently, which means that π might not be the product of µ i .
A coarse correlated equilibrium (CCE) is a correlated proﬁle that all the agents have no incentive to deviate from by playing a different independent policy:
Deﬁnition 2 ( (cid:15) -approximate coarse correlated equilibrium (CCE) for n -player MG) .
A correlated strategy proﬁle π is an (cid:15) -approximate coarse correlated equilibrium if we have for all i ∈ [ n ]
[Equation]
where µ − i is the marginal distribution of π over the joint policy of all agents other than i .
Remark 1.
Our deﬁnition of correlated strategy proﬁle and CCEs is slightly different from [35].
This is because we are considering with policy classes while [35] does not.
In fact, our deﬁnition is more strict in the sense that a correlated proﬁle satisfying our deﬁnition must also satisfy theirs.
Specially, if a CCE π satisﬁes π = (cid:81) i ∈ [ n ] µ i , it is also called a Nash Equilibrium (NE).
We will use our algorithm as an example to show that if a decentralized algorithm can achieve low regrets under
Deﬁnition 1, we will be able to ﬁnd a CCE by running the algorithm independently for each agent just like in classic matrix games.
2. F UNCTION A PPROXIMATION
To deal with the potentially large or even inﬁnite state and action space, we consider learning with general value function approximation in this paper [24, 25].
We assume the player is given a function class F = F 1 ×·
· ·×F H +1 ( F h ⊆ ( S ×A×B → [0 , V max ]) ) to approximate action-value functions.
Since there is no reward in state s H +1 , we let f H +
( s, a, b )
=
for all s ∈ S , a ∈ A , b ∈ B , f ∈ F .
To measure the size of F , we use |F| to denote its cardinality when F is ﬁnite.
For inﬁnite function classes, we use (cid:15) -covering number to measure its size, which is deﬁned as follows:
Deﬁnition 3 ( (cid:15) -covering number) .
The (cid:15) -covering number of F , denoted by N F ( (cid:15) ) , is the minimum integer n such that there exists a subset F (cid:48)
⊂ F with |F (cid:48) | = n and for any f ∈ F there exists f (cid:48) ∈ F (cid:48) such that max h ∈
[ H ] (cid:107) f
h − f (cid:48) h (cid:107) ∞ ≤ (cid:15) .
In addition to the size, we also need to impose some complexity assumption on the structure of the function class to achieve small generalization error.
Here we introduce one of such structure complexity measures called Distributional Eluder (DE) dimension [25], which we will utilize in our subsequent analysis.
First let us deﬁne independence between distributions:
Deﬁnition 4 ( (cid:15) -independence between distributions) .
Let W be a function class deﬁned on X , and ρ, ρ 1 , · · · , ρ n be probability measures over X .
We say ρ is (cid:15) -independent of { ρ 1 , · · · , ρ n } with respect to W if there exists w ∈ W such that (cid:112)(cid:80) ni
=
( E ρ i [ w ]) 2 ≤ (cid:15) but | E ρ [ w ] | > (cid:15) .
From the deﬁnition we can see that a probability distribution ρ is independent from { ρ 1 , · · · , ρ n } if there exists a discriminator function in W such that the function values are small at { ρ 1 , · · · , ρ n } while large at ρ .
DE dimension is simply the length of the longest sequence of independent probability measures that the function class can discriminate:
Deﬁnition 5 (Distributional Eluder (DE) dimension) .
Let W be a function class deﬁned on X , and
Q be a family of probability measures over X .
The distributional Eluder dimension dim DE ( W , Q , (cid:15) ) is the length of the longest sequence { ρ 1 , · · · , ρ n } ⊂ Q such that there exists (cid:15) (cid:48) ≥ (cid:15) where ρ i is (cid:15) (cid:48) -independent of { ρ 1 , · · · , ρ i − 1 } for all i ∈ [ n ] .
Eluder dimension, another commonly-used complexity measure proposed by [43], is a special case of DE dimension.
If we choose Q = { δ x ( · ) | x ∈ X} where δ x ( · ) is the dirac measure centered at x , then the Eluder dimension can be formulated as
[Equation]
where W − W = { w 1 − w 2 : w 1 , w 2 ∈ W} .
Many function classes in MDPs are known to have low
Eluder dimension, including linear MDPs [28], generalized linear complete models [52] and kernel
MDPs [25].
We also assume the existence of an auxiliary function class G =
G 1 ×· · ·×G H ( G h ⊆ ( S ×A×B → [0 , V max ]) ) to capture the results of applying Bellman operators on F as in [25, 27].
When F satisﬁes completeness (Assumption 3), we can simply choose G = F .
A LGORITHM : DORIS
Policy revealing setting.
Recall that in decentralized policy learning setting, the player is also able to observe some information of the opponent, denoted by J h , aside from its own actions and rewards.
There have been works studying the case where J h = ∅ [49] and J h
= b h [27, 23] in two-player zero-sum games.
However, their benchmark is the Nash value of the Markov game, i.e., V µ ∗ × ν ∗( s 1 ) where µ ∗ × ν ∗ is an NE, which is strictly weaker than our benchmark max µ ∈ Π (cid:80) Kt =1 V µ × ν t( s 1 ) in two-player zero-sum games.
In fact, [33] has showed achieving low regrets under Deﬁnition 1 is exponentially hard in tabular cases when the opponent’s policies are not revealed (see Appendix E. for details).
Therefore in this paper we let J h
= { b h , ν h } just like [33] and call this information structure policy revealing setting.
That said, even in policy revealing setting, the challenge of nonstationarity still exists because the opponent’s policy can be adversarial and only gets revealed after the player plays a policy.
Thus from the perspective of the player, the transition kernel P νh ( ·| s, a ) :
= E b ∼ ν h ( s ) P h ( ·| s, a, b ) still changes in an unpredictable way across episodes.
In addition, the problem of how to balance exploration and exploitation with general function approximation also remains due to the unknown transition probability.
In this section we propose DORIS , an algorithm that is capable of handling both these challenges and achieving a √ K regret upper bound in policy revealing setting. Remark 2.
When the opponent’s policy is not revealed but changes slowly, we indeed can infer the opponent’s policy approximately via the procedures in [39, 46] and this can be viewed as an **approximate policy revealing condition** in practice.
DORIS .
Intuitively, our algorithm is an actor-critic / mirror descent (Hedge) algorithm where each policy µ in Π is regarded as an expert and the performance of each expert at episode t is given by the value function of V µ
× ν t( s 1 ) .
We call it Decentralized Optimistic hypeRpolicy mIrror deScent ( DORIS ).
DORIS possesses three important features, whose details are shown in Algorithm 1:
• Hyperpolicy and Hedge: Motivated from the adversarial bandit literature [1, 20, 30] and no-regret learning works [33], DORIS maintains a distribution p over the policies in Π , which we call hyperpolicy , to combat the nonstaionarity.
The hyperpolicy is updated using Hedge, with the reward of each policy µ being an estimation of the value function V µ × ν t s 1 ) .
This is equivalent to running mirror ascent algorithm over the policy space Π with the gradient being V µ × ν t ( s 1 )
• Optimism:
However, we do not have access to the exact value function since the transition probability is unknown, which forces us to deal with the exploration-exploitation tradeoff.
Here we utilize the Optimism in the Face of Uncertainty principle [2, 28, 25, 27, 23] and choose our estimation V t ( µ ) to be optimistic with respect to the true value V µ × ν t( s 1 ) .
In this way DORIS will prefer policies with more uncertainty and thus encourage exploration in the Markov game.
• Optimistic policy evaluation with general function approximation: Finally we need to design an efﬁcient method to obtain such optimistic estimation V t ( µ ) with general function approximation.
Here we propose OptLSPE to accomplish this task. In
short, OptLSPE constructs a conﬁdence set for the target action-value function Q
µ × ν based on the player’s local information and
chooses an optimistic estimation from the conﬁdence set
, as shown in
Algorithm The construction of the conﬁdence set utilizes the fact that Q µ ×
ν h
satisﬁes the Bellman equation [41]:
[Equation]
+1 ( s (cid:48) , µ, ν )] , where Q µ × ν h +1 ( s (cid:48) , µ, ν ) = E a (cid:48) ∼ µ ( ·| s (cid:48) ) ,b (cid:48) ∼ ν ( ·| s (cid:48) )
[ Q µ × ν h +1 ( s (cid:48) , a (cid:48) , b (cid:48) )] .
We call T µ,νh the Bellman operator induced by µ × ν at the h -th step.
Then the construction rule of B D ( µ, ν ) is based on least-squares policy evaluation with slackness β as below:
[Equation]
where L D is the empirical Bellman residuals on D :
[Equation]
Input learning rate η conﬁdence parameter for t = 1 , · · · , K do
V t ( µ ) ← OptLSPE ( µ, ν t , D 1: t − 1 , F , G , β ) , ∀ µ ∈ Π .
p t +1 ( µ ) ∝ p t ( µ ) · exp( η · V t ( µ )) , ∀ µ ∈ Π .
Decentralized Algorithm. Here we want to highlight that DORIS is a decentralized algorithm because the player can run DORIS based only on its local information, i.e., { s h , a h , J h , r h } , and we do not make any assumptions on the policies of the opponent. We also discuss the computational complexity of DORIS in Appendix E.3.
Comparison with OPMD [33].
The hyperpolicy and Hedge part of DORIS indeed follows OPMD proposed by [33], which is an algorithm for no-regret learning in tabular MG.
The novelty of DORIS lies in the new policy evaluation algorithm specially designed for the policy revealing setting that can tackle general function approximation.
We also need to propose new techniques to analyze the performance of DORIS , which is more complicated than tabular cases.
See Appendix E.1 for more detailed comparison.
3. DORIS IN S ELF -P LAY S ETTING
Apart from decentralized learning setting with a possibly adversarial opponent, we are also interested in the self-play setting where we can control all the agents and need to ﬁnd an equilibrium for the nagent general-sum Markov game.
Inspired by the existing relationships between no-regret learning and
CCEs in matrix games [17, 6, 10], a natural idea is to simply let all agents run DORIS independently.
To achieve this, we assume each agent i is given a value function class F i = F 1 ,i × · · · × F H,i and an auxiliary function class
G i =
G 1 ,i × · · · × G H,i as in DORIS , and run DORIS with learning rate η i and conﬁdence parameter β i by viewing the other agents as its opponent.
Suppose the policies played by agent i during K episodes are { µ ti } Kt =1 , then we output the ﬁnal joint policy as a uniform mixture:
[Equation]
See Algorithm 3 for more details.
Remark 3.
Algorithm 3 is also a decentralized algorithm since every agent runs their local algorithm independently without coordination.
The only step that requires centralized control is the output process where all the agents need to share the same iteration index, which is also required in the existing decentralized algorithms [35, 26].
T HEORETICAL G UARANTEES
In this section we analyze the theoretical performance of DORIS in decentralized policy learning and self-play setting.
We ﬁrst introduce a new complexity measure for function classes, called Bellman
Evaluation Eluder (BEE) dimension, and then illustrate the regret and sample complexity bounds based on this new measure.
4. B ELLMAN E VALUATION E LUDER D IMENSION
Motivated from Bellman Eluder (BE) dimension in classic MDPs and its variants in MGs [25, 27, 23], we propose a new measure speciﬁcally tailored to the decentralized policy learning setting, called
Bellman Evaluation Eluder (BEE) dimension.
First, for any function class F , we deﬁne ( I−T Π , Π (cid:48) h ) F to be the Bellman residuals induced by the policies in Π and Π (cid:48) :
[Equation]
Then Bellman Evaluation Eluder (BEE) dimension is the DE dimension of the Bellman residuals induced by the policy class Π and Π (cid:48) on function class F :
Deﬁnition 6.
The (cid:15) -Bellman Evaluation Eluder dimension of function class F on distribution family
Q with respect to the policy class Π × Π (cid:48) is deﬁned as follows:
[Equation]
BEE dimension is able to capture the generalization error of evaluating value function V µ × ν where µ ∈ Π , ν ∈ Π (cid:48) , which is one of the most essential tasks in decentralized policy space optimization as shown in DORIS .
Similar to [25, 27, 23], we mainly consider two distribution families for Q here:
• Q 1 = {Q 1 h } h ∈
[ H ] : the collection of all probability measures over S ×
A × B at each step when executing ( µ, ν
) ∈ Π × Π (cid:48) .
• Q 2 =
{Q 2 h } h ∈ [ H ] : the collection of all probability measures that put measure 1 on a single state-action pair ( s, a, b ) at each step.
We also use dim BEE ( F , (cid:15), Π , Π (cid:48) ) to denote min { dim BEE ( F , (cid:15), Π , Π (cid:48) , Q 1 ) , dim BEE ( F , (cid:15), Π , Π (cid:48) , Q 2 ) } for simplicity in the following discussion.
Relation with Eluder dimension.
To illustrate the generality of BEE dimension, we show that all function classes with low Eluder dimension also have low BEE dimension, as long as completeness (Assumption 3) is satisﬁed.
More speciﬁcally, we have the following proposition and its proof is deferred to Appendix F:
Proposition 1.
Assume F satisﬁes completeness, i.e., T µ,νh
[Equation]
[ H ] .
Then for all (cid:15) > 0 , we have
[Equation]
Inequality (4) shows that BEE dimension is always upper bounded by Eluder dimension when completeness is satisﬁed.
With Proposition 1, Appendix H validates that kernel
Markov games (including tabular Markov games and linear Markov games) and generalized linear complete models all have small Bellman Evaluation Eluder Dimension.
Furthermore, in this case the upper bound of
BEE dimension does not depend on Π and Π (cid:48) , which is a desirable property when Π and Π (cid:48) is large.
Comparison with multi-agent BE dimension.
Jin et al.
[27], Huang et al.
[23] also propose a variant of BE dimension for Markov games called multi-agent BE dimension.
However, this complexity measure and its analysis techniques are not applicable in our setting because DORIS is different from their algorithms in terms of conﬁdence set construction and optimism.
See Appendix E.1 for details.
D ECENTRALIZED P OLICY L EARNING R EGRET
Next we present the regret analysis for DORIS in decentralized policy learning setting.
For simplicity, we focus on ﬁnite Π here:
Assumption 1 (Finite player’s policy class) .
We assume Π is ﬁnite.
We consider two cases, the oblivious opponent (i.e., the opponent determines { ν t } Kt =1 secretly before the game starts) and the adaptive opponent (i.e., the opponent determines its policy adaptively as the game goes on) separately.
The difference between these two cases lies in the policy evaluation step in DORIS .
The policy ν t of an oblivious opponent does not depend on the collected dataset D 1: t −and thus V µ,ν t is easier to evaluate.
However, for an adaptive opponent, ν t will be chosen adaptively based on D 1: t − 1 and we need to introduce an additional union bound over Π (cid:48) when analyzing the evaluation error of V µ,ν t .
Oblivious opponent.
To attain accurate value function estimation, we ﬁrst need to introduce two standard assumptions, realizability and generalized completeness, on F and G [25, 27].
Here realizability refers to that all the true action value functions belong to F and generalized completeness means that G contains all the results of applying Bellman operator to the functions in F .
Assumption 2 (Realizability and generalized completeness) .
Assume that for any h ∈ [ H ] , µ ∈ Π , ν ∈ { ν 1 , · · · , ν K } , f h +1 ∈
F h
+1 , we have Q µ × ν h [Equation]
Remark 4.
Some existing works [57, 23] assume the completeness assumption, which can also be generalized to our setting:
Assumption 3.
Assume for any h ∈
[ H ] , µ ∈ Π , ν ∈ Π [Equation]
, we [Equation]
We want to clarify that Assumption 3 is stronger than generalized completeness in Assumption 2 since if Assumption 3 holds, we can simply let G = F to satisfy generalized completeness.
Appendix G shows that realizability and generalized completeness are satisﬁed in many examples including tabular MGs, linear MGs and kernel MGs.
With the above assumptions, we have Theoremto characterize the regret of DORIS when the opponent is oblivious, whose proof sketch is deferred to Appendix I. To simplify writing, we use the following notations in Theorem 1:
[Equation]
Theorem 1 (Regret of Oblivious Adversary) .
Under Assumption 1,2, there exists an absolute constant c such that for any δ ∈ (0 , 1] , K ∈ N , if we choose β = cV 2max log( N cov | Π | /δ ) and η = (cid:112) log | Π | / ( KV 2max ) in DORIS , then with probability at least 1 − δ , we have:
[Equation]
The √ K regret bound in Theorem 1 is consistent with the rate in tabular case [33] and suggests that the uniform mixture of the output policies { µ t } Kt =1 is an (cid:15) -approximate best policy in hindsight when
K = (cid:101) O
(1 /(cid:15) 2 ) .
The complexity of the problem affects the regret through the covering number and the
BEE dimension, implying that BEE dimension indeed captures the essence of this problem.
Further, in oblivious setting, the regret bound in (5) does not depend on Π (cid:48) directly (the upper bound of the
BEE dimension is also independent of Π (cid:48) in some special cases as shown in Proposition 1) and thus
Theorem 1 can still hold when Π (cid:48) is inﬁnite, as long as Assumptions 2 is satisﬁed.
Adaptive Opponent.
In the adaptive setting, we ﬁrst need to modify Assumption 2 to hold for all ν ∈ Π (cid:48) since ν t depends on the collected data (recall that Π is the player’s policy class and Π (cid:48) is the opponent’s policy class):
Assumption 4 (Uniform realizability and generalized completeness) .
Assume that for any h ∈ [ H ] , µ ∈ Π , ν ∈ Π (cid:48) ,
f h +1 ∈
F h
+1 , we have Q µ × ν h
[Equation]
Further, as we have mentioned before, we need to introduce a union bound over the policies in Π (cid:48) in our analysis and thus we also assume Π (cid:48) to be ﬁnite for simplicity.
Assumption 5 (Finite opponent’s policy class) .
We assume Π (cid:48) is ﬁnite.
Remark 5.
It is straightforward to generalize our analysis to inﬁnite Π (cid:48) by replacing | Π (cid:48) | with the covering number of Π (cid:48) .
However, the regret still depends on the size of Π (cid:48) , which is not the case in tabular setting [33].
This dependency originates from our model-free type of policy evaluation algorithm (Algorithm 2) and is inevitable for DORIS in general.
That said, when the Markov game has special structures (e.g., the Markov games in Appendix C and D), we can avoid this dependency.
With the above assumptions, we have Theorem 2 to show that DORIS can still achieve sublinear regret in adaptive setting, whose proof is deferred to Section I:
Theorem 2 (Regret of Adaptive Adversary) .
Under Assumption 1,4,5, there exists an absolute constant c such that for any δ ∈ (0 , 1] , K ∈ N , choosing β = cV 2max log( N cov |
Π || Π (cid:48)
| /δ ) and η = (cid:112) log | Π | / ( KV 2max ) in DORIS , then with probability at least 1 − δ we have: [Equation]
[Equation]
We can see that in adaptive setting the regret also scales with √ K , implying that DORIS can still ﬁnd an (cid:15) -approximate best policy in hindsight with (cid:101) O
(1 /(cid:15) 2 ) episodes even when the opponent is adaptive.
Compared to Theorem 1, Theorem 2 has an additional log | Π (cid:48) | in the upper bound (6), which comes from the union bound over Π (cid:48) in the analysis.
Intuitions on the regret bounds.
The regrets in Theorem 1 and Theorem 2 can be decomposed to two parts, the online learning error incurred by Hedge and the cumulative value function estimation error incurred by OptLSPE .
From the online learning literature [20], the online learning error is
O ( V max (cid:112) K log | Π | ) by viewing the policies in Π as experts and V t ( µ ) as the reward function of expert µ .
For the estimation error, we utilize BEE dimensions to bridge V t ( µ t ) − V π t 1 ( s 1 ) with the function’s empirical Bellman residuals on D 1: t − 1 .
This further incurs (cid:101) O
( V max √ Kd BEE ) in the results.
Our technical contribution mainly lies in bounding the cumulative value function estimation error with the newly proposed BEE dimensions, which is different from [25] where they focus on bounding the cumulative distance from the optimal value function.
Comparison with existing works.
There have been works studying decentralized policy learning.
However, most of them [7, 53, 49, 27, 23] only competes against the Nash value in a two-player zero-sum games, which is a much weaker baseline than ours.
[33] can achieve √ K regret under
Deﬁnition 1, but they are restricted in tabular cases and the bound becomes vacuous with more complicated cases like linear MGs and kernel MGs in Appendix H. DORIS is the ﬁrst algorithm that can achieve √ K regret under Deﬁnition 1 with general function approximation and is capable of tackling all models with low BEE dimension, including linear MGs, kernel MGs and generalized linear complete models.
More details are deferred to Appendix E.1.
4. S ELF -P LAY S AMPLE C OMPLEXITY
Our previous discussion assumes the opponent is arbitrary or even adversary.
A natural question is to ask whether there are any additional guarantees if the player and opponent run DORIS simultaneously, which is exactly the self-play setting.
The following corollary answers this question afﬁrmatively and shows that Algorithm 3 can ﬁnd an approximate CCE (cid:98) π efﬁciently:
Corollary 1.
Suppose Assumption 1,4 hold for all the agents i and its corresponding F i , G i ,
Π i , Π − i .
Then for any δ ∈ (0 , 1] , (cid:15) > 0 , if we choose
[Equation]
where d BEE ,i and N cov ,i are deﬁned respectively as d BEE
[Equation]
dim BEE [Equation] and set β i =
cV 2max log( N cov ,i | Π i
||
Π − i | n/δ ) , η i =
(cid:112) log |
Π i | / ( KV 2max ) , then with probability at least 1 − δ , (cid:98) π is (cid:15) -approximate
CCE.
The proof is deferred to Appendix K. Corollary 1 shows that if we run DORIS independently for each agent, we are able to ﬁnd an (cid:15)
-approximate
CCE with (cid:101) O
(1 /(cid:15) 2 ) samples.
This can be regarded as a counterpart in Markov games to the classic connection between no-regret learning algorithms and equilibria in matrix games.
However, this guarantee does not hold if an algorithm can only achieve low regrets with respect to the Nash values.
Avoiding curse of multiagents.
The sample complexity in (7) avoids exponential scaling with the number of agents n and only scales with max i ∈ [ n ] d BEE ,i , max i ∈ [ n ] N cov ,i and (cid:80) nj =1 log | Π j | , suggesting that statistically Algorithm 3 is able to escape the curse-of-multiagents problem in the literature [26].
Nevertheless, the input dimension of functions in F i and G i may scale with the number of agents, leading to the computational inefﬁciency of OptLSPE .
We comment that ﬁnding computational efﬁcient algorithms is beyond the scope of this paper and we leave it to future works.
Comparison with existing algorithms.
There have been many works studying how to ﬁnd equilibria in Markov games.
However, most of them are focused on centralized two-player zero-sum games [3, 56, 27, 23] rather than decentralized algorithms.
For decentralized algorithms, existing literature mainly handle with potential Markov games [60, 32, 12] and two-player zero-sum games [11, 44, 54].
[35, 26] are able to tackle decentralized multi-agent general-sum Markov games while their algorithms are restricted to tabular cases.
Algorithm 3 can deal with more general cases with function approximation and policy classes in multi-agent general-sum games.
Furthermore, compared to the above works, DORIS has an additional advantage of robustness to adversaries since all the benign agents can exploit the opponents and achieve no-regret learning.
Extensions.
Although Theorem 1, Theorem 2 and Corollary 1 are aimed at Markov games, DORIS can be applied to a much larger scope of problems.
Two such problems are ﬁnding the optimal policy in constrained MDPs and vector-valued MDPs.
We will investigate these two problems in
Appendix C and D, where we demonstrate how to convert such problems into Markov games with a ﬁctitious opponent by duality so that DORIS is ready to use.
A DORIS IN S ELF -P LAY S ETTING
Here we present the pseudocode of Algorithm 3.
Input : learning rate { η i } ni =1 , conﬁdence parameter { β i } ni =1 .
Initialize p 1 i ∈ ∆ | Π i | to be uniform over Π i for all i ∈ [ n ] .
for t = 1 , · · · , K do
Collect samples:
Agent i samples µ ti from p ti .
Run µ t = (cid:81) ni =1 µ ti and collect D t,i = { s t 1 , a t 1 , r t 1 ,i , · · · , s tH +1 } for each agent i .
Update policy distribution:
All agents reveal their policies µ ti .
V ti ( µ i ) ← OptLSPE ( µ i , µ t − i , D 1: t − 1 ,i , F i , G i , β i ) , ∀ µ i ∈
Π
i , i ∈ [ n ] .
p t +1 i ( µ i )
∝ p ti ( µ i ) · exp( η i · V ti ( µ i )) , ∀ µ i ∈
Π
i , i ∈ [ n ] .
end for
Output : (cid:98) π ∼ Unif ( { (cid:81) i ∈ [ n ] µ 1 i , · · · , (cid:81) i ∈ [ n ] µ Ki } ) .
B R ELATED WORKS
In this section we supplement the related literature.
Decentralized learning with an adversarial opponent.
There have been a few works studying decentralized policy learning in the presence of a possibly adversarial opponent.
[7] proposes R-max and is able to attain an average game value close to the Nash value in tabular MGs.
More recently, [53, 49] improve the regret bounds in tabular cases and [27, 23] extend the results to general function approximation setting.
However, these works only compete against the Nash value of the game and are unable to exploit the opponent.
A more related paper is [33], which develops a provably efﬁcient algorithm that achieves a sublinear regret against the best ﬁxed policy in hindsight.
But there results are only limited to the tabular case.
Our work extends the results in [33] to the setting with general function approximation, which requires novel technical analysis.
Finding equilibria in self-play Markov games.
Our work is closely related to the recent literature on ﬁnding equilibria in Markov games via reinforcement learning.
Most of the existing works focus on two-player zero-sum games and consider centralized algorithms with unknown model dynamics.
For example, [53, 3] utilize optimism to tackle the exploration-expoitation tradeoff and ﬁnd Nash equilibria in tabular cases, and [56, 27, 23] extend the results to linear and general function approximation setting.
Furthermore, under the decentralized setting with well-explored data, [11, 60, 44, 54, 32, 12] utilize independent policy gradient algorithms to deal with potential
Markov games and two-player zero-sum games.
Meanwhile, under the online setting, [4, 35, 26] have designed algorithms named V-learning, which are able to ﬁnd CCE in multi-agent general-sum games.
However, there results are only limited to the tabular case.
Constrained Markov decision process.
[15, 13] propose a series of primal-dual algorithms for
CMDPs which achieve √ K bound on regrets and constraint violations in tabular and linear approximation cases.
[34] reduces the constraint violation to (cid:101) O
(1) by adding slackness to the algorithm and achieves zero violation when a strictly safe policy is known; [55] further avoids such requirement with the price of worsened regrets.
Nevertheless, these improvements are only discussed in the tabular case.
Approchability for vector-valued Markov decision process.
[36] ﬁrst introduces the approachability task for VMDPs but does not provide an algorithm with polynomial sample complexity.
Then [58] proposes a couple of primal-dual algorithms to solve this task and achieves a (cid:101)
O ( (cid:15) − 2 ) sample complexity in the tabular case.
More recently, [37] utilizes reward-free reinforcement learning to tackle the problem and studies both the tabular and linear approximation cases, achieving roughly the same sample complexity as [58].
C E XTENSION : C ONSTRAINED M ARKOV D ECISION P ROCESS
Although DORIS is designed to solve Markov games, there are quite a lot of other problems which
DORIS can tackle with small adaptation.
In this section we investigate an important scenario in practice called constrained Markov decision process (CMDP).
By converting CMDPs into a maximin problem via Lagrangian multiplier, we will be able to view it as a zero-sum Markov game and apply
DORIS
readily.
Constrained Markov decision process.
Consider the Constrained Markov Decision Process (CMDP) [13] M CMDP =
( S , A , { P h } Hh =1 , { r h } Hh =1 , { g h } Hh =1 , H ) where S is the state space, A is the action space, H is the length of each episode, P h : S ×
A → ∆( S ) is the transition function at the h -th step, r h : S ×
A → R + is the reward function and g h : S ×
A → [0 , 1] is the utility function at the h -th step.
We assume the reward r h is also bounded in [0 , 1] for simplicity and thus
V max = H .
Then given a policy µ = { µ h : S → ∆ A } h ∈ [ H ] , we can deﬁne the value function V µr,h and action-value function Q µr,h with respect to the reward function r as follows:
[Equation]
The value function V µg,h and action-value function Q µg,h with respect to the utility function g can be deﬁned similarly.
Another related concept is the state-action visitation distribution, which can be deﬁned as
[Equation]
where Pr µ denotes the distribution of the trajectory induced by executing policy µ in M CMDP .
Learning objective.
In CMDPs, the player aims to solve a constrained problem where the objective function is the expected total rewards and the constraint is on the expected total utilities:
Problem 1: Optimization problem of CMDP
[Equation]
where b ∈ (0 , H ] to avoid triviality.
Denote the optimal policy for (8) by µ ∗ CMDP , then the regret can be deﬁned as the performance gap with respect to µ ∗ CMDP :
[Equation]
However, since utility information is only revealed after a policy is decided, it is impossible for each policy to satisfy the constraints.
Therefore, like [13], we allow each policy to violate the constraint in each episode and focus on minimizing total constraint violations over K episodes:
[Equation]
Achieving sublinear violations in (10) implies that if we sample a policy uniformly from { µ t } Kt =1 , its constraint violation can be arbitrarily small given large enough K .
Therefore, if an algorithm can achieve sublinear regret in (9) and sublinear violations in (10) at the same time, this algorithm will be able to ﬁnd a good approximate policy to µ ∗ CMDP . C
[Equation]
To solve Problem 1 with DORIS , we ﬁrst need to convert it into a Markov game.
A natural idea is to apply the Lagrangian multiplier Y ∈ R + to Problem 1, which brings about the equivalent maximin problem below:
[Equation]
Although Problem 1 is non-concave in µ , there have been works indicating that strong duality still holds for Problem 1 when the policy class is described by a good parametrization [40].
Therefore, here we assume strong duality holds and it is straightforward to generalize our analysis to the case where there exists a duality gap:
Assumption 6 (Strong duality) .
Assume strong duality holds for Problem 1, i.e.,
[Equation]
Remark 6.
One example case where strong duality (12) holds is when policy class Π satisﬁes global realizability.
Let µ ∗ glo denote the solution to max µ h ( ·| s ) ∈ ∆
A min Y ≥
L CMDP ( µ, Y ) .
[13] shows that max µ
[Equation] [Equation]
min Y
[Equation]
L CMDP ( µ, Y ) satisﬁes strong duality, and thus as long as µ ∗ glo ∈ Π ,
Problem 1 also has strong duality.
Further, let D ( Y ) :
= max µ ∈ Π L CMDP ( µ, Y ) denote the dual function and suppose the optimal dual variable is Y ∗ =
arg min Y ≥
D ( Y ) .
To ensure Y ∗ is bounded, we need to assume the standard
Slater’s Condition holds:
Assumption 7.
There exists λ sla > 0 and (cid:101) µ ∈ Π such that V (cid:101) µg, 1 ( s 1 ) ≥ b + λ sla .
Then the following lemma shows that Assumption 7 implies bounded optimal dual variable, whose proof is deferred to Appendix L.1:
Lemma 1.
Suppose Assumption 6,7 hold, then we have 0 ≤ Y ∗ ≤ H/λ sla .
Now we are ready to adapt DORIS into a primal-dual algorithm to solve Problem 1.
Notice that the maximin problem (11) can be viewed as a zero-sum Markov game where the player’s policy is µ and the reward function for the player is r h ( s, a )
+ Y g h
( s, a ) .
The opponent’s action is Y ∈ R + which remains the same throughout a single episode.
With this formulation, we can simply run DORIS on the player, assuming the player is given function classes {F r , G r } and {F g , G g } to approximate Q µr,h and Q µg,h respectively.
In the meanwhile, we run online projected gradient descent on the opponent so that its action Y can capture the total violation so far.
This new algorithm is called DORIS-C and shown in Algorithm 4.
It consists of the three steps below in each iteration.
For the policy evaluation task in the second step, DORIS-C runs a single-agent version of OptLSPE to estimate V µr, 1 ( s 1 ) and V µg, 1 ( s 1 ) separately, which is essential for DORIS-C to deal with the inﬁnity of the opponent’s policy class, i.e., R + .
• The player plays a policy µ t sampled from its hyperpolicy p t and collects a trajectory.
• The player runs OptLSPE-C to obtain optimistic value function estimations V tr ( µ ) , V tg ( µ ) for all µ ∈ Π and updates the hyperpolicy using Hedge with the rewards being V tr ( µ ) + Y t V tg ( µ ) .
The
construction
rule for
B
D (
µ )
is still based on
relaxed least-squares policy evaluation:
[Equation]
where L D is the empirical Bellman residuals on D :
[Equation]
• The dual variable is updated using online projected gradient descent.
Input : learning rate η, α , conﬁdence parameter β r , β g , projection length χ .
Initialize p 1 ∈ R | Π | to be uniform over Π , Y 1 ← 0 .
for t = 1 , · · · , K do
Collect samples:
The player samples µ t from p t .
Run µ t and collect D rt
= { s t 1 , a t 1 , r t 1 , · · · , s tH +1 } , D gt = { s t 1 , a t 1 , g t 1 , · · · , s tH +1 } .
Update policy distribution:
V tr ( µ ) ← OptLSPE-C ( µ, D r 1: t − 1 , F r , G r , β r ) , ∀ µ ∈ Π .
V tg ( µ ) ← OptLSPE-C ( µ, D g 1: t − 1 , F g , G g , β g ) , ∀ µ ∈ Π .
p t +1 ( µ ) ∝ p t ( µ ) · exp( η · ( V tr ( µ )
+ Y t V tg ( µ ))) , ∀ µ ∈ Π .
Update dual variable:
Y t +1 ← Proj [0 ,χ ] ( Y t + α ( b − V tg ( µ t ))) .
end for
Algorithm
Construct B D ( µ ) based on D via (13).
Select ¯ V ← max f ∈B D ( µ ) f ( s 1 , µ ) .
return ¯ V .
C. T HEORETICAL G UARANTEES
Next we provide the regret and constraint violation bounds for DORIS-C .
Here we also consider the case where Π is ﬁnite, i.e., Assumption 1 is true.
However, we can see that here the opponent is adaptive and its policy class is inﬁnite, suggesting that Assumption 5 is violated.
Fortunately, since the opponent only affects the reward function, the player can simply ﬁrst estimate V µr, 1 ( s 1 ) and V µg, 1 ( s 1 ) respectively and then use their weighted sum to approximate the target value function
V µr, 1 ( s 1 ) + Y · V µg, 1 ( s 1 ) .
In this way, DORIS-C circumvents introducing a union bound on Y and thus can work even when the number of possible values for Y is inﬁnite.
We also need to introduce the realizability and general completeness assumptions on the function classes as before:
Assumption 8 (Realizability and generalized completeness in CMDP) .
Assume that for any h ∈ [ H ] , µ ∈ Π , f rh +1 ∈ F rh +1 , f gh +1 ∈ F g , we have
[Equation]
Here T µ,rh is the Bellman operator at step h with respect to r :
[Equation]
where f h +1 ( s (cid:48) , µ ) =
E a (cid:48) ∼ µ ( ·| s )
[ f h +1 ( s (cid:48) , a (cid:48) )] .
T µ,gh is deﬁned similarly.
We can see that (14) simply says that all the action value functions with respect to r ( g ) belong to F r ( F g ) and G r ( G g ) contains all the results of applying Bellman operator with respect to r ( g ) to the functions in F r ( F g ).
In addition, as a simpliﬁed case of Deﬁnition 6, BEE dimension for single-agent setting can be deﬁned as follows:
Deﬁnition 7.
The single-agent (cid:15) -Bellman Evaluation Eluder dimension of function class F on distribution family Q with respect to the policy class Π and the reward function r is deﬁned as follows:
[Equation]
] dim DE (( I − T Π ,r h ) F , Q h , (cid:15) ) , where ( I − T Π ,r h ) F := { f h − T µ,rh f h +1 : f ∈ F , µ ∈ Π } .
We also let dim BEE ( F , (cid:15), Π , r ) denote min { dim BEE ( F , (cid:15), Π , r, Q 1 ) , dim BEE ( F , (cid:15), Π , r, Q 2 ) } as before.
dim BEE ( F , (cid:15), Π , g, Q ) and dim BEE ( F , (cid:15), Π , g ) are deﬁned similarly but with respect to the utility function g .
Now we can present Theorem 3 which shows that DORIS-C is capable of achieving sublinear regret and constraint violation for Problem 1.
We also use the following notations to simplify writing:
[Equation]
Theorem 3.
Under Assumption 6,7,1,8, there exists an absolute constant c such that for any δ ∈ (0 , 1] ,
K ∈ N , if we choose β r = cH 2 log( N cov ,r | Π | /δ ) , β g = cH 2 log( N cov ,g | Π | /δ ) , α = 1 / √ K , χ
= 2 H/λ sla and η = (cid:112) log | Π | / ( K ( χ + 1) 2 H 2 ) in DORIS-C , then with probability at least 1 − δ , we have:
[Equation]
where
[Equation]
The bounds in (15) and (16) show that both the regret and constraint violation of DORIS-C scale with √
K .
This implies that for any (cid:15) > 0 , if (cid:98) µ is sampled uniformly from { µ t } Kt =1 and K ≥ (cid:101)
O
(1 /(cid:15) 2 ) , (cid:98) µ will be an (cid:15) near-optimal policy with high probability in the sense that
[Equation]
In addition, compared to the results in Theorem 1 and Theorem 2, (15) and (16) have an extra term scaling with 1 /λ sla .
This is because DORIS-C is a primal-dual algorithm and λ sla characterizes the regularity of this constrained optimization problem.
The proof of the regret bound is similar to Theorem 1 and Theorem 2 by viewing V µr, 1 ( s 1 )+ Y V µg, 1 ( s 1 ) as the target value function and decomposing the regret into cumulative estimation error and online learning error.
To bound the constraint violation, we need to utilize the strong duality and the property of online projected gradient descent.
See Appendix L for more details.
Comparison with existing algorithms.
There has been a line of works studying the exploration and exploitation in CMDPs.
[15, 13] propose a series of algorithms which can achieve √ K bound on regrets and constraint violations.
However, they focus on tabular cases or linear function approximation and do not consider policy classes while DORIS-C can deal with nonlinear function approximation and policy classes.
As an interesting follow-up, [34] reduces the constraint violation to (cid:101) O
(1) by adding slackness to the algorithm and achieves zero violation when a strictly safe policy is known; [55] further avoids such requirement with the price of worsened regrets.
However, these improvements are all limited in tabular cases and we leave the consideration of their general function approximation counterpart to future works.
D E XTENSION : V ECTOR - VALUED M ARKOV D ECISION PROCESS
Another setting where DORIS can play a role is the approachability task for vector-valued Markov decision process (VMDP)
Similar to CMDP, we convert it into a zero-sum Markov game by Fenchel’s duality and then adapt DORIS properly to solve it.
Vector-valued Markov decision process.
Consider the Vector-valued Markov decision process (VMDP) [58] M VMDP =
( S , A , { P h } H h =1 , r , H ) where r = { r h : S ×
A → [0 , 1] d } H h
=1 is a collection of d -dimensional reward functions and the rest of the components are deﬁned the same as in Section C.
Then given a policy µ ∈ Π , we can deﬁne the corresponding d -dimensional value function V µh : S → [0 , H ] d and action-value function Q µh : S ×
A → [0 , H ] d as follows:
[Equation]
Learning objective.
In this paper we study the approachability task [36] in VMDP where the player needs to learn a policy whose expected cumulative reward vector lies in a convex target set C .
We consider a more general agnostic version [58, 37] where we do not assume the existence of such policies and the player learns to minimize the Euclidean distance between expected reward and the target set C :
Problem 2: Approachability for VMDP
[Equation]
where dist( x , C ) is the Euclidean distance between point x and set C .
The approachability for VMDP is a natural objective in multi-task reinforcement learning where each dimension of the reward can be regarded as a task.
It is important in many practical domains such as robotics, autonomous vehicles and recommendation systems [58].
Therefore, ﬁnding the optimal policy for Problem 2 efﬁciently is of great signiﬁcance in modern reinforcement learning.
D. A LGORITHM : DORIS-V
To deal with Probelm 2, we ﬁrst convert Problem 2 into a Markov game as we have done in
Appendix C. By Fenchel’s duality of the distance function, we know Problem 2 is equivalent to the following minimax problem:
[Equation]
where B ( r ) is the d -dimensional Euclidean ball of radius r centered at the origin.
Regarding µ as the player’s policy and θ as the opponent, we can again view this minimax problem as a Markov game where the reward function for the player is (cid:104) θ , r h ( s, a ) (cid:105) .
Consider the general function approximation case where the player is given function classes F :=
{F jh } H,dh,j =1 , G := {G jh } H,dh,j =1 to approximate
Q µh ( F jh and G jh are the j -th dimension of F h and G h ), we can run DORIS for the player while the opponent will update θ via online projected gradient ascent just like DORIS-C .
We call this new algorithm DORIS-V , which is shown in Algorithm 6 and also consists of three steps in each iteration.
For the policy evaluation task here, we apply OptLSPE-V and construct a conﬁdence set for each dimension of the function class separately, and let the ﬁnal conﬁdence set be their intersection.
Therefore the construction rule for B D ( µ ) is given as:
[Equation]
where for any j ∈ [ d ] and h ∈ [ H ] ,
[Equation]
and r jh is the j -the dimension of r h .
In addition, since here we want to minimize the distance,
OptLSPE-V
will output a pessimistic estimation of the target value function instead
of an optimistic one
• The player plays a policy µ t sampled from its hyperpolicy p t and collects a trajectory.
• The player runs OptLSPE-V to obtain pessimistic value function estimations (cid:104) θ t , V t ( µ ) (cid:105) for all µ ∈ Π and updates the hyperpolicy using Hedge.
• The dual variable is updated using online projected gradient ascent.
D. T HEORETICAL G UARANTEES
We still consider ﬁnite policy class Π here.
Notice that in the ﬁctitious MG of VMDP, the opponent’s policy class is also inﬁnite, i.e., B (1) .
However, since the player only needs to estimate V µ 1 ( s 1 ) , which is independent of θ , DORIS-V can
Algorithm
Input : learning rate η, α t , conﬁdence parameter β .
Initialize p 1 ∈ R | Π | to be uniform over Π , θ 1 ← 0 .
for t = 1 , · · · , K do
Collect samples:
The learner samples µ t from p t .
Run µ t and collect D t = { s t 1 , a t 1 , r t 1 , · · · , s tH +1 } .
Update policy distribution:
V t ( µ ) ← OptLSPE-V ( µ, D 1: t − 1 , F , G , β, θ t ) , ∀ µ ∈ Π .
p t +1 ( µ ) ∝ p t ( µ ) · exp( − η (cid:104) V t ( µ ) , θ t (cid:105) ) , ∀ µ ∈ Π .
Update dual variable: θ t +1 ← Proj B (1) ( θ t + α t ( V t ( µ t ) − arg max x ∈C (cid:104) θ t , x (cid:105) )) .
end for
Output : (cid:98) µ uniformly sampled from µ 1 , · · · , µ K .
Algorithm
Construct B D ( µ ) based on D via (17).
Select V ← f 1 ( s 1 , µ ) , where f = arg min f (cid:48) ∈B D ( µ ) (cid:104) f (cid:48) 1 ( s 1 , µ ) , θ (cid:105) .
return V .
In addition, we need to introduce the realizability and generalized completeness assumptions in this speciﬁc setting, which is simply a vectorized version as before:
Assumption 9 (Realizability and generalized completeness in VMDP) .
Assume that for any h ∈ [ H ] , j ∈ [ d ] , µ ∈ Π ,
f h +1 ∈
F h
+1 , we have Q µ,jh ∈ F h,j , T
µ,jh f jh +1 ∈
G jh , where Q µ,jh is the j -the dimension of Q µh and T µ,jh is the j -th dimensional Bellman operator at step h deﬁned in (18).
Here T µ,jh is deﬁned as:
[Equation]
In addition, the BEE dimension for VMDP can be deﬁned as the maximum BEE dimension among all d dimensions:
Deﬁnition 8.
The d -dimensional (cid:15) -Bellman Evaluation Eluder dimension of function class F on distribution family Q with respect to the policy class Π is deﬁned as follows:
[Equation]
where ( I − T Π ,j h )
[Equation]
We also use dim BEE ( F , (cid:15), Π) to denote min { dim BEE ( F , (cid:15), Π , Q 1 ) , dim BEE ( F , (cid:15), Π , Q 2 ) } as before.
The next theorem shows that DORIS-V is able to ﬁnd a near optimal policy for Problem 2 with polynomial samples, where we use the following notations to simplify writing:
[Equation]
Theorem 4.
Under Assumption 1,9, there exists an absolute constant c such that for any δ ∈ (0 , 1] ,
K ∈ N , if we choose β = cH 2 log( N cov ,
V | Π | d/δ ) , α t = 2 / ( H √ dt ) , and η = (cid:112) log | Π | / ( KH 2 d ) in DORIS-V , then with probability at least 1 − δ , we have:
[Equation]
The bound in (19) shows that for any (cid:15) > 0 , if K ≥ (cid:101) O ( d/(cid:15) 2 ) , (cid:98) µ will be an (cid:15) near-optimal policy with high probability.
Compared to the results in Theorem 1 and Theorem 2, there is an additional term d .
This is because the reward is d -dimensional and we are indeed evaluating d scalar value functions in
OptLSPE-V .
The proof is similar to that of Theorem 3 and utilizes the fact that both µ and θ are updated via no-regret online learning algorithms (Hedge for µ and online projected gradient ascent for θ ).
See
Appendix M for more details.
Paper
Setting
[Table]
Tian et al.
Tabular
[Table]
Liu et al.
Tabular
[Table]
Jin et al.
General FA
[Table]
Huang et al.
This work
[Table]
Jin et al.
[27], Huang et al.
[23] are able to deal with general function approximation, but they can only compete against
Paper
Setting
[Table]
Jin
[Table]
al.
[Table] [Table]
Mao et al.
[Table]
Jin et al.
[Table]
Huang et al.
This work
General FA
[Table]
general function approximation setting.
We can see that Jin et al.
[26], Mao et al.
[35] studies tabular cases.
Their algorithms are decentralized and can still work when the number of players is larger than 2.
Jin et al.
[27], Huang et al.
[23] works on general function approximation setting but their algorithms are centralized and limited to two-player zero-sum games.
In comparison, our work can handle multi-agent ( ≥ 2 players) general-sum games with general function approximation and the algorithm is decentralized.
Comparison with existing algorithms.
[58] has also proposed algorithms for approachability tasks in tabular cases and achieve the same sub-optimality gap with respect to d and K as Theorem 4.
[37] studies the tabular and linear approximation cases, achieving √ K regret as well.
Their sample complexity does not scale with d because they have normalized the reward vector to lie in B (1) in tabular cases and B ( √ d lin ) in d lin -dimensional linear VMDPs.
Compared to the above works,
DORIS-V is able to tackle the more general cases with nonlinear function approximation and policy classes while retaining the sample efﬁciency.
E D ISCUSSION E. C OMPARISON WITH C LOSELY -R ELATED W ORKS
In this section we provide a more detailed comparison between DORIS and some closely-related works in this section.
First we would like to summarize the comparison with decentralized learning literature and self-play literature in Table 1 and 2.
Next we will clarify the novelty of our work given some relarted works.
Novelty given Liu et al.
The idea of maintaining a hyperpolicy and utilizing Hedge to update it in DORIS is inspired from OPMD proposed in Liu et al.
However, the policy evaluation algorithm in Liu et al.
[33] can only work in tabular cases and our novelty lies in the new optimism and policy evaluation step (Algorithm 2) specially designed for the policy revealing setting.
Note that the extension to the setting of general function approximation is not trivial.
Combining existing techniques on reinforcement learning with general function approxiamtion (for example, Bellman
Eluder dimension in Jin et al.
[25]) with Liu et al.
[33] does not lead to our work because the optimism in DORIS is (i) global (in the sense that optimism is only true for s 1 , which differs from
Liu et al.
[33]) and (ii) policy-pair speciﬁc (that is, our conﬁdence set is only optimistic with respect to V µ,ν k
( s 1 ) , which differs from Jin et al.
More speciﬁcally, Liu et al.
[33] attains optimism via adding a bonus term β for each step h and state-action pair ( s, a, b ) in value iteration as follows:
[Equation]
This guarantees that ¯ V µ,ν k h ( s, a, b ) is optimistic with respect to the true value function V µ
[Equation]
for each h, s, a, b .
However, DORIS picks the most optimistic estimation from the constructed conﬁdence set directly:
[Equation]
This can only guarantee that ¯ V µ,ν k is optimistic with respect to V µ,ν k
[Equation]
Thus bounding the regret will be harder in our case with only global optimism.
In addition, although Jin et al.
[25] also uses global optimism, it is optimistic with respect to the ﬁxed optimal value function V ∗ 1 ( s 1 ) and only needs to construct one conﬁdence set of it.
Nevertheless, to tackle the non-stationary optimal policy in the decentralized setting which keeps changing across episodes due to the adversarial opponent, DORIS needs to construct a conﬁdence set of V µ,ν k( s 1 ) for each policy µ in the policy class and the estimation ¯ V µ,ν k is only optimistic with respect to V
[Equation]
for each µ respectively.
Therefore, the analysis techniques in Jin et al.
[25] cannot be applied directly here.
We need to decompose the regret in a different way and propose a new complexity measure (i.e., BEE dimension) to bound the regret.
Novelty given Jin et al.
[27], Huang et al.
Jin et al.
[23], Huang et al.
[23] also design algorithms and utilize analysis techniques based on Bellman-eluder-type complexity (i.e., multi-agent
BE dimension [27, 23]) in Markov games.
Here we want to clarify that although the BEE dimension is also inspired from Bellman Eluder dimension proposed in Jin et al.
[25], DORIS is very different from Jin et al.
[27], Huang et al.
[23] in terms of conﬁdence set construction and optimism , which makes their analysis techniques not applicable here either.
Jin et al.
[27], Huang et al.
[23] consider zero-sum Markov games in the centralized setting with general function approximation.
The algorithms in these works are based on (i) constructing conﬁdence regions of the optimal value function (i.e., Nash value function in zero-sum games) or model and (ii) solving the Nash equilibrium with respect to the optimistic function/model.
As a result, their algorithms can be regarded as running optimistic greedy policies in games and the estimated value functions are always optimistic estimates of the optimal value function V ∗ 1 ( s 1 ) of the underlying game.
In contrast, in the decentralized setting, one unique challenge faced by DORIS is that the optimal policy is indeed changing across the episodes because we cannot control the opponent and the opponent can be adversarially adjusting its own policy.
Therefore, there does not exist such a ﬁxed optimal value function that we can run optimism with respect to.
More importantly, from the view of the single agent, the environment is adversarially changing due to the opponent.
Such nonstationarity does not appear in these works.
To deal with this challenge, DORIS is based on (i) constructing the conﬁdence region for policy evaluation problems, (ii) running mirror descent over the space of policies.
As a result, DORIS is more like a decentralized policy optimization algorithm and the value functions maintained by the DORIS are only optimistic with respect to the value functions associated with the current policy pair ( µ, ν k ) , which changes at each iteration.
More importantly, such a different version of optimism leads to a different regret decomposition.
Speciﬁcally, in (22), we show that the regret is upper bounded by the policy evaluation error (cid:80) Kt =1 ( ¯ V t ( µ t ) − V π t 1 ( s 1 )) and online learning error induced (cid:80) Kt =1 ( ¯ V t ( µ ∗ ) − ¯ V t ( µ t )) by mirror descent.
Bounding the evaluation error (cid:80) Kt =1 ( ¯ V t ( µ t ) − V π t 1 ( s 1 )) incurred by achieving optimism in policy evaluation has not been considered in Jin et al.
[27], Huang et al.
Multi-agent BE dimension [27, 23] cannot be applied here either because it measures the Bellman residuals f h ( s, a, b ) − r h
+1 ( s, a, b ) − min ν f h +1 ( s, µ, ν ) and can only help bound (cid:80) Kt =1 ( ¯ V t ( µ t )
− min ν V µ t ,ν( s 1 )) when the policy ν t played by the opponent is a pessimistic best response of µ t .
In our case ν t is arbitrary (typically not the best response of µ t ) and the value we want to bound is also different, therefore multi-agent BE dimension is not applicable and we have to propose BEE dimension to evaluate the complexity of policy evaluation tasks with general function approximation.
Along with the new measure, we have also identiﬁed common function classes with low BEE dimension in the paper to illustrate the capacity of BEE dimension.
E. L OWER B OUND IN L IU ET AL . [33]
Here we present a lower bound from Liu et al.
Theorem 5 (Liu et al.
[33, Theorem 4]) .
There exists a Markov game with S, A = O ( H ) and an opponent who chooses policy uniformly at random from an unknown set of H Markov policies in each episode, such that when the opponent’s policy is not revealed, the regret for competing with the best ﬁxed Markov policy in hindsight is Ω(min { K, 2 H } /H ) . The above lower bound shows that if the opponent’s policy is not revealed, even when the opponent only plays a ﬁnite number of Markov policies, the exponential regret lower bound for competing with the best Markov policy in hindsight is inevitable, which validates the necessity of policy revealing condition. E. C OMPUTATIONAL C OMPLEXITY OF DORIS
There are mainly two steps in DORIS that require computation, optimistic policy evaluation via
OptLSPE and hyperpolicy update via Hedge.
Assuming the policy class is ﬁnite facilitates the second step, but even with ﬁnite policy class, OptLSPE is still computationally inefﬁcient.
This is due to the global optimism step in OptLSPE , i.e., constructing the conﬁdence set (Equation (3)) and ﬁnding the most optimistic estimation.
This is a common issue of algorithms with general function approximation even in single-agent MDPs.
For example, the global optimism step of the algorithms in [24, 25, 14, 23, 27] are all computationally inefﬁcient and hard to implement.
However, if we only consider linear MGs, computationally efﬁcient algorithms are possible since we can use local optimism and implement OptLSPE by an analog of LSVI-UCB [28], which is computationally efﬁcient.
In addition, if there is a computationally efﬁcient solver for optimistic policy evaluation with general function approximation in single-agent MDPs, we believe that we can also utilize it here since the conﬁdence set update rule (Equation (3)) is similar to single-agent MDPs.
That said, in this work we mainly focus on the statistical complexity of learning the Markov game and thus computationally efﬁcient algorithms are left as future works.
F P ROOFS OF P ROPOSITION
From the completeness assumption, we know that there exists g h ∈ F h such that g h
= T µ,νh [Equation] 1 , which implies that
[Equation]
In other words, ( I − T Π , Π (cid:48) h ) F ⊆ F h − F h .
Therefore, from the deﬁnition of dim BEE ( F , (cid:15), Π , Π (cid:48) )
we have
[Equation]
where the last step comes from the deﬁnition of dim E and Q 2 h is the dirac distribution family.
This concludes our proof.
G E XAMPLES FOR R EALIZABILITY , G ENERALIZED C OMPLETENESS AND C OVERING N UMBER
In this section we illustrate practical examples where realizability and generalized completeness hold and the covering number is upper bounded at the same time.
In speciﬁc, we will consider tabular
MGs, linear MGs and kernel MGs.
G. T ABULAR MG S
For tabular MGs, we let F h
= { f | f : S ×
A × B (cid:55)→ [0 , V max ] } and G h
=
F h for all h ∈
[ H ] .
Then it is obvious that Q µ × ν h ∈ F h and T µ,ν f
h +1 ∈ G h for any f ∈ F , h ∈
[ H ] , µ, ν , which implies that realizability and generalized completeness are satisﬁed.
In addition, notice that in this case we have
[Equation]
This suggests that the size of F and G is also not too large.
G. L INEAR MG S
In this subsection we consider linear MGs.
Here we generalize the deﬁnition of linear MDPs in classic MDPs [28] to Markov games:
Deﬁnition 9 (Linear MGs) .
We say an MG is linear of dimension d if for each h ∈ [ H ] , there exists a feature mapping φ h : S ×A×B (cid:55)→ R d and d unknown signed measures ψ h
=
( ψ (1) h , · · · , ψ ( d ) h ) over
S and an unknown vector θ h ∈ R d such that P h ( ·| s, a, b ) =
φ h ( s, a, b ) (cid:62) ψ h ( · ) and r h ( s, a, b ) =
φ h ( s, a, b ) (cid:62) θ h for all ( s, a, b ) ∈ S ×
A × B .
Without loss of generality, we assume (cid:107) φ h ( s, a, b ) (cid:107) ≤ 1 for all s ∈ S , a ∈ A , b ∈ B and (cid:107)
ψ h ( S ) (cid:107) ≤ √ d, θ h ≤ √ d for all h .
Let F h
=
G h
= { φ h ( · ) (cid:62) w | w ∈ R d , (cid:107) w (cid:107) ≤
( H − h +1) √ d, 0 ≤ φ h ( · ) (cid:62) w ≤
H − h + 1 } .
Realizability.
We have for any µ, ν ,
Q µ × ν h
[Equation]
where w µ × ν h = θ
[Equation]
[-+
-]+
[Equation] S V µ × ν h +1 ( s (cid:48) )
dψ h
( s (cid:48) ) and thus (cid:107) w µ ×
ν
h
(cid:107) ≤ ( H − h + 1) √ d .
Therefore, Q µ × ν h ∈
F h , which means that realizability holds.
Generalized completeness.
For any f h +1 ∈ F h +1 , we have
[Equation]
[Equation]
[ f [Equation] [Equation] Since (cid:107) f h +1 (cid:107) ∞ ≤ H − h , we have (cid:107) θ h
+ (cid:82) S f h
+1 ( s (cid:48) , µ, ν ) dψ
h
( s (cid:48) )
(cid:107) ≤ ( H − h + 1) √ d , which indicates T µ,ν f
h +1 ∈ G h and thus generalized completeness is satisﬁed.
Covering number.
First notice that from the literature [51], the covering number of a l 2 -norm ball can be bounded as log N B (( H − h +1) √ d ) ( (cid:15) )
≤ d log(3 H √ d/(cid:15) ) .
Therefore, there exists W ⊂
B (( H − h + 1) √ d ) where log |W| ≤ d log(3 H √ d/(cid:15) ) such that for any w ∈ B (( H − h + 1) √ d ) , there exists w (cid:48)
∈ W satisfying (cid:107) w (cid:48) − w (cid:107) ≤ (cid:15) .
Now let F (cid:48)
h
= { φ h ( · ) (cid:62) w | w ∈ W} .
For any f h ∈ F h , suppose f h ( · ) =
φ h ( · ) (cid:62) w f h .
Then we know there exists f (cid:48) h ( · ) =
φ h ( · ) (cid:62) w (cid:48)
f h ∈ F (cid:48)
h where (cid:107) w (cid:48)
f h − w f h (cid:107) ≤ (cid:15) , which implies
[Equation]
Therefore log N F h ( (cid:15) ) ≤ log |F (cid:48)
h
| = log |W| ≤ d log(3 H √ d/(cid:15) ) .
G. K ERNEL MG S
In this subsection we show that kernel MGs also satisfy realizability and generalized completeness naturally.
In addition, when a kernel MG has a bounded effective dimension, its covering number will also be bounded.
First we generalize the deﬁnition of kernel MDPs [25] to MGs as follows.
Deﬁnition 10 (Kernel MGs) .
In a kernel MDP, for each step h ∈
[ H ] , there exist feature mapping φ h : S ×
A × B (cid:55)→ H and ψ h : S (cid:55)→ H where H is a separable Hilbert space such that
P h ( s (cid:48) | s, a, b ) =
(cid:104)
φ h ( s, a, b ) , ψ h ( s (cid:48) )
(cid:105) H for all s ∈ S , a ∈ A , b ∈ B , s (cid:48) ∈ S .
Besides, the reward function os linear in φ , i.e., r h ( s, a, b ) =
(cid:104)
φ h ( s, a, b ) , θ h (cid:105) H for some θ h ∈ H .
Moreover, a kernel
MG satisﬁes the following regularization conditions:
• (cid:107) θ h (cid:107) H ≤ 1 , (cid:107) φ h ( s, a, b ) (cid:107) H ≤ 1 , for all s ∈ S , a ∈ A , b ∈ B , h ∈
[ H ] .
• (cid:107) (cid:80) s ∈S V ( s ) ψ h ( s ) (cid:107) H ≤ 1 , for all function V : S (cid:55)→ [0 , 1] ,
h ∈
[ H ] .
Remark 7.
It can be observed that tabular and linear MGs are special cases of kernel MGs.
Therefore, the following discussion applies to tabular and linear MGs as well.
Then we let F h
=
G h
= { φ h ( · ) (cid:62) w | w ∈ B H ( H − h + 1) } where B H ( r ) is a ball with radius r in
H .
Following the same arguments in linear MGs, we can validate that realizability and generalized completeness are satisﬁed in kernel MGs.
Covering number.
Before bounding the covering number of F h , we need introduce a new measure to evaluate the complexity of a Hilbert space since H might be inﬁnite dimensional.
Here we use the effective dimension [14, 25], which is deﬁned as follows:
Deﬁnition 11 ( (cid:15) -effective dimension of a set) .
The (cid:15) -effective dimension of a set X is the minimum integer d eﬀ ( X , (cid:15) )
= n such that
[Equation]
Remark 8.
When X is ﬁnite dimensional, suppose its dimension is d .
Then its effective dimension can be upper bounded by O (cid:0)
d log (cid:0) 1 + R 2 /(cid:15)
(cid:1)(cid:1) where R is the norm bound of X [14].
In addition, even when X is inﬁnite dimensional, if the eigenspectrum of the covariance matrices concentrates in a low-dimension subspace, the effective dimension of X can still be small [48].
We call a kernel MG is of effective dimension d ( (cid:15) ) if d eﬀ ( X h , (cid:15) ) ≤ d ( (cid:15) ) for all h and (cid:15) where
X h
= { φ h ( s, a, b ) : ( s, a, b ) ∈ S ×
A × B} .
Then the following proposition shows that the covering number of F h is upper bounded by the effective dimension of the kernel
MG:
Proposition 2.
If the kernel MG has effective dimension d ( (cid:15) ) , then
[Equation]
Proof.
Suppose dim E ( F h , (cid:15) ) = n .
Then by the deﬁnition of Eluder dimension, there exists a sequence { φ i } ni =1 such that for any w 1 , w 2 ∈ B H ( H − h +1) , φ ∈ X h , if (cid:80) ni
=1 ( (cid:104)
φ i , w 1 − w 2 (cid:105) )
2 ≤ (cid:15) 2 , then |(cid:104) φ, w 1 − w 2 (cid:105)| ≤ (cid:15) .
Therefore, the covering number of kernel MGs can be reduced to covering the projection of B H ( H − h + 1) onto the space spanned by { φ i } n i =1 , whose dimension is at most n .
From the literature [51], the covering number of such space is O ( n log (1 + nH/(cid:15) )) , which implies [Equation] N F h (
[Equation]
) ≤ O (cid:0) n log(1 + nH/(cid:15) )
(cid:1) . Finally, by the proof of Proposition 3, we know n ≤ d ( (cid:15)/ 2 H ) , which concludes the proof.
H E XAMPLES FOR BEE D IMENSION
In this section we will show that kernel MGs (including tabular MGs and linear MGs) and generalized linear complete models have low BEE dimensions.
H. K ERNEL MG S
Consider the kernel MG deﬁned in Deﬁnition 10 and F h
= { φ h ( · ) (cid:62) w | w ∈ B H ( H − h + 1) } , then we have the following proposition showing that the BEE dimension of a kernel MG is upper bounded by its effective dimension (Deﬁnition 11):
Proposition 3.
If the kernel MG has effective dimension d ( (cid:15) ) , then for any policy classes Π and Π (cid:48) , we have d BEE ( F , (cid:15), Π , Π (cid:48) )
≤ d ( (cid:15)/ 2 H ) .
Proof.
First in Appendix G we have showed that F satisﬁes completeness.
By Proposition 1, we have d BEE ( F , (cid:15), Π , Π (cid:48) )
≤ max h ∈
[ H ] dim E ( F h , (cid:15) ) .
Therefore we only need to bound dim E ( F h , (cid:15) ) for each h ∈
[ H ] .
Suppose dim E ( F h , (cid:15) )
=
k > d ( (cid:15)/ 2 H ) .
Then by the deﬁnition of Eluder dimension, there exists a sequence φ 1 , · · · , φ k and { w 1 ,i } ki =1 , { w 2 ,i } ki =1 where φ
i ∈ X h
= { φ h ( s, a, b ) : ( s, a, b ) ∈ S ×
A × B} , w 1 ,i , w 2 ,i ∈
B H ( H − h + 1) for all i such that for any t ∈ [ k ] : [Equation] (cid:88) [Equation]
20) |(cid:104)
[Equation]
21) where (cid:15) (cid:48) ≥ (cid:15) .
Let Σ t denote (cid:80) t − 1 i =
φ
i φ (cid:62) i + (cid:15) 2 4 H 2 · I .
Then we have for any t ∈ [ k ]
[Equation]
On the other hand, by Cauchy-Schwartz inequality we know
[Equation]
This implies for all t ∈ [ k ]
[Equation]
Therefore, applying elliptical potential lemma (e.g., Lemma 5.6 and Lemma F.3 in [14]), we have for any t ∈ [ k ] log det
[Equation]
However, by the deﬁnition of effective dimension, we know when n = d eﬀ ( X h , (cid:15) 2 H ) ,
[Equation]
i =
φ i φ (cid:62) i (cid:19) ≤ ne − 1 .
This is a contradiction since n ≤ d ( (cid:15)/ 2 H ) <
k and log 32 > e − 1 .
Therefore we have dim E ( F h , (cid:15) )
≤ d ( (cid:15)/ 2 H ) for all h ∈
[ H ] , which implies d BEE ( F , (cid:15), Π , Π (cid:48) )
≤ d ( (cid:15)/ 2 H ) .
This concludes our proof.
Tabular MGs.
Tabular MGs are a special case of kernel MGs where the feature vectors are |S||A||B| -dimensional one-hot vectors.
From the standard elliptical potential lemma, we know d ( (cid:15) )
=
(cid:101) O ( |S||A||B| ) for tabular MDPs, suggesting their BEE dimension is also upper bounded (cid:101) O ( |S||A||B| ) .
Linear MGs.
When the feature vectors are d -dimensional, we can recover linear MGs.
Similarly, by the standard elliptical potential lemma, we have the BEE dimension of linear MGs is upper bounded (cid:101) O ( d ) .
H. G ENERALIZED L INEAR C OMPLETE M ODELS
An important variant of linear MDPs is the generalized linear complete models proposed by [52].
Here we also generalize it into Markov games:
Deﬁnition 12 (Generalized linear complete models) .
In d -dimensional generalized linear complete models, for each step h ∈
[ H ] , there exists a feature mapping φ h : S ×
A × B (cid:55)→ R d and a link function σ such that:
• for the generalized linear function class F h = { σ ( φ h ( · ) (cid:62) w ) | w ∈ W} where W ⊂ R d , realizability and completeness are both satisﬁed; • the link function is strictly monotone, i e., there exist 0 < c 1 < c 2 < ∞
such that σ (cid:48) ∈
[ c 1 , c 2 ] .
• φ h , w satisfy the regularization conditions: (cid:107) φ h ( s, a, b ) (cid:107) ≤ R, (cid:107) w (cid:107) ≤ R for all s, a, b, h where R > 0 is a constant.
When the link function is σ ( x ) = x , the generalized linear complete models reduce to the linear complete models, which contain instances such as linear MGs and LQRs.
The following proposition shows that generalized linear complete models also have low BEE dimensions:
Proposition 4.
If a generalized linear complete model has dimension d , then for any policy classes Π and Π (cid:48) , its BEE dimension can be bounded as follows:
[Equation]
Proof.
The proof is similar to Proposition 3, except (20) and (21) become
[Equation]
Then repeat the arguments in the proof of Proposition 3, we have dim E ( F h , (cid:15) ) ≤ (cid:101)
O ( dc 22 /c 21 ) for all h ∈ [ H ] .
Since F satisﬁes completeness, we can use Proposition 1 and obtain
[Equation]
I P ROOF OF T HEOREM 1 AND T HEOREM
In this section we present the proof for Theorem 1 and Theorem 2.
We ﬁrst consider the oblivious setting.
Let µ ∗ = arg max µ ∈ Π (cid:80) Kt =1 V µ × ν t( s 1 ) and we can decompose the regret into the following terms:
[Equation]
+
[Equation]
V t
[Equation]
p
[Equation]
(cid:105) (cid:124) (cid:123)(cid:122) (cid:125) (2)
[Equation]
+
[Equation]
K
[Equation]
t
[Equation]
(cid:104) V t , p t (cid:105) −
K (cid:88) t =
V t ( µ t ) (cid:124) (cid:123)(cid:122) (cid:125) (3) (cid:19) + (cid:18) K (cid:88) t =
V t ( µ t ) −
K (cid:88) t =
V π t 1 ( s 1 ) (cid:124) (cid:123)(cid:122) (cid:125) (4) (cid:19) .
Our proof bounds these terms separately and mainly consists of three steps:
• Prove V t ( µ ) is an optimistic estimation of V µ × ν t( s 1 ) for all t ∈
[ K ] and µ ∈ Π , which implies that term (1) ≤ 0 .
• Bound term (4), the cumulative estimation error (cid:80) Kt =1 V t ( µ t ) − V π t 1 ( s 1 ) .
In this step we utilize the newly proposed complexity measure BEE dimension to bridge the cumulative estimation error and the empirical Bellman residuals occurred in OptLSPE .
• Bound term (2) using the existing results of online learning error induced by Hedge and bound (3) by noticing that it is a martingale difference sequence.
I. S TEP 1: P ROVE O PTIMISM
First we can show that the constructed set B D 1: t − 1 ( µ, ν t ) is not vacuous in the sense that the true action-value function Q µ,ν t belongs to it with high probability
Lemma 2.
With probability at least 1 − δ/ 4 , we have for all t ∈ [ K ] , µ ∈ Π , Q µ,ν t ∈ B D 1: t − 1 ( µ, ν t ) .
Proof.
See Appendix J.1.
Then since V t ( µ ) = max f ∈B D 1: t − 1 ( µ,ν t ) f ( s 1 , µ, ν t ) , we know for all t ∈
[ K ] and µ ∈ Π ,
[Equation]
In particular, we have for all t ∈ [ K ] ,
[Equation]
I.
[Equation]
OUND E STIMATION E RROR
Next we need to show the estimation error (cid:80) Kt =1 V t ( µ t ) − V π t 1 ( s 1 ) is small.
Let f t,µ = [Equation]
, µ, ν t ) .
Then using standard concentration inequalities, we can have the following lemma which says that empirical Bellman residuals are indeed close to true residuals with high probability.
Recall that here π k = µ k × ν k .
Lemma 3.
With probability at least 1 − δ/ 4 , we have for all t ∈
[ K ] , h ∈ [ H ] and µ ∈ Π ,
[Equation]
Proof.
See Appendix J.2.
Besides, using performance difference lemma we can easily bridge V t ( µ t ) − V π t 1 ( s 1 ) with Bellman residuals, whose proof is deferred to Appendix J.3:
Lemma 4.
For any t ∈ [ K ] , we have
[Equation]
Therefore, from Lemma 4 we can obtain
[Equation]
µ
[Equation]
Notice that in (26) we need to bound the Bellman residuals of f t,µ t h weighted by policy π t .
However, in Lemma 3, we can only bound the Bellman residuals weighted by π 1: t − 1 .
Fortunately, we can utilize the inherent low BEE dimension to bridge these two values with the help of the following technical lemma:
Lemma 5 ([25]) .
Given a function class Φ deﬁned on X with φ ( x ) ≤ C for all ( φ, x ) ∈ Φ × X , and a family of probability measures Q over X .
Suppose sequence { φ t } Kt =1 ⊂ Φ and { ρ t } Kt =1 ⊂ Q satisfy that for all t ∈ [ K ] , (cid:80) t − 1 k
=
( E ρ k [ φ t ]) 2 ≤ β .
Then for all t ∈ [ K ] and w > 0 , t
[Equation]
Invoking Lemma 5 with Q = Q 1 h , Φ =
( I − T Π , Π (cid:48) h ) F and w = (cid:112) 1 /K , conditioning on the event (24) in Lemma 3 holds true, we have
K (cid:88)
[Equation]
Similarly, invoking Lemma 5 with Q = Q 2 h , Φ =
( I − T Π , Π (cid:48) h ) F and w = (cid:112) 1 /K , conditioning on the event (25) in Lemma 3 holds true, we have with probability at least 1 − δ/ 4 ,
K
[Equation]
where the ﬁrst inequality comes from standard martingale difference concentration.
Therefore, combining (27) and (28)
[Equation]
we have:
[Equation]
Substitute the above bounds into (26) and we have:
[Equation]
I. S TEP 3: B OUND THE REGRET
Now we only need to bound the online learning error.
Notice that p t is updated using Hedge with reward V t .
Since 0 ≤ V t ≤ V max and there are | Π | policies, we have from the online learning literature [20] that
[Equation]
In addition, suppose F k denotes the ﬁltration induced by { ν 1 } ∪ ( ∪ ki =1 { µ i , D i , ν i +1 } ) .
Then we can observe that (cid:104) V t , p t (cid:105) − V t ( µ t ) ∈ F t .
In addition, we have V t ∈ F t − 1 since the estimation of V t only utilizes D 1: t − 1 , which implies
[Equation]
Therefore (3) is a martingale difference sequence and by Azuma-Hoeffding’s inequality we have with probability at least 1 − δ/ 4 ,
[Equation]
Substituting (23), (29), (30), and (31) into (22) concludes our proof for Theorem 1.
For the adaptive setting, we can simply repeat the above arguments.
The only difference is that now ν t can depend on
D 1: t − 1 and thus we need to introduce a union bound over Π (cid:48) when proving Lemma 2 and Lemma 3.
This will incur an additional log | Π (cid:48) | in β and thus also in the regret bound.
This concludes our proof.
J P ROOFS OF L EMMAS IN A PPENDIX I J. P ROOF OF L EMMA
Let V ρ be a ρ -cover of G with respect to (cid:107) · (cid:107) ∞ .
Consider an arbitrary ﬁxed tuple ( µ, t, h, g ) ∈
Π × [ K ] ×
[ H ] × G .
Deﬁne W t,k ( h, g, µ ) as follows:
[Equation]
h +1 ( s kh +1 , µ, ν t ))− ( Q µ,ν t h ( s kh
[Equation]
a [Equation] b kh
[Equation] − [Equation] Q
[Equation]
ν t
[Equation]
1 ( s kh +1 , µ, ν t )) 2 , and F k,h be the ﬁltration induced by { ν 1 , · · · , ν K }∪{ s i 1 , a i 1 , b i 1 , r i 1 , · · · , s iH +1 } k − 1 i =1 ∪{ s k 1 , a k 1 , b k 1 , r k 1 , · · · , s kh , a kh , b kh } .
Then we have for all k ≤ t − 1 ,
E [ W t,k
( h, g, µ )
[Equation]
F
[Equation]
[Equation]
h )( s
[Equation]
and
Var [ W t,k ( h, g, µ ) | F k,h ] ≤ 4 V 2max E [ W t,k ( h, g, µ ) | F k,h ] .
By Freedman’s inequality, with probability at least 1 − δ/ 4 , we have (cid:12)(cid:12)(cid:12)(cid:12) [Equation] k = [Equation]
[Equation]
g
[Equation]
) −
[Equation]
(cid:88)
[Equation]
≤ O (cid:32)
V max (cid:118)(cid:117)(cid:117)(cid:116) log
[Equation]
(cid:88)
[Equation]
[Equation]
h
[Equation]
(cid:33) .
By taking union bound over Π ×
[ K ] ×
[ H ] × V ρ and the non-negativity of (cid:80) t − 1 k =
[( g h − Q µ,ν t h ) ( s kh , a kh , b kh )] 2 , we have with probability at least 1 − δ/ 4 , for all ( µ, k, h, g ) ∈
Π × [ K ] ×
[ H ] × V ρ , − t − 1 (cid:88) k
=
W t,k ( h, g, µ ) ≤
O ( V 2max ι ) , where ι = log( HK |V ρ || Π | /δ ) .
This implies for all ( µ, t, h, g ) ∈
Π × [ K ] ×
[ H ] × G , t − 1 (cid:88) k
=
[Equation]
+ O (
[Equation]
ι +
[Equation]
tρ
[Equation]
Choose ρ = V max /K
and we know that with probability at least 1 − δ for all µ ∈ Π and t ∈
[ K ] ,
Q µ,ν t ∈ B D 1: t − 1 ( µ, ν t ) .
This concludes our proof.
J. P ROOF OF L EMMA
Let Z ρ be a ρ -cover of F with respect to (cid:107) · (cid:107) ∞ .
Consider an arbitrary ﬁxed tuple ( µ, t, h, f ) ∈
Π × [ K ] ×
[ H ] × F .
Let
X t,k ( h, f, µ ) :=(
f h ( s kh , a
[Equation]
, b kh ) −
[Equation]
− f
[Equation]
+
( s
[Equation] +1 , µ, ν t )) − (( T µ,ν t h f h +1 ) ( s kh , a kh , b kh ) − r kh − f h +1 ( s kh +1 , µ, ν t )) 2 , and F k,h be the ﬁltration induced by { ν 1 , · · · , ν K }∪{ s i 1 , a i 1 , b i 1 , r i 1 , · · · , s iH +1 } k − 1 i =1 ∪{ s k 1 , a k 1 , b k 1 , r k 1 , · · · , s kh , a kh , b kh } .
Then we have for all k ≤ t − 1 ,
E [ X t,k ( h, f, µ ) | F k,h ] =
[( f h − T µ,ν t h f h +1 ) ( s kh , a kh , b kh )] 2 , and
Var [ X t,k
( h, f, µ )
| F k,h ] ≤ 4 V 2max E [ X t,k
( h, f, µ ) | F k,h ] .
By Freedman’s inequality, with probability at least 1 − δ , (cid:12) (cid:12)(cid:12)(cid:12) t − 1 (cid:88) k
= X t,k ( h, f, µ ) − t − 1 (cid:88) k = [( f h − T µ,ν t h f h +1 ) ( s kh , a kh , b kh )](cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
By taking union bound over Π ×
[ K ] ×
[ H ] × Z ρ , we have with probability at least 1 − δ , for all ( µ, t, h, f ) ∈
Π × [ K ] ×
[ H ] × Z ρ ,
[Equation]
where ι = log( HK |Z ρ || Π | /δ ) .
Conditioned on the above event being true, we consider an arbitrary pair ( h, t, µ ) ∈
[ H ] ×
[ K ] × Π .
By the deﬁnition of B D 1: t − 1 ( µ, ν t ) and Assumption 2, we have: t − 1 (cid:88) k =
[Equation]
k
[Equation]
≤ β
[Equation]
Let l t,µ = arg min l ∈Z ρ max h ∈
[ H ] (cid:107) f
t,µh − l t,µh (cid:107) ∞ .
By the deﬁnition of Z ρ , we have
[Equation]
By (32), we know:
[Equation]
[Equation]
Combining (33) and (34), we obtain
[Equation]
This implies that
[Equation]
Choose ρ = V max /K
and we can obtain (b).
For (a), simply let F k,h be the ﬁltration induced by { ν 1 , · · · , ν K } ∪ { µ i , s i 1 , a
i 1 , b i 1 , r i 1 , · · · , s iH +1 } k − 1 i =1 ∪ µ k and repeat the above arguments, which concludes our proof.
J. P ROOF OF L EMMA
First notice that V t ( µ t ) =
f t,µ t( s 1 , µ t , ν t ) .
Therefore, we have
[Equation]
E s 2 ∼ P 1 ( ·| s 1 ,a 1 ,b 1 ) [ f t,µ t( s 2 , µ t , ν t )] − E s 2 ∼ P 1 ( ·| s 1 ,a 1 ,b 1 )
[ V π t 2 ( s 2 )] (cid:3) + E a 1 ∼ µ t ( ·| s 1 ) ,b 1 ∼ ν t ( ·| s 1 ) [( f t,µ t− T µ t ,ν tf t,µ t)( s 1 , a 1 , b 1 )]
=
E s 2 ∼ π t [ f t,µ t( s 2 ,
[Equation]
t
[Equation]
ν t
[Equation]
π t 2 ( s
[Equation]
[Equation]
E π t [( f t,µ t− T µ t ,ν tf t,µ t)( s 1 , a 1 , b 1 )] .
Repeat the above procedures and we can obtain Lemma 4.
This concludes our proof.
K P ROOF OF C OROLLARY
From Theorem 2, we have with probability at least 1 − δ , for all i ∈ [ n ]
[Equation]
By the deﬁnition of (cid:98) π , this is equivalent to
[Equation]
where (cid:98) µ − i is uniformly sampled from { µ t − i } Kt =1 and thus is the marginal distribution of (cid:98) π over the agents other than i .
Therefore, by the deﬁnition of CCE in (2), (cid:98) π is (cid:15) -approximate
CCE with probability at least 1 − δ , which concludes our proof.
L P ROOF OF T HEOREM
In this section we present the proof for Theorem 3.
Our proof mainly consists of four steps:
• Prove V t r ( µ ) and V t g ( µ ) are optimistic estimations of V µ r, 1 ( s 1 ) and V µ g, 1 ( s 1 ) for all t ∈
[ K ] and µ ∈ Π . •
Bound the total estimation error (cid:80) Kt =1 V tr ( µ t ) − V µ t r, 1 ( s 1 ) and (cid:80) Kt =1 V tg ( µ t ) − V µ t g, 1 ( s 1 ) .
Bound the regret by decomposing it into estimation error and online learning error induced by Hedge.
Bound the constraint violation by strong duality.
Step 1: Prove optimism.
First we can show that the constructed set B D r 1: t − 1 ( µ ) ( B D g 1: t − 1 ( µ ) ) is not vacuous in the sense that the true action-value function Q µr ( Q µg ) belongs to it with high probability:
Lemma 6.
With probability at least 1 − δ/ 4 , we have for all t ∈
[ K ] and µ ∈ Π ,
Q µr ∈ B D r 1: t − 1 ( µ ) , Q µg ∈ B D g 1: t − 1 ( µ ) .
Proof.
The proof is almost the same as Lemma 2 and thus is omitted here.
Then since V tr ( µ ) = max f ∈B D g 1: t − 1 ( µ ) f ( s 1 , µ ) , we know for all t ∈ [ K ] and µ ∈ Π , [Equation] Similarly, we know V tg ( µ ) ≥ V µg, 1 ( s 1 ) .
Step 2: Bound estimation error.
Next we need to show the estimation error (cid:80) Kt =1 V tr ( µ t ) −
V µ t r, 1 ( s 1 ) and (cid:80) K t =1 V t g ( µ t ) − V µ t g, 1 ( s 1 ) are small.
Let f t,µ,r = arg max f ∈B D r 1: t − 1 ( µ )
f
( s 1 , µ ) and f t,µ,g = arg max f ∈B D g 1: t − 1 ( µ ) f
( s 1 , µ ) .
Then we have
Lemma 7.
With probability at least 1 − δ/ 4 , we have for all t ∈
[ K ] , h ∈ [ H ] and µ ∈ Π , ( a )
[Equation]
Proof.
The proof is almost the same as Lemma 3 and thus is omitted here.
Besides, using performance difference lemma we can easily bridge V tr ( µ t ) − V µ t r, 1 ( s 1 ) and V tg ( µ t ) −
V µ t g, 1 ( s 1 ) with Bellman residuals [Equation] whose proof is also omitted:
Lemma 8.
For any t ∈ [ K ] [Equation] we have
[Equation]
Therefore, from Lemma 8 we can obtain for any t ∈
[ K ] ,
[Equation]
which implies
K (cid:88)
[Equation]
[Equation]
Similar to Section I, from Lemma 5, conditioning on the event in Lemma 7 holds true, we have with probability at least 1 − δ/
[Equation]
Substitute the above bounds into (35) and we have:
[Equation]
Similarly, we have
[Equation]
Step 3:
Bound the regret.
Now we can bound the regret.
We ﬁrst decompose the ﬁctitious total regret (cid:80) Kt =1 ( V µ ∗
[Equation]
r
[Equation]
( s 1 ) + Y t V tg ( µ ∗ CMDP )) − (cid:80) Kt [-=
-]=1 ( V µ t r
[Equation]
1 ( s 1 ) + Y t V tg ( µ t )) to the following terms:
[Equation]
( V µ ∗ CMDP r,( s 1 ) + Y t V tg ( µ ∗ CMDP )) − K (cid:88) t = ( V µ t r, 1 ( s 1 ) + Y t V tg ( µ t )) =
(cid:18) K (cid:88) t = V µ ∗ CMDP r,( s 1 ) − K (cid:88) t = V tr ( µ ∗ CMDP ) (cid:124) (cid:123)(cid:122) (cid:125) (1) (cid:19) + (cid:18) K (cid:88) t =
( V tr ( µ ∗ CMDP )
+ Y t V tg ( µ ∗ CMDP ))
−
K (cid:88) t
= (cid:104) V tr
+ Y t V tg , p t (cid:105) (cid:124) (cid:123)(cid:122) (cid:125) (2) (cid:19) + (cid:18) K (cid:88) t
= (cid:104) V tr
+ Y t V tg , p t (cid:105) −
K (cid:88) t
= ( V tr ( µ t )
+ Y t V tg ( µ t ))
(cid:124) (cid:123)(cid:122) (cid:125) (3) (cid:19)
+ (cid:18) K (cid:88) t =
V tr ( µ t ) −
K (cid:88) t =
V µ t r, 1 ( s 1 ) (cid:124) (cid:123)(cid:122) (cid:125) (4) (cid:19) .
From Lemma 6, we know (1) ≤ 0 .
Since p t is updated using Hedge with loss function V t , we have (2) ≤ H (1 + χ ) (cid:112) K log | Π | .
(3) is a martingale difference sequence, which implies (3) ≤ O (cid:0) H (1 + χ ) (cid:112) K log(1 /δ )
(cid:1) with probability at least 1 − δ/ 4 .
Finally, Step 2 has bounded term (4) in (36), which implies K
[Equation] [Equation]
Now we only need to bound − (cid:80) Kt
=1 Y t ( V tg ( µ ∗
[Equation]
) − V tg ( µ t )) if we want to bound the regret (cid:80) Kt =1 ( V µ ∗ CMDP r,( s 1 ) − V µ t r, 1 ( s 1 )) .
In fact, updating the dual variable Y t with projected gradient descent guarantees us the following lemma:
Lemma 9.
Suppose the events in Lemma 6 hold true, we have
[Equation]
Proof.
See Appendix L.2.
Substituting Lemma 9 into (38), we can obtain the bound on Regret ( K ) :
[Equation]
Step 4: Constraint Violation Analysis.
Next we need to bound the constraint violation.
First notice that (cid:80) Kt =1 Y t ( b − V tg ( µ t )) is indeed not far from (cid:80) Kt =1 Y ( b − V tg ( µ t )) for any Y ∈ [0 , χ ] , as shown in the following lemma whose proof is deferred to Appendix L.3:
Lemma 10.
For any Y ∈ [0 , χ ] , we have
[Equation]
Substituting Lemma 10 into (38) and notice that b ≤ V µ ∗ CMDP g,( s 1 ) ≤ V tg ( µ ∗ CMDP ) , we have for any
Y ∈ [0 , χ ] , K
[Equation]
Combining the above inequality with (37), we have
K (cid:88)
[Equation]
BEE [Equation] , where (cid:15)
[Equation]
Choose Y as
[Equation]
χ
[Equation]
then we can bound the summation of regret and constraint violation as follows:
[Equation]
Further, when Assumption 6 and Assumption 7 hold, we have the following lemma showing that an upper bound on ( V µ ∗ CMDP r,( s 1 ) − 1 K (cid:80) Kt =
[Equation]
t r, 1 ( s 1 )) + χ [ b − 1 K (cid:80) Kt =1 V µ t g, 1 ( s 1 )] + implies an upper bound on [ b − 1 K (cid:80) Kt =1 V µ t g, 1 ( s 1 )]
+ :
Lemma 11.
Suppose Assumption 6 and Assumption 7 hold and 2 Y ∗ ≤ C ∗ .
If { µ t } Kt =1 ⊆ Π satisﬁes
[Equation]
Then
[Equation]
See Appendix L.4 for the proof.
Combining Lemma 11, Lemma 1 and (39), we have
[Equation]
(cid:19) .
This concludes our proof.
L. P ROOF OF L EMMA
Notice that D ( Y ∗ ) =
V µ ∗ CMDP r,( s 1 ) , which suggests:
[Equation]
This implies that
[Equation]
which concludes our proof.
L. P ROOF OF L EMMA
Notice that we have:
[Equation]
where the last step is due to optimism and V µ ∗ CMDP g,
[Equation]
≥ b .
This implies that
[Equation]
This concludes our proof.
L. P ROOF OF L EMMA
Notice that we have for any t ∈ [ K ] and Y ∈ [0 , χ ] :
[Equation]
Repeating the above expansion procedures, we have
[Equation]
which is equivalent to
[Equation]
This concludes our proof.
L. P ROOF OF L EMMA
First we extend Π in a reasonable way to make the policy class more structured while not changing its optimal policy.
Deﬁne the set of state-action visitation distributions induced by the policy Π as follows:
[Equation]
Let conv( P Π ) denote the convex hull of P Π , i.e., for any d ∈ conv( P Π ) , there exists { w µ } µ ∈ Π ≥such that for any h ∈ [ H ] , s ∈ S .a
∈ A , we have
[Equation]
As a special case, there exists d (cid:48) h
( s, a ) ∈ conv( P Π ) such that for any h ∈
[ H ] , s ∈ S .a
∈ A ,
[Equation]
Notice that there exists a one-to-one mapping from state-action visitation distributions to policies [41].
Let conv(Π) denote the policy class that induces conv( P Π ) , and then there exists µ (cid:48) such that d (cid:48)
=
d µ (cid:48) , which implies
[Equation]
Therefore, the condition of this lemma says
[Equation]
Next we show that µ ∗ CMDP is still the optimal policy in conv(Π) when Assumption 6, i.e., strong duality, holds.
First notice that
[Equation]
However, given Y ≥ 0 , L CMDP ( d, Y ) is linear in d , which means the maximum is always attained at the vertices of conv( P Π ) , i.e., P Π .
Therefore we know
[Equation]
which suggests
[Equation]
By strong duality, we have
[Equation]
Combining (41),(42) and (43), we know all the inequalities have to take equality, which implies
[Equation]
Besides, strong duality also holds for max µ ∈ {+conv(Π) min Y ≥ 
+}
L CMDP ( µ, Y ) .
Now let v ( τ ) :
= max µ ∈ conv( [Equation] { V µr [Equation]
s
[Equation]
|
[Equation]
µg
[Equation]
[Equation]
s 1 ) ≥ b + τ } , then we have for any µ ∈ conv(Π) ,
[Equation]
( s
[Equation]
, where the third step comes from strong duality.
Therefore, for any µ ∈ conv(Π) and τ ∈ R which satisﬁes V µg, 1 ( s 1 ) ≥ b + τ , we have
[Equation]
This implies that for any τ ∈ R , V µ ∗ CMDP r,
[Equation]
) − τY ∗ ≥ v ( τ ) .
Pick τ = (cid:101) τ := − [ b − V µ (cid:48) g, 1 ( s 1 )]
+ , then we have
[Equation]
On the other hand, (40) is equivalent to
[Equation]
Thus we have ( C ∗ − Y ∗ ) | (cid:101) τ |
≤ δ , which means that
[Equation]
Recall that V µ (cid:48) g, 1 ( s 1 ) =
1 K (cid:80) Kt =1 V µ t g, 1 ( s 1 ) , which concludes our proof.
M P ROOF OF T HEOREM
In this section we present the proof for Theorem 4.
Our proof mainly consists of four steps:
• Prove (cid:104) V t ( µ ) , θ t (cid:105) is a pessimistic estimations of (cid:104) V µ 1 ( s 1 ) , θ t (cid:105) for all t ∈
[ K ] and µ ∈ Π . •
Bound the total estimation error (cid:107) 1 K (cid:80)
Kt =1 V t ( µ t ) − V µ t 1 ( s 1 ) (cid:107) .
• Bound dist( V (cid:98) µ 1 ( s 1 ) , C ) .
Step 1: Prove pessimism.
First we can show that the true action-value function Q µ belongs to the constructed set B D 1: t − 1 ( µ ) with high probability:
Lemma 12.
With probability at least 1 − δ/ 4 , we have for all t ∈
[ K ] and µ ∈ Π , Q µ ∈ B D 1: t − 1 ( µ ) .
Proof.
Repeat the arguments in the proof of Lemma 2 for each dimension j ∈ [ d ] and the lemma follows directly.
Then since V t ( µ ) = f 1 ( s 1 , µ ) where f = arg min f (cid:48) ∈B D 1: t − 1 ( µ ) (cid:104) f (cid:48) 1 ( s 1 , µ ) , θ t (cid:105) , we know for all t ∈
[ K ] and µ ∈ Π ,
[Equation]
Step 2: Bound estimation error.
Next we need to show the estimation error (cid:107) 1 K (cid:80) Kt =1 V t ( µ t ) −
V µ t 1 ( s 1 ) (cid:107) is small.
Let f t,µ = arg min f ∈B D 1: t − 1 ( µ ) (cid:104) f
1 ( s 1 , µ ) , θ t (cid:105) .
Let f t,µ,j denotes the j -the dimension of f t,µ .
Then we have
Lemma 13.
With probability at least 1 − δ/ 4 , we have for all t ∈
[ K ] , h ∈ [ H ] , j ∈ [ d ] and µ ∈ Π ,
[Equation]
Proof.
Repeat the arguments in the proof of Lemma 3 for each dimension j ∈ [ d ] and the lemma follows directly.
Besides, using performance difference lemma we have:
Lemma 14.
For any t ∈ [ K ] and j ∈ [ d ] , we have
[Equation]
where V t,j ( µ t ) is the j -th dimension of V t ( µ t ) .
Therefore, from Lemma 14 we can obtain for any t ∈ [ K ] and j ∈ [ d ]
[Equation]
Similar to Section I, from Lemma 5, conditioning on the event in Lemma 13 holds true, with probability at least 1 − δ/ 4 , we have for any j ∈ [ d ] and h ∈ [ H ] ,
[Equation]
Substitute the above bounds into (44) and we have for any j ∈ [ d ] :
[Equation]
, which implies if the event in Lemma 13 is true,
[Equation]
Step 3:
Bound the distance.
Now we can bound the distance dist( V (cid:98) µ ( s 1 ) , C ) .
First since (cid:98) µ is sampled uniformly from { µ t } Kt =1 , we know dist
[Equation]
µ
[Equation]
, C
[Equation]
= dist
[Equation]
By Fenchel’s duality, we know dist
[Equation]
[Equation]
where the second step is due to max[ f 1 + f 2 ] ≤ max f 1 + max f 2 .
Notice by Cauchy-Schwartz inequality and Step 2, we have
[Equation]
Now we only need to bound max θ ∈ B
[Equation] [Equation]
1 K
[Equation]
Kt = [Equation] V
[Equation]
− max x ∈C [Equation] θ , x [Equation]
Recall that we update θ t using online gradient descent.
Using the conclusions from the online learning literature [20], we know
[Equation]
Further, notice that p t is updated via Hedge with loss function being (cid:104) θ t , V t ( µ ) (cid:105) , similarly to the analysis in Section I, we have with probability at least 1 − δ ,
[Equation]
where µ ∗ VMDP = arg min µ ∈ Π dist( V µ 1 ( s 1 ) , C ) .
Let P ( V µ ∗ VMDP( s 1 )) denote the projection of
V µ ∗ VMDP( s 1 ) onto C .
Conditioning on the event of Lemma 12 holds [Equation] we have
[Equation]
Therefore we have
[Equation]
where the second step is due to P ( V µ ∗ VMDP( s 1 )) ∈ C , the third step is from Cauchy-Schwartz inequality, and the last step is from the deﬁnition of µ ∗ VMDP .
In conclusion, we have with probability at least 1 − δ ,
[Equation]
This concludes our proof.