W HEN C AN W E L EARN G ENERAL -S UM M ARKOV G AMES WITH A L ARGE N UMBER OF P LAYERS S AMPLE -E FFICIENTLY ?
A BSTRACT
Multi-agent reinforcement learning has made substantial empirical progresses in solving games with a large number of players.
However, theoretically, the best known sample complexity for ﬁnding a Nash equilibrium in general-sum games scales exponentially in the number of players due to the size of the joint action space, and there is a matching exponential lower bound.
This paper investigates what learning goals admit better sample complexities in the setting of m -player general-sum Markov games with H steps, S states, and A i actions per player.
First, we design algorithms for learning an ε -Coarse Correlated Equilibrium (CCE) in (cid:101)
O ( H 5 S max i ≤ m A
i /ε 2 ) episodes, and an ε -Correlated Equilibrium (CE) in (cid:101)
O ( H 6 S max i ≤ m
A
i /ε 2 ) episodes.
This is the ﬁrst line of results for learning CCE and CE with sample complexities polynomial in max i ≤ m A i .
Our algorithm for learning CE integrates an adversarial bandit subroutine which minimizes a weighted swap regret, along with several novel designs in the outer loop.
Second, we consider the important special case of Markov Potential Games, and design an algorithm that learns an ε -approximate Nash equilibrium within (cid:101)
O ( S (cid:80) i
≤ m A
i /ε 3 ) episodes (when only highlighting the dependence on S , A i , and ε ), which only depends linearly in (cid:80) i
≤ m A i and signiﬁcantly improves over existing efﬁcient algorithms in the ε dependence.
Overall, our results shed light on what equilibria or structural assumptions on the game may enable sample-efﬁcient learning with many players.
I NTRODUCTION
Multi-agent reinforcement learning (RL) has achieved substantial recent successes in solving artiﬁcial intelligence challenges such as GO (Silver et al., 2016; 2018), multi-player games with team play such as Starcraft (Vinyals et al., 2019) and
Dota2 (Berner et al., 2019), behavior learning in social interactions (Baker et al., 2019), and economic simulation (Zheng et al., 2020; Trott et al., 2021).
In many applications, multi-agent RL is able to yield high quality policies for multi-player games with a large number of players (Wang et al., 2016; Yang et al., 2018).
Despite these empirical progresses, theoretical understanding of when we can sample-efﬁciently solve multi-player games with a large number of players remains elusive, especially in the setting of multi-player Markov games.
A main bottleneck here is the exponential blow-up of the joint action space —The total number of joint actions in a generic game with simultaneous plays is equal to the product of the number of actions for each player, which scales exponentially in the number of players.
Such an exponential dependence is indeed known to be unavoidable in the worst-case for certain standard problems.
For example, for learning an approximate Nash equilibrium from payoff queries in an one-step multi-player general-sum game, the query complexity lower bound of Chen et al. and Rubinstein (2016) shows that at least exponentially many queries (samples) is required, even when each player only has two possible actions and the query is noiseless.
Moreover, for learning Nash equilibrium in Markov games, the best existing sample complexity upper bound also scales with the size of the joint action space (Liu et al., 2021).
Nevertheless, these exponential lower bounds do not completely rule out interesting theoretical inquiries—there may well be other notions of equilibria or additional structures within the game that allow us to learn with a better sample complexity.
This motivates us to ask the following
Question : When can we solve general-sum Markov games with sample complexity milder than exponential in the number of players?
This paper makes steps towards answering the above question by considering multi-player generalsum Markov games (MGs) with m players, H steps, S states, and A i actions per player.
We make two lines of investigations: (1) Can we learn alternative notions of equilibria with better sample complexity than learning Nash; (2) Can the Nash equilibrium be learned with better sample complexity under additional structural assumptions on the game.
This paper makes contributions on both ends, which we summarize as follows
• We ﬁrst design an algorithm that learns the ε -approximate Coarse Correlated Equilibrium (CCE) with (cid:101) O
( H 5 S max i ∈ [ m ] A
i /ε 2 ) episodes of play (Section 3).
Our algorithm CCE-VL EARNING is a multi-player adaptation of the Nash V-Learning algorithm of Bai et al.
• We design an algorithm CE-V-L EARNING which learns the stricter notion of ε -approximate Correlated Equilibrium (CE) with (cid:101)
O
( H 6 max i ∈ [ m ] A 2 i /ε 2 ) episodes of play (Section 4).
For Markov games, these are the ﬁrst line of sample complexity results for learning CE and CCE that only scales polynomially with max i ∈ [ m ] A i , and improves signiﬁcantly in the A i dependency over the current best algorithm which scales with (cid:81) i ∈
[ m ] A i .
Technically, our algorithm CE-V-L EARNING makes several major modiﬁcations over CCEV-L EARNING in order to learn the CE (Section 4.2).
Notably, inspired by the connection between CE and low swap-regret learning, we use a mixed-expert Follow-The-Regularized Leader algorithm within its inner loop to achieve low swap-regret for a particular adversarial bandit problem.
Our analysis also contains new results for adversarial bandits on weighted swap regret and weighted regret with predicable weights, which may be of independent interest.
Finally, we consider learning Nash equilibrium in Markov Potential Games (MPGs), an important subclass of general-sum Markov games.
By a reduction to single-agent RL, we design an algorithm N ASH -CA that achieves (cid:101)
O (Φ max H 3 S (cid:80) i ∈
[ m ] A
i /ε 3 ) sample complexity, where Φ max ≤
Hm is the bound on the potential function (Section 5).
Compared with the recent result of Leonardos et al. (2021), we signiﬁcantly improves the ε dependence from their 1 /ε 6 .
1. R ELATED WORK
Learning equilibria in general-sum games
The sample (query) complexity of learning Nash,
CE, and CCE from samples in one-step (i.e. normal form) general-sum games with m players and A i actions per player has been studied extensively in literature (Hart & Mas-Colell, 2000;
Hart, 2005; Stoltz, 2005; Cesa-Bianchi & Lugosi, 2006; Blum & Mansour, 2007; Fearnley et al., 2015; Babichenko & Barman, 2015; Chen et al., 2015; Fearnley & Savani, 2016; Goldberg & Roth, 2016; Babichenko, 2016; Rubinstein, 2016; Hart & Nisan, 2018).
It is known that learning Nash equilibrium requires exponential in m samples in the worst case (Rubinstein, 2016), whereas CE and
CCE admit efﬁcient poly( m, max i ≤ m A i ) -sample complexity algorithms by independent no-regret learning (Hart & Mas-Colell, 2000; Hart, 2005; Syrgkanis et al., 2015; Goldberg & Roth, 2016;
Chen & Peng, 2020; Daskalakis et al., 2021).
Our results for learning CE and CCE can be seen as extension of these works into Markov games.
We remark that even when the game is fully known, the computational complexity for ﬁnding Nash in general-sum games is PPAD-hard (Daskalakis, 2013).
Markov games Markov games (Shapley, 1953; Littman, 1994) is a widely used framework for game playing with sequential decision making, e.g. in multi-agent reinforcement learning.
Algorithms with asymptotic convergence have been proposed in the early works of Hu & Wellman (2003); Littman (2001); Hansen et al. (2013).
A recent line of work studies the non-asymptotic sample complexity for learning Nash in two-player zero-sum Markov games (Bai & Jin, 2020; Xie et al., 2020; Bai et al., 2020; Zhang et al., 2020; Liu et al., 2021; Chen et al., 2021; Jin et al., 2021; Huang et al., 2021) and learning various equilibria in general-sum Markov games (Liu et al., 2021; Bai et al., 2021), building on techniques for learning single-agent Markov Decision Processes sample-efﬁciently (Azar et al., 2017; Jin et al., 2018).
Learning the Nash equilibrium in general-sum Markov games are much harder than that in zero-sum Markov games.
Liu et al.
(2021) present the ﬁrst line of results for learning Nash, CE, and CCE in general-sum Markov games; however their sample complexity scales with (cid:81) i ≤ m A i due to the model-based nature of their algorithm.
Algorithms for computing CE in extensive-form games has been widely studied (Von Stengel & Forges, 2008; Celli et al., 2020; Farina et al., 2021; Morrill et al., 2021), though we remark Markov games and extensive-form games are different frameworks and our results do not imply each other.
Markov potential games Lastly, a recent line of works considers Markov potential games (Macua et al., 2018;
Leonardos et al., 2021; Zhang et al., 2021), a subset of general-sum Markov games in which the Nash equilibrium admits more efﬁcient algorithms.
Leonardos et al.
(2021) gives a sample-efﬁcient algorithm based on the policy gradient method (Agarwal et al., 2021).
The special case of Markov cooperative games is studied empirically in e.g. Lowe et al.
Yu et al.
For one step potential games, Kleinberg et al. ; Palaiopanos et al. ; Cohen et al. (2017a) show the convergence to Nash equilibria of no-regret dynamics.
P RELIMINARIES
We present preliminaries for multi-player general-sum Markov games as well as the solution concept of (approximate) Nash equilibrium.
Alternative solution concepts and other concrete subclasses of
Markov games considered in this paper will be deﬁned in the later sections.
Markov games A multi-player general sum Markov game (MG; Shapley (1953); Littman (1994)) with m players can be described by a tuple
MG ( H, S , {A i } mi =1 , P , { r i } mi =1 ) , where H is the episode length, S is the state space with |S| = S ,
A i is the action space for the i th player with |A
i | =
A i .
Without loss of generality, we assume A i =
[ A i ] .
We let a :=
( a 1 , · · · , a m ) denote the vector of joint actions taken by all the players and
A =
A 1 × · · · × A m denote the joint action space.
Throughout this paper we assume that S and A i are ﬁnite.
The transition probability P =
{ P h } h ∈ [ H ] is the collection of transition matrices, where P h ( ·| s, a ) ∈
∆ S denotes the distribution of the next state when actions a are taken at state s at step h .
The rewards r i = { r h,i } h ∈
[ H ] ,i ∈ [ m ] is the collection of reward functions for the i th player, where r h,i ( s, a ) ∈ [0 , 1] gives the deterministic 1 reward of i th player if actions a are taken at state s at step h .
Without loss of generality, we assume the initial state s 1 is deterministic.
A key feature of general-sum games is that the rewards r i are in general different for each player i , and the goal of each player is to maximize her own cumulative reward.
Markov product policy, value function A product policy is a collection of m policies π := { π i } i ∈ [ m ] where π i is the general (potentially history-dependent) policy for the i -th player.
We ﬁrst focus on the case of Markov product policies, in which π i = { π h,i : S → ∆ A i } h ∈ [ H ] , and π h,i ( a i | s ) is the probability for the i th player to take action a i at state s at step h .
For a policy π and i ∈ [ m ] , we use π − i :=
{ π j } j ∈ [ m ] ,j (cid:54) =
i to denote the policy of all
but the i th player.
The value function V πh,i ( s ) :
S → R is deﬁned as the expected cumulative reward for the i th player when policy π is taken starting from state s and step h :
[Equation]
Best response & Nash equilibrium For any product policy π = { π i } i ∈ [ m ] , the best response for the i th player against π − i is deﬁned as any policy π † such that V π † ,π − i 1 ,
i
[Equation]
For any Markov product policy, this best response is guaranteed to exist (and be Markov) as the above maximization problem is equivalent to solving a Markov Decision Process (MDP) for the i th player.
We will also use the notation V † ,π − i 1 ,i ( s 1 ) to denote the above value function V π † ,π − i 1 ,i ( s 1 ) .
We say π is a Nash equilibrium (e.g. Nash (1951); Pérolat et al.
if all players play the best response against other players, i.e., for all i ∈ [ m ] ,
[Equation]
Note that in general-sum MGs, there may exist multiple Nash equilibrium policies with different value functions, unlike in two-player zero-sum MGs (Shapley, 1953).
To measure the suboptimality of any policy π , we deﬁne the NE-gap as
[Equation]
For any ε ≥ 0 , we say π is ε -approximate Nash equilibrium ( ε -Nash) if NE-gap ( π ) ≤ ε .
General correlated policy & Its best response A general correlated policy π is a set of H maps π :=
{ π h : Ω × ( S × A ) h − 1 × S → ∆ A } h ∈ [ H ] .
The ﬁrst argument of π h is a random variable ω ∈ Ω sampled from some underlying distribution, and the other arguments contain all the history information and the current state information (unlike Markov policies in which the policies only depend on the current state information).
The output of π h is a general distribution of actions in A =
A 1 ×· · ·×A m (unlike product policies in which the action distribution is a product distribution).
For any correlated policy π = { π h } h ∈
[ H ] and any player i , we can deﬁne a marginal policy π − i as a set of H maps π − i :=
{ π h, − i : Ω × ( S × A ) h − 1 × S →
∆ A − i } h ∈
[ H ] where A − i :=
A 1 × · · · × A i − 1 ×
A i +1 × · · · × A m , and the output of π h, − i is deﬁned as the marginal distribution of the output of π h restricted to the space
A − i .
For any general correlated policy π , we can deﬁne its initial state value function V π 1 ,i ( s 1 ) similar as (1).
The best response value of the i th player against π is V † ,π − i 1 ,i ( s 1 ) =
sup µ i V µ i ,π − i 1 ,i ( s 1 ) , where V µ i ,π − i 1 ,i ( s 1 ) is the value function of the policy ( µ i , π − i )
(the i th player plays according to general policy µ i , and all other players play according to π − i ), and the supremum is taken over all general policy µ i of the i th player.
Learning setting Throughout this paper we consider the interactive learning (i.e. exploration) setting where algorithms are able to play episodes within the MG and observe the realized transitions and rewards.
Our focus is on the PAC sample complexity (i.e. number of episodes of play) for any learning algorithm to output an approximate equilibrium.
2. E XPONENTIAL LOWER BOUND FOR LEARNING APPROXIMATE N ASH EQUILIBRIUM
The focus of this paper is the setting where the number of players m is large.
Intuitively, as the joint action space has size |A| =
(cid:81) mi =1 A i which scales exponentially in
m (if each A
i ≥ 2 ), naive algorithms for learning Nash equilibrium may learn all r i ( a ) by enumeratively querying all a ∈ A , and this costs exponential in m samples.
Unfortunately, recent work shows that such exponential in m dependence is unavoidable in the worst-case for any algorithm—there is an exp(Ω( m )) sample complexity lower bound for learning approximate Nash, even in one-step general-sum games (Chen et al., 2015; Rubinstein, 2016)
(see Proposition A.2 for formal statement).
This suggests that the Nash equilibrium as a solution concept may be too hard to learn efﬁciently for MGs with a large number of players, and calls for alternative solution concepts or additional structural assumptions on the game in order to achieve an improved m dependence.
E FFICIENT L EARNING OF C OARSE C ORRELATED E QUILIBRIA (CCE)
Given the difﬁculty of learning Nash when the number of players m is large , we consider learning other relaxed notions of equilibria for general-sum MGs.
Two standard notions of equilibria for games are the Correlated Equilibrium (CE) and Coarse Correlated Equilibrium (CCE), and they satisfy { Nash } ⊂ { CE } ⊂ { CCE } for general-sum MGs (Nisan et al., 2007).
We begin by considering learning CCE (most relaxed notion above) for Markov games.
Deﬁnition 1 ( ε -approximate CCE for general-sum MGs) .
We say a (general) correlated policy π is an ε -approximate Coarse Correlated Equilibrium ( ε -CCE) if
[Equation]
We say π is an (exact) CCE if the above is satisﬁed with ε = 0 .
The following result shows that there exists an algorithm that can learn an ε -approximate CCE in general-sum Markov games within (cid:101)
O ( H 5 S max i ∈ [ m ] A
i /ε 2 ) episodes of play.
Theorem 2 (Learning ε -approximate CCE for general-sum MGs) .
Suppose we run the CCE-VL EARNING algorithm (Algorithm 4) for all m players and K ≥ O
[Equation]
episodes ( ι = log( m max i ∈ [ m ] A i HSK/ ( pε )) is a log factor). Then with probability at least 1 − p , the certiﬁed policy (cid:98) π deﬁned in Algorithm 2 is an ε -CCE, i.e. max i ∈ [ m ] ( V † , (cid:98)
[Equation]
Mild dependence on action space For small enough ε , the sample complexity featured in Theorem 2 scales as (cid:101)
O ( H 5 S max i ∈ [ m ] A i /ε 2 ) .
Most notably, this is the ﬁrst algorithm that scales with max i ∈ [ m ] A i , and exhibits a sharp difference in learning Nash and learning CCE in view of the exp(Ω( m )) lower bound for learning Nash in Proposition A.2.
Indeed, existing algorithms such as
Multi-Nash-VI Algorithm with CCE subroutine (Liu et al., 2021) does require (cid:101) O
( H 4 S 2 (cid:81) mi =1 A
i /ε 2 ) episodes of play, which scales with (cid:81) i ∈
[ m ] A i due to its model-based nature.
We achieve signiﬁcantly better dependence on A i and also S , though slightly worse H dependence.
Our CCE-V-L EARNING algorithm (deferred to Appendix B.1 due to space limit) is a multi-player adaptation of the Nash V-Learning algorithm of Bai et al.
; Tian et al.
(2021) for learning Nash equilibria in two-player zero-sum MGs.
Similar as Bai et al.
(2020), we show that this algorithm enjoys a “no-regret” like guarantee for each player at each ( h, s ) (Lemma B.3). We also adopted the choice of hyperparameters in Tian et al.
(2021) so that the sample complexity has a slightly better dependence in H .
When combined with the “certiﬁed correlated policy” procedure (Algorithm 2), our algorithm outputs a policy that is ε -CCE.
Our certiﬁed policy procedure is adapted from the certiﬁed policy of Bai et al.
(2020), and differs in that ours output a correlated policy for all the players whereas Bai et al. (2020) outputs a product policy.
The key feature enabling this max i ∈ [ m ] A i dependence is that this algorithm attempts to learn the state value function V , which avoids learning the state action value function Q (as in Liu et al.
(2021)) that requires sample size scales as (cid:81)
i ∈ [ m ] A i . The proof of Theorem 2 is in Appendix B.
E FFICIENT L EARNING OF C ORRELATED E QUILIBRIA (CE)
In this section, we move on to considering the harder problem of learning Correlated Equilibria (CE).
We ﬁrst present the deﬁnition of CE in Markov games.
Deﬁnition 3 (Strategy modiﬁcation for i th player) .
A strategy modiﬁcation φ := { φ h,s } ( h,s ) ∈
[ H ] ×S for player
i is a set of H × S functions φ h,s : ( S × A )
h − 1 × A i
→ A i .
A strategy modiﬁcation φ can be composed with any policy π to give a modiﬁed policy φ (cid:5) π deﬁned as follows:
At any step h and state s with the history information τ h − 1 =
( s 1 , a 1 , · · · , s h − 1 , a h − 1 ) , if π chooses to play a = ( a 1 , . .
. , a m ) , the modiﬁed policy φ (cid:5) π will play ( a 1 , . . .
, a i − 1 , φ h,s ( τ h − 1 , a i ) , a i +1 , . .
. , a m ) .
We use Φ i denote the set of all possible strategy modiﬁcations for player i .
Deﬁnition 4 ( ε -approximate CE for general-sum MGs) .
We say a (general) correlated policy π is an ε -approximate CE ( ε -CE) if
[Equation]
We say π is an (exact) CE if the above is satisﬁed with ε = 0 .
Our deﬁnition of CE follows (Liu et al., 2021) and is a natural generalization of the CE for the well-studied special case of one-step (i.e. normal form) games (Nisan et al., 2007).
4. A LGORITHM DESCRIPTION
Our algorithm CE-V-L EARNING (Algorithm 1) builds further on top of CCE-V-L EARNING and
Nash V-Learning, and makes several novel modiﬁcations in order to learn the CE.
The key feature of
CE-V-L EARNING is that it uses a weighted swap regret algorithm (mixed-expert FTRL) for every ( s, h, i ) .
At a high-level, CE-V-L EARNING consists of the following steps:
• Line 6-11 (Sample action using mixed-expert FTRL):
For each ( h, s ) we maintain A i “subexperts” indexed by b (cid:48) ∈
[ A i ] (Each sub-expert represents an independent “expert” that runs her own FTRL algorithm).
Sub-expert b (cid:48)
ﬁrst computes an action distribution q b (cid:48) ( · ) ∈ ∆
A i via Follow-the-Regularized-Leader (FTRL; Line 8).
Then we employ a two-step sampling procedure to obtain the action: First sample a sub-expert b from a suitable distribution µ computed from { q b (cid:48) } b (cid:48) ∈
[ A i ] , then sample the actual action a h,i from q b .
• Line 13-17 (Take action and record observations
): Player i takes action a h,i and observes other player’s actions, the reward, and the next state.
Sub-expert b then computes a loss estimator and weight according to the observations, which will be used in future FTRL updates.
• Line 19 (Optimistic value update): Updates the optimistic estimate of the value V h,i using step-size α t and bonus β t .
Finally, after executing Algorithm 1 for K episodes, we use the certiﬁed correlated policy procedure (Algorithm 2) to obtain our ﬁnal output policy (cid:98) π .
This procedure is a direct modiﬁcation of the certiﬁed policy procedure of (Bai et al., 2020) and outputs a correlated policy (because the randomly sampled k and l in line 1 and line 4 of Algorithm 2 are used by all the players) instead of product policy.
The same procedure is also used for learning CCEs earlier in Section 3.
Here we specify the hyperparameters used in Algorithm 1:
[Equation]
The constants α jt used in Algorithm 2 is deﬁned as
[Equation]
Note that for any t ≥ 1 , { α jt } 1 ≤ j ≤ t sums to one and deﬁnes a distribution over [ t ] .
4. O VERVIEW OF TECHNIQUES
Here we brieﬂy overview the techniques used in Algorithm 1.
Minimizing swap regret via mixed-expert FTRL
The key technical advance in our Algorithmover CCE-V-L EARNING and Nash V-Learning is the use of mixed-expert FTRL (Line 6-11).
The purpose of this is to allow the algorithm to achieve low swap regret at each ( h, s ) in a suitable sense—For one-step (normal form) games, it is known that combining low-swap-regret learning for each player leads to an approximate CE (Stoltz, 2005; Cesa-Bianchi & Lugosi, 2006).
To integrate this into Markov games, we utilize a celebrated reduction from low-swap-regret learning to usual low-regret learning (Blum & Mansour, 2007), which for any bandit problem with A i actions maintains
A i sub-experts each running its own FTRL algorithm.
Our particular application builds upon the two-step randomization scheme of Ito (2020) which ﬁrst samples a sub-expert b and the action from this sub-expert.
The distribution µ ( · ) for sampling the sub-expert is carefully chosen by solving a linear system (Line 10) so that µ also coincides with the (marginal) distribution of the sampled action, from which the reduction follows.
FTRL with predictable weights Applied naively, the above reduction does not directly work for our purpose, as our analysis requires minimizing the weighted swap regret with weights α it , whereas the reduction of Ito (2020) relies crucially on the vanilla (average) regret.
We address this challenge by using a slightly modiﬁed FTRL algorithm for each sub-expert that takes in random but predictable weights (i.e. depending fully on prior information and “external” randomness).
We present the analysis for such FTRL algorithm in Appendix F.4, and the consequent analysis for the weighted swap regret in Appendix F.1-F.3, both of which may be of independent interest.
Proposal distributions Finally, a nuanced but important new design in CE-V-L EARNING is that all sub-experts compute a proposal action distribution to sample the sub-expert and the associated action. Require:
Hyperparameters: { α jt } 1 ≤ j ≤ t ≤ K , { α t } 1 ≤ t ≤ K , { η t } 1 ≤ t ≤ K , { ¯ β t } 1 ≤ t ≤ K .
1: Initialize:
For any ( s, a, h ) , set V h,i ( s ) ← H , N h ( s ) ← 0 .
Set µ h ( a | s ) ← 1 /A i , q b (cid:48)
h
( a | s ) ← 1 /A
i , (cid:96) b (cid:48) h,t ( a | s ) ← 0 , N b (cid:48) h
( s ) ← 0 for all ( b (cid:48)
, a, h, s, t ) ∈
[ A i ] × [ A i ] ×
[ H ] × S ×
[ K ] .
2: for episode k = 1 , . . .
, K do 3:
Receive s 1 .
4: for step h = 1 , . . .
, H do 5: //
Compute proposal action distributions by FTRL 6:
Update accumulator t :
=
N h
( s h )
← N h
( s h )
+ 1 .
Set u t ← α tt /α 1 t .
Let t b (cid:48) ← N b (cid:48)
h
( s h ) for all b (cid:48) ∈
[ A i ] for shorthand.
Compute the action distribution for all sub-experts b (cid:48) ∈
[ A i ] : q b (cid:48) h ( a | s h )
∝ a exp (cid:0) − ( η t b (cid:48)
/u t ) (cid:80) t b (cid:48) τ =
[Table]
w h,τ ( b (cid:48) | s h ) (cid:96) b (cid:48) h,
[Table]
( a | s h ) (cid:1) .
Sample sub-expert b and action from q b ( · ) 10:
Compute µ h ( ·| s h ) ∈ ∆ [ A i ] by solving µ h ( ·| s h ) =
(cid:80)
A i b (cid:48)
=1 µ h ( b (cid:48) | s h ) q b (cid:48)
h ( ·| s h ) .
Sample sub-expert b ∼ µ h ( ·| s h ) , and then action a h,i ∼ q bh ( ·| s h ) .
12: // Take action and feed the observations to sub-expert b 13:
Take action a h,i , observe the actions a h, − i from all other players.
Observe reward r h,i =
r h,i ( s h , a h,i , a h, − i ) and the next state s h +1 from the environment.
Update accumulator for the sampled sub-expert: t b := N bh ( s h ) ← N bh ( s h ) + 1 .
Compute and update loss estimator (cid:96) bh,t b ( a | s h ) ← (cid:2)
H − h + 1 − ( r h,i + min { V h
+1 ,
i ( s h +1 ) , H − h } ) (cid:3) /H
· 1 { a h,i = a } q bh ( a | s h ) + η t b .
Set w h,t b ( b | s h ) ← u t .
Optimistic value update 19:
V h,i ( s h ) ← (1 − α t ) V h,i ( s h ) + α t (cid:0) r h,i
( s h , a h,i , a h, − i )
+ V h
+1 ,
i ( s h +1 )
+ β t (cid:1) .
1: Sample k ← Uniform ([ K ]) .
2: for step h = 1 , . . .
, H do 3:
Observe s h , and set t ← N kh
( s h ) (the value of N h ( s h ) at the beginning of the k ’th episode). Sample l ∈ [ t ] with P ( l = j ) = α jt (c.f. Eq. Update k ← k lh ( s h ) (the episode at the end of which the state s h is observed exactly l times).
Jointly take action ( a h, 1 , a h, 2 , . .
. , a h,m ) ∼ (cid:81) mi =1 µ kh,i ( ·| s h ) ,
where µ kh,i ( ·| s h ) is the policy µ h,i ( ·| s h ) at the beginning of the k ’th episode.
Then, only the sampled sub-expert takes this action, and all other proposal distributions are discarded.
This is different from the original algorithms of (Blum & Mansour, 2007; Ito, 2020) in which the
FTRL update come after the sub-expert sampling and only happens for the sampled sub-expert.
Our design is required here as otherwise the sub-experts are required to predict the next time when it is sampled in order to compute the weighted FTRL update, which is impossible.
4. T HEORETICAL GUARANTEE
We are now ready to present the theoretical guarantee for our CE-V-L EARNING algorithm.
Theorem 5 (Learning ε -approximate CE for general-sum MGs) .
Suppose we run the CE-VL EARNING algorithm (Algorithm 1) for all m players for K ≥ O
[Equation]
episodes ( ι = log( m max i ∈ [ m ] A i HSK/ ( pε )) is a log factor).
Then with probability at least 1 − p , the certiﬁed correlated policy (cid:98) π deﬁned in Algorithm 2 is an ε -CE, i.e. max
[Equation]
Discussions To the best of our knowledge, Theorem 5 presents the ﬁrst result for learning CE that scales polynomially with max i ∈ [ m ] A i , which is signiﬁcantly better than the best known existing algorithm of Multi-Nash-VI with CE subroutine (Liu et al., 2021) whose sample complexity scales with (cid:81) i ∈
[ m ] A i .
Similar as in Theorem 2, this follows as our CE-V-L EARNING attempts to learn the state value function V instead of the state-action value function Q .
We also observe that our sample complexity for learning CE is higher than for learning CCE by a factor of (cid:101)
O
( H max i ∈ [ m ] A i ) ; the additional max i ∈ [ m ] A i factor is expected as CE is a strictly harder notion of equilibrium.
The proof of Theorem 5 can be found in Appendix C.
L EARNING N ASH E QUILIBRIA IN M ARKOV P OTENTIAL G AMES
In this section, we consider learning Nash equilibria in Markov Potential Games (MPGs), an important subclass of general-sum MGs.
Despite the curse of number of players of learning Nash in general-sum
MGs, recent work shows that learning Nash in MPGs does not require sample size exponential in m , by using stochastic policy gradient based algorithms (Leonardos et al., 2021;
Zhang et al., 2021).
In this section, we provide an alternative algorithm N ASH -CA that also achieves a mild dependence on m and an improved dependence on ε by a simple reduction to single-agent learning.
5. M ARKOV POTENTIAL GAMES
We ﬁrst present the deﬁnition of MPGs.
Our deﬁnition is the ﬁnite-horizon variant 2 of the deﬁnitions of Macua et al.
; Leonardos et al.
; Zhang et al.
and is slightly more general as we only require (4) on the total return.
Throughout this section, π denotes a Markov product policy.
Deﬁnition 6.
(Markov potential games)
A general-sum Markov game is a Markov potential game if there exists a potential function Φ mapping any product policy to a real number in [0 , Φ max ] , such that for any i ∈ [ m ] , any two policies π i , π (cid:48)
i of the i th player, and any policy π − i of other players, the difference of the value functions of the i th player with policies ( π i , π − i ) and ( π (cid:48)
i , π − i ) is equals the difference of the potential function on the same policies, i.e.,
[Equation]
Note that the range of the potential function Φ max admits a trivial upper bound
Φ max ≤ mH (this can be seen by varying π i for one i at a time).
An important example of MPGs is Markov Cooperative
Games (MCGs) where all players share the same reward r
i ≡ r .
5. A LGORITHM AND THEORETICAL GUARANTEE
We present a simple algorithm N ASH -CA (Nash Coordinate Ascent) for learning an ε -Nash in MPGs.
As its name suggests, the algorithm operates by solving single-agent Markov Decision Processes (MDPs) one player at a time, and intrinsically performing coordinate ascent on the potential function of the Markov game.
Due to the potential structure of MPGs and the boundedness of the potential function, the local improvements of players across the steps can have an accumulative effect on the potential function, and the algorithm is guaranteed to stop after a bounded number of steps.
We give the full description of the N ASH -CA in Algorithm 3.
We remark that N ASH -CA is additionally guaranteed to output a pure-strategy Nash equilibrium (cf.
Appendix D for deﬁnition).
Theorem 7 (Sample complexity for N ASH -CA) .
For Markov potential games, with probability at least 1 − p , Algorithm 3 terminates within 4Φ max /ε steps of the while loop, and outputs an ε -approximate (pure-strategy)
Nash equilibrium.
The total episodes of play is at most
[Equation]
where ι = log( mHSK max 1 ≤ i ≤ m A i εp ) is a log factor.
Require: Error tolerance ε do 3:
Execute 4: for player i = 1 ,
, m do 5:
Fix π − i , and let the i th player run UCBVI-UPLOW (Algorithm 7) for K i =
Θ( H 3 SA i ι ε+
H 3 S 2 A
i ι 2 ε ) episodes and get a new deterministic policy (cid:98) π i . Execute policy π π for N Θ episodes and obtain i π π is the empirical average return under policy
Set ∆ i ← (cid:98) V
i π π
− (cid:98) V
8: if max m
>
/ 2 then 9:
Update π j ← where j = arg max i [ m
∆ i .
10: else 11: return π
Discussions For small enough ε , the sample complexity for the N ASH -CA algorithm in the above theorem is (cid:101) O
(Φ max H 3 S (cid:80) i ≤ m A
i /ε 3 ) .
As Φ max ≤ mH , this at most scales with the number of players as m (cid:80) i ≤ m A i , which is much better than the exponential in m sample complexity for general-sum MGs without additional structures.
Compared with recent results on learning Nash via policy gradients (Leonardos et al., 2021; Zhang et al., 2021), the N ASH -CA algorithm also achieves poly( m, max i ≤ m A i ) dependence, and signiﬁcantly improves on the ε dependence from their ε −to ε − 3 .
In addition, our algorithm does not require assumptions on bounded distribution mismatch coefﬁcient as they do, due to the exploration nature of our single-agent MDP subroutine.
Also, compared with the sample complexity bound (cid:101) O
( H 4 S 2 (cid:81) mi =1 A i /ε 2 ) of the Nash-VI algorithm (Liu et al., 2021) for general-sum MGs (not restricted to MPGs), our N ASH -CA algorithm doesn’t suffer from the exponential dependence on m thanks to the MPG structure.
We do achieve a looser in the dependence on ε , yet overall our sample complexity is still better unless ε < ( (cid:80) mi =1 A i ) /
( (cid:81) mi =1 A i ) is exponentially small.
The proof of Theorem 7 can be found in
Appendix D. A lower bound To accompany Theorem 7, we establish a sample complexity lower bound of Ω( H 2 (cid:80) mi =1 A i /ε 2 ) for learning pure-strategy Nash in MCGs and hence MPGs (Theorem E.1 in
Appendix E).
This lower bound improves in the A i dependence over the naive reduction to singleplayer MDPs (Domingues et al., 2021), which gives Ω( H 3 S max i ∈ [ m ]
A i /ε 2 ) , though is loose on the S, H dependence.
The improved A i dependence is achieved by constructing a novel class of hard instances of on one-step games (Lemma E.2), which may be of further technical interest.
However, there is still a large gap between these lower bounds and the best current upper bound of either our (cid:101) O
( (cid:80) mi =1 A
i /ε 3 ) or the (cid:101) O
( (cid:81) mi =1 A
i /ε 2 ) of Liu et al.
(2021), which we leave as future work.
C ONCLUSION
This paper investigates the question of when can we solve general-sum Markov games (MGs) sampleefﬁciently with a mild dependence on the number of players.
Our results show that this is possible for learning approximate (Coarse)
Correlated Equilibria in general-sum MGs, as well as learning approximate Nash equilibrium in Markov potential games.
In both cases, our sample complexity bounds improve over existing results in many aspects.
Our work opens up many interesting directions for future work, such as sharper algorithms for both problems, sample complexity lower bounds, or how to perform sample-efﬁcient learning in general-sum MGs with function approximations.
In addition to Markov potential games, it would also be interesting to explore alternative structural assumptions that permit sample-efﬁcient learning. A CKNOWLEDGEMENT
Ziang Song is partially supported by the elite undergraduate training program of School of Mathematical Sciences in Peking University
A E XPONENTIAL IN m L OWER B OUND FOR L EARNING N ASH IN G ENERAL - SUM MG S
In this section, we give a sample complexity lower bound for computing approximate Nash equilibrium in one-step binary-action general-sum MGs ( H = 1 , S = 1 and A i = 2 ) which has an exponential dependence in m , the number of players.
The result is built on the lower bound of query complexity in Rubinstein (2016).
We use G to denote the one-step Markov game ( H = 1 and S = 1 ), in which there are m players and
A = 2 actions for each player.
We index the players by [ m ] = { 1 , . . .
, m } and denote the actions space of each player by [ A ] = { 1 , 2 } .
Since we restricted attention to binary-action games (i.e.
A = 2 ), the total number of joint actions is 2 m .
We deﬁne a (exact) query as the procedure where the algorithm queries a joint action a ∈ [ A ] m and observes the (deterministic) reward r i ( a ) ∈ [0 , 1] .
We deﬁne the query complexity (Chen et al., 2015) for learning ε -approximate
Nash equilibrium (ANE) as the following.
Deﬁnition A.1 (Query complexity) .
The query complexity QC p (ANE( m, ε )) for learning ε - ANE is deﬁned as the smallest n such that there exists a randomized oracle algorithm
A satisfying the following: for any binary-action, m-player game G , the algorithm A can use no more than n sequential queries of the reward to output an ε - ANE with probability at least 1 − p .
In one-step MGs with deterministic reward, the query complexity is equivalent to the sample complexity, since each query obtains a reward entry.
The following result in Rubinstein (2016) gives a 2 Ω( m ) query complexity lower bound for learning ε 0 -ANE in m -player binary action games. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
L h,i ( s h , a ) ← (1 − α t )
L h,i ( s h , a ) + α t L h,i ( s h , a ) 13:
Set µ h ( ·| s h )
∝ exp[ − ( η t /α t )
L h,i ( s h , · )] .
Proposition A.2 (Corollary 4.5, (Rubinstein, 2016)) .
There exists absolute constants ε 0 > 0 and c > 0 , such that for all m ,
[Equation]
where p = 2 − cm . This result shows that it is impossible for any algorithm to learn an ε 0 -ANE for every binary action game with probability at least (1 − p ) using poly( m, log(1 /p ))
samples: such an algorithm with p = 2 − cm would only use poly( m, log(2 cm ))
= poly( m ) samples, yet the sample complexity lower bound in Proposition A.2 requires at least 2 Ω( m )
=
exp(Ω( m ))
samples.
Since Proposition A. allows ε 0 = Θ(1) , this also rules out the possibility of learning ε -ANE with poly( m, log(1 /p ) , 1 /ε ) samples for all small ε .
B P ROOFS FOR S ECTIONB. A LGORITHM FOR LEARNING CCE IN GENERAL - SUM M ARKOV GAMES
Our algorithm used to learn CCE in general-sum MGs is a combination of Algorithm 4 and Algorithm 2.
In particular, Algorithm 4 computes a set of policies and plays these policies in each episode.
Algorithm 2 used the full history in Algorithm 4 to produce a certiﬁed, general correlated policy which we will show to be a CCE (we will also use the same Algorithm 2 to produce the certiﬁed policy in the algorithm of learning CE).
During the execution of Algorithm 2, if the index t is 0 at some step h , the certiﬁed policy can choose any action at and after step h .
In Algorithm 4, we choose the hyper-parameters as follows:
[Equation]
where c > 0 is some absolute constant, and ι = log( m max i ∈
[ m ] A i HSK pε ) is a log factor.
The choice of η t follows the V-OL algorithm in Tian et al.
(2021) which helps to shave off an H factor in the sample complexity compared with the original Nash V-Learning algorithm in Bai et al.
Here, we have a short comment on the log factor ι .
In fact, we need ι to be C log( m max i ∈ [ m ] A i HSK pε ) for some absolute constant C .
For the cleanness of the results, in this paper, we ignore this difference since this would not harm the correctness of all the results we present.
B. P ROOF OF T HEOREM
We begin with an auxiliary lemma on α jt (its deﬁnition is in (3)).
Lemma B.1 (Lemma 4.1 in Jin et al.
The following properties hold for α jt :
1. 1 √ t ≤ (cid:80) tj =1 α jt √ j ≤ 2 √ t for every t ≥ 1 . 2.
max j ∈ [ t ] α jt ≤ 2 Ht and (cid:80) tj =1 (cid:16) α jt (cid:17) 2 ≤ 2 Ht for every t ≥ 1 . 3.
(cid:80) ∞ t =
j α jt = 1 + 1 H for every j ≥ 1 . 4.
(cid:80) tj =
α jt j ≥ 12 t for every t ≥ 1 .
Property 4 above does not appear in (Jin et al., 2018), for which we provide a quick proof here:
[Equation]
Here, (i) uses α jt is increasing in j for ﬁxed t .
Some notations The following notations will be used repeatedly (throughout this section and the next section). At the beginning of any episode k for a particular state s h , we denote k 1 h ( s h ) < · · · < k N kh
( s h ) h
( s h ) < k to be all the episodes that the state s h was visited, where N kh ( s h ) is the times the state s h has been visited before the start of k -th episode. When there is no confusion, we sometimes will write k j = k jh ( s h ) in short.
For player i , we let V kh,i , V kh,i , µ ki denote the values and policies maintained by Algorithm 4 at the beginning of k -th episode, and a kh denote taken action at step h and episode k .
For any joint policy µ h (over all players), reward function r and value function V , we deﬁne the operators P h and D µ h as
[Equation]
Towards proving Theorem 2, we begin with a simple consequence of the update rule in Algorithm 4, which will be used several times later.
Lemma B.2 (Update rule) .
Fix a state s in time step h and ﬁx an episode k , let t =
N kh ( s ) and suppose s was previously visited at episodes k 1 < · · · < k t < k at the h -th step.
The update rules in Algorithm 4 gives the following equations:
[Equation]
We next present and prove the following lemma which helps to explain why our choice of the bonus term is β t .
The constant c in β t is actually the same with the constant c in this lemma.
Lemma B.3 (Per-state guarantee) .
Fix a state s in time step h and ﬁx an episode k , let t =
N kh ( s ) and suppose s was previously visited at episodes k 1 < · · · < k t < k at the h -th step.
With probability at least 1 − p 2 , for any ( i, s, h, t ) ∈ [ m ] × S × [ H ] ×
[ K ] , there exist a constant c s.t.
[Equation]
Proof of Lemma B.
First, we decompose max µ
[Equation]
) −
[Equation]
+ min {
[Equation]
[Equation]
[Equation] + [Equation] h } (cid:21) into R (cid:63) ( i, s, h, t )
+ U
( i, s, h, t ) where
R (cid:63) ( i, s, h, t ) :
= max µ t (cid:88)
j = α
[Equation]
i (cid:18) r [Equation] i [Equation]
P h min
[Equation]
V
[Equation]
[Equation]
} (cid:19) (
[Equation]
) −
[Equation]
(cid:88)
[Equation]
=
[Equation]
(cid:18)
[Equation]
P
[Equation]
min
[Equation] + [Equation] h } (cid:19) ( s ) , and
U ( i, s, h, t ) := t [Equation] =
[Equation]
(cid:19) ( [Equation] [Equation] =
[Equation]
a
[Equation]
j [Equation] a k [Equation]
+
[Equation]
{ V k j
[Equation]
+
[Equation]
[Equation] j [Equation] +1 )
[Equation]
We ﬁrst bound U ( i, s, h, t ) .
Deﬁne F l as the σ -algebra generated by all the random variables up to the time when s h is observed at the l -th episode.
Recall that for j ≥ 1 , k j = k jh ( s ) =
inf { l > k j − 1 : s is visited at step h in episode l } (with convention k 0 = 0 ).
Then { k j } j ≥ 1 is a sequence of increasing stopping times w.r.t.
{F l } l ≥ 1 .
Deﬁne G j = F k j +1 . So {G j } j ≥ 0 is also a ﬁltration.
Under G j − 1 ( = F k j ), by the deﬁnition of operator D and P , we have
[Equation]
So we can apply Azuma-Hoeffding inequality.
Note that (cid:80) tj =1 ( α jt ) 2 ≤ 2 H/t by Lemma B.1.
Using
Azuma-Hoeffding inequality, we have with probability at least 1 −
[Equation]
i,
[Equation]
After taking a union bound, we have the following statement is true with probability at least 1 − p/ 4 ,
U ( i, s, h, t ) ≤ 2 (cid:112) H 3 ι/t for all ( i, s, h, t ) ∈ [ m ] × S ×
[ H ] ×
[ K ] .
Then we bound R (cid:63) ( i, s, h, t ) .
For ﬁxed ( i, s, h ) , if we deﬁne the loss function
[Equation]
then R (cid:63) ( i, s, h, t ) =
H max µ (cid:80) tj =1 α jt (cid:68) µ − µ k j h,i , (cid:96) j (cid:69) becomes the weighted regret with weight α jt .
Note that the update rule for µ k j h,i ( ·| s ) is essentially performing Follow-the-Regularized-Leader (FTRL) algorithm with changing step size for each state s and each step h to solve an adversarial bandit problem.
Lemma 17 in (Bai et al., 2020) 3 bounds the weight regret with high probability.
Using that lemma, we have with probability at least 1 − p 4 mHS ,
R (cid:63) ( i, s, h, t ) ≤
Hα tt log A i η t
[Equation]
HA
[Equation]
[Equation]
(cid:118)(cid:117)(cid:117) (cid:116)
[Equation]
simultaneously for all t ∈
[ K ] .
By Lemma B.1 and η t = (cid:113) HιA i t , we have with probability at least 1 − p 4 mHS R
[Equation]
i
[Equation]
[Equation]
t
[Equation]
≤
[Equation]
for all
[Equation]
∈ [ K ] . Again, taking a union bound in all ( i, s, h ) ∈ [ m ] × S ×
[ H ] , we have with probability at least 1 − p/ 4 ,
R (cid:63) ( i, s, h, t ) ≤ 10 (cid:112) H 3 A i ι/t + 10 H 2 ι/t for all ( i, s, h, t ) ∈
[ m ] × S ×
[ H ] ×
[ K ] .
Finally, we concluded that with probability at least 1 − p/ 2 , we have
U
[Equation]
cH
[Equation]
t for all ( i, s, h, t ) ∈ [ m ] × S ×
[ H ] ×
[ K ] for some absolute constant c .
Recall that the certiﬁed policy (cid:98) π as in Algorithm 2 is a nested mixture of policies.
We further deﬁne policies { (cid:98) π k h } h ∈ [ H ] ,k ∈
[ K ] in Algorithm 5.
By construction, the relationship between (cid:98) π and (cid:98) π k h is that when players jointly play policy the (cid:98) π , {+they ﬁrst sample k from Uniform ([ K ]) , then they play together the policy (cid:98) π k 
+}
(Algorithm 5 for h = 1 ) with the same sampled k .
As a result, we have the following relationship:
[Equation]
Deﬁnition B.4 (Policy starting from the h -th step) . We deﬁne the policy starting from the h -th step for player i as π ≥ h,i := { π h (cid:48) ,i : Ω × ( S × A ) h (cid:48) − h × S → ∆ A i } Hh (cid:48) =
h .
At each step h (cid:48) ≥ h , π ≥ h,i samples action based on current state, the history starting from the h -th step and a random number ω ∈ Ω .
We use Π ≥ h,i to denote all policies for player i starting from the h -th step.
Similar to Section 2, we can deﬁne general correlated policy starting from the h -th step (where the random numbers may be correlated for different players), and we use Π ≥ h to denote all such general correlated policy starting from the h -th step.
For π ∈ Π ≥ h , we can deﬁne the value function starting from the h -th step as:
[Equation]
We also deﬁne the value function of the best response as:
[Equation]
One example of a policy starting from the h -th step is (cid:98) π kh deﬁned in Algorithm 5, so that we can deﬁne V (cid:98)
π kh h,i ( s ) and V † , (cid:98) π kh, − i h,i ( s ) .
Lemma B.5 (Valid upper and lower bounds) .
We have
[Equation]
for all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] with probability at least 1 − p .
Proof of Lemma B.
We prove this lemma by backward induction over h ∈ [ H + 1] .
The base case of h = H + 1 is true as all the value functions equal 0 by deﬁnition.
Suppose the claim is true for h
+ 1 .
We begin with upper bounding V † , (cid:98) π kh, − i h,i ( s ) . Let t = N kh ( s ) and k j = k jh ( s ) for 1 ≤ j ≤ t to be the j ’th time that s is previously visited. By the deﬁnition of certiﬁed policies (cid:98) π kh,i and by the value iteration formula of MGs, we have for any policy µ i ∈ Π
[Equation]
Here, µ h,i is the policy µ i at the h -th step, and ( µ h +1: H,i | s, a ) is the policy µ i from time h + 1 to H with history information at h -th step to be s h = s and a h
= a .
By the deﬁnition of Π ≥ h,i , we have ( µ h +1:
H,i | s, a ) ∈ Π ≥
h +1 ,i which implies the inequality in the equation above.
By taking supremum over µ i ∈ Π ≥ h,i and using the deﬁnition of the operator D in (6), we have
[Equation]
Conditional on the high probability event in Lemma B.3, we use the inductive hypothesis to obtain
V † , (cid:98) π kh, − i h,i ( s ) ≤ sup µ i t (cid:88)
j = α jt
[Equation]
, − i [
[Equation]
[Equation]
[Equation]
s ) ≤
[Equation]
(cid:88)
j = α
[Equation]
(cid:20)
[Equation]
[Equation]
(cid:16)
[Equation]
a
[Equation]
h , a
[Equation]
j h, − i (cid:17)
[Equation]
[Equation]
h [Equation] 1 ,i ( [Equation]
j h
[Equation]
1 ) , H
[Equation]
h
[Equation]
(cid:21)
[Equation]
(cid:112) H
[Equation]
A
[Equation]
ι
[Equation]
t + cH 2 ι/t ( i ) ≤ t (cid:88) j = α jt (cid:20) r h,i (cid:16) s, a k j h , a k j h, − i (cid:17) + min { V k j h
+1 ,
i ( s k j h +1 ) , H − h } + β j (cid:21) ≤ t (cid:88)
j = α jt (cid:20) r
[Equation]
[Equation]
(cid:16)
[Equation]
, a
[Equation]
j h , [Equation]
k j h,
[Equation]
i (cid:17) [Equation] j h
[Equation]
1 ,i (cid:16) s k
j h +1 (cid:17) + ¯ β j (cid:21) =
V kh,i ( s ) .
Here, (i) uses our choice of β j = c (cid:113) H 3 A i ι j + 2 c
H 2 ι j and 1 √ t ≤ (cid:80) tj {+=
+} α jt √ j , 1 t ≤ (cid:80) tj =1 2 α jt j .
Meanwhile, for V kh,i ( s ) , by the deﬁnition of certiﬁed policy and inductive hypothesis,
[Equation]
Then we note that
[Equation]
is a martingale difference w.r.t.
the ﬁltration {G j } j ≥ 0 , which is deﬁned in the proof of Lemma B.3.
So by Azuma-Hoeffding inequality, with probability at least 1 − [Equation]
(cid:88)
[Equation]
On this event, we have t (cid:88)
j =
[Equation]
Here, (i) uses (cid:80) tj =
α jt β j ≥ 2 (cid:80) tj =
α jt (cid:112) H 3 ι/j ≥ 2 (cid:112) H 3 ι/t.
As a result, the backward induction would work well for all h as long as the inequalities in
Lemma B.3 and (9) hold for all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] .
Taking a union bound in all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] , we have with probability at least 1 − p/ 2 , the inequality in (9) is true simultaneously for all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] .
Therefore the inequalities in Lemma
B.3 and (9) hold simultaneously for all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] with probability at least 1 − p .
This ﬁnishes the proof of this lemma.
Equipped with these lemmas, we are ready to prove Theorem 2.
Proof of Theorem
Conditional on the high probability event in Lemma B.5 (this happens with probability at least 1 − p ), we have
[Equation]
for all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] .
Then, choosing h = 1 and s = s 1 , we have
[Equation]
Moreover, by (7), value function of certiﬁed policy can be decomposed as
[Equation]
where the decomposition is due to the ﬁrst line in the Algorithm 2: sample k ← Uniform( [ K ] ).
So we have
[Equation]
To prove (cid:98) π is an approximate CCE, we only need to bound (cid:80) Kk =1 (cid:16)
V k 1 ,i ( s 1 ) − V k 1 ,i ( s 1 ) (cid:17) .
Letting δ kh,i :
= V kh,i ( s kh )
− V kh,i ( s kh ) and t = N kh ( s kh ) .
Suppose s kh was previously visited at episodes k 1 , . .
k t at the h -th step.
By the update rule,
[Equation]
H
[Equation]
j .
We can use Lemma B.1 which gives (cid:80) tj =1 α jt / √ j ≤ 2 /t and max j ≤ t α jt ≤ 2 H/t to get δ
[Equation]
i ≤ α 0 t H +
[Equation]
where we also uses (cid:80) tj =1 1 /j ≤ 1 + log t .
[-+1 ,i + 4 c (cid:
-]
Taking the summation w.r.t.
k, we begin by the ﬁrst two terms;
[Equation]
where (i) is by changing the order of summation and (ii) is by Lemma B.1.
So
K
[Equation]
K (cid:88) k
=  (cid:113) N kh ( s )
+ 8 cH 3 ι
K (cid:88) k
= 1 + log N kh ( s ) N kh ( s ) .
By pigeonhole argument,
K (cid:88) k =  (cid:113) N k h ( s ) = (cid:88) s ∈S
N Kh ( s ) (cid:88) n = 1 √ n ≤ O (1) √ SK.
Similarly,
K (cid:88) k
= [Equation]
where we assume ι ≥ 1 .
So we have
[Equation]
Recursing this argument for h ∈ [ H ] gives
[Equation]
To conclude,
[Equation]
Therefore, K ≥ O ( H 5 S max i ∈ [ m ] A i ι ε+ SH 4 ι 3 ε ) guarantees that we have V † , (cid:98) π − i [Equation] i [Equation]
s [Equation] ≤ ε for all i ∈ [ m ] .
This complete the proof of Theorem 2.
C P ROOFS FOR S ECTION
In this section we prove Theorem 5. We ﬁrst deﬁne a set of lower value estimates V kh,i ( s ) (along with the upper estimates used in Algorithm 1) via the following update rule:
[Equation]
We emphasize that V kh,i ( s ) are analyses quantities only for simplifying the proof, and are not used by the algorithm.
We restate and use several notations we introduced in the last section. At the beginning of any episode k for a particular state s h , we denote k 1 h ( s h ) < · · · < k N kh
( s h ) h
( s h ) < k to be all the episodes that the state s h was visited, where N kh ( s h ) is the times the state s h has been visited before the start of k -th episode. When there is no confusion, we sometimes will write k j = k jh ( s h ) in short.
For player i , we let V k h,i , V k h,i , µ ki denote the values and policies maintained by Algorithm 4 at the beginning of k -th episode, and a kh denote taken action at step h and episode k .
For any joint policy µ h (over all players), reward function r and value function V , we deﬁne the operators P h and D µ h as
[Equation]
The following lemma is the same as Lemma B.2 in the CCE case (except that for a different algorithm).
Lemma C.1 (Update rule) .
Fix a state s in time step h and ﬁx an episode k , let t =
N kh ( s ) and suppose s was previously visited at episodes k 1 < · · · < k t < k at the h -th step.
The update rule for V h,i ( s ) and V h,i ( s ) in Algorithm 1 and (10) gives the following equations:
[Equation]
We next prove the following lemma which helps explain our choice of the bonus term β t .
The constant c in β t is the same with the constant c in this lemma.
For any policy modiﬁcation ϕ i : A i → A i for the i th player and one-step policy π h : S → ∆ A for any h , the modiﬁed policy ϕ i (cid:5) π h is deﬁned as follows: if π h chooses to play a
= ( a 1 , . . . , a m ) , the modiﬁed policy ϕ i (cid:5) π h will play ( a 1 , . . . , a i − 1 , ϕ i ( a i ) ,
a i +1 , . . . , a m ) . Moreover, for π h,i : S → ∆ A i , policy ϕ i (cid:5) π h,i chooses ϕ i ( a ) when π h,i chooses a .
Lemma C.2 (Per-state guarantee) . Fix a state s in time step h and ﬁx an episode k , let t = N kh ( s ) and suppose s was previously visited at episodes k 1 < · · · < k t < k at the h -th step.
With probability 1 − p 2 , for any ( i, s, h, t ) ∈ [ m ] × S × [ H ] ×
[ K ] , there exist a constant c s.t.
[Equation]
Proof of Lemma C. First, like the prove in Lemma B.3, we decompose sup ϕ i : A i →A i t
[Equation]
=
[Equation]
(cid:18) r
[Equation]
+
[Equation]
+
[Equation]
[Equation]
h } (cid:19) ( s ) − t (cid:88) j = α [Equation]
r
[Equation]
, a
[Equation]
, a
[Equation]
−
[Equation]
(cid:17) + min { V
[Equation]
+
[Equation] ( s [Equation]
j
[Equation]
+1 ) , H − h } (cid:21) into R (cid:63) ( i, s, h, t ) + U ( i, s, h, t ) where
R (cid:63) ( i, s, h, t ) :=
sup ϕ
i : A i →A i t (cid:88) j = α jt D ϕ i (cid:5) µ kjh (cid:18) r h,i + P h min {
V k j h
+1 ,i , H − h } (cid:19) ( s ) − t (cid:88) j = α jt
[Equation]
− i (cid:20) r h,i (cid:16) s, a
[Equation]
j h
[Equation]
a
[Equation]
i (cid:17) + E s h +1 ∼ P h ( ·| s,a kjh , a − i ) min { V k j h +1 , i ( s h +1 ) , H − h } (cid:21) , and
U ( i, s, h, t ) :
= t (cid:88)
j =
[Equation]
(cid:20) r
[Equation]
(cid:16)
[Equation]
[Equation]
(cid:17) [Equation] ∼ [Equation]
[Equation]
a − i )
[Equation]
We ﬁrst bound U ( i, s, h, t ) .
By the same reason in proof of Lemma B.3, we can apply AzumaHoeffding inequality.
Note that (cid:80) tj =1 ( α jt ) 2 ≤ 2 H/t by Lemma B.1.
Using Azuma-Hoeffding inequality, we have with probability at least 1 −
[Equation]
After taking a union bound, the following statement is true with probability at least 1 − p/ 4 ,
U ( i, s, h, t ) ≤ 2 (cid:112) H 3 ι/t for all ( i, s, h, t ) ∈ [ m ] × S ×
[ H ] ×
[ K ] .
Then we bound R (cid:63) ( i, s, h, t ) .
For ﬁxed ( i, s, h ) , we deﬁne loss function (cid:96) j ( a ) [Equation] [Equation] a i
[Equation]
a [Equation] a − i ∼ µ k j h, − i [ H − h + 1 − r h,i ( s, a ) − E
[Equation]
h +1 ∼ P h ( ·| s,a, a − i ) min { V k j h +1 , i ( s h
+
[Equation]
, H − h } ] .
[Table]
Then we have (cid:96) j ( a ) ∈ [0 , 1] for all a and the realized loss function:
[Equation]
h
[Equation]
] ∈ [0 , 1] is an unbiased estimator of (cid:96) j ( a k j h ) .
Then R (cid:63) ( i, s, h, t ) can be written as [Equation]
[Equation]
Now, for any ﬁxed step h and state s , the distributions (cid:8) q bh ( ·| s ) (cid:9) b and visitation counts (cid:8) N bh ( s ) (cid:9) b are only updated at episodes k 1 h ( s ) , . . .
, k th ( s ) .
Further, these updates are exactly equivalent to the mixed-expert FTRL update algorithm which we describe in Algorithm 8.
Therefore, the R (cid:63) ( i, s, h, t ) above can be bounded by the weighted swap regret bound of Lemma F.1 (choosing the log term as ι = 4 log 10 mSAHKp ) to yield that
[Equation]
with probability at least 1 − p/ (4 mSH ) .
Taking a union bound over all ( i, s, h ) ∈ [ m ] × S ×
[ H ] , we have with probability at least 1 − p/ 4 ,
[Equation]
[Equation]
t for all ( i, s, h, t ) ∈ [ m ] × S ×
[ H ] ×
[ K ] .
Finally, we conclude that with probability at least 1 − p/ 2 , we have
U {+( i, s, h, t ) + R (cid:63) ( i, s, h, t ) ≤ cH 
 A i (cid:112) ι/t + cH 2 A+}
i ι/t for all ( i, s, h, t ) ∈ [ m ] × S ×
[ H ] ×
[ K ] for some absolute constant c .
We deﬁne the auxiliary certiﬁed policies (cid:98) π kh in Algorithm 6 (same as Algorithm 5 for the CCE case but repeated here for clarity). Again, we have the following relationship: [Equation]
[Equation] Deﬁnition C.3 (Policy modiﬁcation starting from the h -th step) .
A strategy modiﬁcation starting from the h -th step φ ≥ h :=
{ φ h (cid:48) ,s } ( h (cid:48) ,s ) ∈{ h,h
+1 ,...,H }×S for player i is a set of S × ( H − h + 1) functions φ h (cid:48) ,s : ( S × A ) h (cid:48) − h ×
{+A i → A i . This strategy modiﬁcation φ ≥ h can be composed with any policy π ∈ Π ≥ h (as in Deﬁnition B.4) to give a modiﬁed policy φ ≥ h (cid:5) π deﬁned as follows: At any step h (cid:48) ≥ h and state s with the history information starting from the h -th step τ h : h (cid:48) − 
 =+}
( s h , a h , · · · , s h (cid:48) − 1 , a h (cid:48) − 1 ) , if π chooses to play a = ( a 1 , . .
. , a m ) , the modiﬁed policy φ (cid:5) π will play ( a 1 , . . .
, a i − 1 , φ h (cid:48) ,s ( τ h : h (cid:48) − 1 , a i ) , a i +1 , . . . , a m ) .
We use Φ ≥ h,i denote the set of all such possible strategy modiﬁcations for player i .
For any φ ∈ Φ ≥ h,i , φ (cid:5) (cid:98) π kh also doesn’t depend on the history before the h -th step, so φ (cid:5) (cid:98) π kh ∈ Π ≥ h , which implies that V φ (cid:5) (cid:98) π kh h,i ( s ) is well-deﬁned in (8).
Lemma C.4 (Valid upper and lower bounds) .
We have
[Equation]
for all ( i, k, h, s ) ∈ [ m ] × [ K ] ×
[ H ] × S with probability at least 1 − p/ 2 .
Proof of Lemma C.
We prove this lemma by backward induction over h ∈ [ H + 1] .
The base case of h = H + 1 is true as all the value functions equal 0 .
Suppose the claim is true for h
+ 1 .
We begin with upper bounding sup φ ∈ Φ ≥ h,i V φ
[Equation] π kh [Equation] i ( s ) .
Let t =
N kh ( s ) and k j = k jh ( s ) for 1 ≤ j ≤ t to be the j ’th time that s is previously visited. By the deﬁnition of certiﬁed policies (cid:98) π kh,i and by the value iteration formula of MGs, we have for any φ ∈ Φ ≥ h,i ,
[Equation]
Here, φ h,i is the strategy modiﬁcation function φ at the h -th step, and ( φ ≥ h +1 | s, a ) is the modiﬁcation φ from the h + 1 -th step with history information to be s h = s and a h
= a .
By the deﬁnition of Φ ≥ h,i , we have ( φ ≥ h +1 | s, a ) ∈
Φ ≥ h +1 ,i which implies the above inequality.
By taking supremum over φ ∈ Φ ≥ h,i and using the deﬁnition of the operator D in (6), we have
[Equation]
Condition on the high probability event (with probability at least 1 − p/ 2 ) in Lemma C.2, we can use the inductive hypothesis to obtain
[Equation]
Here, (i) uses our choice of β j = cH 2 A i (cid:113) ιj + 2 c H 2 A
i ι j and 1 √ j ≤ (cid:80) tj =
α jt √ j , 1 t ≤ (cid:80) tj =1 2 α jt j {+, so that cH 
 A i (cid:112) ι/t + cH 2 A i ι/t ≤ (cid:80) α jt β j+} .
Meanwhile, for V kh,i ( s ) , by the deﬁnition of certiﬁed policy and inductive hypothesis,
[Equation]
Note that
[Equation]
j ≥ 1 is a martingale difference sequence w [Equation] r
[Equation]
t. the ﬁltration {G j } j ≥ 0 , which is deﬁned in the proof of
Lemma B.3.
So by Azuma-Hoeffding inequality, with probability at least 1 −
[Equation]
On this event, we have
[Equation]
Here, (i) uses (cid:80) tj =
α jt β j ≥ 2 (cid:80) tj =
α jt (cid:112) H 3 ι/j ≥ 2 (cid:112) H 3 ι/t.
As a result, the backward induction would work well for all h as long as the inequalities in
Lemma C.2 and (12) hold for all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] .
Taking a union bound in all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] , we have with probability at least 1 − p/ 2 , the inequality in (12) is true simultaneously for all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] .
Therefore the inequalities in Lemma B.3 and (12) hold simultaneously for all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] with probability at least 1 − p .
This ﬁnishes the proof of this lemma.
Equipped with these lemmas, we are ready to prove Theorem 5.
Proof of Theorem
Conditional on the high probability event in Lemma C.4 (this happens with probability at least 1 − p ), we have
[Equation]
for all ( i, s, h, k ) ∈ [ m ] × S ×
[ H ] ×
[ K ] .
Then, choosing h = 1 and s = s 1 , we have
[Equation]
Moreover, by (11), value function of certiﬁed policy can be decomposed as
[Equation]
where the decomposition is due to the ﬁrst line in the Algorithm 2: sample k ← Uniform( [ K ] ).
Therefore we have the following bound on sup φ ∈
Φ i V φ (cid:5) (cid:98) π 1 ,i ( s 1 ) − V (cid:98)
π 1 ,i ( s 1 ) :
[Equation]
[Equation]
By Lemma C.4 Letting δ kh,i :=
V kh,i ( s kh )
− V kh,i ( s kh ) and t =
N kh ( s kh ) .
By the update rule, we have
[-j = 
-]
[Equation]
Taking the summation w.r.t.
k, by the same argument in the proof of Theorem 2, we can get
[Equation]
,i ( s 1 ) (cid:17) = 1 K
K (cid:88) k
= δ k 1 ,i ≤ O (1) SH 4 max i ∈
[ m ] A i ι 3 /K + O (cid:32)(cid:114)
H 6 S max i ∈ [ m ] {+A 
+}
i ι/K (cid:33) .
Therefore, if K ≥ O ( H 6 S max i ∈ [ m ] A 2 i ι ε+ H 4 S max i ∈ [ m ] A i ι 3 ε ) , we have max φ ∈
Φ V φ (cid:5) (cid:98) π 1 ,i ( s 1 ) −
V (cid:98) π 1 ,i ( s 1 ) ≤ ε holds for all i ∈ [ m ] , which means (cid:98) π is an ε -approximate CE.
This completes the proof of Theorem 2.
D P ROOFS FOR S ECTION
Here, we ﬁrst deﬁne pure-strategy Nash equilibrium.
We say policy π is a pure strategy (deterministic policy)
if and only if for any ( h, i, s ) ∈ [ H ] ×
[ m ] ×S , π h,i ( a i | s )
=
I a i = a (cid:48) h,i ( s ) for some a (cid:48) h,i ( s ) .
We say π a pure-strategy Nash equilibrium if π is a pure-strategy and is a Nash equilibrium.
Similarly, we say π is a pure strategy ε -approximate Nash equilibrium if π is a pure strategy and NE-gap ( π )
≤ ε .
Pure-strategy Nash equilibrium does not always exist in general-sum MGs, but is guaranteed to exist (as we will see) in Markov Potential Games.
D. E XISTENCE OF PURE - STRATEGY N ASH EQUILIBRIA IN M ARKOV POTENTIAL GAMES
A particular property of MPGs is that, there always exists a pure-strategy Nash equilibrium.
Such a property does not hold for every general-sum MG.
Pure-strategy Nash equilibria are preferred in many scenarios since each player can take deterministic actions.
Proposition D.1.
For any Markov potential games, there exists a pure-strategy Nash equilibrium.
See Theorem 3.1 in Leonardos et al.
(2021) or Proposition 1 in Zhang et al.
for a proof of
Proposition D.1.
D. T HE UCBVI-UPLOW SUB - ROUTINE
In this subsection we consider the problem of learning a near optimal policy in the ﬁxed horizon stochastic reward RL problem
MDP( H, S , A , P , r ) .
The setting is standard ( c.f. Jin et al. and is a special case of Markov games (c.f. Section 2) by setting the number of agents m = 1 . We will use the same notations including policies and value functions as that of the Markov games as do 3: do 4: do 5:
Set t then 7: β ← B
V
V
1 + V s a c f
Eq.
(13)) 8: γ ←
V
− V s a
Q
←
V γ β
Q
← max
P V
− γ β
11: for s do 12: π arg
V s
; V
Receive the initial
15: for do 16:
Take denote estimates policy introduced in Section 2, except that we will omit the sub-scripts i ’s since here we just have a single agent.
We consider the UCBVI-UPLOW algorithm (Algorithm 7), which is adapted from (Xie et al., 2021;
Liu et al., 2021), for learning approximate optimal policy in reinforcement learning problems.
Such an algorithm is used as a sub-routine in Algorithm 3 to learn approximate pure-strategy Nash equilibria in MPGs.
We remark that although in Algorithm 3 we propose to use the UCBVI-UPLOW algorithm to search for a near optimal policy, many alternative algorithms can be used to ﬁnd the near optimal policy (e.g., UCBVI (Azar et al., 2017) or Q-learning (Jin et al., 2018)).
Here we choose the UCBVIUPLOW algorithm because 1) it has a tight sample complexity bound; 2) it outputs a deterministic policy which can be used to ﬁnd a pure-strategy approximate Nash equilibrium.
In the description of Algorithm 7, the (cid:98) P h quantity appeared in lines 8,9,10, and 18 can be viewed either as a set of empirical probability distributions or as an operator: for any ﬁxed ( s h , a h ) , we can view (cid:98) P h ( ·| s h , a h ) as a probability distribution over S ; for any given function V : S → R , we can view (cid:98) P h as an operator by deﬁning ( (cid:98) P h V )( s, a ) :
=
E s (cid:48) ∼ (cid:98) P h ( ·| s,a ) [ V ( s (cid:48) )] .
The (cid:98) V h operator in line 7 is the empirical variance operator deﬁned as (cid:98) V h V
= (cid:98) P
h V 2 − ( (cid:98) P h V ) 2 .
The BONUS function in the algorithm is chosen to be the Bernstein type bonus function
[Equation]
We have the following sample complexity guarantee for the UCBVI-UPLOW algorithm returning an ε -approximate optimal policy.
Lemma D.2.
The UCBVI-UPLOW algorithm always returns a deterministic policy π k (cid:63) .
Moreover, for any p ∈ (0 , 1] , letting ι = log( SAHK/p ) and taking the number of episodes
[Equation]
then with probability at least 1 − p , the returned policy π k (cid:63) is ε -approximate optimal, i.e., sup µ V µ 1 ( s 1 ) − V π k(cid:63)
[Equation]
Proof of Lemma D.
First, π k (cid:63) is obviously deterministic from line 12 of Algorithm 7.
The sample complexity guarantee of the UCBVI-UPLOW algorithm is a consequence of the sample complexity guarantee of the Nash-VI algorithm for learning Nash in zero-sum Markov games as proved in Liu et al.
More speciﬁcally, we denote the rewards and transition matrices by { r h ( s, a )
} h ∈ [ H ] and P = { P h ( s h
+1 | s h , a h ) } h ∈ [ H ] for the MDP( H, S , A , P , r ) .
Such a MDP can be viewed as a zero-sum
Markov game MG( H, S , A , B , P , ¯ r ) , in which ( H, S , A ) are the same as that of the MDP, the action space for the min-player is a singleton B = 1 so that the rewards ¯ r h ( s, a, b ) ≡ r h
( s, a ) do not depend on the action of the min-player and the transition matrices P h ( ·| s, a, b )
=
P h ( ·| s, a 1 ) do not depend on the action of the min-player either.
Then, for any ε -approximate Nash equilibrium ( µ, ν ) of the associated zero-sum Markov game, the max-player’s policy µ must be ε -approximate optimal for the original MDP.
By this correspondence, the UCBVI-UPLOW algorithm is actually a speciﬁc version of the Nash-VI algorithm in Liu et al.
(2021), and
Line 12 in UCBVI-UPLOW is actually a speciﬁc version of line 12 in Nash-VI in Liu et al.
(2021): this is because in this speciﬁc Markov game, Q ( s, a, b ) and
Q ( s, a, b ) only depend on s and a , so that π h ( s ) =
arg max a ∈A Q h ( s, a ) is actually in the CCE set
CCE ( Q h ( s, a, b ) , Q h ( s, a, b )) .
By this reduction and by Theorem 4 in Liu et al.
(2021), this lemma is proved.
D. P ROOF OF T HEOREM
We use superscript t to represent variables at the t -th step (before π is updated) of the while loop.
Because we can choose the log factor as ι = 4 log( mHSK max i ∈ [ m ] A i εp ) (this doesn’t affect the correctness of the theorem), for each execution of UCBVI-UPLOW, by Lemma D.2, it return a ε/ 4 -optimal deterministic policy with probability at least 1 − pε 8 m 2 H .
Taking a union bound, we have
[Equation]
simultaneously for all i ∈ [ m ] and t ≤ 4 mH/ε with probability at least 1 − p/ 2 .
For the empirical estimator (cid:98) V t 1 ,i , it’s bounded in [0 , H ] .
Thus by Hoeffding’s inequality, for ﬁxed i ∈ [ m ] and t
[Equation]
Choosing N = CH 2 ι/ε 2 for some large constant C , we have
[Equation]
Apply this inequality to (cid:98) V t 1 ,i ( (cid:98) π ti , π t − i ) and (cid:98) V 1 ,i ( π t ) and taking a union bound, we have
[Equation]
[Equation] simultaneously for all i ∈ [ m ] and t ≤ 4 mHε with probability at least 1 − p/ 2 .
As a result, by (14) and (15), we have
[Equation]
16) simultaneously for all i ∈ [ m ] and t ≤ 4 mH/ε with probability at least 1 − p .
On this event,
[Equation]
If the while loop doesn’t end after the t -th iteration and t ≤ 4 mH/ε , there exists j t s.t.
∆ tj t ≥ ε/ 2 , so we have
[Equation]
Here, ( i ) follows the deﬁnition of potential function.
Because Φ is bounded, the while loop ends within 4Φ max /ε ≤ 4 mH/ε steps.
Therefore, (16) holds simultaneously for all i ∈ [ m ] and t before the end of while loop with probability at least 1 − p .
Again, on this event, if the while loop stops at the end of t -th step, we have max i ∈ [ m ] ∆ ti ≤ ε/ 2 , then
[Equation]
So the returned policy π t is a ε -approximate Nash equilibrium.
Moreover, since UCBVI-UPLOW outputs a pure-strategy policy and our initial policy is also a pure-strategy policy, we can conclude that with probability at least 1 − p , within 4Φ max /ε steps of the while loop, Algorithm 3 outputs an ε -approximate (pure-strategy)
Nash equilibrium.
Finally, the number of episodes within each step of the while loop is
[Equation]
So the total sample complexity (episodes) is at most
[Equation]
O
[Equation]
This concludes the proof.
E L OWER B OUND OF F INDING A PPROXIMATE P URE -S TRATEGY N ASH E QUILIBRIUM
In this section, we present an result on the sample complexity lower bound for learning a pure-strategy Nash equilibrium in MPGs (a harder task than learning Nash as pure-strategy Nash is a stricter notion).
We remark that our lower bounds are actually constructed on Markov Cooperative Games (MCGs) which is a subset of MPGs.
Note that for MCGs, the potential function Φ is bounded in [0 , H ] , so by Theorem 7, the sample complexity of Nash-CA (Algorithm 3) is (cid:101) O
( (cid:80) mi =1 A i /ε 3 ) highlighting the dependency on ε , m and
A i , i = 1 , . . .
, m .
We would show this (cid:80) mi =1 A i dependency is inevitable for learning ε -approximate pure-strategy Nash equilibrium in MCGs by proving an Ω( (cid:80) mi
=1 A i /ε 2 ) lower bound.
We ﬁrst present our main theorem:
Theorem E.1 (Lower bound for learning pure-strategy Nash in MCGs) .
Suppose A i = 2 k , i = 1 , . . .
, m , H ≥ 2 , S ≥ 3 and m ≥ 4 .
Then, there exists an absolute constant c 0 such that for any ε ≤ 0 .
4 and any online ﬁnetuning algorithm M that outputs a pure-strategy policy (cid:98)
π = ( (cid:98) π 1 , . .
, (cid:98) π m ) , if the number of episodes
[Equation]
then there exists general-sum Markov cooperative game MG on which the algorithm M suffers from ε/ 4 -suboptimality, i.e.
[Equation]
where the expectation E M is w.r.t.
the randomness during the algorithm’s execution within Markov game MG .
This theorem can be viewed as a corollary of the following lemma by a simple reduction.
We would prove this theorem in the next subsection.
One-step (general-sum) game is a game with only one state and one step.
In a one-step game, each player chooses an action simultaneously and then receive it’s own reward.
The Nash equilibrium and NE-gap can be deﬁned similarly in one-step games.
Lemma E.2 (Lower bound for one-step game) .
Suppose A i = [-
-]2 k , i = 1 , . . .
, m and m ≥ 4 .
Then, there exists an absolute constant c 0 such that for any ε ≤ 0 .
4 and any online ﬁnetuning algorithm that outputs a pure strategy (cid:98)
π = ( (cid:98) π 1 , . .
, (cid:98) π m ) , if the number of samples n ≤ c(cid:80)
[Equation]
then there exists a one-step game M with stochastic reward , on which the algorithm suffers from ε/ 4 -suboptimality, i.e.
[Equation]
where the expectation E M is w.r.t.
the randomness during the algorithm’s execution within game M. u
i ( π ) is the expected reward of the i th player when strategy π are taken for each player.
The proof of this lemma is also in the next subsection.
In the proof, we ﬁrst construct a class of one-step games which reward is Bernoulli ( 12 ) or Bernoulli ( 12 + ε ) depending on the taken joint-action.
The proportion of joint-actions with reward Bernoulli ( 12 + ε ) is relatively small.
Most importantly, every pure-strategy ε -approximate Nash equilibrium has reward Bernoulli ( 12 + ε ) .
So in order to ﬁnd an ε -approximate pure-strategy Nash equilibrium, we must explore sufﬁcient joint-actions.
The number of the joint-actions with reward Bernoulli ( 12 + ε ) can be bounded by the covering number of [2 k ] m under Hamming distance.
Then we use KL divergence decomposition (Lemma E.5) to argue rigorously that we need to explore sufﬁcient joint-actions to get an ε -approximate pure-strategy Nash equilibrium.
The rest of this section is organized as follows: We ﬁrst prove Lemma E.2 in Section E.1, and then prove the main Theorem E.1 in Section E.2.
Discussions of Theorem E.1 There’s (cid:80) mi =1 A i dependency 4 in the lower bound of sample complexity for ﬁnding a pure-strategy ε -approximate Nash equilibrium in MCGs.
This bound is novel and improves the existing result.
The existing proof in sample complexity’s lower bound of Markov games (Bai & Jin (2020)) relies on an reduction from Markov games to single-agent MDPs, so the existing lower bound’s dependency on A
i ( i = 1 , . . .
, m ) is max i ∈ [ m ] A i .
Here, we don’t include S factor in our lower bound.
The difﬁculty is that the NE-gap only depends on the player with the most suboptimality.
For a single-agent MDP, if the player can change the policy at each state to improve the expected cumulative reward by ε , then the player can change policy at all state to improve the expected cumulative reward to the utmost extent.
In general-sum Markov games, at different state, maybe different players can change the policy for this state to improve his expected cumulative reward by ε .
However, the deﬁnition NE-gap only allows one player to change the policy.
This difference in nature makes S and (cid:80) mi =1 A i incompatible in the lower bound.
If we consider another notion of suboptimality, i.e., changing maximum to summation:
[Equation]
This deﬁnition of NE-gap is different from the previous deﬁnition.
With NE-gap (cid:48) , if each player can change his policy to improve his expected cumulative reward by ε , the NE-gap (cid:48) would be at least mε .
Then we can similarly deﬁne ε -approximate Nash equilibrium as the policy π such that
NE-gap (cid:48)
( π ) ≤ ε .
We simply point out that with this new deﬁnition of NE-gap (cid:48) and ε -approximate
Nash equilibrium, mimicking the proof of Theorem 2 in Dann & Brunskill (2015), we can prove the sample complexity’s lower bound for learning a pure-strategy
ε -approximate Nash equilibrium in
Markov (cooperative) games is Ω( H 2 S (cid:80) mi =1 A i /ε 2 ) .
E. P ROOF OF L EMMA E.
For convenience, we call the joint-action (in one-step game) that is a Nash equilibrium a Nash strategy.
We begin with a special case of Lemma E.2, i.e. the case when A i = 2 for all i ∈ [ m ] .
Lemma E.3.
Suppose A i = 2 , i = 1 , . . .
, m and m ≥ 4 .
Then, there exists an absolute constant csuch that for any ε ≤ 0 .
4 and any algorithm that outputs a pure strategy (cid:98)
π = ( (cid:98) π 1 , . .
, (cid:98) π m ) , if the number of samples
[Equation]
then there exists a one step game M with stochastic reward on which the algorithm suffers from ε/ 4 -suboptimality, i.e.,
[Equation]
where the expectation E M is w.r.t.
the randomness during the algorithm’s execution within game M. u
i ( π ) is the expected reward of the i th player when strategy π are taken for each player.
The proof of this lemma further relies on the following lemma.
Lemma E.4.
There exists a one-step game for m players where each player has two actions.
The deterministic reward is 0 or 1 and the number of joint actions that have 1 is at most 2 m +1 m .
Moreover, the only pure-strategy Nash equilibria are these joint actions which have reward 1 .
Proof of Lemma E.
We use r ( a ) to denote the reward of (joint) actions a ∈ { 1 , 2 } m and deﬁne hamming distance d ( a , a (cid:48) )
=
# { i : a i (cid:54) = a (cid:48) i } .
To ensure that pure-strategy Nash equilibria must have reward 1, we only need to ensure that for a a ∈ { 1 , 2 } m , there exists one a (cid:48) ∈ { 1 , 2 } m such that
[Equation]
In other words, the set { a : r ( a ) = 1 } is a 1-net of { 1 , 2 } m under the distance d ( · , · ) .
By the deﬁnition of covering number, we only need to prove
[Equation]
Deﬁne K ( m, 1) =
N ( { 1 , 2 } m , d, 1) .
By hamming code (Hamming (1950)), we know that for any integer k ≥ 1 ,
[Equation]
Moreover, we also have K ( n, 1) ≤ 2 K ( n − 1 , 1) by adding 0 and 1 behind the 1-net of { 0 , 1 } n − 1 .
Taking largest k such that 2 k − 1 ≤ n and iterating this construction on the Hamming code we get
K ( m, 1) ≤ 2 m − [log 2 ( m +1)] ≤ 2 m +1 m .
This ends the proof.
The next lemma is KL divergence decomposition (Lemma 15.1 of (Lattimore & Szepesvári, 2020)]), we restate it in one-step games.
Lemma E.5 (KL divergence decomposition in one-step games) .
For any one-step games with stochastic reward and any algorithm.
Let X
=
( a 1 , r 1 , a 2 , r 2 , . .
. , a n , r n ) , where a k is the action (adaptively) chosen by the algorithm at the k th round and r k is the reward received at the k th round after a k is taken.
P and Q are two probability measure for the stochastic reward.
Let N ( a ) be the total number of actions a in the ﬁrst n rounds.
Then
[Equation]
Then we return to the proof of Lemma E.3.
Suppose a game M satisﬁes the condition in Lemma E.4, by permuting the actions of M , we get 2 m games.
They all satisfy the condition in Lemma E.4.
Suppose the reward of the i -th game is r ( i ) ( · ) ( i = 1 , · · · , 2 m ) .
We consider the following family of one-step games with stochastic reward:
Let a =
( a 1 , a 2 , . . . , a m ) .
[Equation]
where in one-step game M ( i ) , the reward is sampled from Bernoulli ( M ( i ) ( a )) if the joint action is a = ( a 1 , a 2 , . .
. , a m ) .
Moreover, we deﬁne M (0) as a game whose reward is sampled from Bernoulli ( 12 ) independent of the action.
We further let ν denote the uniform distribution on { 1 , 2 , . . .
, 2 m } .
Proof of Lemma E.
Fix a one-step game M ( i ) ∈ M ( ε ) , by Lemma E.4, it’s clear that purestrategy Nash equilibria form a set D ( i ) :
=
a ∈ { a : r ( i ) ( a ) = 1 } .
We have # D ( i ) ≤ 2 m +1 /m .
For any online ﬁnetuning algorithm A that outputs a pure strategy (cid:98) π .
Suppose (cid:98) π takes joint-action (cid:98) a = ( (cid:98) a 1 , (cid:98) a 2 , · · · , (cid:98) a m ) .
From the structure of the M ( ε ) , we know
[Equation]
where P i is w.r.t.
the randomness of the algorithm and game M ( i ) .
So we only need to use n , the number of samples to give a lower bound of P i ( (cid:98) a (cid:54)∈ D ( i ) ) .
Since X = { a 1 , r 1 , · · · , a n , r n } is a sufﬁcient statistics for the posterior distribution D ( i ) , we have
[Equation]
Above, (i) follows from the deﬁnition of total variation distance; (ii) uses the fact that under P 0 , we can’t get any information about a (cid:63) , so E i ∼ ν P
( g ( X ) (cid:54)∈ D ( i ) )
= 1 − | D (1) | 2 m ≥ 12 ; (iii) uses Pinsker’s inequality; (iv) uses KL divergence decomposition (Lemma E.5).
For a ∈ D ( i ) , we have
[Equation]
ε,
[Equation] + [Equation] (cid:19)(cid:19) =2 log 1 − 4 ε≤ 4 ε 2 , if ε ∈ (0 , 0 . 4] .
For a (cid:54)∈ D ( i ) , KL( P 0 ( ·| a ) (cid:107) P i ( ·| a ))
= 0 .
So
[Equation]
[Equation]
Above, ( i ) uses Jensen’s inequality, ( ii ) uses the fact that E P
[ N ( a )] is independent of i which gives
[Equation]
where (cid:80) i : a ∈ D ( i ) 1 = | D (1) | is because of the permutation.
Finally, we choose c 0 = 132 , if n ≤ 132 2 mε 2 , then
[Equation]
So there’s a game instance M ( i ) on which the algorithm suffer from ε/ 4 sub-optimality.
The difference between Lemma E.3 and E.2 is the size of action space.
To prove the A i = 2 k case, we need to generalized E.4 to the case each player have 2 k actions.
Lemma E.6.
For all positive integers m and k .
There exists a one-step game for m players where each player has 2 k actions.
The deterministic reward is 0 or 1 and the number of joint actions that have 1 is at most 2 · (2 k ) m km .
Moreover, the only pure-strategy Nash equilibria are these joint actions which has reward 1 .
Proof of Lemma E.
We would prove this lemma by induction.
Without loss of generality, we suppose the action for each player is 1 , 2 , . . .
, 2 k .
First, we deﬁne the hamming distance between two vectors.
[Equation]
The joint action space is [2 k ] m = { 1 , 2 , . . .
, 2 k } m .
Suppose we have an 1 -net of [2 k ] m , D , which means for every y ∈ [2 k ] m , we can ﬁnd a x ∈ D , s.t.
d ( x , y ) ≤ 1 .
If the reward of a game satisfy: r ( a ) = 1 for all a ∈ D, and r ( a ) = 0 for all a (cid:54)∈ D.
Then for every pure strategy a which has reward 0 , we can ﬁnd another pure strategy a (cid:48)
s.t.
r ( a (cid:48) )
=
and d ( a , a (cid:48) )
= 1 .
This means that one player can change to obtain higher reward.
So a is not a pure-strategy Nash equilibrium.
As a result, we can construct a game based on the 1 -net.
The only thing left to be veriﬁed is that the number of joint actions that have reward 1 is at most 2 · (2 k ) m km .
Note that the number of joint actions that have reward 1 is actually | D | , so we need to prove
[Equation]
where N ( · , d, ε ) denotes the covering number.
For k = 1 , from the proof of Lemma E.4, (17) is true.
For k > 1 , we ﬁrst decompose [2 k ] m into k m smaller blocks, i.e.
[Equation]
where { l 1 , l 2 , . . .
, l m } ∈ [ k ] m .
Among these k m blocks, we choose the blocks which satisfy k | l 1 + l 2 + · · · + l m .
Apparently, there are k m − 1 blocks satisfying m | l 1 + l 2 + · · · + l m .
Because (17) is true for k = 1 , we can pick a 1 -net of Ω(1 , 1 , . .
. , 1) with at most 2 · 2 m m elements.
By a translation, (all elements add a constant vector), we can pick a 1 -net of each block with at most 2 · 2 m m elements.
Totally there are at most 2 · 2 m k m − 1 m = 2 · (2 k ) m km elements.
These elements form a set P .
We would prove P is a 1 -net of [2 k ] m .
In fact, for any x ∈ [2 k ] m , suppose x is in Ω( l 1 , l 2 , . . .
, l m ) .
After translating this block to Ω(1 , 1 , . . . , 1) , x is coincided with x 0 .
If x 0 ∈ P , change l 1 to l (cid:48) 1 such that k | l (cid:48) 1 + l 2 + · · · + l m .
Suppose x (cid:48) is the vector in block Ω( l (cid:48) 1 , l 2 , . . .
, l m ) that corresponds to x 0 in Ω(1 , 1 , . .
This gives x (cid:48) ∈ P .
Moreover, d ( x , x (cid:48) ) ≤ 1 because the translation from Ω( l 1 , l 2 , . . .
, l m ) to Ω( l (cid:48) 1 , l 2 , . . .
, l m ) only changes the ﬁrst component.
If x 0 (cid:54)∈ P , we can ﬁnd x (cid:48)
in P ∩ Ω(1 , 1 , . . .
, 1) such that d ( x 0 , x (cid:48) 0 )
= 1 because P | Ω(1 , 1 ,..., 1) is a 1 -net.
Suppose x 0 and x (cid:48)
differs at the j -th component.
We can change l j to l (cid:48)
j s.t.
k | l 1 + l 2 + · · · + l j − 1 + l (cid:48)
j + l j +1 + · · · + l m .
Suppose x (cid:48) 0 corresponds to x (cid:48) in Ω( l 1 , . . .
, l m ) and x (cid:48)(cid:48) in Ω( l 1 , . .
, l j − 1 , l (cid:48)
j , l j +1 , . . .
, l m ) .
By the deﬁnition of P , we know x (cid:48)(cid:48) ∈ P .
Moreover, x and x (cid:48) only differ at the j -th component because of translation and the fact that x 0 and x (cid:48) 0 only differ at the j -th component.
x (cid:48) and x (cid:48)(cid:48) also only differ at the j -th component because of translation.
So we have x and x (cid:48)(cid:48) may only differ at the j -th component, i.e. d ( x , x (cid:48)(cid:48) ) ≤ 1 .
Recall that x (cid:48)(cid:48) ∈ P .
To conclude, we can always ﬁnd another vector y ∈ P such that d ( x , y ) ≤ 1 , this means P is a 1 -net of [2 k ] m .
So (17) is proved.
Using this fundamental lemma and suppose a game M satisﬁes the condition in Lemma E.6, by permuting the actions of M , we get (2 k ) m games. They all satisfy the condition in Lemma E.6. Suppose the reward of the i -th game is r ( i ) ( · ) ( i = 1 , · · · , (2 k ) m ) .
We consider the following family of one-step games with stochastic reward: Let a = ( a 1 , a 2 , . . . , a m ) . [Equation]
where in one-step game M ( i ) , the reward is sampled from Bernoulli ( M ( i ) ( a )) if the joint action is a = ( a 1 , a 2 , . . . , a m ) . Moreover, we deﬁne M (0) as a game whose reward is sampled from Bernoulli ( 12 ) independent of the action.
Then by the same argument in proof of Lemma E.3, we can easily prove Lemma E.2.
E. P ROOF OF T HEOREM E.
We are now ready to prove Theorem E.1 based on Lemma E.2.
Because S ≥ 3 , we construct a class of general-sum Markov games as follows (see ﬁgure 1): the set of all joint-actions A can be divided into two sets A + an A − . Transition Starting from s 1 , for a ∈ A + , P 1 ( s + | s 1 , a ) = 12 + ε 2( H − 1) and P 1 ( s − | s 1 , a ) = 12 − ε 2( H −
For a ∈ A
−
P s + | s a ) = [-
-]P 1 ( s −
|
s
a ) =
For all
h ≥
and
a ∈ A , P h ( s + | s + , a ) = 1 and P h ( s − | s − , a
=
{+Rewards For any a and i ∈ [ m ] , r 1 ,i ( s, a ) = 
+}
r h,i ( s + , a ) = 1 and r h,i ( s − , a ) = 0 , where h > 1 .
This also implies the Markov game is cooperative.
If we choose A + as the joint action with reward Bernoulli ( 12 + ε ) in M ( i ) deﬁned in (18).
By the construction, for any pure-strategy policy π , if the joint actions taken at step 1 is a (cid:54)∈ A + , we have
Require: Hyper-parameters { η t } 1 ≤ t ≤ T , { γ t } 1 ≤ t ≤ T .
Sequence (cid:8) α it (cid:9) 1 ≤ i ≤ t ≤ T .
1: Initialize : Accumulators t b (cid:48) ← 0 for all b (cid:48) ∈
[ A ] .
2: for t = 1 , 2 , . . .
, T do 3:
Compute weight u t ← α tt /α 1 t .
Compute action distributions for all sub-experts b (cid:48) ∈
[ A ] by FTRL:
q b (cid:48) ( a ) ∝ a exp (cid:32) − ( η t b (cid:48) /u t ) · t b (cid:48) (cid:88) τ = w τ ( b (cid:48) )
(cid:98) (cid:96) b (cid:48) τ
( a ) (cid:33) .
Compute p t ∈ ∆
[ A ] by solving the linear system p t ( · ) = (cid:80) Ab (cid:48)
=1 p t ( b (cid:48) )
q b (cid:48) ( · ) .
Sample sub-expert b ∼ p t .
Sample action a t ∼ q b from sub-expert b .
Play the action a t , and observe the (bandit-feedback) loss ˜ (cid:96) t ( a t ) .
Update accumulator for sub-expert b : t b ← t b + 1 .
Compute the loss estimate (cid:98) (cid:96) bt b (
· ) ∈ R
A ≥ 0 as (cid:98) (cid:96) bt b ( a ) ← ˜ (cid:96) t b ( a ) 1 { a = a t } q b ( a ) + γ t b .
Set w t b ( b ) ← u t .
NE-gap ( π ) ≥ ε .
The only useful information in each episode is the transition at step 1 .
Transition from s 1 to s + can be viewed as getting a 1 reward and transition from s 1 to s − can be viewed as getting a 0 reward.
So learning pure-strategy Nash equilibrium of this class of Markov games is equivalent to learning pure-strategy Nash equilibrium of a game for m players with stochastic reward.
As a result, by Lemma E.2, if the number of episodes
[Equation]
then there exists general-sum Markov cooperative game MG on which the algorithm suffers from ε/ 4 -suboptimality, i.e.
[Equation]
This ﬁnish the proof of Theorem E.1.
F A DVERSARIAL B ANDIT WITH L OW W EIGHTED S WAP R EGRET
In this section, we describe and analyze our main algorithm for adversarial bandits with low weighted swap regret.
Our Algorithm, Mixed-expert Follow-The-Regularized-Leader (FTRL) for weighted adversarial bandits, is described in Algorithm 8. Problem setting Throughout this section, we consider a standard (adversarial) bandit problem with action space is [ A ] = { 1 , . . . , A } for some A ≥ 1 .
We assume that the realized loss values ˜ (cid:96) t ∈ [0 , 1] A , and let (cid:96) t ( a ) := E t [˜ (cid:96) t ( a )] denote the expected loss conditioned on (as usual) all information before the sampling of the action a t in Line 6 & 7. The hyperparameters in Algorithmare chosen as [Equation]
where the log factor [Equation] F. M AIN RESULT FOR ADVERSARIAL BANDIT WITH LOW WEIGHTED SWAP REGRET
We ﬁrst deﬁne a strategy modiﬁcation. A strategy modiﬁcation is a function F : [ A ] → [ A ] which can also be applied to any action distribution µ ∈ ∆ A , such that F (cid:5) µ gives the swap distribution which takes action F ( a ) with probability µ ( a ) .
The swap regret (Blum & Mansour, 2007; Ito, 2020) measures the difference between the cumulative realized loss for the algorithm and that for swapped action sequences generated by an arbitrary strategy modiﬁcation F .
Here, we consider a weighted version of the swap regret with some non-negative weights (cid:8) α it (cid:9) 1 ≤ i ≤ t , deﬁned as
[Equation]
We will also consider a slightly modiﬁed version of the swap regret used in our analyses for learning CE, deﬁned as
[Equation]
where p t is t -th action distribution (from which the action a t is sampled from) played by the algorithm. We now state our main result of this section.
Lemma F.1 (Bound on weighted swap regret) .
If we execute Algorithm 8 for T rounds and the weights (cid:8) α it (cid:9) are chosen according to (2) , then with probability at least 1 − p/ 2 , we have the following bounds on the swap regret simultaneously for all t ∈ [ T ] : [Equation] max
[Equation] where C > 0 is some absolute constant. The rest of this section is devoted to proving Lemma F.1, organized as follows. We ﬁrst present some important properties of Algorithm 8 in Section F.2.
We then prove Lemma F.1 in Section F.3 using new auxiliary results on weighted adversarial bandits with predictable weights. Lastly, these auxiliary results are stated and proved in Appendix F.4. F. P ROPERTIES OF A LGORITHM
Solution of the linear system In Line 5, we need to compute p t ∈ ∆ [ A ] by solving the linear system p t ( · ) = (cid:80) Ab (cid:48)
=1 p t ( b (cid:48) )
q b (cid:48) ( · ) .
Such p t ( · ) can be interpreted as the stationary distribution of a Markov chain over [ A ] with transition probability P ( b | a ) =
q a ( b ) .
This guarantees the existence of p t ∈ ∆ [ A ] such that p t ( · ) = (cid:80) Ab (cid:48)
=1 p t ( b (cid:48) )
q b (cid:48) ( · ) .
Moreover, p t ( · ) can be computed efﬁciently (see e.g. Cohen et al.
(2017b)).
FTRL update for each sub-expert Here we show that, the updates for any particular sub-expert b ∈ [ A ] in Algorithm 8 is exactly equivalent to an FTRL update (with loss sequence being the ones only over the episodes in which b is sampled) which we summarize in Algorithm 9. This is important as our proof relies on reducing the weighted swap regret to a linear combination of weighted regrets for each sub-expert b ∈ [ A ] .
To see this, ﬁx any b ∈ [ A ] . When b is sampled in Line 6 at episode t , we have accumulator t b and weight w t b ( b ) =
u t = α tt /α 1 t at the end of episode t .
Action a t is chosen from distribution
[Equation] after w t b ( b ) =
u t = α tt /α 1 t is observed.
The loss estimate (cid:98) (cid:96) t b is
[Equation]
So from the sub-expert b ’s perspective, she is performing FTRL (follow the regularized leader) with changing step size and random weight (We summarize this in Algorithm 9).
Suppose the sub-expert b is chosen at episode t = k 1 , k 2 , . .
k t b , then the weighted regret for sub-expert b becomes
[Equation]
Bounded weight w i ( b ) Recall (cid:8) α it (cid:9) is chosen as in (2).
We have the following bound:
[Equation]
So for any t ≤ T , we have
[Equation]
The weight w n b ( b ) =
α tt /α 1 t have a (non-random) upper bound W .
F. P ROOF OF L EMMA F.
We use b i to denote the sampled sub-expert b at the i -th episode.
Deﬁne G i as the σ -algebra generated by all the random variables observed up to the end of the i -th episode.
First observe that for R swap ( t ) we have the bound
R swap ( t ) = max F :[ A ] → [ A ] t (cid:88) i = α it [ (cid:96) i ( a i ) − (cid:96) i ( F ( a i ))] ≤ max F :[ A ] → [ A ] t (cid:88) i = α it [ (cid:96) i ( a i ) − (cid:96) i ( F ( b i ))] (cid:124) (cid:123)(cid:122) (cid:125) I + max F :[ A ] → [ A ] t (cid:88) i = α it [ (cid:96) i ( F ( b i )) − (cid:96) i ( F ( a i ))] (cid:124) (cid:123)(cid:122) (cid:125) II , also, for (cid:101) R swap ( t ) we have the bound (cid:101) R swap ( t ) = max F :[ A ] → [ A ] t (cid:88) i = α it [ (cid:96) i ( a i ) − (cid:10) F (cid:5) p i , (cid:96) i (cid:11) ] ≤ max F :[ A ] → [ A ] t (cid:88) i = α it [ (cid:96) i ( a i ) − (cid:96) i ( F ( b i ))] (cid:124) (cid:123)(cid:122) (cid:125) I + max F :[ A ] → [ A ] t (cid:88) i = α it [ (cid:96) i ( F ( b i )) − (cid:10) F (cid:5) p i , (cid:96) i (cid:11) ] (cid:124) (cid:123)(cid:122) (cid:125) (cid:101) II ,
Term II and (cid:101) II can be bounded by concentration in a similar fashion; here we ﬁrst focus on term II .
Observe that at the i -th episode, as p i obtained in Line 5 solves the equation
[Equation]
we have b i ∼ p
i ( · ) and a i ∼ q b i ( · ) has the same marginal distribution conditioned on G i − 1 , where
G i − 1 denote the σ -algebra generated by (cid:96) i and all the random variables in the ﬁrst i − 1 episodes.
Therefore, ﬁxing any strategy modiﬁcation F : [ A ] → [ A ] , we have (cid:8) α i t
[ (cid:96) i ( F ( b i )) − (cid:96) i
( F ( a i ))] (cid:9) 1 ≤ i ≤ t is a bounded martingale difference sequence w.r.t.
the ﬁltration {G i } , so by Azuma-Hoeffding inequality and (cid:80) ti =1 ( α jt ) 2 ≤ 2 H/t ,
[Equation]
As there is at most A A strategy modiﬁcations, we can substitute p with p/ (4 A A T ) , and take a union bound to get
[Equation]
simultaneously for all t ∈
[ T ] with probability at least 1 − p/ 4 .
We also note that, by a similar argument (as b i is also distributed according to p i conditioned on the past), we have that
[Equation]
Therefore for bounding both R swap ( t ) and (cid:101) R swap ( t ) , it sufﬁces to bound term I .
We next bound term I .
Deﬁne U
b := (cid:8) i ∈
[ t ] : b i = b (cid:9) and let n tb be the value of t b at the end of the t -th episode, i.e. n tb = # (cid:8) i : b i = b (cid:9) .
We also suppose the sub-expert b was chosen at episode t = k b 1 , k b 2 , . . .
, k bn tb up to episode t .
Then we have
[Equation]
Here the last equation is because our choice of w τ ( b ) =
α k bτ k bτ /α 1 k bτ = α k bτ t /α 1 t which is a simple corollary from the deﬁnition of α it in Eq.
Then because n tb (cid:88)
[Equation]
where R t ( b ) deﬁned in (22) is the weighted regret R t ( b ) for sub-expert b , we can use our result on weighted adversarial bandits with predictable weights (Lemma F.2) to bound this term (The upper bound W of the weight w τ ( b ) can be taken as ( H + T ) 2 H by the calculation of (23).
Moreover, w τ ( b ) =
α k b τ t /α 1 t and { α jt } t j =1 is increasing, so { w τ ( b ) } τ ≥ 1 is non-decreasing.). Recall that our choice of log term is ι =
[Equation]
where p (cid:48) ≤ p/ (4 A ) .
Thus by Lemma F.2, with probability at least 1 − p/ (4 A ) ,
[Equation]
Taking a union bound
[Equation]
we get
[Equation]
with probability at least 1 − p/ 4 .
On this event
[Equation]
we have
[Equation]
[Equation]
Here, (i) uses (26), (ii) uses the α t (cid:48) t is increasing w.r.t.
t (cid:48) and (iii) uses max 0 ≤ t (cid:48) ≤ t α t (cid:48)
t ≤ 2 H/t .
Finally, because (cid:80) b ∈A n tb = t , by the concavity of x (cid:55)→ √ x , we have with probability at least 1 − p/ 4 ,
I =
[Equation]
Combining this with (24) and (25), we ﬁnish the proof.
F. A UXILIARY LEMMAS FOR WEIGHTED ADVERSARIAL BANDIT WITH PREDICTABLE WEIGHTS
In this subsection, we consider the Follow the Regularized Leader (FTRL) algorithm (Lattimore &
Szepesvári, 2020) with
changing step size, 2.
weighted regret with F t − 1 -measurable weights and loss distributions, 3.
high probability regret bound.
We present these results because the predictable weights we would use are potentially unbounded from above; if weights are predictable and also bounded, then there may be an easier analysis.
Interaction protocol
We ﬁrst describe the interaction protocol between the environment and the player for this problem.
At each episode t , the environment adversarially choose a weight w t which takes values in R > 0 , and distributions ( L at ) a ∈
[ A ] , where each L at is a distribution that takes values in P ([0 , 1]) (the space of distributions supported on [0 , 1] ).
Then the player receives the weight w t , chooses an action a t , and observes a loss ˜ (cid:96) t ( a t ) ∼ L a t ,t .
The action chosen a t can be based on all the information in D t ≡ { ( a i , w i , ˜ (cid:96) i ( a i ))
i ≤ t − 1 , w t } .
Then, the environment can observe the player’s action a t and the incurred loss realization ˜ (cid:96) t ( a t ) , and use these information and some external randomness z t to choose the weight and distributions ( w t +1 , ( L a,t +1 ) a ∈ [ A ] ) of the next episode.
We will consider the following variant of FTRL algorithm for the player.
We denote (cid:96) t ≡ ( E ˜ (cid:96) ∼L at [˜ (cid:96) ]) a ∈ [ A ] ∈ [0 , 1]
A which is a random vector but is σ (( L at ) a ∈ [ A ] ) - measurable.
For some θ ∗ ∈ R A , the regret of the player up to episode t is deﬁned as
R t ≡ t (cid:88) i = [Equation] w
[Equation]
( (cid:96)
[Equation] ( a [Equation] ) − (cid:104) θ ∗ , (cid:96) i (cid:105) ) .
Let ( z t ) t ≥ 1 be a sequence of external random variables which are identically and independently distributed as Unif ([0 , 1]) and are independent of all other random variables.
We deﬁne F t = σ ( { a i , ˜ (cid:96) i , z i } i ≤ t ) to be the sigma algebra generated by the random chosen action a i , the random loss ˜ (cid:96) i ( a i ) , and the external random variables z i by episode t .
We assume that the random weights and the random loss distributions ( w t , ( L at ) a ∈ [ A ] ) t ≥ 1 are a ( F t ) t ≥ 1 - predictable sequence , in the sense that ( w t , ( L at ) a ∈ [ A ] ) is F t − 1 -measurable.
Then F t contains all information (random chosen action, random observed loss, random weight, and random loss distributions) before action a t +1 is taken.
We assume the predictable sequence ( w t ) 1 ≤ t ≤ T have a global (non-random) upper bound W .
Then we deﬁne the log term ι = log( AT (cid:100) log 2 W (cid:101) /p ) .
We set
[Equation]
ι
At. F.4. R EGRET BOUND
In the following, we consider to give a high probability weighted regret bound for Algorithm 9.
Lemma F.2.
Let ( w t , ( L at ) a ∈ [ A ] ) t ≥ 1 be any F t -predictable sequence satisfying 1 ≤ min i
≤ t w i ≤ max i ≤ t w i ≤ W for some constant (non-random) W > 0 almost surely.
Moreover, suppose w i is non-decreasing.
Then, following Algorithm 9, with probability at least 1 − 4 p , for any θ ∗ ∈ ∆ A and t ≤ T we have
[Equation]
where ι = log( AT (cid:100) log 2 W (cid:101) /p ) .
Proof.
This lemma follows from the bound in Lemma F.3 and a concentration step that we establish below.
Deﬁne M = (cid:100) log 2 W (cid:101) , and w i ( k )
=
w i 1 (cid:8) w i ≤ 2 k (cid:9) .
Then w i ( k ) is also F i − 1 measurable.
We
Consider sequence (cid:8) w i ( k )( (cid:96) i ( a i ) − (cid:104) θ i , (cid:96) i (cid:105) )
(cid:9) i ≥ 1 .
Since w i is F i − 1 -measurable, E [ w i ( (cid:96) i ( a i ) − (cid:104) θ i , (cid:96) i (cid:105) )
|F i − 1 ] = 0 , which means (cid:8) w i ( k )( (cid:96) i ( a i ) − (cid:104)
θ i , (cid:96)
i (cid:105) )
(cid:9) i ≥ 1 is a martingale difference sequence w.r.t.
ﬁltration ( F i ) .
So by Azuma-Hoeffding inequality,
[Equation]
Taking a union bound, we have
[Equation]
Denote k (cid:48) =
(cid:100) log 2 max i ≤ t w i (cid:101) , then we have (cid:110) ∀ k ∈ [ M ] , t (cid:88) [Equation]
=
[Equation]
⊆ (cid:110) t (cid:88) i =
[Equation]
i
[Equation] [Equation] (cid:96) [Equation]
[Equation]
− (cid:104)
[Equation]
i
[Equation]
i (cid:105) )
[Equation]
max
[Equation]
Then probability bound gives
[Equation]
Taking a union bound, we get
[Equation] with probability at least 1 − p .
Summing the above and the regret bound shown in Lemma F.3, we ﬁnish the proof.
Lemma F.3.
Let ( w t , ( L at ) a ∈ [ A ] ) t ≥ 1 be any F t -predictable sequence satisfying 1 ≤ min i
≤ t w i ≤ max i ≤ t w i ≤ W for some constant (non-random) W > 0 almost surely.
Moreover, suppose w i is non-decreasing.
Then, following Algorithm 9, with probability at least 1 − 3 p , for any θ ∗ ∈ ∆ A and t ≤ T we have
[Equation]
where ι = log( AT (cid:100) log 2 W (cid:101) /p ) .
Proof.
The regret R t ( θ ∗ ) can be decomposed into three terms
[Equation]
(cid:125)
( A ) + t (cid:88) i = w i (cid:68) θ i , (cid:96) i − (cid:98) (cid:96) i (cid:69) (cid:124) (cid:123)(cid:122) (cid:125) ( B ) + t (cid:88) i = w i (cid:68) θ ∗ , (cid:98) (cid:96) i − (cid:96) i (cid:69) (cid:124) (cid:123)(cid:122) (cid:125) ( C ) and we bound ( A ) in Lemma F.5, ( B ) in Lemma F.6 and ( C ) in Lemma F.7.
Setting η t = γ t = (cid:112) ιAt , the conditions in Lemma F.5 and Lemma F.7 are satisﬁed.
Putting them together and take union bound, we have with probability 1 − 3 p
[Equation]
This proves the lemma.
The rest of this section is devoted to the proofs of the Lemmas used in the proofs of Lemma F.3.
We begin the following useful lemma adapted from Lemma 1 in Neu (2015), which is crucial in constructing high probability guarantees.
Lemma F.4.
For any predictable sequence of coefﬁcients c 1 , c 2 , . . .
c t s.t.
c i ∈ [0 , 2 γ
i ]
A w.r.t.
( F i )
i ≥ 1 and ﬁxing t , we have with probability at least 1 − p/
( AT ) ,
[Equation]
Proof.
Deﬁne M = (cid:100) log 2 W (cid:101) , and w i ( k )
=
w i 1 (cid:8) w i ≤ 2 k (cid:9) .
By deﬁnition,
[Equation]
where ( i ) is because z
[Equation]
+ [Equation] ≤
[Equation]
for all z ≥ [Equation]
Deﬁning the sum
[Equation]
Then S i is F i − 1 -measurable since w i , c i , (cid:96) i are F i − 1 -measurable.
Using E i [ · ] to denote the conditional expectation E [ ·|F i ] , we have
[Equation]
where ( i ) is because z 1 log (1 + z 2 ) ≤ log (1 + z 1 z 2 ) for any 0 ≤ z 1 ≤ 1 and z 2 ≥ − 1 .
Here we are using the condition
c i ( a ) ≤ 2 γ i to guarantee the condition is satisﬁed.
Equipped with the above bound [Equation] we can now prove the concentration result [Equation]
≤
[Equation]
=
[Equation]
E
[Equation]
≤
[Equation]
i = (cid:16)(cid:98)
[Equation]
(cid:17)(cid:35)(cid:35) ≤
[Equation]
≤ p
ATM .
So we have
P (cid:16) t [Equation] = [Equation]
≤
[Equation]
(cid:17) ≥ 1 − p/ ( ATM ) .
Taking a union bound,
[Equation] [Equation]
AT ) . Denote [Equation]
i [Equation] and note that
[Equation]
Therefore, we have
[Equation]
This proves the lemma.
Using Lemma F.4, we can bound the ( A )( B )( C ) separately as below.
Lemma F.5.
If η i ≤ 2 γ i for all i ≤ t and { η i /w i } i ≥ 1 is non-increasing, with probability 1 − p , for any t ∈ [ T ] and θ ∗ ∈ ∆ A ,
[Equation]
Proof.
We use the standard analysis of FTRL with changing step size, see for example Exercise 28.12(b) in Lattimore & Szepesvári (2020).
Notice the essential step size is η t /w t which is nonincreasing and the essential loss vector is w t (cid:98) (cid:96) t , we have [Equation] We claim that [Equation]
[Equation]
In fact, applying Theorem 26.13 in Lattimore & Szepesvári (2020) gives that
[Equation]
, for some u i ∈ [0 , 1] .
{+Note that our (cid:98) (cid:96) i only have at most one non-zero entry, i.e. (cid:98) (cid:96) i ( a ) > 
+}
if and only if a = a i .
If θ i ( a i ) ≥ θ i +1 ( a i ) , we have
[Equation]
{+θ i +1 , (cid:98) (cid:96) 
+}
i (cid:69) =
{+η i 2 [ u i θ i ( a i ) + (1 − u i ) θ i +
+}
( a i )] (cid:98) (cid:96)
i ( a i )≤ η i 2 θ
{+i ( a i ) (cid:98) (cid:96) i ( a i ) 2 = η i 2 (cid:68) θ i , (cid:98) (cid:96) 
+}
i (cid:69) .
Otherwise, θ i ( a i ) <
θ i +1 ( a i ) , so we have (cid:68) θ i − θ
i +1 , (cid:98) (cid:96) i (cid:69) =
{+[ θ i ( a i ) − θ i +
+}
( a i )] (cid:98) (cid:96)
i ( a i ) ≤ 0 and − KL ( θ i +1 (cid:107) θ i ) /η
i ≤ 0 .
In both cases, (29) is correct.
As a result, t (cid:88) i =
w i (cid:68) θ i − θ ∗ , (cid:98) (cid:96) i (cid:69) ≤ w t log A η t + t (cid:88) i =
η i w i (cid:68) θ i , (cid:98) l 2 i (cid:69) ≤ w t log A η t + t (cid:88) i = (cid:88) a ∈A η i w i (cid:98) (cid:96) i ( a ) ( i ) ≤ w t log A η t + t (cid:88) i = (cid:88) a ∈A η i w i (cid:96) i ( a ) + max i ≤ t w i ι
[Equation]
where ( i ) is by using Lemma F.4 with c
i ( a ) =
η i .
The any-time guarantee is justiﬁed by taking union bound.
Lemma F.6.
With probability 1 − p , for any t ∈ [ T ] ,
[Equation]
Proof.
We further decompose the left side into
[Equation]
The ﬁrst term is bounded by
[Equation]
≤ A t (cid:88) i = γ i w i . To bound the second term, we use similar argument in the proof of Lemma F.4 , we deﬁne w = max i ≤ t w i , M = (cid:100) log 2 W (cid:101) .
and w i ( k ) =
w i 1 (cid:8) w i ≤ 2 k (cid:9) , notice (cid:68) θ i , (cid:98) (cid:96) i (cid:69) ≤ (cid:88) a ∈A θ i ( a ) 1 { a t = a } θ i ( a ) + γ i ≤ (cid:88)
a ∈A 1 { a i = a } = 1 , thus { w i ( k ) (cid:68) θ i , E i − 1 [ (cid:98) (cid:96)
i ] − (cid:98) (cid:96) i (cid:69) } ti =1 is a bounded martingale difference sequence w.r.t.
the ﬁltration {F i } ti =1 .
By Azuma-Hoeffding inequality, t (cid:88) i = w
[Equation]
k
[Equation]
, E
[Equation]
− (cid:98) (cid:96)
[Equation]
{+(cid:69) ≤ (cid:118)(cid:117)(cid:117)(cid:116) 
+}
ι t (cid:88) i = w
[Equation]
( k )
[Equation]
≤ (cid:113)
[Equation]
[Equation]
)with probability at least 1 − p/TM .
Taking a union bound, we get t (cid:88) i = w i ( k ) (cid:68) θ i , E i − 1 [ (cid:98) (cid:96) i ] − (cid:98) (cid:96) i (cid:69) ≤ √ 2 ι · t 2 2 k , for all ( t, k ) ∈
[ T ] × [ M ] with probability at least 1 − p .
On this event, choosing k (cid:48)
= (cid:100) log 2 w (cid:101) , we have t
[Equation]
This ends the proof.
Lemma F.7.
With probability 1 − p , for any t ∈ [ T ] and any θ ∗ ∈
∆
A , if γ i is non-increasing in i , t (cid:88)
[Equation]
Proof.
Deﬁne a basis { e j } Aj =1 of R A by [Equation]
Then for all the j ∈ [ A ] , apply Lemma F.4 with c i =
γ t e j .
Since now c i ( a ) ≤ γ t ≤ γ i , the condition in Lemma F.4 is satisﬁed.
As a result, for any t ∈ [ T ] and j ∈
[ A ] , we have with probability at least 1 − p/ ( TA ) that
[Equation]
Taking a union bound, we have with probability at least 1 − p ,
[Equation]
Since any θ ∗ is a convex combination of { e j } A j =1 , on this event, we also have [Equation] for all [Equation]
∈
[Equation]
This ﬁnishes the proof.