V-L EARNING —A S IMPLE , E FFICIENT , D ECENTRALIZED A LGORITHM FOR M ULTIAGENT RL
A BSTRACT
A major challenge of multiagent reinforcement learning (MARL) is the curse of multiagents , where the size of the joint action space scales exponentially with the number of agents.
This remains to be a bottleneck for designing efﬁcient MARL algorithms even in a basic scenario with ﬁnitely many states and actions.
This paper resolves this challenge for the model of episodic Markov games.
We design a new class of fully decentralized algorithms—V-learning, which provably learns Nash equilibria (in the two-player zero-sum setting), correlated equilibria and coarse correlated equilibria (in the multiplayer general-sum setting) in a number of samples that only scales with max i ∈ [ m ] A i , where A i is the number of actions for the i th player.
This is in sharp contrast to the size of the joint action space which is (cid:81) mi
=1 A i .
V-learning (in its basic form) is a new class of single-agent RL algorithms that convert any adversarial bandit algorithm with suitable regret guarantees into an RL algorithm.
Similar to the classical Q-learning algorithm, it performs incremental updates to the value functions.
Different from Q-learning, it only maintains the estimates of V-values instead of Q-values.
This key difference allows V-learning to achieve the claimed guarantees in the MARL setting by simply letting all agents run V-learning independently.
I NTRODUCTION
A wide range of modern artiﬁcial intelligence challenges can be cast as multi-agent reinforcement learning (MARL) problems, in which agents learn to make a sequence of decisions in the presence of other agents whose decisions will inﬂuence the outcome and can adapt to the strategies of the agents.
Modern MARL systems have achieved signiﬁcant success recently on a rich set of traditionally challenging tasks, including the game of GO (Silver et al., 2016; 2017), Poker (Brown & Sandholm, 2019), real-time strategy games (Vinyals et al., 2019; OpenAI, 2018), decentralized controls or multiagent robotics systems (Brambilla et al., 2013), autonomous driving (Shalev-Shwartz et al., 2016), as well as complex social scenarios such as hide-and-seek (Baker et al., 2020).
While singleagent RL has been the focus of recent intense theoretical study, MARL has been comparatively underexplored, which leaves several fundamental questions open even in the basic model of Markov games Shapley (1953) with ﬁnitely many states and actions.
One such unique challenge of MARL is the curse of multiagents —let A i be the number of actions for the i th player, then the number of possible joint actions (as well as the number of parameters to specify a Markov game) scales with (cid:81) mi =1 A i , which grows exponentially with the number of agents m .
This remains to be a bottleneck even for the best existing algorithms for learning Markov games.
In fact, a majority of these algorithms adapt the classical single-agent algorithms, such as value iteration or Q-learning, into the multiagent setting Bai et al.
; Liu et al.
(2021), whose sample complexity scales at least linearly with respect to (cid:81) mi =1 A i .
This is prohibitively large in practice even for fairly small multiagent applications, say only ten agents are involved with ten actions available for each agent.
[Table]
Another challenge of the MARL is to design decentralized algorithms.
While a centralized algorithm requires the existence of a centralized controller which gathers all information and jointly optimizes the policies of all agents, a decentralized algorithm allows each agent to only observe her own actions and rewards while optimizing her own policy independently.
Decentralized algorithms are often preferred over centralized algorithms in practice since (1) decentralized algorithms are typically cleaner, easier to implement; (2) decentralized algorithms are more versatile as the individual learners are indifferent to the interaction and the number of other agents; and (3) they are also faster due to less communication required.
While several provable decentralized MARL algorithms have been developed (see, e.g., Zhang et al., 2018; Sayin et al., 2021; Daskalakis et al., 2021), they either have only asymptotic guarantees or work only under certain reachability assumptions (see
Section A).
The existing provably efﬁcient algorithms for general Markov games (without further assumptions) are exclusively centralized algorithms (Bai & Jin, 2020; Xie et al., 2020; Liu et al., 2021).
This motivates us to ask the following open question:
Can we design decentralized MARL algorithms that break the curse of
multiagents?
This paper addresses both challenges mentioned above, and provide the ﬁrst positive answer to this question in the basic setting of tabular episodic Markov games.
We propose a new class of singleagent RL algorithms—V-learning, which converts any adversarial bandit algorithm with suitable regret guarantees into a RL algorithm.
Similar to the classical Q-learning algorithm, V-learning also performs incremental updates to the values.
Different from Q-learning, V-learning only maintains the V-value functions instead of the Q-value functions.
We remark that the number of parameters of
Q-value functions in MARL is O ( S (cid:81) mi =1 A i ) , where S is the number of states, while the number of parameters of V-value functions is only O ( S ) .
This key difference allows V-learning to be readily extended to the MARL setting by simply letting all agents run V-learning independently, which gives a fully decentralized algorithm.
We consider the standard learning objectives in the game theory—Nash equilibrium (NE), correlated equilibrium (CE) and coarse correlated equilibrium (CCE).
Except for the task of ﬁnding a NE in multiplayer general-sum games which is PPAD-complete even for matrix games, we prove that Vlearning ﬁnds a NE for two-player zero-sum Markov games within ˜ O
( H 5 SA/(cid:15) 2 ) episodes, where
H is the length of each episode, A = max i ∈ [ m ]
A i is the maximum number of actions of all agents, and (cid:15) is the error tolerance for the objective.
We further prove that for multiplayer general-sum
Markov games, V-learning coupled with suitable adversarial bandit algorithms is capable of ﬁnding a CCE within ˜ O
( H 5 SA/(cid:15) 2 ) episodes, and a CE within ˜ O
( H 5 SA 2 /(cid:15) 2 ) episodes.
All sample complexities mentioned above do not grow with the number of the agents, which thus break the curse of multiagents (See Table 1 for a summary and Appendix A for a more comprehensive discussion on related literature.).
P RELIMINARIES
We consider the model of Markov Games (MG) Shapley (1953) (also known as stochastic games in the literature) in its most generic—multiplayer general-sum form.
Formally, we denote an tabular episodic MG with m players by a tuple
MG( H, S , {A i } mi =1 , P , { r i } mi =1 ) , where H , S denote the length of each episode and the state space with |S| = S .
A i denotes the action space for the i th player and |A
i | =
A i .
We let a :=
( a 1 , · · · , a m ) denote the (tuple of) joint actions by all m players, and A =
A 1 × . . .
× A m .
P =
{ P h } h ∈ [ H ] is a collection of transition matrices, so that P h ( ·| s, a ) gives the distribution of the next state if actions a are taken at state s at step h , and r i = { r i,h } h ∈
[ H ] is a collection of reward functions for the i th player, so that r i,h ( s, a ) ∈ [0 , 1] gives the deterministic reward received by the i th player if actions a are taken at state s at step h .
2 We remark that since the relation among the rewards of different agents can be arbitrary, this model of MGs incorporates both cooperation and competition.
In each episode, we start with a ﬁxed initial state s 1 .
At each step h ∈
[ H ] , each player i observes state s h ∈ S , picks action a
i,h ∈
A i simultaneously, observes the actions played by other players, 4 and receives her own reward r i,h ( s h , a h ) .
Then the environment transitions to the next state s h +1 ∼ P h ( ·| s h , a h ) .
The episode ends when s H +1 is reached.
Policy, value function A ( random ) policy π i of the i th player is a set of H maps π i :=
(cid:8) π i,h : Ω × ( S × A ) h − 1 × S → A i (cid:9) h ∈
[ H ] , where π i,h maps a random sample ω from a probability space Ω and a history of length h —say τ h :=
( s 1 , a 1 , · · · , s h ) , to an action in A i .
To execute policy π i , we ﬁrst draw a random sample ω at the beginning of the episode.
Then, at each step h , the i th player simply takes action π i,h ( ω, τ h ) .
We note here ω is shared among all steps
h ∈
[ H ] .
ω encodes both the correlation among steps and the individual randomness of each step.
We further say a policy π i is deterministic if π i,h ( ω, τ h )
=
π i,h ( τ h ) which is independent of the choice of ω .
An important subclass of policy is Markov policy , which can be deﬁned as π i :=
(cid:8) π i,h : Ω × S →
A i (cid:9) h ∈
[ H ] .
Instead of depending on the entire history, a Markov policy takes actions only based on the current state.
Furthermore, the randomness in each step of Markov policy is independent.
Therefore, when it is clear from the context, we write Markov policy as π i :=
(cid:8) π i,h : S → ∆ A i (cid:9) h ∈
[ H ] , where ∆ A i denotes the simplex over A i .
We also use notation
π i,h ( a | s ) to denote the probability of the i th agent taking action a at state s at step h .
A joint (potentially correlated) policy is a set of policies { π i } mi =1 , where the same random sample ω is shared among all agents, which we denote as π = π 1 (cid:12) π 2 (cid:12) . . .
(cid:12) π m .
We also denote π − i =
π 1 (cid:12) . . .
π i − 1 (cid:12) π i +1 (cid:12) . . .
(cid:12) π m to be the joint policy excluding the i th player.
A special case of joint policy is the product policy where the random sample has special form ω
= ( ω 1 , . .
ω m )
, and for any i ∈ [ m ] , π i only uses the randomness in ω i , which is independent of remaining { ω j } j (cid:54) =
i , which we denote as π = π 1 × π 2 × . . .
× π m .
We deﬁne the value function V πi, 1 ( s 1 ) as the expected cumulative reward that the i th player will receive if the game starts at initial state s 1 at the 1 st step and all players follow joint policy π :
[Equation]
where the expectation is taken over the randomness in transition and the random sample ω in policy π .
Best response and strategy modiﬁcation For any strategy π − i , the best response of the i th player is deﬁned as a policy of the i th player which is independent of the randomness in π − i , and achieves the highest value for herself conditioned on all other players deploying π − i .
In symbol, the best
We assume players response is the maximizer of max
[Equation]
( s 1 ) whose value we also denote as V † ,π − i i,( s 1 ) for simplicity.
By its deﬁnition, we know the best response can always be achieved at deterministic policies.
A strategy modiﬁcation φ i for the i th player is a set of maps φ i :=
{ φ i,h : ( S × A ) h − 1 × S ×
A i →
A i } , 5 where φ i,h can depend on the history τ h and maps actions in A i to different actions in A i .
For any policy of the i th player π i , the modiﬁed policy (denoted as φ i (cid:5) π i ) changes the action
π i,h ( ω, τ h ) under random sample ω and history τ h to φ i ( τ h , π i,h ( ω, τ h )) .
For any joint policy π , we deﬁne the best strategy modiﬁcation of the i th player as the maximizer of max φ i V ( φ i [Equation] π i ) [Equation]
π −
[Equation] ( s [Equation] Different from the best response, which is completely independent of the randomness in π − i , the best strategy modiﬁcation changes the policy of the i th player while still utilizing the shared randomness among π i and π − i .
Therefore, the best strategy modiﬁcation is more powerful than the best response: formally one can show that
[Equation]
for any policy π .
2. L EARNING OBJECTIVES
A special case of Markov game is Markov Decision Process (MDP).
One can show there always exists an optimal policy π (cid:63) =
argmax π V π 1 ( s 1 ) .
Denote the value of the optimal policy as V (cid:63) .
The objective of learning MDPs is to ﬁnd an (cid:15) -optimal policy π , which satisﬁes V (cid:63) 1 ( s 1 ) − V π 1 ( s 1 ) ≤ (cid:15) .
For Markov games, there are three common learning objectives in the game theory literature—Nash
Equilibrium, Correlated Equilibrium (CE) and Coarse Correlated Equilibrium (CCE).
First, a Nash equilibrium is deﬁned as a product policy where no player can increase her value by changing only her own policy.
Formally,
Deﬁnition 1 (Nash Equilibrium) .
A product policy π is a
Nash equilibrium if
[Equation]
A product policy π is an (cid:15) -approximate
Nash equilibrium
if
[Equation]
We remark that, except for the special case of two-player zero-sum Markov games where reward r 2 ,h
= − r 1 ,h 6 for any h ∈
[ H ] , the Nash equilibrium in general has been proved PPAD-hard to compute Daskalakis (2013).
Therefore, we only present results for ﬁnding Nash equilibria in twoplayer zero-sum MGs in this paper.
Second, a coarse correlated equilibrium is deﬁned as a joint (potentially correlated) policy where no player can increase her value by playing a different independent strategy.
In symbol,
Deﬁnition (Coarse
Correlated
Equilibrium) .
A joint policy π is a
CCE if
[Equation]
A joint policy [Equation] is a [Equation]
-approximate
CCE if
[Equation]
The only difference between Deﬁnition 1 and Deﬁnition 2 is that Nash equilibrium requires the policy π to be a product policy while CCE does not.
Thus, it is clear that CCE is a relaxed notion of
Nash equilibrium, and a Nash equilibrium is always a CCE.
Finally, a correlated equilibrium is deﬁned as a joint (potentially correlated) policy where no player can increase her value by using a strategy modiﬁcation.
In symbol,
Deﬁnition (Correlated
Equilibrium) .
A joint policy
[Equation]
if
[Equation]
policy π
[Equation]
In Markov games, we also have that a Nash equilibrium is a CE, and a CE is a CCE (see Proposition 9 in Appendix C for more details).
V-L EARNING A LGORITHM
In this section, we introduce V-learning algorithm as a new class of single-agent RL algorithms, which converts any adversarial bandit algorithm with suitable regret guarantees into a RL algorithm.
We also present its theoretical guarantees for ﬁnding a nearly optimal policy in the single-agent setting.
Training algorithm To begin with, we describe the V-learning algorithm (Algorithm 1).
It maintains a value V h ( s ) , a counter N h ( s ) , and a policy π h ( ·| s ) for each state s and step h , and initializes them to be the max value, 0 , and uniform distribution respectively.
V-learning also instantiates S × H different adversarial bandit algorithms—one for each ( s, h ) pair.
At each step h in each episode k , the algorithm performs three major steps:
• Policy execution (Line 5-6): the algorithm takes action a h according to the maintained π h , then observes the reward r h and the next state s h +1 , and increases the counter N h
( s h ) by 1 . • V -value update (Line 7-8): the algorithm performs incremental update to the value function:
[Equation]
here α t is the learning rate, and β t is the bonus to promote optimism (and exploration).
The choices of both quantities will be speciﬁed later.
Next, we simply update V h as a truncated version of ˜ V h .
• Policy update (Line 9): the algorithm feeds the action a h and its “loss” H − r h
+ V h
+
( s h
+
H to the ( s h , h ) th adversarial bandit algorithm, and receives the updated policy π h ( ·| s h ) .
Throughout this paper, we will always use the following learning rate α t .
We also deﬁne an auxiliary sequence { α it } ti =1 based on the learning rate, which will be frequently used across the paper.
[Equation]
We remark that our incremental update equation 2 bears signiﬁcant similarity to Q-learning, and our choice of learning rate is precisely the same as the choice in Q-learning (Jin et al., 2018).
However, a key difference is that the V-learning algorithm maintains V-value functions instead of Q-value functions.
This is crucial when extending V-learning to the multiplayer setting where the number of parameters of Q-value functions becomes O ( HS (cid:81) mi =1 A i ) while the number of parameters of
V-value functions is only O ( HS ) .
Since V-learning does not use action-value functions, it resorts to adversarial bandit algorithms to update its policy.
A DV B ANDIT U PDATE
subroutine: Consider a multi-arm bandit problem with adversarial loss, where we denote the action set by B with |B| =
B .
At round t , the learner picks a strategy (distribution over actions) θ t ∈ ∆ B , and the adversary chooses a loss vector (cid:96) t ∈ [0 , 1] B .
Then the learner takes an action b t that is sampled from distribution θ t , and receives a noisy bandit-feedback do
4: set k ← k ih ( s h ) , where i ∈ [ t ] is sampled with probability α it .
action
˜ (cid:96) t ( b t ) ∈ [0 , 1] where E [ ˜ (cid:96) t ( b t ) | (cid:96) t , b t ] = (cid:96) t ( b t ) .
Then, the adversarial bandit algorithm performs updates based on b t and ˜ (cid:96) t ( b t ) , and outputs the strategy for next round θ t +1 , which we abstract as θ t +1 ← A DV B ANDIT U PDATE ( b t , ˜ (cid:96) t ( b t ))
(see Protocol 2).
Output policy We deﬁne the ﬁnal output policy ˆ π of V-learning by how to execute this policy (see
Algorithm 3).
Let V k , N k , π k be the value, counter and policy maintained by V-learning algorithm at the beginning of episode k .
The output policy maintains a scalar k , which is initially uniformly sampled from [ K ] .
At each step h , after observing s h , ˆ π plays a mixture of policy { π k i h ( ·| s h ) } ti = with corresponding probability { α it } ti =1 deﬁned in equation 3.
Here t = N kh
( s h ) is the number of times s h is visited at step h at the beginning of episode k , and k i is short for k ih ( s h ) which is the index of the episode when s h is visited at step h for the i th time.
After that, ˆ π sets k to be the index k ih ( s h ) whose policy is just played within the mixture, and continue the same process for the next step.
This mixture form of output policy ˆ π is mainly due to the incremental updates of V-learning.
One can show that, if omitting the optimistic bonus, V K 1 ( s 1 ) computed in the V-learning algorithm is a stochastic estimate of the value of policy
ˆ π .
We remark that ˆ π is not a Markov policy, but a general random policy (see Deﬁnition in Section 2), which can be written as a set of maps { π h : Ω × S h
→ A i } .
The choice of action at each step h depends on a joint randomness ω ∈ Ω which is shared among all steps, and the history of past states ( s 1 , . . .
, s h ) .
In Section 5, we will further introduce a simple monotone technique that allows
V-learning to output a Markov policy in both the single-agent and the two-player zero-sum setting.
Single-agent guarantees We ﬁrst state our requirement for the adversarial bandit algorithm used in V-learning, which is to have a high probability weighted external regret guarantee as follows.
The weights { α it } ti =1 are deﬁned in equation 3.
Assumption 1.
For any t ∈ N and any δ ∈ (0 , 1) , with probability at least 1 − δ , we have
[Equation]
We further assume the existence of an upper bound Ξ( B, t, log(1 /δ )) ≥ (cid:80) tt (cid:48)
=1 ξ ( B, t (cid:48) , log(1 /δ ))
where (i) ξ ( B, t, log(1 /δ )) is non-decreasing in B for any t, δ ; (ii) Ξ( B, t, log(1 /δ )) is concave in t for any B , δ .
Assumption 1 can be satisﬁed by modifying many existing algorithms with unweighted external regret to the weighted setting.
In particular, we prove that the Follow-the-Regularized-Leader (FTRL) algorithm (Algorithm 5) satisﬁes the Assumption 1 with bounds ξ ( B, t, log(1 /δ )) ≤
O ( (cid:112) HB log( B/δ ) /t ) and Ξ( B, t, log(1 /δ ))
≤
O ( (cid:112) HBt log( B/δ )) .
The H factor comes into the bounds because our choice of weights { α it } in equation 3 involves H .
We refer readers to
Appendix H for more details.
We are now ready to introduce the theoretical guarantees of V-learning for ﬁnding near-optimal policies in the single-agent setting.
Theorem 4.
Suppose subroutine A DV B ANDIT U PDATE satisﬁes Assumption 1.
For any δ ∈ (0 , 1) and K ∈ N , let ι = log( HSAK/δ ) .
Choose learning rate α t according to equation 3 and bonus { β t } Kt =1 so that (cid:80) ti =
α it β i =
Θ( Hξ ( A, t, ι ) + (cid:112) H 3 ι/t ) for any t ∈
[ K ] .
Then, with probability at least 1 − δ , after running Algorithm 1 for K episodes, we have the output policy ˆ π by Algorithm 3 satisﬁes
[Equation]
In particular, when instantiating subroutine A DV B ANDIT U PDATE by FTRL (Algorithm 5), we can choose β t = c · (cid:112) H 3 Aι/t for some absolute constant c , where V (cid:63) 1 ( s 1 ) − V ˆ π 1 ( s 1 ) ≤
O ( (cid:112) H 5 SAι/K ) .
The special cases of Theorem 4 and Theorem 5 (when the subroutine is instantiated by FTRL) were ﬁrstly presented in the conference version of this paper (Bai et al., 2020), with an additional √ H factor in the error.
Theorem 4 characterizes how fast the suboptimality of ˆ π decreases with respect to the total number of episode K .
In particular, to obtain an (cid:15) -optimal output policy ˆ π , we only need to use a number of episodes K =
˜
O ( H 5 SA/(cid:15) 2 ) .
This is H 2 factor larger than the information-theoretic lower bound Ω( H 3 SA/(cid:15) 2 ) in this setting (Jin et al., 2018).
We remark that one extra H factor is due to the incremental update and the use of learning rate in equation 3 which is exactly the same for
Q-learning algorithm (Jin et al., 2018).
The other H factor can be potentially improved by using reﬁned ﬁrst-order regret bound (counterparts of Bernstein concentration) in V-learning.
While V-learning seems to be no better than classical value iteration or Q-learning in the single-agent setting, its true power starts to show up in the multiagent setting: Value iteration and Q-learning require highly nontrivial efforts to adapt them to the multiagent setting, and by design they suffer from the curse of multiagents (Bai et al., 2020;
Liu et al., 2021).
In the following sections, we will show that V-learning can be directly extended to the multiagent setting by simply letting all agents run V-learning independently.
Furthermore, V-learning breaks the curse of multiagents.
T WO - PLAYER Z ERO - SUM M ARKOV G AMES
In this section, we provide the sample efﬁciency guarantee for V-learning to ﬁnd Nash equilibria in two-player zero-sum Markov games.
In the two-player zero-sum setting, we have two agents whose rewards satisfy r 1 ,h = − r 2 ,h for any h ∈
[ H ] .
Our algorithm is simply that both agents run V-learning (Algorithm 1) independently with learning rate α t as speciﬁed in equation 3.
Each player j will uses her own set of bonus { β j,t } that depends on the number of her actions and will be speciﬁed later.
To execute the output policy, both agents simply execute Algorithm 3 independently using their own intermediate policies computed by V-learning.
We have the following theorem for V-learning.
For clean presentation, we denote A = max j ∈ [2] A j .
Theorem 5.
Suppose subroutine A DV B ANDIT U PDATE satisﬁes Assumption 1.
For any δ ∈ (0 , 1) and K ∈ N , let ι = log( HSAK/δ ) .
Choose learning rate α t according to equation 3 and bonus { β j,t } Kt =1 of the j th player so that (cid:80) ti =
α it β j,i =
Θ( Hξ ( A j , t, ι ) + (cid:112) H 3 ι/t ) for any t ∈
[ K ] .
Then, with probability at least 1 − δ , after running Algorithm 1 for K episodes, let ˆ π 1 , ˆ π 2 be the output policies by Algorithm 3 for each player, then we have the product policy ˆ π = ˆ π 1 × ˆ π 2 satisﬁes
[Equation]
When instantiating A DV B ANDIT U PDATE by FTRL (Algorithm 5), we can choose β j,t = c · (cid:112)
H
A j ι/t for some absolute constant c , which leads to max j ∈ [2] [ V † , ˆ π − j j,
[Equation]
O ( (cid:112) H 5 SAι/K ) .
Theorem 5 claims that, to ﬁnd an (cid:15) − approximate Nash equilibrium, we only need to use a number of episodes K =
˜ O ( H 5 SA/(cid:15) 2 ) , where A = max j ∈ [2] A j .
In contrast, value iteration or Q-learning based algorithms require at least Ω( H 3 SA 1 A 2 /(cid:15) 2 ) episodes to ﬁnd Nash equilibria Bai et al.
Liu et al.
Furthermore, V-learning is a fully decentralized algorithm.
To our best knowledge,
V-learning is the only algorithm up to today that achieves sample complexity linear in A for ﬁnding
Nash equilibrium in two-player zero-sum Markov games.
We remark that
V-learning only performs
O (1) operations and calls subroutine
A
[Equation]
ANDIT U PDATE once every time a new sample is observed.
As long as the adversarial bandit algorithm used in V-learning is computationally efﬁcient (which is the case for FTRL),
V-learning itself is also computationally efﬁcient.
M ONOTONIC V-L EARNING
In the previous sections, we present the V-learning algorithm whose output policy (Algorithm 3) is a nested mixture of Markov polices.
Storing such a output policy requires O ( HSA j K ) space for the j th player.
In Section B, we argue this approach has a signiﬁcant advantage over directly storing a general correlated policy when the number of agents is large.
Nevertheless, this space complexity can be undesirable when the number of agents is small.
In this section, we introduce a simple monotonic techique to V-learning, which allows each agent to output a Markov policy when ﬁnding Nash equilibria in the two-player zero-sum setting.
Storing a
Markov policy only takes O ( HSA j ) space for the j th player.
A similar result for the single-agent setting can be immediately obtained by setting the second player in the Markov game to be a dummy player with only a single action to choose from.
Monotonic update Monotonic V-learning is almost the same as V-learning with only the Linein Algorithm 1 changed to
[Equation]
This step guarantees V h ( s h ) to monotonically decrease at each step.
This is helpful because in twoplayer zero-sum Markov games, all Nash equilibria share a unique value which we denote as V (cid:63) .
By design, we can prove that the V-values maintained in V-learning are high probability upper bounds of V (cid:63) (Lemma 18).
This monotonic update allows our V-value estimates to always get closer to V (cid:63) after each update, which improves the accuracy of our V-value estimates.
Markov output policy For an arbitrary ﬁxed ( s, h ) ∈ S ×
[ H ] , let t 1 be the last episode when the value V 1 ,h ( s ) is updated (i.e., strictly decreases), and let t 2 be the last episode when the value
V 2 ,h ( s ) is updated.
Then the output policy for this ( s, h ) has the following form.
[Equation]
where k i denotes the index of episode when state s is visited at step h is visited for the i th time.
Recall that π kj,h ( ·| s ) is the policy maintained by the j th player at the beginning of the k th episode when she runs V-learning.
That is, the new output policy is simply the weighted average of policies computed in the V-learning at each ( s, h ) pair.
Clearly, the policies ˜ π 1 and ˜ π 2 deﬁned by equationare Markov policies.
We remark that although the execution of ˜ π 1 and ˜ π 2 can be fully decentralized, in equation 6 the computation of ˜ π
1 ,h ( ·| s ) depends on t 2 while the computation of ˜ π
2 ,h ( ·| s ) depends on t 1 .
That is, two players need to communicate at the end the indexes of the most recent episodes when their
V -values are updated.
As a result, monotonic V-learning is not fully decentralized.
Theorem 6.
Monotonic V-learning with output policy ˜ π
= ˜
π 1 × ˜ π 2 as speciﬁed by equation 6 has the same theoretical guarantees as Theorem 5 with the same choices of hyperparamters.
Theorem 6 asserts that V-learning can be modiﬁed to output Markov policies when ﬁnding Nash equilibria of two-player zero-sum Markov games.
As a special case, the same technique and results directly apply to the single-agent setting.
A R ELATED WORK
In this section, we focus our attention on theoretical results for the tabular setting, where the numbers of states and actions are ﬁnite.
We acknowledge that there has been much recent work in RL for continuous state spaces (see, e.g., Jiang et al., 2017; Jin et al., 2020; Zanette et al., 2020; Jin et al., 2021a; Xie et al., 2020; Jin et al., 2021b), but this setting is beyond our scope.
Markov games.
Markov Game (MG), also known as stochastic game (Shapley, 1953), is a popular model in multi-agent RL (Littman, 1994).
Early works have mainly focused on ﬁnding Nash equilibria of MGs under strong assumptions, such as known transition and reward (Littman, 2001;
Hu & Wellman, 2003; Hansen et al., 2013; Wei et al., 2020), or certain reachability conditions (Wei et al., 2017; 2021) (e.g., having access to simulators (Jia et al., 2019; Sidford et al., 2020; Zhang et al., 2020))
that alleviate the challenge in exploration.
A recent line of works provide non-asymptotic guarantees for learning two-player zero-sum tabular
MGs without further structural assumptions.
Bai & Jin (2020) and Xie et al.
(2020) develop the ﬁrst provably-efﬁcient learning algorithms in MGs based on optimistic value iteration.
Liu et al.
(2021) improves upon these works and achieve best-known sample complexity for ﬁnding an (cid:15) -
Nash equilibrium— O ( H 3 SA 1 A 2 /(cid:15) 2 ) episodes.
For multiplayer general-sum tabular MGs, Liu et al.
(2021) is the only existing work that provides non-asymptotic guarantees in the exploration setting.
It proposes centralized model-based algorithms based on value-iteration, and shows that Nash equilibria (although computationally inefﬁcient), CCE and CE can be all learned within O ( H 4 S 2 (cid:81) mj
=1 A j /(cid:15) 2 ) episodes.
Note this result suffers from the curse of multiagents.
V-learning—initially coupled with the FTRL algorithm as adversarial bandit subroutine—is ﬁrstly proposed in the conference version of this paper Bai et al.
(2020), for ﬁnding Nash equilibria in the two-player zero-sum setting.
During the preparation of this draft, we note two very recent independent works Song et al.
; Mao & Bas¸ar (2021), whose results partially overlap with the results of this paper in the multiplayer general-sum setting.
In particular, Mao & Bas¸ar (2021) use V-learning with stablized online mirror descent as adversarial bandit subroutine, and learn (cid:15) -
CCE in O
( H 6 SA/(cid:15) 2 ) episodes, where A = max j ∈ [ m ] A j .
This is one H factor larger than what is required in Theorem 7 of this paper.
Song et al.
(2021) considers similar V-learning style algorithms for learning both (cid:15) -CCE and (cid:15) -CE.
For the latter objective, they require O
( H 6 SA 2 /(cid:15) 2 ) episodes which is again one H factor larger than what is required in Theorem 8 of this paper.
Song et al.
(2021) also considers learning Markov potential games by algorithms that are not V-learning, which is beyond the scope of this paper.
We remark that both parallel works have not presented V-learning as a generic class of algorithms which can be coupled with any adversarial bandit algorithms with suitable regret guarantees in a black-box fashion.
Strategic games.
Strategic game is one of the most basic game forms studied in the game theory literature Osborne & Rubinstein (1994).
It can be viewed as Markov games without state and transition.
The fully decentralized algorithm that breaks the curse of multiagents is known in the setting of strategic games.
By independently running no-regret (or no-swap-regret) algorithm for all agents, one can ﬁnd Nash Equilibria (in the two-player zero-sum setting), correlated equilibria and coarse correlated equilibria (in the multiplayer general-sum setting) in a number of samples that only scales with max i ∈ [ m ] A i Cesa-Bianchi & Lugosi (2006); Hart & Mas-Colell (2000); Blum &
Mansour (2007).
However, such successes do not directly extend to the Markov games due to the additional temporal structures involving both states and transition.
In particular, there is no computationally efﬁcient no-regret algorithm for Markov games Radanovic et al.
; Bai et al.
Extensive-form games.
There is another long line of research on MARL based on the model of extensive-form games (EFG) (see, e.g., Koller & Megiddo, 1992; Gilpin & Sandholm, 2006;
Zinkevich et al., 2007; Brown & Sandholm, 2018; 2019; Celli et al., 2020).
EFGs can be viewed as special cases of Markov games where any state at the h th step can be reached from only one state at the ( h − 1) th step (due to tree structure of the game).
Therefore, results on learning EFGs do not directly imply results for learning MGs.
Decentralized MARL There is a long line of empirical works on decentralized MARL (see, e.g.,
Lowe et al., 2017; Iqbal & Sha, 2019; Sunehag et al., 2017; Rashid et al., 2018; Son et al., 2019).
A majority of these works focus on the cooperative setting.
They additionally attack the challenge where each agent can only observe a part of the underlying state, which is beyond the scope of this paper.
For theoretical results, Zhang et al.
(2018) consider the cooperative setting while Sayin et al.
(2021) study the two-player zero-sum Markov games.
Both develop decentralized MARL algorithms but provide only asymptotic guarantees.
Daskalakis et al.
(2021) analyze the convergence rate of independent policy gradient method in episodic two-player zero-sum MGs.
Their result requires the additional reachability assumptions (concentrability) which alleviates the difﬁculty of exploration.
Single-agent RL There is a rich literature on reinforcement learning in MDPs (see e.g. Jaksch et al., 2010; Osband et al., 2014; Azar et al., 2017; Dann et al., 2017; Strehl et al., 2006; Jin et al., 2018; 2021a).
MDPs are special cases of Markov games, where only a single agent interacts with a stochastic environment.
For the tabular episodic setting with nonstationary dynamics and no simulators, the best sample complexity achieved by existing model-based and model-free algorithms are ˜
O ( H 3 SA/(cid:15) 2 ) (achieved by value iteration Azar et al.
and
˜ O ( H 4 SA/(cid:15) 2 ) (achieved by Qlearning Jin et al.
(2018)), respectively, where S is the number of states, A is the number of actions,
H is the length of each episode.
Both of them (nearly) match the lower bound Ω( H 3 SA/(cid:15) 2 )
Jaksch et al.
; Osband & Van Roy (2016); Jin et al.
B M ULTIPLAYER G ENERAL - SUM M ARKOV G AMES
In multiplayer general-sum games, ﬁnding Nash equilibria is computationally hard in general (which is technically PPAD-complete Daskalakis (2013)).
In this section, we focus on ﬁnding two commonly-used alternative notions of equilibria in the game theory—coarse correlated equilibria (CCE), and correlated equilibria (CE).
Both are relaxed notions of Nash equilibira.
B. F INDING COARSE CORRELATED EQUILIBRIA
The algorithm for ﬁnding CCE is again running V-learning (Algorithm 1) independently for each agent j with learning rate α t (as speciﬁed in equation 3) and bonus { β j,t } (to be speciﬁed later).
The major difference from the case of ﬁnding Nash equilibria is that CCE and CE require the output policy to be joint correlated policy.
We achieve this correlation by feeding the same random seed to all agents at the very beginning when they execute the output policy according to Algorithm 3.
That is, while training can be done in the fully decentralized fashion, we require one round of communication at the beginning of the execution to broadcast the shared random seed.
After that, each agent can simply execute her own output policy independently.
During the execution, since the states visited are shared among all agents, shared random seed allows the same index i to be sampled across all agents in the Step 4 of Algorithm 3 at every step.
We denote this correlated joint output policy as ˆ π = ˆ π 1 (cid:12) . . .
(cid:12) ˆ π m .
We remark that to specify a correlated policy in general, we need to specify the probability for taking all action combinations ( a 1 , . .
. , a m ) for each ( s, h ) .
This requires at least Ω( HS (cid:81) mj
=1 A j ) space, which grows exponentially with the number of agents m .
The way V-learning speciﬁes the joint policy only requires agents to store their own intermediate counters and policies computed during training.
This only takes a total of O ( HSK ( (cid:80) mj
=1 A j )) space, which scales only linearly with the number of agents.
Our approach dramatically improve over the former approach in space complexity when the number of agents is large.
We now present the guarantees for V-learning to learn a CCE as follows.
Let A = max j ∈ [ m ] A j .
Theorem 7.
Suppose subroutine A DV B ANDIT U PDATE satisﬁes Assumption 1.
For any δ ∈ (0 , 1) and K ∈ N , let ι = log( mHSAK/δ ) .
Choose learning rate α t according to equation 3 and bonus { β j,t } K t =1 of the j th player so that (cid:80) ti =
α it β j,i =
Θ( Hξ ( A j , t, ι )+ (cid:112) H 3 ι/t ) for any t ∈
[ K ] .
Then, with probability at least 1 − δ , after all the players running Algorithm 1 for K episodes, let ˆ π j be the output policy by Algorithm 3 for the j th player, then we have the joint policy
ˆ π = ˆ π 1 (cid:12) . . .
(cid:12) ˆ π m satisﬁes
[Equation]
When instantiating A DV B ANDIT U PDATE by FTRL (Algorithm 5), we can choose β j,t = c · (cid:112)
H
A j ι/t for some absolute constant c , which leads to max j ∈ [ m ] [ V † , ˆ [Equation] j j,
[Equation]
O ( (cid:112) H 5 SAι/K ) .
Theorem 7 claims that, to ﬁnd an (cid:15) − approximate CCE, V-learning only needs to use a number of episodes K =
˜ O ( H 5 SA/(cid:15) 2 ) , where A = max j ∈ [ m ] A j .
This is in sharp contrast to the prior results for multiplayer general-sum Markov games, which use value iteration based algorithms, and require at least Ω( H 4 S 2 ( (cid:81) mi
=1 A i ) /(cid:15) 2 ) episodes Liu et al.
As a result, V-learning is the ﬁrst algorithm that breaks the curse of multiagents for ﬁnding CCE in Markov games.
B
F INDING CORRELATED EQUILIBRIA
The algorithm for ﬁnding CE is almost the same as the algorithm for ﬁnding CCE except that we now require a different A DV B ANDIT U PDATE subroutine, which has the following high probability weighted swap regret guarantee.
Assumption 2.
For any t ∈ N and any δ > 0 , with probability at least 1 − δ , we have
[Equation]
We assume the existence of an upper bound Ξ sw ( B, t, log(1 /δ )) ≥ (cid:80) tt (cid:48)
=1 ξ sw ( B, t (cid:48) , log(1 /δ ))
where (i) ξ sw ( B, t, log(1 /δ )) is non-decreasing in B for any t, δ ; (ii) Ξ sw ( B, t, log(1 /δ )) is concave in t for any B , δ .
Here Ψ denotes the set { ψ : B → B} which consists of all maps from actions in B to actions in
B .
Meanwhile, for any θ ∈ ∆ B , the term ψ (cid:5) θ ∈ ∆ B denotes the distribution over actions where ψ (cid:5) θ ( b ) =
(cid:80) b (cid:48) :
ψ ( b (cid:48) )
=
b θ ( b (cid:48) ) .
We note that bounded swap regret is a stronger requirement compared to bounded external regret as in equation 4, since by maximizing over a subset of functions in Ψ which map all actions in B to one single action, we recover the external regret by equation 7.
Assumption 2 can be satisﬁed by modifying many existing algorithms with external regret to the swap regret setting.
In particular, we prove that the Follow-the-Regularized-Leader for swap regret (FTRL swap) algorithm
(Algorithm 6) satisﬁes Assumption 2 with bounds ξ sw ( B, t, log(1 /δ )) ≤
O ( B (cid:112) H log( B/δ ) /t ) and Ξ sw ( B, t, log(1 /δ ))
≤ O ( B (cid:112)
Ht log( B/δ )) .
Both bounds have one extra √ B factor comparing to the counterparts in external regret.
We refer readers to Appendix I for more details.
We now present the guarantees for V-learning to learn a CCE as follows.
Let A = max j ∈ [ m ] A j .
Theorem 8.
Suppose subroutine A DV B ANDIT U PDATE satisﬁes Assumption 2.
For any δ ∈ (0 , 1) and K ∈ N , let ι = log( mHSAK/δ ) .
Choose learning rate α t according to equation 3 and bonus { β j,t } Kt =1 of the j th player so that (cid:80) ti =
α it β j,i =
Θ( Hξ sw ( A j , t, ι ) + (cid:112) H 3 ι/t ) for any t ∈
[ K ] .
Then, with probability at least 1 − δ , after all the players running Algorithm 1 for K episodes, let ˆ π j be the output policy by algorithm 3 for the j th player, then we have the joint policy
ˆ π = ˆ π 1 (cid:12) . . .
(cid:12) ˆ π m satisﬁes
[Equation]
When instantiating A DV B ANDIT U PDATE by FTRL swap (Algorithm 6), we can choose β j,t = c ·
A j (cid:112) H 3 ι/t for some absolute constant c , which leads to max j ∈ [ m ] max φ j
[ V φ j (cid:5) ˆ π j [Equation] ( s 1 ) − V ˆ πj, 1 ( s 1 )] ≤
O ( A (cid:112) H
Sι/K ) .
Theorem 8 claims that, to ﬁnd an (cid:15) − approximate CE, V-learning only needs to use a number of episodes K =
˜ O ( H 5 SA 2 /(cid:15) 2 ) , where A = max j ∈ [ m ] A j .
It has an extra A multiplicative factor comparing to the sample complexity of ﬁnding CCE, since CE is a subset of CCE thus ﬁnding CE is expected to be more difﬁcult.
Nevertheless, the sample complexity presented here is far better than value iteration based algorithm, which requires at least Ω( H 4 S 2 ( (cid:81) mi
=1 A i ) /(cid:15) 2 ) episodes for ﬁnding
CE Liu et al.
V-learning is also the ﬁrst algorithm that breaks the curse of multiagents for ﬁnding CE in Markov games.
C N OTATIONS AND B ASIC L EMMAS C. N OTATIONS
In this subsection, we introduce some notations that will be frequently used in appendixes.
Recall that we use V k , N k , π k to denote the value, counter and policy maintained by V-learning algorithm at the beginning of the episode k .
We also introduce a new policy ˆ π kh for a single agent (deﬁned by its execution in Algorithm 4), which can be viewed as a part of the output policy in Algorithm 3.
The deﬁnition of ˆ π kh is very similar to ˆ π except two differences: (1) ˆ π kh is a policy for step h, . .
, H
while ˆ π is a policy for step 1 , . .
, H ; (2) in ˆ π the initial value of k is sampled uniformly at random from [ K ] at the very beginning while in ˆ π kh the initial value of k is given.
We remark that ˆ π kh is a non-Markov policy that does not depends on history before to the h th step.
In symbol, we can express this class of policy as π j :=
{ π j,h (cid:48) : Ω × ( S ×A ) h (cid:48) − h ×S → A j } Hh (cid:48)
=
h .
We call this class of policy the policy starting from the h th step , and denote it as Π h .
Similar to Section 2, we can also deﬁne joint policy π = π 1 (cid:12) . . .
(cid:12) π m and product policy π
=
π
1 × . .
× π m for policies in Π h .
We can also deﬁne value V πh ( s ) for joint policy π ∈ Π h as
[Equation]
This allows us to deﬁne the corresponding best response of π − i as the maximizer of max
[Equation]
( s ) .
We also denote this maximum value as V † ,π [Equation] h ( s ) .
We deﬁne the strategy modiﬁcation for policies starting from the h th step as φ i :=
{ φ i,h (cid:48) : ( S × A ) h (cid:48) −
h × S × A i →
A i } Hh (cid:48)
= h , and denote the set of such strategy modiﬁcation as Φ h .
Finally, for simplicity of notation, we deﬁne two operators P and D as follows:
[Equation]
for any value function V , Q and any one-step Markov policy π .
C. B ASIC LEMMAS
We ﬁrst present a proposition which clarify the relations among the three different kind of equilibria.
In particular, we show that, similar to strategic games, we also have Nash ⊂ CE ⊂ CCE in Markov games.
Proposition 9 (Nash ⊂ CE ⊂ CCE) .
In Markov games, any (cid:15) -approximate
Nash equilibrium is an (cid:15) -approximate CE, and any (cid:15) -approximate CE is an (cid:15) -approximate
CCE.
Proof.
We prove two claims separately.
For Nash ⊂ CE, let π = π 1 × π 2 × · · · π m be an (cid:15)
-approximate
Nash equilibrium, then
[Equation]
Step (a) is because that π is a product policy, where the randomness of different agents are completely independent.
In this case, maximizing over strategy modiﬁcation φ i is equivalent to maximizing over a new independent policy.
Step (b) directly follows from π being an (cid:15) -approximate
Nash equilibrium.
By deﬁnition, this proves that π is also an (cid:15) -approximate CE.
For CE ⊂ CCE, let π = π 1 (cid:12) π 2 (cid:12) · · · π m be
an (cid:15) -approximate CE, then we have
[Equation]
Step (c) is because by deﬁnition of strategy modiﬁcation φ i :
= { φ i,h : ( S×A )
[Equation]
− 1 ×S×A i → A i } , we can consider a subset of strategy modiﬁcation φ (cid:48)
i := { φ (cid:48)
i,h : ( S × A ) h − 1 × S → A i } which modiﬁes the policy ignoring whatever the action π i takes.
It is not hard to see that maxmizing over the strategy modiﬁcation in this subset is equivalent to maximizing over a new independent policy π (cid:48) i .
Therefore, maximizing over all strategy modiﬁcation is greater or equal to maximizing over π (cid:48) i .
Finally, step (d) follows from π being an (cid:15) -approximate CE.
By deﬁnition, this proves that π is also an (cid:15) -approximate
CCE.
Next, we present some basic lemmas that will be used in the proofs of different theorems.
We start by introducing some useful properties of sequence { α it } deﬁned in equation 3.
Lemma 10.
((Jin et al., 2018, Lemma 4.1),(Tian et al., 2021, Lemma 2))
The following properties hold for α it :
1. 1 √ t ≤ (cid:80) ti =
α it √ i ≤ 2 √ t and 1 t ≤ (cid:80) ti =
1 α it i ≤ 2 t for every t ≥ 1 . 2.
max i ∈ [ t ] α it ≤ 2 Ht
and (cid:80) ti =
( α it ) 2 ≤ 2 Ht for every t ≥ 1 . 3.
(cid:80) ∞ t = i α it
= 1 + 1 H for every i ≥ 1 .
Finally, we have the following lemma which express the ˜ V maintained in V-learning in the form of weighted sum of earlier updates.
Lemma 11.
Consider an arbitrary ﬁxed ( s, h, k ) tuple.
Let t =
N kh ( s ) denote the number of times s is visited at step h at the beginning of episode k , and suppose s was previously visited at episodes k 1 , . .
k t < k at the h -th step.
Then the two V-values ˜ V and V in Algorithm 1 satisfy the following equation:
[Equation]
Proof.
The proof follows directly from the update rule in Line 7 Algorithm 1.
Note that α 0 t is equal to zero for any t > 1 and equal to one for t = 0 .
D P ROOFS FOR C OMPUTING CCE IN G ENERAL - SUM MG S
In this section, we give complete proof of Theorem 7.
To avoid repeatedly state the condition of
Theorem 7 in each lemma, we will use
• Condition of the adversarial bandit sub-procudure (Assumption 1) and • Set the bonus { β j,t } Kt =1 of the j th player so that (cid:80) ti =
α it β j,i =
Θ( Hξ ( A j , t, ι )+ (cid:112) H 3 ι/t ) for any t ∈
[ K ] .
throughout the whole section.
The following Lemma is a direct consequence of Assumption 1, which will play an important role in our later analysis.
Lemma 12.
Under Assumption 1, the following event is true with probability at least 1 − δ : for any ( s, h, k ) ∈ S
× [ H ] ×
[ K ] , let t =
N kh ( s ) and suppose s was previously visited at episodes k 1 , . .
k t < k at the h -th step, then for all j ∈ [ m ]
[Equation]
where ι = log( mHSAK/δ ) .
Proof.
By Assumption 1 and the adversarial bandit update step in Algorithm 1, we have that with probability at least 1 − δ , for any ( s, h, k ) ∈ S ×
[ H ] ×
[ K ] ,
[Equation]
which implies the desired result by simple algebraic transformation.
Then we show V is actually an optimistic estimation of the value function of player j ’th
best response to the output policy.
Lemma 13 (Optimism) .
For any δ ∈ (0 , 1] , with probability at least 1 − δ , for any ( s, h, k, j ) ∈
S ×
[ H ] ×
[ K ] × [ m ] , V kj,h ( s ) ≥ V † , ˆ π k − j,h j,h ( s ) .
Proof of Lemma 13.
We prove by backward induction.
The claim is trivially satisﬁed for h = H +1 .
Suppose it is true for h + 1 , consider a ﬁxed state s .
It sufﬁces to show ˜ V kj,h ( s ) ≥ V † , ˆ π k − j,h j,h ( s ) because V kj,h ( s )
= min { ˜ V kj,h ( s ) , H − h + 1 } .
Let t =
N kh ( s ) and suppose s was previously visited at episodes k 1 , . .
k t < k at the h -th step.
Then using Lemma 11, ˜ V kj,h ( s )
=
α 0 t H + t (cid:88)
[Equation]
i =
[Equation]
where ( i ) is by martingale concentration and Lemma 2, ( ii ) is by Lemma 12, ( iii ) is by the deﬁnition of β j,i , and ( iv ) is by induction hypothesis.
Finally, we remark that ( v ) is not directly from Bellman equation since ˆ π k − j,h is non-Markov policy, and the best reponse of a non-Markov policy is not necessary a Markov policy.
We prove ( v ) as follows.
Recalls deﬁnitions for policies in Π h as in Appendix C, by the deﬁnition, we have
[Equation]
where V πj,h +1 ( s, a , s (cid:48) ) for policy π ∈ Π h is deﬁned as:
[Equation]
=
E π (cid:104)(cid:80) Hh (cid:48)
=
h +1 r h (cid:48)
( s h (cid:48) , a h (cid:48) )
(cid:12)(cid:12)(cid:12) s h
= s, a h = a , s h
+1 =
s (cid:48) (cid:105) .
Step (a) uses the relation between ˆ π k − j,h and { ˆ π k i − j,h +1 } i .
Step (b) pushes max inside summation and expectation.
Step (c) is because the Markov nature of Markov game and that { ˆ π k i − j,h +1 } i are policies that does not depend on history at step h , we know the maximization over µ ( h +
): H is achieved at policies in Π h
+1 .
This ﬁnishes the proof.
To proceed with the analysis, we need to introduce two pessimistic V-estimations ˜ V and V that are deﬁned similarly as ˜ V and V .
Formally, let t = N kh ( s ) denote the number of times s is visited at step h at the beginning of episode k , and suppose s was previously visited at episodes k 1 , . .
k t < k at the h -th step.
Then
[Equation]
11) for any player j ∈ [ m ] and k ∈ [ K ] .
We emphasize that ˜ V and V are deﬁned only for the purpose of analysis.
Neither do they inﬂuence the decision made by each agent, nor do the agents need to maintain these quantities when running V-learning.
Equipped with the lower estimations, we are ready to lower bound V
ˆ π kh j,h .
Lemma 14 (Pessimism) .
For any δ ∈ (0 , 1] , with probability at least 1 − δ , the following holds for any ( s, h, k, j ) ∈ S ×
[ H ] ×
[ K ] × [ m ] and any player j , V kj,h ( s ) ≤ V ˆ π kh j,h ( s ) .
Proof of Lemma 14.
We prove by backward induction.
The claim is trivially satisﬁed for h = H +1 .
Suppose it is true for h + 1 , consider a ﬁxed state s .
It sufﬁces to show ˜ V kj,h ( s ) ≤
V
ˆ π kh j,h ( s ) because
V kj,h ( s ) = max { ˜ V k j,h ( s ) , 0 } .
Let t =
N k h ( s ) and suppose s was previously visited at episodes k 1 , . . .
, k t < k at the h -th step.
Then by Equation 10,
[Equation]
where ( i ) is by martingale concentration, ( ii ) is by the deﬁnition of β j,i , and ( iii ) is by induction hypothesis.
To prove Theorem 7, it remains to bound the gap (cid:80)
Kk =1 max j ( V k 1 ,j − V k 1 ,j )( s 1 ) .
Proof of Theorem 7.
Consider player j , we deﬁne δ kj,h :=
V kj,h ( s kh ) − V kj,h ( s kh ) ≥ 0 .
The nonnegativity here is a simple consequence of the update rule and induction.
We want to bound δ kh :=
max j δ kj,h .
Let n kh = N kh (cid:0) s kh (cid:1) and suppose s kh was previously visited at episodes k 1 , . .
k n kh < k at the h -th step.
Now by the update rule of V kj,h ( s kh ) and V kj,h ( s kh ) ,
[Equation]
where in the last step we have used (cid:80) ti =
1 α it β j,i =
Θ( Hξ ( A j , t, ι ) + (cid:112) H 3 ι/t ) .
Now by taking maximum w.r.t.
j on both sides and notice ξ ( B, t, ι ) is non-decreasing in B , we have
[Equation]
Summing the ﬁrst two terms w.r.t.
k ,
K
[Equation]
K
[Equation]
K
[Equation] k [Equation] where ( i ) is by changing the order of summation and ( ii ) is by Lemma 10.
Putting them together,
[Equation]
Recursing this argument for h ∈ [ H ] gives
[Equation]
By pigeonhole argument,
[Equation]
where in the last step we have used concavity.
Finally take the sum w.r
[Equation]
t. h ∈
[ H ] we have
[Equation]
which implies
[Equation]
E P ROOFS FOR C OMPUTING CE IN G ENERAL - SUM MG S
In this section, we give complete proof of Theorem 8.
To avoid repeatedly state the condition of
Theorem 8 in each lemma, we will use
• Condition of the adversarial bandit sub-procudure (Assumption 2) and • Set the bonus { β j,t } Kt =1 of the j th player so that (cid:80) ti =
α it β j,i =
Θ( Hξ sw ( A j , t, ι ) + (cid:112) H 3 ι/t ) for any t ∈
[ K ] .
throughout the whole section.
We begin with a swap regret version of Lemma 12.
Lemma 15.
The following event is true with probability at least 1 − δ : for any ( s, h, k ) ∈ S
× [ H ] ×
[ K ] , let t =
N kh ( s ) and suppose s was previously visited at episodes k 1 , . .
k t < k at the h -th step, then for all j ∈ [ m ]
[Equation]
where ι = log( KHS/δ ) .
Proof.
By Assumption 2 and the adversarial bandit update step in Algorithm 1, we have that with probability at least 1 − δ , for any ( s, h, k ) ∈ S ×
[ H ] ×
[ K ] ,
[Equation]
We begin with proving V is actually an optimistic estimation of the value function under best response.
Lemma 16 (Optimism) .
For any δ ∈ (0 , 1) , with probability at least 1 − δ , the following holds for any ( s, h, k, j ) ∈ S ×
[ H ] ×
[ K ] × [ m ] , V kj,h [Equation] s ) ≥ max
[Equation]
( s ) .
Proof of Lemma 16.
We prove by backward induction.
The claim is trivially satisﬁed for h =
H + 1 .
Suppose it is true for h + 1 , consider a ﬁxed state s .
It sufﬁces to show ˜ V kj,h ( s ) ≥
[Equation]
j [Equation] ( s ) because V kj,h ( s ) =
min { ˜ V kj,h ( s ) , H − h + 1 } .
Let t =
N kh ( s ) and suppose s was previously visited at episodes k 1 , . .
k t < k at the h -th step.
Then using Lemma 11,
[Equation]
[Table]
( s ) where ( i ) is by martingale concentration and Lemma
[Table]
( ii ) is by Lemma
[Table]
( iii ) is by the deﬁnition of β j,
i
[Table]
and ( iv ) is by induction hypothesis.
Finally, ( v ) follows from a similar reasoning as in the proof of Lemma 13, which we omit here.
We still need to lower bound V ˆ π kh j,h .
To do this, we estimate ˜ V and V deﬁned by Equation 10 and
Equation 11.
These quantities are indeed the lower bounds we need.
Lemma 17 (Pessimism) .
For any δ ∈ (0 , 1) , with probability at least 1 − δ , the following holds for any ( s, h, k, j ) ∈ S ×
[ H ] ×
[ K ] × [ m ] , V kj,h ( s ) ≤ V
ˆ π
kh h
( s ) .
Proof of Lemma 17.
We prove by backward induction.
The claim is trivially satisﬁed for h = H +1 .
Suppose it is true for h + 1 , consider a ﬁxed state s .
It sufﬁces to show ˜ V kj,h ( s ) ≤
V
ˆ π kh j,h ( s ) because
V kj,h ( s ) = max { ˜ V kj,h ( s ) , 0 } .
Let t =
N kh ( s ) and suppose s was previously visited at episodes k 1 , . .
k t < k at the h -th step.
Then by equation equation 10,
[Equation]
where ( i ) is by martingale concentration, ( ii ) is by the deﬁnition of β j,i , and ( iii ) is by induction hypothesis.
To prove Theorem 8, it remains to bound the gap (cid:80)
Kk =1 max j ( V k 1 ,j − V k 1 ,j )( s 1 ) .
Proof of Theorem 8.
Consider player j , we deﬁne δ kj,h :=
V kj,h ( s kh ) − V kj,h ( s kh ) ≥ 0 .
The nonnegativity here is a simple consequence of the update rule and induction.
We want to bound δ kh :=
max j δ kj,h .
Let n kh = N kh (cid:0) s kh (cid:1) and suppose s kh was previously visited at episodes k 1 , . .
k n kh < k at the h -th step.
Now by the update rule of V kj,h ( s kh ) and V kj,h ( s kh ) ,
[Equation]
where in the last step we have used (cid:80) ti =
1 α it β j,i =
Θ( Hξ sw ( A j , t, ι ) + (cid:112) H 3 ι/t ) .
Now by taking maximum w.r.t.
j on both sides and notice ξ sw ( B, t, ι ) is non-decreasing in B , we have
[Equation]
Summing the ﬁrst two terms w.r.t.
k ,
K
[Equation]
[Equation]
where ( i ) is by changing the order of summation and ( ii ) is by Lemma 10.
Putting them together,
K (cid:88)
[Equation]
Recursing this argument for h ∈ [ H ] gives
K
[Equation]
By pigeonhole argument,
K (cid:88)
[Equation]
where in the last step we have used concavity.
Finally take the sum w.r.t.
h ∈
[ H ] we have
[Equation]
which implies
[Equation]
F P ROOFS FOR MDP S AND T WO - PLAYER Z ERO - SUM MG S
In this section, we prove the main theorems for V-learning in the setting of single-agent (MDPs) and two-player zero-sum MGs.
Proof of Theorem 5.
To begin with, we notice an equivalent deﬁnition of two-player zero-sum MGs is that the reward function satisﬁes r 1 ,h = 1 − r 2 ,h for all h ∈
[ H ] .
The reason we use this deﬁnition instead of the common version r 1 ,h = − r 2 ,h is we want to make it consistent with our assumption that the reward function takes value in [0 , 1] for any player.
Although this deﬁnition does not satisfy the zero-sum condition, its Nash equilibria are the same as those of the zero-sum version because adding a constant to the reward function of player 2 per step will not change the dynamics of the game.
In order to show ˆ π = ˆ π 1 × ˆ π 2 is an approximate Nash policy, it sufﬁces to control
[Equation]
Since r 1 ,h = 1 − r 2 ,h for all h ∈
[ H ] , with probability at least 1 − δ
[Equation]
where the last inequality follows from Theorem 7.
The reason we can use Theorem 7 here is the precondition of Theorem 5 is a special case of the precondition of Theorem 7.
Proof of Theorem 4.
Since MDPs is a subclass of two-player zero-sum MGs by simply choosing the action set of the second player to be a singleton, it sufﬁces to only prove Theorem 5, from which the single-agent guarantee, Theorem 4 trivially follows.
G P ROOFS FOR M ONOTONIC V- LEARNING
In this section, we prove Theorem 6.
The algorithm is V-learning with monotonic update, and the setting we consider is two-player zero-sum Markov games.
As before, we assume r 1 ,h ( s, a )
= 1 − r 2 ,h ( s, a ) for all s, a, h .
The reason for assuming r 1 ,h ( s, a ) =
1 − r 2 ,h ( s, a ) instead of r 1 ,h ( s, a )
= − r 2 ,h ( s, a ) can be found in Appendix F.
For two player zero-sum MGs, we can deﬁne its minimax value function (Nash value function) by the following Bellman equations
[Equation]
Lemma 18 (Optimism of V-estimates) .
With probability at least 1 − δ , for any ( s, h, k, j ) ∈ S × [ H ] ×
[ K ] × [2] ,
[Equation]
where V (cid:63)j,h is the minimax (Nash) value function deﬁned above.
Proof of Lemma 18.
Note that ˜ V kj,h ( s ) ≥ V kj,h ( s ) is straightforward by the update rule of V-learning, and V † , ˜ π − j j,h ( s ) ≥ V (cid:63)j,h ( s ) directly follows from the deﬁnition of minimax value function.
Therefore, we only need to prove the second inequality.
We do this by backward induction.
The claim is true for h = H + 1 .
Assume for any s and k , V kj,h +1 ( s ) ≥ V † , ˜ π − j j,h +1 ( s ) .
For a ﬁxed ( s, h, k ) ∈ S × [ H ] ×
[ K ] , let t =
N kh ( s ) and suppose s was previously visited in episode k 1 , . .
k t < k at the h -th step.
By Bellman equation,
V † , ˜ π − j j,h
[Equation]
where the second inequality follows from our induction hypothesis and the monotonicity of V k , the third inequality follows from Lemma 12, and the last one follows from martingale concentration as well as Lemma 10.
By Lemma 11 and the precondition of Theorem 6, we know the RHS is no larger than ˜ V kj,h ( s ) .
Note that V k can be equivalently deﬁned as
[Equation]
we conclude V kj,h ( s ) ≥ V † , ˜ π − j j,h ( s ) for any k ∈ [ K ] .
Now we are ready to prove Theorem 6.
Proof of Theorem 6.
By the monotonicity of V and Lemma
[Equation]
where the ﬁrst equality follows from the deﬁnition of two-player zero-sum game, i.e., r 1 ,h = 1 − r 2 ,h .
Now we can mimic the proof of Theorem 7.
Deﬁne δ kh
=
˜ V k 1 ,h ( s kh )
+ ˜ V
k 2 ,h ( s kh ) − ( H − h + 1) .
The non-negativity here follows from Lemma 18 as below
[Equation]
Let n kh = N kh (cid:0) s kh (cid:1) and suppose s kh was previously visited at episodes k 1 , . .
k n kh < k at the h -th step.
By Lemma 11 and the fact that r 1 ,h = 1 − r 2 ,h for all h , we have
[Equation]
where in the last step we used (cid:80) ti =
1 α it β j
[Equation]
i =
Θ( Hξ ( A j
[Equation]
t
[Equation]
) + (cid:112) H 3 ι/t ) .
The remaining steps follow exactly the same as the proof of Theorem 7.
As a result, we obtain
[Equation]
which completes the proof.
H A DVERSARIAL B ANDIT WITH W EIGHTED E XTERNAL R EGRET
In this section, we present a Follow-the-Regularized-Leader (FTRL) style algorithm that achieves low weighted (external) regret for the adversarial bandit problem.
Although FTRL is a classial algorithm in the adversarial bandit literature, we did not ﬁnd a good reference of FTRL with changing step size, weighted regret and high probability bound.
For completeness of this work, we provide detailed derivations here.
We present the FTRL algorithm in Algorithm 5.
In Corollary 19, we prove that FTRL satisﬁes the
Assumption 1 with good regret bounds.
Recall that B is the number of actions, and our normalization condition requires loss ˜ l t ∈ [0 , 1] B for any t .
Corollary 19.
By choosing hyperparameter w t = α t (cid:16)(cid:81) ti =2 (1 − α i ) (cid:17) − 1 and η t = γ t = (cid:113) H log B Bt , FTRL (Algorithm 5) satisﬁes Assumption 1 with
[Equation]
To prove Corollary 19, we show a more general weighted regret guarantee which works for any set of weights { w i } ∞ i =1 in addition to { α it } ti =1 .
In particular, a general weighted regret is deﬁned as
[Equation]
Theorem 20.
For any t ≤ K , following Algorithm 5, if η i ≤ 2 γ i and η i is non-increasing for all i ≤ t , let ι = log( B/δ ) , then with probability 1 − 3 δ , we have
[Equation]
We postpone the proof of theorem 20 to the end of this section.
We ﬁrst show how to obtain Corollary 19 from Theorem 20.
Proof of Corollary 19.
The weights { w t } Kt =1 we choose satisfy a nice property: for any t we have
[Equation]
We prove this for i ≤ j and the other case is similar.
By deﬁnition,
[Equation]
i
[Equation]
(cid:89)
[Equation]
i +
[Equation]
and
[Equation]
We can easily verify that the RHS are the same.
Deﬁne ˜ R ( t ) := max θ
[Equation]
∆ B (cid:80) ti =
1 α it [ (cid:104) θ i , (cid:96) i (cid:105) − (cid:104) θ, (cid:96) i (cid:105) ] .
By plugging w t = α t (cid:16)(cid:81) ti =2 (1 − α i ) (cid:17) −into Theorem 20, and using the property above, we have the regret guarantee
[Equation]
By choosing η t = γ t = (cid:113)
H log B Bt and using Lemma 10, we can further upper bound the regret by
[Equation]
To further simplify the above upper bound, consider two cases:
• If Hι/t ≤ 1 , (cid:112) Hι/t ≥ Hι/t and thus ˜ R ( t ) ≤ 10 (cid:112) HBι/t .
If Hι/t ≥ 1 , (cid:112) HBι/t ≥ 1 ≥ ˜ R ( t ) where the last step is by the deﬁnition of ˜ R ( t ) .
Therefore we have ˜ R ( t ) ≤ (cid:112) HBι/t .
Combining the two cases above gives ˜ R ( t ) ≤ 10 (cid:112) HBι/t .
Finally, we pick ξ ( B, t, log(1 /δ )) :
= 10 (cid:112) HBι/t , which is non-decreasing in B .
Since (cid:80) tt (cid:48)
=1 ξ ( B, t, log(1 /δ ))
≤ 20 √ HBtι , we choose Ξ( B, t, log(1 /δ ))
= 20 √ HBtι , which is concave in t .
To prove Theorem 20, we ﬁrst note that the weighted regret equation 14 can be decomposed into three terms
[Equation]
The rest of this section is devoted to bounding three terms above.
We begin with the following useful lemma adapted from Lemma 1 in Neu (2015), which is crucial in achieving high probability guarantees.
Lemma 21.
For any sequence of coefﬁcients c 1 , c 2 , . . .
c t s.t.
c i ∈ [0 , 2 γ
i ] B is F i -measurable, we have with probability 1 − δ ,
[Equation]
Proof.
Deﬁne w = max i ≤ t w i .
By deﬁnition,
[Equation]
wθ [Equation] ) (cid:33) where ( i ) follows from z 1+ z/ 2 ≤ log (1 + z ) for all z ≥ 0 .
Deﬁning the sum
[Equation]
we have
[Equation]
where ( i ) follows from z 1 log (1 + z 2 ) ≤ log (1 + z 1 z 2 )
Equipped with the above bound, we can now prove the concentration result.
[Equation]
We conclude the proof by taking a union bound.
With Lemma 21, we can bound the three terms ( A ) , ( B ) and ( C ) in equation 15 separately as below.
Lemma 22.
For any t ∈ [ K ] , suppose η i ≤ 2 γ i for all i ≤ t .
Then with probability at least 1 − δ , for any θ (cid:63) ∈ ∆ B , t (cid:88) i =
w i (cid:68) θ i − θ (cid:63) , ˆ l i (cid:69) [Equation] t [Equation]
η
[Equation]
Proof.
We use the standard analysis of FTRL with changing step size, see for example Exercise 28.13 in Lattimore & Szepesv´ari (2018).
Notice the essential step size is η t /w t ,
[Equation]
where ( i ) is by using Lemma 21 with c
i ( b ) =
η i for any b .
The any-time guarantee is justifed by taking union bound.
Lemma 23.
For any t ∈ [ K ] , with probability 1 − δ , t (cid:88) i =
[Equation]
t
[Equation]
i =
[Equation]
ι t (cid:88) i =
w 2 i
[Equation]
Proof.
We further decopose it into t (cid:88) i =
w i (cid:68) θ i , l i − ˆ l i (cid:69) = t (cid:88) i = w i (cid:68) θ i , l i −
[Equation]
The ﬁrst term is bounded by
[Equation]
To bound the second term, notice
[Equation]
thus { w i (cid:68) θ i , E i ˆ l i − ˆ l i (cid:69) } ti =1 is a bounded martingale difference sequence w.r.t.
the ﬁltration {F i } ti =1 .
By Azuma-Hoeffding, t
[Equation]
w
[Equation]
Lemma 24.
For any t ∈ [ K ] , with probability 1 − δ , for any θ (cid:63) ∈ ∆ B , if γ i is non-increasing in i , t
[Equation]
Proof.
Deﬁne a basis { e j } Bj =1 of R B by
[Equation]
1 if a = j 0 otherwise
Then for all the j ∈ [ B ] , we can apply Lemma 21 with c i = γ t e j .
Sincee c i ( b ) ≤ γ t ≤ γ i , the condition in Lemma 21 is satisﬁed.
As a result,
[Equation]
Since any θ (cid:63) is a convex combination of { e j } Bj =1 , by taking the union bound over j ∈
[ B ] , we have
[Equation]
Finally we are ready to prove Theorem 20.
Proof of Theorem 20.
Note the conditions in Lemma 22 and Lemma 24 are satisﬁed by assumptions.
Recall the regret decomposition equation 15.
By bounding ( A ) in Lemma 22, ( B ) in Lemma 23 and ( C ) in Lemma 24, with probability 1 − 3 δ , we have that
[Equation]
I A DVERSARIAL B ANDIT WITH W EIGHTED S WAP R EGRET
In this section, we adapt Follow-the-Regularized-Leader (FTRL) algorithm that achieves low weighted swap regret for the adversarial bandit problem.
We follow a similar technique presented in
Blum & Mansour (2007) which adapts external regret algorithms to swap regret algorithms for the unweighted case.
We present the FTRL swap algorithm in Algorithm 6.
Different from FTRL (Algorithm 5),
FTRL swap maintains an additional B × B matrix ˜ θ t ( ·|· ) , and uses its eigenvector when taking actions.
The matrix will be updated similarly to FTRL, with a subtle difference that the loss estimator here ˆ (cid:96) t ( ·| b ) is θ t ( b ) times the loss estimator ˆ (cid:96) t ( · ) in the FTRL algorithm (Line 4 in Algorithm 5).
In Corollary 25, we prove that FTRL swap satisﬁes the Assumption 2 with good swap regret bounds.
Recall that B is the number of actions, and our normalization condition requires loss ˜ l t ∈ [0 , 1] B for any t .
Corollary 25.
By choosing hyperparameter w t = α t (cid:16)(cid:81) ti =2 (1 − α i ) (cid:17) − 1 and η t = γ t = (cid:113) H log B t , FTRL swap
(Algorithm 6) satisﬁes Assumption 2 with
[Equation]
Again, we prove Corollary 25 by showing a more general weighted swap regret guarantee which works for any set of weights { w i } ∞ i =1 in addition to { α it } ti =1 .
A general weighted swap regret is deﬁned as
[Equation]
Theorem 26.
For any t ≤ K , following Algorithm 6, if η i ≤ 2 γ i and η i is non-increasing for all i ≤ t , let ι = log( B 2 /δ ) , then with probability 1 − 3 δ , we have
[Equation]
We postpone the proof of Theorem 26 to the end of this section.
We show ﬁrst how Theoremdirectly implies Corollary 25.
Proof of Corollary 25.
As shown in the proof of Corollary 19, the weights { w t } Kt =1 we choose satisﬁes a nice property: for any t we have
[Equation]
Deﬁne ˜ R swap ( t ) := max ψ ∈ Ψ (cid:80) ti =
1 α it [ (cid:104) θ
i , l i (cid:105) − (cid:104) ψ (cid:5) θ i , l i (cid:105) ] .
Plugging our choice of w i = α t (cid:16)(cid:81) ti =2 (1 − α i ) (cid:17) − 1 into Theorem 26, we have
[Equation]
By choosing η t = γ t = (cid:113) H log B t and using Lemma 10, we can further upper bound the swap regret by
[Equation]
Hι/t + Hι/t.
To further simplify the above upper bound, consider two cases:
• If Hι/t ≤ 1 , (cid:112) Hι/t ≥ Hι/t and thus ˜ R swap ( t ) ≤ 10 B (cid:112)
Hι/t .
• If Hι/t ≥ 1 , B (cid:112) Hι/t ≥ 1 ≥ ˜ R swap ( t ) where the last step is by the deﬁnition of ˜ R swap ( t ) .
Therefore we have ˜ R swap ( t ) ≤ B (cid:112)
Hι/t .
Combine the above two cases, ˜ R swap ( t ) ≤ 10 B (cid:112)
Hι/t .
Finally, we pick ξ sw ( B, t, log(1 /δ )) :
= 10 B (cid:112)
Hι/t , which is non-decreasing in B .
On the other hand, since (cid:80) tt (cid:48)
=1 ξ sw ( B, t, log(1 /δ ))
≤ 20 B √ Htι , we choose Ξ sw ( B, t, log(1 /δ ))
= 20 B √ Htι , which is concave in t .
To prove Theorem 26, we again ﬁrst decompose the swap regret.
We ﬁrst note that by Line 7 of
Algorithm 6, we have:
[Equation]
On the other hand, by the deﬁnition of strategy modiﬁcation Ψ , we have min ψ
[Equation]
Ψ t [Equation] w
[Equation]
min
[Equation]
w
[Equation] Therefore, we have the following decomposition of the swap regret
[Equation]
For the remaining proof, we bound term ( A ) , ( B ) , ( C ) separately in Lemma 27, Lemma 28,
Lemma 29.
Lemma 27.
For any t ∈ [ K ] , suppose η i ≤ 2 γ i for all i ≤ t .
The with probability 1 − δ , for any θ (cid:63)
[Equation]
Proof.
Similar to Lemma 22, we have, t (cid:88)
[Equation]
b
[Equation]
Summing over b and using the fact that (cid:80) b ∈B ˜ θ i ( b (cid:48) | b ) θ i ( b )
= θ
i ( b (cid:48)
[Equation]
(cid:88) b ∈B t (cid:88) i =
w i (cid:68)
[Equation] i ( ·| b ) − θ (cid:63) ( ·| b ) , ˆ l i ( ·| b ) (cid:69) ≤ w t B log B η t + t (cid:88) i = (cid:88) b (cid:48) ∈B η i w i ˆ l i ( b (cid:48) | b ) θ i ( b ) ( i ) ≤ w t B log B
η t + t (cid:88) i = (cid:88) b (cid:48) ∈B
η i w i l i ( b (cid:48) )
+ 1 2 max i ≤ t w i ι ≤ w t B log B η t + B t (cid:88)
i =
η i w i + 1 2 max i ≤ t w i ι where ( i ) is by using Lemma 21 with c
i ( b ) =
η i .
Notice the quantity ˆ l i ( b (cid:48) | b ) θ i ( b ) actually doesn’t depend on b , so
it is well-deﬁned even after we take the summation with respect to b .
The any-time guarantee is justiﬁed by taking union bound.
Lemma 28.
For any t ∈ [ K ] , with probability 1 − δ , (cid:88) b ∈B t (cid:88) i =
w i (cid:68) ˜
θ i ( ·| b ) , θ i ( b ) l i ( · ) − ˆ l i ( ·| b ) (cid:69) ≤ B t (cid:88) i =
γ i w i + B (cid:118)(cid:117)(cid:117)(cid:116) 2 ι t (cid:88) i =
w 2 i .
Proof.
We further decompose it into t
[Equation]
The ﬁrst term is bounded by
[Equation]
So by taking the sum with respect to b
[Equation]
we have
[Equation]
(cid:29) ≤ (cid:88) b (cid:48) ∈B t (cid:88) i =
[Equation]
[Equation]
(cid:48) ) ≤
[Equation] t (cid:88) i = γ i w i .
To bound the second term, notice ˜ θ
i ( b (cid:48) | b ) θ i ( b ) ≤ θ i ( b (cid:48) ) for any b, b (cid:48) ∈ B , (cid:68)
˜
θ i ( ·| b ) , ˆ l i ( ·| b ) (cid:69) ≤ (cid:88)
b
[Equation] [Equation]
(cid:88) b (cid:48)
[Equation]
1 , thus { w i (cid:68) ˜
θ i ( ·| b ) , E i ˆ l i ( ·| b ) − ˆ l i ( ·| b ) (cid:69) } ti =1 is a bounded martingale difference sequence w.r.t.
the ﬁltration {F i } ti =1 .
By Azuma-Hoeffding, t (cid:88) i = w i
[Equation]
θ
[Equation]
| b ) , E
[Equation]
| [Equation]
) −
[Equation] ≤ (cid:118)(cid:117)(cid:117)(cid:116) [Equation] ι t (cid:88) i =
[Equation]
w 2 i .
The proof is completed by taking the summation with respect to b and a union bound.
Lemma 29.
For any t ∈ [ K ] , suppose γ i is non-increasing in i , then with probability 1 − δ , and any θ (cid:63) , (cid:88) b ∈B t (cid:88)
i =
w i (cid:68) θ (cid:63) ( ·| b
[Equation]
Proof.
The proof follows from Lemma 24 and taking the summation with respect to b .
Finally, we are ready to prove Theorem
[Equation]
Proof of Theorem 26.
Recall the decomposition of swap regret equation 17.
We bound ( A ) in
Lemma 27, ( B ) in Lemma 28 and ( C ) in Lemma 29.
Putting everything together, we have
[Equation]