P OSTERIOR SAMPLING FOR MULTI - AGENT REINFORCEMENT LEARNING : SOLVING EXTENSIVE GAMES WITH IMPERFECT INFORMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Posterior sampling for reinforcement learning (PSRL) is a useful framework for making decisions in an unknown environment.
PSRL maintains a posterior distribution of the environment and then makes planning on an environment sampled from the posterior distribution.
Though PSRL works well on single-agent reinforcement learning problems, how to apply PSRL to multi-agent reinforcement learning problems is largely unexplored.
In this work, we extend PSRL to twoplayer zero-sum extensive-games with imperfect information (TZIEG), which is a class of multi-agent systems.
Technically, we combine PSRL with counterfactual regret minimization (CFR), which is a leading algorithm for TZIEG with a known environment.
Our main contribution is a novel design of interaction strategies.
With our interaction strategies, our algorithm provably converges to the Nash Equilibrium at a rate of O ( √ log T=T ) .
Empirical results show that our algorithm works well.
I NTRODUCTION
Reinforcement Learning (RL) (Sutton & Barto, 2018) provides a framework for decision-making problems in an unknown environment, such as robotics control.
In an RL problem, agents improve their strategies by gaining information from iterative interactions with the environment.
One typical target in designing RL algorithms is to reduce the number of interactions needed to ﬁnd good strategies. Thus, how to reduce the number of samples by designing efﬁcient interaction strategies is one of the key challenges in RL.
Posterior sampling for RL (PSRL) (Strens, 2000) provides a useful framework for deciding how to interact with the environment.
PSRL originates from the famous bandit algorithm Thompson
Sampling (Russo et al., 2018), which uses samples from posterior distributions of bandit parameters to calculate current policy.
PSRL also maintains a posterior distribution for the underlying environment and uses an environment which is sampled from this posterior to compute its interaction strategies.
The interaction strategies are then used to interact with the environment to collect data.
The design of the interaction strategies depends on the speciﬁc property of the task.
For example, in a single-agent RL (SARL) problem, PSRL takes the strategy with the maximum expected reward on the sampled environment as the interaction strategy (Osband et al., 2013). Theoretical and empirical results (Osband & Van Roy, 2016)
both demonstrate that PSRL is one of the near-optimal methods for SARL.
Moreover, although PSRL is a Bayesian-style algorithm, empirical evaluation (Chapelle & Li, 2011) and theoretical analysis on the multi-armed bandit problems (Agrawal & Goyal, 2017) suggest that it also enjoys good performance for a problem with ﬁxed parameters.
However, applying PSRL to multi-agent RL (MARL) requires additional design on the interaction strategies.
This is because the goal of MARL is quite different from that of SARL.
In an MARL problem, each agent still aims to maximize its own reward, but the reward of an agent’s strategy relies not only on the environment, but also on the strategies of other agents.
Therefore, in MARL, the goal of learning is generally referred to ﬁnding a Nash Equilibrium (NE) where no agent is willing to deviate its strategy individually.
So we should design the interaction strategies with which the agents can ﬁnd or approximate the NE efﬁciently.
More speciﬁcally, we consider the RL problems in imperfect information extensive games (Osborne & Rubinstein, 1994).
Extensive games provide a uniﬁed model for sequential decision-making problems in which agents take actions in turn.
Imperfect information here means that agents can keep their own private information, such as the private cards in poker games.
Games with imperfect information are also fundamental to many practical applications such as economics and security.
In particular, we concentrate on two-player zero-sum imperfect information games (TZIEG) where there are two players gaining opposite rewards and a chance player to model the transitions of the environment.
When the environment (i.e. the transition functions of the chance player and the reward functions) is known, counterfactual regret minimization (CFR) (Zinkevich et al., 2008) is the leading algorithm in approximating the NE in a TZIEG.
However, in the RL setting where the environment is unknown, CFR is not applicable.
In this work, we present a posterior sampling algorithm for TZIEGs with the technique of CFR.
That is, we apply CFR to the environment sampled from the posterior distribution.
Our main contribution is a novel design of interaction strategies for the RL problem of TZIEGs.
With the proposed strategies, we show that our algorithm can provably converge to an approximate NE at a rate of O ( √ log T=T ) .
Empirical results show that our algorithm works well.
P RELIMINARY
In this section, we formally deﬁne the task of two-player zero-sum imperfect information games (TZIEG) under Reinforcement Learning; and then brieﬂy review two closely related techniques, namely counterfactual regret minimization (CFR) (Zinkevich et al., 2008) and posterior sampling for reinforcement learning (PSRL) (Osband et al., 2013), which inspire our solution.
2. P ROBLEM FORMULATION OF TZIEG
We start from the deﬁnition of general N -player extensive games (See (Osborne & Rubinstein, 1994, pg. 200) for a formal deﬁnition.), which include TZIEG as a nontivial special case when N = 2 .
Deﬁnition 1 (Extensive game) .
An extensive game has the following components:
(cid:15)
A ﬁnite set of players that includes N agents and a chance player C representing the "Nature" of the game.
(cid:15) A ﬁnite set H of sequences that satisﬁes: 1) The empty sequence is a member of H ; 2) If a sequence f a 1 ; (cid:1) (cid:1) (cid:1) ; a k g belongs to H , then for all 1 (cid:20) l < k , f a 1 ; (cid:1) (cid:1) (cid:1) ; a l g [-and P ( h ) 
-]is a member of H . Here
each member of H is a history and each component of a history is an action taken by a player. The set of available actions after a history is denoted by (cid:11) ( h ) =
f a : ( h; a ) 2 H g and the set of terminal histories is denoted by Z
(cid:15) A function P such that P ( h ) is the player who takes the action after history h . H i represents the set of all h that P ( h ) =
i . (cid:15) A function c (cid:3) that is the strategy of the chance player, i
e., c (cid:3) ( h; a ) is the probability of action a occurs after h if P ( h ) = C . (cid:15) For each player i (besides the chance player), a partition I i of H .
I i is an information partition of player i ; a set
{+I 
 I+} i is a subset of H such that if h 1 ; h 2 2 I , then player i cannot distinguish them. (cid:15)
A reward function r (cid:3) where r (cid:3) ( h; i ) is the distribution of the reward of player i at h 2 Z . We assume the rewards are bounded in [ (cid:0) 1 ; 1] .
For convenience, let A = max h j (cid:11) ( h ) j be the maximal size of actions for one history.
A strategy (cid:27) i for player
i is a mapping from H i to the distribution over valid actions.
We use (cid:27) i ( h; a ) to represent the probability of taking action a at h 2 H i and (cid:27) i ( h ) to be the vector of (cid:27) i ( h; a ) ; a 2 (cid:11) ( h ) .
And a strategy proﬁle (cid:27) consists of the strategies of all players in [ N ] := f 1 ; (cid:1) (cid:1) (cid:1) ; N g , i.e., (cid:27) =
f (cid:27) i g i 2 [ N ] .
We use (cid:27) (cid:0) i to refer to the strategies of all players except i .
Since player i cannot distinguish h 1 ; h 2 2 I
I i , (cid:27) i ( h 1 ) and (cid:27) i ( h 2 ) must be the same.
For the clarity of notation, we use d (cid:3) to denote the corresponding ( c (cid:3) ; r (cid:3) ) .
Let u i ( h j (cid:27); d (cid:3) ) denote the expected reward of player
i 2 [ N ] at history h under strategy (cid:27) .
For convenience, let u i ( (cid:27) j d (cid:3) )
=
u i ( h r j (cid:27); d (cid:3) ) where h r is the root of H and u i ( h j r (cid:3) ) is the expected reward of player i for h 2 Z .
We use (cid:25) (cid:27) ( h j d (cid:3) ) to denote the probability of reaching h with (cid:27) and c (cid:3) .
It is easy to see that we can decompose (cid:25) (cid:27) ( h j d (cid:3) ) into the product of the contribution of each player. That is, (cid:25) (cid:27) ( h j d (cid:3) )
=
∏ i 2 [ N ] [fCg (cid:25) i(cid:27) ( h j d (cid:3) ) .
We use D ( h ) to refer to the depth of h in the game tree and D i ( h ) to refer to the number of h ’s ancestors whose player is i .
Obviously, we have D ( h ) =
1+ ∑ i 2f N g[fCg D i ( h ) .
And let D = max h D ( h ) and D i = max
h D
i
( h ) .
With the above notations, then in a TZIEG there are two-players besides the chance player (i.e., N = 2 ), player 1 and player 2 , and u 1 ( h j r (cid:3) ) + u 2 ( h j r (cid:3) ) =
for all histories h 2 Z .
Nash Equilibrium and exploitability : In a multi-agent system, a solution is often referred to a Nash Equilibrium (NE) (Osborne & Rubinstein, 1994).
In a TZIEG, (cid:27) =
( (cid:27) 1 ; (cid:27) 2 ) is a NE if and only if u i ( (cid:27) j d (cid:3) )
=
max (cid:27) (cid:3) ;i u i ( (cid:27) (cid:3) [Equation] i [Equation]
j d [Equation] Our target is to approximate NE.
More speciﬁcally, in TZIEGs, the approximation error of (cid:27) =
( (cid:27) 1 ; (cid:27) 2 ) is usually measured by its exploitability:
[Equation]
If the environment of a TZEIG, i.e. d (cid:3) , is known for the players, we can directly use counterfactural regret minimization (CFR) (Zinkevich et al., 2008) to minimize the exploitability of this TZEIG, as brieﬂy reviewed in Sec. 2.2.
In this paper, we concentrate on the more challenging and realistic yet largely unexplored setting of
TZEIG, where the environment d (cid:3) is unknown and moreover d (cid:3) is subject to some intrinsic uncertainty.
For example, poker games with unknown deck fall into this setting.
In practice, this setting is not uncommon in industrial organisation with unknown entry, exit and ﬁrm-speciﬁc sources (Ericson & Pakes, 1995).
In this setting, players have to interact with the unknown environment to gain sufﬁcient knowledge of the game for making optimal decisions, thereby becoming a reinforcement learning (RL) task.
In particular, the task of ﬁnding approximate NEs for TZIEGs with unknown d (cid:3) is a Multi-Agent Reinforcement Learning (MARL) task (Bu¸soniu et al., 2010).
Moreover, to consider the intrinsic uncertainty of the unknown environment, we adopt a Bayesian formulation for our TZEIG task, which can ﬂexibly incorporate the prior information and enjoys various potential beneﬁts (Russo & Van Roy, 2014).
Formally, we consider the setting where the chance player and the reward functions follow a prior distribution P 0 .
That is, the underlying d (cid:3) = ( c (cid:3) ; r (cid:3) ) is sampled from P 0 ( c; r ) . Here c and r are not necessarily independent.
After playing t games, players collect some samples from d (cid:3) = ( c (cid:3) ; r (cid:3) ) and they can get the posterior distribution, denoted as P t .
For example, in the case where r (cid:3) ( h; i ) is a Bernoulli distribution and its prior is a Beta distribution, the posterior distribution P t ( r ) is also a Beta distribution.
Similarly if the prior for c (cid:3) is a Dirichlet distribution, then P t ( c ) is a Dirichlet distribution since c (cid:3) ( h ) is a multinomial distribution for h 2 H C .
To solve the above TZIEG problems, we present a method that draws inspirations from the solutions for two simpliﬁed settings, as brieﬂy reviewed below.
2. C OUNTERFACTUAL REGRET MINIMIZATION (CFR)
As stated above, when the parameters of the game, i.e. d (cid:3) = ( c (cid:3) ; r (cid:3) ) , are known 1 , counterfactual regret minimization (CFR) (Zinkevich et al., 2008) provides an effective solution to TZIEGs with state-of-the-art performance.
Formally, CFR is a self-play algorithm that generates a sequence of strategy proﬁles, f (cid:27) t g Tt =1 , by minimizing the following regrets:
[Equation]
For convenience, we write (cid:22) (cid:27) T = 1 T ∑
[Equation] (cid:27) t if (cid:22) (cid:27)
iT ( I ) =
[Equation]
∑ [Equation] i(cid:27)t [Equation]
I [Equation]
One important observation (Zinkevich et al., 2008) is that in a TZIEG the exploitability is:
[Equation]
Algorithm while t < T do
Sample d t and ~ d t from the posterior P t .
for all i 2 f 1 ; 2 g do
Select (cid:27) t by exploiting CFR to minimize the regret: max (cid:27) i
∑ t (cid:20) T u i ( (cid:27) i ; (cid:27) (cid:0) i t j d t ) (cid:0)
∑ t (cid:20) T u i ( (cid:27) t j d t ) .
end for
Calculate interaction strategies ^ (cid:27) 1 ;T and ^ (cid:27) 2 ;T with Eq.(6). Use ^ (cid:27) 1 ;T and ^ (cid:27) 2 ;T to interact with the environment to gather data and then compute P t +1 .
end while
Output Therefore, CFR makes (cid:22) (cid:27) T converge to the NE by minimizing R (cid:3) 2. P OSTERIOR SAMPLING FOR REINFORCEMENT LEARNING (PSRL) Posterior sampling for reinforcement learning (PSRL)
(Osband et al., 2013) provides an effective method for solving RL problems when the environment has uncertainty.
Formally, PSRL applies to the Bayesian RL setting with a given prior distribution over the transition and reward function; and the agents can access this prior distribution and then update the posterior distribution using the observations collected from interaction with the environment.
PSRL provides a framework on how to select the strategy to interact with the environment under the Bayesian setting.
The process of PSRL can be decomposed into two steps: (1) sampling one environment from the posterior and computing strategies for agents according to the sampled environment; (2) using the computed strategies to interact with the underlying environment and updating the posterior distribution with the collected data. The two steps are iterated. The strategies that are used to play games are called interaction strategies. For different kinds of RL problems, the interaction strategies for PSRL are different.
For example,
PSRL chooses the strategy with the maximum expected value as the interaction strategy in the singleagent RL problems (SARL).
However, this cannot be trivially extended to MARL since the learning goal turns to be the NE. Hence, in order to apply PSRL to our problem of TZEISs, we need to design of proper interaction strategies.  M ETHOD In this section, we formally present our method, which conjoins the merits of PSRL and CFR and can efﬁciently compute the approximate NE for TZIEG tasks.
The key is our design of a proper interaction strategy, which can coordinate with CFR to interact with the environment.
Before diving into details, we given an overview of our algorithm. We call two games interacted with the environment as one episode. In episode t , we sample a d t from posterior distribution P t and then apply CFR to d t to get a policy tuple ( (cid:27) 1 t ; (cid:27) 2 t ) . Then we sample another ~ d t to calculate the interaction strategies.
Then we use the interaction strategies to interact with the environment to collect data and update the posterior. Our algorithm can converge to the NE at a rate of O ( √ log( T ) =T ) . The time complexity of computing the interaction strategies is linear to j H j .
Here we introduce our method in detail.
The algorithm is presented in Alg.1.
The detailed format of the interaction strategy will be given soon in Eq.
To compute the approximate NE, we adopt a CFR algorithm to minimize the following regret for episode T :
[Equation]
(3) where d t is sampled from the posterior distribution at episode t (i.e., P t ).
Then we take (cid:22) (cid:27) =
T ∑ t (cid:20) T (cid:27) t as the output strategy for our algorithm.
Obviously, simply minimizing ^ R T will not make the exploitability expl ((cid:22) (cid:27) j d (cid:3) ) small, as d (cid:3) can be very different with d t , so we need the interaction strategy to be efﬁcient enough to make sure the difference between d t and d (cid:3) is relatively small.
The following equation establishes a relation between the exploitability expl ((cid:22) (cid:27) j d (cid:3) ) and regret ^ R iT :
[Equation]
Here by ﬁxing that the player (cid:0) i plays the strategy (cid:27) (cid:0) i t at episode t , 2 we use (cid:27) (cid:3) ;iT = arg max [Equation] i [Equation]
) to denote player i ’s optimal strategy in the underlying game d (cid:3) , and (cid:27) ′ iT =
arg max (cid:27) i
∑ t (cid:20) T u i ( (cid:27) i ; (cid:27) (cid:0) i t j d t ) to denote player
i ’s optimal strategy when the game at episode t is d t .
For convenience, let G iT =
1 T ∑ t (cid:20) T ( u i ( (cid:27) (cid:3) ;iT ; (cid:27) (cid:0)
i t j d (cid:3) ) (cid:0) u i ( (cid:27) ′ iT ; (cid:27) (cid:0) i t j d t )) denote the gap between exploitability and the regret from CFR. As we can minimize ^ R iT by CFR, so we only need to minimize G iT in order to minimize expl ((cid:22) (cid:27) j d (cid:3) ) . Obviously, we can design interaction strategies to reduce the uncertainty of the posterior, i.e., the gap between d (cid:3) and d t , which will make G iT decrease as well.
The remaining challenge is to design interaction strategies to minimize G iT efﬁciently.
In episode t , we ﬁrst draw ~ d t = (~ c t ; ~ r t ) (cid:24) P t .
Then for i 2 f 1 ; 2 g , we compute the strategy that maximizes the cumulative reward gaps between games sampled from the posterior:
[Equation]
Interaction strategy: We adopt the following interaction strategies for episode T :
[Equation]
and
[Equation]
[Equation]
The computation of ~ (cid:27) it can be implemented in time O
( j H j ) .
To make the whole procedure clear, we use a simple toy game to show the game tree at episode t in Fig.
We present d t , ~ d t and (cid:27) t . The interaction strategy is then calculated from these quantities. It needs to be emphasized that the strategies (cid:27) 1 t and (cid:27) 2 t are generated by CFR in episode t (cid:0) 1 and they are used as the opponents’ strategies in the interaction strategies. With the interaction strategies (~ (cid:27) 1 T ; (cid:27) 2 T ) and ( (cid:27) 1 T ; ~ (cid:27) 2 T ) , we can prove the following bound on the exploitability expl ((cid:22) (cid:27) ) .
Theorem 1.
Let (cid:24) i =
[Equation]
{+[Equation] j (cid:25) i(cid:27) i ( I ) . Use E d (cid:3) to represent the expectation over all the prior distribution P 
 ( d (cid:3) ) .+}
If the true game is sampled from a prior P 0 over the chance player nodes and terminal nodes, then for (cid:22) (cid:27) T computed by Alg. 1, we have
[Equation]
Here (cid:24) i is a game-dependent parameter, related to the structure of the game. Its deﬁnition comes from Corrollary 2 in Burch (2018). Generally, we have that
[Equation]
The present theorem is signiﬁcant at least in the following aspects.
Firstly, the per episode running time is linear to the size of game tree and the bound is sublinear to T .
Thus, we can expect our algorithm to reach a certain approximate error in a ﬁnite time.
Secondly, our theorem holds for any prior distribution over d (cid:3) .
In practical TZIEGs, it is possible that the priors for h 1 and h 2 , h 1 ; h 2 2 H C , are independent.
Our theorem and algorithms can also be applied to such situations.
Lastly, our interaction strategies ^ (cid:27) 1 ;T and ^ (cid:27) 2 ;T only contribute to the bound for G iT , which can be treated as the error for interaction strategy’s exploring the environment.
If we apply PSRL to a single-agent tree game, the Bayesian regret might be considered as some error caused by interacting with the environment.
Using the analysis in (Osband et al., 2013), we can get that PSRL enjoys an averaged Bayesian regret bound of order O (
[Equation]
for a general prior.
Therefore, our bound for G iT has a comparable order to the bound for the average
Bayesian regret in single-agent tree games.
3. P ROOF SKETCH OF T HEOREM
Before giving the detailed proof, we introduce some additional notations.
For episode t , we generate two trajectories by interacting with the environment.
More speciﬁcally, we use T i;t ( i 2 f 1 ; 2 g ) to denote the trajectory generated by ^ (cid:27) i;t in environment d (cid:3) .
We use E T i;t to denote the expectation over all trajectories for episode t .
Then we use T C i;t =
f h C 1 ;t ; h C 2 ;t ; :::; h C m i;t ;t g to denote the trajectory for the chance player in episode t , and here m i;t denotes the length of T C i;t .
Furthermore, we denote the terminal node for T i;t as z i;t .
Besides, we denote the collection of T 1 ; 1 ; T 2 ; 1 :::; T 1 ;t (cid:0) 1 ; T 2 ;t (cid:0) 1 and the related rewards as H t , which represents all the observations before episode t .
For each history h , we further use n t ( h ) to denote the count that h has been visited in H t .
Further, we use E T i;t to denoting the expectation over all possible trajectories T i;t .
Below we give the key part for the proof.
Obviously, we need to bound the regret of CFR, i.e., ^ R iT , and G iT .
We can directly apply the technique in Theorem 1 of (Burch, 2018, pg. 34) to bound ^ R iT with Eq.
Next we show the key part for bounding G iT .
Using the deﬁnition of (cid:27) ′ iT , we have:
[Equation]
And then, in Lemma 1 and 2, we decompose the above bound into the weighted sum of j c (cid:3) ( h ) (cid:0) c t ( h ) j and j r (cid:3) ( h ) (cid:0) r t ( h ) j .
And soon later we will show how each term decreases as the number of episodes increases.
Lemma 1.
At episode T , with ^ (cid:27) deﬁned in Eq.
(6) , we can upper bound the expectation of G iT :
[Equation]
Lemma 1 decomposes the expectation of G iT into two terms, representing the reward difference between d (cid:3) and ~ d t and the difference between d (cid:3) and d t .
Below we give an intuitive sketch for bounding the ﬁrst term u i (~ (cid:27) it ; (cid:27) (cid:0) i t j ~ d t ) (cid:0) u i (~ (cid:27) it ; (cid:27) (cid:0)
i t
j d (cid:3) ) .
Lemma 2.
At episode T , the expectation of u i (~ (cid:27) [Equation] [Equation] (cid:27) (cid:0) i t j d (cid:3) ) can be bounded by the summation of the difference between d (cid:3) and ~ d t :
[Equation]
According to the deﬁnition of the expectation E ~ d;d (cid:3) E T i;t , we can see that Eq.
(11) is a weighted sum of j c t ( h ) (cid:0) c (cid:3) ( h ) j and j u i ( h j ~ r t ) (cid:0)
u i ( h j r (cid:3) )
j .
Recall that u
i ( h j r ) refers to the expectation of r ( h ) for player i .
Intuitively, we can use concentration bound on j c t ( h ) (cid:0) c (cid:3) ( h ) j , so for h with a large weight, we should visit it for more times.
Notice that the weight in Eq.
(11) is essentially the probability of reaching h under our interaction strategy ^ (cid:27) and the real environment c (cid:3) .
Hence if we use ^ (cid:27) to interact with the environment, we can expect our algorithm can visit h with large weight for sufﬁcient times.
To simplify the derivation, we tentatively assume that ~ d t and d (cid:3) are identically distributed for nodes h C j;i and z i;t conditioning on H t .
That is, for any node h , with Pr referring to the probability of some event, we here assume that
[Equation]
In fact this assumption fails when h is reached, because the probability to reach h is inﬂuenced by d (cid:3) and ~ d .
We will remove this assumption and provide a rigorous proof in Appendix A. For ( h C j;i ; a ) and z i;t , we can insert the empirical mean estimations (cid:22) c t ( h C j;i ; a ) and (cid:22) u it ( z i;t ) and use the frequentists’ concentration bound (Hoeffding, 1994; Weissman et al., 2003).
Then for any (cid:14) 2 (0 ; 1) , we have the following inequalities:
[Equation]
E H t { [Equation] [Equation] √[-2 ln(
-]
[Equation] log( [Equation] (cid:14) ) max
[Equation] n [Equation] z i
[Equation]
t
[Equation]
1) (cid:12)(cid:12)(cid:12) [Equation] [Equation] j Z j (cid:14):
Then for a history h 2 Z [ H C , we have n t ( h ) ∑ n = i √
=i (cid:20) √ n t ( h ) .
Using the Jensen’s inequality to the summation over Z and H C , we can get the following bound:
T ∑
[Equation]
We can apply the same method to T ∑
[Equation] H
[Equation] } and get Eq.
Combining the results of Eq. (7) and (8), we can ﬁnish the proof of the theorem 1. [Figure]
R ELATED W ORK
Other methods for TZIEGs under unknown environment : There also exit some works on TZIEGs under an unknown environment. Fictitious play (FP) (Brown, 1951) is another popular algorithm for approximating NE.
In FP, the agent takes the best response to the average strategy of its opponent.
Heinrich et al.
(2015) extend FP to TZIEGs.
Though it may be easier to combine FP with other machine learning techniques than CFR, when the chance player is known, the convergence rate of FP is usually worse than CFR variants.
Monte Carlo CFR with outcome sampling (MCCFR-OS) (Lanctot et al., 2009) can also be applied to TZIEGs to approximate NE in a modelfree style.
It uses Monte Carlo estimates of the environment to conduct CFR and can converge to the NE.
Since it is updated without a model of the environment, it is much less efﬁcient than modelbased methods.
There is also work that applies SARL methods to TZIEGs.
For example, Srinivasan et al.
adapts actor-critic to games in a model-free style. MDP : SARL problems are often formalized as the Markov Decision Process (MDP).
In the simplest MDP with no transitions, i.e. the Multi-armed bandit problems, the problem-dependent regret upper bound of PSRL (also named Thompson Sampling in bandit problems) has been carefully analyzed (Agrawal & Goyal, 2017).
The problem-dependent bounds for general MDP is still an open problem.
Besides PSRL, there is another kind of provable algorithms for MDP (Jaksch et al., 2010;
Azar et al., 2013) following the Optimism in the Face of Uncertainty principle.
They estimate the uncertainty of the underlying MDP and then use the currently optimal policy to interact with the environment.
Stochastic Games : the stochastic game (Littman, 1994) is also one kind of multi-agent systems.
In a stochastic game, players take actions at each state and then the environment transits to a new state and returns immediate rewards.
Nash Q-learning (Hu & Wellman, 2003) converges to approximate
NE by extending Q-learning to games, but it lacks ﬁnite-time analysis.
Some other work (Szepesvári & Littman, 1996; Perolat et al., 2015; Wei et al., 2017) concentrates on two-player zero-sum stochastic games in RL setting.
This kind of games don’t involve imperfect information, and this makes them different from TZIEG.
E XPERIMENTS
To empirically evaluate our algorithm, we test it on imperfect-information poker games.
In this section, we ﬁrst introduce our baseline methods and then present the details of the games.
Finally, we show the results.
We choose three kinds of methods as our baselines.
The ﬁrst one is Fictitious Play (FP) and the second is Monte Carlo CFR with outcome sampling (MCCFR). Both are algorithms for solving MARL.
Thus we can compare the performance of our algorithm and these existing methods.
We choose two variants of our algorithm as the other kind of baselines, which is used to compare different choices of interaction strategies.
Details of baselines are given below:
(cid:15) Fictitious self-play (FSP): FSP is another popular algorithm to solve games in the RL setting.
In FP, when d (cid:3) is known, each player chooses the best response of its opponent’s average strategy.
When d (cid:3) is not known, we need other RL algorithms to learn the best response.
We combine FSP with two kinds of RL algorithms: 1) FSP with a ﬁtted-Q iteration algorithm (FSP-ﬁtted-Q): we follow (Heinrich et al., 2015) to use a Fitted-Q iteration to learn the best response.
We use the same hyperparameters as reported in Heinrich et al.
(2015); 2) FSP with PSRL (FSP-PSRL): we use a combination of FP and PSRL to give a new baseline:
In episode t , we compute player i ’s best response under d t (cid:24) P t , that is, arg max (cid:27) i ∑ t ′ <
t u i ( (cid:27) i ; (cid:27) (cid:0) i t ′ j d t ) .
(cid:15) Monte Carlo CFR with outcome sampling (MCCFR-OS): MCCFR-OS uses samples of the game tree to conduct CFR.
Estimated counterfactual values are used to update the policies and the average polices can converge to NE.
This method can be applied to TZIEGs under the RL setting.
We use ϵ -greedy with ϵ = 0 : 1 as the exploration strategy for MCCFR-OS in our experiments.
(cid:15) Variants of Alg.
1: Though we have proved the convergence of Alg. 1 with interaction strategy (~ (cid:27) i ; (cid:27) (cid:0) i ) , the proposed method does not necessarily work well in practice.
In our experiments, we evaluate four interaction strategies: 1) Random: the players take actions randomly; 2) Naive: the players use the output of the CFR procedure, i.e., (cid:27) t , to interact with the environment; 3) the best response to (cid:27) (cid:0) i t (Bestresp): (~ (cid:27) ′ it ; (cid:27) (cid:0) i t ) where ~ (cid:27) ′ it is the best response to (cid:27) (cid:0) i t under ~ d t i.e., ~ (cid:27) ′ it = arg max (cid:27) i u i ( (cid:27) i ; (cid:27) (cid:0) i t j ~ d t ) ; 4) Default: the interaction strategies in Eq.
We test these algorithms on variants of Leduc Hold’em poker (Southey et al., 2012) which is widely used in imperfect-information game solving.
We generate games by keeping the tree structure of the
Leduc Hold’em poker and replacing c and r by randomly generated functions.
More speciﬁcally, when generating the tree structure, to control the sizes of the generated game tree, we restrict each player not to bid more than 4 or 5 times the big blind.
The numbers of histories in the generated games are 9435 and 34776 respectively.
The reward function r i ( h ) is a binary distribution.
With a probability p the value of r 1 ( h ) is (cid:0) 1 and with probability 1 (cid:0) p , the value is 1 .
The prior P 0 ( r 1 ( h )) is a uniform [0 ; 1] distribution over parameter p . Obviously, r 2 ( h ) = (cid:0) r 1 ( h ) .
Let e d denote the vector in R d with every element is 1 .
c ( h ) is sampled from Dirichlet ( e j A ( h ) j ) .
We generate 20 variants for Leduc(4) and Leduc(5) respectively.
And on each generated game, each algorithm updates its strategies for 10000 times, and after each update, it interacts with the environment for 2 rounds of games.
The result is shown in Fig.
As Fig. 2 shows, the exploitability of naive CFR fails to decrease after 10000 rounds on both Leduc(4) and Leduc(5).
This might be caused by the lack of efﬁcient exploration of the environment.
MCCFR-OS and FSP-Fitted-Q have poor performances comparing to other algorithms.
This may be caused by the data-inefﬁciency of model-free methods and the inefﬁcient exploration strategies.
Random interaction and FP can gradually decrease the exploitability, but our algorithm decrease at a higher speed.
Thus the empirical result shows that our algorithm outperforms baselines on the two games.
C ONCLUSIONS AND DISCUSSIONS
In this work, we consider the problem of posterior sampling for TZIEGs, which is a class of multiagent reinforcement learning problems.
By a novel design of interaction staregies, we conjoin the merits of PSRL and CFR and present a provably convergent algorithm for TZIEGs.
Our algorithm empirically works well.
In the future, there are various directions to improve the result.
For example, our bound is a Bayesian bound describing the expected performance.
Considering one sample from the prior, Frequentists’ methods such as UCBVI (Azar et al., 2013) also give a high probability regret bound for SARL of a similar order to PSRL.
Further, comparing with the worst-case bound, the problem-dependent performance is much more important.
Though it is possible that our method has a better performance on a speciﬁc TZIEG than the bound in Theorem 1, our algorithm is very possibly not the best in the sense of problem-dependent performance.
Another direction is that our method heavily relies on the structure of TZIEGs and the solution concept of Nash Equilibrium.
Thus, further work is needed to extend posterior sampling to more complicated multi-agent systems, such as stochastic games (Littman, 1994) and extensive games with more than two players.
A P ROOF FOR THEOREM
Let (cid:22) (cid:27) =
1 T ∑ t (cid:20) T (cid:27) t denote the average strategy tuple up to episode T .
We can decompose the exploitability at episode T into the CFR regret and an extra exploration term:
[Equation]
where ^ R iT , (cid:27) (cid:3) ;iT and (cid:27) ′ iT are formally deﬁned as:
[Equation]
Here ^ R iT is the regret from CFR.
And (cid:27) (cid:3) ;iT and (cid:27)
iT represent the optimal strategies under d (cid:3) and d t respectively, ﬁxing that the its opponent plays the strategy (cid:27) (cid:0) i t at episode t .
Recall that d t is sampled from the posterior distribution P t .
The proof for this decomposition is given below:
Proof.
Inserting a term u i ( (cid:27) ′ iT ; (cid:27) (cid:0) i t j d t ) for episode t , we have: ∑
[Equation]
Moreover, with u 1 ( (cid:27) t j d t ) + u 2 ( (cid:27) t j d t ) = 0 and the deﬁnition of expl , we ﬁnish the proof.
By directly applying the result in (Burch, 2018), we can upper bound the CFR regret with
[Equation]
where (cid:24) i =
∑ Dj = √ [Equation] i(cid:27) i ( I ) represents the complexity of the game tree.
For convenience, let G iT
=
1 T ∑ t (cid:20) T ( u i ( (cid:27) (cid:3) ;iT ; (cid:27) (cid:0)
i t j d (cid:3) ) (cid:0) u i ( (cid:27) ′ iT ; (cid:27) (cid:0) i t j d t )) denote the gap between exploitability and the regret from CFR. Since we have upper bounded
^ R iT , we only need to bound G iT
and then we can upper bound the exploitability of the average strategy (cid:22) (cid:27) .
Obviously, G iT represents the difference between d (cid:3) and d t .
Thus we need to design a suitable interaction strategy to make sure that G i T is small.
Using the deﬁnition of (cid:27) ′
i T , we have
[Equation]
We select the interaction strategy as follows. First, we draw ~ d t (cid:24) P t .
For i 2 f 1 ; 2 g , we compute
[Equation]
And then we use (~ (cid:27) 1 t ; (cid:27) 2 t ) and (~ (cid:27) 2 t ; (cid:27) 1 t ) to interact with the environment.
Following lemma provides an upper bound on G it .
Then we get below lemma:
Lemma 3.
With ~ (cid:27) it deﬁned in Eq. [Equation]
the expectation of G iT can be bounded by: [Equation]
[Equation]
This lemma decompose G iT into two terms, which can be bounded with careful analysis of the posterior distribution.
We ﬁrst give the proof for this lemma.
Proof.
We have
[Equation]
(cid:12)(cid:12)(cid:12) H T } + E H T {
E ~ d T ;d T [ u i (~ (cid:27) iT ; (cid:27) (cid:0) i T j ~ d T ) (cid:0) u i (~ (cid:27) iT ; (cid:27) (cid:0) i T j d T ) ]
(cid:12)(cid:12)(cid:12) H T } 3 ⃝(cid:20) E H T {
E ~ d T [ max ~ (cid:27) i ( T (cid:0) 1 ∑ t = ( u i (~ (cid:27) i ; (cid:27) (cid:0) i t j ~ d T ) (cid:0) u
i (~ (cid:27) i ; (cid:27) (cid:0) i t j d t )) )]
(cid:12)(cid:12)(cid:12) H T } + E H T {
E ~ d T ;d T [ u i (~ (cid:27) i T ; (cid:27) (cid:0) i T j ~ d T ) (cid:0) u i (~ (cid:27) i T ; (cid:27) (cid:0) i T j d T ) ]
(cid:12)(cid:12)(cid:12) H T } 4 ⃝ =
E H T (cid:0){
E ~ d T (cid:0) 1 ;d T (cid:0) [ T (cid:0) 1 ∑ t = ( u i (~ (cid:27) iT (cid:0) 1 ; (cid:27) (cid:0) i t j ~ d T (cid:0) 1 ) (cid:0) u i (~ (cid:27) iT (cid:0) 1 ; (cid:27) (cid:0) i t j d t )) ]
(cid:12)(cid:12)(cid:12) H T (cid:0)} + E H T { E ~ [Equation]
T ; [Equation] T [ u i (~ (cid:27) iT ; (cid:27) (cid:0) i T j ~ d T ) (cid:0) u i (~ (cid:27) iT ; (cid:27) (cid:0) i t j d T ) ] (cid:12)(cid:12)(cid:12) H T } 5 ⃝(cid:20) T
∑ t =
E H t {
E ~ d t ;d t [ u i (~ (cid:27) it ; (cid:27) (cid:0) i t j ~ [Equation] t ) (cid:0) u i (~ (cid:27) it ; (cid:27) (cid:0) i t [Equation]
t ) ] (cid:12)(cid:12)(cid:12) H t } 6 ⃝ =
T ∑ t =
E H t { E
[Equation] t ; [Equation] [ u i (~ (cid:27) it ; (cid:27) (cid:0) i t j ~
[Equation]
t ) (cid:0) u i (
[Equation]
(cid:27) it ; (cid:27) (cid:0) i t j d (cid:3) ) ] (cid:12)(cid:12)(cid:12)
[Equation]
+
[Equation]
∑ [Equation] E [Equation]
[Equation]
[Equation] [ u [Equation] it ;
[Equation] (cid:0) i t [Equation] (cid:3) ) (cid:0) u
[Equation]
(cid:27)
[Equation]
; (cid:27) (cid:0)
i
[Equation]
j
[Equation] [Equation]
H t } : Here equality 1 ⃝ holds since d (cid:3) and ~ d T are identical distributed conditioning on H t . Equation 2 ⃝ holds due to the deﬁnition of ~ (cid:27) i T .
Then we get 3 ⃝ with the maximum function.
Equality 4 ⃝ holds with the deﬁnition of ~ (cid:27) iT (cid:0) 1 .
Inequality 5 ⃝ holds by using the induction to episode t .
Finally we get equality 6 ⃝ by inserting u i (~ (cid:27) it ; (cid:27) (cid:0) i t j d (cid:3) ) .
Therefore, we ﬁnish the proof.
Then we only to give upper bounds for the above two terms.
Also mentioned in sec.3.1, we introduce some additional notations.
For episode t , we generate two trajectories by interacting with the environment.
More speciﬁcally, we use T i;t ( i 2 f 1 ; 2 g ) to denote the trajectory generated by ^ (cid:27) i;t with d (cid:3) .
We use E T i;t to denote the expectation over all trajectories for episode t .
Then we denote
T C i;t = f
h C 1 ;t ; h C 2 ;t ; :::; h C m i;t ;t g the trajectory for the chance player in episode t , and here m i;t denotes the length of T C i;t .
Furthermore, we denote the terminal node for episode t as z i;t .
Besides, we denote the collection of T 1 ; 1 ; T 2 ; 1 :::; T 1 ;t (cid:0) 1 ; T 2 ;t (cid:0) 1 as H t , which represents all the observations before episode t .
For each history h , we further use n t ( h ) to denote the count that h has been visited in H t .
Further, we use E T i;t to denoting the expectation over all possible trajectories T i;t .
Then we concentrate on the ﬁrst term T ∑
[Equation]
H t } and the second term has similar proof.
Since the strategy tuple is the same for the two utilities, we can decompose their difference with below lemma.
Lemma 4.
At episode t , the expectation of u i (~ (cid:27) it ; (cid:27) (cid:0) i t j ~ d t ) (cid:0) u i (~ (cid:27) it ; (cid:27) (cid:0) i t
j d (cid:3) ) can be upper bounded:
[Equation]
Proof.
From the root node to h C 1 ;t , players take actions according to (~ (cid:27) it ; (cid:27) (cid:0) i t ) .
Thus we should have
E H t {
E ~ d t ;d (cid:3) [ u i (~ (cid:27) it ; (cid:27) (cid:0) i t j ~ d t ) (cid:0) u i (~ (cid:27) it ; (cid:27) (cid:0) i t
j d (cid:3) ) ] (cid:12)(cid:12)(cid:12) H t } =
E H t {
E ~ d t ;d (cid:3) E T i;t [ u i ( h C 1 ;t j ~ (cid:27) it ; (cid:27) (cid:0) i t ; ~ d t ) (cid:0) u
i ( h C 1 ;t j ~ (cid:27) it ; (cid:27) (cid:0) i t ; d (cid:3) ) ]
(cid:12)(cid:12)(cid:12) H t } =
E
[Equation]
t 8< :
[Equation]
~ [Equation] (cid:3) E T i
[Equation] t  ∑ [Equation] (cid:11) ( h
[Equation]
1 ;t ) (~ c t ( h C 1 ;t ; a ) u i ( h C 1 ;t j ~ (cid:27) it ; (cid:27) (cid:0) i t ; ~ d t ) (cid:0) c (cid:3) ( h C 1 ;t ; a ) u i ( h C 1 ;t j ~ (cid:27) it ; (cid:27) (cid:0) i t ; d (cid:3) )) 5 (cid:12)(cid:12)(cid:12) H t 9= ; 1 ⃝ = E
[Equation]
8<
[Equation]
∑
[Equation]
(cid:11) (
[Equation]
[Equation]
[Equation]
t
[Equation]
a ) (cid:0) c (cid:3) ( h
[Table]
[Table]
t ; a ))
u i ( h
[Table]
[Table]
t j ~ (cid:27) it ; (cid:27) (cid:0) i t ; ~ d t ) 5 (cid:12)(cid:12)(cid:12) H t 9= ; + E H t 8< : E ~ d t ;d (cid:3) E T i;t  ∑ a 2 (cid:11) ( h C 1 ;t ) c (cid:3) ( h C 1 ;t ; a )( u i ( h C 1 ;t j ~ (cid:27) it ; (cid:27) (cid:0) i t ; ~ d t ) (cid:0) u
i ( h C 1 ;t j ~ (cid:27) it ; (cid:27) (cid:0) i t ; d (cid:3) ))
5 (cid:12)(cid:12)(cid:12) H t 9= ; 2 ⃝ =
E H t 8< : E ~ d t ;d (cid:3) E T i;t  ∑ a 2 (cid:11) ( h C 1 ;t ) (~ c t ( h C 1 ;t ; a ) (cid:0) c (cid:3) ( h C 1 ;t ; a ))
u i ( h C 1 ;t j ~ (cid:27) it ; (cid:27) (cid:0) i t ; ~ d t ) 5 (cid:12)(cid:12)(cid:12) H t 9= ; + E H t {
E ~ d t ;d (cid:3) E T i;t [ u i ( h C 2 ;t j ~ (cid:27) i t ; (cid:27) (cid:0) i t ; ~ d t ) (cid:0) u
i ( h C 2 ;t j ~ (cid:27) i t ; (cid:27) (cid:0) i t ; d (cid:3) ) ]
(cid:12)(cid:12)(cid:12) H t } 3 ⃝ =
E H t 8 < : E ~ d t ;d (cid:3) E T i;t 4 m i;t
∑ j = ∑ a 2 (cid:11) ( h C j;t ) (~ c t ( h C j;t ; a ) (cid:0) c (cid:3) ( h C j;t ; a ))
u i ( h C j;t
j ~ (cid:27) it ; (cid:27) (cid:0) i t ; ~ d t ) 5 (cid:12)(cid:12)(cid:12) H t 9 = ; + E H t {
E ~ d t ;d (cid:3) E T i;t [ u i ( z i;t j ~ r t ) (cid:0) u i ( z i;t j r (cid:3) ) ]
(cid:12)(cid:12)(cid:12) H t } :
Here equation 1 ⃝ holds by inserting a term c (cid:3) ( h C 1 ;t ; a ) u i ( h C 1 ;t j ~ (cid:27) it ; (cid:27) (cid:0) i t ; ~ d t ) ; equation 2 ⃝ holds since that h C 2 ;t is reached following the underlying transition c (cid:3) ( h C 1 ;t ; and equation 3 ⃝ holds by using induction.
Therefore we ﬁnish the proof.
We upper bound the term u
i ( z i;t j ~ r t ) (cid:0) u i ( z i;t j r (cid:3) )
ﬁrst.
We can refer to the technique of previous work in PSRL.
Recall that in episode t , players reaches terminal node
z i;t with a visited count n t ( z i;t ) .
We denote that (cid:22) u
it ( z i;t ) as the empirical mean of u i ( z i;t j r (cid:3) ) .
Simply we insert (cid:22) u
it ( z i;t ) to get
[Equation]
First we consider the second one j (cid:22) u it ( z i;t ) (cid:0) u i ( z i;t j r (cid:3) )
j
and we have similar bound for the ﬁrst term. Conditioning on r (cid:3) ( z i;t ) , we can apply the Chernoff-Hoeffding bound (Hoeffding, 1994).
For (cid:14) 2 (0 ; 1) Pr
[Equation]
where Pr denote the probability.
Then we use the above inequality to get below lemma:
Lemma 5.
At episode t , the expectation of j (cid:22) u it ( z i;t ) (cid:0) u i ( z i;t j r (cid:3) )
j can be bounded by
[Equation]
Proof.
Notice that Eq.
14 holds conditioning on r (cid:3) ( z i;t ) and the expectation is taken over the prior P 0 .
Then we need to carefully apply the Eq.
For the convenience of notation, we use (cid:25) t ( h j d (cid:3) ) to represent (cid:25) ~ (cid:27) it ;(cid:27) (cid:0) i t ( h j d (cid:3) ) .
We further use I ( (cid:1) ) to indicate the identical function.
Then we expand the expectation into integration
[Equation]
r (cid:3) )
j ] (cid:12)(cid:12)(cid:12) H t } 1 ⃝ =
∑ z 2 Z ∫ j (cid:22) u it ( z ) (cid:0) u i ( z j r (cid:3) ) j (cid:25) t ( z j d (cid:3) ) Pr ( d (cid:3) ; d t jH t ) Pr ( H t ) d ( d (cid:3) ; d t ; H t ) 2 ⃝(cid:20)
∑ z 2 Z ∫ √ ln(
[Equation]
=(cid:14) ) 2 max( n t ( z )
[Equation]
1) (cid:25) t ( z j d (cid:3) )
Pr (
[Equation]
(cid:3)
[Equation]
d t jH t ) Pr ( H t ) d
( d (cid:3)
[Equation]
d t ; H t ) + ∑ z
[Equation]
Z ∫
[Equation]
( j (cid:22)
[Equation]
it ( z ) (cid:0)
[Equation]
i ( z j r (cid:3) ) j (cid:21) √ ln(2 =(cid:14) ) 2 max( n t ( z )
[Equation]
Pr ( d (cid:3) jH t ) Pr ( H t ) d
( d (cid:3) ; H t ) (15) 3 ⃝
[Equation]
E H t { E ~
[Equation]
t
[Equation]
(cid:3) [√ 2 log(
[Equation]
(cid:14) ) max( n t ( z i
[Equation]
t )
[Equation]
(cid:12)(cid:12)(cid:12) H t } + ∑ z
[Equation]
Z ∫
[Equation]
I ( j (cid:22) u it ( z ) (cid:0) u i ( z j r (cid:3) )
j (cid:21) √ ln
[Equation]
(cid:14) ) 2 max( n t ( z ) ; 1) )
Pr ( H t j d (cid:3) ) {+P 
+}
( d (cid:3) ) d ( d (cid:3) ; H t )
[Equation]
Here we extend the expectation into integration in equation 1 ⃝ .
Inequality 2 ⃝ holds by separating the integral space.
Equation 3 ⃝ uses the Bayes rule and thus we can apply Eq.
(14) to get equation 4 ⃝ .
Therefore we ﬁnish the proof.
For another term j u i ( z i;t j ~ r t ) (cid:0) (cid:22) u it ( z i;t ) j , we can still apply the technique in Lemma 5 to get below lemma:
Lemma 6.
At episode t , the expectation of j u i ( z i;t j ~ r t ) (cid:0) (cid:22) u it ( z i;t )
j can be bounded by
[Equation]
H t
[Equation]
Proof.
We can directly prove that
[Equation]
The above proof is almost the same as that in Lemma 5 expect inequality (16).
Since d (cid:3) and ~ d t are identically distributed conditioning on H t , then we apply below equality to Eq.
[Equation]
Then we ﬁnish the proof.
Hence we combine the results in Lemma 5 and 6 and get the conclusion that for any (cid:14) 2 (0 ; 1) ,
E H t
[Equation]
Using a pigeon-hole principle and choosing (cid:14) =
1 =
( j Z j T ) , we have below lemma:
Lemma 7.
At episode T , the expectation of the summation of u i ( z i;t j ~ r t ) (cid:0) u i ( z i;t j r (cid:3) ) over the prior distribution P 0 has an order of:
[Equation]
Then we consider chance player node h C j;t .
We also denote (cid:22) c
( h C j;t ; a ) as the empirical mean of chance player’s probability to choose a at h C j;t .
Notice that the utility is bounded in [ (cid:0) 1 ; 1] .
We have ∑
[Equation] [Equation]
h C
[Equation]
t [Equation] Then conditioning on c (cid:3) ( h C j;t ; a ) , we use the concentration bound for L 1 norm (i.e. the deviation inequality (Weissman et al., 2003) to get that for (cid:14) 2 (0 ; 1) [Equation]
<
[Equation]
Similar to the analysis in r , we give below lemma:
Lemma 8.
At episode t , the expectation of j (cid:22) c ( h C j;t ; a ) (cid:0) c (cid:3) ( h C j;t ; a ) j can be bounded by ∑
[Equation]
Proof.
We use similar techniques to get E H t
[Equation]
=
∑
[Equation] ∑ [Equation] ) (cid:20)
∑ h 2 H C ∫ √ 2 ln(
A =(cid:14) ) max( n t ( h ) ; 1) (cid:25) t ( s j d (cid:3) )
Pr ( d (cid:3) ; d t jH t ) Pr ( H t ) d
( d (cid:3) ; d t ; H t )
+ ∑ h 2 H C ∫
I @
∑ a 2 (cid:11) ( h ) j (cid:22) c
( h; a ) (cid:0) c (cid:3) ( h; a ) j (cid:21) √ 2 ln(
A =(cid:14) ) max( n t ( h ) ; 1)
A Pr ( d (cid:3) jH t ) Pr ( H t ) d
( d (cid:3) ; H t ) (17) =
E H t {
E ~ d t ;d (cid:3) [√ 2 ln(2 A =(cid:14) ) max( n t ( h ) ; 1) ] (cid:12)(cid:12)(cid:12) H t } + ∑ h 2 H C ∫
I @
∑ a 2 (cid:11) ( h ) j (cid:22) c
( h; a ) (cid:0) c (cid:3) ( h; a ) j (cid:21) √ 2 ln(
A =(cid:14) ) max( n t ( h ) ; 1)
A Pr ( H t j d (cid:3) ) P
( d (cid:3) )
d ( d (cid:3) ; H t ) (cid:20) E H t {
{+E ~ d t ;d (cid:3) [√ 2 log(
+}
A =(cid:14) ) max( n t ( z i;t ) ; 1) ] (cid:12) (cid:12)(cid:12)
H t } + j H C j (cid:14):
The proof process above is similar to that of Lemma 5.
Once again, we get below lemma:
Lemma 9.
At episode t , the expectation of j ~ c ( h C j;t ; a ) (cid:0) (cid:22) c ( h C j;t ; a ) j can be bounded by ∑
[Equation]
Proof.
We use similar techniques to get E H t
[Equation] =
∑
[Equation]
∑
[Equation]
The proof process here is similar to that of Lemma 6.
Next, we use a pigeon-hole principle and choosing (cid:14) =
1 =
( j H C j T ) , we have below lemma:
Lemma 10.
At episode T , the expectation of the summation of j ~ c t ( h C j;t ; a ) (cid:0) c (cid:3) ( h C j;t ; a ) j over the prior distribution P 0 has an order of:
[Equation]
Therefore we use the conclusion in Lemma7 and 10 to get
T ∑
[Equation]
The similar proof can be applied to the second term to get the same upper bound by simply replacing ~ d t with d t :
T ∑
[Equation]
Sum the analysis together, we get to the conclusion that
[Equation]
E d (cid:3) ;d T [
[Equation]
iT (cid:12)(cid:12)(cid:12)
[Equation]