I NTRODUCING C OORDINATION IN C ONCURRENT R EINFORCEMENT L EARNING
A BSTRACT
Research on exploration in reinforcement learning has mostly focused on problems with a single agent interacting with an environment.
However many problems are better addressed by the concurrent reinforcement learning paradigm, where multiple agents operate in a common environment.
Recent work has tackled the challenge of exploration in this particular setting (Dimakopoulou & Van Roy, 2018;
Dimakopoulou et al., 2018).
Nonetheless, they do not completely leverage the characteristics of this framework and agents end up behaving independently from each other.
In this work we argue that coordination among concurrent agents is crucial for efficient exploration.
We introduce coordination in Thompson Sampling based methods by drawing correlated samples from an agent’s posterior.
We apply this idea to extend existing exploration schemes such as randomized least squares value iteration (RLSVI).
Empirical results on simple toy tasks emphasize the merits of our approach and call attention to coordination as a key objective for efficient exploration in concurrent reinforcement learning.
I NTRODUCTION
At the heart of reinforcement learning is the exploration-exploitation trade-off that describes an agent’s dilemma to balance maximizing its cumulative rewards and improving its knowledge of the environment.
While there have been a lot of attempts in recent years to improve practical exploration (Bellemare et al., 2016; Osband et al., 2016a; Pathak et al., 2017; Burda et al., 2019), existing work has mainly focused on fully sequential problems where a single agent interacts with an environment.
However, it is sometimes beneficial or even necessary to have multiple agents interacting concurrently with the environment.
For example, clinical trials are run in different phases, in each of them a group of patients are studied simultaneously, results are then collected to facilitate the design of the next phase (Perchet et al., 2015).
Likewise, in many commercial applications a company might interact with many customers at the same time before gathering their feedback to improve its strategy.
This setting corresponds to concurrent reinforcement learning (Silver et al., 2013; Pazis & Parr, 2016), a framework where multiple agents interact with an environment while sharing data to learn in parallel.
To learn efficiently within this paradigm agents should coordinate their exploratory effort which adds a layer of complexity to the already difficult exploration problem.
There has been some interest in designing exploration methods suited for concurrent learning (Dimakopoulou & Van Roy, 2018;
Dimakopoulou et al., 2018;
Amin et al., 2021) however none of them actually achieve coordinated exploration .
That is, agents do not leverage other agents acting at the same time to influence their decisions.
Instead, agents will usually behave independently from each other conditioned on the history.
In the end, these techniques fail to fully satisfy the diversity property
that
Dimakopoulou &
Van Roy (2018) deemed necessary for efficient exploration in concurrent reinforcement learning.
Our aim in this work is to highlight the importance of coordination in concurrent exploration.
As an introduction we provide a regret bound for Thompson Sampling based algorithms for concurrent learning such as seed sampling (Dimakopoulou & Van Roy, 2018).
Then, we propose a simple coordination mechanism to introduce cooperation between parallel agents.
This mechanism is first used to create a bandit algorithm for multi-armed bandits with Gaussian rewards.
We also use the same strategy to extend Randomized Least Square Value Iteration (RLSVI; Wen (2014);
Osband et al.
(2016b; 2018)).
We evaluate the empirical performance of these algorithms on tabular problems and show that it leads to improved performance over non-coordinated baselines.
Altogether, given the growing interest in concurrent reinforcement learning (Kalashnikov et al., 2018; Bodnar et al., 2020; Kalashnikov et al., 2021), our work demonstrates the relevance of coordination as a research direction to improve exploration in reinforcement learning.
P ROBLEM F ORMULATION
We consider a Markov decision process (MDP) M represented by a tuple ⟨S , A , P M , r M , H, ρ 1 ⟩ with S the state space,
A the finite set of actions, P M : S ×
A ← [0 , 1] S the transition probability distribution, r M : S × A (cid:55)→ [0 , 1] the reward function, H ∈ N ∗ the horizon and ρ 1 the initial state distribution.
We will write S = |S| and A = |A| for the number of states and actions.
Let G ∈ N ∗ be the number of agents that operate in parallel.
At the beginning of episode k , the g -th agent begins at state s gk, 1 sampled according to ρ 1 .
Then, at depth h each agent takes an action a gk,h , gets a reward r gkh ∼ r
( s gk,h , a gk,h ) and transits to a new state s gk,h +1 ∼ P ( s gk,h , a gk,h ) .
We assume that agents share information in real time.
We also write H kh the concatenated history for all agents up to episode k at depth h .
Given a policy π , we define the value function for each agent g at depth h to be the expected sum of rewards to be collected from the current state when following policy π :
[Equation]
We will also write V M,πH +1 ( s )
=
for all s ∈ S .
The Bellman Equation applied in MDP M states that:
[Equation]
We denote by M ∗ the true MDP with unknown rewards and transition R ∗ and P ∗ .
Let π ∗ be an optimal policy.
We define the regret incurred by an agent g following policy π up to time T by Regret
[Equation]
where ∆ g k denotes regret over the k th episode
[Equation]
We model the initial uncertainty of the environment with a prior distribution ϕ and assume that M ∗ is a sample from that prior.
We then define the Bayesian regret as:
[Equation]
The per-agent regret experienced by the population of agents is given by, for K =
⌊ T/H ⌋ :
[Equation]
g = ∆ gk (cid:3) .
H OW CAN A TEAM OF AGENTS EXPLORE EFFICIENTLY ?
In this section we take a closer look at exploration in the concurrent setting and the potential benefits of coordination.
do do 4: Compute gk max
V M gk ,π k,6: end
H do 8:
Act: a gk,h gk Observe r gkh
10: end 3. C ONCURRENT EXPLORATION
Posterior Sampling for Reinforcement Learning (PSRL, (Strens, 2000; Osband et al., 2013)) extends
Thompson Sampling (Thompson, 1933) to reinforcement learning and operates by sampling a statistically plausible model of the environment given a prior and data previously collected.
The agent then acts according to this model to gather more information and update its posterior.
A natural extension of PSRL is to draw multiple samples from the posterior – one for each concurrent agent – as done in Algorithm 1.
Dimakopoulou & Van Roy (2018)’s seed sampling algorithm is an instance of this idea, agents draw multiple independent samples from the posterior.
Our first contribution is to provide the following regret bound for Concurrent PSRL
Theorem 1.
The mean Bayes regret of Algorithm 1 is bounded by
[Equation]
Proof.
All proofs are in the appendix.
This result shows that the average regret of concurrent posterior sampling scales in O (1 / √ G ) and that the total regret of Algorithm 1 scales in O ( HS √ AGT ) .
Asymptotically it is as efficient to run concurrent posterior sampling with G agents for T timesteps as it is to run posterior sampling with a single agent for GT timesteps.
Similar results were found in the bandit setting by Contal et al.
and Kandasamy et al.
3. T HE ISSUE WITH INDEPENDENT SAMPLING
Concurrent Posterior Sampling requires that each sample is drawn from the posterior but it does not impose any constraint on their joint distribution.
This gives us some flexibility in the choice of this joint distribution.
Options include
Draw a single sample from the posterior that is then used for every agent.
Draw independent samples from the posterior.
• Draw correlated samples from the posterior.
Dimakopoulou & Van Roy (2018) showed that the first option is inefficient because agents do not work together to explore faster.
Instead, their algorithm follows the second option and draw independent samples from the posterior.
Though they argue it allows agents to diversify their exploratory effort, independence does not guarantee that agents will coordinate.
There remains some probability that some actions are repeated while other equally promising actions are ignored.
In the context of exploration, it would mean trying few state-action pairs several times as opposed to trying more the Gamification and state-action pairs fewer times.
The former provides less information about the environment and leads to sub-optimal exploration.
We propose to reach the favored behavior by adding correlation between samples from the posterior so that parallel agents can cooperate.
The main idea in this paper is to introduce coordination through the following principle.
Principle 1.
Let s be a state such that all actions in s are equally likely to be the optimal (from the exploration algorithm point-of-view) at episode k .
We define the probability distribution
[Equation]
Q s,k,h is the distribution of actions sampled by agents in s at episode k and timestep h .
Then a coordinated exploration algorithm should maximize the entropy of Q s,k,h .
In words, Principle 1 expresses that when all actions in a state are equally promising, agents should try all possible actions roughly the same number of times.
We will see in the section how this can help us design new coordinated exploration algorithms.
B ANDIT WITH G AUSSIAN REWARDS
To demonstrate the benefits of coordination we start with a simple multi-armed bandit problem with
A arms.
The reward function for each arm is normally distributed with a known variance σ 2 .
Agents are uncertain about the mean reward for each arm over which they share a common N ( µ 0 , σ 0 ) prior.
4. C OORDINATED T HOMPSON S AMPLING
To derive an implementation of Algorithm 1 that validates Principle 1 we must find a way to influence the joint distribution of posterior samples without affecting their marginals.
To do so we introduce the concept of a preference function .
Let f k : (cid:74) 1 , · · · , G (cid:75)
→ A be a preference function.
For each agent g , f k ( g ) encodes its affinity for a specific action at timestep k .
We design our algorithm so that when all possible actions are equally likely to be optimal under the posterior agent g will pull arm f k ( g ) .
By enforcing that the set { f k ( g ) } g ∈ (cid:74) 1 , ··· ,G (cid:75) covers the action space as well as possible we can validate Principle 1 and promote coordination between agents.
Algorithm 2 presents Coordinated Thompson
Sampling (CoordTS) an extension of Thompson Sampling with a coordination mechanism.
This mechanism relies on the relationship between sampling from a normal distribution and sampling from a standard normal distribution.
For any arm with posterior N ( µ k , σ 2 k )
[Equation]
Thompson Sampling proceeds by selecting the arm with the highest posterior sample.
When all arms have the same posterior distribution (i.e µ 1 = · · · = µ N and σ 1 = · · · = σ N ) this maximum is achieved by the arm with the highest sample ϵ .
We can create a preference for a specific arm by assigning the maximum of ϵ to this arm.
Dimakopoulou & Van Roy (2018)’s seed sampling algorithm also made use of Equation [-equation 
-]8 and they referred to ϵ as a seed , in the remainder of this paper we will choose instead the term standard normal vector .
The sampling procedure used to draw a preference function at each timestep is given by Algorithm 3.
The algorithm ensures that preferences are distributed almost equally among all actions.
The following lemma shows that if preferences are sampled according to Algorithm 3 then Principlewill be verified.
Lemma 1.
We define
[Equation]
P k and Q k correspond to the distribution of preferences and actions at timestep k.
Then, Algorithm 3 maximizes the entropy of P k .
Furthermore, when all arms have the same distribution under the posterior then the entropy of Q k is also maximized.
1: Input: Gaussian prior p : N ( µ 0 , σ 20 I n ) , G ∈ N + number of agents, A ∈ N + number of arms, f k : (cid:74) 1 , · · · , G (cid:75)
→ A preference functions 2: for k = 1 , · · · do 3: for agent g = 1 , · · · , G do 4: //
Sample a standard normal vector 5:
Sample ϵ g ∼ N (0 , I n ) 6: j ← arg max i ϵ g
[ i ] 7: m ← f k ( g ) 8:
Swap values by setting ϵ g [ m ] =
ϵ g
[ j ] and ϵ g [ j ] =
ϵ g
[ m ] 9: //
Compute posterior sample 10: ˜ θ kh = µ k + σ k ϵ g 11: end for 12:
Act: a gk = arg max a ∈A ˜ θ gk [ a ] , ∀ g 13:
Observe r ( a gk ) , ∀ g 14:
Update: p ← P (cid:16) · | p, a 1 k , r ( a 1 k ) , · · · , a Gk , r ( a Gk ) (cid:17) 15: end for do end
4. A NALYSIS
To better understand the behavior of CoordTS we study the simpler two armed bandit setting ( A = 2 ,
A = { a 1 , a 2 } ).
Let g be an agent with a preference for action a 1 . At each timestep, g will draw independently ϵ 1 , ϵ 2 ∼ N (0 , 1) then use ϵ = (max( ϵ 1 , ϵ 2 ) , min( ϵ 1 , ϵ 2 )) to compute a sample from the posterior.
Because of this reshuffling, ϵ is no longer Gaussian. The distribution of ϵ is made more precise by the following lemma.
Lemma 2.
Let X 1 , X 2 be two iid random variables distributed according to N (0 , 1) .
Let Z =
( X, Y ) with X = max( X 1 , X 2 ) and Y = min( X 1 , X 2 ) .
Then X and Y follow a skew normal distribution with scale +1 and − 1 and Z follows a half-normal distribution with support on { ( x, y ) ∈
R 2 | x ≥ y } .
[Figure] [Figure] Figure 1 shows the joint distribution and marginals of ϵ .
If instead g were to have a preference for action a 2 then its standard normal vector’s marginals would also follow a skew normal distribution with scale − 1 and +1 and its joint distribution would be the missing half of the truncated normal distribution in Figure 1b.
Altogether, while ϵ is not Gaussian when the preference of g is known, it is when the preference of g has been marginalized.
Algorithm 2 achieves coordination by enforcing a good coverage of the posterior, ensuring that when agents sample their standard normal vector they draw as many times from the two halves of the distribution {+separated by the line { ϵ 
+}
= ϵ 2 } .
The condition mentioned in Principle 1 may never be verified.
We may wonder how CoordTS behave outside of this rare event and if it still allows agents to coordinate.
To answer that question we can numerically estimate the probability of choosing each action.
Figure 2 shows the probability of selecting action a 1 for independent and coordinated sampling as a function of the posterior distribution of the two arms.
We observe that an agent with a preference for an action will select it more often than it would if agents were independent.
This shows that, even when actions do not have the exact same probability to be optimal, parallel agents can coordinate.
This highlights the relevance of the term preference to describe the agent’s affinity for a particular action.
We now consider the general case ( A ≥ 2 ).
Let e
= ( e 1 , · · · , e N ) be the canonical basis of the
Euclidean space R N .
For i ∈ (cid:74) 1 , · · · , N (cid:75) we define F i = { x ∈ R N |∀
j, x T e i ≥ x T e j } .
The distribution of the standard normal vector of an agent with a preference for action a i will be a truncated normal distribution with support on F i .
Algorithm 2 ensures that at each timestep standard normal vectors sampled are distributed almost equally among each F i .
This is made more precise by the following lemma which follows directly from Lemma
Lemma 3.
Consider a team of G agents and N k
( F i ) the number of standard normal vectors sampled at timestep k that belong to subspace F i .
Then, we have ∀ i, j ∈ (cid:74) 1 , ..., N (cid:75) ,
| N k
( F i ) − N k
( F j ) | ≤ 1 .
Because ( F i ) i ∈ (cid:74) 1 , ··· ,N (cid:75) define a partition of R N , standard normal vectors sampled by Algorithm 4 are normally distributed when agents’ preferences have been marginalized.
4. E MPIRICAL EVALUATION
We now evaluate CoordTS empirical performance on our Gaussian bandit problem.
As a baseline, we compare with Upper Confidence Bounds (UCB) and Thompson Sampling with independent agents.
We use the following parameters for the problem: µ 0 = 0 , σ 0 = 300 , σ = 10 , A = 500 .
Figures 3a and 3b show the mean regret as a function of the number of concurrent agents after 50 episodes, results are averaged over 500 runs.
We observe a clear improvement using correlated sampling over independent sampling.
Both algorithms perform similarly when the number of agents is small however the gap widens as the number of agents grows.
With 10 agents CoordTS provides a 0 .
5% improvement over independent Thompson Sampling, this gap increases to almost 40% withagents.
As a whole, our experiments underline the potential of coordinated exploration.
C OORDINATION WITH LINEAR FUNCTION APPROXIMATION
The previous section provided a simple example where coordination between agent can be beneficial.
In this section we extend those ideas and focus on the more general reinforcement learning problem [Figure] [Figure] with linear function approximation.
We assume that we are provided a state representation ϕ h :
S ×
A → R d .
We write Φ h ∈ R SA ×
d the feature matrix Φ h :=
[ ϕ h ( s, a )] ( s,a ) ∈S×A .
We assume that the Q-function can be approximated by a linear function with weight vector θ h ∈ R d such that
[Equation]
Randomized Least-Squares Value Iteration (RLSVI, Wen, 2014;
Osband et al., 2016b) has been introduced as a computationally tractable way to approximate PSRL with linear function approximation.
RLSVI uses random perturbations to approximate a posterior over value functions.
RLSVI extends naturally to the concurrent setting by drawing multiple independent samples.
5. C OORDINATED RLSVI
To add a coordination mechanism to RLSVI we start by generalizing the concept of preference previously introduced.
Let f k,h : (cid:74) 1 , · · · , G (cid:75) × S
→ A be a preference function, f k,h ( g, s ) encodes the preference of agent g at state s , episode k and timestep h .
We also need the generalization of
Equation 8, for µ ∈ R d , Σ ∈ R d × d
[Equation]
where D ∈ R d × d is the square root of Σ , i.e Σ = DD ⊤ .
Following Principle 1, we wish to coordinate agents when all actions in a state are equally promising.
At state s , RLSVI acts according the greedy action arg max α ∈A (cid:0) Φ˜ θ (cid:1) ( s, α ) where ˜ θ is a sample from the posterior.
All actions in s are equally likely to be selected if we have
[Equation]
with equality in distribution and θ h the posterior distribution over value functions learned by RLSVI.
The computation of a standard normal vector is made more precise by the following lemma
Lemma 4.
Let g be an agent in state s at timestep k .
Let N (¯ θ, Σ) be the posterior distribution over value functions for the current timestep, ϵ ∼ N (0 , I d ) and ˜ θ
= ¯ θ + Dϵ , where D is the square root of Σ .
If all actions in s are equally likely to be the greedy action chosen by RLSVI we have, with a = f k
[Equation] g, s
[Equation]
[Equation]
where ψ s,α = D ⊤ ϕ s,α ∥ D ⊤ ϕ s,α ∥ 2 .
If ψ s,a is different from every other ψ s,α , the greedy action a is unique.
This lemma shows that if an agent has a preference for an action a its standard normal vector will be distributed according to a truncated Gaussian with support on F s,a = { x ∈ R d |∀ α ̸
= a, ψ ⊤ s,a x ≥ ψ ⊤ s,α x } .
This result generalizes CoordTS’ sampling procedure to the linear function approximation setting.
Because { F s,a } a ∈ A define a partition of R d , we recover the fact that when agents’ preferences are marginalized standard normal vectors are normally distributed. [Figure]
We are now ready to present a new coordinated algorithm that extends RLSVI.
This algorithm that we call Coordinated RLSVI (CoordRLSVI) is made up of two parts.
The first part describes the update of the posterior distribution and is presented in the appendix as Algorithm 4.
It assumes a prior N (0 , λI ) over weights and injects Gaussian noise with variance σ 2 .
The only difference with the original RLSVI algorithm is that the algorithm returns the mean and covariance matrix of the posterior instead of samples.
To obtain a complete exploration algorithm, action selection must be done according to the procedure specified by Algorithm 5.
Algorithm 5 adds two steps to RLSVI’s greedy action selection.
In the first step the algorithm samples a preference for each agent at its current state.
To keep track of preferences assigned during the episode a buffer B is created.
Two states within a certain distance are considered similar and share the same entry in the buffer to monitor preferences.
This distance is defined using a metric ∥ · ∥ and a radius r > 0 .
In the second step a standard normal vector is drawn for each agent at its current state.
5. E MPIRICAL EVALUATION
In this section we present empirical results that support the effectiveness of CoordRLSVI.
Our testing environment is a full binary tree of height H as displayed by Figure 5.
Each agent starts at the root s 0 then they can transition left or right at every step.
All states have zero reward except for the leaves.
At each leaf c the agent chooses a last action to observe a gaussian reward with unknown mean µ c and known variance ν 2 after which the episode ends.
Agents share a common prior N ( µ 0 , Σ 0 ) over rewards mean.
In this environment coordination can be useful as agents can cooperate in order to reach individual leaves faster.
We compare RLSVI and CoordRLSVI on this environment with a one-hot encoding for features and the following hyperparameters λ = 50 , σ = 25 , H = 6 , ν = 25 , Σ
= ν 2 I, µ 0 = 0 .
Results are displayed in Figures 4 and 4a, each data point was averaged over 500 seeds.
The two algorithms perform similarly when the number of agents is low.
As the number of agents increases, CoordRLSVI average regret decreases faster, with 60 agents, CoordRLSVI mean regret is 43% smaller than RLSVI’s.
By coordinating agents, CoordRLSVI is faster to achieve a lower regret, highlighting the benefit of coordination for linear function approximation.
D ISCUSSION
Concurrent reinforcement learning presents an enticing offer; trading time with computational cost by increasing the number concurrent of agents.
However, the increased computational requirements can become restrictive.
Coordination can provide a solution to this problem by maximizing the efficiency of concurrent reinforcement learning algorithms.
In this work we presented the first algorithms that achieve coordination in reinforcement learning by drawing correlated samples from a posterior distribution.
While these algorithms performed better than existing ones, much work remains to be done to capitalize on coordinated exploration.
First, it is unclear how we can mathematically model coordinated algorithms to better understand their behavior and derive sharper regret bounds.
Then, moving beyond linear function approximation will be an important hurdle, yet it is where coordination will be most valuable.
As concurrent reinforcement is becoming more sought after, it is in complex applications with intractable state and action spaces that it will shine the most.
A PPENDIX B P ROOFS
Theorem 1.
The mean Bayes regret of Algorithm 1 is bounded by
[Equation]
Proof.
To simplify notations, we write:
[Equation]
We have:
[Equation]
The second equality is true since M k,g and M ∗ are identically distributed given the history
H k 1 .
Hence the average Bayesian regret can be written as:
[Equation]
This new term can then decomposed as, with x gk,h :=
( s gkh , a gkh
[Equation]
Let ¯ r k
and ¯ P k be the empirical reward and transition at timestep k .
Wede fine the confidence set M k for episode k :
[Equation]
Where β 1 k := (cid:113) 7log(2 SAt k /δ )
2max { 1 ,N tk ( s,a ) } and β 2 := (cid:113) 14 S log(
At k /δ ) max { 1 ,N tk ( s,a ) } are chosen so that M ∗ and M k,g belong to M k with high probability and t k = ( k − 1) H +
Lemma 5.
Lemma 17 from Auer et al.
Let δ ∈ (0 , 1) , β 1 k , β 2 chosen as previously, then we have
[Equation]
[Equation]
Because M ∗ and M k,g are identically distributed we have 1 { M k,g / ∈ M k } = 1 { M ∗ / ∈ M k }
=
P ( M
[Equation]
/ ∈
M k ) .
By choosing δ = 1 K we have
[Equation]
For the other term we have
[Equation]
where ρ gk,h ( s ) is the probability to reach state s at depth h following policy π gk .
We have
[Equation]
Now, what remains is bounding the term
[Equation]
We have
[Equation]
Which gives the desired result
[Equation]
Lemma 1.
We define
[Equation]
P k and Q k correspond to the distribution of preferences and actions at timestep k.
Then, Algorithm 3 maximizes the entropy of P k .
Furthermore, when all arms have the same distribution under the posterior then the entropy of Q k is also maximized.
Proof.
Let C
=
(cid:110) p ∈ [0 , 1]
A | (cid:80) i p i
= 1 , ∀ i, p i ∈ [0 , 1 G , · · · , G − 1 G , 1] (cid:111) , C is the space of discrete probability distribution with values in { 0 , 1 G , · · · , G − 1 G , 1 } .
Let H be the entropy function, for any distribution p , H ( p ) =
− (cid:80)
i p
i log p
i
We define ν =
⌊ GA ⌋ and D = (cid:110) p ∈
C|∀ i, p i = νG or p i =
ν +1 G (cid:111) .
It is easy to see that at each timestep the distribution of preferences sampled by Algorithm 3 is an element of D .
It is also clear that the entropy is a constant over D .
We want to show that this constant is the maximum that can be attained for probability distributions in
C .
Let p 1 , p 2 ∈ [0 , 1]
A be two probability distributions, we define the distance ∥·∥ ∞ by ∥ p 1 − p 2 ∥ ∞ = max i | p 1 i − p 2 i | .
We define also q = [ 1 A , · · · , 1 A ] the uniform distribution over [0 , 1] A .
The set of distributions of C that minimizes the distance to the uniform distribution with the distance ∥ · ∥ ∞ is D
[Equation]
If it was not the case, there would be a distribution p with p i >
ν +1 G or p i < νG which minimizes this distance.
This is not possible because we would have ∥ p − q ∥ ∞ > 1 G and the distance between probability distributions in D and p is less than 1 G .
The solution is exactly D because all probability distributions in D have the same distance to the uniform distribution.
For any distribution p ∈ C we have
[Equation]
Because the uniform distribution maximizes the entropy over all possible distributions in [0 , 1] A , the distribution p with maximal entropy in C minimizes the KL divergence KL ( p || q ) .
We can conclude using Pinsker’s inequality by noting that arg min p ∈C ∥ p − q ∥ ∞ = arg min p ∈C KL ( p || q ) .
When all arms have the same distribution under the posterior we have Q k = P k and the entropy of
Q k is also maximized.
Lemma 3.
Consider a team of G agents and N k
( F i ) the number of standard normal vectors sampled at timestep k that belong to subspace F i .
Then, we have ∀ i, j ∈ (cid:74) 1 , ..., N (cid:75) ,
| N k
( F i ) − N k
( F j ) | ≤ 1 .
Proof.
This lemma is a direct consequence of the previous lemma.
Each preference f k ( g )
=
a i leads to a standard normal vector ϵ ∈ F i .
We have (cid:80) g 1 { f k ( g )= a } =
⌊ GA ⌋ or (cid:80) g 1 { f k ( g )= a } =
⌊ GA ⌋ + 1 , so for i ∈ (cid:74) 1 , · · · , N (cid:75) , N k ( F i ) =
⌊ G A ⌋ or ⌊ G A ⌋ + 1 which gives the desired result.
Lemma 2.
Let X 1 , X 2 be two iid random variables distributed according to N (0 , 1) .
Let Z =
( X, Y ) with X = max( X 1 , X 2 ) and Y = min( X 1 , X 2 ) .
Then X and Y follow a skew normal distribution with scale +1 and − 1 and Z follows a half-normal distribution with support on { ( x, y ) ∈
R 2 | x ≥ y } .
Proof.
Let f and F denote the PDF and CDF of N (0 , 1) .
For x ∈ R
F X
[Equation]
P
[Equation]
and X
[Equation]
[Equation]
The density function of X is now given by
[Equation]
Which is the density function of a skew normal distribution with shape 1 .
Similarly for y ∈ R we have
[Equation]
The density function of Y is now given by
[Equation]
Because we have 1 − F ( y ) =
F ( − y ) .
This density function corresponds to a skew normal distribution with shape − 1 .
Finally with support on ( x, y ) ∈ { ( x, y ) ∈
R 2 | x ≥ y }
[Equation]
the CDF joint distribution is given by
[Equation]
And the density function is given by
[Equation]
Which concludes the proof.
Lemma 4.
Let g be an agent in state s at timestep k .
Let N (¯ θ, Σ) be the posterior distribution over value functions for the current timestep, ϵ ∼ N (0 , I d ) and ˜ θ
= ¯ θ + Dϵ , where D is the square root of Σ .
If all actions in s are equally likely to be the greedy action chosen by RLSVI we have, with a = f k
( g, s ) :
[Equation]
) where ψ s,α = D ⊤ ϕ s,α ∥ D ⊤ ϕ s,α ∥ 2 .
If ψ s,a is different from every other ψ s,α , the greedy action a is unique.
Proof.
For any action a j ∈ A , actions a i and a j having the same value under the posterior means that
[Equation]
with equality in distribution.
Taking the expectation and variance in the last equality gives
[Equation]
The second equation can be simplified as
[Equation]
We get
[Equation]
Hence we have
[Equation]
ϵ
[Equation]