C ONTEXTUAL M ULTI -A RMED B ANDITS WITH C OMMUNICATION C ONSTRAINTS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider a remote Contextual Multi-Armed Bandit (CMAB) problem, in which the decision-maker observes the context and the reward, but must communicate the actions to be taken by the agents over a rate-limited communication channel.
This can model, for example, a personalized ad placement application, where the content owner observes the individual visitors to its website, and hence has the context information, but must convey the ads that must be shown to each visitor to a separate entity that manages the marketing content.
In this Rate-Constrained
CMAB (RC-CMAB) problem, the constraint on the communication rate between the decision-maker and the agents imposes a trade-off between the number of bits sent per agent and the acquired average reward.
We are particularly interested in the scenario in which the number of agents and the number of possible actions are large, while the communication budget is limited.
Consequently, it can be considered as a policy compression problem, where the distortion metric is induced by the learning objectives.
We ﬁrst consider the fundamental information theoretic limits of this problem by letting the number of agents go to inﬁnity, and study the regret that can be achieved.
In particular, we identify two distinct rate regions resulting in linear and sub-linear regret behaviour, respectively.
Then, we propose a practical coding scheme, and provide numerical results for the achieved regret.
I NTRODUCTION
In the last few years, synergies between Machine Learning (ML) and communication networks have attracted a lot of interest in the research community, thanks to the fruitful interplay of the two ﬁelds in emerging applications from Internet of things (IoT) to autonomous vehicles and other edge services.
In most of these applications, both the generated data and the processing power is distributed across a network of physically distant devices, thus a reliable communication infrastructure is pivotal to run ML algorithms that can leverage the collected distributed knowledge (Park et al., 2019).
To this end, a lot of recent works have tried to redesign networks and to efﬁciently represent information to support distributed ML applications, where the activities of data collection, processing, learning and inference are performed in different geographical locations, and should consider limited communication, memory, or processing resources, as well as addressing privacy issues.
In contrast to the insatiable growth in our desire to gather more data and intelligence, available communication resources (bandwidth and power, in particular) are highly limited, and must be shared among many different devices and applications.
This requires the design of highly communicationefﬁcient distributed learning algorithms , particularly for edge applications.
Information theory, in particular the rate-distortion theory, have laid the fundamental limits of efﬁcient data compression with the aim to reconstructing the source signal with the highest ﬁdelity (Cover & Thomas, 2006b).
However, in the aforementioned applications, the goal is often not to reconstruct the source signal, but to make some inference based on that.
This requires task-oriented compression , ﬁltering out the unnecessary information for the target application, and thus decreasing the number of bits that have to be transmitted over the communication networks.
This approach should target the questions of what is the most useful information that has to be sent, and how to represent it, in order to meet the application requirements consuming the minimum amount of network resources.
Our goal in this paper is to investigate a theoretically grounded method to efﬁciently transmit data in a Contextual Multi-Armed Bandit (CMAB) problem, in which the context information is available to a decision-maker, whereas the actions can be taken by a remote entity, called controller , controlling a multitude of agents.
We assume that a limited communication link is available between the decisionmaker and the controller to communicate at each round the intended actions.
The controller must decide on the actions to take based on the message received over the channel, while the decisionmaker observes the rewards at each round, and updates its policy accordingly.
This scenario can model, for example, a personalized ad placement application, where the content owner observes the individual visitors to its website; and hence, has the context information, but must convey the ads that must be shown to each visitor to a separate entity that manages the marketing content.
This will require communicating hundreds or thousands of adds to be placed at each round, from among a large set of possible adds, within the communication resource and delay constraints of the underlying communication channel, which is quantiﬁed as the number of bits available per agent.
This problem may arise in other similar applications of CMABs with communication constraints between the decision-maker and the controller (Bouneffouf & Rish, 2019).
1. R ELATED W ORK
Given the amount of data that is generated by machines, sensors and mobile devices, the design of distributed learning algorithms is a hot topic in the ML literature.
These algorithms often impose communication constraints among agents, requiring the design of methods that would allow efﬁcient representation of messages to be exchanged.
While rate-distortion theory deals with efﬁcient lossy transmission of signals (Cover & Thomas, 2006b), in ML applications, we typically do not need to reconstruct the underlying signal, but make some inference based on that.
These applications can be modeled through distributed hypothesis testing (Berger, Sep. 1979; Ahlswede & Csisz´ar, 1986) and estimation (Zhang et al., 2013; Xu & Raginsky, 2017) problems under rate constraints.
In parallel to the theoretical rate-distortion analysis, signiﬁcant research efforts have been invested in the design of practical data compression algorithms, focusing on speciﬁc information sources, such as JPEG and BPG for image compression, or MPEG and H.264 for video compression.
While adapting these tools to speciﬁc inference tasks is difﬁcult, recently deep learning techniques have been employed to learn task-speciﬁc compression algorithms (Torfason et al., 2018; Jankowski et al., 2021), which achieve signiﬁcant efﬁciency by bypassing image reconstruction.
While the above mainly focus on the inference task through supervised learning, here we consider the CMAB problem.
There is a growing literature on multi-agent Reinforcement Learning (RL) problems with communication links (Foerster et al., 2016; Sukhbaatar et al., 2016; Havrylov & Titov, 2017; Lazaridou et al., 2017).
These papers consider a multi-agent partially observable
Markov decision process (POMDP), where the agents collaborate to resolve a speciﬁc task.
In addition to the usual reward signals, agents can also beneﬁt from the available communication links to better cooperate and coordinate their actions.
It is shown that communication can help overcome the inherent non-stationarity of the multi-agent environment.
Our problem can be considered as a special case of this general RL formulation, where the state at each time is independent of the past states and actions.
Moreover, we focus on a particular setting in which the communication is one-way, from the decision-maker that observes the state and the reward, towards the controller that takes the actions.
This formulation is different from the existing results in the literature involving multi-agent Multi-Armed Bandit (MAB).
In Agarwal et al.
(2021), each agent can pull an arm and communicate with others.
They do not consider the contextual case, and focus on a particular communication scheme, where each agent shares the index of the best arm according to its experience.
Another related formulation is proposed in Hanna et al.
(2021), where a pool of agents collaborate to solve a common MAB problem with a rate-constrained communication channel from the agents to the server.
In this case, agents observe their rewards and upload them to the server, which in turn updates the policy used to instruct them.
In Park & Faradonbeh (2021), the authors consider a partially observable CMAB scenario, where the agent has only partial information about the context.
However, this paper does not consider any communication constraint, and the partial/ noisy view of the context is generated by nature.
Differently from the existing literature, our goal here is to identify the fundamental information theoretic limits of learning with communication constraints in this particular scenario.
P ROBLEM F ORMULATION 2. C ONTEXTUAL M ULTI -A RMED B ANDIT (CMAB) P ROBLEM
We consider N agents, which experience independent realizations of the same CMAB problem.
The
CMAB is a sequential decision game in which the environment imposes a probability distribution
P S over a set of contexts, or states, S , which is ﬁnite in our case.
The game proceeds in rounds, and at each round t = 1 , . .
, T , a realization of the state s t,i ∈ S is sampled from the distribution P S for each agent i ∈ { 1 , . . .
, N } .
At each time step t , and for each agent i , states are sampled iid according to P S .
In the usual CMAB setting, the decision-maker would observe the states { s t,i } Ni =1 of the agents, and choose an action (or arm) a t,i ∈ { 1 , . . .
, K }
=
A , for each agent, where K is the total number of available actions, with probability π t,i ( a t,i | s t,i ) given by a (possibly) stochastic policy π t : S → ∆ K .
Here ∆ K is the K-dimensional simplex, containing all possible distributions over the set of actions.
Once the actions have been taken by all the agents, at each round t the environment returns rewards for all the agents following independent realizations of the same reward process, r t,i =
r ( s t,i , a t,i ) ∼ P R ( r | s t,i , a t,i ) , ∀ i ∈ { 1 , . . .
, N } , which depends on the state and the action of the corresponding agent.
Based on the history of returns up to round t − 1 , the decision-maker can optimize its policy to maximize the sum of the rewards G = (cid:80) Tt =1 (cid:80) Ni =1 r t,i .
2. R ATE -C ONSTRAINED CMAB
In our modiﬁed version, the process of observing the system states is spatially separated from the process of taking the actions.
The environment states, { s t,i } Ni =1 , are observed by a central entity, called the decision-maker, that has to communicate to the controller over a rateconstrained communication channel, at each round t , the information about the actions { a t,i } Ni = the agents should take.
The decision-maker can exploit the knowledge accumulated from the states and rewards observed by all the agents up to round t − 1 , denoted by H ( t − 1) =
(cid:110) ( { s 1 ,i } Ni =1 , { a 1 ,i } Ni =1 , { r 1 ,i } Ni =1 ) , . . .
, ( { s t − 1 ,i } Ni =1 , { a t − 1 ,i } Ni =1 , { r t − 1 ,i } Ni =1 ) , (cid:111) ∈ H ( t − 1) , to optimize the policy to be used at round t .
Consequently, the problem is to communicate the action distribution, i.e., the policy π t ( a | s t ) , which depends on the speciﬁc state realizations observed in round t , to the controller within the available communication resources while inciting the minimal impact on the performance of the learning algorithm.
Speciﬁcally, the decision-maker employs function f ( N ) t : H ( t − 1) ×S N → { 1 , 2 , . . .
, B } to map the history up to time t and the states of the agents at time t to a message index to be transmitted over the channel.
The controller, on the other hand, employs a function g ( N ) t : { 1 , 2 , . . .
, B } → A N to map the received message to a set of actions for the agents.
In general, both functions f ( N ) t and g ( N ) t can be stochastic.
Average per period regret achieved by sequences (cid:110)
f ( N ) t , g ( N ) t (cid:111)
T t
=1 is given by ρ ( N ) ( T )
= 1 T E (cid:34) T (cid:88) t =
N (cid:88) i =
r ( s t,i , a [Equation] (1) where a t,i =
g t,i ( m ( t )) is the action taken by agent i based on message m ( t ) =
f ( N ) t (cid:16) H ( t − 1) , { s t,i } Ni =1 (cid:17) transmitted in round t , a ∗ ( s t,i ) is the action with maximum mean reward in state s t,i , i.e., the optimal action, and the expectation is taken with respect to the state distribution P S and reward distribution P R .
We say that, for a given time period T and N agents, an average regret- communication rate pair ( ρ, R ) is achievable if there exist functions (cid:110)
f ( N ) t , g ( N ) t (cid:111)
T t =1 as deﬁned above with rate 1 N log 2 B ≤ R and regret ρ ( N ) ( T ) ≤ ρ .
If a sufﬁciently large rate is available for communication, i.e., R ≥ log K , then the intended action for each agent can be reliably conveyed to the controller.
Otherwise, to achieve the learning goal while satisfying the rate constraint, the decision-maker must apply a lossy compression scheme, such that the action distribution adopted by the pool of agents resembles the intended policy as much as possible.
S OLUTION
In this section, we present the proposed solution to the problem.
First of all, we brieﬂy discuss the algorithm adopted by the decision-maker to solve the CMAB.
Then, the communication scheme for the RC-CMAB with a particular distortion function is provided, which is inherited by the learning task.
We then identify two distinct rate regions resulting in linear and sub-linear regret, and propose a more practical coding scheme.
3. T HOMPSON S AMPLING
In the proposed solution to the CMAB problem, the decision-maker adopts Thompson Sampling (TS) Thompson (1935).
In particular, it makes use of one TS instance for each state s ∈ S .
Consequently, the decision-maker maintains an estimate of the distribution p s,at ( µ ) of the mean reward µ s,a ∈ D ⊆ R of action a in state s at time t .
To take the decision in state s t , the decisionmaker would sample µ s t ,a t ∼ p s t ,a t , ∀ a ∈ A , and takes the action a ∗ = arg max a ∈A { µ s t ,a t } .
This procedure is repeated for each agent i ∈ { 1 , . . .
, N } .
After receiving the rewards { r t,i } Ni =1 , the decision-maker can update its belief on µ s,a , i.e., the probabilities p s,at ( µ ) , in order to minimize the regret.
We notice that this strategy induces a probability distribution π t ( a | s ) over the actions that is π t ( a | s ) =
(cid:82) D p s,at ( µ ) (cid:81) Kj =1 ,j (cid:54) =
a P s,jt ( µ ) dµ, where P s,jt ( µ ) is the Cumulative Distribution Function (CDF) of µ j , where the random variables µ s,a are considered independently distributed.
However, the constraint on the rate imposed by the RC-CMAB formulation makes it infeasible for the decision-maker to sample, through the pool of agents, the actions directly from the true distribution π t ( a | s ) .
The agents have to use a proxy Q t ( a | s ) , which is the one obtained from the message received.
This problem is similar to approximate TS, where a proxy distribution is used to sample the actions, or the reward means, given that the true distribution is too complex to sample from.
In that case, the bottleneck is given by the complexity of the mean reward distributions, whereas in this work, it is imposed by the limited-rate communication channel between the decision-maker and the controller.
3. O PTIMAL S OLUTION FOR THE RC-CMAB
We model the environment as a Discrete Memoryless Source (DMS), that generates states from a ﬁnite alphabet S with probability P S , emitting sequences of N symbols s N =
( s 1 , . . .
, s N ) , one per agent.
We then denote with ˆ Q s N ( m ) the empirical probability of state m in s N .
We also consider the sequence of actions a N , and denote with ˆ Q z N ( m, j ) the empirical joint probability of the pair ( m, j ) in z N =
(( s 1 , a 1 ) , . .
, ( s N , a N )) .
The whole picture can be seen in Fig.
In the ﬁgure above, the actions taken by the agents are denoted by ˆ a to indicate that they can differ from a .
However, we are interested in the probability distributions generating the sequences, thus we will denote with A the random variables indicating both the actions at the controller and decision-maker side.
We assume that the distribution P S is known (or accurately estimated).
The decision-maker can observe the realization s N of the contexts, and its task is to transmit an index u ∈ { 1 , . . .
, B } , and the agents can generate from u a sequence a N , such that ˆ Q s N a N is close to P SA ( sa ) =
P S ( s ) π ( a | s ) , where closeness depends on a distortion measure E
[ d ( ˆ Q S N A N , P SA )] , which in general is not an average of a per-letter distortion measure.
The problem is a compression task in which the server has complete (or partial) knowledge of the states s N , and wants to transmit a conditional probability distribution π A | S to the agents, consuming the minimum amount of bits, in such a way that the empirical distributions ˆ Q s N a N given by the sequence induced by the agents is close to the joint distribution P SA induced by the policy.
For a distortion function d ( Q SA , P SA ) that is 1) nonnegative, 2) upper bounded by a constant D max , 3) continuous in Q SA , and 4) convex in Q SA , in Kramer & Savari (2007), the authors provide the rate-distortion function R ( D ) , i.e., the minimum rate R = log 2 B N bits per symbol such that E [ d ( ˆ Q S N A N , P SA )] ≤ D , in the limit when N is arbitrarily large.
The solution is given by
[Equation]
where Q SA = P S Q A | S is the joint probability induced by the environment distribution P S and policy Q A | S , which depends on the information sent by the decision-maker.
As we can see, in the asymptotic limit of N agents, the problem admits a single-letter solution, which also serves as a lower bound on the ﬁnite agent scenario.
The studied RC-CMAB model described in Sec. 2.2 ﬁts into this framework, and in the following section we identify an appropriate distortion function and provide its characterization.
3. T HE KL-D IVERGENCE AS D ISTORTION F UNCTION
When applying TS to the RC-CMAB problem with limited communication rate, the decision-maker may not be able to induce the controller to take samples from the true policy π s,at .
In Phan et al.
(2019), the authors provide some theoretical guidelines to construct approximate sampling policies to make the posteriors, i.e., p s,at , concentrate achieving sub-linear regret.
In particular, they studied the case in which the sampling distribution Q differs from the target posterior π , using D α ( π, Q ) as distortion measure, which denotes the α -divergence between the two, and is deﬁned as
[Equation]
In Phan et al.
(2019), it is shown in Theorem 1 that, for α > 0 , the condition D α ( π, Q ) <
δ with δ > 0 , cannot guarantee that the posterior π converges in sub-linear time, even for very small values of δ .
On the contrary, for α ≤ 0 , the authors provide a scheme that can guarantee sub-linear regret.
In particular, they suggest to introduce an exploration term ρ t ∈ o (1) such that (cid:80) ∞ t =1 ρ t = ∞ , and the actions are sampled from Q t with probability 1 − ρ t , and uniformly at random with probability ρ t , while D α ( π t , Q t ) <
δ for all t .
In this case, it is possible to obtain a sub-linear regret.
Consequently, in order to solve the proposed RC-CMAB problem, we decide to investigate this strategy with α = 0 , which leads to the reverse KL-divergence from Q to π , or equivalently, the KL-divergence from π to Q , deﬁned as D KL ( Q, π )
= (cid:80) x ∈X Q ( x ) log Q ( x ) π ( x ) , when x takes values in the discrete set X .
Consequently, to ﬁnd the optimal constrained policy Q t ( a | s ) , we need to ﬁnd a solution to Eq.
(2), which is a rate-distortion optimization problem, when the distortion function is given by the reverse
KL-divergence.
We can rewrite the optimization objective of Eq.
(2) as a double minimization problem (Sec. 10.8, (Cover & Thomas, 2006a))
[Equation]
Following (Lemma 10.8.1, (Cover & Thomas, 2006a)), the marginal Q ∗ ( y ) =
(cid:80) x P ( x ) Q ( y | x ) has the property
[Equation]
that is, it minimizes the KL-divergence between the joint and the product P ( x ) Q ( y ) ,
[Equation]
This means that ˜ Q ( a ) obtained by solving Eq.
(4) is indeed the marginal over the actions induced by
Q ( a | s ) .
Exploiting this formulation, it is possible to apply the iterative Blahut-Arimoto algorithm to solve the problem and ﬁnd the solution.
In particular, given that the two sets ∆ K and ∆ S are made of probability distributions, and the target measure is the KL-divergence, the algorithm does converge to the minimum (Csisz´ar & Tusn´ady, 1984).
The process is initialized by setting a random ˜
Q 0 ( a ) , which is used as a ﬁxed point to compute
[Equation]
From Q ∗ 1 ( a | s ) , we compute the optimal Q ∗ 1 ( a ) by solving Eq.
(5), which is simply the marginal
Q ∗ 1 ( a ) =
(cid:80) s P ( s ) Q ∗ 1 ( a | s ) .
The process is iterated until convergence.
We now solve the inner minimization problem, i.e., Eq.
(6) with ﬁxed ˜ Q ( a ) .
As we can see, the constraint on the distortion tries to keep Q A | S close to π A | S , which is the target distribution.
By minimizing the mutual information, we want Q A | S as close to Q A as possible, which is the marginal known by the agent (it is sent once at the beginning of the round and does not depend on the state realizations), in order to reduce the number of necessary bits to be transmitted.
The solution to Eq.
(2) is a conditional distribution Q A | S that is a combination of the target policy conditioned on S , and the marginal distribution.
We now discuss some simple corner cases.
If the maximum acceptable distortion is 0 , we need
Q A | S =
π A | S , thus the rate is equal to E P S (cid:2) D KL (cid:0)
π A | S
|| π A (cid:1)(cid:3) .
If S and A are independent, i.e., the policy does not depend on the state, we have π A | S =
π A , thus the decision-maker does not need to send any bits to the agents, which can just sample A ∼ π A .
Moreover, if we can ﬁnd a distribution
Q A such that E P S (cid:2) D KL (cid:0) Q A || π A | S (cid:1)(cid:3) ≤ D , then the best strategy is not to convey any message to the controller, as the constraint is already satisﬁed with rate R ( D ) = 0 .
The problem is to ﬁnd, among all conditional distributions Q ( a | s ) that satisfy the constraint, the one with the minimum divergence from Q ( a ) , in order to minimize the rate log 2 B N , i.e., the number of bits to be consumed per agent, for an arbitrarily large N , in the general case.
We solve the problem using the Lagrangian multipliers, and obtain the shape of the optimal distribution given by
[Equation]
where γ ∗ is s.t.
E P S
[ D KL ( Q γ ∗ || P )]
= D .
The derivation is provided in Appendix A.1.
3. A SYMPTOTIC R EGRET
To prove the results on the achievable regret, we need additional assumptions, which are contained in the Assumption 1 in Russo (2016), which states that rewards have to be distributed following canonical exponential families, and the priors used by TS over the average rewards have to be uniformly bounded, i.e., bounded away from zero ∀ ( s, a ) .
The proofs of both Lemmas are reported in the Appendix D.
In the following, we provide the minimum rate needed to achieve sub-linear regret in all states s ∈ S .
We now deﬁne H ( A ∗ ) as the marginal entropy of the optimal arm, computed based on the marginal π ∗ ( a ) =
(cid:80) s P S ( s ) π ∗ ( a | s ) , and deﬁned as H ( A ∗ )
= (cid:80) a π ∗ ( a ) log 1 π ∗ ( a ) , and we prove that it is the minimum rate required to achieve sub-linear regret.
Lemma 3.1.
If R < H ( A ∗ ) , then it is not possible to convey a policy Q t ( s, a ) that achieves sublinear regret in all states s ∈ S .
The following Lemma provides the achievibility part.
Lemma 3.2.
If R > H ( A ∗ ) , then achieving sub-linear regret is possible in all states s ∈ S .
The consequence of this second Lemma is that, even if the exact TS policy π t cannot be transmitted ∀ t , as long as sufﬁcient rate is available
, i.e., R > H ( A ∗ ) , it is still possible to achieve sub-linear regret. Following the notation introduced in Sec. 2.2, this implies that, as T → ∞ , and for all sub-linear regrets ρ , the regret-communication pair ( ρ, R ) is achievable as long as R > H ( A ∗ ) . Moreover, we argue that the policy construction found in Sec.
3.3 can achieve better empirical performance w.r.t.
the scheme used to prove Lemma 3.2.
The above analysis allows us to characterize an information theoretical bound on the optimal performance, but does not provide a constructive communication scheme.
To ﬁnd a practical coding scheme, we propose a solution that is based on state reduction and computes a compact state representation.
In essence, the decision-maker constructs a message containing the new state representations ˆ s ( s ) ∈ ˆ S of s , one for each agent, and send it over the channel.
Once the agents have received the message, they can sample the actions according to a common policy Q ˆ s ( a | ˆ s ) , which is deﬁned on the compressed state space ˆ S .
If the rate constraint imposes B bits per agent, it means that it is possible to transmit at most 2 B different states to each agent.
The idea is to group the states into 2 B =
M clusters ˆ s 1 , . . .
, ˆ s M , minimizing (cid:80) s P ( s ) D KL ( Q ˆ s ( a | ˆ s )
||
π ( a | s )) , where Q ˆ s ( a | ˆ s ) is the policy deﬁned on the state ˆ s ( s ) ∈ ˆ S .
To ﬁnd the clusters and relative policies we employ the well-known Lloyd algorithm, which is an iterative process to group states into 2 B clusters.
First of all, knowing the policy π , the decisionmaker maps each state s i to a K-dimensional point α i =
π ( ·| s i ) ∈
∆ K , ﬁnding |S| =
L different points α 1 , . .
. , α L .
Then, it generates 2 B =
M random points µ 1 , . .
. , µ M ∈ ∆ K as initial centroids, i.e., representative policies, and iterates over the following two steps:
Assign to each point α i the class j ∗ ∈ { 1 , . . .
, M }
such that j ∗ = arg min
j D KL ( µ j || α i ) , i e., minimizing the divergence between the representative µ j ∗ and the original policy, which is the point α i .
For each cluster j , we now deﬁne with S j the set containing the states associated to the policies in the cluster.
Update µ 1 , . .
. , µ M such that µ j = arg min µ ∈ ∆ K (cid:80) s ∈ S j P ( s ) D KL ( µ || π ( ·| s )) , which is still a convex optimization problem, and can be solved again applying the Lagrangian multipliers.
The solution is
[Equation]
[Equation] ) where the product has to be considered element-wise, A ( S j ) is the sum of the probabilities of states in S j , i.e., A ( S j )
=
(cid:80) s ∈S j P ( s ) , and Z is the normalizing factor.
After computing the new centroids, we go back to step (1).
The derivation of Eq.
(8) is provided in
Appendix A.2.
The process continues until the new solution does not decrease the KL-divergence between the clustered and the target policy D KL ( Q ˆ s (ˆ s, a ) || P ( s, a ))
= (cid:80) Mj
=1 (cid:80) s ∈S j P ( s ) D KL ( µ j || π ( ·| s )) .
Observation Note that the controller is assumed to know the 2 NR policies from which it samples the actions of the agents.
This can be transmitted at the beginning of each round.
In this case, the scheme is efﬁcient as long as N log 2 K >> BLλ log 2 K , where λ is the number of bits used to represent the values of the Probability Mass Function (PMF) Q ˆ s ( ·| ˆ s ) .
For this reason, we provide a scheme where the new policy is updated not at every transmission, but just when the new target π has changed considerably.
In particular, if we denote with π cls the policy deﬁned over the compressed state representation, with π last the last policy used to compute π cls , and with π the updated target policy, we compute and transmit π cls every time D KL ( π last || π ) > β .
N UMERICAL RESULTS
In this section, we provide numerical results in support of our theoretical analysis.
Additional experiments and details can be found in Appendix C
In this ﬁrst experiment, we analyze the rate-distortion function, and the related optimal Q ( a | s ) , when the distortion is given by the KL-divergence in three different problems, when the numbers of states, L , and of actions, K , are both equal to 16 .
The target policies π ( a | s ) have a one-to-one relation with the states, and are generated such that π ( a = i | s = j ) = 0 .
99 if j = i , and uniformly distributed otherwise.
We refer to this as the Deterministic case.
In the second experiment, the target policies are generated in a similar way, but the other actions’ probabilities take random values in (0 , 0 . 05] , and then normalized.
This, we call the Random Deterministic case.
In the third experiment, the states are grouped in 4 blocks, such that π ( a = i | s = j )
= 0 .
99 if (cid:98) j/ 4 (cid:99)
=
i , and uniformly distributed otherwise.
This case is denoted as the Block one.
In all three experiments, the distribution P S is uniform over the state space S .
The rate-distortion curves are reported in Fig.
2a.
As we can see, in the Deterministic case, the point with zero distortion has R (0) ∼ H( P S ), where H( P S ) is [Figure] the Shannon entropy of P S , as expected.
Indeed, in this case, the action distributions are strongly correlated with the state, thus we need an accurate knowledge of it.
In the second experiment, the starting point is similar, but the curve decreases more rapidly, caused by the random values of the other actions’ probabilities.
In the third experiment, the zero distortion point is achieved at R (0) ∼bits, since, similarly to the Deterministic experiment, action probabilities are correlated with the state realization, but are grouped into four different cases.
Again, given the uniform distribution over the state, R (0) ∼ H(Unif [4] ), where Unif [4] is the uniform distribution over a set of 4 elements.
In Fig.
2c, the approximate distributions for state s = 8 are reported for the Deterministic and Random
Deterministic cases, when R = 1 .
4. C ONTEXTUAL M ULTI -A RMED B ANDIT
We now analyze the RC-CMAB problem presented in Sec. 2, and apply the clustered policy schemes to solve it.
In particular, we compare the performance of the Perfect agent, which applies TS without any rate constraint, thus admits samples from the true posterior π , with the performance of the Comm , Cluster , and Marginal agents.
The Comm agent uses the optimal scheme provided in Sec. 3.2, the Cluster agent implements the practical coding scheme provided in Sec. 3.5, with B = (cid:100) R (cid:101) bits per agent, where R is the rate adopted by the Comm agent; the Marginal agent adopts the marginal over the states, computed from the target policy π and the environment distribution P S , and serves as a lower bound on the performance.
State distribution P S is uniform over L =states, and there are K = 16 actions, N = 100 agents, and the total number of rounds is T = 200 .
The threshold for changing the clustered policy is set to β = 0 . 2 , and the communication rate constraint to R = 2 . 5 .
In this experiment, for each state s i ∈ S , the best reward is given by the arm a j such that i = j , with i, j ∈ { 0 , . . .
In particular, the reward behind arm i when in state j is a Bernoulli random variable with parameter µ j = 0 .
8 if i = j , whereas µ j ∼ Unif [0 , 0 .
75] if i (cid:54) =
j .
In this case, the best action response is strongly correlated with the state realization, thus a sufﬁciently high rate is required to sample from the target policy π .
As it can be observed from Fig. 3(a), the theoretical rate to transmit the target policy is related to the amount of information the agent is learning from the environment, and it is computed using Eq.
Indeed, during the ﬁrst ∼ 40 iterations, the rate is below R = 2 . 5 .
As the learning process continues, the required rate for reliable transmission increases, as the mutual information between S and A increases.
We highlight that, in order to fairly represent the 16 different actions, one would need 4 bits.
Indeed, another way of looking at Eq.
(2) is through bottleneck principle (Igl et al., 2019), which is used to constrain the mutual information between the states and the actions, and to encourage exploration, that can be exploited to reduce the required rate when training multi-agent systems.
In Fig 3 (b), the KL-divergence between the last policy used to compute the compressed state representation, i.e., π last in Sec. 3.5, and the target known by the decision-maker is reported.
The trend shows that, at the beginning of the learning process, it oscillates rapidly, as the target policy is changing signiﬁcantly between two iterations.
In gray, the threshold β = 2 is highlighted, indicating when a new cluster policy has to be sent.
In this experiment, the decision-maker had to send a new policy, on average, once every ∼ 7 rounds.
Fig.
4 (a) reports the regret at state s = 10 (the other plots can be seen in Appendix C.2), for the four different agents, as deﬁned in Eq.
We can see that both the Cluster and the Comm agents can achieve sub-linear regret in this particular state.
In
Fig.
4 (b), the average rewards obtained by the agents are reported, for the different states separately.
We can see that the developed scheme is still able to achieve good performance.
[Figure] [Figure]
C ONCLUSION
We have studied the RC-CMAB problem, in which an intelligent entity, i.e., the decision-maker, observes the contexts of N parallel CMAB processes, and has to decide on the actions depending on the current contexts and the past actions and rewards.
However, the actions are implemented by a controller that is connected to the decision-maker through a rate-constrained communication link.
First, we cast the problem into the proper information-theoretic framework, formulating it as a policy compression problem, and provided the optimal compression scheme in the limit of an inﬁnite number of agents, when the adopted distortion measure is the KL-divergence between the compressed policy adopted by the controller and the policy of the decision-maker.
We then characterize the minimum needed rate to obtain sub-linear regret, and prove that any rates above this threshold can achieve it. In the end, we designed a practical coding scheme to transmit the actions for a ﬁnite N , which relies on a compressed state representation Finally, we evaluated the performances of the policy obtained through the asymptotic information theoretic formulation, and the one obtained through the clustering scheme, and observed a close gap between the two.
We numerically showed the relation between the asymptotic rate bound and the learning phase of agents, showing how it is possible to save communication resources when training a multi-agent system like the one considered.
We believe that this work can serve as a ﬁrst step towards understanding the fundamental performance limits of multi-agent decision-making problems under communication constraints, and highlights the intimate relation between the communication scheme and the learning process.
Ongoing work include deriving an information theoretic converse result on the regret performance, and generalizing the framework to reinforcement learning problems.
A PPENDIX A P OLICIES D ERIVATION A. O PTIMAL POLICY
To solve this problem we solve the related Lagrangian
L ( Q ( a | s ) , λ, µ ) =
(cid:88)
[Equation]
where the Lagrangian multiplier λ has to be optimized to meet the constraints on the divergence, whereas µ ensures that the solution is a probability distributions, i.e., the elements sum up to one.
The positivity constraints on the terms are already satisﬁed by the fact that the solution has an exponential shape.
We ﬁrst take the derivative of the Lagrangian w.r.t.
to the terms Q ( a | s ) and set it to zero
[Equation]
ﬁnding
[Equation]
and now we rewrite with γ = 11+ λ , γ ∈ [0 , 1] , and µ such that e − ( µ + λ ) 1+ λ is the normalizing factor.
The solution has thus the shape
[Equation]
By the convexity of KL-Divergence and its triangular inequality, we know the solution lies on the boundary of the constraints, i.e., when E P S [ D KL ( Q γ ∗ || P )]
= D .
Consequently, if we can numerically ﬁnd
A. U PDATE C ENTROIDS
Again, we compute the optimal centroids by solving the Lagrangian
[Equation]
taking its derivative and solving the equality
[Equation]
ﬁnding
[Equation]
where Z is the normalizing factor, obtaining the shape expressed in Eq.
B P RESENTED ALGORITHMS
In this appendix, the algorithms for the theoretically-optimal and cluster policies are described.
B. T HEORETICAL O PTIMAL P OLICY
This is the pseudocode of the algorithm at the decision-maker side, when adopting the informationtheoretic inspired compressed policy.
The only task for the controller is to decode the received message, and to communicate the actions to the agents.
We deﬁne with [ A ] the set { 1 , . . .
, A } , and with Unif B the uniform probability over the set B .
Input rate R , n° agents N , n° actions K and state distribution P S
Initialize Thompson Sampling policy π , and compressed policy Q ( ·| s ) to Unif [ K ] foreach t ∈ 1 , . .
, T do
Observe s Nt
=
( s t, 1 , . . .
, s t,N )
Sample a Nt
=
( a t, 1 , . .
. , a t,N ) from Q ( ·| s )
Code a Nt into u t = f t ( a Nt ) and transmit to the controller
Observe r Nt =
( r t, 1 , . . .
, r t,N )
Update action posteriors π ( ·| s ) using the Thompson Sampling Algorithm
Update Q ( a | s ) using the Blahut-Arimoto Algorithm (Iterate over Eq.
5 and Eq. 6)
B.
C LUSTER P OLICY
This is the pseudocode of the algorithm at the decision-maker side, when adopting the cluster policy.
The task for the controller is to sample, at time t and for each agent i , the action a t,i according to the last received centroid µ j ( s t,i ) , whose components convey the action probabilities.
Algorithm
Input per-agent bit B , n° agents N , n° actions K and state distribution P S , threshold β
Initialize Thompson Sampling policy π
Compute the centroids µ j for j = 1 , . .
, 2 B using π with Lloyd algorithm and the rule in Eq.
Transmit the centroids µ j to all agents.
π last = π foreach t ∈ 1 , . . .
, T do Observe s Nt
=
( s t, 1 , . . .
, s t,N ) if D KL (cid:0) π
last || π (cid:1) >
β then Update the centroids µ j using π with Lloyd algorithm and the rule in Eq.
Transmit the centroids µ j to all agents.
∀ s t,i compute the index j ( s t,i ) indicating its belonging cluster Transmit the vector j Nt =
( j ( s t, 1 ) , . . .
, j ( s t,N )) Observe r
Nt
=
( r t, 1 , . . .
, r t,N ) Update action posteriors π ( ·| s ) using the Thompson Sampling Algorithm C E XPERIMENTS
In this section, additional experiments are provided to clarify the proposed scheme, and quantify the trade-offs between rate and regret.
C. S TATE R EPRESENTATION
In Fig. 5 we report the policy clustering results when applying the coding scheme explained in Sec. 3.5, when R = 2 bits can be used to represent 20 randomly generated policies.
In the ﬁgure, it is possible to see the 4 policies representative of the 4 compressed state representations.
As we can see, the policy centroids found by the coding scheme try to fairly resemble the shapes of all the policies within the same cluster.
In this section we reported some more details on the RC-CMAB experiments presented in Sec. 4.2.
[Figure]
C. 8 -B LOCK E XPERIMENT
In this second RC-CMAB experiment, the setting is similar to the one presented above, but the best action response is not a one-to-one mapping with the state.
Again, a ∈ { 0 , . . .
, 15 } and s ∈ { 0 , . . .
, 15 } , but the Bernoulli parameter µ a ( s ) for action a in state s is 0 .
8 if (cid:98) s 2 (cid:99) = a , and sampled uniformly in (0 , 0 .
75] otherwise.
Thus, the best action responses are grouped into 8 different classes, depended on the state realization.
In this case, the rate is limited to R = 2 .
[Figure]
[Figure]
C. R EGRET VS N UMBER OF C LASSES
In this last experiment, the environment is the same as the one described in Sec.
4.2, a part form the number of agents, that here is N = 50 .
The task is to quantify the effect of the number of per-agent bits B in the cluster policy on the achievable regret, which varies from 1 to 4 .
As we can see from the plots below, when B is equal to 1 or 2 , which in turn means that the number of clusters is, respectively, 2 and 4 , the regret is not sub-linear in several states.
However, the policy with just 2 bits can achieve performance comparable with the 3 and 4 bits policies in some states, e.g., state 4 and 6 .
This is due to the fact that even in those cases with not enough bits, the posteriors still converges to good solutions, thus if a cluster contains, for example, one state, it will convey the best policy, achieving sub-linear regret.
On the contrary, when a cluster contains more states, the representative policy can fail to achieve optimal performance, if those are substantially different. D R EGRET B OUND
To prove our statements, we deﬁne with H q ( X ) and I q ( X ; Y ) the marginal entropy and mutual information w.r.t.
to the joint probability q , i.e., H
( A ∗ ) =
H π ∗ ( A ) .
We start by proving Lemma D.1.
Lemma D.1.
If the exact Thompson Sampling policies π t ( a | s ) achieve sub-linear Bayesian regret for all state s ∈ S , then lim t →∞ I π t ( S ; A ) =
lim t →∞ H π t
( A ) =
H ( A ∗ ) .
Proof.
First of all, we write I π t ( S ; A ) =
H π t ( A ) − H π t
( A | S ) .
Following Theorem 2 from Kalkanli & Ozgur (2020), if π t ( a | s ) achieves sub-linear regret ∀ s ∈ S , then ∀ s ∈ S
[Equation]
when a
[Equation]
is the optimal arm
[Equation]
Consequently, we have that in the limit π t ( a | s ) is a deterministic function, thus
[Equation]
which concludes our proof.
Then, we prove Lemma 3.1, which is repeated below.
Lemma 3.1 If R < H
( A ∗ ) , then it is not possible to convey a policy Q t ( s, a ) that achieves sublinear regret in all state s ∈ S .
Proof.
Following Lemma 10 in Phan et al.
(2019), if lim t →∞ π t ( a ∗ | s )
= 1 and D KL ( Q t || π t ) <
(cid:15) , ∀ t , for some (cid:15) > 0 , then lim t →∞ Q t ( a ∗ | s )
= 1 , inducing lim t →∞ D KL ( Q t || π t )
= 0 .
Now, for any policy Q t , if the minimal rate to convey it, ˆ R t = I Q t ( S ; A ) , is below the threshold of the optimal policy, i.e., ˆ R t < H
( A ∗ ) , by Eq.
(2), we have
[Equation]
which holds also in the limit
Otherwise, we would have a policy Q t s.t.
D KL ( Q t || π ∗ )
= 0 , which could be conveyed at a rate strictly below that characterized by Eq.
(2), which would contradict with the deﬁnition of the rate-distortion function.
Consequently, by Lemma 10 in Phan et al.
(2019) and Eq.
(10), there cannot be any (cid:15) > 0 s.t.
lim t →∞ D KL ( Q t || π t ) < (cid:15) ; and hence, lim t →∞ D KL ( Q t || π t )
= ∞ .
However, if lim t →∞ D KL ( Q t || π t ) = ∞ , ∃ ( s, a ) ∈ S
×
A s.t.
lim t →∞ Q t ( a | s ) =
c > 0 , when a (cid:54) =
a ∗ .
This implies that, at step t , Q t ( s, a ) plays a sub-optimal arm in state s with constant probability, and so it can not achieve sub-linear regret in all states.
Finally, we provide a proof for Lemma 3.2, which we repeat below.
Lemma 3.2 If R > H
( A ∗ ) , then achieving sub-linear regret is possible in all states s ∈ S .
Proof.
{+We deﬁne by R π t the rate needed to convey the TS policy perfectly to the controller at time t , and let δ > 
+}
s.t. R =
H ( A ∗ )
+ δ , where R is the available communication rate.
We now provide a scheme that guarantees sub-linear regret.
First, generate ρ t parameters as in Sec. 3.3.
As long as R π t > R , with probability ρ t play a t uniformly at random, and with probability 1 − ρ t , play according to a policy Q t ( a | s ) , which satisﬁes the rate-distortion constraint I ( A ; S ) <
R under Q t ( a | s ) , which can be transmitted to the controller, and will have a bounded reverse KL divergence from the TS policy π t at time t .
Following Lemmain Russo (2016), in this way enough exploration is guaranteed for the TS policy to concentrate, and there exists a ﬁnite t 0 s.t.
∀ t > t 0 , R π t < H
( A ∗ )
+ δ .
This means that, for the ﬁrst t 0 rounds, both the target and the approximating policies are playing sub-optimal arms with non-zero probabilities.
However, their average gap is within a constant, given the average rewards are bounded within [0 , 1] .
Then, ∀ t > t 0 it is possible to play the exact TS policy, leading to the optimal policy for all future steps, and hence, a sub-linear regret.