Original: Learning from Distributed Users in Contextual Linear Bandits Without Sharing the ContextAnonymous Author(s)
Revised: Learning from Distributed Users in Contextual Linear Bandits Without Sharing the Context

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Contextual linear bandits is a rich and theoretically important model that has manypractical applications.
Revised: Contextual linear bandits is a rich and theoretically important model that has many practical applications.

Original: Recently, this setup gained a lot of interest in applicationsover wireless where communication constraints can be a performance bottleneck,especially when the contexts come from a large d -dimensional space.
Revised: Recently, this setup gained a lot of interest in applications over wireless where communication constraints can be a performance bottleneck, especially when the contexts come from a large d -dimensional space.

Original: In this paper,we consider a distributed memoryless contextual linear bandit learning problem,where the agents who observe the contexts and take actions are geographicallyseparated from the learner who performs the learning while not seeing the contexts.
Revised: In this paper, we consider a distributed memoryless contextual linear bandit learning problem, where the agents who observe the contexts and take actions are geographically separated from the learner who performs the learning while not seeing the contexts.

Original: We assume that contexts are generated from a distribution and propose a methodthat uses ≈ 5 d bits per context for the case of unknown context distribution and0 bits per context if the context distribution is known, while achieving nearly thesame regret bound as if the contexts were directly observable.
Revised: We assume that contexts are generated from a distribution and propose a method that uses ⇡ 5 d bits per context for the case of unknown context distribution and 0 bits per context if the context distribution is known, while achieving nearly the same regret bound as if the contexts were directly observable.

Original: The former boundimproves upon existing bounds by a log( T ) factor, where T is the length of thehorizon, while the latter achieves information theoretical tightness.
Revised: The former bound improves upon existing bounds by a log( T ) factor, where T is the length of the horizon, while the latter achieves information theoretical tightness.

Original:  Introduction
Revised:  Introduction

Original: Contextual linear bandits offer a sequential decision-making framework that combines fundamentaltheoretical importance with signiﬁcant practical popularity [8], as it offers a tractable way to captureside information (context), as well as a potentially inﬁnite set of decisions (actions).
Revised: Contextual linear bandits offer a sequential decision-making framework that combines fundamental theoretical importance with signiﬁcant practical popularity [8], as it offers a tractable way to capture side information (context), as well as a potentially inﬁnite set of decisions (actions).

Original: The most prominent application is in recommendation systems [30], but it has also been used in applications suchas virtual support agents [39], clinical trials [12], transportation systems [9], wireless optimization[26, 25], health [10], robotics [31] and online education [34].
Revised: The most prominent application is in recommendation systems [30], but it has also been used in applications such as virtual support agents [39], clinical trials [12], transportation systems [9], wireless optimization [26, 25], health [10], robotics [31] and online education [34].

Original: In this paper, we develop algorithms that support the deployment of contextual linear bandits indistributed settings.
Revised: In this paper we develop algorithms that support the deployment of contextual linear bandits in distributed settings.

Original: In particular, we consider the case where a central learner wishes to solve acontextual linear bandit problem with the help of transient agents.
Revised: In particular, we consider the case where a central learner wishes to solve a contextual linear bandit problem with the help of transient agents.

Original: That is, we assume that the agentsdo not keep memory of past actions and may not be present for the whole duration of learning;learning in our setup can happen thanks to the persistent presence of the central learner.
Revised: That is, we assume that the agents do not keep memory of past actions and may not be present for the whole duration of learning; learning in our setup can happen thanks to the persistent presence of the central learner.

Original: We view thecentral learner as a “knowledge repository”, that accumulates knowledge from the experience of thetransient agents and makes it available to next agents.
Revised: We view the central learner as a “knowledge repository”, that accumulates knowledge from the experience of the transient agents and makes it available to next agents.

Original: The central learner, through the information itkeeps, could help passing by devices decide how to perform an action, for example: passing by dronesdecide how to perform a manoeuver; agricultural robots decide what amounts of substances such aspesticids to release; and passing by mobile devices decide which local restaurants to recommend.
Revised: The central learner, through the information it keeps, could help passing by devices decide how to perform an action, for example: passing by drones decide how to perform a manoeuver; agricultural robots decide what amounts of substances such as pesticids to release; and passing by mobile devices decide which local restaurants to recommend.

Original: The main challenge we try to address in this paper is the efﬁcient communication of the context theagents experience.
Revised: The main challenge we try to address in this paper is the efﬁcient communication of the context the agents experience.

Original: More speciﬁcally, in our setup, each time an agent joins, she receives from thecentral learner information on the system, such as current estimates of the system parameters; sheobserves her current context, selects and plays an action and collects the corresponding reward.
Revised: More speciﬁcally, in our setup, each time an agent joins, she receives from the 36th Conference on Neural Information Systems (NeurIPS central learner information on the system, such as current estimates of the system parameters; she observes her current context, selects and plays an action and collects the corresponding reward.

Original: Notethat although the distributed agent knows her context, the action she plays and the observed reward,the central learner does not - and needs this information to update its estimate of the system parameters.
Revised: Note that although the distributed agent knows her context, the action she plays and the observed reward, the central learner does not - and needs this information to update its estimate of the system parameters.

Original: The context in particular can be communication heavy - in the examples we mentioned before, fordrones the context could be their navigation capabilities, physical attributes, and enviromental factorssuch as wind speed; for agricultural robots, it could be images that indicate state of plants and sensormeasurements such as of soil consistency; for restaurant recommendations, it could be the personaldietary preferences and restrictions, budget, and emotional state.
Revised: The context in particular can be communication heavy - in the examples we mentioned before, for drones the context could be their navigation capabilities, physical attributes, and enviromental factors such as wind speed; for agricultural robots, it could be images that indicate state of plants and sensor measurements such as of soil consistency; for restaurant recommendations, it could be the personal dietary preferences and restrictions, budget, and emotional state.

Original: Moreover, because of geographicalseparation, the central learner may not have any other way to learn the context beyond communication.
Revised: Moreover, because of geographical separation, the central learner may not have any other way to learn the context beyond communication.

Original: Unlike the reward, that is usually a single scalar value, the context can be a vector of a large dimensiond from an inﬁnite alphabet, and thus, communicating the context efﬁciently is heavily nontrivial.
Revised: Unlike the reward, that is usually a single scalar value, the context can be a vector of a large dimension d from an inﬁnite alphabet, and thus, communicating the context efﬁciently is heavily nontrivial.

Original: The technical question we ask is, how many bits do we need to convey per context to solve the linearbandit problem without downgrading the performance as compared to the non-distributed setting?
Revised: The technical question we ask is, how many bits do we need to convey per context to solve the linear bandit problem without downgrading the performance as compared to the non-distributed setting?

Original: In this paper, we design algorithms that support this goal.
Revised: In this paper, we design algorithms that support this goal.

Original: We note that our algorithms optimizethe uplink communication (from the agents to the central learner), and assume unlimited (cost-free)downlink communication.
Revised: We note that our algorithms optimize the uplink communication (from the agents to the central learner), and assume unlimited (cost-free) downlink communication.

Original: This is a standard assumption in wireless [7, 33, 21] for several reasons:uplink wireless links tend to be much more bandwidth restricted, since several users may be sharingthe same channel; uplink communication may also be battery-powered and thus more expensive tosustain; in our particular case, the agents may have less incentive to communicate (provide theirfeedback) than the central learner (who needs to learn).
Revised: This is a standard assumption in wireless [7, 33, 21] for several reasons: uplink wireless links tend to be much more bandwidth restricted, since several users may be sharing the same channel; uplink communication may also be battery-powered and thus more expensive to sustain; in our particular case, the agents may have less incentive to communicate (provide their feedback) than the central learner (who needs to learn).

Original: Having said that, we note that our algorithms(in Sections 3 and 4) make frugal use of the downlink channels, only using them to transmit systemparameters.
Revised: Having said that, we note that our algorithms (in Sections 3 and 4) make frugal use of the downlink channels, only using them to transmit system parameters.

Original: Below we summarize our main contributions:
Revised: Below we summarize our main contributions:

Original: We show the surprising result that, if the central learner knows the distribution of the contexts,we do not need to communicate the context at all - the agent does not need to send any information
Revised: We show the surprising result that, if the central learner knows the distribution of the contexts, we do not need to communicate the context at all - the agent does not need to send any information

Original: on the actual context she observes and the action she plays.
Revised: on the actual context she observes and the action she plays.

Original: It is sufﬁcient for the agent to just send 1 bit to convey quantized information on her observed reward and nothing else.
Revised: It is sufﬁcient for the agent to just send 1 bit to convey quantized information on her observed reward and nothing else.

Original: But for this verylimited communication, the central learner can learn a policy that achieves the same order of regret asif full information about the context and reward is received.
Revised: But for this very limited communication, the central learner can learn a policy that achieves the same order of regret as if full information about the context and reward is received.

Original: This result holds for nearly all contextdistributions and it is the best we can hope for - zero bits of communication for the context.2.
Revised: This result holds for nearly all context distributions and it is the best we can hope for - zero bits of communication for the context.

Original: If the central learner has no knowledge of the context distribution, we show that ≈ 5 d bits percontext (where d is the context dimension) is sufﬁcient to achieve the same order regret as knowingthe context in full precision.
Revised: If the central learner has no knowledge of the context distribution, we show that ⇡ 5 d bits per context (where d is the context dimension) is sufﬁcient to achieve the same order regret as knowing the context in full precision.

Original: Note that previous algorithms, that rely on constructing 1 /T -net for theset of feature vectors, use O ( d log T ) bits per context to achieve the same order regret, where T isthe length of the horizon [24], and require time complexity of O ( T d ) which is exponential in d .
Revised: Note that previous algorithms, that rely on constructing 1 /T -net for the set of feature vectors, use O ( d log T ) bits per context to achieve the same order regret, where T is the length of the horizon [24], and require time complexity of O ( T d ) which is exponential in d .

Original: Related Work and Distinction.
Revised: Related Work and Distinction.

Original: Contextual linear bandits is a rich and important model that hasattracted signiﬁcant interest both in theory and applications [8, 24].
Revised: Contextual linear bandits is a rich and important model that has attracted signiﬁcant interest both in theory and applications [8, 24].

Revised: Popular algorithms for this setup include LinUCB [1, 37] and cotextual Thompson sampling [2].

Revised: Under Assumption 1, these algorithms achieve a regret of ˜ O

Revised: ( d p T ) , where d is the dimension of an unknown system parameter and T is the time horizon, while the best known lower bound for this setup is ⌦ ( d p T ) [37].

Revised: These algorithms assume perfect knowledge of the contexts and rewards.

Original: Within this space, our workfocuses on operation under communications constraints in a distributed setting.
Revised: Within this space, our work focuses on operation under communications constraints in a distributed setting.

Original: There is large body of work focusing on distributed linear contextual bandits settings, but mainlywithin the framework of federated learning, where batched algorithms have been proposed forcommunication efﬁciency [43, 41, 6, 5, 23] that aggregate together observations and parameterlearning across a large number of iterations.
Revised: There is large body of work focusing on distributed linear contextual bandits settings, but mainly within the framework of federated learning, where batched algorithms have been proposed for communication efﬁciency [43, 41, 6, 5, 23] that aggregate together observations and parameter learning across a large number of iterations.

Original: This is possible because in federated learning, the agentsthemselves wish to learn the system parameters, remain active playing multiple actions throughout thelearning process, and exchange information with the goal of speeding up their learning [43, 41].
Revised: This is possible because in federated learning, the agents themselves wish to learn the system parameters, remain active playing multiple actions throughout the learning process, and exchange information with the goal of speeding up their learning [43, 41].

Original: Incontrast, in our setup batched algorithms cannot reduce the communication cost because each agentonly plays a single action; this may be because agents are transient, but also because they may not beinterested in learning - this may not be a task that the agents wish to consistently perform - and thusdo not wish to devote resources to it.
Revised: In contrast, in our setup batched algorithms cannot reduce the communication cost because each agent only plays a single action; this may be because agents are transient, but also because they may not be interested in learning - this may not be a task that the agents wish to consistently perform - and thus do not wish to devote resources to it.

Original: For example, an agent may wish to try a restaurant in a specialoccasion, but would not be interested in sampling multiple restaurants/learning recommendationsystem parameters.
Revised: For example, an agent may wish to try a restaurant in a special occasion, but would not be interested in sampling multiple restaurants/learning recommendation system parameters.

Original: In other words, we consider a scenario where the user beneﬁts from receivingan action (or policy) from the central learner, e.g., a recommendation.
Revised: In other words, we consider a scenario where the user beneﬁts from receiving an action (or policy) from the central learner, e.g., a recommendation.

Original: In response, the user givesfeedback to the central learner in terms of (compressed) context/reward.
Revised: In response, the user gives feedback to the central learner in terms of (compressed) context/reward.

Original: The compression operationsbeneﬁt the user by helping reduce her communication cost.
Revised: The compression operations beneﬁt the user by helping reduce her communication cost.

Original: In principle, the user is not requiredto respond.
Revised: In principle, the user is not required to respond.

Original: But the central learner will be able to learn whenever there is a feedback; creating anincentive for the user to respond could be an interesting future topic.
Revised: But the central learner will be able to learn whenever there is a feedback; creating an incentive for the user to respond could be an interesting future topic.

Original: Our setup supports a different(and complementary) set of applications than the federated learning framework, and requires a newset of algorithms that operate without requiring agents to keep memory of past actions.
Revised: Our setup supports a different (and complementary) set of applications than the federated learning framework, and requires a new set of algorithms that operate without requiring agents to keep memory of past actions.

Revised: There is a long line of research on compression for machine learning and distributed optimization, e.g., compression for distributed gradient descent [40, 3, 32, 18], and distributed inference [19].

Revised: However, such schemes are not optimized for active learning applications.

Revised: Our compression schemes can be seen as quantization schemes for contexts and rewards tailored to active learning applications.

Original:  To the best of our knowledge, our framework has not been examined before for linear contextualbandits.
Revised: Our work also differs from traditional vector compression schemes [15] that aim to reconstruct the data potentially with some distortion (achieve rate-distortion trade-offs). In our case, we do not aim to reconstruct the data, but instead to distinguish the best arm for each context. Indeed, using 0 bits, as we do in Section 3, we cannot reconstruct a meaningful estimate of the context. To the best of our knowledge, our framework has not been examined before for linear contextual bandits.

Original: Work in the literature has examined compression for distributed memoryless MABs [21],but only for rewards (scalar values) and not the contexts (large vectors), and thus these techniquesalso do not extend to our case.
Revised: Work in the literature has examined compression for distributed memoryless MABs [21], but only for rewards (scalar values) and not the contexts (large vectors), and thus these techniques also do not extend to our case.

Original: Additional less-closely related work is reviewed in App. A.

Original: Paper organization.
Revised: Paper organization.

Original: Section 2 reviews our notation and problem formulation; Section 3 providesand analyzes our algorithm for known and Section 4 for unknown context distributions.
Revised: Section 2 reviews our notation and problem formulation; Section 3 provides and analyzes our algorithm for known and Section 4 for unknown context distributions.

Original:  Notation and Problem Formulation
Revised:  Notation and Problem Formulation

Original: Notation.
Revised: Notation.

Original: We use the following notation throughout the paper.
Revised: We use the following notation throughout the paper.

Original: For a vector X we use X i or ( X ) i todenote the i -th element of the vector X ; similarly for a matrix V , we use V ij or ( V ) ij to denote theelement at row i , and column j .
Revised: For a vector X we use X i or ( X ) i to denote the i -th element of the vector X ; similarly for a matrix V , we use V ij or ( V ) ij to denote the element at row i , and column j .

Original: We use (cid:107) V (cid:107) 2 to denote the matrix spectral norm.
Revised: We use k V k 2 to denote the matrix spectral norm.

Original: For a functionf , we denote its domain and range by dom ( f ) , ran ( f ) respectively.
Revised: For a function f , we denote its domain and range by dom ( f ) , ran ( f ) respectively.

Original: When dom ( f ) ⊆ R , we usef ( X ) for a vector X ∈ R d to denote f ( X ) :=
Revised: When dom ( f ) ✓ R , we use f ( X ) for a vector X 2 R d to denote f ( X ) : =

Original: [ f ( X 1 ) , ..., f ( X d )] , i.e., the function f is appliedelement-wise; for example we use X 2 to denote the element-wise square of X .
Revised: [ f ( X 1 ) , ..., f ( X d )] , i.e., the function f is applied element-wise; for example we use X 2 to denote the element-wise square of X .

Original: We denote the inverseof a function f by f − 1 ; if f is not one-to-one, with abuse of notation we use f − 1 to denote a functionthat satisﬁes f ( f − 1 ( x ))
Revised: We denote the inverse of a function f by f  1 ; if f is not one-to-one, with abuse of notation we use f  1 to denote a function that satisﬁes f ( f  1 ( x ))

Original: = x ∀ x ∈ ran ( f ) (this is justiﬁed due to the axiom of choice [22]).
Revised: = x 8 x 2 ran ( f ) (this is justiﬁed due to the axiom of choice [22]).

Original: For amatrix V , we use V − 1 to denote its inverse; if V is singular, we use V − 1 to denote its pseudo-inverse.
Revised: For a matrix V , we use V  1 to denote its inverse; if V is singular, we use V  1 to denote its pseudo-inverse.

Original: We use [ N ] for N ∈ N to denote { 1 , ..., N } , and { X a } a ∈A to denote the set { ( a, X a ) | a ∈ A }.
Revised: We use [ N ] for N 2 N to denote { 1 , ..., N } , and { X a } a 2 A to denote the set { ( a, X a ) | a 2 A }.

Original: Wesay that y
Revised: We say that y

Original: = O ( f ( x )) if there is x 0 and a constant C such that y ≤ Cf ( x ) ∀ x > x 0 ; we also use˜ O ( f ( x )) to omit log factors.
Revised: = O ( f ( x )) if there is x 0 and a constant C such that y  Cf ( x ) 8 x > x 0 ; we also use ˜ O ( f ( x )) to omit log factors.

Original: Contextual Linear Bandits.
Revised: Contextual Linear Bandits.

Original: We consider a contextual linear bandits problem over a horizon oflength T [8], where at each iteration t = 1 , ..., T , an agent, taking into account the context, choosesan action a t ∈ A and receives a reward r t .
Revised: We consider a contextual linear bandits problem over a horizon of length T [8], where at each iteration t = 1 , ..., T , an agent, taking into account the context, chooses an action a t 2 A and receives a reward r t .

Original: For each action a ∈ A , the agent has access to acorresponding feature vector X t,a ∈ R d .
Revised: For each action a 2 A , the agent has access to a corresponding feature vector X t,a 2 R d .

Original: The set of all such vectors { X t,a } a ∈A is the context attime t , and the agent can use it to decide which action a t to play.
Revised: The set of all such vectors { X t,a } a 2 A is the context at time t , and the agent can use it to decide which action a t to play.

Original: We assume that the context isgenerated from a distribution, i.e., given a , X t,a is generated from a distribution P a .
Revised: We assume that the context is generated from a distribution, i.e., given a , X t,a is generated from a distribution P a .

Original: As a speciﬁcexample, we could have that a ∈ R d and X t,a is generated from a Gaussian distribution withzero mean and covariance matrix || a || 2{+
+} I , where I is the identity matrix, i.e., P a = N (0 , || a || 2 I ) .
Revised: As a speciﬁc example, we could have that a 2 R d and X t,a is generated from a Gaussian distribution with zero mean and covariance matrix || a || {+
+} I , where I is the identity matrix, i.e., P a = N (0 , || a || 2 I ) .

Original: The selection of a t may depend not only on the current context { X t,a } a ∈A but also on the history
Revised: The selection of a t may depend not only on the current context { X t,a } a 2 A but also on the history

Original: H t (cid:44) {{ X 1 ,a } a ∈A , a 1 , r 1 , ..., { X t − 1 ,a } a ∈A , a t − 1 , r t − 1 } , namely, all previously selected actions,observed contexts and rewards.
Revised: H t , {{ X 1 ,a } a 2 A , a 1 , r 1 , ..., { X t  1 ,a } a 2 A , a t  1 , r t  1 } , namely, all previously selected actions, observed contexts and rewards.

Original: Once an action is selected, the reward is generated according to
Revised: Once an action is selected, the reward is generated according to

Original: [Equation]
Revised: [Equation]

Original: (1) where (cid:104) .
Revised: where h .,

Original: (cid:105) denotes the dot product, θ (cid:63) is an unknown (but ﬁxed) parameter vector in R d , andη t is noise.
Revised: i denotes the dot product, ✓ ? is an unknown (but ﬁxed) parameter vector in R d , and ⌘ t is noise.

Original: We assume that the noise follows an unknown distribution with E [ η t |F t ] = 0 and122 E [exp( λη t ) |F t ] ≤ exp( λ 2 / 2) ∀ λ ∈ R , where F t = σ ( { X 1 ,a } a ∈A , a 1 , r 1 , ..., { X t,a } a ∈A , a t ) is theﬁltration [13] of historic information up to time t , and σ ( X ) is the σ -algebra generated by X [13].
Revised: We assume that the noise follows an unknown distribution with E [ ⌘ t |F t ] = 0 and E [exp( ⌘ t ) |F t ]  exp(  2 / 2) 8  2 R , where F t =  ( { X 1 ,a } a 2 A , a 1 , r 1 , ..., { X t,a } a 2 A , a t ) is the ﬁltration [13] of historic information up to time t , and  ( X ) is the  -algebra generated by X [13].

Original: The objective is to minimize the regret R T over a horizon of length T , where
Revised: The objective is to minimize the regret R T over a horizon of length T , where

Original: [Equation]
Revised: [Equation]

Original: The best performing algorithms for this problem, such as LinUCB and contextual Thompson sampling,achieve a worst case regret of ˜ O
Revised: The best performing algorithms for this problem, such as LinUCB and contextual Thompson sampling, achieve a worst case regret of ˜ O

Original: ( d √ T ) [29, 28, 1, 2].
Revised: ( d p T ) [29, 28, 1, 2].

Original: The best known lower bound is Ω( d √ T ) [37].
Revised: The best known lower bound is ⌦ ( d p T ) [37].

Original: In the rest of this paper, we make the following assumptions that are standard in the literature [24].
Revised: In the rest of this paper, we make the following assumptions that are standard in the literature [24].

Original: Assumption 1.
Revised: Assumption 1.

Original: We consider contextual linear bandits that satisfy:
Revised: We consider contextual linear bandits that satisfy:

Original: [Equation]
Revised: [Equation]

Original: The boundedness assumption on r t can be relaxed using [21], which only requires approximately 3 . 5bits on average to send r t , even if it is unbounded.
Revised: The boundedness assumption on r t can be relaxed using [21], which only requires approximately 3 .bits on average to send r t , even if it is unbounded.

Original: Memoryless Distributed Contextual Linear Bandits.
Revised: Memoryless Distributed Contextual Linear Bandits.

Original: We consider a distributed setting that consistsof a central learner communicating with geographically separated agents.
Revised: We consider a distributed setting that consists of a central learner communicating with geographically separated agents.

Original: For example, the agentsare drones that interact with a trafﬁc policeman (central learner) as they ﬂy by.
Revised: For example, the agents are drones that interact with a trafﬁc policeman (central learner) as they ﬂy by.

Original: We assume that theagents do not keep memory of past actions and may not be present for the whole duration of learning;learning in our setup can happen thanks to the persistent presence of the central learner.
Revised: We assume that the agents do not keep memory of past actions and may not be present for the whole duration of learning; learning in our setup can happen thanks to the persistent presence of the central learner.

Original: At each time t , t = 1 . . .
Revised: At each time t , t = 1 . . .

Original: T , a distributed agent joins the system; she receives from the central learnerinformation on the system, such as the current estimate of the parameter vector θ (cid:63) or the history
Revised: T , a distributed agent joins the system; she receives from the central learner information on the system, such as the current estimate of the parameter vector ✓ ? or the history

Original: H t ; she observes the current context { X t,a } a ∈A , selects and plays an action a t and collects thecorresponding reward r t .
Revised: H t ; she observes the current context { X t,a } a 2 A , selects and plays an action a t and collects the corresponding reward r t .

Original: Note that although the distributed agent knows the context { X t,a } a ∈A , theaction a t and the observed reward r t , the central learner does not.
Revised: Note that although the distributed agent knows the context { X t,a } a 2 A , the action a t and the observed reward r t , the central learner does not.

Original: The central learner may need thisinformation to update its estimate of the system parameters, such as the unknown parameter vectorθ ∗ , and the history H t +1 .
Revised: The central learner may need this information to update its estimate of the system parameters, such as the unknown parameter vector ✓ ⇤ , and the history H t +1 .

Original: However, we assume that the agent is restricted to utilize a communicationconstrained channel and thus may not be able to send the full information to the central learner.
Revised: However, we assume that the agent is restricted to utilize a communicationconstrained channel and thus may not be able to send the full information to the central learner.

Original: The main question we ask in this paper is: can we design a compression scheme, where the agentsends to the central learner only one message using B t bits (for as small as possible a value of B t ) thatenables the central learner to learn equally well (experience the same order of regret) as if there wereno communication constraints?
Revised: The main question we ask in this paper is: can we design a compression scheme, where the agent sends to the central learner only one message using B t bits (for as small as possible a value of B t ) that enables the central learner to learn equally well (experience the same order of regret) as if there were no communication constraints?

Original: With no communication constraints the agent could send unquantizedthe full information {{ X t,a } a ∈A , a t , r t } .
Revised: With no communication constraints the agent could send unquantized the full information {{ X t,a } a 2 A , a t , r t } .

Original: Instead, the agent transmits a message that could be afunction of all locally available information at the agent.
Revised: Instead, the agent transmits a message that could be a function of all locally available information at the agent.

Original: For example, it could be a function of( H t , { X t,a t } a ∈A , a t , r t ) , if the agent had received H t from the central learner.
Revised: For example, it could be a function of ( H t , { X t,a t } a 2 A , a t , r t ) , if the agent had received H t from the central learner.

Original: It could also be afunction of just ( X t,a t , r t ) , which could be sufﬁcient if the central learner employs an algorithm suchas LinUCB [1, 37].
Revised: It could also be a function of just ( X t,a t , r t ) , which could be sufﬁcient if the central learner employs an algorithm such as LinUCB [1, 37].

Original: In summary, we set the following goal.
Revised: In summary, we set the following goal.

Original: Goal.
Revised: Goal.

Original: Design contextual linear bandit schemes for the memoryless distributed setting that achievethe best known regret of O ( d (cid:112) T log( T )) , while communicating a small number of bits B t .
Revised: Design contextual linear bandit schemes for the memoryless distributed setting that achieve the best known regret of O ( d p T log( T )) , while communicating a small number of bits B t .

Original: We only impose communication constraints on the uplink communication (from the agents to thecentral learner) and assume no cost downlink communication (see discussion in Secttion 1).
Revised: We only impose communication constraints on the uplink communication (from the agents to the central learner) and assume no cost downlink communication (see discussion in Secttion 1).

Original: Stochastic Quantizer (SQ) [16].
Revised: Stochastic Quantizer (SQ) [16].

Original: Our proposed algorithms use stochastic quantization, that we nextreview.
Revised: Our proposed algorithms use stochastic quantization, that we next review.

Original: We deﬁne SQ (cid:96) , (cid:96) ∈ N to be a quantizer, that uses log( (cid:96) +1) bits, consisting of an encoder anddecoder described as following.
Revised: We deﬁne SQ ` , ` 2 N to be a quantizer, that uses log( ` +1) bits, consisting of an encoder and decoder described as following.

Original: The encoder ξ (cid:96) takes a value x ∈ [0 , (cid:96) ] and outputs an integer valueξ (cid:96) = [Equation] x
Revised: The encoder ⇠ ` takes a value x 2 [0 , ` ] and outputs an integer value

Original: [Equation]
Revised: [Equation]

Original: with probability [Equation] [Equation] The output ξ (cid:96) is represented with log( (cid:96) + 1) bits.
Revised: The output ⇠ ` is represented with log( ` + 1) bits.

Original: The decoder D (cid:96) takes as input the binary representation of ξ (cid:96) ( x ) and outputs the real value ξ (cid:96) ( x ) .
Revised: The decoder D ` takes as input the binary representation of ⇠ ` ( x ) and outputs the real value ⇠ ` ( x ) .

Original: The composition of the encoder ξ (cid:96) , the binarymapping, and decoder D (cid:96) is denoted by SQ (cid:96) .
Revised: The composition of the encoder ⇠ ` , the binary mapping, and decoder D ` is denoted by SQ ` .

Original: We notice that since the decoder only inverts the binarymapping operation, we have that SQ (cid:96) = 
Revised: We notice that since the decoder only inverts the binary mapping operation, we have that SQ ` = ⇠ ` .

Original: ξ (cid:96) . When SQ (cid:96) is applied at the agents side, the agent encodesits data, x , as ξ (cid:96) ( x ) , then sends the corresponding binary mapping to the central learner that applies
Revised: When SQ ` is applied at the agents side, the agent encodes its data, x , as ⇠ ` ( x ) , then sends the corresponding binary mapping to the central learner that applies

Original: D (cid:96) to get SQ (cid:96) ( x ) .
Revised: D ` to get SQ ` ( x ) .

Original: With slightly abuse of notation, this operation is described in the paper, by sayingthat the agent sends SQ (cid:96) to the central learner.
Revised: With slightly abuse of notation, this operation is described in the paper, by saying that the agent sends SQ ` to the central learner.

Original: The quantizer SQ (cid:96) is a form of dithering [16] and it has the following properties
Revised: The quantizer SQ ` is a form of dithering [16] and it has the following properties

Original: E [ SQ (cid:96) ( x ) | x ] = (cid:98) x (cid:99) ( (cid:100) x (cid:101)

Original: [Equation]
Revised: [Equation]

Original: + (cid:100) [Equation] (cid:101) ( x
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (cid:98) x (cid:99) ) =

Original: x ( (cid:100) x (cid:101) − (cid:98) x (cid:99) )

Original: = x, and | SQ (cid:96) ( x ) − x | ≤ 1 .

Original: In particular, it conveys an unbiased estimate of the input with a difference that is bounded byalmost surely.
Revised: In particular, it conveys an unbiased estimate of the input with a difference that is bounded byalmost surely.

Original: We also deﬁne a generalization of SQ (cid:96) denoted by SQ [ a,b ] (cid:96) where the input x of the encoder is in [ a, b ] instead of [0 , (cid:96) ] .
Revised: We also deﬁne a generalization of SQ ` denoted by SQ [ a,b ] ` where the input x of the encoder is in [ a, b ] instead of [0 , ` ] .

Original: The encoder ﬁrst shifts and scales x using ˜ x
Revised: The encoder ﬁrst shifts and scales x using ˜ x

Original: = (cid:96) b − a ( x − a ) to make it lie in [0 , (cid:96) ] , then feeds ˜ x to the encoder in (3).
Revised: = ` b  a ( x  a ) to make it lie in [0 , ` ] , then feeds ˜ x to the encoder in (3).

Original: This operation is inverted at the decoder.
Revised: This operation is inverted at the decoder.

Original: It is easy to see that SQ [ a,b ] (cid:96) satisﬁes
Revised: It is easy to see that SQ [ a,b ] ` satisﬁes

Original: [Equation]
Revised: [Equation]

Original:  Contextual Linear Bandits with Known Context Distribution
Revised:  Contextual Linear Bandits with Known Context Distribution

Original: In this section, we show that if the central learner knows the distributions for the vectors X t,a , thenthe agent does not need to convey the speciﬁc realization of the vector X t,a she observes at all - it issufﬁcient to just send 1 bit to convey some information on the observed reward and nothing else.
Revised: In this section, we show that if the central learner knows the distributions for the vectors X t,a , then the agent does not need to convey the speciﬁc realization of the vector X t,a she observes at all - it is sufﬁcient to just send 1 bit to convey some information on the observed reward and nothing else.

Original: Butfor this very limited communication, the central learner can experience the same order regret, as whenreceiving in full precision all the information that the agents have, namely, R T
Revised: But for this very limited communication, the central learner can experience the same order regret, as when receiving in full precision all the information that the agents have, namely, R T

Original: = O ( d √ T log T ) .
Revised: = O ( d p T log T ) .

Original: Algorithm 1, that we describe in this section, provides a method to achieve this.
Revised: Algorithm 1, that we describe in this section, provides a method to achieve this.

Original: Algorithm 1 is clearlyoptimal, as we cannot hope to use less than zero bits for the vector X t,a .
Revised: Algorithm 1 is clearly optimal, as we cannot hope to use less than zero bits for the vector X t,a .

Original: Remark 1.
Revised: Remark 1.

Original: Knowledge of the distribution of X t,a is possible in practice, since many times thecontext may be capturing well studied statistics (e.g., male or female, age, weight, income, race,dietary restrictions, emotional state, etc) - the advent of large data has made and will continue to makesuch distributions available.
Revised: Knowledge of the distribution of X t,a is possible in practice, since many times the context may be capturing well studied statistics (e.g., male or female, age, weight, income, race, dietary restrictions, emotional state, etc) - the advent of large data has made and will continue to make such distributions available.

Original: Similarly, actions may be ﬁnite (eg., restaurants to visit) or well described(e.g. , released amounts of substances), and thus the distribution of X t,a could be derived.
Revised: Similarly, actions may be ﬁnite (eg., restaurants to visit) or well described (e.g., released amounts of substances), and thus the distribution of X t,a could be derived.

Original: When thedistribution is approximately known, we provide later in this section a bound on the misspeﬁcationperformance penalty in terms of regret.
Revised: When the distribution is approximately known, we provide later in this section a bound on the misspeﬁcation performance penalty in terms of regret.

Original: Main Idea.
Revised: Main Idea.

Original: The intuition behind Algorithm 1 is that it reduces the multi-context linear bandit problemto a single context problem.
Revised: The intuition behind Algorithm 1 is that it reduces the multi-context linear bandit problem to a single context problem.

Original: In particular, it calls as a subroutine an algorithm we term Λ , that servesas a placeholder for any current (or future) bandit algorithm that achieves regret O ( d √ T log T ) forthe case of a single context (for example, LinUCB [1, 37]).
Revised: In particular, it calls as a subroutine an algorithm we term ⇤ , that serves as a placeholder for any current (or future) bandit algorithm that achieves regret O ( d p T log T ) for the case of a single context (for example, LinUCB [1, 37]).

Original: The central learner uses Λ to convey tothe agents the information they need to select a good action.
Revised: The central learner uses ⇤ to convey to the agents the information they need to select a good action.

Original: Our aim is to parametrize the singlecontext problem appropriately, so that, by solving it we also solve our original problem.
Revised: Our aim is to parametrize the single context problem appropriately, so that, by solving it we also solve our original problem.

Original: Recall that in a single context problem, at each iteration t , any standard linear bandit algorithm Λselects a feature vector (an action) x t from a set of allowable actions X , and observes a reward
Revised: Recall that in a single context problem, at each iteration t , any standard linear bandit algorithm ⇤ selects a feature vector (an action) x t from a set of allowable actions X , and observes a reward

Original: [Equation]
Revised: [Equation]

Original: (4) where θ (cid:48) (cid:63) is an uknown parameter and η t is noise that satisﬁes the same assumptions as in (1).
Revised: where ✓ 0 ? is an uknown parameter and ⌘ t is noise that satisﬁes the same assumptions as in (1).

Original: Theobjective of Λ is to minimize the standard linear regret R T (Λ) over a horizon of length T , namely
Revised: The objective of ⇤ is to minimize the standard linear regret R T ( ⇤ ) over a horizon of length T , namely

Original: [Equation]
Revised: [Equation]

Original: Our reduction proceeds as follows.
Revised: Our reduction proceeds as follows.

Original: We assume that Λ operates over the same horizon of length T andis parametrized by an unknown parameter θ (cid:48) (cid:63) .
Revised: We assume that ⇤ operates over the same horizon of length T and is parametrized by an unknown parameter ✓ 0 ? .

Original: We will design the action set X that we provide to Λusing our knowledge of the distributions P a 2 as we will describe later in (7).
Revised: We will design the action set X that we provide to ⇤ using our knowledge of the distributions P a 2 as we will describe later in (7).

Original: During each iteration,the central learner asks Λ to select an action x t ∈ X and then provides to Λ a reward for this action(our design ensures that this reward satisﬁes (4) with θ (cid:48) (cid:63) = θ (cid:63) ).
Revised: During each iteration, the central learner asks ⇤ to select an action x t 2 X and then provides to ⇤ a reward for this action (our design ensures that this reward satisﬁes (4) with ✓ 0 ? = ✓ ? ).

Original: Λ operates with this information,oblivious to what else the central learner does.
Revised: ⇤ operates with this information, oblivious to what else the central learner does.

Original: Yet, the action x t is never actually played: the centrallearner uses the selected action x t to create an updated estimate of the parameter vector ˆ θ t , as wewill describe later, and only sends this parameter vector estimate to the distributed agent.
Revised: Yet, the action x t is never actually played: the central learner uses the selected action x t to create an updated estimate of the parameter vector ˆ ✓ t , as we will describe later, and only sends this parameter vector estimate to the distributed agent.

Original: The agentobserves her context, selects what action to play, and sends back her observed quantized reward tothe central learner.
Revised: The agent observes her context, selects what action to play, and sends back her observed quantized reward to the central learner.

Original: This is the reward that the central learner provides to Λ .
Revised: This is the reward that the central learner provides to ⇤ .

Original: We design the set X andthe agent operation to satisfy that: (4) holds; and R T − R T (Λ) is small, where R T is the regret forour original multi-context problem and R T (Λ) the regret of Λ .
Revised: We design the set X and the agent operation to satisfy that: (4) holds; and R T  R T ( ⇤ ) is small, where R T is the regret for our original multi-context problem and R T ( ⇤ ) the regret of ⇤ .

Original: We next try to provide some intuitionon how we achieve this.
Revised: We next try to provide some intuition on how we achieve this.

Original: We ﬁrst describe how we construct the set X .
Revised: We ﬁrst describe how we construct the set X .

Original: Let Θ be the set of all values that θ (cid:63) could possiblytake.
Revised: Let ⇥ be the set of all values that ✓ ? could possibly take.

Original: For each possible parameter vector value θ ∈ Θ the central learner considers the quantity
Revised: For each possible parameter vector value ✓ 2 ⇥ the central learner considers the quantity

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where x a is the random variable that follows the distribution P a .
Revised: where x a is the random variable that follows the distribution P a .

Original: Ties in (6) can be broken uniformlyat random.
Revised: Ties in (6) can be broken uniformly at random.

Original: In fact any pre-selected choice function would work as long as the same function is alsoused in step 12 of Algorithm 1.
Revised: In fact any pre-selected choice function would work as long as the same function is also used in step 12 of Algorithm 1.

Original: Note that the function X (cid:63) : R d → R d can be computed ofﬂine beforethe learning starts, see Example 1.
Revised: Note that the function X ? : R d ! R d can be computed ofﬂine before the learning starts, see Example 1.

Original: We then use
Revised: We then use

Original: [Equation]
Revised: [Equation]

Original: Intuitively, for each value of θ , we optimistically assume that the distributed agent may select the bestpossible realization X t,a for this θ (that has the expectation in (6)), and receive the associated reward;accordingly, we restrict the action space X of Λ to only contain the expectation of these “best” X t,a .
Revised: Intuitively, for each value of ✓ , we optimistically assume that the distributed agent may select the best possible realization X t,a for this ✓ (that has the expectation in (6)), and receive the associated reward; accordingly, we restrict the action space X of ⇤ to only contain the expectation of these “best” X t,a .

Original: The vector x t ∈ X may not actually be the vector corresponding to the action the agent selects; it isonly used to convey to the agent an estimate of the unknown parameter ˆ θ t that satisﬁes x t = 
Revised: The vector x t 2 X may not actually be the vector corresponding to the action the agent selects; it is only used to convey to the agent an estimate of the unknown parameter ˆ ✓ t that satisﬁes x t = X ?

Original: X (cid:63) (ˆ θ t ) .
Revised: (ˆ ✓ t ) .

Original: Although the central learner does not control which action the agent plays, this is inﬂuenced by ˆ θ t ;we show in App.
Revised: Although the central learner does not control which action the agent plays, this is inﬂuenced by ˆ ✓ t ; we show in App.

Original: B that X t,a t is an unbiased estimate of x t , and the generated reward follows thelinear model in (4) with θ (cid:48) (cid:63)
Revised: A that X t,a t is an unbiased estimate of x t , and the generated reward follows the linear model in (4) with ✓ 0 ?

Original: = θ (cid:63) .
Revised: = ✓ ? .

Original: In Theorem 1, we prove that
Revised: In Theorem 1, we prove that

Original: [Equation]
Revised: [Equation]

Original: Hence, if Λ converges to selecting the best action for the single context problem, we will havethat ˆ θ t converges to θ (cid:63) if the maximizer in (8) is unique.
Revised: Hence, if ⇤ converges to selecting the best action for the single context problem, we will have that ˆ ✓ t converges to ✓ ? if the maximizer in (8) is unique.

Original: If there are multiple values for θ with
Revised: If there are multiple values for ✓ with

Original: X (cid:63) ( θ ) = 
Revised: X ? ( ✓ ) = X ?

Original: X (cid:63) ( θ (cid:63) ) , we show in the proof of Theorem 1 that they all lead to the same expected rewardfor the original multi-context problem.
Revised: ( ✓ ? ) , we show in the proof of Theorem 1 that they all lead to the same expected reward for the original multi-context problem.

Original: Example 1.
Revised: Example 1.

Original: Consider the case where d = 1 , A = { 1 , 2 } , X t,a ∈ {− 1 , 1 } ∀ a ∈ A , Θ = {− 1 , 1 } ,θ (cid:63) = 1 and X t, 1 takes the value − 1 with probability p and 1 otherwise, while X t, 2 takes the values− 1 with probability q and 1 otherwise.
Revised: Consider the case where d = 1 , A = { 1 , 2 } , X t,a 2 {  1 , 1 } 8 a 2 A , ⇥ = {  1 , 1 } , ✓ ? = 1 and X t, 1 takes the value  1 with probability p and 1 otherwise, while X t, 2 takes the values  1 with probability q and 1 otherwise.

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that if arg max X t,a (cid:104) X t,a , 1 (cid:105) (cid:54) = 1 , it must be the case that both X t, 1 and
Revised: where we use the fact that if arg max X t,a h X t,a , 1 i 6 = 1 , it must be the case that both X t, 1 and

Original: X t, 2 are − 1 .
Revised: X t, 2 are  1 .

Original: Thus, X (cid:63) (1) =
Revised: Thus, X ? (1) =

Original: E [arg max X t,a (cid:104) X t,a , 1 (cid:105) ] {+= 
+}
Revised: E [arg max X t,a h X t,a , 1 i ] {+= 
+}

Original: = 1 − 2 pq , and similarly X (cid:63) ( − 1) =− 1 + 2(1 − p )(1 − q ) , and hence, X = { 1 − 2 pq, − 1 + 2(1 − p )(1 − q ) } .
Revised:  2 pq , and similarly X ? (  1) =  1 + 2(1  p )(1  q ) , and hence, X = { 1  2 pq,  1 + 2(1  p )(1  q ) } .

Original: If Λ decides to pickx t = 1 − 2 pq , we have that ˆ θ t = 1 , otherwise ˆ θ t = − 1 .
Revised: If ⇤ decides to pick x t = 1  2 pq , we have that ˆ ✓ t = 1 , otherwise ˆ ✓ t =  1 .

Original: This estimate ˆ θ t is then conveyed to theagent to help her pick the action.
Revised: This estimate ˆ ✓ t is then conveyed to the agent to help her pick the action.

Original: Algorithm Operation.
Revised: Algorithm Operation.

Original: The pseudo-code is provided in Algorithm 1.
Revised: The pseudo-code is provided in Algorithm 1.

Original: • First, the central learner calculates the function
Revised: • First, the central learner calculates the function

Original: [Equation]
Revised: [Equation]

Original: and creates the action set X
Revised: and creates the action set X

Original: = { X (cid:63) ( θ ) | θ ∈ Θ } that algorithm Λ is going to use.•
Revised: = { X ? ( ✓ ) | ✓ 2 ⇥ } that algorithm ⇤ is going to use.

Original:  At each time t , based on past history, Λ decides on a next action x t ∈ X .
Revised: • At each time t , based on past history, ⇤ decides on a next action x t 2 X .

Original: The central learner usesx t to calculate the new update ˆ θ t = X − 1 ( x t ) , where X − 1 is the inverse of X (cid:63) (see Section 2).• The agent receives ˆ θ t from the central learner, observes her context, plays an action a t =arg max a ∈A (cid:104) X t,a , ˆ θ t (cid:105) , and observes the reward r t .
Revised: The central learner uses x t to calculate the new update ˆ ✓ t = X  1 ( x t ) , where X  1 is the inverse of X ? (see Section 2). • The agent receives ˆ ✓ t from the central learner, observes her context, plays an action a t = arg max a 2 A h X t,a , ˆ ✓ t i , and observes the reward r t .

Original: She then quantizes the reward using a stochasticquantizer SQ 1 (see Section 2), and communicates the outcome using one bit to the central learner.•
Revised: She then quantizes the reward using a stochastic quantizer SQ 1 (see Section 2), and communicates the outcome using one bit to the central learner.

Original:  The central learner provides the quantized reward as input to Λ .
Revised: • The central learner provides the quantized reward as input to ⇤ .

Original: Note that Λ is oblivious to whatactions are actually played; it treats thereceived reward as corresponding to the action x t it haddecided. Send ˆ θ t = X − 1 ( x t ) to agent. Agent : 10:
Revised: Note that ⇤ is oblivious to what actions are actually played; it treats the

Original: Receive ˆ θ t from the central learner. Observe context realization { X t,a } a ∈A . Pull arm a t = arg max a ∈A (cid:104) X t,a , ˆ θ t (cid:105) and receive reward r t . Send ˆ r t = SQ 1 ( r t ) to the central learner using 1-bit.
Revised: received reward as corresponding to the action x t it had decided.

Original: The following theorem proves that Algorithm 1 achieves a regret R T (Λ) + O ( √ T log T ) , where
Revised: The following theorem proves that Algorithm 1 achieves a regret R T ( ⇤ ) + O ( p T log T ) , where

Original: R T (Λ) is the regret of Λ in (5).
Revised: R T ( ⇤ ) is the regret of ⇤ in (5).

Original: Hence, if Λ satisﬁes the best known regret bound of O ( d √ T log T ) ,e.g., LinUCB, Algorithm 1 achieves a regret of O ( d √ T log T ) .
Revised: Hence, if ⇤ satisﬁes the best known regret bound of O ( d p T log T ) , e.g., LinUCB, Algorithm 1 achieves a regret of O ( d p T log T ) .

Original: The theorem holds under the mildset of assumptions that we stated in Section 2.
Revised: The theorem holds under the mild set of assumptions that we stated in Section 2.

Original: Theorem 1.
Revised: Theorem 1.

Original: Algorithm 1 uses 1 bit per reward and 0 bits per context.
Revised: Algorithm 1 uses 1 bit per reward and 0 bits per context.

Original: Under Assumption 1, itachieves a regret R T
Revised: Under Assumption 1, it achieves a regret R T

Original: = R T (Λ) + O ( √ T log T ) with probability at least 1 − 1 T .
Revised: = R T ( ⇤ ) + O ( p T log T ) with probability at least 1  1 T .

Original: Proof outline.
Revised: Proof outline.

Original: The complete proof is available in App.
Revised: The complete proof is available in App.

Original: B. We next provide a short outline.
Revised: A. We next provide a short outline.

Original: Fromthe deﬁnition of X (cid:63) in (10), we notice the following.
Revised: From the deﬁnition of X ? in (10), we notice the following.

Original: Recall that the distributed agent receives ˆ θ tfrom the central learner, and pulls the best action for this ˆ θ t , i.e., a t = arg max a ∈A (cid:104) X t,a , ˆ θ t (cid:105) . 
Revised: Recall that the distributed agent receives ˆ ✓ t from the central learner, and pulls the best action for this ˆ ✓ t , i.e., a t = arg max a 2 A h X t,a , ˆ ✓ t i . We 2: Initialize P a } a a

Original: We show that conditioned on x t , the associated vector Xt,a t is an unbiased estimate of x t with a smallvariance.
Revised: A } h x ✓ i X { X

Revised: ( ✓

Revised: ✓ 2 ⇥

Revised: , ˆ r 0 = 0 .

Revised: 3: Let X  1 be an inverse of X ? .

Revised: 4: for t = 1 : T do 5:

Revised: Central learner : 6:

Revised: Receive ˆ r t  1 and provide it to ⇤ .

Revised: 7: ⇤ , using the history ( x 1 , ˆ r

Revised: , x t

Revised: 1 , ˆ r t  1 ) , selects x t .

Revised: Send ˆ ✓

Revised: X to agent.

Revised: Agent : 10:

Revised: Receive ˆ t from the central learner.

Revised: Observe context realization { X t,a } a 2 A .

Revised: Pull arm

Original:  Given this, we prove that ˆ r t satisﬁes (6), and thus the rewards observed by Λ are generatedaccording to a linear bandit model with unknown parameter that is the same as θ (cid:63) .
Revised: = arg max a 2 A h X t,a , t i and receive t . 13: Send ˆ r t = SQ r t ) to the central learner using 1-bit. show that conditioned on x t , the associated vector X t,a t is an unbiased estimate of x t with a small variance. Given this, we prove that ˆ r t satisﬁes (6), and thus the rewards observed by ⇤ are generated according to a linear bandit model with unknown parameter that is the same as ✓ ? .

Original: We next decompose the difference
Revised: We next decompose the difference

Original: R T − R T (Λ) to two terms:

Original: [Equation]

Original: T
Revised: R

Original: [Equation]
Revised: [Equation]

Revised: to two terms:

Original: [Equation]
Revised: [Equation]

Original: To bound the ﬁrst term, we show that the unbiasdness property togetherwith Assumption 1 implies that Σ T is a martingale with bounded difference.
Revised: To bound the ﬁrst term, we show that the unbiasdness property together with Assumption 1 implies that ⌃ T is a martingale with bounded difference.

Original: This implies that | Σ T | =
Revised: This implies that | ⌃ T | =

Original: O ( √ T log T ) with high probability.
Revised: O ( p T log T ) with high probability.

Original: To bound Σ (cid:48) T , we ﬁrst show thatarg max x ∈X (cid:104) x, θ (cid:63) (cid:105) =
Revised: To bound ⌃ 0 T , we ﬁrst show that arg max x 2 X h x, ✓ ? i =

Original: X (cid:63) ( θ (cid:63) )
Revised: X ? ( ✓ ? )

Original: (we note that this is why the algorithm converges to ˆ θ t that is equalto, or results in the same expected reward as, θ (cid:63) ).
Revised: (we note that this is why the algorithm converges to ˆ ✓ t that is equal to, or results in the same expected reward as, ✓ ? ).

Original: Then, following a similar approach, we can showthat Σ (cid:48) T is a martingale with bounded difference which implies that | Σ (cid:48) T | =
Revised: Then, following a similar approach, we can show that ⌃ 0 T is a martingale with bounded difference which implies that | ⌃ 0 T | =

Original: O ( √ T log T ) with highprobability.
Revised: O ( p T log T ) with high probability.

Original: [Equation]
Revised: ⇤

Original: Downlink Communication.
Revised: Downlink Communication.

Original: The downlink cost of our scheme is O ( d ) (see App.
Revised: The downlink cost of our scheme is O ( d ) (see App.

Original: B for discussion).
Revised: A for discussion).

Original: Operation Complexity.
Revised: Operation Complexity.

Original: The main complexity that our algorithm adds beyond the complexity of Λ ,is the computation of the function X (cid:63) .
Revised: The main complexity that our algorithm adds beyond the complexity of ⇤ , is the computation of the function X ? .

Revised: The time-complexity of X ⇤ ( ✓ ) depends on the context distribution.

Revised: While computing X ⇤ ( ✓ ) can be computationally expensive in worst-case scenarios, it can be computed/approximated efﬁciently for many practical distributions even in a closed form.

Revised: We give the following examples:

Revised: • For d = 1 , ✓ > 0 , we have that X ?

Original: For many practical distributions P a  , this can be calculatedefﬁciently.
Revised: ( ✓ ) is the expectation of the maximum of multiple random variables, i.e., X ? ( ✓ ) = E x a ⇠ P a [max a 2 A x a ] , which can be computed/approximated efﬁciently if the distributions P a are given in a closed form. • If { P a } a 2 A are continuous distributions, then X ⇤ ( ✓ ) , ✓ 6 = 0 can be expressed as

Revised: [Equation]

Revised: For many distributions, the previous expression can be computed/approximated efﬁciently.

Original: We discuss this in more details in App.
Revised: For instance, consider the case where d  1 , x a are independent, identically distributed d -dimensional Gaussian vectors with mean µ and covariance matrix ⌃ = U T DU , where D is a diagonal matrix and U is upper triangular.

Original: B.
Revised: The expectation in (11) is equal to ( Q ( h x a  µ, ✓ i kp DU ✓ k 2 )) |A|  1 , where Q ( c ) = 1 p 2 ⇡ R 1 c exp(  12 x 2 ) dx . Hence, X ⇤ ( ✓ ) can be approximated efﬁciently in that case. • For discrete distributions, X ⇤ ( ✓ ) can be computed efﬁciently depending on the number of mass points of the distribution and if the distribution has structures/properties to simplify the expression.

Original: Imperfect Knowledge of Distributions.
Revised: Imperfect Knowledge of Distributions.

Original: Since we only use the distributions to calculate X (cid:63) ,imperfect knowledge of distribution only affects us in the degree that it affects the calculation of X (cid:63) .
Revised: Since we only use the distributions to calculate X ? , imperfect knowledge of distribution only affects us in the degree that it affects the calculation of X ? .

Original: Suppose that we have an estimate ˜ X (cid:63) of X (cid:63) that satisﬁes
Revised: Suppose that we have an estimate ˜ X ? of X ? that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Using Theorem 1 we prove in App.
Revised: Using Theorem 1 we prove in App.

Original: B the following corollary.
Revised: A the following corollary.

Original: Corollary 1.
Revised: Corollary 1.

Original: Suppose we are given ˜ X (cid:63) that satisﬁes (11).
Revised: Suppose we are given ˜ X ? that satisﬁes (12).

Original: Then, there exists an algorithm Λ forwhich Algorithm 1 achieves R T =
Revised: Then, there exists an algorithm ⇤ for which Algorithm 1 achieves R T =

Original: ˜ O( d √ T
Revised: ˜ O

Original:  + (cid:15)T √ d ) with probability at least 1 − 1 T .
Revised: ( d p T + ✏ T p d ) with probability at least 1  1 T .

Original: Privacy.
Revised: Privacy.

Original: Our result may be useful for applications beyond communication efﬁciency; indeed, thecontext may contain private information (e.g., personal preferences, ﬁnancial information, etc); useof our algorithm enables to not share this private information at all with the central learner, withoutimpeding the learning process.
Revised: Our result may be useful for applications beyond communication efﬁciency; indeed, the context may contain private information (e.g., personal preferences, ﬁnancial information, etc); use of our algorithm enables to not share this private information at all with the central learner, without impeding the learning process.

Original: Surprisingly, work in [48], motivated from privacy considerations, hasshown that if an agent adds a small amount of zero mean noise to the true context before sending it tothe central learner, this can severely affect the regret in some cases - and yet our algorithm essentiallyenables to “guess” the context with no regret penalty if the distributions are known.
Revised: Surprisingly, work in [48], motivated from privacy considerations, has shown that if an agent adds a small amount of zero mean noise to the true context before sending it to the central learner, this can severely affect the regret in some cases - and yet our algorithm essentially enables to “guess” the context with no regret penalty if the distributions are known.

Original: Although addinga zero mean noise to the observed feature vector conveys an unbiased estimate of the observation, thedifference between this and our case is technical and mainly due to the fact that the unbiasdness isrequired to hold conditioned on the central learner observation (noisy context); see App.
Revised: Although adding a zero mean noise to the observed feature vector conveys an unbiased estimate of the observation, the difference between this and our case is technical and mainly due to the fact that the unbiasdness is required to hold conditioned on the central learner observation (noisy context).

Original: B for moredetails.
Revised: Note that we do not make formal privacy claims in this paper, but simply observe that our approach could potentially be leveraged for privacy purposes. It is true that the reward can reveal some information about the context, e.g., if all the actions result in small reward for context and large reward for another context. However, privatizing the reward (which implies a private context in our case) is much easier than privatizing the context and there are many proposed optimal algorithms with little to no regret loss, e.g., see [20, 38, 44, 35]. This is not the case when privatizing the context.

Original: We also discuss potential uses of our algorithm in preserving the context  privacy in App B.
Revised: In fact it was shown in [42] that privatizing the context can lead to linear regret and relaxed deﬁnitions of privacy are proposed to avoid this.

Original:  Contextual Linear Bandits with Unknown Context Distribution
Revised:  Contextual Linear Bandits with Unknown Context Distribution

Original: We now consider the case where the learner does not know the context distributions, and thus
Revised: We now consider the case where the learner does not know the context distributions, and thus

Original: Algorithm 1 that uses zero bits for the context cannot be applied.
Revised: Algorithm 1 that uses zero bits for the context cannot be applied.

Original: In this case, related literatureconjectures a lower bound of Ω( d ) [46, 47] – which is discouraging since it is probably impossible toestablish an algorithm with communication logarithmically depending on d .
Revised: In this case, related literature conjectures a lower bound of ⌦ ( d ) [46, 47] – which is discouraging since it is probably impossible to establish an algorithm with communication logarithmically depending on d .

Original: Additionally, in practicewe use 32 d bits to convey full precision values - thus this conjecture indicates that in practice we maynot be able to achieve order improvements in terms of bits communicated, without performance loss.
Revised: Additionally, in practice we use 32 d bits to convey full precision values - thus this conjecture indicates that in practice we may not be able to achieve order improvements in terms of bits communicated, without performance loss.

Original: In this section, we provide Algorithm 2 that uses ≈ 5 d bits per context and achieves (optimal) regret
Revised: In this section, we provide Algorithm 2 that uses ⇡ 5 d bits per context and achieves (optimal) regret

Original: R T=
Revised: R T

Original:  O ( d √ T log T ) .
Revised: = O ( d p T log T ) .

Original: We believe Algorithm 2 is interesting for two reasons:1.
Revised: We believe Algorithm 2 is interesting for two reasons:

Original: In theory, we need an inﬁnite number of bits to convey full precision values- we prove that aconstant number of bits per dimension per context is sufﬁcient.
Revised: In theory, we need an inﬁnite number of bits to convey full precision values- we prove that a constant number of bits per dimension per context is sufﬁcient.

Original: Previously best-known algorithms,which rely on constructing 1 /T -net
Revised: Previously best-known algorithms, which rely on constructing 1 /T -net

Original: for the set of feature vectors, use O ( d log T ) bits per context,which goes to inﬁnity as T goes to inﬁnity.
Revised: for the set of feature vectors, use O ( d log T ) bits per context, which goes to inﬁnity as T goes to inﬁnity.

Original: Moreover, these algorithms require exponential complexity[24] while ours is computationally efﬁcient.2.
Revised: Moreover, these algorithms require exponential complexity [24] while ours is computationally efﬁcient.

Original: In practice, especially for large values of d , reducing the number of bits conveyed from 32 d to ≈ 5 dis quite signiﬁcant - this is a reduction by a factor of six, which implies six times less communication.
Revised: In practice, especially for large values of d , reducing the number of bits conveyed from 32 d to ⇡ 5 d is quite signiﬁcant - this is a reduction by a factor of six, which implies six times less communication.

Original: Main Idea.
Revised: Main Idea.

Original: The intuition behind Algorithm 2 is the following.
Revised: The intuition behind Algorithm 2 is the following.

Original: The central learner is going to usean estimate of the d × d least-squares matrix V t =
Revised: The central learner is going to use an estimate of the d ⇥ d least-squares matrix V t =

Original: (cid:80) ti =
Revised: P ti =

Original: 1 X i,a i X Ti,a i to update her estimates for theparameter vector θ (cid:63) .
Revised: X i,a i X Ti,a i to update her estimates for the parameter vector ✓ ? .

Original: Thus, when quantizing the vector X t,a , we want to make sure that not only thisvector is conveyed with sufﬁcient accuracy, but also that the central learner can calculate the matrix
Revised: Thus, when quantizing the vector X t,a , we want to make sure that not only this vector is conveyed with sufﬁcient accuracy, but also that the central learner can calculate the matrix

Original: V t accurately.
Revised: V t accurately.

Original: In particular, we would like the central learner to be able to calculate an unbiasedestimator for each entry of X t,a and each entry of the matrix V t .
Revised: In particular, we would like the central learner to be able to calculate an unbiased estimator for each entry of X t,a and each entry of the matrix V t .

Original: Our algorithm achieves this byquantizing the feature vectors X t,a t , and also the diagonal (only the diagonal) entries of the leastsquares matrix V t .
Revised: Our algorithm achieves this by quantizing the feature vectors X t,a t , and also the diagonal (only the diagonal) entries of the least squares matrix V t .

Original: We prove that by doing so, with only ≈ 5 d bits we can provide an unbiasedestimate and guarantee an O ( 1 √ d ) quantization error for each entryin the matrix almost surely.
Revised: We prove that by doing so, with only ⇡ 5 d bits we can provide an unbiased estimate and guarantee an O ( 1 p d ) quantization error for each entry

Revised: in the matrix almost surely.

Original: Quantization Scheme.
Revised: Quantization Scheme.

Original: We here describe the proposed quantization scheme
Revised: We here describe the proposed quantization scheme.

Original: • To quantize X t,a t : Let m (cid:44) (cid:100)√ d (cid:101) .
Revised: • To quantize X t,a t : Let m , d p d e .

Original: We ﬁrst send the sign of each coordinate of X t,a t using d bits,namely, we send the vector s t =
Revised: We ﬁrst send the sign of each coordinate of X t,a t using d bits, namely, we send the vector s t =

Original: X t,a t / | X t,a t | .
Revised: X t,a t / | X t,a t | .

Original: To quantize the magnitude | X t,a t | , we scale eachcoordinate of | X t,a t | by m and quantize it using a Stochastic Quantizer (SQ) 3 with m + 1 levels inthe interval [0 , m ] .
Revised: To quantize the magnitude | X t,a t | , we scale each coordinate of | X t,a t | by m and quantize it using a Stochastic Quantizer (SQ) 3 with m + 1 levels in 3 As described in (3) in Section 2, SQ maps value x to an integer value, namely b x c with probability d x e  x and d x e with probability x  b x c . the interval [0 , m ] .

Original: Let X t (cid:44) SQ m ( m | X t,a t | ) denote the resulting SQ outputs, we note that X t takesnon-negative integer values and lies in a norm-1 ball of radius 2 d (this holds since the original vectorlies in a norm-2 ball of radius 1 and the error in each coordinate is at most 1 /m ).
Revised: Let X t , SQ m ( m | X t,a t | ) denote the resulting SQ outputs, we note that X t takes non-negative integer values and lies in a norm-1 ball of radius 2 d (this holds since the original vector lies in a norm-2 ball of radius 1 and the error in each coordinate is at most 1 /m ).

Original: That is, it holds thatX t ∈ Q
Revised: That is, it holds that X t 2 Q

Original: = { x ∈ N d |(cid:107) x (cid:107) 1 ≤ 2 d } .
Revised: = { x 2 N d | k x k 1  2 d } .

Original: We then use any enumeration h : Q → [ |Q| ] of this set to encodeX t using log( |Q| ) bits.• To quantize X t,a t X Tt,a t : Let X 2 t,a t denote a vector that collects the diagonal entries of X t,a t X Tt,a t .
Revised: We then use any enumeration h : Q ! [ |Q| ] of this set to encode X t using log( |Q| ) bits. • To quantize X t,a t X Tt,a t : Let X 2 t,a t denote a vector that collects the diagonal entries of X t,a t X Tt,a t .

Original: Let ˆ X t (cid:44) s t X t /m be the estimate of X t,a t that the central learner retrieves.
Revised: Let ˆ X t , s t X t /m be the estimate of X t,a t that the central learner retrieves.

Original: Note that ˆ X 2 t is notan unbiased estimate of X 2 t,a t ; however, ( X 2 t,a t − ˆ X 2 t ) i ≤ 3 /m for all coordinates i (proved in App. C).
Revised: Note that ˆ X 2 t is not an unbiased estimate of X 2 t,a t ; however, ( X 2 t,a t  ˆ X 2 t ) i  3 /m for all coordinates i (proved in App. B).

Original: Our scheme simply conveys the difference X 2 t,a t − ˆ X 2 t with 1 bit per coordinate using a SQ [ − 3 /m, 3 /m ]quantizer.
Revised: Our scheme simply conveys the difference X 2 t,a t  ˆ X 2 t with 1 bit per coordinate using a SQ [  3 /m, 3 /m ]quantizer.

Original: The central learner and distributed agent operations are presented in Algorithm 2.
Revised: The central learner and distributed agent operations are presented in Algorithm 2.

Original: Example 2.
Revised: Example 2.

Original: Consider the case where d = 5 .
Revised: Consider the case where d = 5 .

Original: Then each coordinate of | X t,a t | is scaled by 3 andquantized using SQ 3 to one of the values 0 , 1 , 2 , 3 to get X t .
Revised: Then each coordinate of | X t,a t | is scaled by 3 and quantized using SQ 3 to one of the values 0 , 1 , 2 , 3 to get X t .

Original: The function h then maps the values for
Revised: The function h then maps the values for

Original: X t that satisfy the (cid:107) X t (cid:107) 1 ≤ 10 to a unique value (a code) in the set [ |Q| ] .
Revised: X t that satisfy the k X t k 1  10 to a unique value (a code) in the set [ |Q| ] .

Original: For instance the value 3 . 1is not given a code, where 1 is the vector of all ones.
Revised: For instance the value 3 . 1 is not given a code, where 1 is the vector of all ones.

Original: However, note that for | X t,a t | to be mapped to3 . 1 , we must have 3 | ( X t,a t ) i | ≥ 2 for all coordinates i , which cannot happen since it implies that(cid:107) X t,a t (cid:107) 2 ≥ 2 (cid:112) 5 / 6 > 1 which contradicts Assumption 1. Algorithm Performance.
Revised: However, note that for | X t,a t | to be mapped to 3 .

Original: Theorem 2, stated next, holds under Assumption 1 in Section 2 and someadditional regulatory assumptions on the distributions P a provided in Assumption 2.

Original: Assumption 2.

Original: There exist constants c, c (cid:48) such that for any sequence θ 1 , ..., θ T , where θ t dependsonly on H t , with probability at least 1 − c (cid:48) T , it holds that

Original: [Equation]

Original: where a t = arg max a ∈A (cid:104) X t,a , θ t (cid:105) , and I is the identity matrix.

Original: We note that several common assumptions in the literature imply (12), for example, boundedeigenvalues for the covariance matrix of X t,a t [11, 27, 17].

Original: Such assumptions hold for a widerange of distributions, including subgaussian distribitions with bounded density [36].

Original: We discuss thechallenge in relaxing this assumption in App. C.

Original: Theorem 2. Algorithm 2satisﬁes that for all t : X  t ∈ Q ; and B t ≤ 1 + log 2 (2 d + 1) + 5 . 03 d bits. Under assumptions 1, 2, it achieves a regret R T
Revised: 1 , we must have| ( X t,a t ) i |  2 for all coordinates i , which cannot happen since it implies that k X t,a t k 2 p 5 / 6 > 1 which contradicts Assumption 1.

Original: = O ( d √ T log T )

Original: with probability at least 1 − 1 T .

Original: Proof Outline.

Original: To bound the number of bits B t , we ﬁrst bound the size of Q by formulating astandard counting problem: we ﬁnd the number of non-negative integer solutions for a linear equation. To bound the regret R T , we start by proving that our quantization scheme guarantees some desirableproperties, namely, unbiasedness and O ( 1 √ d ) quantization error for each vector coordinate. We thenupper bound the regret in terms of (cid:107) ˆ θ t − θ (cid:63) (cid:107) 2 and show that this difference can be decomposed as(cid:107) [Equation]
Revised: Algorithm

Original: 1: Input: underlying set of actions A , and time horizon T .
Revised: 1: Input: underlying set of actions A , and time horizon T .

Original: 2: ˆ θ 0 = 0 , ˜ V 

Revised: 2: ˆ ✓ 0 = 0 , ˜ V

Original: = 0 , u 0 = 0 , m = 
Revised: = 0 , u 0 = 0 , m = dp

Original: (cid:100)√ d (cid:101) .
Revised: d e .

Original: 3: Let h be an enumeration of the set Q = { x ∈ N d |(cid:107) x (cid:107) 1 ≤ 2 d } .
Revised: 3: Let h be an enumeration of the set Q = { x 2 N d | k x k 1  2 d } .

Original: 4: for t = 1 : T do 5:
Revised: 4: for t = 1 : T do 5:

Original: Agent : 6:
Revised: Agent : 6:

Original: Receive ˆ θ t − 1 from the central learner.
Revised: Receive ˆ ✓ t  1 from the central learner.

Original: Observe context realization { X t,a } a ∈A .
Revised: Observe context realization { X t,a } a 2 A .

Original: Pull arm a t = arg max a ∈A (cid:104) X t,a , ˆ θ t − 1 (cid:105) and receive reward r t . 
Revised: Pull arm a t = arg max a 2 A h X t,a , ˆ ✓ t  1 i and receive reward r t . 9:

Original: Compute the signs s t = X t,a t / | X t,a t | of X t,a t . 10:
Revised: Compute the signs s t = X t,a t / | X t,a t | of X t,a t . 10:

Original: Let X t =
Revised: Let X t =

Original: SQ m ( m | X t,a t | ) .
Revised: SQ m ( m | X t,a t | ) .

Original: 11: e 2 t = SQ [ − 3 /m, 3 /m ]( X 2 t,a t − ˆ X 2 t ) , where ˆ X t = s t X t /m .
Revised: 11: e 2 t = SQ [  3 /m, 3 /m ]( X 2 t,a t  ˆ X 2 t ) , where ˆ X t = s t X t /m .

Original: Send to the central learner h ( X t ) , s t , and e 2 t using log 2 ( |Q| ) , d , and d bits, respectively.
Revised: Send to the central learner h ( X t ) , s t , and e 2 t using log 2 ( |Q| ) , d , and d bits, respectively.

Original: Send ˆ r t = SQ 1 ( r t ) using 1-bit.
Revised: Send ˆ r t = SQ 1 ( r t ) using 1-bit.

Original: Central learner : 15:
Revised: Central learner : 15:

Original: Receive X t , s t , e 2 t , and ˆ r t from the distributed agent.
Revised: Receive X t , s t , e 2 t , and ˆ r t from the distributed agent.

Original: 16: ˆ X t = s t X t /m , ˆ X ( D ) t
Revised: [Equation] [Equation] 17: u t

Original: = ˆ X 2 t + e 2 t .
Revised: [Equation]

Original: 17: u t ← u t − 1 + ˆ r t ˆ X t . 18: ˜V t ← ˜ V t − 1 + ˆ X t ˆ X
Revised: 18: ˜

Original: Tt − diag ( ˆ X t ˆ X Tt ) + diag ( ˆ X ( D ) t ) .
Revised: [Equation]

Original: 19: ˆ θ t ← ˜ V − 1 t u t .
Revised: 19: ˆ ✓ [Equation]

Original: Send ˆ θ t to the next agent.
Revised: Send ˆ ✓ t to the next agent.

Revised: Algorithm Performance.

Revised: Theorem 2, stated next, holds under Assumption 1 in Section 2 and some additional regulatory assumptions on the distributions P a provided in Assumption 2.

Revised: Assumption 2.

Original:  where E t captures the error in estimating the matrix X t,a t X Tt,a t , e t is the error in estimating X t,at , andη (cid:48) t  is a noise that satisﬁes the same properties as η t .
Revised: There exist constants c, c 0 such that for any sequence ✓ 1 , ..., ✓ T , where ✓ t depends only on H t , with probability at least 1  c 0 T , it holds that [Equation] where a t = arg max a 2 A h X t,a , ✓ t i , and I is the identity matrix. We note that several common assumptions in the literature imply (13), for example, bounded eigenvalues for the covariance matrix of X t,a t [11, 27, 17].

Revised: Such assumptions hold for a wide range of distributions, including subgaussian distribitions with bounded density [36].

Revised: Challenge in relaxing assumption 2 (diversity assumption).

Revised: The main challenge in relaxing the diversity assumption for LinUCB (or Thompson sampling) based algorithms is that the regret of those algorithms is bounded as ˜ O

Revised: ( p T k ˆ ✓ T  ✓ ⇤ k V T ) .

Revised: Without quantization, the quantity k ˆ ✓ T  ✓ ⇤ k V T grows slowly and is nearly a constant; however, without the diversity assumption, the quantization error can make k ˆ ✓ T  ✓ ⇤ k 2 V T to grow as p T in the worst case.

Revised: This is due to the fact that sub-optimal arms do not have large number of pulls, hence, we do not have good estimate of ✓ ?

Revised: on those direction; on the other hand, the quantization errors in estimating V T is accumulated in all directions.

Revised: As a result, the regret bound increases by a factor of T 1 / 4 .

Revised: We leave it as a future work to either relax the diversity assumption (which is required in our paper only in the case of unknown context distribution) or else show that removing it will unavoidably increase the regret order.

Revised: Theorem 2.

Revised: Algorithm 2 satisﬁes that for all t : X t 2 Q ; and B t  1 + log 2 (2 d + 1) + 5 .

Revised: 03 d bits.

Revised: Under assumptions 1, 2, it achieves a regret R T

Revised: = O

Revised: ( d p T log T ) with probability at least 1  1 T .

Revised: Proof Outline.

Revised: To bound the number of bits B t , we ﬁrst bound the size of Q by formulating a standard counting problem: we ﬁnd the number of non-negative integer solutions for a linear equation.

Revised: To bound the regret R T , we start by proving that our quantization scheme guarantees some desirable properties, namely, unbiasedness and O ( 1 p d ) quantization error for each vector coordinate.

Revised: We then upper bound the regret in terms of k ˆ ✓ t  ✓ ?

Revised: k 2 and show that this difference can be decomposed as

Revised: [Equation]

Revised: [Equation]

Revised: where E t captures the error in estimating the matrix X t,a t X Tt,a t , e t is the error in estimating X t,a t , and ⌘ 0 t is a noise that satisﬁes the same properties as ⌘ t .

Original: Using Assumption 2, we prove that V − 1 t grows as
Revised: Using Assumption 2, we prove that V  1 t grows as

Original: O ( dt ) with high probability, and from the unbiasdness and boundedness of all error quantities we showthat they grow as O ( √ t log t ) with high probability.
Revised: O ( dt ) with high probability, and from the unbiasdness and boundedness of all error quantities we show that they grow as O ( p t log t ) with high probability.

Original: This implies that (cid:107) ˆ θ t − θ (cid:63) (cid:107) 

Revised: This implies that k ˆ ✓ t  ✓ ? k

Original: = O ( d (cid:113) log t t ) ,and hence, R T
Revised: = O ( d q log t t ) , and hence, R T

Original: = O ( d √ T log T ) .
Revised: = O ( d p T log T ) .

Original: The complete proof is provided in App.
Revised: The complete proof is provided in App.

Original: C.
Revised: B.

Original: (cid:3)
Revised: ⇤

Original: Algorithm Complexity.
Revised: Algorithm Complexity.

Original: If we do not count the quantization operations, it is easy to see that thecomplexity of the rest of the algorithm is dominated by the complexity of computing V − 1 t which canbe done in O ( d 2 . 373 ) [4].
Revised: If we do not count the quantization operations, it is easy to see that the complexity of the rest of the algorithm is dominated by the complexity of computing V  1 t which can be done in O ( d 2 . 373 ) [4].

Original: For the quantization, we note that each coordinate of X t can be computedin ˜ O
Revised: For the quantization, we note that each coordinate of X t can be computed in ˜ O

Original: (1) time 4 .
Revised: (1) time 4 .

Original: Moreover, the computation of h ( x ) for x ∈ Q can be done in constant time withhigh probability using hash tables, where h is the enumeration function in Step 3.
Revised: Moreover, the computation of h ( x ) for x 2 Q can be done in constant time with high probability using hash tables, where h is the enumeration function in Step 3.

Original: Hence, the addedcomputational complexity is almost linear in d .
Revised: Hence, the added computational complexity is almost linear in d .

Original: Although a hash table for h can consume Ω(2 5 d )memory, by sacriﬁcing a constant factor in the number of bits, we can ﬁnd enumeration functionsthat can be stored efﬁciently.
Revised: Although a hash table for h can consume ⌦ (2 5 d ) memory, by sacriﬁcing a constant factor in the number of bits, we can ﬁnd enumeration functions that can be stored efﬁciently.

Original: As an example, consider the scheme in [14] that can ﬁnd an one-to-onefunction h : Q → N + which can be stored and computed efﬁciently, but only gives guarantees inexpectation that E [log( h ( x ))]
Revised: As an example, consider the scheme in [14] that can ﬁnd an one-to-one function h : Q ! N + which can be stored and computed efﬁciently, but only gives guarantees in expectation that E [log( h ( x ))]

Original: = O ( d ) for all x ∈ Q .
Revised: = O ( d ) for all x 2 Q .

Original: Downlink Communication Cost.
Revised: Downlink Communication Cost.

Original: Although we assume no-cost downlink communication, as wasalso the case for Algorithm 1, the downlink in Algorithm 2 is only used to send the updated parametervector ˆ θ t to the agents.
Revised: Although we assume no-cost downlink communication, as was also the case for Algorithm 1, the downlink in Algorithm 2 is only used to send the updated parameter vector ˆ ✓ t to the agents.

Original: If desired, these estimates can be quantized using the same method as for
Revised: If desired, these estimates can be quantized using the same method as for

Original: X t,a t , which (following a similar proof to that of Theorem 2) can be shown to not affect the order ofthe regret while reducing the downlink communication to ≈ 5 d bits per iteration.
Revised: X t,a t , which (following a similar proof to that of Theorem 2) can be shown to not affect the order of the regret while reducing the downlink communication to ⇡ 5 d bits per iteration.

Original: Ofﬂoading To Agents.
Revised: Ofﬂoading To Agents.

Original: For applications where the agents wish to computationally help the centrallearner, the central learner may simply aggregate information to keep track of u t , ˜ V t and broadcastthese values to the agents; the estimate ˆ θ t can be calculated at each agent.
Revised: For applications where the agents wish to computationally help the central learner, the central learner may simply aggregate information to keep track of u t , ˜ V t and broadcast these values to the agents; the estimate ˆ ✓ t can be calculated at each agent.

Original: Moving the computationalload to the agents does not affect the regret order or the number of bits communicated on the uplink.
Revised: Moving the computational load to the agents does not affect the regret order or the number of bits communicated on the uplink.

Original: Remark 2.
Revised: Remark 2.

Original: Under the regulatory assumptions in [17], the regret bound can be improved by a factorof (cid:112) log( K ) /d , where K = |A| is the number of actions.
Revised: Under the regulatory assumptions in [17], the regret bound can be improved by a factor of p log( K ) /d , where K = |A| is the number of actions.

Original: However, this does not improve the regretin the worst case as the worst case number of actions is O ( C d ) , C > 1 [24].Checklist
Revised: However, this does not improve the regret in the worst case as the worst case number of actions is O ( C d ) , C > 1 [24].

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [Yes] SeeApp. A.
Revised: Societal Impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Theorem 1and Theorem 2.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See App.

Original: B andApp.

Original: C.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [N/A]
Revised: Results in this work can be used in decision making systems which can potentially lead to biased decisions against racial, sex, or minority groups if used without care.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [N/A] If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: Limitations.

Original: (a) If your work uses existing assets, did you cite the creators? [N/A] (b) Did you mention the license of the assets? [N/A]
Revised: The results in this paper apply under Assumption 1 (and Assumption 2 for Section 4).

Original: (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: Relaxing those assumptions is an interesting future direction.

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]

Original: information or offensive content?

Original: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A] [Table]
Revised: Acknowledgment.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review [Table] Board (IRB) approvals, if applicable? [N/A]
Revised: LY is supported in part by DARPA grant HR00112190130, NSF Award 2221871.

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?

Original: [N/A]