Original: Learning in Distributed  Contextual Linear Bandits Without Sharing the Context
Revised: Learning from Distributed Users in Contextual Linear Bandits Without Sharing the Context

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Contextual linear bandits is a rich and theoretically important model that has manypractical applications.
Revised: Contextual linear bandits is a rich and theoretically important model that has manypractical applications.

Original: Recently, this setup gained a lot of interest in applicationsover wireless where communication constraints can be a performance bottleneck,especially when the contexts come from a large d -dimensional space.
Revised: Recently, this setup gained a lot of interest in applicationsover wireless where communication constraints can be a performance bottleneck,especially when the contexts come from a large d -dimensional space.

Original: In this paper,we consider a distributed memoryless contextual linear bandit learning problem,where the agents who observe the contexts and take actions are geographicallyseparated from the learner who performs the learning while not seeing the contexts.
Revised: In this paper,we consider a distributed memoryless contextual linear bandit learning problem,where the agents who observe the contexts and take actions are geographicallyseparated from the learner who performs the learning while not seeing the contexts.

Original: We assume that contexts are generated from a distribution and propose a methodthat uses ≈ 5 d bits per context for the case of unknown context distribution and0 bits per context if the context distribution is known, while achieving nearly thesame regret bound as if the contexts were directly observable.
Revised: We assume that contexts are generated from a distribution and propose a methodthat uses ≈ 5 d bits per context for the case of unknown context distribution and0 bits per context if the context distribution is known, while achieving nearly thesame regret bound as if the contexts were directly observable.

Original: The former boundimproves upon existing bounds by a log( T ) factor, where T is the length of thehorizon, while the latter achieves information theoretical tightness.
Revised: The former boundimproves upon existing bounds by a log( T ) factor, where T is the length of thehorizon, while the latter achieves information theoretical tightness.

Original:  Introduction
Revised:  Introduction

Original: Contextual linear bandits offer a sequential decision-making framework that combines fundamentaltheoretical importance with signiﬁcant practical popularity [7], as it offers a tractable way to captureside information (context), as well as a potentially inﬁnite set of decisions (actions).
Revised: Contextual linear bandits offer a sequential decision-making framework that combines fundamentaltheoretical importance with signiﬁcant practical popularity [8], as it offers a tractable way to captureside information (context), as well as a potentially inﬁnite set of decisions (actions).

Original: The most prominent application is in recommendation systems [23], but it has also been used in applications suchas virtual support agents [29], clinical trials [11], transportation systems [8], wireless optimization[21, 20], health [9], robotics [24] and online education [26].
Revised: The most prominent application is in recommendation systems [30], but it has also been used in applications suchas virtual support agents [39], clinical trials [12], transportation systems [9], wireless optimization[26, 25], health [10], robotics [31] and online education [34].

Original: In this paper, we develop algorithms that support the deployment of contextual linear bandits indistributed settings.
Revised: In this paper, we develop algorithms that support the deployment of contextual linear bandits indistributed settings.

Original: In particular, we consider the case where a central learner wishes to solve acontextual linear bandit problem with the help of transient agents.
Revised: In particular, we consider the case where a central learner wishes to solve acontextual linear bandit problem with the help of transient agents.

Original: That is, we assume that the agentsdo not keep memory of past actions and may not be present for the whole duration of learning;learning in our setup can happen thanks to the persistent presence of the central learner.
Revised: That is, we assume that the agentsdo not keep memory of past actions and may not be present for the whole duration of learning;learning in our setup can happen thanks to the persistent presence of the central learner.

Original: We view thecentral learner as a “knowledge repository”, that accumulates knowledge from the experience of thetransient agents and makes it available to next agents.
Revised: We view thecentral learner as a “knowledge repository”, that accumulates knowledge from the experience of thetransient agents and makes it available to next agents.

Original: The central agent, through the information itkeeps, could help passing by devices decide how to perform an action, for example: passing by dronesdecide how to perform a manoeuver; agricultural robots decide what amounts of substances such aspesticids to release; and passing by mobile devices decide which local restaurants to recommend.
Revised: The central learner, through the information itkeeps, could help passing by devices decide how to perform an action, for example: passing by dronesdecide how to perform a manoeuver; agricultural robots decide what amounts of substances such aspesticids to release; and passing by mobile devices decide which local restaurants to recommend.

Original: The main challenge we try to address in this paper is the efﬁcient communication of the context theagents experience.
Revised: The main challenge we try to address in this paper is the efﬁcient communication of the context theagents experience.

Original: More speciﬁcally, in our setup, each time an agent joins, she receives from thecentral learner information on the system, such as current estimates of the system parameters; sheobserves her current context, selects and plays an action and collects the corresponding reward.
Revised: More speciﬁcally, in our setup, each time an agent joins, she receives from thecentral learner information on the system, such as current estimates of the system parameters; sheobserves her current context, selects and plays an action and collects the corresponding reward.

Original: Notethat although the distributed agent knows her context, the action she plays and the observed reward,the central learner does not - and needs this information to update its estimate of the system parameters.
Revised: Notethat although the distributed agent knows her context, the action she plays and the observed reward,the central learner does not - and needs this information to update its estimate of the system parameters.

Original: The context in particular can be communication heavy - in the examples we mentioned before, fordrones the context could be their navigation capabilities, physical attributes, and enviromental factorssuch as wind speed; for agricultural robots, it could be images that indicate state of plants and sensormeasurements such as of soil consistency; for restaurant recommendations, it could be the personaldietary preferences and restrictions, budget, and emotional state.
Revised: The context in particular can be communication heavy - in the examples we mentioned before, fordrones the context could be their navigation capabilities, physical attributes, and enviromental factorssuch as wind speed; for agricultural robots, it could be images that indicate state of plants and sensormeasurements such as of soil consistency; for restaurant recommendations, it could be the personaldietary preferences and restrictions, budget, and emotional state.

Original: Moreover, because of geographicalseparation, the central agent may not have any other way to learn the context beyond communication.
Revised: Moreover, because of geographicalseparation, the central learner may not have any other way to learn the context beyond communication.

Original: Unlike the reward, that is usually a single scalar value, the context can be a vector of a large dimensiond from an inﬁnite alphabet, and thus, communicating the context efﬁciently is heavily nontrivial.
Revised: Unlike the reward, that is usually a single scalar value, the context can be a vector of a large dimensiond from an inﬁnite alphabet, and thus, communicating the context efﬁciently is heavily nontrivial.

Original: The technical question we ask is, how many bits do we need to convey per context to solve the linearbandit problem without downgrading the performance as compared to the non-distributed setting?
Revised: The technical question we ask is, how many bits do we need to convey per context to solve the linearbandit problem without downgrading the performance as compared to the non-distributed setting?

Original: In this paper, we design algorithms that support this goal.
Revised: In this paper, we design algorithms that support this goal.

Original: We note that our algorithms optimize theuplink communication (from the agents to the  learner), and assume unlimited (cost-free) downlinkcommunication.
Revised: We note that our algorithms optimizethe uplink communication (from the agents to the central learner), and assume unlimited (cost-free)downlink communication.

Original: This is a standard assumption in wireless [6, 25, 16] for several reasons: uplinkwireless links tend to be much more bandwidth restricted, since several users may be sharing the samechannel; uplink communication may also be battery-powered and thus more expensive to sustain; inour particular case, the agents may have less incentive to communicate (provide their feedback) thanthe learner (who needs to learn).
Revised: This is a standard assumption in wireless [7, 33, 21] for several reasons:uplink wireless links tend to be much more bandwidth restricted, since several users may be sharingthe same channel; uplink communication may also be battery-powered and thus more expensive tosustain; in our particular case, the agents may have less incentive to communicate (provide theirfeedback) than the central learner (who needs to learn).

Original: Having said that, we note that our algorithms (in Sections 3 and 4)make frugal use of the downlink channels, only using them to transmit system parameters.
Revised: Having said that, we note that our algorithms(in Sections 3 and 4) make frugal use of the downlink channels, only using them to transmit systemparameters.

Original: Below we summarize our main contributions:
Revised: Below we summarize our main contributions:

Original: We show the surprising result that, if the central agent knows the distribution of the contexts, wedo not need to communicate the context at all - the agent does not need to send any information onthe actual context she observes and the action she plays.
Revised: We show the surprising result that, if the central learner knows the distribution of the contexts,we do not need to communicate the context at all - the agent does not need to send any information on the actual context she observes and the action she plays.

Original: It is sufﬁcient for the agent to just send 1 bitto convey quantized information on her observed reward and nothing else.
Revised: It is sufﬁcient for the agent to just send 1 bit to convey quantized information on her observed reward and nothing else.

Original: But for this very limitedcommunication, the central learnercan learn a policy that achieves the same order of regret as if
Revised: But for this verylimited communication, the central learner

Original:  full information about the context and reward is received.
Revised: can learn a policy that achieves the same order of regret asif full information about the context and reward is received.

Original: This result holds for nearly all context distributions and it is the best we can hope for - zero bits of communication for the context.
Revised: This result holds for nearly all contextdistributions and it is the best we can hope for - zero bits of communication for the context.2.

Original: If the central agent has no knowledge of the context distribution, we show that ≈ 5 d bits percontext (where d is the context dimension) is sufﬁcient to achieve the same order regret as knowing the context in full precision.
Revised: If the central learner has no knowledge of the context distribution, we show that ≈ 5 d bits percontext (where d is the context dimension) is sufﬁcient to achieve the same order regret as knowingthe context in full precision.

Original: Note that previous algorithms use O ( d log T ) bits per context to achievethe same order regret, where T is the length of the horizon [19], and require exponential complexity 
Revised: Note that previous algorithms, that rely on constructing 1 /T -net for theset of feature vectors, use O ( d log T ) bits per context to achieve the same order regret, where T isthe length of the horizon [24], and require time complexity of O ( T d ) which is exponential in d .

Original: Related Work and Distinction.
Revised: Related Work and Distinction.

Original: Contextual linear bandits is a rich and important model that hasattracted signiﬁcant interest both in theory and applications [7, 19].
Revised: Contextual linear bandits is a rich and important model that hasattracted signiﬁcant interest both in theory and applications [8, 24].

Original: Within this space, our workfocuses on operation under communications constraints in a distributed setting.
Revised: Within this space, our workfocuses on operation under communications constraints in a distributed setting.

Original: There is large body of work focusing on distributed linear contextual bandits settings, but mainlywithin the framework of federated learning, where batched algorithms have been proposed forcommunication efﬁciency [31, 30, 5, 4, 18] that aggregate together observations and parameterlearning across a large number of iterations.
Revised: There is large body of work focusing on distributed linear contextual bandits settings, but mainlywithin the framework of federated learning, where batched algorithms have been proposed forcommunication efﬁciency [43, 41, 6, 5, 23] that aggregate together observations and parameterlearning across a large number of iterations.

Original: This is possible because in federated learning, the agentsthemselves wish to learn the system parameters, remain active playing multiple actions throughout thelearning process, and exchange information with the goal of speeding up their learning [31, 30].
Revised: This is possible because in federated learning, the agentsthemselves wish to learn the system parameters, remain active playing multiple actions throughout thelearning process, and exchange information with the goal of speeding up their learning [43, 41].

Original: Incontrast, in our setup batched algorithms cannot reduce the communication cost because each agentonly plays a single action; this may be because agents are transient, but also because they may not beinterested in learning - this may not be a task that the agents wish to consistently perform - and thusdo not wish to devote resources to it.
Revised: Incontrast, in our setup batched algorithms cannot reduce the communication cost because each agentonly plays a single action; this may be because agents are transient, but also because they may not beinterested in learning - this may not be a task that the agents wish to consistently perform - and thusdo not wish to devote resources to it.

Original: For example, an agent may wish to try a restaurant in a specialoccasion, but would not be interested in sampling multiple restaurants/learning recommendationsystem parameters.
Revised: For example, an agent may wish to try a restaurant in a specialoccasion, but would not be interested in sampling multiple restaurants/learning recommendationsystem parameters.

Revised: In other words, we consider a scenario where the user beneﬁts from receivingan action (or policy) from the central learner, e.g., a recommendation.

Revised: In response, the user givesfeedback to the central learner in terms of (compressed) context/reward.

Revised: The compression operationsbeneﬁt the user by helping reduce her communication cost.

Revised: In principle, the user is not requiredto respond.

Revised: But the central learner will be able to learn whenever there is a feedback; creating anincentive for the user to respond could be an interesting future topic.

Original: Our setup supports a different (and complementary) set of applications than thefederated learning framework, and requires a new set of algorithms that operate without requiringagents to keep memory of past actions.
Revised: Our setup supports a different(and complementary) set of applications than the federated learning framework, and requires a newset of algorithms that operate without requiring agents to keep memory of past actions.

Original: To the best of our knowledge, our framework has not been examined before for linear contextualbandits.
Revised: To the best of our knowledge, our framework has not been examined before for linear contextualbandits.

Original: Work in the literature has examined compression for distributed memoryless MABs [16],but only for rewards (scalar values) and not the contexts (large vectors), and thus these techniquesalso do not extend to our case.
Revised: Work in the literature has examined compression for distributed memoryless MABs [21],but only for rewards (scalar values) and not the contexts (large vectors), and thus these techniquesalso do not extend to our case.

Original: Additional less-closely related work is reviewed in App. A.
Revised: Additional less-closely related work is reviewed in App. A.

Original: Paper organization.
Revised: Paper organization.

Original: Section 2 reviews our notation and problem formulation; Section 3 providesand analyzes our algorithm for known and Section 4 for unknown context distributions.
Revised: Section 2 reviews our notation and problem formulation; Section 3 providesand analyzes our algorithm for known and Section 4 for unknown context distributions.

Original:  Notation and Problem Formulation
Revised:  Notation and Problem Formulation

Original: Notation.
Revised: Notation.

Original: We use the following notation throughout the paper.
Revised: We use the following notation throughout the paper.

Original: For a vector X we use X i or ( X ) i todenote the i -th element of the vector X ; similarly for a matrix V , we use V ij or ( V ) ij to denote theelement at row i , and column j .
Revised: For a vector X we use X i or ( X ) i todenote the i -th element of the vector X ; similarly for a matrix V , we use V ij or ( V ) ij to denote theelement at row i , and column j .

Original: We use (cid:107) V (cid:107) 2 to denote the matrix spectral norm.
Revised: We use (cid:107) V (cid:107) 2 to denote the matrix spectral norm.

Original: For a functionf , we denote its domain and range by dom ( f ) , ran ( f ) respectively.
Revised: For a functionf , we denote its domain and range by dom ( f ) , ran ( f ) respectively.

Original: When dom ( f ) ⊆
Revised: When dom ( f ) ⊆

Original: R , we usef ( X ) for a vector X ∈ R d to denote f ( X ) :=
Revised: R , we usef ( X ) for a vector X ∈ R d to denote f ( X ) :=

Original: [ f ( X 1 ) , ..., f ( X d )] , i.e., the function f is appliedelement-wise; for example we use X 2 to denote the element-wise square of X .
Revised: [ f ( X 1 ) , ..., f ( X d )] , i.e., the function f is appliedelement-wise; for example we use X 2 to denote the element-wise square of X .

Original: We denote the inverseof a function f by f − 1 ; if f is not one-to-one, with abuse of notation we use f − 1 to denote a functionthat satisﬁes f ( f − 1 ( x ))
Revised: We denote the inverseof a function f by f − 1 ; if f is not one-to-one, with abuse of notation we use f − 1 to denote a functionthat satisﬁes f ( f − 1 ( x ))

Original: = x ∀ x ∈ ran ( f ) (this is justiﬁed due to the axiom of choice [17]).
Revised: = x ∀ x ∈ ran ( f ) (this is justiﬁed due to the axiom of choice [22]).

Original: For amatrix V , we use V − 1 to denote its inverse; if V is singular, we use V − 1 to denote its pseudo-inverse.
Revised: For amatrix V , we use V − 1 to denote its inverse; if V is singular, we use V − 1 to denote its pseudo-inverse.

Original: We use [ N ] for N ∈ N to denote { 1 , ..., N } , and { X a } a ∈A to denote the set { ( a, X a ) | a ∈ A }.
Revised: We use [ N ] for N ∈ N to denote { 1 , ..., N } , and { X a } a ∈A to denote the set { ( a, X a ) | a ∈ A }.

Original: Wesay that y
Revised: Wesay that y

Original: = O ( f ( x )) if there is x 0 and a constant C such that y ≤ Cf ( x ) ∀ x > x 0 ; we also use˜ O ( f ( x )) to omit log factors.
Revised: = O ( f ( x )) if there is x 0 and a constant C such that y ≤ Cf ( x ) ∀ x > x 0 ; we also use˜ O ( f ( x )) to omit log factors.

Original: Contextual Linear Bandits.
Revised: Contextual Linear Bandits.

Original: We consider a contextual linear bandits problem over a horizon oflength T [7], where at each iteration t = 1 , ..., T , an agent, taking into account the context, choosesan action a t ∈ A and receives a reward r t .
Revised: We consider a contextual linear bandits problem over a horizon oflength T [8], where at each iteration t = 1 , ..., T , an agent, taking into account the context, choosesan action a t ∈ A and receives a reward r t .

Original: For each action a ∈ A , the agent has access to acorresponding feature vector X t,a ∈ R d .
Revised: For each action a ∈ A , the agent has access to acorresponding feature vector X t,a ∈ R d .

Original: The set of all such vectors { X t,a } a ∈A is the context attime t , and the agent can use it to decide which action a t to play.
Revised: The set of all such vectors { X t,a } a ∈A is the context attime t , and the agent can use it to decide which action a t to play.

Original: We assume that the context isgenerated from a distribution, i.e., given a , X t,a is generated from a distribution P a .
Revised: We assume that the context isgenerated from a distribution, i.e., given a , X t,a is generated from a distribution P a .

Original: As a speciﬁcexample, we could have that a ∈ R d and X t,a is generated from a Gaussian distribution withzero mean and covariance matrix || a || 2 I , where I is the identity matrix, i.e., P a = N (0 , || a || 2 I ) .
Revised: As a speciﬁcexample, we could have that a ∈ R d and X t,a is generated from a Gaussian distribution withzero mean and covariance matrix || a || 2 I , where I is the identity matrix, i.e., P a = N (0 , || a || 2 I ) .

Original: The selection of a t may depend not only on the current context { X t,a } a ∈A but also on the history
Revised: The selection of a t may depend not only on the current context { X t,a } a ∈A but also on the history

Original: H t (cid:44) {{ X 1 ,a } a ∈A , a 1 , r 1 , ..., { X t − 1 ,a } a ∈A , a t − 1 , r t − 1 } , namely, all previously selected actions,observed contexts and rewards.
Revised: H t (cid:44) {{ X 1 ,a } a ∈A , a 1 , r 1 , ..., { X t − 1 ,a } a ∈A , a t − 1 , r t − 1 } , namely, all previously selected actions,observed contexts and rewards.

Original: Once an action is selected, the reward is generated according to
Revised: Once an action is selected, the reward is generated according to

Original: [Equation]
Revised: [Equation]

Original:  where (cid:104) .
Revised: (1) where (cid:104) .

Original: (cid:105) denotes the dot product, θ (cid:63) is an unknown (but ﬁxed) parameter vector in R d , andη t is noise.
Revised: (cid:105) denotes the dot product, θ (cid:63) is an unknown (but ﬁxed) parameter vector in R d , andη t is noise.

Original: We assume that the noise follows an unknown distribution with E [ η t |F t ] = 0 and 115 E [exp( λη t ) |F t ] ≤ exp( λ 2 / 2) ∀ λ ∈ R , where F t = σ ( { X 1 ,a } a ∈A , a 1 , r 1 , ..., { X t,a } a ∈A , a t ) is theﬁltration [12] of historic information up to time t , and σ ( X ) is the σ -algebra generated by X [12].
Revised: We assume that the noise follows an unknown distribution with E [ η t |F t ] = 0 and 122 E [exp( λη t ) |F t ] ≤ exp( λ 2 / 2) ∀ λ ∈ R , where F t = σ ( { X 1 ,a } a ∈A , a 1 , r 1 , ..., { X t,a } a ∈A , a t ) is theﬁltration [13] of historic information up to time t , and σ ( X ) is the σ -algebra generated by X [13].

Original: The objective is to minimize the regret R T over a horizon of length T , where
Revised: The objective is to minimize the regret R T over a horizon of length T , where

Original: [Equation]
Revised: [Equation]

Original: The best performing algorithms for this problem, such as LinUCB[1, 28] and contextual Thompsonsampling [2], achieve a worst case regret of  O ( d √ Tlog T ) [19].
Revised: The best performing algorithms for this problem, such as LinUCB and contextual Thompson sampling,achieve a worst case regret of ˜ O ( d √ T ) [29, 28, 1, 2].

Original: The best known lower bound isΩ( d √ T ) [28].
Revised: The best known lower bound is Ω( d √ T ) [37].

Original: In the rest of this paper, we make the following assumptions that are standard in the literature [19].
Revised: In the rest of this paper, we make the following assumptions that are standard in the literature [24].

Original: Assumption 1.
Revised: Assumption 1.

Original: We consider contextual linear bandits that satisfy:
Revised: We consider contextual linear bandits that satisfy:

Original: [Equation]
Revised: [Equation]

Revised: The boundedness assumption on r t can be relaxed using [21], which only requires approximately 3 .

Revised: 5bits on average to send r t , even if it is unbounded.

Original: Memoryless Distributed Contextual Linear Bandits.
Revised: Memoryless Distributed Contextual Linear Bandits.

Original: We consider a distributed setting that consistsof a central learner communicating with geographically separated agents.
Revised: We consider a distributed setting that consistsof a central learner communicating with geographically separated agents.

Original: For example, the agentsare drones that interact with a trafﬁc policeman (central learner) as they ﬂy by.
Revised: For example, the agentsare drones that interact with a trafﬁc policeman (central learner) as they ﬂy by.

Original: We assume that theagents do not keep memory of past actions and may not be present for the whole duration of learning;learning in our setup can happen thanks to the persistent presence of the central learner.
Revised: We assume that theagents do not keep memory of past actions and may not be present for the whole duration of learning;learning in our setup can happen thanks to the persistent presence of the central learner.

Original: At each time t , t = 1 . . .
Revised: At each time t , t = 1 . . .

Original: T , a distributed agent joins the system; she receives from the central learnerinformation on the system, such as the current estimate of the parameter vector θ (cid:63) or the history
Revised: T , a distributed agent joins the system; she receives from the central learnerinformation on the system, such as the current estimate of the parameter vector θ (cid:63) or the history

Original: H t ; she observes the current context { X t,a } a ∈A , selects and plays an action a t and collects thecorresponding reward r t .
Revised: H t ; she observes the current context { X t,a } a ∈A , selects and plays an action a t and collects thecorresponding reward r t .

Original: Note that although the distributed agent knows the context { X t,a } a ∈A , theaction a t and the observed reward r t , the central learner does not.
Revised: Note that although the distributed agent knows the context { X t,a } a ∈A , theaction a t and the observed reward r t , the central learner does not.

Original: The central learner may need thisinformation to update its estimate of the system parameters, such as the unknown parameter vectorθ ∗ , and the history H t +1 .
Revised: The central learner may need thisinformation to update its estimate of the system parameters, such as the unknown parameter vectorθ ∗ , and the history H t +1 .

Original: However, we assume that the agent is restricted to utilize a communicationconstrained channel and thus may not be able to send the full information to the central learner.
Revised: However, we assume that the agent is restricted to utilize a communicationconstrained channel and thus may not be able to send the full information to the central learner.

Original: The main question we ask in this paper is: can we design a compression scheme, where the agentsends to the  learner only one message using B t bits (for as small as possible a value of B t ) thatenables the central learner to learn equally well (experience the same order of regret) as if there wereno communication constraints?
Revised: The main question we ask in this paper is: can we design a compression scheme, where the agentsends to the central learner only one message using B t bits (for as small as possible a value of B t ) thatenables the central learner to learn equally well (experience the same order of regret) as if there wereno communication constraints?

Original: With no communication constraints the agent could send unquantizedthe full information {{ X t,a } a ∈A , a t , r t } .
Revised: With no communication constraints the agent could send unquantizedthe full information {{ X t,a } a ∈A , a t , r t } .

Original: Instead, the agent transmits a message that could be afunction of all locally available information at the agent.
Revised: Instead, the agent transmits a message that could be afunction of all locally available information at the agent.

Original: For example, it could be a function of( H t , { X t,a t } a ∈A , a t , r t ) , if the agent had received H t from the  learner.
Revised: For example, it could be a function of( H t , { X t,a t } a ∈A , a t , r t ) , if the agent had received H t from the central learner.

Original: It could also be a function ofjust ( X t,a t , r t ) , which could be sufﬁcient if the central learner employs an algorithm such as LinUCB[1, 28].
Revised: It could also be afunction of just ( X t,a t , r t ) , which could be sufﬁcient if the central learner employs an algorithm suchas LinUCB [1, 37].

Original: In summary, we set the following goal.
Revised: In summary, we set the following goal.

Original: Goal.
Revised: Goal.

Original: Design contextual linear bandit schemes for the memoryless distributed setting that achievethe best known regret of O ( d (cid:112) T log( T )) , while communicating a small number of bits B t .
Revised: Design contextual linear bandit schemes for the memoryless distributed setting that achievethe best known regret of O ( d (cid:112) T log( T )) , while communicating a small number of bits B t .

Original: We only impose communication constraints on the uplink communication (from the agents to thecentral learner) and assume no cost downlink communication (see discussion in Secttion 1).
Revised: We only impose communication constraints on the uplink communication (from the agents to thecentral learner) and assume no cost downlink communication (see discussion in Secttion 1).

Original: Stochastic Quantizer (SQ) [14].
Revised: Stochastic Quantizer (SQ) [16].

Original: Our proposed algorithms use stochastic quantization, that we nextreview.
Revised: Our proposed algorithms use stochastic quantization, that we nextreview.

Original: We deﬁne SQ (cid:96) , (cid:96) ∈ N to be a quantizer, that uses log( (cid:96) +1) bits, consisting of an encoder anddecoder described as following.
Revised: We deﬁne SQ (cid:96) , (cid:96) ∈ N to be a quantizer, that uses log( (cid:96) +1) bits, consisting of an encoder anddecoder described as following.

Original: The encoder ξ (cid:96) takes a value x ∈ [0 , (cid:96) ] and outputs an integer valueξ (cid:96) 
Revised: The encoder ξ (cid:96) takes a value x ∈ [0 , (cid:96) ] and outputs an integer valueξ (cid:96) = [Equation]

Original: = (cid:26) (cid:98) x (cid:99) with probability (cid:100) x (cid:101) − x (cid:100) x (cid:101) with probability x − (cid:98) x (cid:99) .
Revised: x [Equation] with probability [Equation]

Original: [Equation]
Revised: [Equation]

Original: The output ξ (cid:96) is represented with log( (cid:96) + 1) bits.
Revised: The output ξ (cid:96) is represented with log( (cid:96) + 1) bits.

Original: The decoder D (cid:96) takes as input the binary representation of ξ (cid:96) ( x ) and outputs the real value ξ (cid:96) ( x ) .
Revised: The decoder D (cid:96) takes as input the binary representation of ξ (cid:96) ( x ) and outputs the real value ξ (cid:96) ( x ) .

Original: The composition of the encoder ξ (cid:96) , the binarymapping, and decoder D (cid:96) is denoted by SQ (cid:96) .
Revised: The composition of the encoder ξ (cid:96) , the binarymapping, and decoder D (cid:96) is denoted by SQ (cid:96) .

Original: We notice that since the decoder only inverts the binarymapping operation, we have that SQ (cid:96) =
Revised: We notice that since the decoder only inverts the binarymapping operation, we have that SQ (cid:96) =

Original: ξ (cid:96) .
Revised: ξ (cid:96) .

Original: When SQ (cid:96) is applied at the agents side, the agent encodesits data, x , as ξ (cid:96) ( x ) , then sends the corresponding binary mapping to the central learner that applies
Revised: When SQ (cid:96) is applied at the agents side, the agent encodesits data, x , as ξ (cid:96) ( x ) , then sends the corresponding binary mapping to the central learner that applies

Original: D (cid:96) to get SQ (cid:96) ( x ) .
Revised: D (cid:96) to get SQ (cid:96) ( x ) .

Original: With slightly abuse of notation, this operation is described in the paper, by sayingthat the agent sends SQ (cid:96) to the central learner.
Revised: With slightly abuse of notation, this operation is described in the paper, by sayingthat the agent sends SQ (cid:96) to the central learner.

Original: The quantizer SQ (cid:96) is a form of dithering [14] and it has the following properties
Revised: The quantizer SQ (cid:96) is a form of dithering [16] and it has the following properties

Original: E [ SQ 
Revised: E [ SQ (cid:96) ( x ) | x ] = (cid:98) x (cid:99) ( (cid:100) x (cid:101) [Equation] + (cid:100)

Original: [Equation]
Revised: [Equation]

Original: and | SQ
Revised: (cid:101) ( x

Original: [Equation]
Revised: [Equation]

Revised: (cid:98) x (cid:99) ) =

Revised: x ( (cid:100) x (cid:101) − (cid:98) x (cid:99) )

Revised: = x, and | SQ (cid:96) ( x ) − x | ≤ 1 .

Original: In particular, it conveys an unbiased estimate of the input with a difference that is bounded by 1almost surely.
Revised: In particular, it conveys an unbiased estimate of the input with a difference that is bounded byalmost surely.

Original: We also deﬁne a generalization of SQ (cid:96) denoted by SQ [ a,b ] (cid:96) where the input x of theencoder is in [ a, b ] instead of [0 , (cid:96) ] .
Revised: We also deﬁne a generalization of SQ (cid:96) denoted by SQ [ a,b ] (cid:96) where the input x of the encoder is in [ a, b ] instead of [0 , (cid:96) ] .

Original: The encoder ﬁrst shifts and scales x using ˜ x
Revised: The encoder ﬁrst shifts and scales x using ˜ x

Original: = (cid:96) b − a ( x − a ) tomake it lie in [0 , (cid:96) ] , then feeds ˜ x to the encoder in (3).
Revised: = (cid:96) b − a ( x − a ) to make it lie in [0 , (cid:96) ] , then feeds ˜ x to the encoder in (3).

Original: This operation is inverted at the decoder.
Revised: This operation is inverted at the decoder.

Original: It iseasy to see that SQ [ a,b ] (cid:96) satisﬁes
Revised: It is easy to see that SQ [ a,b ] (cid:96) satisﬁes

Original: [Equation]
Revised: [Equation]

Original:  Contextual Linear Bandits with Known Context Distribution
Revised:  Contextual Linear Bandits with Known Context Distribution

Original: In this section, we show that if the central learner knows the distributions for the vectors X t,a , thenthe agent does not need to convey the speciﬁc realization of the vector X t,a she observes at all - it issufﬁcient to just send 1 bit to convey some information on the observed reward and nothing else.
Revised: In this section, we show that if the central learner knows the distributions for the vectors X t,a , thenthe agent does not need to convey the speciﬁc realization of the vector X t,a she observes at all - it issufﬁcient to just send 1 bit to convey some information on the observed reward and nothing else.

Original: Butfor this very limited communication, the central learner can experience the same order regret, as whenreceiving in full precision all the information that the agents have, namely, R T
Revised: Butfor this very limited communication, the central learner can experience the same order regret, as whenreceiving in full precision all the information that the agents have, namely, R T

Original: =
Revised: =

Original: O ( d √ T log T ) .
Revised: O ( d √ T log T ) .

Original: Algorithm 1, that we describe in this section, provides a method to achieve this.
Revised: Algorithm 1, that we describe in this section, provides a method to achieve this.

Original: Algorithm 1 is clearlyoptimal, as we cannot hope to use less than zero bits for the vector X t,a .
Revised: Algorithm 1 is clearlyoptimal, as we cannot hope to use less than zero bits for the vector X t,a .

Original: Remark 1.
Revised: Remark 1.

Original: Knowledge of the distribution of X t,a is possible in practice, since many times thecontext may be capturing well studied statistics (e.g., male or female, age, weight, income, race,dietary restrictions, emotional state, etc) - the advent of large data has made and will continue to makesuch distributions available.
Revised: Knowledge of the distribution of X t,a is possible in practice, since many times thecontext may be capturing well studied statistics (e.g., male or female, age, weight, income, race,dietary restrictions, emotional state, etc) - the advent of large data has made and will continue to makesuch distributions available.

Original: Similarly, actions may be ﬁnite (eg., restaurants to visit) or well described(e.g.
Revised: Similarly, actions may be ﬁnite (eg., restaurants to visit) or well described(e.g.

Original: , released amounts of substances), and thus the distribution of X t,a could be derived.
Revised: , released amounts of substances), and thus the distribution of X t,a could be derived.

Original: When thedistribution is approximately known, we provide later in this section a bound on the misspeﬁcationperformance penalty in terms of regret.
Revised: When thedistribution is approximately known, we provide later in this section a bound on the misspeﬁcationperformance penalty in terms of regret.

Original: Main Idea.
Revised: Main Idea.

Original: The intuition behind Algorithm 1 is that it reduces the multi-context linear bandit problemto a single context problem.
Revised: The intuition behind Algorithm 1 is that it reduces the multi-context linear bandit problemto a single context problem.

Original: In particular, it calls as a subroutine an algorithm we term Λ , that servesas a placeholder for any current (or future) bandit algorithm that achieves regret O ( d √ T log T ) forthe case of a single context (for example, LinUCB [1, 28]).
Revised: In particular, it calls as a subroutine an algorithm we term Λ , that servesas a placeholder for any current (or future) bandit algorithm that achieves regret O ( d √ T log T ) forthe case of a single context (for example, LinUCB [1, 37]).

Original: The central learner uses Λ to convey tothe agents the information they need to select a good action.
Revised: The central learner uses Λ to convey tothe agents the information they need to select a good action.

Original: Our aim is to parametrize the singlecontext problem appropriately, so that, by solving it we also solve our original problem.
Revised: Our aim is to parametrize the singlecontext problem appropriately, so that, by solving it we also solve our original problem.

Original: Recall that in a single context problem, at each iteration t , any standard linear bandit algorithm Λselects a feature vector (an action) x t from a set of allowable actions X , and observes a reward
Revised: Recall that in a single context problem, at each iteration t , any standard linear bandit algorithm Λselects a feature vector (an action) x t from a set of allowable actions X , and observes a reward

Original: [Equation]
Revised: [Equation]

Original:  where θ  (cid:63)(Λ) is an uknown parametervector and η t is noise that satisﬁes the same assumptions as in(1).
Revised: (4) where θ (cid:48) (cid:63) is an uknown parameter and η t is noise that satisﬁes the same assumptions as in (1).

Original: The objective of Λ is to minimize the standard linear regretexpression R T (Λ) over a horizon oflength T , namely
Revised: Theobjective of Λ is to minimize the standard linear regret R T (Λ) over a horizon of length T , namely

Original: [Equation]
Revised: [Equation]

Original: Our reduction proceeds as follows.
Revised: Our reduction proceeds as follows.

Original: We assume that Λ operates over the same horizon of length T andis parametrized by an unknown parameter θ  (cid:63)(Λ) .
Revised: We assume that Λ operates over the same horizon of length T andis parametrized by an unknown parameter θ (cid:48) (cid:63) .

Original: We will design the action set X that we provide toΛ using our knowledge of the distributions P a 2 as we will describe later in (7).
Revised: We will design the action set X that we provide to Λusing our knowledge of the distributions P a 2 as we will describe later in (7).

Original: During each iteration,the central learner asks Λ to select an action x t ∈ X and then provides to Λ a reward for this action(our design ensures that this reward satisﬁes (4) with θ  (cid:63)(Λ) = θ (cid:63) ).
Revised: During each iteration,the central learner asks Λ to select an action x t ∈ X and then provides to Λ a reward for this action(our design ensures that this reward satisﬁes (4) with θ (cid:48) (cid:63) = θ (cid:63) ).

Original: Λ operates with this information,oblivious to what else the central learner does.
Revised: Λ operates with this information,oblivious to what else the central learner does.

Original: Yet, the action x t is never actually played: the centrallearner uses the selected action x t to create an updated estimate of the parameter vector ˆ θ t , as wewill describe later, and only sends this parameter vector estimate to the distributed agent.
Revised: Yet, the action x t is never actually played: the centrallearner uses the selected action x t to create an updated estimate of the parameter vector ˆ θ t , as wewill describe later, and only sends this parameter vector estimate to the distributed agent.

Original: The agentobserves her context, selects what action to play, and sends back her observed quantized reward tothe central learner.
Revised: The agentobserves her context, selects what action to play, and sends back her observed quantized reward tothe central learner.

Original: This is the reward that the central learner provides to Λ .
Revised: This is the reward that the central learner provides to Λ .

Original: We design the set X andthe agent operation to satisfy that: (4) holds; and R T − R T (Λ) is small, where R T is the regret forour original multi-context problem and R T (Λ) the regret of Λ .
Revised: We design the set X andthe agent operation to satisfy that: (4) holds; and R T − R T (Λ) is small, where R T is the regret forour original multi-context problem and R T (Λ) the regret of Λ .

Original: We next try to provide some intuitionon how we achieve this.
Revised: We next try to provide some intuitionon how we achieve this.

Original: We ﬁrst describe how we construct the set X .
Revised: We ﬁrst describe how we construct the set X .

Original: Let Θ be the set of all values that θ (cid:63) could possiblytake.
Revised: Let Θ be the set of all values that θ (cid:63) could possiblytake.

Original: For each possible parameter vector value θ ∈ Θ the central learner considers the quantity
Revised: For each possible parameter vector value θ ∈ Θ the central learner considers the quantity

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where x a is the random variable that follows the distribution P a .
Revised: where x a is the random variable that follows the distribution P a .

Revised: Ties in (6) can be broken uniformlyat random.

Revised: In fact any pre-selected choice function would work as long as the same function is alsoused in step 12 of Algorithm 1.

Original: Note that the function X (cid:63) : R d →209 R d can be computed ofﬂine before the learning starts, see Example 1.
Revised: Note that the function X (cid:63) : R d → R d can be computed ofﬂine beforethe learning starts, see Example 1.

Original: We then use
Revised: We then use

Original: [Equation]
Revised: [Equation]

Original: Intuitively, for each value of θ , we optimistically assume that the distributed agent may select the bestpossible realization X t,a for this θ (that has the expectation in (6)), and receive the associated reward;accordingly, we restrict the action space X of Λ to only contain the expectation of these “best” X t,a .
Revised: Intuitively, for each value of θ , we optimistically assume that the distributed agent may select the bestpossible realization X t,a for this θ (that has the expectation in (6)), and receive the associated reward;accordingly, we restrict the action space X of Λ to only contain the expectation of these “best” X t,a .

Original: The vector x t ∈ X may not actually be the vector corresponding to the action the agent selects; it isonly used to convey to the agent an estimate of the unknown parameter ˆ θ t that satisﬁes x t =
Revised: The vector x t ∈ X may not actually be the vector corresponding to the action the agent selects; it isonly used to convey to the agent an estimate of the unknown parameter ˆ θ t that satisﬁes x t =

Original: X (cid:63) ( ˆ θ t ) .
Revised: X (cid:63) (ˆ θ t ) .

Original: Although the  learner does not control which action the agent plays, this is inﬂuenced by ˆ θ t ; we showin App.
Revised: Although the central learner does not control which action the agent plays, this is inﬂuenced by ˆ θ t ;we show in App.

Original: B that X t,a t is an unbiased estimate of x t , and the generated reward follows the linear modelin (4) with θ  (cid:63)(Λ)
Revised: B that X t,a t is an unbiased estimate of x t , and the generated reward follows thelinear model in (4) with θ (cid:48) (cid:63)

Original: = θ (cid:63) .
Revised: = θ (cid:63) .

Original: In Theorem 1, we prove that
Revised: In Theorem 1, we prove that

Original: [Equation]
Revised: [Equation]

Original: Hence, if Λ converges to selecting the best action for the single context problem, we will have thatˆ θ t = θ (cid:63) .
Revised: Hence, if Λ converges to selecting the best action for the single context problem, we will havethat ˆ θ t converges to θ (cid:63) if the maximizer in (8) is unique.

Original: If there are multiple values for θ with X (cid:63) ( θ ) =
Revised: If there are multiple values for θ with X (cid:63) ( θ ) =

Original: X (cid:63) ( θ (cid:63) ) , we show in the proof of Theorem 1that they all lead to the same expected reward for the original multi-context problem.
Revised: X (cid:63) ( θ (cid:63) ) , we show in the proof of Theorem 1 that they all lead to the same expected rewardfor the original multi-context problem.

Original: Example 1.
Revised: Example 1.

Original: Consider the case where d = 1 , A = { 1 , 2 } , X t,a ∈ {− 1 , 1 } ∀ a ∈ A , Θ = {− 1 , 1 } ,θ (cid:63) = 1 and X t, 1 takes the value − 1 with probability p and 1 otherwise, while X t, 2 takes the values− 1 with probability q and 1 otherwise.
Revised: Consider the case where d = 1 , A = { 1 , 2 } , X t,a ∈ {− 1 , 1 } ∀ a ∈ A , Θ = {− 1 , 1 } ,θ (cid:63) = 1 and X t, 1 takes the value − 1 with probability p and 1 otherwise, while X t, 2 takes the values− 1 with probability q and 1 otherwise.

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that if arg max X t,a (cid:104) X t,a , 1 (cid:105) (cid:54) = 1 , it must be the case that both X t, 1 and
Revised: where we use the fact that if arg max X t,a (cid:104) X t,a , 1 (cid:105) (cid:54) = 1 , it must be the case that both X t, 1 and

Original: X t, 2 are − 1 .
Revised: X t, 2 are − 1 .

Original: Thus, X (cid:63) (1) =
Revised: Thus, X (cid:63) (1) =

Original: E [arg max X t,a (cid:104) X t,a , 1 (cid:105) ]
Revised: E [arg max X t,a (cid:104) X t,a , 1 (cid:105) ]

Original: = 1 − 2 pq , and similarly X (cid:63) ( − 1) =− 1 + 2(1 − p )(1 − q ) , and hence, X = { 1 − 2 pq, − 1 + 2(1 − p )(1 − q ) } .
Revised: = 1 − 2 pq , and similarly X (cid:63) ( − 1) =− 1 + 2(1 − p )(1 − q ) , and hence, X = { 1 − 2 pq, − 1 + 2(1 − p )(1 − q ) } .

Original: If Λ decides to pickx t = 1 − 2 pq , we have that ˆ θ t = 1 , otherwise ˆ θ t = − 1 .
Revised: If Λ decides to pickx t = 1 − 2 pq , we have that ˆ θ t = 1 , otherwise ˆ θ t = − 1 .

Original: This estimate ˆ θ t is then conveyed to theagent to help her pick the action.
Revised: This estimate ˆ θ t is then conveyed to theagent to help her pick the action.

Original: Algorithm Operation.
Revised: Algorithm Operation.

Original: The pseudo-code is provided in Algorithm 1.
Revised: The pseudo-code is provided in Algorithm 1.

Original: • First, the central learner calculates the function
Revised: • First, the central learner calculates the function

Original: [Equation]
Revised: [Equation]

Original: and creates the action set X
Revised: and creates the action set X

Original: = { X (cid:63) ( θ ) | θ ∈ Θ } that algorithm Λ is going to use.•
Revised: = { X (cid:63) ( θ ) | θ ∈ Θ } that algorithm Λ is going to use.•

Original: At each time t , based on past history, Λ decides on a next action x t ∈ X .
Revised: At each time t , based on past history, Λ decides on a next action x t ∈ X .

Original: The central learner usesx t to calculate the new update ˆ θ t = X − 1 ( x t ) , where X − 1 is the inverse of X (cid:63) (see Section 2).• The agent receives ˆ θ t from the central learner, observes her context, plays an action a t = [Equation] max 
Revised: The central learner usesx t to calculate the new update ˆ θ t = X − 1 ( x t ) , where X − 1 is the inverse of X (cid:63) (see Section 2).• The agent receives ˆ θ t from the central learner, observes her context, plays an action a t =arg max a ∈A (cid:104) X t,a , ˆ θ t (cid:105) , and observes the reward r t .

Original: [Equation]

Original: (cid:104) X t,a , ˆ θ t (cid:105) , and observes the reward r t .

Original: She then quantizes the reward using a stochasticquantizer SQ 1 (see Section 2), and communicates the outcome using one bit to the central learner.•
Revised: She then quantizes the reward using a stochasticquantizer SQ 1 (see Section 2), and communicates the outcome using one bit to the central learner.•

Original: The  learner provides the quantized reward as input to Λ .
Revised: The central learner provides the quantized reward as input to Λ .

Original: Note that Λ is oblivious to what actionsare actually played; it treats the received reward as corresponding to the action x t it had decided.
Revised: Note that Λ is oblivious to whatactions are actually played; it treats the received reward as corresponding to the action x t it haddecided. Send ˆ θ t = X − 1 ( x t ) to agent. Agent : 10:

Revised: Receive ˆ θ t from the central learner.

Original:  The following theorem proves that Algorithm 1 achieves a regret R T (Λ) + O ( √ T log T ) , where
Revised: Observe context realization { X t,a } a ∈A . Pull arm a t = arg max a ∈A (cid:104) X t,a , ˆ θ t (cid:105) and receive reward r t . Send ˆ r t = SQ 1 ( r t ) to the central learner using 1-bit. The following theorem proves that Algorithm 1 achieves a regret R T (Λ) + O ( √ T log T ) , where

Original: R T (Λ) is the regret of Λ in (5).
Revised: R T (Λ) is the regret of Λ in (5).

Original: Hence, if Λ satisﬁes the best known regret bound of O ( d √ T log T ) ,e.g., LinUCB, Algorithm 1 achieves a regret of O ( d √ T log T ) .
Revised: Hence, if Λ satisﬁes the best known regret bound of O ( d √ T log T ) ,e.g., LinUCB, Algorithm 1 achieves a regret of O ( d √ T log T ) .

Original: The theorem holds under the mildset of assumptions that we stated in Section 2.
Revised: The theorem holds under the mildset of assumptions that we stated in Section 2.

Original: Theorem 1.
Revised: Theorem 1.

Original: Algorithm 1 uses 1 bit per reward and 0 bits per context.
Revised: Algorithm 1 uses 1 bit per reward and 0 bits per context.

Original: Under Assumption 1, itachieves a regret R T
Revised: Under Assumption 1, itachieves a regret R T

Original: = R T (Λ) + O ( √ T log T ) with probability at least 1 − 1 T .
Revised: = R T (Λ) + O ( √ T log T ) with probability at least 1 − 1 T .

Original: Proof outline.
Revised: Proof outline.

Original: The complete proof is available in App.
Revised: The complete proof is available in App.

Original: B. We next provide a short outline.
Revised: B. We next provide a short outline.

Original: Fromthe deﬁnition of X (cid:63) in (10), we notice the following.
Revised: Fromthe deﬁnition of X (cid:63) in (10), we notice the following.

Original: Recall that the distributed agent receives ˆ θ tfrom the central learner, and pulls the best action for this ˆ θ t , i.e., a t = arg max a ∈A (cid:104) X t,a , ˆ θ t (cid:105) .
Revised: Recall that the distributed agent receives ˆ θ tfrom the central learner, and pulls the best action for this ˆ θ t , i.e., a t = arg max a ∈A (cid:104) X t,a , ˆ θ t (cid:105) .

Original: Weshow that conditioned on x t , the associated vector X t,a t is an unbiased estimate of x t with a smallvariance.
Revised: We show that conditioned on x t , the associated vector X t,a t is an unbiased estimate of x t with a smallvariance.

Original: Given this, we prove that ˆ r t satisﬁes (6), and thus the rewards observed by Λ are generatedaccording to a linear bandit model with unknown parameter that is the same as θ (cid:63) .
Revised: Given this, we prove that ˆ r t satisﬁes (6), and thus the rewards observed by Λ are generatedaccording to a linear bandit model with unknown parameter that is the same as θ (cid:63) .

Original: We next decompose the difference
Revised: We next decompose the difference

Original: R T − R T (Λ) to two terms:Σ T =
Revised: R T − R T (Λ) to two terms:

Original: [Equation]
Revised: [Equation]

Original: Tt =1 (cid:104) arg max X t,a (cid:104) X t,a , θ (cid:63) (cid:105) , θ (cid:63) (cid:105) − (cid:104) x t , θ (cid:63) (cid:105) and Σ (cid:48) T = (cid:80)
Revised: T [Equation]

Original: Tt =1 (cid:104) arg max X t,a (cid:104) X t,a , ˆ θ t (cid:105) , θ (cid:63) (cid:105) −max x ∈X (cid:104) x, θ (cid:63) (cid:105) .
Revised: [Equation]

Original: To bound the ﬁrst term, we show that the unbiasdness property togetherwith Assumption 1 implies that Σ T is a martingale with bounded difference.
Revised: To bound the ﬁrst term, we show that the unbiasdness property togetherwith Assumption 1 implies that Σ T is a martingale with bounded difference.

Original: This implies that | Σ T | =
Revised: This implies that | Σ T | =

Original: O ( √ T log T ) with high probability.
Revised: O ( √ T log T ) with high probability.

Original: To bound Σ (cid:48) T , we ﬁrst show thatarg [Equation] =
Revised: To bound Σ (cid:48) T , we ﬁrst show thatarg max x ∈X (cid:104) x, θ (cid:63) (cid:105) =

Original: X (cid:63) ( θ (cid:63) )
Revised: X (cid:63) ( θ (cid:63) )

Original: (we note that this is why the algorithm converges to ˆ θ t that is equalto, or results in the same expected reward as, θ (cid:63) ).
Revised: (we note that this is why the algorithm converges to ˆ θ t that is equalto, or results in the same expected reward as, θ (cid:63) ).

Original: Then, following a similar approach, we can showthat Σ (cid:48) T is a martingale with bounded difference which implies that | Σ (cid:48) T | =
Revised: Then, following a similar approach, we can showthat Σ (cid:48) T is a martingale with bounded difference which implies that | Σ (cid:48) T | =

Original: O ( √ T log T ) with highprobability.
Revised: O ( √ T log T ) with highprobability.

Original: (cid:3)
Revised: [Equation]

Original: Downlink Communication.
Revised: Downlink Communication.

Original: Note that in our setup we assume that the central learner does not haveany communication constraints when communicating with the distributed agents. Yet our algorithmmakes frugal use of this ability: the learner only sends the updated parameter vector ˆ θ t . We canquantize ˆ θ t without performance loss if the downlink were also communication constrained using≈ 5 d bits and an approach similar to the one in Algorithm 2 - yet we do not expand on this in thispaper, as our focus is in minimizing uplink communication costs.
Revised: The downlink cost of our scheme is O ( d ) (see App. B for discussion).

Original: Operation Complexity.
Revised: Operation Complexity.

Original: The main complexity that our algorithm adds beyond the complexity of Λ ,is the computation of the function X (cid:63) .
Revised: The main complexity that our algorithm adds beyond the complexity of Λ ,is the computation of the function X (cid:63) .

Original: Depending on the distributions P a , this can be calculatedin closed form efﬁciently.
Revised: For many practical distributions P a , this can be calculatedefﬁciently.

Original: For example, for d = 1 , θ > 0 , we have that X (cid:63) ( θ ) is the expectationof the maximum of multiple random variables, i.e., X (cid:63) ( θ ) = E x a ∼P a [max a ∈A x a ] , which can becomputed/approximated efﬁciently if the distributions P a are given in a closed form.
Revised: We discuss this in more details in App. B.

Original: Imperfect Knowledge of Distributions.
Revised: Imperfect Knowledge of Distributions.

Original: Since we only use the distributions to calculate X (cid:63) ,imperfect knowledge of distribution only affects us in the degree that it affects the calculation of X (cid:63) .
Revised: Since we only use the distributions to calculate X (cid:63) ,imperfect knowledge of distribution only affects us in the degree that it affects the calculation of X (cid:63) .

Original: Suppose that we have an estimate ˜ X (cid:63) of X (cid:63) that satisﬁes
Revised: Suppose that we have an estimate ˜ X (cid:63) of X (cid:63) that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Using Theorem 1 we prove in App.
Revised: Using Theorem 1 we prove in App.

Original: B the following corollary.
Revised: B the following corollary.

Original: Corollary 1.
Revised: Corollary 1.

Original: Suppose we are given ˜ X (cid:63) that satisﬁes (11).
Revised: Suppose we are given ˜ X (cid:63) that satisﬁes (11).

Original: Then, there exists an algorithm Λ forwhich Algorithm 1 achieves R T =
Revised: Then, there exists an algorithm Λ forwhich Algorithm 1 achieves R T =

Original: ˜ O ( d √ T
Revised: ˜ O ( d √ T

Original: + (cid:15)T √ d ) with probability at least 1 − 1 T .
Revised: + (cid:15)T √ d ) with probability at least 1 − 1 T .

Original: Privacy.
Revised: Privacy.

Original: Our result may be useful for applications beyond communication efﬁciency; indeed, thecontext may contain private information (e.g., personal preferences, ﬁnancial information, etc); useof our algorithm enables to not share this private information at all with the central learner, withoutimpeding the learning process.
Revised: Our result may be useful for applications beyond communication efﬁciency; indeed, thecontext may contain private information (e.g., personal preferences, ﬁnancial information, etc); useof our algorithm enables to not share this private information at all with the central learner, withoutimpeding the learning process.

Original: Surprisingly, work in [34], motivated from privacy considerations, hasshown that if an agent adds a small amount of zero mean noise to the true context before sending it tothe central learner, this can severely affect the regret in some cases - and yet our algorithm essentiallyenables to “guess” the context with no regret penalty if the distributions are known.
Revised: Surprisingly, work in [48], motivated from privacy considerations, hasshown that if an agent adds a small amount of zero mean noise to the true context before sending it tothe central learner, this can severely affect the regret in some cases - and yet our algorithm essentiallyenables to “guess” the context with no regret penalty if the distributions are known.

Original: Although addinga zero mean noise to the observed feature vector conveys an unbiased estimate of the observation, thedifference between this and our case is technical and mainly due to the fact that the unbiasdness isrequired to hold conditioned on the  learner observation (noisy context); see App.
Revised: Although addinga zero mean noise to the observed feature vector conveys an unbiased estimate of the observation, thedifference between this and our case is technical and mainly due to the fact that the unbiasdness isrequired to hold conditioned on the central learner observation (noisy context); see App.

Original: B for more details.
Revised: B for moredetails.

Revised: We also discuss potential uses of our algorithm in preserving the context privacy in App B.

Original:  Contextual Linear Bandits with Unknown Context Distribution
Revised:  Contextual Linear Bandits with Unknown Context Distribution

Original: We now consider the case where the learner does not know the context distributions, and thus
Revised: We now consider the case where the learner does not know the context distributions, and thus

Original: Algorithm 1 that uses zero bits for the context cannot be applied.
Revised: Algorithm 1 that uses zero bits for the context cannot be applied.

Original: In this case, related literatureconjectures a lower bound of Ω( d ) [32, 33] – which is discouraging since it is probably impossible toestablish an algorithm with communication logarithmically depending on d .
Revised: In this case, related literatureconjectures a lower bound of Ω( d ) [46, 47] – which is discouraging since it is probably impossible toestablish an algorithm with communication logarithmically depending on d .

Original: Additionally, in practicewe use 32 d bits to convey full precision values - thus this conjecture indicates that in practice we maynot be able to achieve order improvements in terms of bits communicated, without performance loss.
Revised: Additionally, in practicewe use 32 d bits to convey full precision values - thus this conjecture indicates that in practice we maynot be able to achieve order improvements in terms of bits communicated, without performance loss.

Original: In this section, we provide Algorithm 2 that uses ≈ 5 d bits per context and achieves (optimal) regret
Revised: In this section, we provide Algorithm 2 that uses ≈ 5 d bits per context and achieves (optimal) regret

Original: R T =
Revised: R T =

Original: O ( d √ T log T ) .
Revised: O ( d √ T log T ) .

Original: We believe Algorithm 2 is interesting for two reasons:
Revised: We believe Algorithm 2 is interesting for two reasons:1.

Original: In theory, we need an inﬁnite number of bits to convey full precision values- we prove that aconstant number of bits per dimension per context is sufﬁcient.
Revised: In theory, we need an inﬁnite number of bits to convey full precision values- we prove that aconstant number of bits per dimension per context is sufﬁcient.

Original: Previously best-known algorithms use
Revised: Previously best-known algorithms,which rely on constructing 1 /T -net

Original:  O ( d log T ) bits per context, which goes to inﬁnity as T goes to inﬁnity.
Revised: for the set of feature vectors, use O ( d log T ) bits per context,which goes to inﬁnity as T goes to inﬁnity.

Original: Moreover, these algorithmsrequire exponential complexity [19] while ours is computationally efﬁcient.
Revised: Moreover, these algorithms require exponential complexity[24] while ours is computationally efﬁcient.2.

Original: In practice, especially for large values of d , reducing the number of bits conveyed from 32 d to ≈ 5 dis quite signiﬁcant - this is a reduction by a factor of six, which implies six times less communication.
Revised: In practice, especially for large values of d , reducing the number of bits conveyed from 32 d to ≈ 5 dis quite signiﬁcant - this is a reduction by a factor of six, which implies six times less communication.

Original: Main Idea.
Revised: Main Idea.

Original: The intuition behind Algorithm 2 is the following.
Revised: The intuition behind Algorithm 2 is the following.

Original: The central learner is going to usean estimate of the d × d least-squares matrix V t = (cid:80) t i =
Revised: The central learner is going to usean estimate of the d × d least-squares matrix V t = (cid:80) ti =

Original: 1 X i,a i X T i,a i to update her estimates for theparameter vector θ (cid:63) .
Revised: 1 X i,a i X Ti,a i to update her estimates for theparameter vector θ (cid:63) .

Original: Thus, when quantizing the vector X t,a , we want to make sure that not onlythis vector is conveyed with sufﬁcient accuracy, but also that the  learner can calculate the matrix
Revised: Thus, when quantizing the vector X t,a , we want to make sure that not only thisvector is conveyed with sufﬁcient accuracy, but also that the central learner can calculate the matrix

Original: V t accurately.
Revised: V t accurately.

Original: In particular, we would like the central learner to be able to calculate an unbiasedestimator for each entry of X t,a and each entry of the matrix V t .
Revised: In particular, we would like the central learner to be able to calculate an unbiasedestimator for each entry of X t,a and each entry of the matrix V t .

Original: Our algorithm achieves this byquantizing the feature vectors X t,a t , and also the diagonal (only the diagonal) entries of the least squares matrix V t .
Revised: Our algorithm achieves this byquantizing the feature vectors X t,a t , and also the diagonal (only the diagonal) entries of the leastsquares matrix V t .

Original: We prove that by doing so, with only ≈ 5 d bits we can provide an unbiasedestimate and guarantee an O ( 1 √ d ) quantization error for each entry in the matrix almost surely.
Revised: We prove that by doing so, with only ≈ 5 d bits we can provide an unbiasedestimate and guarantee an O ( 1 √ d ) quantization error for each entry in the matrix almost surely.

Original: Quantization Scheme.
Revised: Quantization Scheme.

Original: We here describe the proposed quantization scheme.• To quantize X t,a t : Let m (cid:44) (cid:100)√ d (cid:101) .
Revised: We here describe the proposed quantization scheme • To quantize X t,a t : Let m (cid:44) (cid:100)√ d (cid:101) .

Original: We ﬁrst send the sign of each coordinate of X t,a t using d bits,namely, we send the vector s t =
Revised: We ﬁrst send the sign of each coordinate of X t,a t using d bits,namely, we send the vector s t =

Original: X t,a t / | X t,a t | .
Revised: X t,a t / | X t,a t | .

Original: To quantize the magnitude | X t,a t | , we scale eachcoordinate of | X t,a t | by m and quantize it using a Stochastic Quantizer (SQ) 3 with m + 1 levels inthe interval [0 , m ] .
Revised: To quantize the magnitude | X t,a t | , we scale eachcoordinate of | X t,a t | by m and quantize it using a Stochastic Quantizer (SQ) 3 with m + 1 levels inthe interval [0 , m ] .

Original: Let X t (cid:44) SQ m ( m | X t,a t | ) denote the resulting SQ outputs, we note that X t takesnon-negative integer values and lies in a norm-1 ball of radius 2 d (this holds since the original vectorlies in a norm-2 ball of radius 1 and the error in each coordinate is at most 1 /m ).
Revised: Let X t (cid:44) SQ m ( m | X t,a t | ) denote the resulting SQ outputs, we note that X t takesnon-negative integer values and lies in a norm-1 ball of radius 2 d (this holds since the original vectorlies in a norm-2 ball of radius 1 and the error in each coordinate is at most 1 /m ).

Original: That is, it holds thatX t ∈ Q
Revised: That is, it holds thatX t ∈ Q

Original: = { x ∈ N d |(cid:107)
Revised: = { x ∈ N d |(cid:107)

Original: x (cid:107) 1 ≤ 2 d } .
Revised: x (cid:107) 1 ≤ 2 d } .

Original: We then use any enumeration h : Q → [ |Q| ] of this set to encodeX t using log( |Q| ) bits.• To quantize X t,a t X Tt,a t : Let X 2 t,a t denote a vector that collects the diagonal entries of X t,a t X Tt,a t .Let
Revised: We then use any enumeration h : Q → [ |Q| ] of this set to encodeX t using log( |Q| ) bits.• To quantize X t,a t X Tt,a t : Let X 2 t,a t denote a vector that collects the diagonal entries of X t,a t X Tt,a t .

Original:  ˆ X t (cid:44) s t X t /m be the estimate of X t,a t that the central learner retrieves.
Revised: Let ˆ X t (cid:44) s t X t /m be the estimate of X t,a t that the central learner retrieves.

Original: Note that ˆ X 2 t is notan unbiased estimate of X 2 t,a t ; however, ( X 2 t,a t − ˆ X 2 t ) i ≤ 3 /m for all coordinates i (proved inApp.
Revised: Note that ˆ X 2 t is notan unbiased estimate of X 2 t,a t ; however, ( X 2 t,a t − ˆ X 2 t ) i ≤ 3 /m for all coordinates i (proved in

Original:  C).
Revised: App. C).

Original: Our scheme simply conveys the difference X 2 t,a t − ˆ X 2 t with 1 bit per coordinate using a
Revised: Our scheme simply conveys the difference X 2 t,a t − ˆ X 2 t with 1 bit per coordinate using a

Original: SQ [ − 3 /m, 3 /m ]quantizer.
Revised: SQ [ − 3 /m, 3 /m ]quantizer.

Original: The central learner and distributed agent operations are presented in Algorithm 2.
Revised: The central learner and distributed agent operations are presented in Algorithm 2.

Original: Example 2.
Revised: Example 2.

Original: Consider the case where d = 5 .
Revised: Consider the case where d = 5 .

Original: Then each coordinate of | X t,a t | is scaled by 3 andquantized using SQ 3 to one of the values 0 , 1 , 2 , 3 to get X t .
Revised: Then each coordinate of | X t,a t | is scaled by 3 andquantized using SQ 3 to one of the values 0 , 1 , 2 , 3 to get X t .

Original: The function h then maps the values for
Revised: The function h then maps the values for

Original: X t that satisfy the (cid:107) X t (cid:107) 1 ≤ 10 to a unique value (a code) in the set [ |Q| ] .
Revised: X t that satisfy the (cid:107) X t (cid:107) 1 ≤ 10 to a unique value (a code) in the set [ |Q| ] .

Original: For instance the value 3 .
Revised: For instance the value 3 .

Original: 1is not given a code, where 1 is the vector of all ones.
Revised: 1is not given a code, where 1 is the vector of all ones.

Original: However, note that for | X t,a t | to be mapped to3 . 1 , we must have 3 | ( X t,a t ) i
Revised: However, note that for | X t,a t | to be mapped to3 . 1 , we must have 3 | ( X t,a t ) i

Original: | ≥ 2 for all coordinates i , which cannot happen since it implies that(cid:107)
Revised: | ≥ 2 for all coordinates i , which cannot happen since it implies that(cid:107)

Original: X t,a t (cid:107) 2 ≥ 2 (cid:112) 5 / 6 > 1 which contradicts Assumption 1.Observe : Pull a ∈A
Revised: X t,a t (cid:107) 2 ≥ 2 (cid:112) 5 / 6 > 1 which contradicts Assumption 1.

Original: X t,a

Original: θ t −

Original: reward

Original: : Compute the

Original: t =

Original: X t,a t / | X t,a t | of X t,a t . 10:

Original: Let X t =

Original: SQ m ( m | X t,a t | ) .

Original: 11: e 2 t = SQ [

Original: [Equation]

Original: 3 /m, 3 /m ]( X 2 t,a t − ˆ X 2 t ) , where ˆ X t =

Original: s t X t /m .

Original: Send to the central learner h ( X t ) , s t , and e 2 t using log 2 ( |Q| ) , d , and d bits, respectively.

Original: Send ˆ r t = SQ 1 ( r t ) using 1-bit.

Original: Central learner : 15:

Original: Receive X t , s t , e 2 t , and ˆ r t from the distributed agent.

Original: [Equation]

Original: [Equation]

Original: ←

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ←

Original: [Equation]

Original: Send ˆ θ t to the next agent.

Original: Algorithm Performance.
Revised: Algorithm Performance.

Original: Theorem 2, stated next, holds under Assumption 1 in Section 2 and someadditional regulatory assumptions on the distributions P a provided in Assumption 2.
Revised: Theorem 2, stated next, holds under Assumption 1 in Section 2 and someadditional regulatory assumptions on the distributions P a provided in Assumption 2.

Original: Assumption 2.
Revised: Assumption 2.

Original: There exist constants c, c (cid:48) such that for any sequence θ 1 , ..., θ T , where θ t dependsonly on H t , with probability at least 1 − c (cid:48) T , it holds that
Revised: There exist constants c, c (cid:48) such that for any sequence θ 1 , ..., θ T , where θ t dependsonly on H t , with probability at least 1 − c (cid:48) T , it holds that

Original: [Equation]
Revised: [Equation]

Original: where a t = arg max a ∈A (cid:104) X t,a , θ t (cid:105) , and I is the identity matrix.
Revised: where a t = arg max a ∈A (cid:104) X t,a , θ t (cid:105) , and I is the identity matrix.

Original: We note that several common assumptions in the literature imply (12), for example, boundedeigenvalues for the covariance matrix of X t,a t [10, 22, 15].
Revised: We note that several common assumptions in the literature imply (12), for example, boundedeigenvalues for the covariance matrix of X t,a t [11, 27, 17].

Original: Such assumptions hold for a wide rangeof distributions, including subgaussian distribitions with bounded density [27].
Revised: Such assumptions hold for a widerange of distributions, including subgaussian distribitions with bounded density [36].

Revised: We discuss thechallenge in relaxing this assumption in App. C.

Original: Theorem 2.
Revised: Theorem 2.

Original: Algorithm 2 satisﬁes that for all t : X t ∈ Q ; and B t ≤ 1 + log 2 (2 d + 1) + 5 .
Revised: Algorithm 2 satisﬁes that for all t : X t ∈ Q ; and B t ≤ 1 + log 2 (2 d + 1) + 5 .

Original: 03 d bits.
Revised: 03 d bits.

Original: Under assumptions 1, 2, it achieves a regret R T
Revised: Under assumptions 1, 2, it achieves a regret R T

Original: = O ( d √ T log T )
Revised: = O ( d √ T log T )

Original: with probability at least 1 − 1 T .
Revised: with probability at least 1 − 1 T .

Original: Proof Outline.
Revised: Proof Outline.

Original: To bound the number of bits B t , we ﬁrst bound the size of Q by formulating astandard counting problem: we ﬁnd the number of non-negative integer solutions for a linear equation.
Revised: To bound the number of bits B t , we ﬁrst bound the size of Q by formulating astandard counting problem: we ﬁnd the number of non-negative integer solutions for a linear equation.

Original: To bound the regret R T , we start by proving that our quantization scheme guarantees some desirableproperties, namely, unbiasedness and O ( 1 √ d ) quantization error for each vector coordinate.
Revised: To bound the regret R T , we start by proving that our quantization scheme guarantees some desirableproperties, namely, unbiasedness and O ( 1 √ d ) quantization error for each vector coordinate.

Original: We thenupper bound the regret in terms of (cid:107) ˆ θ t − θ (cid:63) (cid:107) 2 and show that this difference can be decomposed as(cid:107)
Revised: We thenupper bound the regret in terms of (cid:107) ˆ θ t − θ (cid:63) (cid:107) 2 and show that this difference can be decomposed as(cid:107)

Original: [Equation]
Revised: [Equation]

Revised: 1: Input: underlying set of actions A , and time horizon T .

Original: (cid:63) (cid:107) 2 = (cid:107) V − 1 t{+
 = 0 , u 0 = 0 , m = (cid:100)√ d (cid:101) . 3: Let h be an enumeration of the set Q = { x ∈ N d |(cid:107) x+} (cid:107)  2 ( (cid:107) t (cid:88) i =
Revised: 2: ˆ θ 0 = 0 , ˜ V {+
 = 0 , u 0 = 0 , m = (cid:100)√ d (cid:101) . 3: Let h be an enumeration of the set Q = { x ∈ N d |(cid:107) x+} (cid:107) 1 ≤ 2 d } .

Original: [-E i (cid:107) 
-]
Revised: [-E i (cid:107) 
-]4: for t = 1 : T do 5: Agent : 6: Receive ˆ θ t − 1 from the central learner. Observe context realization { X t,a } a ∈A .

Original: + (1 + | η i | ) (cid:107) t (cid:88) i
Revised: Pull arm a t = arg max a ∈A (cid:104) X t,a , ˆ θ t − 1 (cid:105) and receive reward r t . Compute the signs s t = X t,a t / | X t,a t | of X t,a t . 10: Let X t = SQ m ( m | X t,a t | ) .

Original:  =  e [-i (cid:107) 
-]
Revised: 11: e 2 t = SQ [ − 3 /m, 3 /m ]( X 2 t,a t − ˆ X 2 t ) , where ˆ X t = s t X t /m . Send to the central learner h ( X t ) , s t , and e [-i (cid:107) 
-]2 t using log 2 ( |Q| ) , d , and d bits, respectively. Send ˆ r t = SQ 1 ( r t ) using 1-bit.

Original: + (cid:107) t(cid:88) i
Revised: Central learner : 15: Receive X t , s t , e 2 t , and ˆ r t from the distributed agent. 16: ˆ X t = s t X t /m , ˆ X ( D ) t

Original: = 
Revised: = ˆ X 2 t + e 2 t . 17: u t ← u t − 1 + ˆ r t ˆ X t . 18: ˜ V t ← ˜ V t − 1 + ˆ X t

Original: ˆη i X
Revised: ˆ X

Revised: Tt − diag ( ˆ X t ˆ X Tt )

Revised: + diag ( ˆ X ( D ) t ) .

Revised: 19: ˆ θ t ← ˜ V − 1 t u t .

Original: i,a i (cid:107) 2 , (13) where E t captures the error in estimating the matrix X t,a t X Tt,a t , e t is the error in estimating X t,a t , andη (cid:48) t is a noise that satisﬁes the same properties as η t .
Revised: Send ˆ θ t to the next agent. where E t captures the error in estimating the matrix X t,a t X Tt,a t , e t is the error in estimating X t,a t , andη (cid:48) t is a noise that satisﬁes the same properties as η t .

Original: Using Assumption 2, we prove that V − 1 t grows as
Revised: Using Assumption 2, we prove that V − 1 t grows as

Original: O ( dt ) with high probability, and from the unbiasdness and boundedness of all error quantities we showthat they grow as O ( √ t log t ) with high probability.
Revised: O ( dt ) with high probability, and from the unbiasdness and boundedness of all error quantities we showthat they grow as O ( √ t log t ) with high probability.

Original: This implies that (cid:107) ˆ θ t − θ (cid:63) (cid:107) 

Revised: This implies that (cid:107) ˆ θ t − θ (cid:63) (cid:107)

Original: = O ( d (cid:113) log t t ) ,and hence, R T
Revised: = O ( d (cid:113) log t t ) ,and hence, R T

Original: = O ( d √ T log T ) .
Revised: = O ( d √ T log T ) .

Original: The complete proof is provided in App.
Revised: The complete proof is provided in App.

Original: C.
Revised: C.

Original: (cid:3)
Revised: (cid:3)

Original: Algorithm Complexity.
Revised: Algorithm Complexity.

Original: If we do not count the quantization operations, it is easy to see that thecomplexity of the rest of the algorithm is dominated by the complexity of computing V − 1 t which canbe done in O ( d 2 . 373 ) [3].
Revised: If we do not count the quantization operations, it is easy to see that thecomplexity of the rest of the algorithm is dominated by the complexity of computing V − 1 t which canbe done in O ( d 2 . 373 ) [4].

Original: For the quantization, we note that each coordinate of X t can be computedin ˜ O
Revised: For the quantization, we note that each coordinate of X t can be computedin ˜ O

Original: (1) time 4 .
Revised: (1) time 4 .

Original: Moreover, the computation of h ( x ) for x ∈ Q can be done in constant time withhigh probability using hash tables, where h is the enumeration function in Step 3.
Revised: Moreover, the computation of h ( x ) for x ∈ Q can be done in constant time withhigh probability using hash tables, where h is the enumeration function in Step 3.

Original: Hence, the addedcomputational complexity is almost linear in d .
Revised: Hence, the addedcomputational complexity is almost linear in d .

Original: Although a hash table for h can consume Ω(2 5 d )memory, by sacriﬁcing a constant factor in the number of bits, we can ﬁnd enumeration functionsthat can be stored efﬁciently.
Revised: Although a hash table for h can consume Ω(2 5 d )memory, by sacriﬁcing a constant factor in the number of bits, we can ﬁnd enumeration functionsthat can be stored efﬁciently.

Original: As an example, consider the scheme in [13] that can ﬁnd an one-to-onefunction h : Q → N + which can be stored and computed efﬁciently, but only gives guarantees inexpectation that E [log( h ( x ))]
Revised: As an example, consider the scheme in [14] that can ﬁnd an one-to-onefunction h : Q → N + which can be stored and computed efﬁciently, but only gives guarantees inexpectation that E [log( h ( x ))]

Original: = O ( d ) for all x ∈ Q .
Revised: = O ( d ) for all x ∈ Q .

Original: Downlink Communication Cost.
Revised: Downlink Communication Cost.

Original: Although we assume no-cost downlink communication, as wasalso the case for Algorithm 1, the downlink in Algorithm 2 is only used to send the updated parametervector ˆ θ t to the agents.
Revised: Although we assume no-cost downlink communication, as wasalso the case for Algorithm 1, the downlink in Algorithm 2 is only used to send the updated parametervector ˆ θ t to the agents.

Original: If desired, these estimates can be quantized using the same method as for
Revised: If desired, these estimates can be quantized using the same method as for

Original: X t,a t , which (following a similar proof to that of Theorem 2) can be shown to not affect the order ofthe regret while reducing the downlink communication to ≈ 5 d bits per iteration.
Revised: X t,a t , which (following a similar proof to that of Theorem 2) can be shown to not affect the order ofthe regret while reducing the downlink communication to ≈ 5 d bits per iteration.

Original: Ofﬂoading To Agents.
Revised: Ofﬂoading To Agents.

Original: For applications where the agents wish to computationally help the centrallearner, the central learner may simply aggregate information to keep track of u t , ˜ V t and broadcastthese values to the agents; the estimate ˆ θ t can be calculated at each agent.
Revised: For applications where the agents wish to computationally help the centrallearner, the central learner may simply aggregate information to keep track of u t , ˜ V t and broadcastthese values to the agents; the estimate ˆ θ t can be calculated at each agent.

Original: Moving the computationalload to the agents does not affect the regret order or the number of bits communicated on the uplink.
Revised: Moving the computationalload to the agents does not affect the regret order or the number of bits communicated on the uplink.

Original: Remark 2.
Revised: Remark 2.

Original: Under the regulatory assumptions in [15], the regret bound can be improved by a factorof (cid:112) log( K ) /d , where K = |A| is the number of actions.
Revised: Under the regulatory assumptions in [17], the regret bound can be improved by a factorof (cid:112) log( K ) /d , where K = |A| is the number of actions.

Original: However, this does not improve the regretin the worst case as the worst case number of actions is O ( C d ) , C > 1 [19].
Revised: However, this does not improve the regretin the worst case as the worst case number of actions is O ( C d ) , C > 1 [24].

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeApp. A.
Revised: [Yes] SeeApp. A.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Theorem and Theorem 2.
Revised: [Yes] See Theorem 1and Theorem 2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See App.
Revised: [Yes] See App.

Original: B and App. C.
Revised: B andApp. C.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original:  (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: [Table] (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original:  (c) Did you include the estimated hourly wage paid to participants and the total amount 
Revised: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]