Original: A Simple and Provably Efﬁcient Algorithm for Asynchronous Federated Contextual Linear Bandits
Revised: A Simple and Provably Efﬁcient Algorithm for Asynchronous Federated Contextual Linear Bandits

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We study federated contextual linear bandits, where M agents cooperate with eachother to solve a global contextual linear bandit problem with the help of a centralserver.
Revised: We study federated contextual linear bandits, where M agents cooperate with eachother to solve a global contextual linear bandit problem with the help of a centralserver.

Original: We consider the asynchronous setting, where all agents work independentlyand the communication between one agent and the server will not trigger otheragents’ communication.
Revised: We consider the asynchronous setting, where all agents work independentlyand the communication between one agent and the server will not trigger otheragents’ communication.

Original: We propose a simple algorithm named FedLinUCB basedon the principle of optimism.
Revised: We propose a simple algorithm named FedLinUCB basedon the principle of optimism.

Original: We prove that the regret of FedLinUCB is boundedby e
Revised: We prove that the regret of FedLinUCB is boundedby e

Original: O ( d qP
Revised: O ( d qP

Original: Mm =1 T m ) and the communication complexity is e
Revised: Mm =1 T m ) and the communication complexity is e

Original: O ( dM 2 ) , where d isthe dimension of the contextual vector and T m is the total number of interactionswith the environment by m -th agent.
Revised: O ( dM 2 ) , where d isthe dimension of the contextual vector and T m is the total number of interactionswith the environment by m -th agent.

Original: To the best of our knowledge, this is the ﬁrstprovably efﬁcient algorithm that allows fully asynchronous communication forfederated contextual linear bandits, while achieving the same regret guarantee as inthe single-agent setting.
Revised: To the best of our knowledge, this is the ﬁrstprovably efﬁcient algorithm that allows fully asynchronous communication forfederated contextual linear bandits, while achieving the same regret guarantee as inthe single-agent setting.

Original:  Introduction
Revised:  Introduction

Original: Contextual linear bandit is a canonical model in sequential decision making with partial informationfeedback that has found vast applications in real-world domains such as recommendation systems (Liet al., 2010a,b; Gentile et al., 2014; Li et al., 2020), clinical trials (Wang, 1991; Durand et al.
Revised: Contextual linear bandit is a canonical model in sequential decision making with partial informationfeedback that has found vast applications in real-world domains such as recommendation systems(Li et al., 2010a,b; Gentile et al., 2014; Li et al., 2020), clinical trials (Wang, 1991; Durand et al.

Original: , 2018)and economics (Jagadeesan et al., 2021; Li et al., 2022).
Revised: and economics (Jagadeesan et al., 2021; Li et al., 2022).

Original: Most existing works on contextual linearbandits focus on either the single-agent setting (Auer, 2002;
Revised: Most existing works on contextuallinear bandits focus on either the single-agent setting (Auer, 2002;

Original: Abe et al., 2003; Dani et al., 2008;
Revised: Abe et al., 2003; Dani et al., 2008;

Original: Li et al., 2010a; Rusmevichientong and Tsitsiklis, 2010; Chu et al., 2011; Abbasi-Yadkori et al.
Revised: Li et al., 2010a; Rusmevichientong and Tsitsiklis, 2010; Chu et al., 2011; Abbasi-Yadkori et al.

Original: Agrawal and Goyal, 2013) or multi-agent settings where communications between agentsare instant and unrestricted (Cesa-Bianchi et al., 2013; Li et al., 2016; Wu et al., 2016;
Revised: Agrawal and Goyal, 2013) or multi-agent settings where communications between agentsare instant and unrestricted (Cesa-Bianchi et al., 2013; Li et al., 2016; Wu et al., 2016;

Original: Li et al.,2021).
Revised: Li et al.,2021).

Original: Due to the increasing amount of data being distributed across a large number of local agents(e.g., clients, users, edge devices), federated learning (McMahan et al., 2017; Karimireddy et al.
Revised: Due to the increasing amount of data being distributed across a large number of local agents(e.g., clients, users, edge devices), federated learning (McMahan et al., 2017; Karimireddy et al.

Original: ,2020) has become an emerging paradigm for distributed machine learning, where agents can jointlylearn a global model without sharing their own localized data.
Revised: ,2020) has become an emerging paradigm for distributed machine learning, where agents can jointlylearn a global model without sharing their own localized data.

Original: This motivates the development ofdistributed/federated linear bandits (Wang et al., 2019; Huang et al., 2021; Li and Wang, 2022a),which enables a collection of agents to cooperate with each other to solve a global linear banditproblem while enjoying performance guarantees comparable to those in the classical single-agentcentralized setting.
Revised: This motivates the development ofdistributed/federated linear bandits (Wang et al., 2019; Huang et al., 2021; Li and Wang, 2021), whichenables a collection of agents to cooperate with each other to solve a global linear bandit problemwhile enjoying performance guarantees comparable to those in the classical single-agent centralizedsetting

Original: However, most existing federated linear bandits algorithms are limited to the synchronous setting (Wang et al., 2019; Dubey and Pentland, 2020; Huang et al., 2021), where all the agents have toﬁrst upload their local data to the server upon the request of the server, and the agents will downloadthe latest data from the server after all uploads are complete.
Revised: However, most existing federated linear bandits algorithms are limited to the synchronous setting (Wang et al., 2019; Dubey and Pentland, 2020; Huang et al., 2021), where all the agents have toﬁrst upload their local data to the server upon the request of the server, and the agents will downloadthe latest data from the server after all uploads are complete.

Original: This requires full participation of theagents and global synchronization mandated by the server, which is impractical in many real-worldapplication scenarios.
Revised: This requires full participation of theagents and global synchronization mandated by the server, which is impractical in many real-worldapplication scenarios.

Original: The only notable exception is Li and Wang (2022a), where an asynchronousfederated is proposed
Revised: The only notable exception is Li and Wang (2021), where an asynchronousfederated is proposed

Original: Nevertheless in their  the uploadby one agent may trigger the download from the server to all other agents.
Revised: Nevertheless in their , the upload one agent may trigger the download from the server to all other agents.

Original: Therefore, the communicationsbetween different agents and the server are not totally independent.
Revised: Therefore, the communicationsbetween different agents and the server are not totally independent.

Original: Moreover, they make a stringentregularity assumption on the contexts, which basically requires the contexts to be stochastic ratherthan adversarial as in standard contextual linear bandits.
Revised: Moreover, their regret guaranteerelies on a stringent regularity assumption on the contexts, which basically requires the contexts to bestochastic rather than adversarial as in standard contextual linear bandits.

Original: That said, their theoretical proof is actually ﬂawed as they ignored some unique challenges caused by asynchronous communication (see

Original: Appendix A.1 for details).

Original: Therefore, how to design a truly asynchronous contextual linear banditalgorithm remains an open problem.
Revised: Therefore, how to design atruly asynchronous contextual linear bandit algorithm remains an open problem.

Original: In this work, we resolve the above open problem by proposing a simple algorithm for asynchronousfederated contextual linear bandits over a star-shaped communication network.
Revised: In this work, we resolve the above open problem by proposing a simple algorithm for asynchronousfederated contextual linear bandits over a star-shaped communication network.

Original: Our algorithm isbased on the principle of optimism (Abbasi-Yadkori et al., 2011) and enjoys the following advantages:(i)
Revised: Our algorithm isbased on the principle of optimism (Abbasi-Yadkori et al., 2011).

Original:  Each agent can decide whether or not to participate in each round.
Revised: The communication protocol ofour algorithm enjoys the following features: (i) Each agent can decide whether or not to participatein each round.

Original: Full participation is not required,thus it allows temporarily ofﬂine agents.
Revised: Full participation is not required, thus it allows temporarily ofﬂine agents; and (ii)

Original: This is much more ﬂexible than existing algorithms forfederated linear bandits in Wang et al. (2019); Dubey and Pentland (2020) ; Huang et al. (2021) whereall agents are required to participate in each round; and (ii) the communication between each agentand the server is asynchronous and totally independent of other agents.
Revised: The communication between each agent and the server is asynchronous and totally independentof other agents.

Original: There is no need of globalsynchronization or mandatory coordination by the server, in contrast to Li and Wang (2022a) whereeach agent might be asked by the server to download data.
Revised: There is no need of global synchronization or mandatory coordination by theserver.

Original: In particular, the communication betweenthe agent and the server is triggered by a matrix determinant-based criterion that can be computedindependently by each agent.
Revised: In particular, the communication between the agent and the server is triggered by a matrixdeterminant-based criterion that can be computed independently by each agent.

Original: Our algorithm design not only allows the agents to independentlyoperate and synchronize with the server, but also ensures low communication complexity (i.e., totalnumber of rounds of communication between agents and the server) and low switching cost (i.e.,total number of local model updates for all agents) (Abbasi-Yadkori et al., 2011).
Revised: Our algorithmdesign not only allows the agents to independently operate and synchronize with the server, but alsoensures low communication complexity (i.e., total number of rounds of communication betweenagents and the server) and low switching cost (i.e., total number of local model updates for allagents) (Abbasi-Yadkori et al., 2011).

Original: While being simple, our algorithm design introduces a challenge in the regret analysis.
Revised: While being simple, our algorithm design introduces a challenge in the regret analysis.

Original: Since theorder of the interaction between the agent and the environment is not ﬁxed, standard martingale-basedconcentration inequality cannot be directly applied.
Revised: Since theorder of the interaction between the agent and the environment is not ﬁxed, standard martingale-basedconcentration inequality cannot be directly applied.

Original: Speciﬁcally, this challenge arises due to themismatch between the partial data information collected by the central server and the true order of thedata generated from the interaction with the environment, as is explained in detail in Section 5 andillustrated by Figure 1.
Revised: Speciﬁcally, this challenge arises due to themismatch between the partial data information collected by the central server and the true order of thedata generated from the interaction with the environment, as is explained in detail in Section 5 andillustrated by Figure 1.

Original: We address this challenge by a novel proof technique, which ﬁrst establishesthe local concentration of each agent’s data and then relates it to the “virtual” global concentration ofall data via the determinant-based criterion.
Revised: We address this challenge by a novel proof technique, which ﬁrst establishesthe local concentration of each agent’s data and then relates it to the “virtual” global concentration ofall data via the determinant-based criterion.

Original: Based on this proof technique, we are able to obtain tightenough conﬁdence bounds that leads to a nearly optimal regret.
Revised: Based on this proof technique, we are able to obtain tightenough conﬁdence bounds that leads to a nearly optimal regret.

Original: Moreover, our theoretical analysisrelies only on minimal assumptions that are standard for contextual linear bandits, relaxing the strongassumptions made in Li and Wang (2022a).

Original: Main contributions.
Revised: Main contributions.

Original: Our contributions are highlighted as follows:
Revised: Our contributions are highlighted as follows:

Original: • We devise a simple algorithm named FedLinUCB that achieves near-optimal regret, low communication complexity and low switching cost simultaneously for asynchronous federated contextual linearbandits.
Revised: • We devise a simple algorithm named FedLinUCB that achieves near-optimal regret, low communication complexity and low switching cost simultaneously for asynchronous federated contextual linearbandits.

Original: In detail, we prove that our algorithms achieves a near-optimal e O ( d p T ) regret with merelye O ( dM 2 ) total communication complexity and e O ( dM 2 ) total switching cost.
Revised: In detail, we prove that our algorithms achieves a near-optimal e O ( d p T ) regret with merely e O ( dM 2 ) total communication complexity and e O ( dM 2 ) total switching cost.

Original: Here M is the numberof agents, d is the dimension of the context and T =
Revised: Here M is the numberof agents, d is the dimension of the context and T =

Original: P Mm =1 T m is the total number of rounds withT m being the number of rounds that agent m participates in.
Revised: P Mm =1 T m is the total number of rounds with T m being the number of rounds that agent m participates in.

Original: When degenerated to single-agentbandits, the regret of our algorithm matches the optimal regret
Revised: When degenerated to single-agent bandits, the regret of our algorithm matches the optimal regret

Original: e
Revised: e

Original: O ( d p T ) (Abbasi-Yadkori et al.,
Revised: [Equation] Abbasi-Yadkori et al.

Original:  We also prove an ⌦ ( M/ log( T/M )) lower bound for the communication complexity.
Revised: 2011).• We also prove an ⌦ ( M/ log( T/M )) lower bound for the communication complexity.

Original: Togetherwith the O ( dM 2 ) upper bound of our algorithm, it suggests that there is only an e O ( dM ) gapbetween the upper and lower bounds of the communication complexity.•
Revised: Togetherwith the O ( dM 2 ) upper bound of our algorithm, it suggests that there is only an e O ( dM ) gapbetween the upper and lower bounds of the communication complexity.•

Original: We identify the issue of ill-deﬁned ﬁltration caused by the unﬁxed order of interactions betweenagents and the environment, which is absent in previous synchronous or single-agent settings.
Revised: We identify the issue of ill-deﬁned ﬁltration caused by the unﬁxed order of interactions betweenagents and the environment, which is absent in previous synchronous or single-agent settings.

Original: Wetackle this unique challenge by connecting the local concentration of each local agent and theglobal concentration of the aggregated data from all agents.
Revised: Wetackle this unique challenge by connecting the local concentration of each local agent and theglobal concentration of the aggregated data from all agents.

Original: We believe this proof technique is ofindependent interest for the analysis of other asynchronous bandit problems.
Revised: We believe this proof technique is ofindependent interest for the analysis of other asynchronous bandit problems.

Original: Notation.
Revised: Notation.

Original: For any positive integer n , we denote the set { 1 , 2 , . . . , n } by [ n ] .
Revised: For any positive integer n , we denote the set { 1 , 2 , . . . , n } by [ n ] .

Original: We use I to denote thed ⇥ d identity matrix.
Revised: We use I to denote thed ⇥ d identity matrix.

Original: We use O to hide universal constants and e O to further hide poly-logarithmicterms.
Revised: We use O to hide universal constants and e O to further hide poly-logarithmic terms.

Original: For any vector x 2 R d and positive semi-deﬁnite ⌃ 2 R d ⇥ d , we denote k x k ⌃ =
Revised: For any vector x 2 R d and positive semi-deﬁnite ⌃ 2 R d ⇥ d , we denote k x k ⌃ =

Original: p x > ⌃ x .
Revised: p x > ⌃ x .

Revised:  Related Work

Revised: In this section, we review related work on distributed/federated bandit algorithms based on the type ofbandits: (1) multi-armed bandits; and (2) stochastic linear bandits; and (3) contextual linear bandits.

Original: Setting
Revised: Setting

Original: Algorithm
Revised: Algorithm

Original: Regret
Revised: Regret

Original: Communication
Revised: Communication

Original: Low-switching
Revised: Low-switching

Original: Noextraassump.oncontexts

Original: Allowfreeparticipation

Original: Single-agent
Revised: Single-agent

Original: OFUL d p T log T
Revised: OFUL d p T log T

Original: N/A 8 (Abbasi-Yadkorietal.,2011)
Revised: N/A 3 (Abbasi-Yadkori et al., 2011)

Original: Federated
Revised: Federated

Original: DisLinUCB d p T log 2 T dM 3 /  8 (Sync.)
Revised: DisLinUCB d p T log 2 T dM 3 /8 (synchr.)

Original: (Wangetal.,2019)
Revised: (Wang et al., 2019)

Original: Federated
Revised: Federated

Original: Async-LinUCB d p T log T dM 2 log T 8 (Async.)
Revised: Async-LinUCB d p T log T dM 2 log T 8 (Asynchr.)

Original: (LiandWang,2022a)
Revised: (Li and Wang, 2021)

Original: Federated
Revised: Federated

Original: FedLinUCB d p T log T dM 2 log T 3 (Async.)
Revised: FedLinUCB d p T log T dM 2 log T 3 (Asynchr.)

Original: (Our Algorithm 1)  Related Work We review related work on distributed/federated bandit algorithms based on the type of bandits: (1)multi-armed bandits, (2) stochastic linear bandits and (3) contextual linear bandits.
Revised: (ours)

Original: Distributed/federated multi-armed bandits.
Revised: Distributed/federated multi-armed bandits.

Original: There is a vast literature on distributed/federatedmulti-armed bandits (MABs) (Liu and Zhao, 2010; Szorenyi et al., 2013; Landgren et al., 2016;
Revised: There is a vast literature on distributed/federatedmulti-armed bandits (MABs) (Liu and Zhao, 2010; Szorenyi et al., 2013; Landgren et al., 2016;

Original: Chakraborty et al., 2017; Landgren et al., 2018; Martínez-Rubio et al., 2019; Sankararaman et al.
Revised: Chakraborty et al., 2017; Landgren et al., 2018; Martínez-Rubio et al., 2019; Sankararaman et al.

Original: Wang et al., 2019, 2020; Zhu et al., 2021), to mention a few.
Revised: Wang et al., 2019, 2020; Zhu et al., 2021), to mention a few.

Original: However, none of these algorithmscan be directly applied to linear bandits, needless to say contextual linear bandits with inﬁnite decisionsets.
Revised: However, none of these algorithmscan be directly applied to linear bandits, needless to say contextual linear bandits with inﬁnite decisionsets.

Original: Distributed/federated stochastic linear bandits.
Revised: Distributed/federated stochastic linear bandits.

Original: In distributed/federated stochastic linear bandits,the decision set is ﬁxed across all the rounds t 2 [ T ] and all the agents m 2 [ M ] .
Revised: In distributed/federated stochastic linear bandits,the decision set is ﬁxed across all the rounds t 2 [ T ] and all the agents m 2 [ M ] .

Original: Wang et al.
Revised: Wang et al.

Original: (2019)proposed the DELD algorithm for distributed stochastic linear bandits on both star-shaped network and
Revised: (2019)proposed the DELD algorithm for distributed stochastic linear bandits on both star-shaped network and

Original: P2P network.
Revised: P2P network.

Original: Huang et al.
Revised: Huang et al.

Original: (2021) proposed an arm elimination-based algorithm called Fed-PE forfederated stochastic linear bandits on the star-shaped network.
Revised: (2021) proposed an arm elimination-based algorithm called Fed-PE forfederated stochastic linear bandits on the star-shaped network.

Original: Both algorithms are in the synchronoussetting and require full participation of the agents upon the server’s request.
Revised: Both algorithms are in the synchronoussetting and require full participation of the agents upon the server’s request.

Original: Distributed/federated contextual linear bandits.
Revised: Distributed/federated contextual linear bandits.

Original: The contextual linear bandit is more general andchallenging than stochastic linear bandits, because the decision sets can vary for each t and m .
Revised: The contextual linear bandit is more general andchallenging than stochastic linear bandits, because the decision sets can vary for each t and m .

Original: Inthis setting, Korda et al.
Revised: Inthis setting, Korda et al.

Original: (2016) considered a P2P network and proposed the DCB algorithm based onthe OFUL algorithm in Abbasi-Yadkori et al.
Revised: (2016) considered a P2P network and proposed the DCB algorithm based onthe OFUL algorithm in Abbasi-Yadkori et al.

Original: Wang et al.
Revised: Wang et al.

Original: (2019) considered both star-shapedand P2P communication networks and achieved the near-optimal e
Revised: (2019) considered both star-shapedand P2P communication networks and achieved the near-optimal e

Original: O ( d p T ) regret in the synchronoussetting.
Revised: O ( d p T ) regret in the synchronoussetting.

Original: 1 Dubey and Pentland (2020) further introduced the differential privacy guarantee into thesetting of Wang et al.
Revised: 1 Dubey and Pentland (2020) further introduced the differential privacy guarantee into thesetting of Wang et al.

Original: Li and Wang (2022b) extended distributed contextual linear bandits togeneralized linear bandits (Filippi et al., 2010;
Revised: Li and Wang (2022) extended distributed contextual linear bandits togeneralized linear bandits (Filippi et al., 2010;

Original: Jun et al., 2017) in the synchronous setting.
Revised: Jun et al., 2017) in the synchronous setting.

Original: Li and
Revised: Li and

Original: Wang (2022a) proposed the ﬁrst asynchronous algorithm for federated contextual linear bandits withthe star-shaped graph and achieve near-optimal e
Revised: Wang (2021) proposed the ﬁrst asynchronous algorithm for federated contextual linear bandits withthe star-shaped graph and achieve near-optimal e

Original: O ( d p T ) regret.
Revised: O ( d p T ) regret.

Original: However, their setting is differentfrom ours in two aspects: (1) the upload triggered by an agent will lead the server to trigger downloadpossibly for all the agents in their setting.
Revised: However, their setting is differentfrom ours in two aspects: (1) the upload triggered by an agent will lead the server to trigger downloadpossibly for all the agents in their setting.

Original: In contrast, the upload triggered by an agent will onlylead to download to the same agent in our setting; (2) their regret guarantee relies on a stringentregularity assumption on the contexts, which basically requires the contexts to be stochastic.
Revised: In contrast, the upload triggered by an agent will onlylead to download to the same agent in our setting; (2) their regret guarantee relies on a stringentregularity assumption on the contexts, which basically requires the contexts to be stochastic.

Original: Asa comparison, the contexts in our setting can be even adversarial, which is exactly the setting ofcontextual linear bandits (Abbasi-Yadkori et al., 2011; Li et al., 2019).
Revised: Asa comparison, the contexts in our setting can be even adversarial, which is exactly the setting ofcontextual linear bandits (Abbasi-Yadkori et al., 2011; Li et al., 2019).

Original: This difference in the settingmakes our algorithm a truly asynchronous contextual linear bandit algorithm but also makes ourregret analysis more challenging.
Revised: This difference in the settingmakes our algorithm a truly asynchronous contextual linear bandit algorithm but also makes ourregret analysis more challenging.

Original: For better comparison, we compare our work with the most related contextual linear bandit algorithmsin Table 1. 
Revised: For better comparison, we compare our work with the most related contextual linear bandit algorithmsin Table 1. e

Original:  Preliminaries
Revised:  Preliminaries

Original: [Figure]

Original: Federated contextual linear bandits.
Revised: Federated contextual linear bandits.

Original: We consider the federated contextual linear bandits as follows: At each round t 2 [ T ] , an arbitrary agent m t 2 [ M ] is active for participation.
Revised: We consider the federated contextual linear bandits as follows:at each round t 2 [ T ] , an arbitrary agent m t 2 [ M ] is active for participation.

Original: This agent receives adecision set D t ⇢ R d , picks an action x t 2 D t, and receives a random reward r t .
Revised: This agent receivesa decision set D t ⇢ R d , picks an action x t 2 D t and receives a random reward r t .

Original: We assume thatthe reward r t satisﬁes r t = h x t , ✓ ⇤ i + ⌘ t for all t 2 [ T ] , where ⌘ t is  conditionally independent of  x tgiven x 1: t 1 , m 1: t , r 1: t  1 .
Revised: We assume thereward r t satisﬁes r t = h x t , ✓ ⇤ i + ⌘ t for all t 2 [ T ] , where ⌘ t is a conditionally independent of 123 x t given x 1: t , m 1: t , r 1: t  1 .

Original: More speciﬁcally, we make the following assumption on ⌘ t , ✓ ⇤ and D t , 130 which is a standard assumption in the contextual linear bandit literature (Abbasi-Yadkori et al., 2011;
Revised: More speciﬁcally, we make the following assumption on ⌘ t , ✓ ⇤ and D t ,which is a standard assumption in the contextual linear bandit literature (Abbasi-Yadkori et al., 2011;

Original: Wang et al., 2019; Dubey and Pentland, 2020).
Revised: Wang et al., 2019; Dubey and Pentland, 2020).

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: The noise ⌘ t is R -sub-Gaussian conditioning on x 1: t , m 1: t and r 1: t  1 , i.e.,
Revised: The noise ⌘ t is R -sub-Gaussian conditioning on x 1: t , m 1: t and r 1: t  1 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: for any  2 R .
Revised: for any  2 R .

Original: We also assume that k ✓ ⇤ k 2  S and k x k 2  L for all action x 2 D t, for all t 2 [ T ] .
Revised: We also assume that k ✓ ⇤ k 2  S and k x k 2  L for all action x 2 D t .

Original: Notably, we assume m t can be arbitrary for all t , which basically says that each agent can decidewhether and when to participate or not.
Revised: Notably, we assume m t can be arbitrary for all t , which basically says that each agent can decidewhether and when to participate or not.

Original: 2 Our setting is more general than the synchronous setting in Wang et al. ; Dubey and Pentland (2020), which requires a round-robin participation of allagents.
Revised: 2 Our setting is more general than the synchronous setting (Wang et al., 2019; Dubey and Pentland, 2020) which requires a round-robin participation of allagents.

Original: Learning objective.
Revised: Learning objective.

Original: The goal of the agents is to collaboratively minimize the cumulative regretdeﬁned as
Revised: The goal of the agents is to collaboratively minimize the cumulative regretwhich is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: To achieve sucha goal, we allow the agents to collaborate via  communication through the centralserver.
Revised: To achieve such goal, we allow the agents to collaborate via the communication through the centralserver.

Original: Below we will explain the details of the communication model.
Revised: Below we will explain the details of the communication model.

Original: Communication model.
Revised: Communication model.

Original: We consider a star-shaped communication network (Wang et al., 2019;
Revised: We consider a star-shaped communication network (Wang et al., 2019;

Original: Dubey and Pentland, 2020) consisting of a central server and M agents, where each agent cancommunicate with the server by uploading and downloading data.
Revised: Dubey and Pentland, 2020) consisting of a central server and M agents, where each agent cancommunicate with the server by uploading and downloading data.

Original: However, any pair of agentscannot communicate with each other directly.
Revised: However, any two agents cannot communicate with each other directly.

Original: We deﬁne the communication complexity as the totalnumber of communication rounds between agents and the server (counting both the uploads anddownloads) (Wang et al., 2019; Dubey and Pentland, 2020; Li and Wang, 2022a).
Revised: We deﬁne the communication complexity as the totalnumber of communication rounds between agents and the server (counting both the uploads anddownloads) (Wang et al., 2019; Dubey and Pentland, 2020; Li and Wang, 2021).

Original: For simplicity, weassume that there is no latency in the communication channel.
Revised: For simplicity, weassume that there is no latency in the communication channel.

Original: We consider the asynchronous setting, where the communication protocol satisﬁes: (1) each agentcan decide whether or not to participate in each round.
Revised: We consider the asynchronous setting, where the communication protocol satisﬁes: (1) each agentcan decide whether or not to participate in each round.

Original: Full participation is not required, whichallows temporarily ofﬂine agents; and (2) the communication between each agent and the server isasynchronous and independent of other agents without mandatory download by the server.
Revised: Full participation is not required, whichallows temporarily ofﬂine agents; and (2) the communication between each agent and the server isasynchronous and independent of other agents without mandatory download by the server.

Original: Switching cost.
Revised: Switching cost.

Original: The notion of switching cost in online learning and bandits refers to the number oftimes the agent switches its policy (i.e., decision rule)
Revised: The notion of switching cost in online learning and bandits refers to the number oftimes the agent switches its policy (i.e., decision rule)

Original: (Kalai and Vempala, 2005; Abbasi-Yadkoriet al., 2011; Dekel et al., 2014; Ruan et al., 2021).
Revised: (Kalai and Vempala, 2005; Abbasi-Yadkoriet al., 2011; Dekel et al., 2014; Ruan et al., 2021).

Original: In the context of linear bandits, it correspondsto the number of times the agent updates its policy of selecting an action from the decision set(Abbasi-Yadkori et al., 2011).
Revised: In the context of linear bandits, it correspondsto the number of times the agent updates its policy of selecting an action from the decision set(Abbasi-Yadkori et al., 2011).

Original: Algorithms with low switching cost are preferred in practice sinceeach policy switching might cause additional computational overhead.
Revised: Algorithms with low switching cost are preferred in practice sinceeach policy switching will cause additional computational overhead.

Original:  The Proposed Algorithm
Revised:  The Proposed Algorithm

Original: We propose a simple algorithm based on the principle of optimism that enables collaboration amongagents through asynchronous communications with the central server.
Revised: In this section, we propose a simple algorithm based on the principle of optimism that enablescollaboration among agents through asynchronous communications with the central server.

Original: The main algorithm is displayedin Algorithm 1.
Revised: The mainalgorithm is displayed in Algorithm 1.

Original: For clarity, we ﬁrst summarize the related notations in Table 2.
Revised: For clarity, we ﬁrst summarize the related notations in Table 2.

Original: Speciﬁcally, ineach round t 2 [ T ] , agent m t participates and interacts with the environment (Line 3).
Revised: Speciﬁcally, in round t 2 [ T ] , agent m t participates and interacts with the environment (Line 3).

Original: The environment speciﬁes the decision set D t (Line 4), and the agent m t selects the action based onits current optimisitic estimate of the reward (Line 5).
Revised: The environment speciﬁes the decision set D t (Line 4), and the agent m t selects the action based onits current optimisitic estimate of the reward (Line 5).

Original: Here the bonus term  k x k ⌃  1 mt,t reﬂects theuncertainty of the estimated reward h b ✓ m t ,t , x i and encourages exploration.
Revised: Here the bonus term  k x k ⌃  1 mt,t reﬂects theuncertainty of the estimated reward h b ✓ m t ,t , x i and encourages exploration.

Original: After receiving the true 167 reward r t from the environment, agent m t then updates its local data (Line 7).
Revised: After receiving the true 161 reward r t from the environment, agent m t then updates its local data (Line 7).

Original: 1: Initialize ⌃ m, 1 = ⌃ ser 1 =  I , b ✓ m, 1 = 0 , b loc m, 

Revised: 1: Initialize ⌃ m, 1 = ⌃ ser 1 =  I , b ✓ m, 1 = 0 , u loc m,

Original: = 0 and ⌃ loc m, 0 = 0 for all m 2 [ M ] 2: for round t = 1 , 2 , . .
Revised: = 0 and ⌃ loc m, 0 = 0 for all m 2 [ M ] 2: for round t = 1 , 2 , . .

Original: , T do 3:
Revised: , T do 3:

Original: Agent m t is active 4:
Revised: Agent m t is active 4:

Original: Receive D t from the environment 5:
Revised: Receive D t from the environment 5:

Original: Select x t argmax x 2 D t h b ✓ m t ,t , x i +  k x k ⌃  1 mt,t /* Optimistic decision */ 6:
Revised: Select x t argmax x 2 D t h b ✓ m t ,t , x i +  k x k ⌃  1 mt,t /* Optimistic decision */ 6:

Original: Receive r t from environment 7: ⌃ loc m t ,t ⌃ loc m t ,t  1 + x t x > t , b loc m t ,t b loc m t ,t  1 + r t x t /* Local update */ 8: if det( ⌃ m t ,t + ⌃ loc m t ,t ) > (1 + ↵ ) det( ⌃ m t ,t ) then 9:
Revised: Receive r t from environment 7: ⌃ loc m t ,t ⌃ loc m t ,t  1 + x t x > t , u loc m t ,t u loc m t ,t  1 + r t x t /* Local update */ 8: if det( ⌃ m t ,t + ⌃ loc m t ,t ) > (1 + ↵ ) det( ⌃ m t ,t ) then 9:

Original: Agent m t sends ⌃ loc m t ,t and b loc m t ,t to server /* Upload */ 10: ⌃ ser t ⌃ ser t + ⌃ loc m t ,t , b ser t b ser t + b loc m t ,t /* Global update */ 11: ⌃ loc m t ,t 0 , b loc m t ,t 12:
Revised: Agent m t sends ⌃ loc m t ,t and u loc m t ,t to server /* Upload */ 10: ⌃ ser t ⌃ ser t + ⌃ loc m t ,t , u ser t u ser t + u loc m t ,t /* Global update */ 11: ⌃ loc m t ,t 0 , u loc m t ,t 12:

Original: Server sends ⌃ ser t and b ser t back to agent m t /* Download */ 13: ⌃ m t ,t + ⌃ ser t , b m t ,t + b ser t 14: b ✓ m t ,t + ⌃  1 m t ,t +1 b m t ,t + /* Compute estimate */ 15: else 16: ⌃ m t ,t + ⌃ m,t , b m t ,t + b m,t , b ✓ m t ,t + b ✓ m t ,t 17: end if 18: for other inactive agent m 2 [ M ] \ { m t } do 19: ⌃ m,t + ⌃ m,t , b m,t + b m,t , b ✓ m,t + b ✓ m,t 20: end for 21: end for
Revised: Server sends ⌃ ser t and u ser t back to agent m t /* Download */ 13: ⌃ m t ,t + ⌃ ser t , u m t ,t + u ser t 14: b ✓ m,t + ⌃  1 m t ,t +1 u m t ,t + /* Compute estimate */ 15: else 16: ⌃ m t ,t + ⌃ m,t , u m t ,t + u m,t , b ✓ m t ,t + b ✓ m t ,t 17: end if 18: for other inactive agent m 2 [ M ] \ { m t } do 19: ⌃ m,t + ⌃ m,t , u m,t + u m,t , b ✓ m,t + b ✓ m,t 20: end for 21: end for

Original: Notation
Revised: Notation

Original: Meaning b ✓ m,t estimate of ✓ ⇤ ⌃ m,t , b m,t data used to compute b ✓ m,t ⌃ loc m,t , b loc m,t local data for agent m ⌃ ser m,t , b ser m,t data stored at the server
Revised: Meaning b ✓ m,t estimate of ✓ ⇤ ⌃ m,t , u m,t data used to compute b ✓ m,t ⌃ loc m,t , u loc m,t local data for agent m ⌃ ser m,t , u ser m,t data stored at the server

Original: The key component of the algorithm is the matrix determinant-based criterion (Line 8), whichevaluates the information accumulated in currentlocal data.
Revised: The key component of the algorithm is the matrix determinant-based criterion (Line 8), whichevaluates the information accumulated in currentlocal data.

Original: If the criterion is satisﬁed, it suggeststhat the local data would help signiﬁcantly reduce the uncertainty of estimating the model ✓ ⇤ .
Revised: If the criterion is satisﬁed, it suggeststhat the local data would help signiﬁcantly reduce the uncertainty of estimating the model ✓ ⇤ .

Original: Therefore, agent m t will share its progress byuploading the local data to the server (Line 9) sothat it can beneﬁt other agents.
Revised: Therefore, agent m t will share its progress byuploading the local data to the server (Line 9) sothat it can beneﬁt other agents.

Original: Then the serverupdates the global data accordingly (Line 10).
Revised: Then the serverupdates the global data accordingly (Line 10).

Original: Afterwards, agent m t downloads the latest global data from the server (Line 12) and updates itslocal data and model (Line 13-14).
Revised: Afterwards, agent m t downloads the latest global data from the server (Line 12) and updates itslocal data and model (Line 13-14).

Original: If the criterion in Line 8 is not met, then the communicationbetween the agent and the server will not be triggered, and the local data remains unchanged foragent m t (Line 16).
Revised: If the criterion in Line 8 is not met, then the communicationbetween the agent and the server will not be triggered, and the local data remains unchanged foragent m t (Line 16).

Original: Finally, all the other inactive agents remain unchanged (Line 19).
Revised: Finally, all the other inactive agents remain unchanged (Line 19).

Original: Note that in Algorithm 1, the communication between the agent and the server (Line 9 and 12)involves only the active agent in that round, which is completely independent of other agents.
Revised: Note that in Algorithm 1, the communication between the agent and the server (Line 9 and 12)involves only the active agent in that round, which is completely independent of other agents.

Original: This isin sharp contrast to existing algorithms.
Revised: This isin sharp contrast to existing algorithms.

Original: For example, in the main algorithm in Li and Wang (2022a),upload by any agent may trigger other agents to download the latest data, while our algorithm doesnot mandate this.
Revised: For example, in the main algorithm in Li and Wang (2021),upload by any agent may trigger other agents to download the latest data, while our algorithm doesnot mandate this.

Original: On the other hand, many existing algorithms for multi-agent settings (e.g., Wanget al.
Revised: On the other hand, many existing algorithms for multi-agent settings (e.g., Wanget al.

Original: (2019)) require all agents to interact with the environment in each round, which essentiallyrequire full participation of all the agents.
Revised: (2019)) require all agents to interact with the environment in each round, which essentiallyrequire full participation of all the agents.

Original: The determinant-based criterion in Line 8 has been a long-standing design trick in contextual linearbandits that can help address the issue of unknown time horizon and reduce the switching cost(Abbasi-Yadkori et al., 2011; Ruan et al., 2021).
Revised: The determinant-based criterion in Line 8 has been a long-standing design trick in contextual linearbandits that can help address the issue of unknown time horizon and reduce the switching cost(Abbasi-Yadkori et al., 2011; Ruan et al., 2021).

Original: For multi-agent bandits, such a criterion has alsobeen used to control the communications complexity (Wang et al., 2019; Dubey and Pentland, 2020;
Revised: For multi-agent bandits, such a criterion has alsobeen used to control the communications complexity (Wang et al., 2019; Dubey and Pentland, 2020;

Original: Li and Wang, 2022a).
Revised: Li and Wang, 2021).

Original: This is because the need for policy switching or communication essentiallyreﬂects the same fact: enough information has been collected and it is time to update the (local)model.
Revised: This is because the need for policy switching or communication essentiallyreﬂects the same fact: enough information has been collected and it is time to update the (local)model.

Original: Therefore, achieving low communication complexity and low switching cost are uniﬁedin our FedlinUCB algorithm in the sense that the communication complexity is exactly twice theswitching cost.
Revised: Therefore, achieving low communication complexity and low switching cost are uniﬁedin our FedlinUCB algorithm in the sense that the communication complexity is exactly twice theswitching cost.

Original: Furthermore, using lazy update makes our algorithm amenable for analysis, whichwill be clear later in Section 6.
Revised: Furthermore, using lazy update makes our algorithm amenable for analysis, whichwill be clear later in Section 6.

Original: In addition, we leave ↵ > 0 as a tuning parameter as it controls thetrade-off between the regret and the communication complexity.
Revised: In addition, we leave ↵ > 0 as a tuning parameter as it controls thetrade-off between the regret and the communication complexity.

Original:  Theoretical Results
Revised:  Theoretical Results

Original: We now present our main result on the theoretical guarantee of Algorithm 1.
Revised: We now present our main result on the theoretical guarantee of Algorithm 1.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: Under Assumption 3.1 for Algorithm 1, if we set the conﬁdence radius  = p  S +( p 1 +M ↵ + M p 2 ↵ )
Revised: Under Assumption 3.1 for Algorithm 1, if we set the conﬁdence radius  = p  S +( p 1 +

Original: [Equation]
Revised: [Equation]

Original: [Equation] 
Revised: +

Original: [Equation]
Revised: [Equation]

Original:  /  ⌘
Revised: ⇥

Original: [Equation]
Revised: [Equation]

Original: ⌘ , then with probability atleast 1   , the regret in the ﬁrst T rounds can be upper bounded by
Revised: , then with probabilityat least 1   , the regret in the ﬁrst T rounds can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: Moreover, the communication complexity and switching cost are both bounded by 2 log 2 · d ( M +1 / ↵ ) log(1 + TL 2 / (  d )) .
Revised: Moreover, the communication complexity and switching cost are both bounded by 2 log 2 · d ( M +1 / ↵ ) log(1 + TL 2 / (  d )) .

Original: Remark 5.2.
Revised: Remark 5.2.

Original: Theorem5.1 suggests that if we set the parameters ↵ = 1 /M 2 and  = 1 /S 2in Algorithm 1, then its regret is bounded by e
Revised: Theorem suggests that if we set the parameters ↵ = 1 /M 2 and  = 1 /S 2 in Algorithm 1, then its regret can be bounded by e

Original: O ( Rd p T ) and the corresponding communicationcomplexity and switching cost are both bounded by e
Revised: O ( Rd p T ) and the corresponding communicationcomplexity and switching cost are both bounded by e

Original: O ( dM 2 ) .
Revised: O ( dM 2 ) .

Original: This choice of parameters yields theregret bound and the communication complexity presented in Table 1.
Revised: This choice of parameters yields theregret bound and the communication complexity presented in Table 1.

Original: As a complement, we also provide a lower bound for the communication complexity as stated in thefollowing theorem. See Appendix C for the proof.
Revised: As a complement, we also provide a lower bound for the communication complexity as stated in thefollowing theorem (see Appendix C for the proof).

Original: Theorem 5.3.
Revised: Theorem 5.3.

Original: For any algorithm Alg with expected communication complexity less than
Revised: For any algorithm Alg with expected communication complexity less than

Original: O ( M/ log( T/M )) , there exist a linear bandit instance with R =
Revised: O ( M/ log( T/M )) , there exist a linear bandit instance with R =

Original: L = S = 1 such that for T  Md ,the expected regret for algorithm Alg is at least ⌦ ( d p MT ) .
Revised: L = S = 1 such that for T  Md ,the expected regret for algorithm Alg is at least ⌦ ( d p MT ) .

Original: Remark 5.4.
Revised: Remark 5.4.

Original: Suppose each agent runs the OFUL algorithm (Abbasi-Yadkori et al., 2011) separately,then each agent m 

Revised: Suppose each agent runs the OFUL algorithm (Abbasi-Yadkori et al., 2011) separately,then each agent m

Original: [ M ] admits an e
Revised: [ M ] has an e

Original: O ( d p T m ) regret, where T m is the number of rounds thatagent m participates in.
Revised: O ( d p T m ) regret, where T m is the number of rounds that agent mparticipates.

Original: Thus the total regret of M agents is upper bounded by
Revised: Then the total regret of M agents is upper bounded by

Original: [Equation]
Revised: [Equation]

Original: e
Revised: e

Original: O ( p T m ) = e
Revised: O ( p T m )  e

Original: O ( d p MT ) .
Revised: O ( d p MT ) .

Original: Theorem 5.3 implies that for any algorithm Alg , if its communication complexity is lessthan O ( M/ log( T/M )) , then its regret cannot be better than naively running M independent OFULalgorithms.
Revised: Theorem 5.4 says that for any algorithm Alg , if its communication complexity is less than O ( M/ log( T/M )) , then its regret cannot be better than naively running M independent OFULalgorithms.

Original: In other words, Theorem5.3 suggests that in order to perform better than the single-agentalgorithm through collaboration, an ⌦ ( M ) communication complexity is necessary.
Revised: In other words, Theorem suggests that in order to perform better than the single-agentalgorithm through collaboration, an ⌦ ( M ) communication complexity is necessary.

Original:  Overview of the Proof
Revised:  Overview of the Proof

Original: When analyzing the performance of
Revised: When analyzing the performance of

Original: FedLinUCB , we face a unique challengecaused by the asynchronous communication,as illustrated in Figure 1.
Revised: FedLinUCB , we face a unique challengecaused by the asynchronous communication, which is illustrated in Figure 1.

Original: Here ( x m,t , ⌘ m,t )denotes the decision and the noise for agentm in its own t -th round.
Revised: Here( x m,t , ⌘ m,t ) denotes the decision and the noisefor agent m in its own t -th round.

Original: Speciﬁcally, in thesynchronous setting, the ﬁltration is generatedby all the data collected by all agents, i.e., F 5 =  { x m,t , ⌘ m,t } 5, 5 t =1,m =1 , as marked by thegreen dashed rectangle.
Revised: Speciﬁcally,if in a synchronous setting, the ﬁltration isgenerated by all the data collected by all agents,i.e., F 5 =  { x m,t , ⌘ m,t } 5 t =1 , as displayed bythose in the green dashed rectangle.

Original: This kind of ﬁltrationis well-deﬁned since all agents share their datawith each other at the end of each round.
Revised: This kindof ﬁltration is well-deﬁned since all agentsshare their data with each other at the end ofeach round.

Original: Insharp contrast,  in our asynchronous setting, the data at the server can be generated by an irregularset of data from the agents, as marked by  the blue rectangles.
Revised: In sharp contrast, if in our asynchronous setting, the data at the server can be generatedby a irregular set of data from the agents, as illustrated by those in the blue rectangles in Figure1.

Original: Such data pattern can be arbitraryand depends on the data collected in all previous rounds, which prevents us from deﬁning a ﬁxedﬁltration as wecan do in the synchronous setting.
Revised: Such data pattern can be arbitrary and depends on the data collected in all previous rounds, whichprevents us from deﬁning a ﬁxed ﬁltration as we do in the synchronous setting.

Original: Since the application of standard martingaleconcentration inequalities relies on the well-deﬁned ﬁltration,  they cannot be directly applied to ourasynchronous setting.
Revised: Yet applications ofcommon martingale concentration inequalities rely on such a well-deﬁned ﬁltration, so they cannotbe directly used in our asynchronous setting.

Original: To circumvent the above issue, we need to analyze the concentration property of the local data foreach agent and then relate it to the concentrationof the global data, so that we can control the sum ofthe bonuses and hence the regret.
Revised: To circumvent the above issue, we need to analyze the concentration property of the local data foreach agent and then relating it to the concentration global data, so that we can control the bonussum and hence the regret.

Original: This requires a careful quantitative comparison of the local andglobal data covariance matrices, which is enabled by our design of determinant-based criterion.
Revised: This requires a careful quantitative comparison of the local and globalcovariance matrices, which is enabled by our design of determinant-based criterion.

Original: Thedetails will be further explained in Section 6.2.
Revised: The details willbe further explained in Section 6.2.

Original: Next, we present the key ingredients in the proof of Theorem 5.1.
Revised: Next, we present the key ingredients in the proof of Theorem 5.2.

Original: Remark 6.1.
Revised: Remark 6.1.

Original: Recall the notations in Table 2, where the values of those matrices and vectors mightchange within each round.
Revised: Recall the notations in Table 2, where the values of those matrices and vectors mightchange during each round.

Original: To eliminate the possible confusion, from now on we follow the conventionthat all matrices and vectors in the analysis correspond to their values at the end of each round in
Revised: To eliminate the possible confusion, from now on we follow the conventionthat all matrices and vectors in the analysis correspond to their values at the end of each round in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: 6.1 Analysis for communication complexity and switching cost
Revised: 6.1 Analysis for communication complexity and switching cost

Original: We ﬁrst analyze the communication complexity and switching cost of Algorithm 1.
Revised: We ﬁrst analyze the communication complexity and switching cost of Algorithm 1.

Original: For each i  0 ,we deﬁne
Revised: For each i  0 ,we deﬁne

Original: [Equation]
Revised: [Equation]

Original: We divide the set of all rounds into epochs { ⌧ i , ⌧ i + 1 , . . .
Revised: We divide the set of all rounds into epochs { ⌧ i , ⌧ i +1 , . . .

Original: , ⌧ i +1  1 } for each i  0 .
Revised: , ⌧ i +1  1 } for each i  0 .

Original: Then thecommunication complexity within each epoch can be bounded using the following lemma.
Revised: Then the 249 communication complexity within each epoch can be bounded using the following lemma.

Original: Lemma 6.2.
Revised: Lemma 6.2.

Original: Under the setting of Theorem 5.1, for each epoch from round ⌧ i to round ⌧ i +1  1 , thenumber of communications in this epoch is upper bounded by 2( M + 1 / ↵ ) .
Revised: Under the setting of Theorem 5.2, for each epoch from round ⌧ i to round ⌧ i +1  1 , thenumber of communications in this epoch is upper bounded by 2( M + 1 / ↵ ) .

Original: Proof of Theorem 5.1: communication complexity and switching cost.
Revised: Proof of Theorem 5.2: communication complexity and switching cost.

Original: It sufﬁces to bound the number of epochs.
Revised: It sufﬁces to bound the number of epochs.

Original: By Assumption 3.1, we have k x t k 2  L for all t 2 [ T ] .
Revised: By Assumption 3.1, we have k x t k 2  L for all t 2 [ T ] .

Original: Since ⌃ ser T is positive deﬁnite,bythe inequality of arithmetic and geometric means, we have
Revised: Since ⌃ ser T is positive deﬁnite,by inequality of arithmetic and geometric means, we have

Original: [Equation]
Revised: [Equation]

Original: Then recalling the deﬁnition of epochs based on (6.1), we have
Revised: Then recalling the deﬁnition of epochs based on (6.1), we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, the total number of epochs is bounded by log 2 · d log(1 + TL 2 / (  d )) .
Revised: Therefore, the total number of epochs is bounded by log 2 · d log(1 + TL 2 / (  d )) .

Original: Now applying
Revised: Now applying

Original: Lemma 6.2, the total communication complexity is bounded by 2 log 2 · d ( M + 1 / ↵ ) log(1 +
Revised: Lemma 6.2, the total communication complexity is bounded by 2 log 2 · d ( M + 1 / ↵ ) log(1 +

Original: TL 2 / (  d )) .
Revised: TL 2 / (  d )) .

Original: Note that in Algorithm 1, each agent only switch its policy after communicating with theserver, so the switching cost is exactly equal to half of the communication complexity.
Revised: Note that in Algorithm 1, each agent only switch its policy after they have communicatedwith the server, so the switching cost is exactly equal to half of the communication complexity.

Original: This ﬁnishesthe proof for the claim on communication complexity and switching cost in Theorem 5.1.
Revised: Thisﬁnishes the proof for the claim on communication complexity and switching cost in Theorem 5.2.

Original: 6.2 Analysis for regret upper bound
Revised: 6.2 Analysis for regret upper bound

Original: The regret analysis for Theorem 5.1 is much more involved, and it relies on a series of intermediatelemmas establishing the concentration.
Revised: The regret analysis for Theorem 5.2 is much more involved, and it relies on a sequence of intermediatelemmas characterizing the concentration properties.

Original: Total information.
Revised: Total information.

Original: We deﬁne the following auxiliary matrices and vectors that contain all theinformation up to round t :
Revised: We deﬁne the following auxiliary matrices and vectors that contain all theinformation up to round t :

Original: [Equation]
Revised: [Equation]

Original: where ⌘ i :=
Revised: where ⌘ t :=

Original: r i  h x i , ✓ ⇤ i is a R -sub-Gaussian noise by Assumption 3.1.
Revised: r t  h x t , ✓ ⇤ i is a R-sub-Gaussian noise by Assumption 3.1.

Original: In our setting,  ⌃ all t , b all t , u all tare not accessible by the agents due to asynchronous communication, and we only use them tofacilitate the analysis.
Revised: In our setting, all of the 267 ⌃ all t , u all t , b all t are not accessible to the agents due to asynchronous communication, and we only usethem to facilitate the analysis.

Original: With this notation, we can further deﬁne thefollowing omnipotent estimate:
Revised: With this notation, we can further deﬁne the omnipotent estimate:

Original: [Equation]
Revised: [Equation]

Original: As a direct application of the self-normalized martingale concentration inequality (Abbasi-Yadkoriet al., 2011), we have thefollowing global conﬁdence bound due to the  concentration of ⌃ all t and b all t .
Revised: As a direct application of the self-normalized martingale concentration inequality (Abbasi-Yadkoriet al., 2011), we have the global conﬁdence bound due to the global concentration of ⌃ all t and u all t .

Original: Lemma 6.3 (Global conﬁdence bound; Theorem 2, Abbasi-Yadkori et al. 2011) .
Revised: Lemma 6.3 (Global conﬁdence bound; Theorem 2, Abbasi-Yadkori et al. 2011) .

Original: With probability atleast 1   , for each round t 2 [ T ] , the estimate [Equation]
Revised: With probability atleast 1   , for each round t 2 [ T ] , the estimate b ✓ all t satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: ⌃ all t  R q d log  (1 + TL 2 / ) /
Revised: all t  R q d log  [Equation] [Equation]

Original:   + p S.
Revised: [Equation] p

Revised: [Equation]

Original: Per-agent information.
Revised: Per-agent information.

Original: Next, for each agent m 

Revised: Next, for each agent m

Original: [ M ] , we denote the rounds when agent mcommunicate with the server (i.e., upload and download data) as { t m, 1 , t m, 2 , ... } .
Revised: [ M ] , we denote the rounds when agent mcommunicate with the server (i.e., upload and download data) as { t m, 1 , t m, 2 , ... } .

Original: For simplicity, atthe end of round t , we denote by N m ( t ) the last round when agent m communicated with the server(so if agent m communicated with the server in round t , then N m ( t )
Revised: For simplicity, atthe end of round t , we denote by N m ( t ) the last round when agent m communicated with the server(so if agent m communicated with the server in round t , then N m ( t )

Original: = t ).
Revised: = t ).

Original: With this notation, foreach round t and agent m 2 [ M ] , the data that has been uploaded by agent m is then 

Revised: With this notation, foreach round t and agent m 2 [ M ] , the data that has been uploaded by agent m is then

Original: [Equation]
Revised: [Equation]

Original: Correspondingly, the local data that has not been uploaded to the server is
Revised: Correspondingly, the local data that has not been uploaded to the server is

Original: [Equation]
Revised: [Equation]

Original: Again, applying the self-normalized martingale concentration (Abbasi-Yadkori et al., 2011) togetherwith a union bound, we can get the per-agent local concentration.
Revised: Again, applying the self-normalized martingale concentration Abbasi-Yadkori et al. (2011) with aunion bound, we can get the per-agent local concentration.

Original: Lemma 6.4 (Local concentration) .
Revised: Lemma 6.4 (Local concentration) .

Original: Under the setting of Theorem 5.1, with probability at least 1   ,for each round t 2 [ T ] and each agent m 

Revised: Under the setting of Theorem 5.2, with probability at least 1   ,for each round t 2 [ T ] and each agent m

Original: [ M ] , it holds that
Revised: [ M ] , it holds that

Original: [Equation]
Revised: [Equation]

Original: Moreover, based on our determinant-based communication criterion, we have the following lemmathat describes the quantitative relationship among the local data, uploaded data and global data.
Revised: Moreover, based on our determinant-based communication criterion, we have the following lemmathat describes the quantitative relationship among the local data, uploaded data and global data.

Original: Lemma 6.5 (Covariance comparison) .
Revised: Lemma 6.5 (Covariance comparison) .

Original: Under the setting of Theorem 5.1, it holds that [Equation] for each agent m 
 
Revised: Under the setting of Theorem 5.2, for each agent m 
 [ M ] , itholds that  I +

Original: [ M ] .
Revised: P

Original: Moreover, for any 1  t 1 < t 2  T , if agent m is the only active agentfrom round t 1 to t 2  1 and agent m only communicates with the server at round t 1 , then for allt 1 + 1   t  t 2  ,  itfurther holds that [Equation]
Revised: Mm 0 =1 ⌃ up m 0 ,t ⌫ 1 ↵ ⌃ loc m,t . Moreover, if agent m is the only active agent from round t 1to t 2  1 and agent m only communicates with the server at round t 1 , then for all t 1 +1  t  t 2  1 , 288 it holds that ⌃ m,t ⌫ 1 1+ M ↵ ⌃ all t .

Original: Combining the above results, we obtain the local conﬁdence bound which then leads to the per-roundregret in each round, as summarized in the following lemma.
Revised: Combining the above results, we further have the local conﬁdence bound which then leads to theone-step regret in each round. These are summarized in the following lemma.

Original: Lemma 6.6 (Local conﬁdence bound and per-round regret) .
Revised: Lemma 6.6 (Local conﬁdence bound and one-step regret) .

Original: Under the setting of Theorem 5.1, withprobability at least 1   , for each t 2 [ T ] , the estimate 
Revised: Under the setting of Theorem 5.2, withprobability at least 1   , for each t 2 [ T ] , the estimate b ✓ m,t +1 satisﬁes that k ✓ ⇤

Original: [Equation]
Revised: [Equation]

Original: 300 for all m 2 [ M ] . Consequently, for each round t 2 [ T ] , the regret in round t satisﬁes
Revised: Consequently, for each round t 2 [ T ] , the regret in round t is upper bounded by

Original: [Equation]
Revised: [Equation]

Original: Now, we are ready to prove the regret bound in Theorem 5.1.
Revised: Now, we are ready to prove the regret bound in Theorem 5.2.

Original: Proof of Theorem 5.1: regret.
Revised: Proof of Theorem 5.2: regret.

Original: Firstly, according to Lemma 6.6, the regret in ﬁrst T round can bedecomposed and upper bounded by
Revised: Firstly, according to Lemma 6.6, the regret in ﬁrst T round can bedecomposed and upper bounded by

Original: [Equation]
Revised: [Equation]

Original: Now, we only need to control the summation of the upper conﬁdence bonus term 2  k x t k ⌃  1 mt,t andwe focus on the agent-action sequence { ( m t , x t ) } Tt =1 .
Revised: Now, we only need to control the summation of the upper conﬁdence bonus term 2  k x t k ⌃  1 mt,t andwe focus on the agent-action sequence { ( m t , x t ) } Tt =1 .

Original: Notice that if an agent m communicate with the server at time t 1 and t 2 , then the order of actions between round t 1 and t 2 will not effect thecovariance matrix for agent m .
Revised: Notice that if an agent m communicate withthe server at time t 1 and t 2 , then the order of actions between round t 1 and t 2 will not effect thecovariance matrix for agent m .

Original: In addition, since agent m does not upload new data between round t 1and t 2 , the order of actions from agent m will also not affect other agents’ covariance matrix.
Revised: In addition, since agent m does not upload new data between round t 1and t 2 , the order of actions from agent m will also not affect other agents’ covariance matrix.

Original: Thus,without effect the covariance matrix and the corresponding upper conﬁdence bonus, we can alwaysreorder the sequence of active agents such that each agent communicates with the server and staysactive until the next agent kicks in to communicate with the server.
Revised: Thus,without effect the covariance matrix and the corresponding upper conﬁdence bonus, we can alwaysreorder the sequence of active agents such that each agent communicates with the server and staysactive until the next agent kicks in to communicate with the server.

Original: Such reordering is valid accordingto the communication protocol as each agent has only local updates between communications withthe server.
Revised: Such reordering is valid according to the communication protocol as each agent has only local updates between communications withthe server.

Original: More speciﬁcally, we assume that the sequence of round that the active agent communicates withserver is
Revised: More speciﬁcally, we assume that the sequence of round that the active agent communicates withserver is

Original: 0 = t 0 < t 1 < t 2 < · · · < t N = T + 1 4 , and from round t
Revised: 0 = t 0 < t 1 < t 2 < · · · < t N = T + 1 4 , and from round t

Original: i + 1 to t i +1  1 there is onlyone agent active, that is, m t i
Revised: i + 1 to t i +1  1 there is onlyone agent active, that is, m t i

Original: = m
Revised: = m

Original: t i +1 = · · · = m t
Revised: t i +1 = · · · = m t

Original: i +1  1 .
Revised: i +1  1 .

Original: Then we apply Lemma 6.5 and get that the 317 bonus term for each round ti < t
Revised: Then we apply Lemma 6.5 and get that the 310 bonus term for each round t

Original:  < t i +1 can be controlled by
Revised: i < t < t i +1 can be controlled by

Original: [Equation]
Revised: [Equation]

Original: In addition, to control the bonus term for rounds { t i } Ni =1 , we deﬁne T i = min { t 2 [ T ] | det( ⌃ all t ) 2 i  d } .
Revised: In addition, to control the bonus term for rounds { t i } Ni =1 , we deﬁne [Equation]

Original: For each time interval from T i to T i +1 , if an agent m communicate with the server more thanonce, e.g., agent m communicates with the server at round T i, 1 and T i, 2 such that T i  T i, 1 < T i,2 < 321 T
Revised: For each time interval from T i to T i +1 , if an agent m communicate with the server more than once,e.g., agent m communicates with the server at round T i, 1 and T i, 2 such that T i  T i, 1 < T i,

Original:  i +1 , then for the latter round T i, 2 , the bonus term can be controlled by
Revised: 2 < T i +1 ,then for the latter round T i, 2 , the bonus term can be controlled by

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality holds due Lemma 6.5 and agent m communicate with the serverat the previous round T i, 1 , the second inequality holds due to Lemma D.4 with the fact thatdet ⌃ all T
Revised: where the ﬁrst inequality holds due Lemma 6.5 and agent m communicate with the serverat the previous round T i, 1 , the second inequality holds due to Lemma D.4 with the fact that

Original: i +1  1 / det( ⌃ all T i, 1 )
Revised: [Equation]

Original:   2 i +1  d / (2 i  d ) = 2 , and the last inequality holds due to the factthat ⌃ all T i +1  1 ⌫ ⌃ all T i, 2 .
Revised: )  2 i +1  d / (2 i  d ) = 2 , and the last inequality holds due to the factthat ⌃ all T i +1  1 ⌫ ⌃ all T i, 2 .

Original: On the other hand, for each time interval from T i to T i +1 , the bonus termfor the ﬁrst communication can always be trivially bounded by 1 for all agent m .
Revised: On the other hand, for each time interval from T i to T i +1 , the bonus termfor the ﬁrst communication can always be trivially bounded by 1 for all agent m .

Original: Therefore, thesummation of the regret over ﬁrst communication can be upper bounded by 1 .
Revised: Therefore, thesummation of the regret over ﬁrst communication can be upper bounded by 1 .

Original: In addition, since thenorm of each action k x k 2  L and it implies that we have
Revised: In addition, since thenorm of each action k x k 2  L and it implies that we have

Original: [Equation]
Revised: [Equation]

Original: which implies that the number of different intervals is at most d log(1 + TL 2 /  ) .
Revised: which implies that the number of different intervals is at most d log(1 + TL 2 /  ) .

Original: Combining theupper bound of regret in (6.6), (6.7) and (6.8), we have
Revised: Combining theupper bound of regret in (6.4), (6.5) and (6.6), we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from a standard elliptic potential argument (Abbasi-Yadkori et al.,2011).
Revised: where the last inequality follows from a standard elliptic potential argument (Abbasi-Yadkori et al.,2011).

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: In this work, we study federated contextual linear bandit problem with fully asynchronous communication.
Revised: In this work, we study federated contextual linear bandit problem with fully asynchronous communication.

Original: We propose a simple and provably efﬁcient algorithm named FedLinUCB .
Revised: We propose a simple and provably efﬁcient algorithm named FedLinUCB .

Original: We provethat FedLinUCB obtains a near-optimal regret of order e
Revised: We provethat FedLinUCB obtains a near-optimal regret of order e

Original: O ( d p T ) with e
Revised: O ( d p T ) with e

Original: O ( dM 2 ) communicationcomplexity.
Revised: O ( dM 2 ) communication complexity.

Original: We also prove a lower bound on the communication complexity, which suggests thatan ⌦ ( M ) communication complexity is necessary for achieving a near-optimal regret.
Revised: We also prove a lower bound on the communication complexity, which suggests that an⌦ ( M ) communication complexity is necessary for achieving a near-optimal regret.

Original: There stillexists an O ( dM ) gap between the upper and lower bounds for the communication complexity andwe leave it as a future work to close this gap.
Revised: There still existsan O ( dM ) gap between the upper and lower bounds for the communication complexity and weleave it as a future work to close this gap.

Original: Another important direction for future work is tostudy federated linear bandits with a decentralized communication network without a central server(i.e., P2P networks).
Revised: Another important direction for future work is to studyfederated linear bandits with a decentralized communication network without a central server (i.e., P2P networks).

Original: In addition, there are potential privacy concerns when the agents upload anddownload data from the server, and it remains an open problem to devise provably efﬁcient algorithmfor asynchronous federated linear bandits with privacy guarantees.