Original: A Simple and Provably Efficient Algorithm for Asynchronous Federated Contextual Linear Bandits
Revised: A Simple and Provably Efﬁcient Algorithm for Asynchronous Federated Contextual Linear Bandits

Original: Abstract
Revised: Abstract

Original: We study federated contextual linear bandits, where M agents cooperate with each other to solve a global contextual linear bandit problem with the help of a central server.
Revised: We study federated contextual linear bandits, where M agents cooperate with each other to solve a global contextual linear bandit problem with the help of a central server.

Original: We consider the asynchronous setting, where all agents work independently and the communication between one agent and the server will not trigger other agents’ communication.
Revised: We consider the asynchronous setting, where all agents work independently and the communication between one agent and the server will not trigger other agents’ communication.

Original: We propose a simple algorithm named FedLinUCB based on the principle of optimism.
Revised: We propose a simple algorithm named FedLinUCB based on the principle of optimism.

Original: We prove that the regret of FedLinUCB is bounded by (cid:101)
Revised: We prove that the regret of FedLinUCB is bounded by e

Original: O ( d (cid:113)(cid:80)
Revised: O ( d qP

Original: Mm =1 T m ) and the communication complexity is (cid:101) O ( dM 2 ) , where d is the dimension of the contextual vector and T m is the total number of interactions with the environment by m -th agent.
Revised: Mm =1 T m ) and the communication complexity is e O ( dM 2 ) , where d is the dimension of the contextual vector and T m is the total number of interactions with the environment by m -th agent.

Original: To the best of our knowledge, this is the first provably efficient algorithm that allows fully asynchronous communication for federated contextual linear bandits, while achieving the same regret guarantee as in the single-agent setting.
Revised: To the best of our knowledge, this is the ﬁrst provably efﬁcient algorithm that allows fully asynchronous communication for federated contextual linear bandits, while achieving the same regret guarantee as in the single-agent setting.

Original:  Introduction
Revised:  Introduction

Original: Contextual linear bandit is a canonical model in sequential decision making with partial information feedback that has found vast applications in real-world domains such as recommendation systems (Li et al., 2010a,b; Gentile et al., 2014; Li et al., 2020), clinical trials (Wang, 1991; Durand et al., 2018) and economics (Jagadeesan et al., 2021;
Revised: Contextual linear bandit is a canonical model in sequential decision making with partial information feedback that has found vast applications in real-world domains such as recommendation systems (Li et al., 2010a,b; Gentile et al., 2014; Li et al., 2020), clinical trials (Wang, 1991; Durand et al., 2018) and economics (Jagadeesan et al., 2021;

Original: Li et al., 2022).
Revised: Li et al., 2022).

Original: Most existing works on contextual linear bandits focus on either the single-agent setting (Auer, 2002;
Revised: Most existing works on contextual linear bandits focus on either the single-agent setting (Auer, 2002;

Original: Abe et al., 2003; Dani et al., 2008;
Revised: Abe et al., 2003; Dani et al., 2008;

Original: Li et al., 2010a; Rusmevichientong and Tsitsiklis, 2010;
Revised: Li et al., 2010a; Rusmevichientong and Tsitsiklis, 2010;

Original: Chu et al., 2011; Abbasi-Yadkori et al., 2011; Agrawal and Goyal, 2013) or multi-agent settings where communications between agents are instant and unrestricted (Cesa-Bianchi et al., 2013; Li et al., 2016; Wu et al., 2016;
Revised: Chu et al., 2011; Abbasi-Yadkori et al., 2011; Agrawal and Goyal, 2013) or multi-agent settings where communications between agents are instant and unrestricted (Cesa-Bianchi et al., 2013; Li et al., 2016; Wu et al., 2016;

Original: Li et al., 2021).
Revised: Li et al., 2021).

Original: Due to the increasing amount of data being distributed across a large number of local agents (e.g., clients, users, edge devices), federated learning (McMahan et al., 2017; Karimireddy et al., 2020) has become an emerging paradigm for distributed machine learning, where agents can jointly learn a global model without sharing their own localized data.
Revised: Due to the increasing amount of data being distributed across a large number of local agents (e.g., clients, users, edge devices), federated learning (McMahan et al., 2017; Karimireddy et al., 2020) has become an emerging paradigm for distributed machine learning, where agents can jointly learn a global model without sharing their own localized data.

Original: This motivates the development of distributed/federated linear bandits (Wang et al., 2019; Huang et al., 2021; Li and Wang, 2022a), which enables a collection of agents to cooperate with each other to solve a global linear bandit problem while enjoying performance guarantees comparable to those in the classical single-agent centralized setting.
Revised: This motivates the development of distributed/federated linear bandits (Wang et al., 2019; Huang et al., 2021; Li and Wang, 2022a), which enables a collection of agents to cooperate with each other to solve a global linear bandit problem while enjoying performance guarantees comparable to those in the classical single-agent centralized setting.

Original: However, most existing federated linear bandits algorithms are limited to the synchronous setting (Wang et al., 2019; Dubey and Pentland, 2020;
Revised: However, most existing federated linear bandits algorithms are limited to the synchronous setting (Wang et al., 2019; Dubey and Pentland, 2020;

Original: Huang et al., 2021), where all the agents have to first upload their local data to the server upon the request of the server, and the agents will download the latest data from the server after all uploads are complete.
Revised: Huang et al., 2021), where all the agents have to ﬁrst upload their local data to the server upon the request of the server, and the agents will download the latest data from the server after all uploads are complete.

Original: This requires full participation of the agents and global synchronization mandated by the server, which is impractical in many real-world application scenarios.
Revised: This requires full participation of the agents and global synchronization mandated by the server, which is impractical in many real-world application scenarios.

Original: The only notable exception is Li and Wang (2022a), where an asynchronous federated linear bandit algorithm is proposed.
Revised: The only notable exception is Li and Wang (2022a), where an asynchronous federated linear bandit algorithm is proposed.

Original: Nevertheless, in their algorithm, the upload by one agent may trigger the download from the server to all other agents.
Revised: Nevertheless, in their algorithm, the upload by one agent may trigger the download from the server to all other agents.

Original: Therefore, the communications between different agents and the server are not totally independent.
Revised: Therefore, the communications between different agents and the server are not totally independent.

Original: Moreover, they make a stringent regularity assumption on the contexts, which basically requires the contexts to be stochastic rather than adversarial as in standard contextual linear bandits.
Revised: Moreover, they make a stringent regularity assumption on the contexts, which basically requires the contexts to be stochastic rather than adversarial as in standard contextual linear bandits.

Original: That said, their theoretical proof is actually flawed as they ignored some unique challenges caused by asynchronous communication (see
Revised: That said, their theoretical proof is actually ﬂawed as they ignored some unique challenges caused by asynchronous communication (see

Original: Appendix A.1 for details).
Revised: Appendix A.1 for details).

Original: Therefore, how to design a truly asynchronous contextual linear bandit algorithm remains an open problem.
Revised: Therefore, how to design a truly asynchronous contextual linear bandit algorithm remains an open problem.

Original: In this work, we resolve the above open problem by proposing a simple algorithm for asynchronous federated contextual linear bandits over a star-shaped communication network.
Revised: In this work, we resolve the above open problem by proposing a simple algorithm for asynchronous federated contextual linear bandits over a star-shaped communication network.

Original: Our algorithm is based on the principle of optimism (Abbasi-Yadkori et al., 2011) and enjoys the following advantages:
Revised: Our algorithm is based on the principle of optimism (Abbasi-Yadkori et al., 2011) and enjoys the following advantages:

Original: (i)
Revised: (i)

Original: Each agent can decide whether or not to participate in each round.
Revised: Each agent can decide whether or not to participate in each round.

Original: Full participation is not required,
Revised: Full participation is not required

Original:  thus
Revised: , thus

Original: it allows temporarily offline agents.
Revised: it allows temporarily ofﬂine agents.

Original: This is much more flexible than existing algorithms for federated linear bandits in Wang et al.
Revised: This is much more ﬂexible than existing algorithms for federated linear bandits in Wang et al.

Original: ; Dubey and Pentland 
Revised: (2019); Dubey and Pentland (2020)

Original: ; Huang et al
Revised: ; Huang et al.

Original: ) where all agents are required to participate in each round; and (ii) the communication between each agent and the server is asynchronous and totally independent of other agents.
Revised: (2021) where all agents are required to participate in each round; and (ii) the communication between each agent and the server is asynchronous and totally independent of other agents.

Original: There is no need of global synchronization or mandatory coordination by the server, in contrast to Li and Wang (2022a) where each agent might be asked by the server to download data.
Revised: There is no need of global synchronization or mandatory coordination by the server, in contrast to Li and Wang (2022a) where each agent might be asked by the server to download data.

Original: In particular, the communication between the agent and the server is triggered by a matrix determinant-based criterion that can be computed independently by each agent.
Revised: In particular, the communication between the agent and the server is triggered by a matrix determinant-based criterion that can be computed independently by each agent.

Original: Our algorithm design not only allows the agents to independently operate and synchronize with the server, but also ensures low communication complexity (i.e., total number of rounds of communication between agents and the server) and low switching cost (i.e., total number of local model updates for all agents)
Revised: Our algorithm design not only allows the agents to independently operate and synchronize with the server, but also ensures low communication complexity (i.e., total number of rounds of communication between agents and the server) and low switching cost (i.e., total number of local model updates for all agents)

Original: (Abbasi-Yadkori et al., 2011).
Revised: (Abbasi-Yadkori et al., 2011).

Original: While being simple, our algorithm design introduces a challenge in the regret analysis.
Revised: While being simple, our algorithm design introduces a challenge in the regret analysis.

Original: Since the order of the interaction between the agent and the environment is not fixed, standard martingale-based concentration inequality cannot be directly applied.
Revised: Since the order of the interaction between the agent and the environment is not ﬁxed, standard martingale-based concentration inequality cannot be directly applied.

Original: Specifically, this challenge arises due to the mismatch between the partial data information collected by the central server and the true order of the data generated from the interaction with the environment, as is explained in detail in Section 5 and illustrated by Figure 1.
Revised: Speciﬁcally, this challenge arises due to the mismatch between the partial data information collected by the central server and the true order of the data generated from the interaction with the environment, as is explained in detail in Section 5 and illustrated by Figure 1.

Original: We address this challenge by a novel proof technique, which first establishes the local concentration of each agent’s data and then relates it to the “virtual” global concentration of all data via the determinant-based criterion.
Revised: We address this challenge by a novel proof technique, which ﬁrst establishes the local concentration of each agent’s data and then relates it to the “virtual” global concentration of all data via the determinant-based criterion.

Original: Based on this proof technique, we are able to obtain tight enough confidence bounds that lead to a nearly optimal regret.
Revised: Based on this proof technique, we are able to obtain tight enough conﬁdence bounds that lead to a nearly optimal regret.

Original: Moreover, our theoretical analysis relies only on minimal assumptions that are standard for contextual linear bandits, relaxing the strong assumptions made in Li and Wang (2022a).
Revised: Moreover, our theoretical analysis relies only on minimal assumptions that are standard for contextual linear bandits, relaxing the strong assumptions made in Li and Wang (2022a).

Original: Main contributions.
Revised: Main contributions.

Original: Our contributions are highlighted as follows:
Revised: Our contributions are highlighted as follows

Original:  • We devise a simple algorithm named FedLinUCB that achieves near-optimal regret, low communication complexity and low switching cost simultaneously for asynchronous federated contextual linear bandits.
Revised: : • We devise a simple algorithm named FedLinUCB that achieves near-optimal regret, low communication complexity and low switching cost simultaneously for asynchronous federated contextual linear bandits.

Original: In detail, we prove that our algorithms achieves a near-optimal(cid:101)
Revised: In detail, we prove that our algorithms achieves a near-optimal

Original:  O ( d √ T) regret with merely (cid:101) O ( dM 2 ) total communication complexity and (cid:101) O ( dM 2 ) total switching cost.
Revised: e O ( d p T regret with merely e O ( dM 2 ) total communication complexity and e O ( dM 2 ) total switching cost.

Original: Here M is the number of agents, d is the dimension of the context and T =
Revised: Here M is the number of agents, d is the dimension of the context and T =

Original: (cid:80) Mm =1 T m is the total number of rounds with T m being the number of rounds that agent m participates in.
Revised: P Mm =1 T m is the total number of rounds with T m being the number of rounds that agent m participates in.

Original: When degenerated to single-agent bandits, the regret of our algorithm matches the optimal regret(cid:101)
Revised: When degenerated to single-agent bandits, the regret of our algorithm matches the optimal regret

Original: O
Revised: e

Original:  ( d √ T ) (Abbasi-Yadkori et al., 2011).
Revised: O ( d p T ) (Abbasi-Yadkori et al., 2011)

Original: • We also prove an Ω( M/ log( T/M )) lower bound for the communication complexity.
Revised: • We also prove an ⌦ ( M/ log( T/M )) lower bound for the communication complexity.

Original: Together with the O ( dM 2 ) upper bound of our algorithm, it suggests that there is only an (cid:101) O ( dM ) gap between the upper and lower bounds of the communication complexity.
Revised: Together with the O ( dM 2 ) upper bound of our algorithm, it suggests that there is only an e O ( dM ) gap between the upper and lower bounds of the communication complexity.

Original: • We identify the issue of ill-defined filtration caused by the unfixed order of interactions between agents and the environment, which is absent in previous synchronous or single-agent settings.
Revised: • We identify the issue of ill-deﬁned ﬁltration caused by the unﬁxed order of interactions between agents and the environment, which is absent in previous synchronous or single-agent settings.

Original: We tackle this unique challenge by connecting the local concentration of each local agent’s data and the global concentration of the aggregated data from all agents.
Revised: We tackle this unique challenge by connecting the local concentration of each local agent’s data and the global concentration of the aggregated data from all agents.

Original: We believe this proof technique is of independent interest for the analysis of other asynchronous bandit problems.
Revised: We believe this proof technique is of independent interest for the analysis of other asynchronous bandit problems.

Original: Notation.
Revised: Notation.

Original: For any positive integer n , we denote the set { 1 , 2 , . . . , n } by [ n ] .
Revised: For any positive integer n , we denote the set { 1 , 2 , . . . , n } by [ n ] .

Original: We use I to denote the d × d identity matrix.
Revised: We use I to denote the d ⇥ d identity matrix.

Original: We use O to hide universal constants and (cid:101) O to further hide poly-logarithmic terms.
Revised: We use O to hide universal constants and e O to further hide poly-logarithmic terms.

Original: For any vector x ∈ R d and positive semi-definite Σ ∈ R d × d , we denote ∥ x ∥ Σ = √ x⊤ Σx .
Revised: For any vector x 2 R d and positive semi-deﬁnite ⌃ 2 R d ⇥ d , we denote k x k ⌃ = p x > ⌃ x .

Original:  Related Work
Revised:  Related Work

Original: We review related work on distributed/federated bandit algorithms stratified by the type of bandits: (1) multi-armed bandits, (2) stochastic linear bandits and (3) contextual linear bandits.
Revised: We review related work on distributed/federated bandit algorithms stratiﬁed by the type of bandits: (1) multi-armed bandits, (2) stochastic linear bandits and (3) contextual linear bandits.

Original: Distributed/federated multi-armed bandits.
Revised: Distributed/federated multi-armed bandits.

Original: There is a vast literature on distributed/federated multi-armed bandits (MABs) (Liu and Zhao, 2010; Szorenyi et al., 2013; Landgren et al., 2016;
Revised: There is a vast literature on distributed/federated multi-armed bandits (MABs) (Liu and Zhao, 2010; Szorenyi et al., 2013; Landgren et al., 2016;

Original: Chakraborty et al., 2017; Landgren et al., 2018; Martínez-Rubio et al., 2019; Sankararaman et al., 2019; Wang et al., 2019, 2020; Zhu et al., 2021), to mention a few.
Revised: Chakraborty et al., 2017; Landgren et al., 2018; Martínez-Rubio et al., 2019; Sankararaman et al., 2019; Wang et al., 2019, 2020; Zhu et al., 2021), to mention a few.

Original: However, none of these algorithms can be directly applied to linear bandits, needless to say contextual linear bandits with infinite decision sets.
Revised: However, none of these algorithms can be directly applied to linear bandits, needless to say contextual linear bandits with inﬁnite decision sets.

Original: Distributed/federated stochastic linear bandits.
Revised: Distributed/federated stochastic linear bandits.

Original: In distributed/federated stochastic linear bandits, the decision set is fixed across all the rounds t ∈ [ T ] and all the agents m ∈ [ M ] .
Revised: In distributed/federated stochastic linear bandits, the decision set is ﬁxed across all the rounds t 2 [ T ] and all the agents m 2 [ M ] .

Original: Wang et al.
Revised: Wang et al.

Original: (2019) proposed the DELD algorithm for distributed stochastic linear bandits on both star-shaped network and
Revised: (2019) proposed the DELD algorithm for distributed stochastic linear bandits on both star-shaped network and

Original: P2P network.
Revised: P2P network.

Original: Huang et al.
Revised: Huang et al.

Original: (2021) proposed an arm elimination-based algorithm called Fed-PE for federated stochastic linear bandits on the star-shaped network.
Revised: (2021) proposed an arm elimination-based algorithm called Fed-PE for federated stochastic linear bandits on the star-shaped network.

Original: Both algorithms are in the synchronous setting and require full participation of the agents upon the server’s request.
Revised: Both algorithms are in the synchronous setting and require full participation of the agents upon the server’s request.

Original: Distributed/federated contextual linear bandits.
Revised: Distributed/federated contextual linear bandits.

Original: The contextual linear bandit is more general and challenging than stochastic linear bandits, because the decision sets can vary for each t and m .
Revised: The contextual linear bandit is more general and challenging than stochastic linear bandits, because the decision sets can vary for each t and m .

Original: In this setting, Korda et al.
Revised: In this setting, Korda et al.

Original: (2016) considered a P2P network and proposed the DCB algorithm based on the OFUL algorithm in Abbasi-Yadkori et al.
Revised: (2016) considered a P2P network and proposed the DCB algorithm based on the OFUL algorithm in Abbasi-Yadkori et al.

Original: Wang et al.
Revised: Wang et al.

Original: (2019) considered both star-shaped and P2P communication networks and achieved the near-optimal (cid:101)
Revised: (2019) considered both star-shaped and P2P communication networks and achieved the near-optimal e

Original: O ( d √ T ) regret in the synchronous setting.
Revised: O ( d p T ) regret in the synchronous setting.

Original: 2 Dubey and Pentland (2020) further introduced the differential privacy guarantee into the setting of Wang et al.
Revised: 2 Dubey and Pentland (2020) further introduced the differential privacy guarantee into the setting of Wang et al.

Original: Li and Wang (2022b) extended distributed contextual linear bandits to generalized linear bandits (Filippi et al., 2010; Jun et al., 2017) in the synchronous setting.
Revised: Li and Wang (2022b) extended distributed contextual linear bandits to generalized linear bandits (Filippi et al., 2010; Jun et al., 2017) in the synchronous setting.

Original: Li and
Revised: Li and

Original: Wang (2022a) proposed the first asynchronous algorithm for federated contextual linear bandits with the star-shaped graph and achieved a near-optimal (cid:101)
Revised: Wang (2022a) proposed the ﬁrst asynchronous algorithm for federated contextual linear bandits with the star-shaped graph and achieved a near-optimal e

Original: O ( d √ T ) regret.
Revised: O ( d p T ) regret.

Original: However, their setting is different from ours in two aspects: (1) the upload triggered by an agent will lead the server to trigger download possibly for all the agents in their setting.
Revised: However, their setting is different from ours in two aspects: (1) the upload triggered by an agent will lead the server to trigger download possibly for all the agents in their setting.

Original: In contrast, the upload triggered by an agent will only lead to download to the same agent in our setting; (2) their regret guarantee relies on a stringent regularity assumption on the contexts, which basically requires the contexts to be stochastic.
Revised: In contrast, the upload triggered by an agent will only lead to download to the same agent in our setting; (2) their regret guarantee relies on a stringent regularity assumption on the contexts, which basically requires the contexts to be stochastic.

Original: As a comparison, the contexts in our setting can be even adversarial, which is exactly the standard setting of contextual linear bandits (Abbasi-Yadkori et al., 2011; Li et al., 2019).
Revised: As a comparison, the contexts in our setting can be even adversarial, which is exactly the standard setting of contextual linear bandits (Abbasi-Yadkori et al., 2011; Li et al., 2019).

Original: This difference in the setting makes our algorithm a truly asynchronous contextual linear bandit algorithm but also makes our regret analysis more challenging.
Revised: This difference in the setting makes our algorithm a truly asynchronous contextual linear bandit algorithm but also makes our regret analysis more challenging.

Original: For better comparison, we compare our work with the most related contextual linear bandit algorithms in Table 1.
Revised: For better comparison, we compare our work with the most related contextual linear bandit algorithms in Table 1.

Original:  Preliminaries
Revised:  Preliminaries

Original: Federated contextual linear bandits.
Revised: Federated contextual linear bandits.

Original: We consider the federated contextual linear bandits as follows:
Revised: We consider the federated contextual linear bandits as follows:

Original: At each round t ∈ [ T ] , an arbitrary agent m t ∈ [ M ] is active for participation.
Revised: At each round t 2 [ T ] , an arbitrary agent m t 2 [ M ] is active for participation.

Original: This agent receives a decision set D t ⊂ R d , picks an action x t ∈ D t , and receives a random reward r t .
Revised: This agent receives a decision set D t ⇢ R d , picks an action x t 2 D t , and receives a random reward r t .

Original: We assume that
Revised: We assume that

Original: Setting
Revised: Setting

Original: Algorithm
Revised: Algorithm

Original: Regret
Revised: Regret

Original: Communication
Revised: Communication

Original: Low-switching
Revised: Low-switching

Original: Noextraassump.oncontexts
Revised: Noextraassump.oncontexts

Original: Allowfreeparticipation
Revised: Allowfreeparticipation

Original: Single-agent
Revised: Single-agent

Original: OFUL d √ T log T
Revised: OFUL d p T log T

Original: N/A  ✓ ✓ ✘ (Abbasi-Yadkorietal.,2011)
Revised: N/A  8 (Abbasi-Yadkorietal.,2011)

Original: Federated
Revised: Federated

Original: DisLinUCB d √ T log 2 T dM 3 /✘ ✓ ✘ (Sync.)
Revised: DisLinUCB d p T log 2 T dM 3 /  8 (Sync.)

Original: (Wangetal.,2019)
Revised: (Wangetal.,2019)

Original: Federated
Revised: Federated

Original: Async-LinUCB d √ T log T dM 2 log T  ✘ ✘ ✘ (Async.)
Revised: Async-LinUCB d p T log T dM 2 log T  8 (Async.)

Original: (LiandWang,2022a)
Revised: (LiandWang,2022a)

Original: Federated
Revised: Federated

Original: FedLinUCB d √ T log T dM 2 log T  ✓ ✓ ✓ (Async.)
Revised: FedLinUCB d p T log T dM 2 log T  3 (Async.)

Original: (Our Algorithm 1) the reward r t satisfies r t = ⟨ x t , θ ∗ ⟩ + η t for all t ∈ [ T ] , where η t is conditionally independent of x t given x 1: t − 1 , m 1: t , r 1: t − 1 .
Revised: (Our Algorithm 1) the reward r t satisﬁes r t = h x t , ✓ ⇤ i + ⌘ t for all t 2 [ T ] , where ⌘ t is conditionally independent of x t given x 1: t  1 , m 1: t , r 1: t  1 .

Original: More specifically, we make the following assumption on η t , θ ∗ and D t , which is a standard assumption in the contextual linear bandit literature (Abbasi-Yadkori et al., 2011;
Revised: More speciﬁcally, we make the following assumption on ⌘ t , ✓ ⇤ and D t , which is a standard assumption in the contextual linear bandit literature (Abbasi-Yadkori et al., 2011;

Original: Wang et al., 2019; Dubey and Pentland, 2020).
Revised: Wang et al., 2019; Dubey and Pentland, 2020).

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: The noise η t is R -sub-Gaussian conditioning on x 1: t , m 1: t and r 1: t − 1 , i.e.,[Equation] ≤
Revised: The noise ⌘ t is R -sub-Gaussian conditioning on x 1: t , m 1: t and r 1: t  1 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: 2) , for any λ ∈ R .
Revised: for any  2 R .

Original: We also assume that ∥ θ ∗ ∥ 2 ≤ S and ∥ x ∥ 2 ≤ L for all action x ∈ D t , for all t ∈ [ T ] .
Revised: We also assume that k ✓ ⇤ k 2  S and k x k 2  L for all action x 2 D t , for all t 2 [ T ] .

Original: Notably, we assume m t can be arbitrary for all t , which basically says that each agent can decide whether and when to participate or not.
Revised: Notably, we assume m t can be arbitrary for all t , which basically says that each agent can decide whether and when to participate or not.

Original: 3 Our setting is more general than the synchronous setting in
Revised: 3 Our setting is more general than the synchronous setting in

Original: Wang et al.
Revised: Wang et al.

Original: ; Dubey and Pentland (2020), which requires a round-robin participation of all agents.
Revised: ; Dubey and Pentland (2020), which requires a round-robin participation of all agents.

Original: Learning objective.
Revised: Learning objective.

Original: The goal of the agents is to collaboratively minimize the cumulative regret defined as
Revised: The goal of the agents is to collaboratively minimize the cumulative regret deﬁned as

Original: [Equation]
Revised: [Equation]

Original: To achieve such a goal, we allow the agents to collaborate via communication through the central server.
Revised: To achieve such a goal, we allow the agents to collaborate via communication through the central server.

Original: Below we will explain the details of the communication model.
Revised: Below we will explain the details of the communication model.

Original: Communication model.
Revised: Communication model.

Original: We consider a star-shaped communication network (Wang et al., 2019;
Revised: We consider a star-shaped communication network (Wang et al., 2019;

Original: Dubey and Pentland, 2020) consisting of a central server and M agents, where each agent can communicate with the server by uploading and downloading data.
Revised: Dubey and Pentland, 2020) consisting of a central server and M agents, where each agent can communicate with the server by uploading and downloading data.

Original: However, any pair of agents cannot directly communicate with each other.
Revised: However, any pair of agents cannot directly communicate with each other.

Original: We define the communication complexity as the total number of communication rounds between agents and the server (counting both the uploads and downloads) (Wang et al., 2019; Dubey and Pentland, 2020; Li and Wang, 2022a).
Revised: We deﬁne the communication complexity as the total number of communication rounds between agents and the server (counting both the uploads and downloads) (Wang et al., 2019; Dubey and Pentland, 2020; Li and Wang, 2022a).

Original: For simplicity, we assume that there is no latency in the communication channel.
Revised: For simplicity, we assume that there is no latency in the communication channel.

Original: We consider the asynchronous setting, where the communication protocol satisfies: (1) each agent can decide whether or not to participate in each round.
Revised: We consider the asynchronous setting, where the communication protocol satisﬁes: (1) each agent can decide whether or not to participate in each round.

Original: Full participation is not required, which allows temporarily offline agents; and (2) the communication between each agent and the server is asynchronous and independent of other agents without mandatory download required by the server.
Revised: Full participation is not required, which allows temporarily ofﬂine agents; and (2) the communication between each agent and the server is asynchronous and independent of other agents without mandatory download required by the server.

Original: Switching cost.
Revised: Switching cost.

Original: The notion of switching cost in online learning and bandits refers to the number of times the agent switches its policy (i.e., decision rule)
Revised: The notion of switching cost in online learning and bandits refers to the number of times the agent switches its policy (i.e., decision rule)

Original: (Kalai and Vempala, 2005; Abbasi-Yadkori et al., 2011; Dekel et al., 2014; Ruan et al., 2021).
Revised: (Kalai and Vempala, 2005; Abbasi-Yadkori et al., 2011; Dekel et al., 2014; Ruan et al., 2021).

Original: In the context of linear bandits, it corresponds to the number of times the agent updates its policy of selecting an action from the decision set (Abbasi-Yadkori et al., 2011).
Revised: In the context of linear bandits, it corresponds to the number of times the agent updates its policy of selecting an action from the decision set (Abbasi-Yadkori et al., 2011).

Original: Algorithms with low switching cost are preferred in practice since each policy switching might cause additional computational overhead.
Revised: Algorithms with low switching cost are preferred in practice since each policy switching might cause additional computational overhead.

Original: 1: Initialize Σ m, 1 = Σ ser [-
-] = λ I , (cid:98) θ m, 1 = 0 , b loc m, 

Revised: 1: Initialize ⌃ m, 1 = ⌃ ser [-
-]1 =  I , b ✓ m, 1 = 0 , b loc m,

Original: = 0 and Σ loc m, 0 = 0 for all m ∈ [ M ] 2: for round t = 1 , 2 , . .
Revised: = 0 and ⌃ loc m, 0 = 0 for all m 2 [ M ] 2: for round t = 1 , 2 , . .

Original: , T do 3:
Revised: , T do 3:

Original: Agent m t is active 4:
Revised: Agent m t is active 4:

Original: Receive D t from the environment 5:
Revised: Receive D t from the environment 5:

Original: Select x t← argmax x ∈ D t ⟨ (cid:98) θ m t ,t , x ⟩ + β ∥ x ∥ Σ − 1 mt,t /* Optimistic decision */ 6:
Revised: Select x t argmax x 2 D t h b ✓ m t ,t , x i +  k x k ⌃  1 mt,t /* Optimistic decision */ 6:

Original: Receive r t from environment 7: Σ loc m t ,t ← Σ loc m t ,t − 1 + x t x ⊤ t , b loc m t ,t← b loc m t ,t − 1 + r t x t /* Local update */ 8: if det( Σ m t ,t + Σ loc m t ,t ) > (1 + α ) det( Σ m t ,t ) then 9:
Revised: Receive r t from environment 7: ⌃ loc m t ,t ⌃ loc m t ,t  1 + x t x > t , b loc m t ,t b loc m t ,t  1 + r t x t /* Local update */ 8: if det( ⌃ m t ,t + ⌃ loc m t ,t ) > (1 + ↵ ) det( ⌃ m t ,t ) then 9:

Original: Agent m t sends Σ loc m t ,t and b loc m t ,t to server /* Upload */ 10: Σ ser t ← Σ ser t + Σ loc m t ,t , b ser t← b ser t + b loc m t ,t / * Global update */ 11: Σ loc m t ,t← 0 , b loc m t ,t← 0 12:
Revised: Agent m t sends ⌃ loc m t ,t and b loc m t ,t to server /* Upload */ 10: ⌃ ser t ⌃ ser t + ⌃ loc m t ,t , b ser t b ser t + b loc m t ,t /* Global update */ 11: ⌃ loc m t ,t 0 , b loc m t ,t 12:

Original: Server sends Σ ser t and b ser t back to agent m t /* Download */ 13: Σ m t ,t +1 ← Σ ser t , b m t ,t +1 ← b ser t 14: (cid:98) θ m t ,t +1 ← Σ − 1 m t ,t +1 b m t ,t + /* Compute estimate */ 15: else 16: Σ m t ,t +1 ← Σ m,t , b m t ,t +1 ← b m,t , (cid:98) θ m t ,t +1 ← (cid:98) θ m t ,t 17: end if 18: for other inactive agent m ∈ [ M ] \ { m t } do 19: Σ m,t +1 ← Σ m,t , b m,t +1 ← b m,t , (cid:98) θ m,t +1 ← (cid:98) θ m,t 20: end for 21: end for
Revised: Server sends ⌃ ser t and b ser t back to agent m t /* Download */ 13: ⌃ m t ,t + ⌃ ser t , b m t ,t + b ser t 14: b ✓ m t ,t + ⌃  1 m t ,t +1 b m t ,t + /* Compute estimate */ 15: else 16: ⌃ m t ,t + ⌃ m,t , b m t ,t + b m,t , b ✓ m t ,t + b ✓ m t ,t 17: end if 18: for other inactive agent m 2 [ M ] \ { m t } do 19: ⌃ m,t + ⌃ m,t , b m,t + b m,t , b ✓ m,t + b ✓ m,t 20: end for 21: end for

Original:  The Proposed Algorithm
Revised:  The Proposed Algorithm

Original: We propose a simple algorithm based on the principle of optimism that enables collaboration among agents through asynchronous communications with the central server.
Revised: We propose a simple algorithm based on the principle of optimism that enables collaboration among agents through asynchronous communications with the central server.

Original: The main algorithm is displayed in Algorithm 1.
Revised: The main algorithm is displayed in Algorithm 1.

Original: For clarity, we first summarize the related notations in Table 2.
Revised: For clarity, we ﬁrst summarize the related notations in Table 2.

Original: [Table]
Revised: Notation Meaning b ✓ m,t estimate of ✓ ⇤ ⌃ m,t , b m,t data used to compute b ✓ m,t ⌃ loc m,t , b loc m,t local data for agent m ⌃ ser m,t , b ser m,t data stored at the server

Original: Specifically, in each round t ∈ [ T ] , agent m t participates and interacts with the environment (Line 3).
Revised: Speciﬁcally, in each round t 2 [ T ] , agent m t participates and interacts with the environment (Line 3).

Original: The environment specifies the decision set D t (Line 4), and the agent m t selects the action based on its current optimisitic estimate of the reward (Line 5).
Revised: The environment speciﬁes the decision set D t (Line 4), and the agent m t selects the action based on its current optimisitic estimate of the reward (Line 5).

Original: Here the bonus term β ∥ x ∥ Σ − 1 mt,t reflects the uncertainty of the estimated reward ⟨ (cid:98) θ m t ,t , x ⟩ and encourages exploration.
Revised: Here the bonus term  k x k ⌃  1 mt,t reﬂects the uncertainty of the estimated reward h b ✓ m t ,t , x i and encourages ex- ploration.

Original: After receiving the true reward r t from the environment, agent m t then updates its local data (Line 7).
Revised: After receiving the true reward r t from the environment, agent m t then updates its local data (Line 7).

Original: The key component of the algorithm is the matrix determinant-based criterion (Line 8), which evaluates the information accumulated in current local data.
Revised: The key component of the algorithm is the matrix determinant-based criterion (Line 8), which evaluates the information accumulated in current local data.

Original: If the criterion is satisfied, it suggests that the local data would help significantly reduce the uncertainty of estimating the model θ ∗ .
Revised: If the criterion is satisﬁed, it suggests that the local data would help signiﬁcantly reduce the uncertainty of estimating the model ✓ ⇤ .

Original: Therefore, agent m t will share its progress by uploading the local data to the server (Line 9) so that it can benefit other agents.
Revised: Therefore, agent m t will share its progress by uploading the local data to the server (Line 9) so that it can beneﬁt other agents.

Original: Then the server updates the global data accordingly (Line 10).
Revised: Then the server updates the global data accordingly (Line 10).

Original: Afterwards, agent m t downloads the latest global data from the server (Line 12) and updates its local data and model (Line 13-14).
Revised: Afterwards, agent m t downloads the latest global data from the server (Line 12) and updates its local data and model (Line 13-14).

Original: If the criterion in Line 8 is not met, then the communication between the agent and the server will not be triggered, and the local data remains local and unshared for agent m t (Line 16).
Revised: If the criterion in Line 8 is not met, then the communication between the agent and the server will not be triggered, and the local data remains local and unshared for agent m t (Line 16).

Original: Finally, all the other inactive agents remain unchanged (Line 19).
Revised: Finally, all the other inactive agents remain unchanged (Line 19).

Original: Note that in Algorithm 1, the communication between the agent and the server (Line 9 and 12) involves only the active agent in that round, which is completely independent of other agents.
Revised: Note that in Algorithm 1, the communication between the agent and the server (Line 9 and 12) involves only the active agent in that round, which is completely independent of other agents.

Original: This is in sharp contrast to existing algorithms.
Revised: This is in sharp contrast to existing algorithms.

Original: For example, in the main algorithm in Li and Wang (2022a), upload by any agent may trigger other agents to download the latest data, while our algorithm does not mandate this.
Revised: For example, in the main algorithm in Li and Wang (2022a), upload by any agent may trigger other agents to download the latest data, while our algorithm does not mandate this.

Original: On the other hand, many existing algorithms for multi-agent settings (e.g., Wang et al.
Revised: On the other hand, many existing algorithms for multi-agent settings (e.g., Wang et al.

Original: (2019)) require all agents to interact with the environment in each round, which essentially require full participation of all the agents.
Revised: (2019)) require all agents to interact with the environment in each round, which essentially require full participation of all the agents.

Original: The determinant-based criterion in Line 8 has been a long-standing design trick in contextual linear bandits that can help address the issue of unknown time horizon and reduce the switching cost (Abbasi-Yadkori et al., 2011; Ruan et al., 2021).
Revised: The determinant-based criterion in Line 8 has been a long-standing design trick in contextual linear bandits that can help address the issue of unknown time horizon and reduce the switching cost (Abbasi-Yadkori et al., 2011; Ruan et al., 2021).

Original: For multi-agent bandits, such a criterion has also been used to control the communications complexity (Wang et al., 2019; Dubey and Pentland, 2020;
Revised: For multi-agent bandits, such a criterion has also been used to control the communications complexity (Wang et al., 2019; Dubey and Pentland, 2020;

Original: Li and Wang, 2022a).
Revised: Li and Wang, 2022a).

Original: This is because the need for policy switching or communication essentially reflects the same fact: enough information has been collected and it is time to update the (local) model.
Revised: This is because the need for policy switching or communication essentially reﬂects the same fact: enough information has been collected and it is time to update the (local) model.

Original: Indeed, achieving low communication complexity and low switching cost are unified in our
Revised: Indeed, achieving low communication complexity and low switching cost are uniﬁed in our

Original: FedlinUCB algorithm in the sense that the communication complexity is exactly twice the switching cost.
Revised: FedlinUCB algorithm in the sense that the communication complexity is exactly twice the switching cost.

Original: Furthermore, using lazy update makes our algorithm amenable for analysis, which will be clear later in Section 6.
Revised: Furthermore, using lazy update makes our algorithm amenable for analysis, which will be clear later in Section 6.

Original: In addition, we leave α > 0 as a tuning parameter as it controls the trade-off between the regret and the communication complexity.
Revised: In addition, we leave ↵ > 0 as a tuning parameter as it controls the trade-off between the regret and the communication complexity.

Original:  Theoretical Results
Revised:  Theoretical Results

Original: We now present our main result on the theoretical guarantee of Algorithm 1.
Revised: We now present our main result on the theoretical guarantee of Algorithm 1.

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: Under Assumption 3.1 for Algorithm 1, if we set the confidence radius β = √ λS + ( √ 1 +Mα + M √ 2 α ) (cid:16) R (cid:114) d log (cid:16)(cid:0) 1 + TL 2 / (min( α, 1) λ ) (cid:1) /δ
Revised: Under Assumption 3.1 for Algorithm 1, if we set the conﬁdence radius  = p  S + ( p 1 + M ↵ + M p 2 ↵ ) ⇣ [Equation]

Original: (cid:17) + √ λS (cid:17) , then with probability at least 1 − δ , the regret in the first T rounds can be upper bounded by
Revised: + [Equation] 2 / (min( ↵ , 1)  [Equation]

Original: Regret(
Revised:  [Equation] ⌘

Original: [Equation]
Revised: [Equation]

Revised: ⌘ , then with probability at least 1   , the regret in the ﬁrst T rounds can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: + TL 2 /λ ) . Moreover, the communication complexity and switching cost are both bounded by 2 log 2 · d ( M + 1 /α ) log(1 + TL 2 / ( λd )) .
Revised: Moreover, the communication complexity and switching cost are both bounded by 2 log 2 · d ( M + 1 / ↵ ) log(1 + TL 2 / (  d )) .

Original: Remark 5.2.
Revised: Remark 5.2.

Original: Theorem 5.1 suggests that if we set the parameters α = 1 /M 2 and λ = 1 /Sin Algorithm 1, then its regret is bounded by (cid:101) O
Revised: Theorem 5.1 suggests that if we set the parameters ↵ = 1 /M 2 and  = 1 /Sin Algorithm 1, then its regret is bounded by e

Original:  ( Rd √ T ) and the corresponding communication complexity and switching cost are both bounded by (cid:101) O ( dM 2 ) .
Revised: O ( Rd p T ) and the corresponding communication complexity and switching cost are both bounded by e O ( dM 2 ) .

Original: This choice of parameters yields the regret bound and the communication complexity presented in Table 1.
Revised: This choice of parameters yields the regret bound and the communication complexity presented in Table 1.

Original: As a complement, we also provide a lower bound for the communication complexity as stated in the following theorem.
Revised: As a complement, we also provide a lower bound for the communication complexity as stated in the following theorem.

Original: See Appendix D for the proof.
Revised: See Appendix D for the proof.

Original: Theorem 5.3.
Revised: Theorem 5.3.

Original: For any algorithm Alg with expected communication complexity less than
Revised: For any algorithm Alg with expected communication complexity less than

Original: O ( M/ log( T/M )) , there exist a linear bandit instance with R =
Revised: O ( M/ log( T/M )) , there exist a linear bandit instance with R =

Original: L = S = 1 such that for T ≥ Md , the expected regret for algorithm Alg is at least Ω( d √ MT ) .
Revised: L = S = 1 such that for T  Md , the expected regret for algorithm Alg is at least ⌦ ( d p MT ) .

Original: Remark 5.4.
Revised: Remark 5.4.

Original: Suppose each agent runs the OFUL algorithm (Abbasi-Yadkori et al., 2011) separately, then each agent m ∈{+
+}
Revised: Suppose each agent runs the OFUL algorithm (Abbasi-Yadkori et al., 2011) separately, then each agent m {+
+}

Original: [ M ] admits an (cid:101)
Revised: [ M ] admits an e

Original: O ( d √ T m ) regret, where T m is the number of rounds that agent m participates in.
Revised: O ( d p T m ) regret, where T m is the number of rounds that agent m participates in.

Original: Thus the total regret of M agents is upper bounded by(cid:80) M m =1 (cid:101) O ( √ T m )
Revised: Thus the total regret of M agents is upper bounded by

Original:  = (cid:101) O 
Revised: P Mm = e O ( p T m ) = e

Original:  ( d √ MT ) .
Revised: O ( d p MT ) .

Original: Theorem 5.3 implies that for any algorithm Alg , if its communication complexity is less than O ( M/ log( T/M )) , then its regret cannot be better than naively running M independent OFUL algorithms.
Revised: Theorem 5.3 implies that for any algorithm Alg , if its communication complexity is less than O ( M/ log( T/M )) , then its regret cannot be better than naively running M independent OFUL algorithms.

Original: In other words, Theorem 5.3 suggests that in order to improve the performance through collaboration, an Ω( M ) communication complexity is necessary.
Revised: In other words, Theorem 5.3 suggests that in order to improve the performance through collaboration, an ⌦ ( M ) communication complexity is necessary.

Original:  Overview of the Proof
Revised:  Overview of the Proof

Original: When analyzing the performance of FedLinUCB , we face a unique challenge caused by the asynchronous communication, as illustrated in Figure 1.
Revised: When analyzing the performance of FedLinUCB , we face a unique challenge caused by the asynchronous communication, as illustrated in Figure 1.

Original: Here ( x m,t , η m,t ) denotes the decision and the noise for agent m in its own t -th round.
Revised: Here ( x m,t , ⌘ m,t ) denotes the decision and the noise for agent m in its own t -th round.

Original: Specifically, in the synchronous setting, the filtration is generated by all the data collected by all agents, i.e., F 5 = σ { x m,t , η m,t } 5 , 5 t =1 ,m =1 , as marked by the green dashed rectangle.
Revised: Speciﬁcally, in the synchronous setting, the ﬁltration is generated by all the data collected by all agents, i.e., F 5 =  { x m,t , ⌘ m,t } 5 , 5 t =1 ,m =1 , as marked by the green dashed rectangle.

Original: This kind of filtration is well-defined since all agents share their data with each other at the end of each round.
Revised: This kind of ﬁltration is well-deﬁned since all agents share their data with each other at the end of each round.

Original: In sharp contrast, in our asynchronous setting, the data at the server can be generated by an irregular set of data from the agents, as marked by the blue rectangles.
Revised: In sharp contrast, in our asynchronous setting, the data at the server can be generated by an irregular set of data from the agents, as marked by the blue rectangles.

Original: Such data pattern can be arbitrary and depends on the data collected in all previous rounds, which prevents us from defining a fixed filtration as we can do in the synchronous setting.
Revised: Such data pattern can be arbitrary and depends on the data collected in all previous rounds, which prevents us from deﬁning a ﬁxed ﬁltration as we can do in the synchronous setting.

Original: Since the application of standard martingale concentration inequalities relies on the well-defined filtration, they cannot be directly applied to our asynchronous setting.
Revised: Since the application of standard martingale concentration inequalities relies on the well-deﬁned ﬁltration, they cannot be directly applied to our asynchronous setting.

Original: To circumvent the above issue, we need to analyze the concentration property of the local data for each agent and then relate it to the concentration of the global data, so that we can control the sum of the bonuses and hence the regret.
Revised: To circumvent the above issue, we need to analyze the concentration property of the local data for each agent and then relate it to the concentration of the global data, so that we can control the sum of the bonuses and hence the regret.

Original: This requires a careful quantitative comparison of the local and global data covariance matrices, which is enabled by our design of determinant-based criterion.
Revised: This requires a careful quantitative comparison of the local and global data covariance matrices, which is enabled by our design of determinant-based criterion.

Original: The details will be further explained in
Revised: The details will be further explained in

Original: Section 6.2.
Revised: Section 6.2.

Original: In the next subsection, we present the key ingredients of the proof of Theorem 5.1.
Revised: In the next subsection, we present the key ingredients of the proof of Theorem 5.1.

Original: Remark 6.1.
Revised: Remark 6.1.

Original: Recall the notations in Table 2, where the values of those matrices and vectors might change within each round.
Revised: Recall the notations in Table 2, where the values of those matrices and vectors might change within each round.

Original: To eliminate the possible confusion, from now on we follow the convention that all matrices and vectors in the analysis correspond to their values at the end of each round in
Revised: To eliminate the possible confusion, from now on we follow the convention that all matrices and vectors in the analysis correspond to their values at the end of each round in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: 6.1 Analysis for communication complexity and switching cost
Revised: 6.1 Analysis for communication complexity and switching cost

Original: We first analyze the communication complexity and switching cost of Algorithm 1.
Revised: We ﬁrst analyze the communication complexity and switching cost of Algorithm 1.

Original: For each i ≥ 0 , we define
Revised: For each i  0 , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: We divide the set of all rounds into epochs { τ i , τ i + 1 , . . .
Revised: We divide the set of all rounds into epochs { ⌧ i , ⌧ i + 1 , . . .

Original: , τ i +1 − 1 } for each i ≥ 0 .
Revised: , ⌧ i +1  1 } for each i  0 .

Original: Then the communication complexity within each epoch can be bounded using the following lemma.
Revised: Then the communication complexity within each epoch can be bounded using the following lemma.

Original: Lemma 6.2.
Revised: Lemma 6.2.

Original: Under the setting of Theorem 5.1, for each epoch from round τ i to round τ i +1 − 1 , the number of communications in this epoch is upper bounded by 2( M + 1 /α ) .
Revised: Under the setting of Theorem 5.1, for each epoch from round ⌧ i to round ⌧ i +1  1 , the number of communications in this epoch is upper bounded by 2( M + 1 / ↵ ) .

Original: Proof of Theorem 5.1: communication complexity and switching cost.
Revised: Proof of Theorem 5.1: communication complexity and switching cost.

Original: It suffices to bound the number of epochs.
Revised: It sufﬁces to bound the number of epochs.

Original: By Assumption 3.1, we have ∥ x t ∥ 2 ≤ L for all t ∈ [ T ] .
Revised: By Assumption 3.1, we have k x t k 2  L for all t 2 [ T ] .

Original: Since Σ ser T is positive definite, by the inequality of arithmetic and geometric means, we have[Equation] (cid:19)(cid:19) [Equation]
Revised: Since ⌃ ser T is positive deﬁnite, by the inequality of arithmetic and geometric means, we have

Original: λd

Original: [Equation]
Revised: [Equation]

Original: Then recalling the definition of epochs based on (6.1), we have
Revised: Then recalling the deﬁnition of epochs based on (6.1), we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, the total number of epochs is bounded by log 2 · d log(1 + TL 2 / ( λd )) .
Revised: Therefore, the total number of epochs is bounded by log 2 · d log(1 + TL 2 / (  d )) .

Original: Now applying
Revised: Now applying

Original: Lemma 6.2, the total communication complexity is bounded by 2 log 2 · d ( M + 1 /α ) log(1 +
Revised: Lemma 6.2, the total communication complexity is bounded by 2 log 2 · d ( M + 1 / ↵ ) log(1 +

Original: TL 2 / ( λd )) .
Revised: TL 2 / (  d )) .

Original: Note that in Algorithm 1, each agent only switch its policy after communicating with the server, so the switching cost is exactly equal to half of the communication complexity.
Revised: Note that in Algorithm 1, each agent only switch its policy after communicating with the server, so the switching cost is exactly equal to half of the communication complexity.

Original: This finishes the proof for the claim on communication complexity and switching cost in Theorem 5.1.
Revised: This ﬁnishes the proof for the claim on communication complexity and switching cost in Theorem 5.1.

Original: 6.2 Analysis for regret upper bound
Revised: 6.2 Analysis for regret upper bound

Original: The regret analysis for Theorem 5.1 is much more involved, and it relies on a series of intermediate lemmas establishing the concentration.
Revised: The regret analysis for Theorem 5.1 is much more involved, and it relies on a series of intermediate lemmas establishing the concentration.

Original: Total information.
Revised: Total information.

Original: We define the following auxiliary matrices and vectors that contain all the information up to round t :
Revised: We deﬁne the following auxiliary matrices and vectors that contain all the information up to round t :

Original: [Equation]
Revised: [Equation]

Original: where η i := r i − ⟨ x i , θ ∗ ⟩ is a R -sub-Gaussian noise by Assumption 3.1.
Revised: where ⌘ i := r i  h x i , ✓ ⇤ i is a R -sub-Gaussian noise by Assumption 3.1.

Original: In our setting, Σ all t , b all t , u all t are not accessible by the agents due to asynchronous communication, and we only use them to facilitate the analysis.
Revised: In our setting, ⌃ all t , b all t , u all t are not accessible by the agents due to asynchronous communication, and we only use them to facilitate the analysis.

Original: With this notation, we can further define the following omnipotent estimate:
Revised: With this notation, we can further deﬁne the following omnipotent estimate:

Original: [Equation]
Revised: [Equation]

Original: As a direct application of the self-normalized martingale concentration inequality (Abbasi-Yadkori et al., 2011), we have the following global confidence bound due to the concentration of Σ all t and b all t .
Revised: As a direct application of the self-normalized martingale concentration inequality (Abbasi-Yadkori et al., 2011), we have the following global conﬁdence bound due to the concentration of ⌃ all t and b all t .

Original: Lemma 6.3 (Global confidence bound; Theorem 2, Abbasi-Yadkori et al. 2011) .
Revised: Lemma 6.3 (Global conﬁdence bound; Theorem 2, Abbasi-Yadkori et al. 2011) .

Original: With probability at least 1 − δ , for each round t ∈ [ T ] , the estimate(cid:98) θ all t in (6.3) satisfies
Revised: With probability at least 1   , for each round t 2 [ T ] , the estimate

Original: [Equation]
Revised: [Equation]

Original: Per-agent information.
Revised: Per-agent information.

Original: Next, for each agent m ∈{+
+} [ M ] , we denote the rounds when agent m communicate with the server (i.e., upload and download data) as { t m, 1 , t m, 2 , ... } .
Revised: Next, for each agent m {+
+} [ M ] , we denote the rounds when agent m communicate with the server (i.e., upload and download data) as { t m, 1 , t m, 2 , ... } .

Original: For simplicity, at the end of round t , we denote by N m ( t ) the last round when agent m communicated with the server (so if agent m communicated with the server in round t , then N m ( t )
Revised: For simplicity, at the end of round t , we denote by N m ( t ) the last round when agent m communicated with the server (so if agent m communicated with the server in round t , then N m ( t )

Original: = t ).
Revised: = t ).

Original: With this notation, for each round t and agent m ∈ [ M ] , the data that has been uploaded by agent m is then
Revised: With this notation, for each round t and agent m 2 [ M ] , the data that has been uploaded by agent m is then⌃ [Equation] t [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Original: Correspondingly, the local data that has not been uploaded to the server is
Revised: N [Equation] X

Original: [Equation]
Revised: [Equation]

Original: m
Revised: x [Equation] ⌘

Original: [Equation]
Revised: [Equation]

Original:  t (cid:88)
Revised: Correspondingly, the local data that has not been uploaded to the server is ⌃ loc m,t = t X j = N m ( t )+

Original: [Equation]
Revised: [Equation]

Original: N m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: t (cid:88)
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original:  Again, applying the self-normalized martingale concentration (Abbasi-Yadkori et al., 2011) together with a union bound, we can get the per-agent local concentration.
Revised: [Equation] Again, applying the self-normalized martingale concentration (Abbasi-Yadkori et al., 2011) together with a union bound, we can get the per-agent local concentration.

Original: Lemma 6.4 (Local concentration) .
Revised: Lemma 6.4 (Local concentration) .

Original: Under the setting of Theorem 5.1, with probability at least 1 − δ , for each round t ∈ [ T ] and each agent m ∈{+
+}
Revised: Under the setting of Theorem 5.1, with probability at least 1   , for each round t 2 [ T ] and each agent m {+
+}

Original: [ M ] , it holds that
Revised: [ M ] , it holds that

Original: [Equation]
Revised: [Equation]

Original: Moreover, based on our determinant-based communication criterion, we have the following lemma describing the quantitative relationship among the local data, uploaded data and global data.
Revised: Moreover, based on our determinant-based communication criterion, we have the following lemma describing the quantitative relationship among the local data, uploaded data and global data.

Original: Lemma 6.5 (Covariance comparison) .
Revised: Lemma 6.5 (Covariance comparison) .

Original: Under the setting of Theorem 5.1, it holds that
Revised: Under the setting of Theorem 5.1, it holds that

Original: [Equation]
Revised: [Equation]

Original: for each agent m ∈{+
+}
Revised: for each agent m {+
+}

Original: [ M ] .
Revised: [ M ] .

Original: Moreover, for any 1 ≤ t 1 < t 2 ≤ T , if agent m is the only active agent from round t 1 to t 2 − 1 and agent m only communicates with the server at round t 1 , then for all t 1 + 1 ≤ t ≤ t 2 , it further holds that
Revised: Moreover, for any 1  t 1 < t 2  T , if agent m is the only active agent from round t 1 to t 2  1 and agent m only communicates with the server at round t 1 , then for all t 1 + 1  t  t 2 , it further holds that

Original: [Equation]
Revised: [Equation]

Original: Combining the above results, we obtain the local confidence bound, which then leads to the per-round regret in each round, as summarized in the following lemma.
Revised: Combining the above results, we obtain the local conﬁdence bound, which then leads to the per-round regret in each round, as summarized in the following lemma.

Original: Lemma 6.6 (Local confidence bound and per-round regret) .
Revised: Lemma 6.6 (Local conﬁdence bound and per-round regret) .

Original: Under the setting of Theorem 5.1, with probability at least 1 − δ , for each t ∈ [ T ] , the estimate (cid:98) θ m,t +1 satisfies that ∥ θ ∗ − (cid:98) θ m,t +1 ∥ Σ m,t +1 ≤ β for all m ∈ [ M ] .
Revised: Under the setting of Theorem 5.1, with probability at least 1   , for each t 2 [ T ] , the estimate b ✓ m,t +1 satisﬁes that k ✓ ⇤  b ✓ m,t +1 k ⌃ m,t +1   for all m 2 [ M ] .

Original: Consequently, for each round t ∈ [ T ] , the regret in round t satisfies
Revised: Consequently, for each round t 2 [ T ] , the regret in round t satisﬁes [Equation] q

Original: [Equation]
Revised: [Equation]

Original: Now, we are ready to prove the regret bound in Theorem 5.1.
Revised: Now, we are ready to prove the regret bound in Theorem 5.1.

Original: Proof of Theorem 5.1: regret.
Revised: Proof of Theorem 5.1: regret.

Original: First, according to Lemma 6.6, the regret in first T round can be decomposed and upper bounded by
Revised: First, according to Lemma 6.6, the regret in ﬁrst T round can be decomposed and upper bounded by

Original: [Equation]
Revised: [Equation]

Original: Now, we only need to control the summation of the upper confidence bonus term 2 β ∥ x t ∥ Σ − 1 mt,t
Revised: Now, we only need to control the summation of the upper conﬁdence bonus term 2  k x t k ⌃  1 mt,t

Original: and we focus on the agent-action sequence { ( m t , x t ) }
Revised: and we focus on the agent-action sequence { ( m t , x t ) }

Original: Tt =1 .
Revised: Tt =1 .

Original: Notice that if an agent m communicate with the server at time t 1 and t 2 , then the order of actions between round t 1 and t 2 will not effect the covariance matrix for agent m .
Revised: Notice that if an agent m communicate with the server at time t 1 and t 2 , then the order of actions between round t 1 and t 2 will not effect the covariance matrix for agent m .

Original: In addition, since agent m does not upload new data between round tand t 2 , the order of actions from agent m will also not affect other agents’ covariance matrix.
Revised: In addition, since agent m does not upload new data between round tand t 2 , the order of actions from agent m will also not affect other agents’ covariance matrix.

Original: Thus, without effect the covariance matrix and the corresponding upper confidence bonus, we can always reorder the sequence of active agents such that each agent communicates with the server and stays active until the next agent kicks in to communicate with the server.
Revised: Thus, without effect the covariance matrix and the corresponding upper conﬁdence bonus, we can always reorder the sequence of active agents such that each agent communicates with the server and stays active until the next agent kicks in to communicate with the server.

Original: Such reordering is valid according to the communication protocol as each agent has only local updates between communications with the server.
Revised: Such reordering is valid according to the communication protocol as each agent has only local updates between communications with the server.

Original: More specifically, we assume that the sequence of round that the active agent communicates with server is 0 = t 0 < t 1 < t 2 < · · · < t N = T + 1 5 , and from round t i + 1 to t i +1 − 1 there is only one agent active, that is, m t i
Revised: More speciﬁcally, we assume that the sequence of round that the active agent communicates with server is 0 = t 0 < t 1 < t 2 < · · · < t N = T + 1 5 , and from round t i + 1 to t i +1  1 there is only one agent active, that is, m t i

Original: = m
Revised: = m

Original: t i +1 = · · · = m t
Revised: t i +1 = · · · = m t

Original: i +1 − 1 .
Revised: i +1  1 .

Original: Then we apply Lemma 6.5 and get that the bonus term for each round t i < t 
Revised: Then we apply Lemma 6.5 and get that the bonus term for each round t i < t < t i [Equation]

Original: < t i +1 can be controlled by
Revised: 1 can be controlled by

Original: [Equation]
Revised: [Equation]

Original: In addition, to control the bonus term for rounds { t i } Ni =1 , we define T i = min { t ∈ [ T ] | det( Σ all t ) ≥ 2 i λ d } .
Revised: In addition, to control the bonus term for rounds { t i } Ni =1 , we deﬁne T i = min { t 2 [ T ] | det( ⌃ all t )  2 i  d } .

Original: For each time interval from T i to T i +1 , if an agent m communicate with the server more than once, e.g., agent m communicates with the server at round T i, 1 and T i, 2 such that T i ≤ T 
Revised: For each time interval from T i to T i +1 , if an agent m communicate with the server more than once, e.g., agent m communicates with the server at round T i, 1 and T i, 2 such that T i  T i, 1 < T i,

Original: i, 1 < T i, 2 < T i +1 , then for the latter round T i, 2 , the bonus term can be controlled by
Revised: 2 < T i +1 , then for the latter round T i, 2 , the bonus term can be controlled by

Original: [Equation]
Revised: [Equation]

Original: where the first inequality holds due Lemma 6.5 and agent m communicate with the server at the previous round T i, 1 , the second inequality holds due to Lemma E.4 with the fact that det Σ
Revised: where the ﬁrst inequality holds due Lemma 6.5 and agent m communicate with the server at the previous round T i, 1 , the second inequality holds due to Lemma E.4 with the fact that det ⌃ all T i +1  1 / det

Original: [Equation]
Revised: [Equation]

Original: / det
Revised: ⌃ all T i,

Original: [Equation]
Revised: [Equation]

Original: Σ all
Revised: 

Original: [Equation]
Revised: [Equation]

Original: i, 1 ) ≤

Original: [Equation]
Revised: [Equation]

Revised:

Original: [Equation]
Revised: [Equation]

Original: 2 i λ d ) = 2 , and the last inequality holds due to the fact that Σ all T i +1 − 1 ⪰ Σ all T i, 2 .
Revised: ) = 2 , and the last inequality holds due to the fact that ⌃ all T i +1  1 ⌫ ⌃ all T i, 2 .

Original: On the other hand, for each time interval from T i to T i +1 , the bonus term for the first communication can always be trivially bounded by 1 for all agent m .
Revised: On the other hand, for each time interval from T i to T i +1 , the bonus term for the ﬁrst communication can always be trivially bounded by 1 for all agent m .

Original: Therefore, the summation of the regret over first communication can be upper bounded by 1 .
Revised: Therefore, the summation of the regret over ﬁrst communication can be upper bounded by 1 .

Original: In addition, since the norm of each action ∥ x ∥ 2 ≤ L and it implies that we have
Revised: In addition, since the norm of each action k x k 2  L and it implies that we have

Original: [Equation]
Revised: [Equation]

Original: which implies that the number of different intervals is at most d log(1 + TL 2 /λ ) .
Revised: which implies that the number of different intervals is at most d log(1 + TL 2 /  ) .

Original: Combining the upper bound of regret in (6.6), (6.7) and (6.8), we have
Revised: Combining the upper bound of regret in (6.6), (6.7) and (6.8), we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from a standard elliptic potential argument (Abbasi-Yadkori et al., 2011).
Revised: where the last inequality follows from a standard elliptic potential argument (Abbasi-Yadkori et al., 2011).

Original: This completes the proof.
Revised: This completes the proof.

Original:  Experiments
Revised:  Experiments

Original: In this section [Figure] we report numerical simulation results on the comparison between our algorithm with other baselines.
Revised: In this section, we report numerical simulation results on the comparison between our algorithm with other baselines.

Original: Specifically, we construct a contextual linear bandit instance with feature dimension d = 25 .
Revised: Speciﬁcally, we construct a contextual linear bandit instance with feature dimension d = 25 .

Original: In each round t , the active agent m t is uniformly sampled from all M agents.
Revised: In each round t , the active agent m t is uniformly sampled from all M agents.

Original: We set the total number of rounds T = 30 , 000 and test for M = 15 and 30 .
Revised: We set the total number of rounds T = 30 , 000 and test for M = 15 and 30 .

Original: We compare our
Revised: We compare our

Original: FedLinUCB with Async-LinUCB (Li and Wang, 2022a) and OFUL Abbasi-Yadkori et al.
Revised: FedLinUCB with Async-LinUCB (Li and Wang, 2022a) and OFUL Abbasi-Yadkori et al.

Original: (2011) with full communication (i.e., the active agent communicates with the server in each round).
Revised: (2011) with full communication (i.e., the active agent communicates with the server in each round).

Original: Due to space limit, further details and more simulation results are deferred to Appendix B.
Revised: Due to space limit, further details and more simulation results are deferred to Appendix B.

Original: The code and data for our experiments can be found on Github 6 .
Revised: The code and data for our experiments can be found on Github 6 .

Original: [Figure]
Revised: [Figure]

Original: It is clear from Figure 2 that FedLinUCB outperforms Async-LinUCB (Li and Wang, 2022a) in terms of regret.
Revised: It is clear from Figure 2 that FedLinUCB outperforms Async-LinUCB (Li and Wang, 2022a) in terms of regret.

Original: Although OFUL with full communication has the smallest regret, its communication cost ( 2 MT ) is much higher than ours.
Revised: Although OFUL with full communication has the smallest regret, its communication cost ( 2 MT ) is much higher than ours.

Original: Furthermore, we plot the log-scaled regret in Figure 3(c) and 3(d) in Appendix B, which show that the average regret of our algorithm actually has a rate very close to the optimal rate of OFUL.
Revised: Furthermore, we plot the log-scaled regret in Figure 3(c) and 3(d) in Appendix B, which show that the average regret of our algorithm actually has a rate very close to the optimal rate of OFUL.

Original: Overall, the numerical simulation corroborates our theoretical results.
Revised: Overall, the numerical simulation corroborates our theoretical results.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: In this work, we study federated contextual linear bandit problem with fully asynchronous communication.
Revised: In this work, we study federated contextual linear bandit problem with fully asynchronous communication.

Original: We propose a simple and provably efficient algorithm named FedLinUCB .
Revised: We propose a simple and provably efﬁcient algorithm named FedLinUCB .

Original: We prove that FedLinUCB obtains a near-optimal regret of order (cid:101)
Revised: We prove that FedLinUCB obtains a near-optimal regret of order e

Original: O ( d √ T ) with (cid:101) O ( dM 2 ) communication complexity.
Revised: O ( d p T ) with e O ( dM 2 ) communication complexity.

Original: We also prove a lower bound on the communication complexity, which suggests that an Ω( M ) communication complexity is necessary for achieving a near-optimal regret.
Revised: We also prove a lower bound on the communication complexity, which suggests that an ⌦ ( M ) communication complexity is necessary for achieving a near-optimal regret.

Original: There still exists an O ( dM ) gap between the upper and lower bounds for the communication complexity and we leave it as a future work to close this gap.
Revised: There still exists an O ( dM ) gap between the upper and lower bounds for the communication complexity and we leave it as a future work to close this gap.

Original: Another important direction for future work is to study federated linear bandits with a decentralized communication network without a central server (i.e., P2P networks).
Revised: Another important direction for future work is to study federated linear bandits with a decentralized communication network without a central server (i.e., P2P networks).

Original: In addition, there are potential privacy concerns when the agents upload and download data from the server, and it remains an open problem to devise provably efficient algorithm for asynchronous federated linear bandits with privacy guarantees.
Revised: In addition, there are potential privacy concerns when the agents upload and download data from the server, and it remains an open problem to devise provably efﬁcient algorithm for asynchronous federated linear bandits with privacy guarantees.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We thank the anonymous reviewers and area chair for their helpful comments.
Revised: We thank the anonymous reviewers and area chair for their helpful comments.

Original: JH and QG are partially supported by the National Science Foundation CAREER Award 1906169 and the Sloan Research
Revised: JH and QG are partially supported by the National Science Foundation CAREER Award 1906169 and the Sloan Research

Original: Fellowship.
Revised: Fellowship.

Original: The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
Revised: The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: This work is seeking to develop a mathematical understanding of federated linear bandits.
Revised: This work is seeking to develop a mathematical understanding of federated linear bandits.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We use existing theorems and cite them properly.
Revised: [Yes] We use existing theorems and cite them properly.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: A Further Discussions

Original: Here we provide further discussions on our results.

Original: We present a detailed comparison with Li and

Original: Wang (2022a) in Appendix A.1, where we first discuss the difference in the algorithmic design, and then elaborate on the concentration issue under the asynchronous setting with a simple example.

Original: In

Original: Appendix A.2, we give an alternative form of our algorithm, where we rewrite Algorithm 1 in an ‘episodic’ fashion.

Original: The purpose is to make it easier for readers to compare our algorithm with existing algorithms for federated linear bandits that are usually expressed in the ‘episodic’ form.

Original: A.1 Comparison with Li and Wang (2022a)

Original: Difference in algorithmic design.

Original: The Async-LinUCB algorithm proposed by Li and Wang (2022a) is not fully asynchronous since in their algorithm, if some agent uploads data to the server, the server will decide if each of the M agents needs to download the data.

Original: If the server decides that an agent needs to download the data, this agent has to first download the data from the server and then update its local policy before further interaction with the environment (i.e., taking the next action).

Original: In other words, if an agent is offline when the server requests a download, the agent cannot take any further action until it goes online and completes the required download and local model update.

Original: In contrast, under the communication protocol in our Algorithm 1, any offline agent can still take action until the trigger of the upload protocol.

Original: It is evident that their asynchronous communication protocol is very restricted.

Original: Concentration issue.

Original: Next, we discuss the concentration issue, and we first illustrate the problem using a multi-arm bandit instance.

Original: Unlike the synchronous case, the reward estimator based on the server-end data can be biased in asynchronous federated linear bandits.

Original: To see so, let us consider the following simple example: The decision set contains two arms, A and B , and suppose for pulling arm A , the agent receives a reward equal to either 1 or − 1 with equal probability.

Original: We assume that there are M agents, and each agent is active for two consecutive rounds.

Original: For each agent m ∈

Original: [ M ] , if the agent has selected the arm A in the first round, then the agent will select again the arm A in the second round only if the agent receives a reward of 1 when pulling arm A in the first round.

Original: In this case, it is easy to show that with probability 0 .

Original: 5 , an agent selects arm A one time with reward − 1 , and with probability 0 .

Original: 25 , an agent selects arm A twice with total reward 2 .

Original: Similarly, with probability 0 .

Original: 25 , an agent selects arm A twice with a total reward of 0 .

Original: In the synchronous setting, all agent will upload their local data to the server at the end of each round.

Original: Thus, taking an average for all data at the server, the expected reward of arm A is 0 , which equals the actual expected reward of arm A .

Original: However, in the asynchronous setting, things become more complicated.

Original: Suppose that for each agent, only selecting arm A twice will trigger the upload protocal.

Original: Then after two active rounds, an agent will upload its data to the server if and only if the agent receives reward 1 in the first round.

Original: Thus among the agents that upload the data, half of them receive a total reward of 2 and the other half receive a total reward of 0 .

Original: In this case, taking an average for all data at the server, the expected reward of arm A is 0 .

Original: 5 , which is a biased estimator compared with the actual expected reward.

Original: Indeed, the above issue could happen in federated linear bandits with the Async-LinUCB algorithm (Li and Wang, 2022a).

Original: Specifically, let us consider a linear bandit instance with dimension d = 2 , and we assume that arm A has context vector

Original: x

Original: A

Original: =

Original: (3 , 0) ⊤ , arm B has context vector x B = (0 , 1 / √ 10) ⊤ , the true model is θ ∗ = 0 , the noise η is a Rademacher random variable, and the parameter λ is set to be 1.

Original: Therefore, the rewards for both arm A and B equal to 1 or − 1 with 0 .

Original: 5 probability.

Original: In this case, based on the principle of optimism in the face of uncertainty, at the beginning, the optimistic estimators for the two arms A, B are 3 β and β/ √ 10 respectively.

Original: Thus, all agents will always choose arm A in the first round, so x 1 = x A .

Original: After choosing arm A at the first round, the optimistic estimator for the two arms A, B in each agent’s second round will be 9 r/ 10+3 β/ √ 10 and β/ √ 10 respectively, where r is the reward received in the first round.

Original: Therefore, with confidence radius β < 1 , each agent will select the arm A (i.e., x 2 = x A ) in the second round only if the agent receives a reward of r =in the first round.

Original: Finally, only choosing arm A twice will increase the determinant of the covariance matrix enough to trigger the upload protocol (e.g., det( λ I + x 1 x ⊤ 1 + x 2 x ⊤ 2 ) / det( λ I ) ≥ 19 ).

Original: As demonstrated above, in the asynchronous setting, the reward estimator based on the server-end data can be biased, which leads to the issue that previous concentration results (e.g., Abbasi-Yadkori

Original: [-1: Initialize Σ m, 1 = Σ ser 
-]
Revised: [-1: Initialize Σ m, 1 = Σ ser 
-]

Original: =

Original: λ

Original: [-I , (cid:98) θ m, 1 = 0 , b loc m, 
-]
Revised: [-I , (cid:98) θ m, 1 = 0 , b loc m, 
-]

Original: = 0 and Σ loc m, 0 = 0 for all m ∈

Original: [ M ] 2: for k = 1 , 2 , . .

Original: , K do 3:

Original: Participation set P k ⊆

Original: [ M ] of arbitrary order 4: for each active agent m ∈ P k do 5:

Original: Receive D m,k from the environment 6:

Original: Select x m,k ← argmax x ∈ D m,k ⟨ (cid:98) θ m,k , x k ⟩ + β ∥ x ∥ Σ − 1 m,k /* Optimistic decision */ 7:

Original: Receive r m,k from environment 8: Σ loc m,k ← Σ loc m,k − 1 + x m,k x ⊤ m,k , b loc m,k ← b loc m,k − 1 + r m,k x m,k /*Local update*/ 9: if det( Σ m,k + Σ loc m,k ) > (1 + α ) det( Σ m,k ) then 10:

Original: Agent m sends Σ loc m,k and b loc m,k to server /* Upload */ 11: Σ ser k ← Σ ser k + Σ loc m,k , b ser k ← b ser k + b loc m,k /

Original: Global update */ 12: Σ loc m,k ← 0 , b loc m,k ←13:

Original: Server sends Σ ser k and b ser k back to agent m /* Download */ 14: Σ m,k +1 ← Σ ser k , b m,k ← b ser k 15: (cid:98) θ m,k +1 ← Σ − 1 m,k +1 b m,k + /* Compute estimate */ 16: else 17: Σ m,k +1 ← Σ m,k , b m,k +1 ← b m,k , (cid:98) θ m,k +1 ← (cid:98) θ m,k 18: end if 19: end for 20: for other inactive agents m ∈

Original: [ M ] \ P k do 21: Σ m,k +1 ← Σ m,k , b m,k +1 ← b m,k , (cid:98) θ m,k +1 ← (cid:98) θ m,k 22: end for 23: end for et al.

Original: (2011)) cannot be directly used for the server’s data.

Original: This is why we need a more dedicated analysis to control this biased error (see Lemma 6.6 for more details).

Original: A.2 An Alternative Form of Algorithm

Original: We introduce an alternative form of Algorithm 1, which is displayed in Algorithm 2.

Original: Algorithmcan be viewed as the episodic 7 version of Algorithm 1, and its form aligns with those of the existing algorithms for federated linear bandits (Wang et al., 2019; Dubey and Pentland, 2020; Huang et al., 2021; Li and Wang, 2022a).

Original: Specifically, in Algorithm 2, for each round (episode) k ∈ [ K ] , the set of active agents is given by P k , where the order of agents in P k can be arbitrary (Line 3).

Original: Then the agents in the set P k participate according to the prefixed order (Line 4).

Original: The operations in the inner loop of Algorithm(i.e.

Original: , decision rule, upload/download, local/global update, and model estimates) are all identical to those in Algorithm 1.

Original: Therefore, Algorithm 2 is indeed equivalent to Algorithm 1 up to relabeling of the participation of the agents, and hence all the theoretical results for Algorithm 1 also hold for

Original: Algorithm 2.

Original: B Experiments

Original: In this section, we provide the remaining details on numerical simulations.

Original: Experiment setup.

Original: We construct two linear bandit instances with dimension d = 25 .

Original: In the first instance, the true model parameter θ ∗ is [1 / √ d, .., 1 / √ d ] ∈ R d .

Original: In the second instance, the true model parameter θ ∗ is generated by uniform random sampling over the space [ − 1 / √ d, 1 / √ d ] d with normalization.

Original: For each round t ∈ [ T ] , the active agent m t is uniform sampled from all M agents

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: et al., 2011) with parameter θ ∗ =

Original: [1 / √ d, .., 1 / √ d ] ∈ R d .

Original: Experiments are run for M = 15 and M = 30 , and results are averaged over 20 runs.

Original: Figures 3(a) and 3(b) present the cumulative regret; Figures 3(c) and 3(d) show the averaged regret (in log scale);

Original: Figures 3(e) and 3(f) compare the communication cost versus number of rounds.

Original: Note that the communication cost of OFUL with full communication is linear with the number of rounds T and is far greater than those of both FedLinUCB and Async-LinUCB.

Original: Therefore, in order to make a clearer comparison between the communication cost of FedLinUCB and Async-LinUCB, OFUL is omitted in Figure 3(e) and 3(f).

Original: and the decision set D t consists of 25 different actions uniformly randomly sampled from the space [ − 1 / √ d, 1 / √ d ] d .

Original: After the active agent m t chooses an action x t , the agent m t receives a reward given by r t = ⟨ x t , θ ∗ ⟩ + η t , where η t is a 0.3-Gaussian noise.

Original: We run simulation on the above linear bandit instance with the total number of rounds T = 30000 (repeating 20 times and taking the average) and the number of agents is set to be 15 or 30.

Original: We implement our FedLinUCB algorithm and compare its performance with Async-LinUCB (Li and Wang, 2022a) and OFUL Abbasi-Yadkori et al.

Original: (2011) with full communication (i.e., the active agent communicates with the server in each

Original: [Figure]

Original: mm un i c a t i o n c o s t

Original: [Figure]

Original: ) with random select θ ∗ .

Original: Experiments are run for

Original: M = 15 and M = 30 , and results are averaged over 20 runs.

Original: Figures 4(a) and 4(b) present the cumulative regret; Figures 4(c) and 4(d) show the averaged regret (in log scale); Figures 4(e) and 4(f) compare the communication cost versus number of rounds.

Original: Note that the communication cost of

Original: OFUL with full communication is linear with the number of rounds T and is far greater than those of both FedLinUCB and Async-LinUCB.

Original: Therefore, in order to make a clearer comparison between the communication cost of FedLinUCB and Async-LinUCB, OFUL is omitted in Figure 4(e) and 4(f).

Original: round).

Original: We set the parameter α = 1 for FedLinUCB and γ U = γ D = 5 for Async-LinUCB to ensure that the communication costs of FedLinUCB and Async-LinUCB have similar magnitudes.

Original: Results.

Original: The results are presented in Figure 3 and 4, suggesting that our algorithm significantly outperforms Async-LinUCB as our algorithm achieves smaller regret while spending less communication cost.

Original: Specifically, Figure 3(a) and 3(b) displays the cumulative regret of our FedLinUCB algorithm, AsyncLinUCB (Li and Wang, 2022a) and OFUL (Abbasi-Yadkori et al., 2011) with full communication.

Original: It can be apparently seen that FedLinUCB outperforms Async-LinUCB in terms of regret.

Original: Next,

Original: Figure 3(c) and 3(d) show the average regret in log scale.

Original: These two plots show that the average regret of our algorithm has a rate very close to the optimal rate of OFUL.

Original: Finally, we plot communication cost versus number of rounds in Figure 3(e) and 3(f), which indicates that the communication cost of FedLinUCB is significantly lower than that of Async-LinUCB.

Original: Together with Figure 3(a) and 3(b), we see that our algorithm achieves lower regret with lower communication cost, compared to

Original: Async-LinUCB.

Original: The experiment for Figure 4 adopts a different θ ∗ (which is randomly selected sampled the space [ − 1 / √ d, 1 / √ d ] d with normalization) from that of Figure 3.

Original: Figure 4 shows almost the same results as Figure 3, indicating our algorithms works in general.

Original: Overall, the simulation corroborates our theoretical results.

Original: It also shows that our algorithm indeed outperforms Async-LinUCB (Li and Wang, 2022a).

Original: For all the experiments, the results are averaged over 20 runs with the error bars chosen as the empirical one standard deviation.

Original: All experiments are conducted on a Macbook with 8-core CPU and 16 GB of memory.

Original: C Missing Proofs in Section

Original: Here we present the proof of the results in Section 6.

Original: C.1 Communication complexity within each epoch

Original: We first present the proof for the bound on the communication complexity within each epoch given in

Original: Lemma 6.2.

Original: Proof of Lemma 6.2.

Original: For each agent m ∈

Original: [ M ] , let n m be the number of communications agent m has made during this epoch, and we denote the communication rounds as t 1 , . .

Original: t n m for simplicity.

Original: Now we consider the data uploaded to the server, and it can be denoted by the value of covariance matrix Σ loc m,t j before communicating with the server.

Original: For each j = 2 , . . . , n m , according to the determinant-based criterion (Line 9) in Algorithm 1, we have

Original: [Equation]

Original: which further implies that

Original: [Equation]

Original: where the inequality holds due to Lemma E.2 together with the fact that the communication in round t 1 updates the covariance matrix so that Σ m,t j ⪰ Σ ser T i .

Original: In addition, we define the sequence of all communications from T i to T i +1 − 1 as t ′ 1 , . . .

Original: , t ′ L .

Original: For each round t ′ j , if the agent m t ′

Original: j have already communicated with the server earlier in this epoch, we have

Original: [Equation]

Original: where the first inequality holds due to Lemma E.1 together with the fact that Σ ser t ′ j − 1 ⪰ Σ ser T i , and the second inequality follows from (C.1).

Original: Now, taking the sum of (C.2) over all round t ′ j , we obtain

Original: [Equation]

Original: Since det( Σ ser ,T i +1 − 1 ) ≤ 2 det( Σ ser ,T i ) , we further have

Original: [Equation]

Original: Each communication between one agent and the server includes one upload and one download, so the communication complexity within one epoch is bounded by 2( M + 1 /α ) .

Original: This finishes the proof.

Original: C.2 Proof for the covariance comparison

Original: Next, we prove the comparison between the covariance matrices given in Lemma 6.5.

Original: Proof of Lemma 6.5.

Original: Fix any round t ∈

Original: [ T ] .

Original: Let t 1 ≤ t be the last round such that agent m is active at round t 1 .

Original: If agent m communicated with the server at this round, then we have

Original: [Equation]

Original: Otherwise, according to determinant-based criterion (Line 9) in Algorithm 1, at the end of each round t 1 , we have

Original: [Equation]

Original: By Lemma E.4, for any non-zero vector x ∈ R d , we have

Original: [Equation]

Original: Rearranging the above yields x ⊤ Σ loc m,t 1 x ≤ α x ⊤ Σ m,t 1 x , which then implies that

Original: [Equation]
Revised: [N/A]

Original: Note that Σ m,t 1 is the downloaded covariance matrix from last communication before round t 1 , so it must satisfy Σ m,t 1 ⪯ Σ ser t 1 .

Original: Therefore, we have

Original: [Equation]

Original: Now, for round t , since agent m is inactive from round t 1 to t , then we have

Original: [Equation]

Original: which yields the first claim in Lemma 6.5.

Original: Next, suppose agent m is the only active agent from round t 1 to t 2 − 1 and agent m only communicates with the server at round t 1 .

Original: Further average the above inequality over all agents m ∈

Original: [ M ] , and we get

Original: [Equation]

Original: which implies that for t 1 + 1 ≤ t ≤ t 2 − 1 , we have

Original: [Equation]

Original: [-where the second equation holds due to the fact that no agent communicate with server from round t 
-]
Revised: [-where the second equation holds due to the fact that no agent communicate with server from round t 
-]

Original: + 1 to t 2 − 1 ,

Original: and the inequality follows from (C.3).

Original: This yields the second claim in Lemma 6.

Original: and finishes the proof. C.3 Proof of the local concentration for agents Recall that the global concentration and corresponding global confidence bound have been shown in Lemma 6.3.
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: Next, we establish the concentration properties of the local data on the agents’ side.

Original: Proof of Lemma 6.4.

Original: For each agent m ∈

Original: [ M ] and any rounds 1 ≤ t 1 ≤ t 2 ≤ T , consider Σ m,t 1 ,t 2 =

Original: [Equation]

Original: I + t 2 (cid:88) i =

Original: t 1 +1 ,m i

Original: = m x i x ⊤ i and u m,t 1 ,t 2 = t 2 (cid:88) i

Original: =

Original: t 1 +1 ,m i

Original: = m x i η i .

Original: By Theorem 2 in Abbasi-Yadkori et al.

Original: (2011), with probability at least 1 − δ/

Original: T 2 M ) , we have

Original: [Equation]

Original: m,t 1 ,t

Original: [Equation]

Original: m,

Original: [Equation]

Original: [Equation]

Original: Then taking an union bound over all agent m ∈

Original: [ M ] and rounds 1 ≤ t 1 ≤ t 2 ≤ T and applying to t 1 = N m ( t ) and t 2 = t for each t ∈

Original: [ T ] , we obtain the desired concentration.

Original: This finishes the proof.

Original: For clarity, we break Lemma 6.6 into two lemmas, Lemma C.1 for local confidence bound and

Original: Lemma C.2 for per-round regret, and prove them separately.

Original: Lemma C.1 (Local confidence bound) .

Original: Under the setting of Theorem 5.1, with probability at least 1 − δ , for each t ∈ [ T ] , the estimate (cid:98) θ m,t +1 satisfies that ∥ θ ∗ − (cid:98) θ m,t +1 ∥ Σ m,t +1 ≤ β .

Original: Proof of Lemma C.1.

Original: Since the estimated vector (cid:98) θ m,t +1 and covariance matrix Σ m,t +1 will keep the same value as in the previous round if the agent m do not communicate with the server, we only need to consider for those round t where agent m communicates with the server.

Original: By the determinant-based criterion (Line 9) in Algorithm 1, if the agent m communicates with the server in round t , then at the end of this round, the covariance matrix Σ m,t +1 and vector b m,t +1 are given by Σ

Original: [Equation]

Original: Therefore, the estimated vector (cid:98) θ m,t +1 is (cid:98) θ m

Original: [Equation]

Original: t +1 = (cid:18)

Original: [Equation]

Original: u

Original: [Equation]

Original: Thus, the difference between (cid:98) θ m,t +1 and the underlying truth θ ∗ can be decomposed as

Original: [Equation]

Original: where the first inequality holds due to that fact that ∥ a + b ∥ Σ ≤ ∥ a ∥ Σ + ∥ b ∥ Σ and the second inequality follows from Σ m,t +1 ≥ λ I .

Original: By the assumption that ∥ θ ∗ ∥ 2 ≤ S , the first term can be controlled by √ λS .

Original: For the second term in (C.5), consider the following decomposition:

Original: M

Original: [Equation]

Original: −

Original: [Equation]

Original: m

Original: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125) B m

Original: [Equation]

Original: (C

Original: [Equation]

Original: [Equation]

Original: For the term A , it follows from (6.5) in Lemma 6.5 that

Original: [Equation]
Revised: [N/A]

Original: where the second inequality holds due to Lemma 6.3.

Original: Next, for each term B m ′ in (C.6), by (6.4) in

Original: Lemma 6.5, we have

Original: [Equation]

Original: which further implies that

Original: [Equation]

Original: Thus

Original: [Equation]

Original: the norm of each term B m ′ can be bounded as

Original: [Equation]

Original: where the first inequality holds due to (C.8) and the second inequality follows from Lemma 6.4.

Original: Finally, combining (C.5), (C.6), (C.7) and (C.9), we obtain

Original: [Equation]

Original: Thus we finish the proof of Lemma C.1.

Original: Lemma C.2 (Per-round regret) .

Original: Under the setting of Theorem 5.1, with probability at least 1 − δ , for each t ∈ [ T ] , the regret in round t satisfies

Original: [Equation]

Original: Proof of Lemma C.2.

Original: First, by Lemma C.1, with probability at least 1 − δ , for each round t ∈ [ T ] and each action x ∈ D t , we have

Original: [Equation]

Original: where the first inequality holds due to the Cauchy-Schwarz inequality and the last inequality follows from Lemma C.1.

Original: (C.10) shows that the estimator for agent m t is always optimistic.

Original: For simplicity, we denote the optimal action at round t as x ∗ = arg max x ∈D t ( θ ∗ ) ⊤ x , and (C.10) further implies

Original: [Equation]

Original: where the first inequality holds due to (C.10), the second inequality follows from the definition of action x t in Algorithm 1, the third inequality applies the Cauchy-Schwarz inequality, and the last inequality is by Lemma C.1.

Original: Thus, we finish the proof of Lemma C.2.

Original: Combining Lemmas C.1 and C.2 yields Lemma 6.

Original: D Proof for Lower Bound

Original: Lemma D.1 (Theorem 3 in Abbasi-Yadkori et al. 2011) .

Original: There exists a constant C > 0 , such that for any normalized linear bandit instance with R =

Original: L = S

Original: = 1 , the expectation of the regret for OFUL algorithm is upper bounded by E [ Regret ( T )] ≤ Cd √ T log T .

Original: Lemma D.2 (Theorem 24.1 in Lattimore and Szepesvári 2020) .

Original: There exists a set of hard-to-learn normalized linear bandit instances with R =

Original: L = S

Original: = 1 , such that for any algorithm Alg and T ≥ d , for a uniformly random instance in the set, the regret is lower bounded by E [ Regret ( T )] ≥ cd √ T for some constant c > 0 .

Original: Theorem 5.3 is an extension of the lower bound result in Wang et al.

Original: (2019, Theorem 2) from multi-arm bandits to linear bandits.

Original: Proof of Theorem 5.3.

Original: For any algorithm Alg for federated bandits, we construct the auxiliary Alg as follows:

Original: For each agent m ∈ [ M ] , it performs Alg until there is a communication between the agent m and the server (upload or download data).

Original: After the communication, the agent m remove all previous information and perform the OFUL Algorithm in Abbasi-Yadkori et al.

Original: In this case, for each agent m ∈

Original: [ M ] , Alg1 do not utilize any information from other agents and it will reduce to a single agent bandit algorithm.

Original: Now, we uniformly randomly select a hard-to-learn instance from the set given in Lemma D.2, and let each agent m ∈

Original: [ M ] be active for T/M different rounds (where we assume T/M is an integer for simplicity).

Original: Since Alg1 reduces to a single agent bandit algorithm, Lemma D.2 implies that the expected regret for agent m with Alg1 is lower bounded by

Original: [Equation]

Original: Taking the sum of (D.1) over all agents m ∈

Original: [ M ] , we obtain

Original: [Equation]

Original: For each agent m ∈ [ M ] , let δ m denote the probability that agent m will communicate with the server.

Original: Notice that before the communication, Alg1 has the same performance as Alg , while for the rounds after the communication, Alg1 executes the OFUL algorithm and Lemma D.1 suggests an

Original: O ( d (cid:112) T/M log( T/M )) upper bounded for the expected regret.

Original: Therefore, the expected regret for agent m with Alg1 is upper bounded by

Original: [Equation]

Original: Taking the sum of (D.3) over all agents m ∈

Original: [ M ] , we obtain

Original: E [ Regret

Original: [Equation]

Original: T

Original: [Equation]

Original: ] =

Original: [Equation]

Original: where δ = (cid:80)

Original: Mm =1 δ m is the expected communication complexity.

Original: Combining (D.2) and (D.4), for any algorithm Alg with communication complexity δ ≤ c/ (2 C ) · M/ log( T/M ) =

Original: O ( M/ log( T/M )) , we have

Original: [Equation]

Original: This finishes the proof of Theorem 5.3.

Original: E Auxiliary Lemmas

Original: To make the analysis self-contained in this paper, here we include the auxiliary lemmas that have been previously used.

Original: Lemma E.1 (Lemma 2.2 in Tie et al. 2011) .

Original: For any positive semi-definite matrices A , B and C , it holds that det( A + B + C ) + det( A ) ≥ det( A + B ) + det( A + C ) .

Original: Lemma E.2 (Lemma 2.3 in Tie et al. 2011) .

Original: For any positive semi-definite matrices A , B and C , it holds that det( A + B + C ) det( A ) ≤ det( A + B ) det( A + C ) .

Original: Theorem E.3 (Theorem 2 in Abbasi-Yadkori et al. 2011) .

Original: Suppose {F t } ∞ t

Original: =

Original: 0 is a filtration.

Original: Let { η t } R t

Original: =1 be a stochastic process in R such that η t is F t -measurable and R -sub-Gaussian conditioning on F t − 1 , i.e, for any c > 0 ,

Original: [Equation]

Original: Let { x t } ∞ t =1 be a stochastic process in R d such that x t is F t − 1 -measurable and ∥ x t ∥ 2 ≤ L .

Original: Let y t

Original: =

Original: ⟨ x t , θ ∗ ⟩ + η t for some θ ∗ ∈ R d s.t.

Original: ∥ θ ∗ ∥ 2 ≤ S .

Original: For any t ≥ 1 , define Σ t =

Original: [Equation]

Original: I + t (cid:88) i

Original: = x

Original: t x ⊤ t , and (cid:98) θ

Original: [Equation]

Original: t (cid:88)

Original: i

Original: [Equation]

Original: x

Original: [Equation]

Original: y i

Original: [Equation]

Original: for some λ > 0 .

Original: Then for any δ > 0 , with probability at least 1 − δ , for all t , we have

Original: [Equation]

Original: Lemma E.4 (Lemma 12 in Abbasi-Yadkori et al.

Original: Suppose A , B ∈ R d × d are two positive definite matrices satisfying that A ⪰ B , then for any x ∈ R d ,

Original: ∥ x ∥

Original: A ≤ ∥ x ∥ B · (cid:112) det( A ) / det( B ) .