Original: Federated Continual Learning with Weighted Inter-client Transfer
Revised: Federated Continual Learning with Weighted Inter-client Transfer

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: There has been a surge of interest in continual learning and federated learning, both of which are important in deep neural networks in realworld scenarios.
Revised: There has been a surge of interest in continual learning and federated learning, both of which are important in deep neural networks in realworld scenarios.

Original: Yet little research has been done regarding the scenario where each client learns on a sequence of tasks from private local data stream.
Revised: Yet little research has been done regarding the scenario where each client learns on a sequence of tasks from private local data stream.

Original: This problem of federated continual learning poses new challenges to continual learning, such as utilizing knowledge from other clients, while preventing interference from irrelevant knowledge.
Revised: This problem of federated continual learning poses new challenges to continual learning, such as utilizing knowledge from other clients, while preventing interference from irrelevant knowledge.

Original: To resolve these issues, we propose a novel federated continual learning framework, Weighted Inter-client Transfer (FedWeIT) , which decomposes the network weights into global federated parameters and sparse taskspeciﬁc parameters, and each client receives selective knowledge from other clients by taking a weighted combination of their task-speciﬁc parameters.
Revised: To resolve these issues, we propose a novel federated continual learning framework, Weighted Inter-client Transfer (FedWeIT) , which decomposes the network weights into global federated parameters and sparse taskspeciﬁc parameters, and each client receives selective knowledge from other clients by taking a weighted combination of their task-speciﬁc parameters.

Original: FedWeIT minimizes interference between incompatible tasks, and also allows positive knowledge transfer across clients during learning.
Revised: FedWeIT minimizes interference between incompatible tasks, and also allows positive knowledge transfer across clients during learning.

Original: We validate our FedWeIT against existing federated learning and continual learning methods under varying degree of task similarity across clients, and our model signiﬁcantly outperforms them with large reduction in the communication cost.
Revised: We validate our FedWeIT against existing federated learning and continual learning methods under varying degree of task similarity across clients, and our model signiﬁcantly outperforms them with large reduction in the communication cost.

Original: 1. Introduction
Revised: 1. Introduction

Original: Continual learning (1; 2; 3; 4; 5) describes a learning scenario where a model continuously trains on a sequence of tasks; it is inspired by the human learning process, as a person learns to perform numerous tasks with large diversity over his/her lifespan, making use of the past knowledge to learn about new tasks without forgetting previously learned ones.Continual learning is a long-studied topic since having , Anonymous Region, Anonymous Country.
Revised: Continual learning (1; 2; 3; 4; 5) describes a learning scenario where a model continuously trains on a sequence of tasks; it is inspired by the human learning process, as a person learns to perform numerous tasks with large diversity over his/her lifespan, making use of the past knowledge to learn about new tasks without forgetting previously learned ones.

Original: Correspondence to: Anonymous Author anon.email@ >. [Table] such an ability leads to the potential of building a general artiﬁcial intelligence.
Revised: Continual learning is a long-studied topic since having such an ability leads to the potential of building a general artiﬁcial intelligence.

Original: However, there are crucial challenges in implementing it with conventional models such as deep neural networks (DNNs), such as catastrophic forgetting , which describes the problem where parameters or semantic representations learned for the past tasks drift to the direction of new tasks during training.
Revised: However, there are crucial challenges in implementing it with conventional models such as deep neural networks (DNNs), such as catastrophic forgetting , which describes the problem where parameters or semantic representations learned for the past tasks drift to the direction of new tasks during training.

Original: The problem has been tackled by various prior work (4; 6; 7; 8).
Revised: The problem has been tackled by various prior work (4; 6; 7; 8).

Original: More recent works tackle other issues, such as scalability or order-robustness (5; 9).
Revised: More recent works tackle other issues, such as scalability or order-robustness (5; 9).

Original: However, all of these models are fundamentally limited in that the models can only learn from its direct experiencethey only learn from the sequence of the tasks they have trained on.
Revised: However, all of these models are fundamentally limited in that the models can only learn from its direct experiencethey only learn from the sequence of the tasks they have trained on.

Original: Contrarily, humans can learn from indirect experience from others, through different means (e.g. verbal communications, books, or various media).
Revised: Contrarily, humans can learn from indirect experience from others, through different means (e.g. verbal communications, books, or various media).

Original: Then wouldn’t it be beneﬁcial to implement such an ability to a continual learning framework, such that multiple models learning on different machines can learn from the knowledge of the tasks that have been already experienced by other clients?
Revised: Then wouldn’t it be beneﬁcial to implement such an ability to a continual learning framework, such that multiple models learning on different machines can learn from the knowledge of the tasks that have been already experienced by other clients?

Original: One problem that arises here, is that due to data privacy and communication cost, it may not be possible to communicate data directly between the clients or between the server and clients.
Revised: One problem that arises here, is that due to data privacy and communication cost, it may not be possible to communicate data directly between the clients or between the server and clients.

Original: Federated learning (10; 11; 12; 13) is a learning paradigm that tackles this issue by communicating the parameters instead of the raw data itself.
Revised: Federated learning (10; 11; 12; 13) is a learning paradigm that tackles this issue by communicating the parameters instead of the raw data itself.

Original: We may have a server that receives the parameters locally trained on multiple clients, aggregates it into a single model parameter, and sends it back to the clients.
Revised: We may have a server that receives the parameters locally trained on multiple clients, aggregates it into a single model parameter, and sends it back to the clients.

Original: Motivated by our intuition on learning from indirect experience, we tackle the problem of Federated Continual Learning (FCL) where we perform continual learning with multiple clients trained on private task sequences, which communicate their task-speciﬁc parameters via a global server.
Revised: Motivated by our intuition on learning from indirect experience, we tackle the problem of Federated Continual Learning (FCL) where we perform continual learning with multiple clients trained on private task sequences, which communicate their task-speciﬁc parameters via a global server.

Original: Yet, the problem of federated continual learning also brings new challenges.
Revised: Yet, the problem of federated continual learning also brings new challenges.

Original: First, there is not only the catastrophic forgetting from continual learning, but also the threat of potential interference from other clients .
Revised: First, there is not only the catastrophic forgetting from continual learning, but also the threat of potential interference from other clients .

Original: Figure 1 (a) describes this challenge with the results of a simple experiment.
Revised: Figure 1 (a) describes this challenge with the results of a simple experiment.

Original: Here, we train a model for MNIST digit recognition while communicating the parameters from another client trained on a different dataset.
Revised: Here, we train a model for MNIST digit recognition while communicating the parameters from another client trained on a different dataset.

Original: When the knowledge transferred from the other client is relevant to the target task (SVHN), the model starts with high accuracy, converge faster and reachhigher accuracy ( green line ), whereas the model underper-
Revised: When the knowledge transferred from the other client is relevant to the target task (SVHN), the model starts with high accuracy, converge faster and reach

Original: [Figure]
Revised: [Figure]

Original: forms the base model if the transferred knowledge is from a task highly different from the target task (CIFAR-10, red line ).
Revised: higher accuracy ( green line ), whereas the model underperforms the base model if the transferred knowledge is from a task highly different from the target task (CIFAR-10, red line ).

Original: Thus, we need to selective utilize knowledge from other clients to minimize the inter-client interference and maximize inter-client knowledge transfer .
Revised: Thus, we need to selective utilize knowledge from other clients to minimize the inter-client interference and maximize inter-client knowledge transfer .

Original: Another problem with the federated learning is efﬁcient communication, as communication cost could become excessively large when utilizing the knowledge of the other clients, since the communication cost could be the main bottleneck in practical scenarios when working with edge devices.
Revised: Another problem with the federated learning is efﬁcient communication, as communication cost could become excessively large when utilizing the knowledge of the other clients, since the communication cost could be the main bottleneck in practical scenarios when working with edge devices.

Original: Thus we want the knowledge to be represented as compactly as possible.
Revised: Thus we want the knowledge to be represented as compactly as possible.

Original: To tackle these challenges, we propose a novel framework for federated continual learning, Federated Weighted Interclient Transfer (FedWeIT) , which decomposes the model parameters into a dense global parameter and sparse taskadaptive parameters.
Revised: To tackle these challenges, we propose a novel framework for federated continual learning, Federated Weighted Interclient Transfer (FedWeIT) , which decomposes the model parameters into a dense global parameter and sparse taskadaptive parameters.

Original: As illustrated in Figure 1 (b), FedWeIT reduces the interference between different tasks since the global parameters ( θ G ) will encode task-generic knowledge, while the task-speciﬁc knowledge will be encoded into the task-adaptive parameters ( A ( t ) c ).
Revised: As illustrated in Figure 1 (b), FedWeIT reduces the interference between different tasks since the global parameters ( θ G ) will encode task-generic knowledge, while the task-speciﬁc knowledge will be encoded into the task-adaptive parameters ( A ( t ) c ).

Original: When we utilize the generic knowledge, we also want the client to selectively utilize task-speciﬁc knowledge obtained at other clients.
Revised: When we utilize the generic knowledge, we also want the client to selectively utilize task-speciﬁc knowledge obtained at other clients.

Original: To this end, we allow each model to take a weighted combination of the task-adaptive parameters broadcast from the server, such that it can select task-speciﬁc knowledge helpful for the task at hand.
Revised: To this end, we allow each model to take a weighted combination of the task-adaptive parameters broadcast from the server, such that it can select task-speciﬁc knowledge helpful for the task at hand.

Original: FedWeIT is communication-efﬁcient, since the task-adaptive parameters are highly sparse and only need to be communicated once when created.
Revised: FedWeIT is communication-efﬁcient, since the task-adaptive parameters are highly sparse and only need to be communicated once when created.

Original: We also perform selective transmission of the parameters to further reduce communication cost.
Revised: We also perform selective transmission of the parameters to further reduce communication cost.

Original: We validate our method on multiple different scenarios with varying degree of task similarity across clients against various federated learning and local continual learning models.
Revised: We validate our method on multiple different scenarios with varying degree of task similarity across clients against various federated learning and local continual learning models.

Original: The results show that our model obtains signiﬁcantly superior performance over all baselines, adapts faster to new tasks, with largely reduced communication cost.
Revised: The results show that our model obtains signiﬁcantly superior performance over all baselines, adapts faster to new tasks, with largely reduced communication cost.

Original: The main contributions of this paper are as follows:
Revised: The main contributions of this paper are as follows:

Original: • We introduce a new problem of Federated Continual Learning (FCL) , where multiple models continuously learn on distributed clients, which poses new challenges such as prevention of inter-client interference and inter-client knowledge transfer.
Revised: • We introduce a new problem of Federated Continual Learning (FCL) , where multiple models continuously learn on distributed clients, which poses new challenges such as prevention of inter-client interference and inter-client knowledge transfer.

Original: • We propose a novel framework for federated continual learning , which allows each client to adaptively update the federated parameter and utilize the past knowledge from other clients, by communicating sparse parameters.
Revised: • We propose a novel framework for federated continual learning , which allows each client to adaptively update the federated parameter and utilize the past knowledge from other clients, by communicating sparse parameters.

Original: • We validate our model under FCL setting with both Overlapped and non-IID task sequences, on which it largely outperforms existing federated learning and local continual learning approaches with signiﬁcantly reduced communication cost .
Revised: • We validate our model under FCL setting with both Overlapped and non-IID task sequences, on which it largely outperforms existing federated learning and local continual learning approaches with signiﬁcantly reduced communication cost .

Original: 2. Related Work
Revised: 2. Related Work

Original: Continual Learning While continual learning (1; 2; 3) is a long-studied topic with a vast literature, we only discuss recent relevant works.
Revised: Continual Learning While continual learning (1; 2; 3) is a long-studied topic with a vast literature, we only discuss recent relevant works.

Original: Regularization-based approaches:
Revised: Regularization-based approaches:

Original: A popular approach for continual learning is to use regularizations that prevent catastrophic forgetting.
Revised: A popular approach for continual learning is to use regularizations that prevent catastrophic forgetting.

Original: EWC (4) leverages Fisher Information Matrix to restrict the change of the model parameters such that the model ﬁnds solution that is good for both previous and the current task, and IMM (6) proposes to learn the posterior distribution for multiple tasks as a mixture of Gaussians.
Revised: EWC (4) leverages Fisher Information Matrix to restrict the change of the model parameters such that the model ﬁnds solution that is good for both previous and the current task, and IMM (6) proposes to learn the posterior distribution for multiple tasks as a mixture of Gaussians.

Original: Architecture-based approaches: PGN (14) progressively expands the networks with ﬁxed number of neurons/ﬁlters at each layer.
Revised: Architecture-based approaches: PGN (14) progressively expands the networks with ﬁxed number of neurons/ﬁlters at each layer.

Original: DEN (15) tackles this issue by expanding the networks size with minimum number of neurons/ﬁlters that are necessary via iterative neuron/ﬁlter pruning and splitting, and RCL (16) tackles the same problem using reinforcement learning.
Revised: DEN (15) tackles this issue by expanding the networks size with minimum number of neurons/ﬁlters that are necessary via iterative neuron/ﬁlter pruning and splitting, and RCL (16) tackles the same problem using reinforcement learning.

Original: APD (9) additively decompose the parameters into shared and taskspeciﬁc parameters to minimize the increase in the network complexity.
Revised: APD (9) additively decompose the parameters into shared and taskspeciﬁc parameters to minimize the increase in the network complexity.

Original: Coreset-based approaches: VCL (17) performs online variational inference by continuously training the model while approximating the likelihood for the coreset, and GEM variants (18; 19) minimize the loss on both of actual dataset and stored episodic memory.
Revised: Coreset-based approaches: VCL (17) performs online variational inference by continuously training the model while approximating the likelihood for the coreset, and GEM variants (18; 19) minimize the loss on both of actual dataset and stored episodic memory.

Original: FRCL (20) memorizes approximated posteriors of previous tasks with sophisticatedly constructed inducing points.
Revised: FRCL (20) memorizes approximated posteriors of previous tasks with sophisticatedly constructed inducing points.

Original: To the best of our knowledge, none of the existing approaches considered the communicability for continual learning of deep neural networks, which we tackle.
Revised: To the best of our knowledge, none of the existing approaches considered the communicability for continual learning of deep neural networks, which we tackle.

Original: CoLLA (21) aims at solving multi-agent lifelong learning with sparse dictionary learning, but it is not applicable to federated learning or continual deep learning.
Revised: CoLLA (21) aims at solving multi-agent lifelong learning with sparse dictionary learning, but it is not applicable to federated learning or continual deep learning.

Original: Federated learning Federated Learning is a distributed learning framework under differential privacy, which aims to learn a global model on a server while aggregating the parameters learned at the clients on their private data.
Revised: Federated learning Federated Learning is a distributed learning framework under differential privacy, which aims to learn a global model on a server while aggregating the parameters learned at the clients on their private data.

Original: There are diverse approaches to aggregate the local models.
Revised: There are diverse approaches to aggregate the local models.

Original: FedAvg (10) aggregates the model trained across multiple clients by computing a weighted average of them based on the number of data points trained.
Revised: FedAvg (10) aggregates the model trained across multiple clients by computing a weighted average of them based on the number of data points trained.

Original: TWAFL (11) and
Revised: TWAFL (11) and

Original: ASO-fed (22) follow weighted averaging in FedAvg while assigning larger weights to newer parameters by leveraging timestamps.
Revised: ASO-fed (22) follow weighted averaging in FedAvg while assigning larger weights to newer parameters by leveraging timestamps.

Original: FedProx (12) trains the local models with a proximal term which restricts their updates to be close to the global model.
Revised: FedProx (12) trains the local models with a proximal term which restricts their updates to be close to the global model.

Original: FedCurv (23) aims to minimize the model disparity across clients during federated learning by adopting a modiﬁed version of EWC (4).
Revised: FedCurv (23) aims to minimize the model disparity across clients during federated learning by adopting a modiﬁed version of EWC (4).

Original: Recent works (13; 24) introduce well-designed aggregation policies by leveraging Bayesian non-parametric methods.
Revised: Recent works (13; 24) introduce well-designed aggregation policies by leveraging Bayesian non-parametric methods.

Original: Another crucial challenge is the reduction of communication cost, as communicating the full network weights may be too costly.
Revised: Another crucial challenge is the reduction of communication cost, as communicating the full network weights may be too costly.

Original: TWAFL (11) tackles this problem by performing layer-wise parameter aggregation, where some layers(i.e.
Revised: TWAFL (11) tackles this problem by performing layer-wise parameter aggregation, where some layers(i.e.

Original: shallow layers) are aggregated at every step, but other layers(i.e.
Revised: shallow layers) are aggregated at every step, but other layers(i.e.

Original: deep layers) are aggregated in the last few steps of a loop.
Revised: deep layers) are aggregated in the last few steps of a loop.

Original: Our method also solves the problem of efﬁcient communication by performing weighted inter-client knowledge transfer.
Revised: Our method also solves the problem of efﬁcient communication by performing weighted inter-client knowledge transfer.

Original: 3. Federated Continual Learning with Weighted Inter-client Transfer
Revised: 3. Federated Continual Learning with Weighted Inter-client Transfer

Original: Motivated by the human learning process from indirect experiences, we introduce a novel continual learning under federated learning setting, which we refer to as Federated Continual Learning (FCL) .
Revised: Motivated by the human learning process from indirect experiences, we introduce a novel continual learning under federated learning setting, which we refer to as Federated Continual Learning (FCL) .

Original: FCL assumes that multiple clients are trained on a sequence of tasks from private data stream, while communicating the learned parameters with a global server.
Revised: FCL assumes that multiple clients are trained on a sequence of tasks from private data stream, while communicating the learned parameters with a global server.

Original: We ﬁrst formally deﬁne the problem in Section 3.1, and then propose naive solutions that straightforwardly combine the existing federated learning and continual learning methods in Section 3.2.
Revised: We ﬁrst formally deﬁne the problem in Section 3.1, and then propose naive solutions that straightforwardly combine the existing federated learning and continual learning methods in Section 3.2.

Original: Then, following Section 3.3 and 3.4, we discuss about two novel challenges that are introduced by federated continual learning, and propose a novel framework, Federated Weighted Inter-client Transfer (FedWeIT) which can effectively handle the two problems while also reducing the client-to-sever communication cost.
Revised: Then, following Section 3.3 and 3.4, we discuss about two novel challenges that are introduced by federated continual learning, and propose a novel framework, Federated Weighted Inter-client Transfer (FedWeIT) which can effectively handle the two problems while also reducing the client-to-sever communication cost.

Original: 3.1. Problem Deﬁnition
Revised: 3.1. Problem Deﬁnition

Original: In the standard continual learning (on a single machine), the model iteratively learns from a sequence of tasks {T (1) , T (2) , ..., T ( T ) } where T ( t ) is a labeled dataset of t th task, T ( t ) =
Revised: In the standard continual learning (on a single machine), the model iteratively learns from a sequence of tasks {T (1) , T (2) , ..., T ( T ) } where T ( t ) is a labeled dataset of t th task, T ( t ) =

Original: { x ( t ) i , y ( t ) i } N t
Revised: { x ( t ) i , y ( t ) i } N t

Original: i =1 , which consists of N t pairs of instances x ( t ) i and their corresponding labels y ( t ) i .
Revised: i =1 , which consists of N t pairs of instances x ( t ) i and their corresponding labels y ( t ) i .

Original: Assuming the most realistic situation, we consider the case where the task sequence is a task stream with an unknown arriving order, such that the model is allowed to access T ( t ) only at the training period of task t which becomes inaccessible afterwards.
Revised: Assuming the most realistic situation, we consider the case where the task sequence is a task stream with an unknown arriving order, such that the model is allowed to access T ( t ) only at the training period of task t which becomes inaccessible afterwards.

Original: Given T ( t ) and the model learned so far, the learning objective at task t is as follows: [Equation] T ( t ) ) , where θ ( t ) ∈ R N × M is a set of the parameters in the model at task t .
Revised: Given T ( t ) and the model learned so far, the learning objective at task t is as follows: minimize θ ( t ) L ( θ ( t ) ; θ ( t − 1) , T ( t ) ) , where θ ( t ) ∈ R N × M is a set of the parameters in the model at task t .

Original: We now extend the conventional continual learning to the federated learning setting with multiple clients and a global server.
Revised: We now extend the conventional continual learning to the federated learning setting with multiple clients and a global server.

Original: Let us assume that we have C clients, where at each client c c ∈ { c 1 , . .
Revised: Let us assume that we have C clients, where at each client c c ∈ { c 1 , . .

Original: . , c C } trains a model on a privately accessible sequence of tasks {T (1) c , T (2) c , ..., T ( t ) c } ⊆ T .
Revised: . , c C } trains a model on a privately accessible sequence of tasks {T (1) c , T (2) c , ..., T ( t ) c } ⊆ T .

Original: Now the goal is to effectively train C continual learning models on their own private task streams, via communicating the model parameters with the global server, which aggregates the parameters sent from each client, and redistributes them to clients.
Revised: Now the goal is to effectively train C continual learning models on their own private task streams, via communicating the model parameters with the global server, which aggregates the parameters sent from each client, and redistributes them to clients.

Original: 3.2. Communicable Continual Learning
Revised: 3.2. Communicable Continual Learning

Original: In conventional federated learning settings, the learning is done with multiple rounds of local learning and parameter aggregation.
Revised: In conventional federated learning settings, the learning is done with multiple rounds of local learning and parameter aggregation.

Original: At each round of communication r , each client c c and the server s perform the following two procedures: local parameter transmission and parameter aggregation & broadcasting .
Revised: At each round of communication r , each client c c and the server s perform the following two procedures: local parameter transmission and parameter aggregation & broadcasting .

Original: In the local parameter transmission step, for a randomly selected subset of clients at round r ,
Revised: In the local parameter transmission step, for a randomly selected subset of clients at round r ,

Original: C ( r ) ⊆ { c 1 , c 2 , ..., c C } , each client c c sends updated parameters θ ( r ) to the server.
Revised: C ( r ) ⊆ { c 1 , c 2 , ..., c C } , each client c c sends updated parameters θ ( r ) to the server.

Original: The server-clients transmission is not done at every client because some of the clients may be temporarily disconnected.
Revised: The server-clients transmission is not done at every client because some of the clients may be temporarily disconnected.

Original: Then the server aggregates the parameters θ ( r ) c sent from the clients into a single parameter.
Revised: Then the server aggregates the parameters θ ( r ) c sent from the clients into a single parameter.

Original: The most popular frameworks for this aggregation are FedAvg (10) and FedProx (12).
Revised: The most popular frameworks for this aggregation are FedAvg (10) and FedProx (12).

Original: However, naive federated continual learning with these two algorithms on local sequences of tasks may result in catastrophic forgetting.
Revised: However, naive federated continual learning with these two algorithms on local sequences of tasks may result in catastrophic forgetting.

Original: One simple solution is to use a regularization-based, such as Elastic Weight Consolidation (EWC) (4), which allows the model to obtain a solution that is optimal for both the previous and the current tasks.
Revised: One simple solution is to use a regularization-based, such as Elastic Weight Consolidation (EWC) (4), which allows the model to obtain a solution that is optimal for both the previous and the current tasks.

Original: There exist other advanced solutions (14; 15; 16; 17; 19) that successfully prevents catastrophic forgetting.
Revised: There exist other advanced solutions (14; 15; 16; 17; 19) that successfully prevents catastrophic forgetting.

Original: However, the prevention of catastrophic forgetting at the client level is an orthogonal problem from [Figure]
Revised: However, the prevention of catastrophic forgetting at the client level is an orthogonal problem from federated learning.

Original: 𝑻𝑻 𝟏𝟏 ( 𝒕𝒕−𝟏𝟏 ) 𝑻𝑻 𝟏𝟏 ( 𝒕𝒕 ) …
Revised: Client 𝒄𝒄𝑻𝑻 𝟏𝟏 ( 𝒕𝒕−𝟏𝟏 ) 𝑻𝑻 𝟏𝟏 ( 𝒕𝒕 ) …

Original: 𝑩𝑩 𝟏𝟏 ( 𝒕𝒕 )
Revised: 𝑩𝑩 𝟏𝟏 ( 𝒕𝒕 )

Original: Client [Figure]
Revised: Client 𝒄𝒄𝑻𝑻 𝟐𝟐 ( 𝒕𝒕−𝟏𝟏 ) 𝑻𝑻 𝟐𝟐 ( 𝒕𝒕 ) …

Original: 𝑻𝑻 𝟐𝟐 ( 𝒕𝒕−𝟏𝟏 ) 𝑻𝑻 𝟐𝟐 ( 𝒕𝒕 )…
Revised: 𝑩𝑩 𝟐𝟐 ( 𝒕𝒕 ) 𝜽𝜽 𝑮𝑮 ( 𝒕𝒕−𝟏𝟏 )

Original: 𝑩𝑩
Revised: CentralServer 𝒎𝒎 𝟏𝟏 ( 𝒕𝒕 ) 𝒎𝒎 𝟐𝟐 ( 𝒕𝒕 ) 𝜮𝜮 ( ) C Client 𝒄𝒄𝑻𝑻 𝟏𝟏 ( 𝒕𝒕−𝟏𝟏 ) 𝑻𝑻 𝟏𝟏 ( 𝒕𝒕 ) … 𝑨𝑨 𝟏𝟏 ( 𝒕𝒕 ) Client 𝒄𝒄𝑻𝑻 𝟐𝟐 ( 𝒕𝒕−𝟏𝟏 ) 𝑻𝑻 𝟐𝟐 ( 𝒕𝒕 ) … 𝑨𝑨 2 ( 𝑡𝑡 )

Original: [Figure]
Revised: CentralServer

Original: 𝒕𝒕 ) [Figure]
Revised: Knowledge Base 𝑨𝑨 𝒄𝒄𝒕𝒕−𝟏𝟏 𝒄𝒄=𝟏𝟏 𝑪𝑪 +

Original: [Figure]
Revised: [Figure]

Revised: client

Original: [Figure]
Revised: [Figure]

Original: Thus we focus on challenges that newly arise in this federated continual learning setting.
Revised: Thus we focus on challenges that newly arise in this federated continual learning setting.

Original: In the federated continual learning framework, the aggregation of the parameters into a global parameter θ G allows inter-client knowledge transfer across clients, since a task T ( q ) i learned at client c i at round q may be similar or related to T ( r ) j learned at client c j at round r .
Revised: In the federated continual learning framework, the aggregation of the parameters into a global parameter θ G allows inter-client knowledge transfer across clients, since a task T ( q ) i learned at client c i at round q may be similar or related to T ( r ) j learned at client c j at round r .

Original: Yet, using a single aggregated parameter θ G may be suboptimal in achieving this goal since knowledge from irrelevant tasks may not to be useful or even hinder the training at each client by altering its parameters into incorrect directions, which we describe as inter-client interference .
Revised: Yet, using a single aggregated parameter θ G may be suboptimal in achieving this goal since knowledge from irrelevant tasks may not to be useful or even hinder the training at each client by altering its parameters into incorrect directions, which we describe as inter-client interference .

Original: Another problem that is also practically important, is the communication-efﬁciency .
Revised: Another problem that is also practically important, is the communication-efﬁciency .

Original: Both the parameter transmission from the client to the server, and server to client will incur large communication cost, which will be problematic for the continual learning setting, since the clients may train on possibly unlimited streams of tasks.
Revised: Both the parameter transmission from the client to the server, and server to client will incur large communication cost, which will be problematic for the continual learning setting, since the clients may train on possibly unlimited streams of tasks.

Original: 3.3. Federated Weighted Inter-client Transfer
Revised: 3.3. Federated Weighted Inter-client Transfer

Original: How can we then maximize the knowledge transfer between clients while minimizing the inter-client interference , and communication cost?
Revised: How can we then maximize the knowledge transfer between clients while minimizing the inter-client interference , and communication cost?

Original: We now describe our model, Federated Weighted Inter-client Transfer (FedWeIT) , which can resolve the these two problems that arise with a naive combination of continual learning approaches with federated learning framework.
Revised: We now describe our model, Federated Weighted Inter-client Transfer (FedWeIT) , which can resolve the these two problems that arise with a naive combination of continual learning approaches with federated learning framework.

Original: The main cause of the problems, as brieﬂy alluded to earlier, is that the knowledge of all tasks learned at multiple clients is stored into a single set of parameters θ G .
Revised: The main cause of the problems, as brieﬂy alluded to earlier, is that the knowledge of all tasks learned at multiple clients is stored into a single set of parameters θ G .

Original: However, for the knowledge transfer to be effective, each client should selectively utilize only the knowledge of the relevant tasks that is trained at other clients.
Revised: However, for the knowledge transfer to be effective, each client should selectively utilize only the knowledge of the relevant tasks that is trained at other clients.

Original: This selective transfer is also the key to minimize the inter-client interference as well as it will disregard the knowledge of irrelevant tasks that may interfere with learning.
Revised: This selective transfer is also the key to minimize the inter-client interference as well as it will disregard the knowledge of irrelevant tasks that may interfere with learning.

Original: We tackle this problem by decomposing the parameters, into three different types of the parameters with different roles: global parameters ( θ G ) that capture the global and generic knowledge across all clients, local base parameters ( B ) which capture generic knowledge for each client, and taskadaptive parameters ( A ) for each speciﬁc task per client.
Revised: We tackle this problem by decomposing the parameters, into three different types of the parameters with different roles: global parameters ( θ G ) that capture the global and generic knowledge across all clients, local base parameters ( B ) which capture generic knowledge for each client, and taskadaptive parameters ( A ) for each speciﬁc task per client.

Original: This decomposition scheme is motivated by (9).
Revised: This decomposition scheme is motivated by (9).

Original: A set of the model parameters θ ( t ) c for task t at continual learning client c c is then deﬁned as follows:
Revised: A set of the model parameters θ ( t ) c for task t at continual learning client c c is then deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: 1) where B ( t ) c ∈ R N × M is the set of base parameters for c th client shared across all tasks in the client, m ( t ) c ∈ R M is a sparse mask which allows to adaptively transform B ( t ) c for the target task, A ( t ) c ∈ R N × M is a sparse matrix of task-adaptive parameters for the task t at client c c .
Revised: where B ( t ) c ∈ R N × M is the set of base parameters for c th client shared across all tasks in the client, m ( t ) c ∈ R M is a sparse mask which allows to adaptively transform B ( t ) c for the target task, A ( t ) c ∈ R N × M is a sparse matrix of task-adaptive parameters for the task t at client c c .

Original: The ﬁrst term allows selective utilization of the global knowledge.
Revised: The ﬁrst term allows selective utilization of the global knowledge.

Original: We want the base parameter B ( t ) c at each client to capture generic knowledge across all tasks across all clients.
Revised: We want the base parameter B ( t ) c at each client to capture generic knowledge across all tasks across all clients.

Original: In Figure 2 (a), we initialize it at each round t with the global parameter from the previous iteration, θ ( t − 1) G which aggregates the parameters sent from the client.
Revised: In Figure 2 (a), we initialize it at each round t with the global parameter from the previous iteration, θ ( t − 1) G which aggregates the parameters sent from the client.

Original: This allows
Revised: This allows

Original: B ( t ) c to also beneﬁt from the global knowledge about all the tasks.
Revised: B ( t ) c to also beneﬁt from the global knowledge about all the tasks.

Original: However, since θ ( t − 1) G also contains knowledge irrelevant to the current task, instead of using it as is, we learn the sparse mask m ( t ) c to select only the relevant parameters for the given task.
Revised: However, since θ ( t − 1) G also contains knowledge irrelevant to the current task, instead of using it as is, we learn the sparse mask m ( t ) c to select only the relevant parameters for the given task.

Original: This sparse parameter selection helps minimize inter-client interference, and also allows for efﬁcient communication.
Revised: This sparse parameter selection helps minimize inter-client interference, and also allows for efﬁcient communication.

Original: The second term is the taskadaptive parameters
Revised: The second term is the taskadaptive parameters

Original: A ( t ) c .
Revised: A ( t ) c .

Original: Since we additively decompose the parameters, this will learn to capture knowledge about the task that is not captured by the ﬁrst term, and thus will capture speciﬁc knowledge about the task T ( t ) c .
Revised: Since we additively decompose the parameters, this will learn to capture knowledge about the task that is not captured by the ﬁrst term, and thus will capture speciﬁc knowledge about the task T ( t ) c .

Original: The ﬁnal term describes inter-client knowledge transfer.
Revised: The ﬁnal term describes inter-client knowledge transfer.

Original: We have aAlgorithm input Dataset {D (1: t ) c } Cc =1 , and Global Parameter θ (0) G output { B c , m (1: t ) c , α (1: t ) c , A (1: t ) c } Cc = 1: Initialize B c to θ (0) G for all c ∈ C ≡ { 1 , ..., C } 2: for task t = 1 , 2 , ... do 3: for round r = 1 , 2 , ..., R do 4:
Revised: We have a input Dataset {D (1: t ) c } Cc =1 , and Global Parameter θ (0) G output { B c , m (1: t ) c , α (1: t ) c , A (1: t ) c } Cc = 1: Initialize B c to θ (0) G for all c ∈ C ≡ { 1 , ..., C } 2: for task t = 1 , 2 , ... do 3: for round r = 1 , 2 , ..., R do 4:

Original: Select communicable clients C ( r ) ⊆
Revised: Select communicable clients C ( r ) ⊆

Original: C 5:
Revised: C 5:

Original: A ( t − 1 ,R ) c ∈C ( r ) and ˆ B ( t,r ) c ∈C ( r ) are transferred from C ( r ) to the central server 6:
Revised: A ( t − 1 ,R ) c ∈C ( r ) and ˆ B ( t,r ) c ∈C ( r ) are transferred from C ( r ) to the central server 6:

Original: Compute θ ( t,r ) G ← 1 |C ( r ) | (cid:80) c ∈C ( r ) ˆ B ( t,r ) c 7:
Revised: Compute θ ( t,r ) G ← 1 |C ( r ) | (cid:80) c ∈C ( r ) ˆ B ( t,r ) c 7:

Original: Distribute θ ( t,r ) G and { A ( t − 1 ,R ) j } j ∈C ( r ) to client c ∈ C ( r ) 8:
Revised: Distribute θ ( t,r ) G and { A ( t − 1 ,R ) j } j ∈C ( r ) to client c ∈ C ( r ) 8:

Original: Minimize Eq.
Revised: Minimize Eq.

Original: (2) for solving each local CL problems 9: end for 10: end for set of parameters that are transmitted from the server, which contain all task-adaptive parameters from all the clients.
Revised: (2) for solving each local CL problems 9: end for 10: end for set of parameters that are transmitted from the server, which contain all task-adaptive parameters from all the clients.

Original: To selectively utilizes these indirect experiences from other clients, we further allocate attention α ( t ) c on these parameters, to take a weighted combination of them.
Revised: To selectively utilizes these indirect experiences from other clients, we further allocate attention α ( t ) c on these parameters, to take a weighted combination of them.

Original: By learning this attention, each client can select only the relevant taskadaptive parameters that help learn the given task.
Revised: By learning this attention, each client can select only the relevant taskadaptive parameters that help learn the given task.

Original: To reduce a burden of the parameter communication, we send previous task-adaptive parameters at each ﬁrst training round per task t .
Revised: To reduce a burden of the parameter communication, we send previous task-adaptive parameters at each ﬁrst training round per task t .

Original: Training.
Revised: Training.

Original: We learn the decomposable parameter θ ( t ) c by optimizing for the following objective: 
Revised: We learn the decomposable parameter θ ( t ) c by optimizing for the following objective: minimize

Original: [Equation]
Revised: [Equation]

Original: where L is a loss function and Ω( · ) is a sparsity-inducing regularization term for the task adaptive parameter and the masking variable (we use (cid:96) 1 -norm regularization), to make them sparse.
Revised: where L is a loss function and Ω( · ) is a sparsity-inducing regularization term for the task adaptive parameter and the masking variable (we use (cid:96) 1 -norm regularization), to make them sparse.

Original: The ﬁnal regularization term is used for retroactive update of the past task-adaptive parameters, which helps the task-adaptive parameters to maintain the original solutions for the target tasks, by reﬂecting the change of the base parameter.
Revised: The ﬁnal regularization term is used for retroactive update of the past task-adaptive parameters, which helps the task-adaptive parameters to maintain the original solutions for the target tasks, by reﬂecting the change of the base parameter.

Original: Here, ∆ B ( t ) c
Revised: Here, ∆ B ( t ) c

Original: [Equation] is the difference between the base parameter at the current and previous timestep, and ∆ A ( i ) c is the difference between the task-adaptive parameter for task i at the current and previous timestep.
Revised: = B ( t ) c − B ( t − 1) c is the difference between the base parameter at the current and previous timestep, and ∆ A ( i ) c is the difference between the task-adaptive parameter for task i at the current and previous timestep.

Original: This regularization is essential for preventing catastrophic forgetting.
Revised: This regularization is essential for preventing catastrophic forgetting.

Original: λ 1 and λ 2 are hyperparameters controlling the effect of the two regularizers.
Revised: λ 1 and λ 2 are hyperparameters controlling the effect of the two regularizers.

Original: 3.4. Efﬁcient Communication via Sparse Parameters
Revised: 3.4. Efﬁcient Communication via Sparse Parameters

Original: FedWeIT learns via server-to-client communication.
Revised: FedWeIT learns via server-to-client communication.

Original: As discussed earlier, a crucial challenge here is to reduce thewe distribute them multiple communication cost.
Revised: As discussed earlier, a crucial challenge here is to reduce the communication cost.

Original: We describe what happens at the client and the server at each step.
Revised: We describe what happens at the client and the server at each step.

Original: Client:
Revised: Client:

Original: At each round r , each client c c updates its base parameter with the nonzero components of the global parameter sent from the server; that is, B c ( i )
Revised: At each round r , each client c c updates its base parameter with the nonzero components of the global parameter sent from the server; that is, B c ( i )

Original: =
Revised: =

Original: θ G ( i ) where i is a nonzero element of the global parameter.
Revised: θ G ( i ) where i is a nonzero element of the global parameter.

Original: After training the model using Eq.
Revised: After training the model using Eq.

Original: (2), it obtains a sparsiﬁed base parameter ˆ B ( t ) c
Revised: (2), it obtains a sparsiﬁed base parameter ˆ B ( t ) c

Original: = B ( t ) c (cid:12) m ( t ) c and the newly learned task-adaptive parameter A ( t ) c .
Revised: = B ( t ) c (cid:12) m ( t ) c and the newly learned task-adaptive parameter A ( t ) c .

Original: Then, the client sends both ˆ B ( t ) c and A ( t ) c to the server.
Revised: Then, the client sends both ˆ B ( t ) c and A ( t ) c to the server.

Original: Since both parameters are highly sparse, this results in the large reduction of the client-to-server communication cost.
Revised: Since both parameters are highly sparse, this results in the large reduction of the client-to-server communication cost.

Original: Server: The server ﬁrst aggregates the sparsiﬁed base parameters sent from all the clients by taking an weighted average of them: θ ( t ) G = 1 C (cid:80) c ˆ B ( t ) c .
Revised: Server: The server ﬁrst aggregates the sparsiﬁed base parameters sent from all the clients by taking an weighted average of them: θ ( t ) G = 1 C (cid:80) c ˆ B ( t ) c .

Original: Then, it broadcasts θ ( t ) G along with all task adaptive parameters { A ( t ) c } Cc =1 to all the clients.
Revised: Then, it broadcasts θ ( t ) G along with all task adaptive parameters { A ( t ) c } Cc =1 to all the clients.

Original: Algorithm 1 describes our FedWeIT algorithm.
Revised: Algorithm 1 describes our FedWeIT algorithm.

Original: 4. Experiments
Revised: 4. Experiments

Original: We validate our FedWeIT under different conﬁgurations of task sequences against baselines which are namely
Revised: We validate our FedWeIT under different conﬁgurations of task sequences against baselines which are namely

Original: Overlapped-CIFAR-100 and NonIID-50. 1) OverlappedCIFAR-100 : We group 100 classes of CIFAR-100 dataset into 20 non-iid superclasses tasks.
Revised: Overlapped-CIFAR-100 and NonIID-50. 1) OverlappedCIFAR-100 : We group 100 classes of CIFAR-100 dataset into 20 non-iid superclasses tasks.

Original: Then, we randomly sample 10 tasks out of 20 tasks and split instances to create a task sequence for each of the clients with overlapping tasks.
Revised: Then, we randomly sample 10 tasks out of 20 tasks and split instances to create a task sequence for each of the clients with overlapping tasks.

Original: NonIID-50 : We use the following eight benchmark datasets:
Revised: NonIID-50 : We use the following eight benchmark datasets:

Original: MNIST (25), CIFAR-10/-100 (26), SVHN (27), FashionMNIST (28), Not-MNIST (29), FaceScrub (30), and TrafﬁcSigns (31).
Revised: MNIST (25), CIFAR-10/-100 (26), SVHN (27), FashionMNIST (28), Not-MNIST (29), FaceScrub (30), and TrafﬁcSigns (31).

Original: We split the classes in the 8 datasets intonon-IID tasks, each of which is composed of 5 classes that are disjoint from the classes used for the other tasks.
Revised: We split the classes in the 8 datasets intonon-IID tasks, each of which is composed of 5 classes that are disjoint from the classes used for the other tasks.

Original: This is a large-scale experiment, since we use 280 , 000 images of 293 classes from 8 heterogeneous datasets.
Revised: This is a large-scale experiment, since we use 280 , 000 images of 293 classes from 8 heterogeneous datasets.

Original: After generating and processing tasks, we randomly distribute them to multiple clients as illustrated in Figure 3.
Revised: After generating and processing tasks, we randomly distribute them to multiple clients as illustrated in Figure 3.

Original: Experimental setup We use a modiﬁed version of
Revised: Experimental setup We use a modiﬁed version of

Original: LeNet (25) for the experiments with both Overlapped-
Revised: LeNet (25) for the experiments with both Overlapped-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: CIFAR-100 and NonIID-50 dataset.
Revised: CIFAR-100 and NonIID-50 dataset.

Original: Further, we use ResNet18 (32) with NonIID-50 dataset.
Revised: Further, we use ResNet18 (32) with NonIID-50 dataset.

Original: We followed other experimental setups from (33) and (9).
Revised: We followed other experimental setups from (33) and (9).

Original: We use an Adam optimizer with adaptive learning rate decay, which decays learning rate by a factor of 3 for every 5 epochs that validation loss does not consecutively decrease.
Revised: We use an Adam optimizer with adaptive learning rate decay, which decays learning rate by a factor of 3 for every 5 epochs that validation loss does not consecutively decrease.

Original: For LeNet with 5 clients, we initialize by 1 e − 3 × 13 at the beginning of each new task.
Revised: For LeNet with 5 clients, we initialize by 1 e − 3 × 13 at the beginning of each new task.

Original: Mini-batch size is 100 , the rounds per task is 20 , an the epoch per round is 1 .
Revised: Mini-batch size is 100 , the rounds per task is 20 , an the epoch per round is 1 .

Original: The setting for ResNet-18 is identical excluding initial learning rate, 1 e − 4 .
Revised: The setting for ResNet-18 is identical excluding initial learning rate, 1 e − 4 .

Original: In the case of experiments with 20 and 100 clients, we set the same settings except reducing minibatch size from 100 to 10 with an initial learning rate 1 e − 4 and exploring client fraction 0 .
Revised: In the case of experiments with 20 and 100 clients, we set the same settings except reducing minibatch size from 100 to 10 with an initial learning rate 1 e − 4 and exploring client fraction 0 .

Original: 25 and 0 .
Revised: 25 and 0 .

Original: 05 , respectively.
Revised: 05 , respectively.

Original: we set λ 1 =
Revised: we set λ 1 =

Original: [1 e − 1 , 4 e − 1 ] and λ 2 =for all experiments.
Revised: [1 e − 1 , 4 e − 1 ] and λ 2 =for all experiments.

Original: Further, we use µ = 5 e − 3 for FedProx, λ = [1 e − 2 , 1 . 0] for EWC and FedCurv.
Revised: Further, we use µ = 5 e − 3 for FedProx, λ = [1 e − 2 , 1 . 0] for EWC and FedCurv.

Original: Baselines and our models 1) Local-EWC :
Revised: Baselines and our models 1) Local-EWC :

Original: Continual learning with EWC (4).
Revised: Continual learning with EWC (4).

Original: 2) Local-APD :
Revised: 2) Local-APD :

Original: Continual learning with APD (9).
Revised: Continual learning with APD (9).

Original: 3) FedCurv : FCL with FedCurv algorithm (23) which reduces the parameter disparity across clients using Fisher Information Matrix.
Revised: 3) FedCurv : FCL with FedCurv algorithm (23) which reduces the parameter disparity across clients using Fisher Information Matrix.

Original: 4) FedCurv-EWC :
Revised: 4) FedCurv-EWC :

Original: Fedcurv algorithm with EWC.
Revised: Fedcurv algorithm with EWC.

Original: 5) Fed-EWC :
Revised: 5) Fed-EWC :

Original: Federated continual learning, that is trained using FedProx (12) algorithm with EWC.
Revised: Federated continual learning, that is trained using FedProx (12) algorithm with EWC.

Original: 6) Fed-APD :
Revised: 6) Fed-APD :

Original: Federated continual learning with
Revised: Federated continual learning with

Original: APD using FedProx algorithm.
Revised: APD using FedProx algorithm.

Original: 7) FedWeIT :
Revised: 7) FedWeIT :

Original: Our federated weighted inter-client transfer algorithm.
Revised: Our federated weighted inter-client transfer algorithm.

Original: 4.1. Experimental Results
Revised: 4.1. Experimental Results

Original: We ﬁrst validate our model on both Overlapped-CIFARand NonIID-50 task sequences against single task learning (STL), continual learning (EWC, APD), federated learning (FedCurv), and naive federated continual learning (FedCurv, FedProx-based) baselines.
Revised: We ﬁrst validate our model on both Overlapped-CIFARand NonIID-50 task sequences against single task learning (STL), continual learning (EWC, APD), federated learning (FedCurv), and naive federated continual learning (FedCurv, FedProx-based) baselines.

Original: Table 1 shows the ﬁnal average per-task performance after the completion of (federated) continual learning on both datasets.
Revised: Table 1 shows the ﬁnal average per-task performance after the completion of (federated) continual learning on both datasets.

Original: We observe that FedProx-based federated continual learning (FCL) approaches degenerate the performance of continual learning (CL) methods over the same methods without federated learning.
Revised: We observe that FedProx-based federated continual learning (FCL) approaches degenerate the performance of continual learning (CL) methods over the same methods without federated learning.

Original: This is because the aggregation of all client parameters that are learned on irrelevant tasks results in severe interference in the learning for each task, which leads to catastrophic forgetting and suboptimal task adaptation.
Revised: This is because the aggregation of all client parameters that are learned on irrelevant tasks results in severe interference in the learning for each task, which leads to catastrophic forgetting and suboptimal task adaptation.

Original: While FedCurv reduces inter-task disparity in parameters, it cannot minimize inter-task interference, which results it to underperform single-machine CL methods.
Revised: While FedCurv reduces inter-task disparity in parameters, it cannot minimize inter-task interference, which results it to underperform single-machine CL methods.

Original: On the other hand, FedWeIT signiﬁcantly outperforms both singlemachine CL baselines and naive FCL baselines on both datasets.
Revised: On the other hand, FedWeIT signiﬁcantly outperforms both singlemachine CL baselines and naive FCL baselines on both datasets.

Original: Even with larger number of clients ( C = 20 ,
Revised: Even with larger number of clients ( C = 20 ,

Original: C = 100 ), FedWeIT consistently outperforms Local-APD (Table 2).
Revised: C = 100 ), FedWeIT consistently outperforms Local-APD (Table 2).

Original: This improvement largely owes to FedWeIT’s ability to selectively utilize the knowledge from other clients to rapidly adapt to the target task, and obtain better ﬁnal performance (Figure 4).
Revised: This improvement largely owes to FedWeIT’s ability to selectively utilize the knowledge from other clients to rapidly adapt to the target task, and obtain better ﬁnal performance (Figure 4).

Original: The fast adaptation to new task
Revised: The fast adaptation to new task

Original: [Figure]

Original: [Figure]

Original: Local-EWC (4)

Original: [Figure]
Revised: [Figure]

Original: Local-APD (9)

Original: [Figure]
Revised: [Figure]

Original: FedCurv (23)

Original: [Figure]

Original: 0.191676 FedCurv

Original: [Table]

Original: [Table]
Revised: [Table]

Original: is another clear advantage of inter-client knowledge transfer.
Revised: is another clear advantage of inter-client knowledge transfer.

Original: To further demonstrate the practicality of our method with larger networks [Table] we experiment on Non-IID dtaset with
Revised: To further demonstrate the practicality of our method with larger networks, we experiment on Non-IID dtaset with

Original: ResNet-18 ( Table 3), on which FedWeIT still signiﬁcantly outperforms the strongest baseline (FedProx-APD) while using fewer parameters.
Revised: ResNet-18 ( Table 3), on which FedWeIT still signiﬁcantly outperforms the strongest baseline (FedProx-APD) while using fewer parameters.

Original: Efﬁciency of FedWeIT We also report the accuracy as a function of network capacity in Table 1, 3, which we measure by the number of parameters used.
Revised: Efﬁciency of FedWeIT We also report the accuracy as a function of network capacity in Table 1, 3, which we measure by the number of parameters used.

Original: We observe that FedWeIT obtains much higher accuracy while utilizing less number of parameters compared to FedProx-APD.
Revised: We observe that FedWeIT obtains much higher accuracy while utilizing less number of parameters compared to FedProx-APD.

Original: This efﬁciency mainly comes from the reuse of task-adaptive parameters from other clients, which is not possible with single-machine CL methods or naive FCL methods.
Revised: This efﬁciency mainly comes from the reuse of task-adaptive parameters from other clients, which is not possible with single-machine CL methods or naive FCL methods.

Original: We also examine the communication cost of each method.
Revised: We also examine the communication cost of each method.

Original: Table 1 reports the client-to-server communication cost (C2S Cost) and, Figure 5 (a) shows the accuracy as function of communication cost.
Revised: Table 1 reports the client-to-server communication cost (C2S Cost) and, Figure 5 (a) shows the accuracy as function of communication cost.

Original: We observe that FedWeIT is signiﬁcantly more communication-efﬁcient than naive FCL baselines although it broadcasts task-adaptive parameters, due to high sparsity of the parameters.
Revised: We observe that FedWeIT is signiﬁcantly more communication-efﬁcient than naive FCL baselines although it broadcasts task-adaptive parameters, due to high sparsity of the parameters.

Original: Ablation study We perform an ablation study to analyze the role of each component of our FedWeIT.
Revised: Ablation study We perform an ablation study to analyze the role of each component of our FedWeIT.

Original: We compare the performance of four different variations of our model.
Revised: We compare the performance of four different variations of our model.

Original: w/o B communication describes the model that does not transfer the base parameter B and only communicates taskadaptive ones.
Revised: w/o B communication describes the model that does not transfer the base parameter B and only communicates taskadaptive ones.

Original: w/o A communication is the model that does not communicate task-adaptive parameters.
Revised: w/o A communication is the model that does not communicate task-adaptive parameters.

Original: w/o A is the model which trains the model only with sparse transmission of local base parameter, and w/o m is the model without the sparse vector mask.
Revised: w/o A is the model which trains the model only with sparse transmission of local base parameter, and w/o m is the model without the sparse vector mask.

Original: As shown in Table 4, without communicating B or A , the model yields signiﬁcantly lower performance compared to the full model since they do not beneﬁt from inter-client knowledge transfer .
Revised: As shown in Table 4, without communicating B or A , the model yields signiﬁcantly lower performance compared to the full model since they do not beneﬁt from inter-client knowledge transfer .

Original: The model w/o A obtains very low performance due to catastrophic forgetting, and the model w/o sparse mask m achieves lower accuracy with larger capacity and cost, which demonstrates the importance of performing selective transmission.
Revised: The model w/o A obtains very low performance due to catastrophic forgetting, and the model w/o sparse mask m achieves lower accuracy with larger capacity and cost, which demonstrates the importance of performing selective transmission.

Original: Catastrophic forgetting Further, we examine how the performance of the past tasks change during continual learning, to see the severity of catastrophic forgetting with each method.
Revised: Catastrophic forgetting Further, we examine how the performance of the past tasks change during continual learning, to see the severity of catastrophic forgetting with each method.

Original: Figure 6 Left shows the performance of FedWeIT and FCL baselines on the 3 rd , 6 th , and 8 th tasks, at the end of training for later tasks.
Revised: Figure 6 Left shows the performance of FedWeIT and FCL baselines on the 3 rd , 6 th , and 8 th tasks, at the end of training for later tasks.

Original: We observe that naive FCL baselines suffer from more severe catastrophic forgetting than local continual learning with EWC because of the inter- client interference , where the knowledge of irrelevant tasks from other clients overwrites the knowledge of the past tasks.
Revised: We observe that naive FCL baselines suffer from more severe catastrophic forgetting than local continual learning with EWC because of the inter- client interference , where the knowledge of irrelevant tasks from other clients overwrites the knowledge of the past tasks.

Original: Contrarily, our model shows no sign of catastrophic forgetting.
Revised: Contrarily, our model shows no sign of catastrophic forgetting.

Original: This is mainly due to the selective utilization of the prior knowledge learned from other clients through the global/task-adaptive parameters, which allows it to effectively alleviate inter-client interference .
Revised: This is mainly due to the selective utilization of the prior knowledge learned from other clients through the global/task-adaptive parameters, which allows it to effectively alleviate inter-client interference .

Original: FedProx-APD also does not suffer from catastrophic forgetting, but they yield inferior performance due to ineffective knowledge transfer.
Revised: FedProx-APD also does not suffer from catastrophic forgetting, but they yield inferior performance due to ineffective knowledge transfer.

Original: We also report Backward Transfer (BWT) , which is a measure on catastrophic forgetting for all models (more positive the better) and our FedWeIT notably prevents the catastrophic forgetting with a BWT of almost zero.
Revised: We also report Backward Transfer (BWT) , which is a measure on catastrophic forgetting for all models (more positive the better) and our FedWeIT notably prevents the catastrophic forgetting with a BWT of almost zero.

Original: Weighted inter-client knowledge transfer By analyzing the attention α in Eq.
Revised: Weighted inter-client knowledge transfer By analyzing the attention α in Eq.

Original: (1), we examine which task parameters from other clients each client selected.
Revised: (1), we examine which task parameters from other clients each client selected.

Original: Figure 5 (b), shows example of the attention weights that are learned for the 0 th split of MNIST and 10 th split of CIFAR-100 .
Revised: Figure 5 (b), shows example of the attention weights that are learned for the 0 th split of MNIST and 10 th split of CIFAR-100 .

Original: We observe that large attentions are allocated to the task parameters from the same dataset (CIFAR-100 utilizes parameters from CIFAR-100 tasks with disjoint classes), or from a similar dataset (MNIST utilizes parameters from Trafﬁc Sign and SVHN).
Revised: We observe that large attentions are allocated to the task parameters from the same dataset (CIFAR-100 utilizes parameters from CIFAR-100 tasks with disjoint classes), or from a similar dataset (MNIST utilizes parameters from Trafﬁc Sign and SVHN).

Original: This shows that FedWeIT effectively selects beneﬁcial parameters to maximize inter-client knowledge transfer.
Revised: This shows that FedWeIT effectively selects beneﬁcial parameters to maximize inter-client knowledge transfer.

Original: This is an impressive result since it does not know which datasets the parameters are trained on.
Revised: This is an impressive result since it does not know which datasets the parameters are trained on.

Original: 4.2. Effect of the Communication Frequency
Revised: 4.2. Effect of the Communication Frequency

Original: We provide an analysis about the effect of the communication frequency of the model, measured by the number of training epochs per communication round.
Revised: We provide an analysis about the effect of the communication frequency of the model, measured by the number of training epochs per communication round.

Original: We run the 4 different FedWeIT given 1 , 2 , 5 , and 20 training epochs per round.
Revised: We run the 4 different FedWeIT given 1 , 2 , 5 , and 20 training epochs per round.

Original: Table 5 shows the performance of our FedWeIT variants.
Revised: Table 5 shows the performance of our FedWeIT variants.

Original: As clients frequently update the model parameters through the communication with the central server, the model gets higher performance while maintaining smaller network capacity since the model with a frequent communication efﬁciently updates the model parameters as transferring the inter-client knowledge.
Revised: As clients frequently update the model parameters through the communication with the central server, the model gets higher performance while maintaining smaller network capacity since the model with a frequent communication efﬁciently updates the model parameters as transferring the inter-client knowledge.

Original: However, it requires much heavier communication costs than the model with sparser communication.
Revised: However, it requires much heavier communication costs than the model with sparser communication.

Original: For example, the model who trains 1 epochs at each round may need to about 16 .
Revised: For example, the model who trains 1 epochs at each round may need to about 16 .

Original: 9 times larger entire communication cost than the model who trainsepochs at each round.
Revised: 9 times larger entire communication cost than the model who trainsepochs at each round.

Original: Hence, there is a trade-off between model performance of federated continual learning and communication efﬁciency whereas FedWeIT variants consistently outperform (federated) continual learning baselines [Table]
Revised: Hence, there is a trade-off between model performance of federated continual learning and communication efﬁciency whereas FedWeIT variants consistently outperform (federated) continual learning baselines.

Original: 5. Conclusion
Revised: 5. Conclusion

Original: We tackled a novel problem of federated continual learning, whose goal is to continuously learn local models at each
Revised: We tackled a novel problem of federated continual learning, whose goal is to continuously learn local models at each

Original: [Table]
Revised: [Table]

Original: client while allowing it to utilize indirect experience (task knowledge) from other clients.
Revised: client while allowing it to utilize indirect experience (task knowledge) from other clients.

Original: This poses new challenges such as inter-client knowledge transfer and prevention of inter-client interference between irrelevant tasks.
Revised: This poses new challenges such as inter-client knowledge transfer and prevention of inter-client interference between irrelevant tasks.

Original: To tackle these challenges, we additively decomposed the model parameters at each client into the global parameters that are shared across all clients, and sparse local task-adaptive parameters that are speciﬁc to each task.
Revised: To tackle these challenges, we additively decomposed the model parameters at each client into the global parameters that are shared across all clients [Table] and sparse local task-adaptive parameters that are speciﬁc to each task.

Original: Further, we allowed each model to selectively update the global task-shared parameters and selectively utilize the task-adaptive parameters from other clients.
Revised: Further, we allowed each model to selectively update the global task-shared parameters and selectively utilize the task-adaptive parameters from other clients.

Original: The experimental validation of our model under various task similarity across clients, against existing federated learning and continual learning baselines shows that our model obtains signiﬁcantly outperforms baselines with reduced communication cost.
Revised: The experimental validation of our model under various task similarity across clients, against existing federated learning and continual learning baselines shows that our model obtains signiﬁcantly outperforms baselines with reduced communication cost.

Original: We believe that federated continual learning is a practically important topic of large interests to both research communities of continual learning and federated learning, that will lead to new research directions.
Revised: We believe that federated continual learning is a practically important topic of large interests to both research communities of continual learning and federated learning, that will lead to new research directions.