Original: Federated Continual Learning with Differentially Private Data Sharing
Revised: Federated Continual Learning with Differentially Private Data Sharing

Original: Abstract
Revised: Abstract

Original: In Federated Learning (FL) many types of skews can occur, including unevenclass distributions, or varying client participation.
Revised: In Federated Learning (FL) many types of skews can occur, including uneven class distributions, or varying client participation.

Original: In addition, new tasks anddata modalities can be encountered as time passes, which leads us to the problemdomain of Federated Continual Learning (FCL).
Revised: In addition, new tasks and data modalities can be encountered as time passes, which leads us to the problem domain of Federated Continual Learning (FCL).

Original: In this work we study how we can adapt some of the simplest, but often most effective, Continual Learning approaches based on replay to FL.
Revised: In this work we study how we can adapt some of the simplest, but often most effective, Continual Learning approaches based on replay to FL.

Original: We focus on temporalshifts in client behaviour, and show that direct application of replay methods leadsto poor results.
Revised: We focus on temporal shifts in client behaviour, and show that direct application of replay methods leads to poor results.

Original: To address these shortcomings, we explore data sharing betweenclients employing differential privacy.
Revised: To address these shortcomings, we explore data sharing between clients employing differential privacy.

Original: This alleviates the shortcomings in currentbaselines, resulting in performance gains in a wide range of cases, with our methodachieving maximum gains of 49%.
Revised: This alleviates the shortcomings in current baselines, resulting in performance gains in a wide range of cases, with our method achieving maximum gains of 49%.

Original:  Introduction
Revised:  Introduction

Original: The interaction of Federated Learning (FL) and Continual Learning (CL) is a underexplored area.
Revised: The interaction of Federated Learning (FL) and Continual Learning (CL) is a underexplored area.

Original: CL focuses on training a model when the underlying data distribution changes in time.
Revised: CL focuses on training a model when the underlying data distribution changes in time.

Original: The trainedmodel needs to perform well on all previously seen data modalities, despite only having access tothe most recent data distribution.
Revised: The trained model needs to perform well on all previously seen data modalities, despite only having access to the most recent data distribution.

Original: Default training of neural networks in CL leads to catastrophicforgetting , where old data modalities are forgotten as the neural network trains on new tasks.
Revised: Default training of neural networks in CL leads to catastrophic forgetting , where old data modalities are forgotten as the neural network trains on new tasks.

Original: A range of research has emerged to tackle this problem.
Revised: A range of research has emerged to tackle this problem.

Original: The developed methods can be broadlycategorised into replay [1], regularisation [2], or parameter isolation [3] techniques.
Revised: The developed methods can be broadly categorised into replay [1], regularisation [2] or parameter isolation [3] techniques.

Original: Simple replaystrategies can involve storing data to form a replay buffer.
Revised: Simple replay strategies can involve storing data to form a replay buffer.

Original: Despite replay buffer methods often havingrelative simplicity, they can achieve very strong performance [4].
Revised: Despite replay buffer methods often having relative simplicity, they can achieve very strong performance [4].

Original: It is this strategy of replay methodswhich we develop for FL in this paper.
Revised: It is this strategy of replay methods which we develop for FL in this paper.

Original: Performing Federated Continual Learning (FCL) introduces new challenges: different clients canexperience different tasks, or clients join and leave the FL system over time causing valuableinformation only held by certain clients being lost.
Revised: Performing Federated Continual Learning (FCL) introduces new challenges: different clients can experience different tasks, or clients join and leave the FL system over time causing valuable information only held by certain clients being lost.

Original: However, replay methods can be limited in theirimpact in FL with the default inability to share replay buffers across clients due to privacy concerns.
Revised: However, replay methods can be limited in their impact in FL with the default inability to share replay buffers across clients due to privacy concerns.

Original: We focus on the challenge of intermittent client participation in FCL.
Revised: We focus on the challenge of intermittent client participation in FCL.

Original: We tackle the scenario where astime passes clients dynamically join a FCL system, bringing with them current task data.
Revised: We tackle the scenario where as time passes clients dynamically join a FCL system, bringing with them current task data.

Original: However,clients which dynamically join do not have access to data belonging to tasks which occurred prior tothem joining.
Revised: However, clients which dynamically join do not have access to data belonging to tasks which occurred prior to them joining.

Original: We model client participation with a FL system as decaying with time.
Revised: We model client participation with a FL system as decaying with time.

Original: This causesproblems, as data pertaining to old tasks is sampled infrequently causing catastrophic forgetting.
Revised: This causes problems, as data pertaining to old tasks is sampled infrequently causing catastrophic forgetting.

Original: We propose a privacy-performance trade-off where users share differentially private (DP) datastatistics with each other enabling access to a common pool of data containing information relating toprior tasks which provides strong performance boosts.
Revised: We propose a privacy-performance trade-off where users share differentially private (DP) data statistics with each other, enabling access to a common pool of data containing information relating to prior tasks which provides strong performance boosts.

Original: The contributions of this paper are as follows:
Revised: The contributions of this paper are as follows:

Original: • We examine a novel scenario in FCL which considers clients dynamically joining the FL system,and having varying participation rates in training rounds.
Revised: • We examine a novel scenario in FCL which considers clients dynamically joining the FL system, and having varying participation rates in training rounds.

Original: This is a challenging situation, as itcan lead to clients which have valuable information on old tasks communicating updates veryinfrequently.
Revised: This is a challenging situation, as it can lead to clients which have valuable information on old tasks communicating updates very infrequently.

Original: To the author’s knowledge this has not been previously studied.•
Revised: To the author’s knowledge this has not been previously studied.

Original:  We propose extending replay methods from using only a client’s local data, to additionally utilisingshared data between clients with DP noise to preserve privacy.
Revised: • We propose extending replay methods from using only a client’s local data, to additionally utilising shared data between clients with DP noise to preserve privacy.

Original: We perform empirical evaluationsand show how performance varies with differing levels of DP noise and client participation.
Revised: We perform empirical evaluations and show how performance varies with differing levels of DP noise and client participation.

Original:  Background2.1 Continual Learning
Revised:  Background 2.1 Continual Learning

Original: Continual Learning focuses on the scenario where we wish to learn from a continuous stream of data,gradually incorporating new knowledge into a model.
Revised: Continual Learning focuses on the scenario where we wish to learn from a continuous stream of data, gradually incorporating new knowledge into a model.

Original: The core challenge arises from the requirementof only having the current data available for training by default.
Revised: The core challenge arises from the requirement of only having the current data available for training by default.

Original: This causes the phenomenon ofcatastrophic forgetting where performance on old tasks degrades over time as new tasks are added.
Revised: This causes the phenomenon of catastrophic forgetting where performance on old tasks degrades over time as new tasks are added.

Original: There are a wide range of setups that can be investigated in CL with three popular scenarios being [5]:
Revised: There are a wide range of setups that can be investigated in CL with three popular scenarios being [5]:

Original: • Task Incremental: In this case, data is presented in a series of tasks that the model needs to solve, and the task-ID is provided both at training and inference time.
Revised: • Task Incremental: In this case, data is presented in a series of tasks that the model needs to solve, and the task-ID is provided both at training and inference time.

Original: • Domain Incremental: Here the task-ID is provided at training time.
Revised: • Domain Incremental: Here the task-ID is provided at training time.

Original: At test time the model needs tocorrectly classify the data, but  not  infer which task it belonged to.• Class Incremental: The final scenario is when the model needs to both correctly classify data at test time and correctly identify the task.
Revised: At test time the model needs to correctly classify the data, but does not need to infer which task it belonged to. • Class Incremental: The final scenario is when the model needs to both correctly classify data at test time and correctly identify the task.

Original: The task-ID is provided only during training.
Revised: The task-ID is provided only during training.

Original: To handle these problems many strategies have been proposed which can be grouped under threecamps: replay, regularisation [2, 6] or parameter isolation [3, 7].
Revised: To handle these problems many strategies have been proposed which can be grouped under three camps: replay, regularisation [2, 6] or parameter isolation [3, 7].

Original: Most relevant to this work are replaymethods.
Revised: Most relevant to this work are replay methods.

Original: They either store raw samples or generate pseudo-samples.
Revised: They either store raw samples or generate pseudo-samples.

Original: In iCarL [1] class exemplarsare stored to best compute class means in feature space.
Revised: In iCarL [1] class exemplars are stored to best compute class means in feature space.

Original: Alternatively, [4] showed a simple yet robustbaseline of training on old samples yields surprisingly high performance, often beating significantlymore complex methods.
Revised: Alternatively, [4] showed a simple yet robust baseline of training on old samples yields surprisingly high performance, often beating significantly more complex methods.

Original: Lastly, training on GAN generated data has been examined [8].
Revised: Lastly, training on GAN generated data has been examined [8].

Original: However,the compute requirements can be high with multiple generative models needed for CL scenarios.
Revised: However, the compute requirements can be high with multiple generative models needed for CL scenarios.

Original: 2.2 Federated Continual Learning
Revised: 2.2 Federated Continual Learning

Original: Research has begun tackling the challenges of performing FCL [9].
Revised: Research has begun tackling the challenges of performing FCL [9].

Original: Recent work in [10] examineddecomposing neural network weights into global and task-specific parameters.
Revised: Recent work in [10] examined decomposing neural network weights into global and task-specific parameters.

Original: Each client thenadaptively weights contributions from other clients by taking a weighted combination of their taskspecific parameters.
Revised: Each client then adaptively weights contributions from other clients by taking a weighted combination of their taskspecific parameters.

Original: The work of [11] used a federated version of prototypical networks to addresscontinual learning problems.
Revised: The work of [11] used a federated version of prototypical networks to address continual learning problems.

Original: In [12] the focus was do detect distribution shifts in a FCL setting, as illdefined task-IDs presents a fresh set of challenges if they are not well specified at training time.
Revised: In [12] the focus was do detect distribution shifts in a FCL setting, as ill defined task-IDs presents a fresh set of challenges if they are not well specified at training time.

Original: The strategy we propose of privatised data sharing can be used standalone, or employed orthogonallyto augment many of the above strategies.
Revised: The strategy we propose of privatised data sharing can be used standalone, or employed orthogonally to augment many of the above strategies.

Original: This presents a privacy-performance trade-off, howeverwe argue that with many works breaking the privacy of FL, privatised data sharing should not beconsidered prohibited in FL.
Revised: This presents a privacy-performance trade-off, however we argue that with many works breaking the privacy of FL, privatised data sharing should not be considered prohibited in FL.

Original: Here, we continue to exploit the decentralised nature of training and usedifferential privacy to provide a base level of protection to any shared data.
Revised: Here, we continue to exploit the decentralised nature of training and use differential privacy to provide a base level of protection to any shared data.

Original: 2.3 Differential Privacy
Revised: 2.3 Differential Privacy

Original: Differential privacy (DP) has emerged as a popular method to preserve the privacy of sensitive datasets[13].
Revised: Differential privacy (DP) has emerged as a popular method to preserve the privacy of sensitive datasets [13].

Original: Through the addition of random noise, DP prevents information leakage about any individualexample.
Revised: Through the addition of random noise, DP prevents information leakage about any individual example.

Original: For the purpose of this paper we consider the classical pure DP , parametrised by ϵ .
Revised: For the purpose of this paper, we consider the classical pure DP , parametrised by ϵ .

Original: Here, a
Revised: Here, a

Original: DP mechanism M with domain X and range R is ϵ -DP, if for all measurable sets S ⊆ R , and for anytwo adjacent databases D and D ′ , the following is satisfied: Pr [ M ( D ) ∈ S ] ≤ e ϵ Pr [ M ( D ′ ) ∈ S ] .
Revised: DP mechanism M with domain X and range R is ϵ -DP, if for all measurable sets S ⊆ R , and for any two adjacent databases D and D ′ , the following is satisfied: Pr [ M ( D ) ∈ S ] ≤ e ϵ Pr [ M ( D ′ ) ∈ S ] .

Original: To achieve ϵ -DP in this work we utilise the Laplace distribution [13].
Revised: To achieve ϵ -DP in this work we utilise the Laplace distribution [13].

Original:  Proposed Algorithm3.1 Problem Definition
Revised:  Proposed Algorithm 3.1 Problem Definition

Original: We now more precisely define the problem setup.
Revised: We now more precisely define the problem setup.

Original: The classical scenario in CL is a series of tasks{T (0) , T (1) , . .
Revised: The classical scenario in CL is a series of tasks {T (0) , T (1) , . .

Original: , T ( N ) } in which T ( i ) is a dataset comprised of K data-label pairs ( x, y ) .
Revised: , T ( N ) } in which T ( i ) is a dataset comprised of K data-label pairs ( x, y ) .

Original: The modelhas access to the data belonging to the current task T ( i ) , but no other tasks.
Revised: The model has access to the data belonging to the current task T ( i ) , but no other tasks.

Original: The goal of our learner isto minimise the loss incurred on the current and all prior tasks.
Revised: The goal of our learner is to minimise the loss incurred on the current and all prior tasks.

Original: However, note we can only optimisethe model using the current task data T ( i ) .
Revised: However, note we can only optimise the model using the current task data T ( i ) .

Original: Once training is finished on T ( i ) and we move to task
Revised: Once training is finished on T ( i ) and we move to task

Original: T ( i +1) the data from T ( i ) is inaccessible.
Revised: T ( i +1) the data from T ( i ) is inaccessible.

Original: Training is conducted in a FL manner and each task T ( i ) brings in new clients C ( i ) which have databelonging to T ( i ) , but they do not have data from prior tasks.
Revised: Training is conducted in a FL manner and each task T ( i ) brings in new clients C ( i ) which have data belonging to T ( i ) , but they do not have data from prior tasks.

Original: Clients which were already present inthe system at prior tasks {C (0) , C (1) , . .
Revised: Clients which were already present in the system at prior tasks {C (0) , C (1) , . .

Original: , C
Revised: , C

Original: ( i − 1) } have their own local data shift to T ( i ) .
Revised: ( i − 1) } have their own local data shift to T ( i ) .

Original: Hence, if aclient joined in task T ( i − 1) , on a task change, would have their current data update to follow T ( i ) .
Revised: Hence, if a client joined in task T ( i − 1) , on a task change, would have their current data update to follow T ( i ) .

Original: Forexample, they may encounter new classes in T ( i ) and loose access to the classes in T ( i − 1) .
Revised: For example, they may encounter new classes in T ( i ) and loose access to the classes in T ( i − 1) .

Original: Finally, aclient’s participation is modelled to be non-constant, and governed by a participation function f p ( N T )which is parameterised by the number of tasks a client has participated in thus far N T .
Revised: Finally, a client’s participation is modelled to be non-constant, and governed by a participation function f p ( N T ) which is parameterised by the number of tasks a client has participated in thus far N T .

Original: 3.2 Algorithm
Revised: 3.2 Algorithm

Original: We tackle this problem by examining privatised data sharing between clients.
Revised: We tackle this problem by examining privatised data sharing between clients.

Original: There is a seriesof works [14, 15, 16] considering data sharing between clients to improve performance, with [15]coining the phrase “hybrid-FL".
Revised: There is a series of works [14, 15, 16] considering data sharing between clients to improve performance, with [15] coining the phrase “hybrid-FL".

Original: The algorithm we propose is based on constructing a replay bufferwith old task data which can be sampled from when new tasks occur.
Revised: The algorithm we propose is based on constructing a replay buffer with old task data which can be sampled from when new tasks occur.

Original: Traditionally in FL, a clientwould apply this using only the locally held data.
Revised: Traditionally in FL, a client would apply this using only the locally held data.

Original: Instead, here clients form both local replay buffersusing locally held data, but also create a global replay buffer which is shared between the participants.
Revised: Instead, here clients form both local replay buffers using locally held data, but also create a global replay buffer which is shared between the participants.

Original: The procedure is simple to implement and execute, and gives strong performance boosts in a range ofsituations which we will empirically show in Section 5.
Revised: The procedure is simple to implement and execute, and gives strong performance boosts in a range of situations which we will empirically show in Section 5.

Original: The algorithm proceeds as follows:
Revised: The algorithm proceeds as follows:

Original: • When a task change occurs from T ( i ) to T ( i +1) , each client with data belonging to the task T ( i +1)computes a local buffer with N samples comprised of averaging N non-overlapping shards of theircurrent training data on a per-class basis.
Revised: • When a task change occurs from T ( i ) to T ( i +1) , each client with data belonging to the task T ( i +1) computes a local buffer with N samples comprised of averaging N non-overlapping shards of their current training data on a per-class basis.

Original: • Then the global buffer is constructed in three steps: 
Revised: • Then the global buffer is constructed in three steps: 1.

Original: Each participating client computes a differentially private mean over their data on a per-classbasis.
Revised: Each participating client computes a differentially private mean over their data on a per-class basis.

Original: We use Laplace DP noise governed by noise parameter ϵ . 
Revised: We use Laplace DP noise governed by noise parameter ϵ . 2.

Original: Each client sends the DP class means to the aggregator.
Revised: Each client sends the DP class means to the aggregator.

Original: The aggregator then re-averages thereceived data to maintain the required total buffer size producing the global buffer.
Revised: The aggregator then re-averages the received data to maintain the required total buffer size producing the global buffer.

Original: The clients then receive the global buffer and train on it in conjunction with the local buffer and current training data.
Revised: The clients then receive the global buffer and train on it in conjunction with the local buffer and current training data.

Original: It is worth highlighting that this method can be trivially used in conjunction with other CL approaches([2, 17, 18]) as during training time we are simply sampling from an extended dataset.
Revised: It is worth highlighting that this method can be trivially used in conjunction with other CL approaches ([2, 17, 18]) as during training time we are simply sampling from an extended dataset.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: We experiment over a sequence of 10 tasks.
Revised: We experiment over a sequence of 10 tasks.

Original: Each task has 150 training rounds before a task changeoccurs.
Revised: Each task has 150 training rounds before a task change occurs.

Original: Each client trains for 1 epoch on their local data in a FL training round.
Revised: Each client trains for 1 epoch on their local data in a FL training round.

Original: The aggregatoraverages all the weights received to make the new global model.
Revised: The aggregator averages all the weights received to make the new global model.

Original: We use the Permuted MNIST and
Revised: We use the Permuted MNIST and

Original: Fashion MNIST datasets.
Revised: Fashion MNIST datasets.

Original: As the Permuted MNIST dataset does not have spatial correlations we usea 4 layer dense neural network.
Revised: As the Permuted MNIST dataset does not have spatial correlations we use a 4 layer dense neural network.

Original: The exact architecture is given in Appendix A.
Revised: The exact architecture is given in Appendix A.

Original: In all experiments 2clients are present in the initial task, and every time a task change occurs, 2 new clients join.
Revised: In all experimentsclients are present in the initial task, and every time a task change occurs, 2 new clients join.

Original: As described above, the client replay buffers are composed of a local and global component.
Revised: As described above, the client replay buffers are composed of a local and global component.

Original: For thelocal component each client generate N = 5 means per class present in the current task.
Revised: For the local component each client generates N = 5 means per class present in the current task.

Original: The globalbuffer size is also set to be N = 5 means per class.
Revised: The global buffer size is also set to be N = 5 means per class.

Original: We investigate when the global buffer is fully distributed amongst all the clients, and compare withthe performance obtained if the aggregator does not send global data back to the clients, but ratherconducts training itself after aggregation on the replay buffer it holds.
Revised: We investigate when the global buffer is fully distributed amongst all the clients, and compare with the performance obtained if the aggregator does not send global data back to the clients, but rather conducts training itself after aggregation on the replay buffer it holds.

Original: In this case the clients will justhave the local component of the replay buffer.
Revised: In this case the clients will just have the local component of the replay buffer.

Original: For the DP noise we used the Laplace mechanism with differing levels of ϵ to obtain privacyperformance trade off results.
Revised: For the DP noise we use the Laplace mechanism with differing levels of ϵ to obtain privacyperformance trade-off results.

Original: We model the client participation as follows:
Revised: We model the client participation as follows:

Original: • We have an initial set of clients present and participating in the FL system.
Revised: • We have an initial set of clients present and participating in the FL system.

Original: They train and participate in each of the 150 training rounds in the current task.
Revised: They train and participate in each of the 150 training rounds in the current task.

Original: • Each time a task change occurs new clients join.
Revised: • Each time a task change occurs new clients join.

Original: New clients have training data pertaining to the current task, and when future task changes occur their data distribution will also update.
Revised: New clients have training data pertaining to the current task, and when future task changes occur their data distribution will also update.

Original: They do not have any data from tasks which occurred before they joined the FL system.
Revised: They do not have any data from tasks which occurred before they joined the FL system.

Original: Similarly, clients already in the system lose direct access to data from prior tasks following the classical CL setups.
Revised: Similarly, clients already in the system lose direct access to data from prior tasks following the classical CL setups.

Original: • Clients participate in all of the training rounds for a task in which they joined.
Revised: • Clients participate in all of the training rounds for a task in which they joined.

Original: However, for subsequent tasks their participation rate drops.
Revised: However, for subsequent tasks their participation rate drops.

Original: The subsequent participation rate is modelled following two distributions.
Revised: The subsequent participation rate is modelled following two distributions.

Original: In the first case, the clients follow an exponential decay distribution.
Revised: In the first case, the clients follow an exponential decay distribution.

Original: Thus, the participation probability on a given FL round of training is given by:
Revised: Thus, the participation probability on a given FL round of training is given by:

Original: [Equation]
Revised: [Equation]

Original: where N T is the number of tasks the client has participated in thus far, and β is the scale parametercontrolling the participation decay profile.
Revised: where N T is the number of tasks the client has participated in thus far and β is the scale parameter controlling the participation decay profile.

Original: In the second case, we model the clients as having a fixed uniform participation chance p pertraining round for tasks N T > 0 .
Revised: In the second case we model the clients as having a fixed uniform participation chance p per training round for tasks N T > 0 .

Original: This presents a challenging scenario in FL, as if data is considered purely on a local level then we willbe sampling early task data infrequently as time passes.
Revised: This presents a challenging scenario in FL, as if data is considered purely on a local level then we will be sampling early task data infrequently as time passes.

Original: This means the model will fit poorly to thoseinitial clients, potentially resulting in them leaving the FL system due to poor model performance.
Revised: This means the model will fit poorly to those initial clients, potentially resulting in them leaving the FL system due to poor model performance.

Original: We examine two styles of CL that can arise in the class-incremental scenario outlined in Section 2.1:
Revised: We examine two styles of CL that can arise in the class-incremental scenario outlined in Section 2.1:

Original: Novel Inputs Scenario:
Revised: Novel Inputs Scenario:

Original: In this situation we assume that all the data is randomly divided betweenclients.
Revised: In this situation we assume that all the data is randomly divided between clients.

Original: The shifts between tasks occurs in the input distribution x .
Revised: The shifts between tasks occurs in the input distribution x .

Original: We use the Permuted MNISTdataset to generate distinct tasks.
Revised: We use the Permuted MNIST dataset to generate distinct tasks.

Original: Performance Difference Compared to Baseline

Original: [Table]

Original: Novel Inputs & Targets Scenario:
Revised: Novel Inputs & Targets Scenario:

Original: Here we look at when each new task is comprised of a new setof classes, so the shifts between tasks involve both changes in the inputs x and also novel targets y .
Revised: Here we look at when each new task is comprised of a new set of classes, so the shifts between tasks involve both changes in the inputs x and also novel targets y .

Original: We use two classes per task, and each task selects the next two classes from MNIST starting withthe initial task having the first two classes
Revised: We use two classes per task, and each task selects the next two classes from MNIST starting with the initial task having the first two classes

Original: y = { 0 , 1 } .
Revised: y = { 0 , 1 } .

Original: At the 5th task, once all the MNIST data isused, we then iterate through the Fashion MNIST dataset in the same manner.
Revised: At the 5th task, once all the MNIST data is used, we then iterate through the Fashion MNIST dataset in the same manner.

Original: Thus, in the end, themodel needs to perform classifications over 20 classes comprised of both the MNIST (with labelsy = { 0 , . .
Revised: Thus, in the end, the model needs to perform classifications over 20 classes comprised of both the MNIST (with labels y = { 0 , . .

Original: and Fashion MNIST datasets (having labels y
Revised: and Fashion MNIST datasets (having labels y

Original: =
Revised: =

Original: On each new task,
Revised: On each new task,

Original: T ( i ) , the new task data is divided randomly between the clients present in the task {C (0) , . . .
Revised: T ( i ) , the new task data is divided randomly between the clients present in the task {C (0) , . . .

Original: , C ( i ) } .
Revised: , C ( i ) } .

Original: Baseline Setup : For our baseline we compare with the setup in which the replay data buffer is keptpurely locally to the client.
Revised: Baseline Setup : For our baseline we compare with the setup in which the replay data buffer is kept purely locally to the client.

Original: The buffer is computed in the same manner via computing per classaverages on the local data.
Revised: The buffer is computed in the same manner via computing per class averages on the local data.

Original: As data is not shared, no DP operations are applied.
Revised: As data is not shared, no DP operations are applied.

Original: We use N = 10averaged samples per class for the baseline configuration.
Revised: We use N =averaged samples per class for the baseline configuration.

Original:  Results
Revised:  Results

Original: We run the described experiments, repeating each configuration 3 times reporting the mean resultfor both models of client participation.
Revised: We run the described experiments, repeating each configuration 3 times reporting the mean result for both types of client participation.

Original: The results are shown in Table 1 highlighting the privacyperformance trade-offs at the different limits of client participation investigated.
Revised: The results are shown in Table 1 highlighting the privacy- Performance Difference Compared to Baseline [Table] performance trade-offs at the different limits of client participation investigated.

Original: Intermediate levelsof participation interpolate between these  values.
Revised: Intermediate levels of participation interpolate between these results and values.

Revised: The key result is that the lower the client participation, the more pronounced the effects of having a shared global buffer become.

Original: Some examples are plotted in Figures 1 and 2.
Revised: For example, for uniform client participation, the joining probabilities were always high enough such that gains are modest when using a global buffer - hence the maximum gain in performance was just 6.61%. However, with exponential client participation, the joining probability can fall very rapidly and thus it becomes crucial to have a distributed global buffer. There are therefore large gains in performance with the largest boost being 49.06% due to the addition of a global buffer. Hence, the proposed algorithm should be applied in a context dependant manner: if in FCL clients are expected to participate with profiles similar to the exponential decay scenario, then deploying a global buffer in the manner described would alleviate many of the underlying issiues.

Revised: However, if clients are participating frequently enough such that old task data is sampled sufficiently from the local replay buffer (such as when we examined a uniform participation profile) then using a global buffer will only give small benefits which may not outweigh the additional communication and privacy costs.

Original: Generally, lowering the level of DP noise results in the accuracy correspondingly rising, particularlywith exponential client modelling.
Revised: Regarding the effects of DP, generally, lowering the level of DP noise results in the accuracy correspondingly rising, particularly with exponential client modelling.

Original: There arealso a number of experimental configurations which had little performance difference between ϵ = 10 . 0 and ϵ = 100 . 0 illustrating that the increased privacydoes not always need to result in a performance drop.
Revised: However, there are a number of experimental configurations which had little performance difference between ϵ = 10 . 0 and ϵ = 100 . 0 illustrating that the increased privacy does not always need to result in a performance drop.

Original: To gain a better intuition as to what the client’sshared data looks like, we show examples from the datasets in the Appndix.
Revised: To gain a better intuition as to what a client’s shared data looks like, we show examples from the datasets in the Appndix.

Original: The gains are more modest in the uniform client participation model: for the Novel Inputs case themaximum gain was 6.61%, and in the Novel Inputs & Targets scenario the improvements were allbelow 1%.

Original: This suggests that in the uniform modelling case the clients are participating frequentlyenough such that old task data is sampled sufficiently from the local replay buffers.

Original: In almost all cases, training with a decentralised global buffer, compared to a global buffer only heldby the aggregator, performed in a similar manner.
Revised: In almost all cases, training with a decentralised global buffer, compared to a global buffer only held by the aggregator, performed in a similar manner.

Original: Hence, the decision of whether to distribute theglobal buffer does not have to be influenced by the performance, but can be taken based on factorssuch as the privacy tolerance of the clients, and the compute resources available to the aggregator.
Revised: Hence, the decision of whether to distribute the global buffer does not have to be influenced by the performance, but can be taken based on factors such as the privacy tolerance of the clients, and the compute resources available to the aggregator.

Original: Inthe bottom section of Table 1 we show the case in which clients participate in every roundrepresenting an upper bound on performance.
Revised: In Table 2 we show the case in which clients participate in every round representing the upper limit of client participation.

Original: In that setup sharing data is of limited benefit, so theperformance is similar to training on purely local data.
Revised: In that setup sharing data is of limited benefit, so the performance is similar to training on purely local data.

Original: We did have an unexpected result when clientsalways participated for the Novel Inputs scenario.
Revised: We did have an unexpected result when clients always participated for the Novel Inputs scenario.

Original: At the lowest levels of DP noise with ϵ = 100 . {+
+}
Revised: At the lowest levels of DP noise with ϵ = 100 . {+
+}

Original: 0over-fitting occurs, with  the low noise configurations performing worse than those with higher levelsof noise, although still performing comparable to, or slightly better than, the baseline accuracy.
Revised: over-fitting occurs, with [Table] the low noise configurations performing worse than those with higher levels of noise, although still performing comparable to, or slightly better than, the baseline accuracy.

Original: We show the full performance curves during training at various levels of client participation in Figures1 and 2 at ϵ = 10 . 0 .
Revised: We show the full performance curves during training at various levels of client participation in Figures 1 and 2 at ϵ = 10 . 0 .

Original: For space constraints we limit ourselves to these configurations to fully plot out.
Revised: For space constraints we limit ourselves to these configurations to fully plot out.

Original: We show both the raw accuracy of our method along with the baseline and highlight the performancedifference.
Revised: We show both the raw accuracy of our method along with the baseline and highlight the performance difference.

Original: We can see in this case that our method outperforms the baseline across the range of βvalues we investigated (0.5 - 2.0).
Revised: We can see in this case that our method outperforms the baseline across the range of β values we investigated (0.5 - 2.0).

Original:  Conclusion
Revised:  Conclusion

Original: In this work we have examined how replay based methods can be extended to work in a FCL settingby sharing data characteristics with DP noise.
Revised: In this work we have examined how replay based methods can be extended to work in a FCL setting by sharing data characteristics with DP noise.

Original: Here we have focused on temporal variation in clientparticipation, with clients participating less in the training process as time passes.
Revised: Here we have focused on temporal variation in client participation, with clients participating less in the training process as time passes.

Original: We saw in this casethat even with relatively high levels of DP noise we often have performance gains due to the lack ofparticipation from clients who hold valuable data.
Revised: We saw in this case that even with relatively high levels of DP noise we often have performance gains due to the lack of participation from clients who hold valuable data.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work has been partially supported by the MORE project (grant agreement 957345), funded bythe EU Horizon 2020 program.
Revised: This work has been partially supported by the MORE project (grant agreement 957345), funded by the EU Horizon 2020 program.

Original: A Appendix: Model Architecture
Revised: A Appendix: Model Architecture

Original: Neural network architecture used for our experiments.
Revised: Neural network architecture used for our experiments.

Original: FC N indicates a fully connected layer with
Revised: FC N indicates a fully connected layer with

Original: N outputs.
Revised: N outputs.

Revised: FC

Original: [Figure]
Revised: [Equation]

Revised: FC 1000 → ReLU →

Original: for Novel Inputs : FC 10for Novel Inputs & Targets : FC[-
-]
Revised: for Novel Inputs : FCfor Novel Inputs & Targets : FC[-
-]

Original: B Appendix: Example Data
Revised: B Appendix: Example Data

Original: We show example data from the MNIST and Fashion MNIST datasets with differing ϵ noise budgets.
Revised: We show example data from the MNIST and Fashion MNIST datasets with differing ϵ noise budgets.

Original: [Figure]
Revised: [Figure]