Original: Reducing Forgetting in Federated Learning with Truncated Cross-Entropy
Revised: Reducing Forgetting in Federated Learning with Truncated Cross-Entropy

Original: Anonymous Author(s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: In federated learning (FL), a global model is learned by aggregating model updatescomputed from a set of client nodes, each having their own data.
Revised: In federated learning (FL), a global model is learned by aggregating model updates computed from a set of client nodes, each having their own data.

Original: A key challengein FL is the heterogeneity of data across clients whose data distributions differfrom one another.
Revised: A key challenge in FL is the heterogeneity of data across clients whose data distributions differ from one another.

Original: Standard FL algorithms perform multiple gradient steps beforesynchronizing the model, which can lead to clients overly minimizing their localobjective and diverging from other client solutions.
Revised: Standard FL algorithms perform multiple gradient steps before synchronizing the model, which can lead to clients overly minimizing their local objective and diverging from other client solutions.

Original: We demonstrate that in such asetting individual client models experience “catastrophic forgetting" with respectto other client data.
Revised: We demonstrate that in such a setting individual client models experience “catastrophic forgetting" with respect to other client data.

Original: We propose a simple yet efficient approach that modifies thecross-entropy objective on a per-client basis such that classes outside a client’s labelset are shielded from abrupt representation change.
Revised: We propose a simple yet efficient approach that modifies the cross-entropy objective on a per-client basis such that classes outside a client’s label set are shielded from abrupt representation change.

Original: Through empirical evaluations,we demonstrate our approach can alleviate this problem, especially under the mostchallenging FL settings with high heterogeneity, low client participation
Revised: Through empirical evaluations, we demonstrate our approach can alleviate this problem, especially under the most challenging FL settings with high heterogeneity, low client participation.

Original:  Introduction
Revised:  Introduction

Original: federated learning (FL) is a distributed machine learning paradigm in which a shared global model islearned from a decentralized set of data located at a number of independent client nodes [1, 2]. 
Revised: federated learning (FL) is a distributed machine learning paradigm in which a shared global model is learned from a decentralized set of data located at a number of independent client nodes [1, 2]. Due to communication constraints

Original: Due tocommunication constraints, FL algorithms typically perform many local gradient update steps beforesynchronizing and in realistic settings where client data  often have non-iid distributions, followingmultiple local update steps this can cause client solutions to “drift" severely  from an optimal globalsolution [3].
Revised: FL algorithms typically perform many local gradient update steps before synchronizing with the global model. FL was conceptualized as a learning technique to train a shared model without sharing user sensitive data but allowing users to benefit from data stored at other nodes, such as phones and tablets of decentralized users. Under realistic settings, client data will often have non-iid distributions and for the case of supervised multi-class classification, users may frequently be missing data from an entire class or multiple classes. The data inhomogeneity across clients frequently induces client drift, a phenomenon in which clients progress too far towards optimizing their own local objective, leading to a solution that has severely "drifted" from an optimal global solution [3].

Original: Continual learning (CL)
Revised: Continual learning (CL)

Original: [4] is another emerging research area in which a learner ispresented with a sequence of tasks to learn in succession.
Revised: [4] is another emerging paradigm in which a learner is presented with a sequence of tasks to be learned in succession.

Original: Similar to how FL clients may have non-iiddata, different tasks in CL also typically contain data drawn from different distributions. 
Revised: Similar to how FL clients may have non-iid data, different tasks in CL also typically contain data drawn from different distributions. In

Original: In CL theproblem of catastrophic forgetting is a focus of study since we want a model to retain its knowledgeof previous tasks after learning a new one.
Revised: CL the problem of catastrophic forgetting is a focus of study since it is desirable for a model to retain its knowledge of previous tasks after learning a new one.

Original: Consider one “round" of FL, in which C random clientsare selected and sent a copy of the current global model. Each client performs a number of localupdate steps to optimize the objective using their local data.
Revised: We consider one round of FL in which C random clients are selected and sent a copy of the current global model and each client performs a number of local update steps to optimize the objective using their local data.

Original: A round ends with an update to theglobal model achieved by the aggregating the updates from each client.
Revised: The round ends with an update to the global model achieved by the aggregating the updates from each client.

Original: In a typical round,as localtraining proceeds, the client model becomes increasingly biased towards that clients dataset andwill experience a catastrophic forgetting with respect to the data from other clients.
Revised: As local training proceeds for a typical round, the client model becomes increasingly biased towards that clients dataset and will experience a catastrophic forgetting with respect to the data from other clients.

Original: In this way wecan draw a connection between the catastrophic forgetting problem  and client drift.
Revised: In this way we can draw a connection between the catastrophic forgetting problem of CL and client drift problem of FL.

Original: We denote thisproblem as local client forgetting .
Revised: We denote this problem as local client forgetting .

Original: Reducing client forgetting would moderate an individual clients increase in loss with respect toother clients data, improving the loss of the client model over the combined data.
Revised: Reducing client forgetting would moderate an individual clients increase in loss with respect to other clients data, improving the loss of the client model over the combined data.

Original: So we submitthat tackling local client forgetting can reduce client drift.
Revised: Therefore we conclude that tackling local client forgetting can reduce client drift.

Original: Many methods to control catastrophicforgetting have been proposed  [5–9] but they are largely impractical for FL.
Revised: Many methods to control catastrophic forgetting have been proposed in the CL literature [5–9] but they are largely impractical for FL.

Original: Experience Replaymethods [7] require access to other clients data violating FL’s data communication constraints.
Revised: Experience Replay methods [7] require access to other clients data violating FL’s data communication constraints.

Original: Many regularization methods such as EWC require communicating additional information and moreover typically require many steps to converge [10] due to additional conflicting objectives.
Revised: Many regularization methods such as EWC require communicating additional information and moreover typically require many steps to converge [10] due to additional conflicting objectives.

Original: For thesupervised CL setting [11, 12] propose a modification of the standard cross entropy (CE) objectivefunction that truncates the softmax denominator, removing terms corresponding to classes from oldtasks.
Revised: For the supervised CL setting [11, 12] propose a modification of the standard cross entropy (CE) objective function that truncates the softmax denominator, removing terms corresponding to classes from old tasks thereby by reducing the bias on the model to avoid predicting old classes.

Original: In FL, as local client optimization proceeds, optimizing the terms in the CE corresponding toclasses not present in that client’s data distribution will encourage absent classes to be forgotten bythe local model.
Revised: In FL, as local client optimization proceeds, optimizing the terms in the CE corresponding to classes not present in that client’s data distribution will encourage absent classes to be forgotten by the local model.

Original: Furthermore, as discussed in [13], it can cause spurious features to emerge.
Revised: Furthermore, as discussed in [13], it can cause spurious features to emerge.

Original: Inspiredby these observations and the parallels between FL and CL we adapt the solution of [11, 12] to FLby correcting each clients loss function based on its class distribution.
Revised: Inspired by these observations and the parallels between FL and CL we adapt the solution of [11, 12] to FL by correcting each clients loss function based on its class distribution.

Original: We show this approach cangreatly reduce client level forgetting in the heterogeneous setting and lead to substantially improvedoverall global model convergence.
Revised: We show this approach can greatly reduce client level forgetting in the heterogeneous setting and lead to substantially improved overall global model convergence.

Original: It also yield optimization more robust to hyper-parameters andnormalization layers.
Revised: It also yield optimization more robust to hyper-parameters and normalization layers.

Original:  Background and Methods
Revised:  Background and Methods

Original: In FL [2] training data is distributed and optimization occurs over K clients with each client k ∈1 , ..., K having data X k drawn from distribution D k .
Revised: In FL [2] training data is distributed and optimization occurs over K clients with each client k ∈ 1 , ..., K having data X k drawn from distribution D k .

Original: We define n k = |
Revised: We define n k = |

Original: X k | and n = (cid:80)
Revised: X k | and n = (cid:80)

Original: Kk =1 n k forn samples.
Revised: Kk =1 n k for n samples.

Original: The data X k at each node may be drawn from different distributions and/or may beunbalanced with some clients possessing more training samples than others.
Revised: The data X k at each node may be drawn from different distributions and/or may be unbalanced with some clients possessing more training samples than others.

Original: The typical objectivefunction for federated optimization is given by
Revised: The typical objective function for federated optimization is given by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  with L ( w , X k ) measuring client k ’s local objective, and w representing the global  parameters.
Revised: ) with L ( w , X k ) measuring client k ’s local objective, and w representing the global model parameters.

Original: In thiswork we will restrict ourselves to the common case where L is the CE loss.
Revised: In this work we will restrict ourselves to the common case where L is the CE loss.

Original: There are many possiblevariations of FL algorithms.
Revised: There are many possible variations of FL algorithms.

Original: In general, they follow the same structure as FedAvg [1]: (a) clientselection: for a set of K clients, K ∗ C are selected at each round { t i } Ti =1 , where 0 < C ≤ 1 is apre-determined fraction.
Revised: In general, they follow the same structure as FedAvg [1]: (a) client selection: for a set of K clients, K ∗ C are selected at each round { t i } Ti =1 , where 0 < C ≤ 1 is a pre-determined fraction.

Original: (b) client updates: At the beginning of round, client models are initializedwith the current weights of the server model.
Revised: (b) client updates: At the beginning of round, client models are initialized with the current weights of the server model.

Original: Each client selected for the round performs E localiterations of SGD.
Revised: Each client selected for the round performs E local iterations of SGD.

Original: (c) server update: The weights of the individual client models are aggregated toform an update to the shared global model.
Revised: (c) server update: The weights of the individual client models are aggregated to form an update to the shared global model.

Original: Truncated Cross Entropy Consider a neural network f : R D → R C where C is the total numberof classes.
Revised: Truncated Cross Entropy Consider a neural network f : R D → R C where C is the total number of classes.

Original: The standard CE is given L CE ( X k , Y k , w ) = − [Equation] x ∈ X log exp [Equation]
Revised: The standard CE is given by equation

Original: f w

Original: [Equation]
Revised: [Equation]

Original: x ) y ( x ) ) (cid:80) c [Equation] exp
Revised: Here y ( x ) is the label of x and C is the set of all classes available to the clients.

Original: [Equation] w ( x ) c ) . Here y ( x )is the label of x and C is the set of all classes available to the clients. We now consider the truncatedcross entropy (TCE) 
Revised: We now consider the truncated cross entropy (TCE) characterized by equation 3 where the denominator is a function of the labels for the client data Y k .

Original: [Equation]
Revised: [Equation]

Original: where the denominator is a function of the labels for the client data Y k .

Original: Specifically, in ourwork we consider C ( Y k ) to be the set unique labels for the client.
Revised: In our work we consider C ( Y k ) to be the set unique labels for the client.

Original: Aggressively optimizing
Revised: Aggressively optimizing

Original: L CE ( X k , Y k ) through multiple gradient steps during a client round can lead to a drastic increase in68 E x,y ∼ D
Revised: L CE ( X k , Y k ) through multiple gradient steps during a client round can lead to a drastic increase in E x,y ∼ D

Original: j ̸ = k [ l CE ( x , y )] where D j are the distribution of clients other than client k .
Revised: j ̸ = k [ l CE ( x , y )] where D j are the distribution of clients other than client k .

Original: The TCE approach,modifies the original local objective function to avoid excessive pressure that drives up the loss ofother client data so that classes not present at the current client are ignored by the local optimization. 
Revised: The TCE approach, modifies the original local objective function to avoid excessive pressure that drives up the loss of other client data so that classes not present at the current client are ignored by the local optimization. [Table] mean per model 0. FedAvg Post Local Update 0.

Original:  This forces each client to learn by adapting the model’s internal representation of the classes presentin its training data, rather than abruptly shifting representations of classes outside its training set [11].
Revised: [Figure] [Figure] This forces each client to learn by adapting the model’s internal representation of the classes present in its training data, rather than abruptly shifting representations of classes outside its training set [11].

Original: We will empirically demonstrate this leads to a reduction in the notion of forgetting defined below.
Revised: In what follows we will empirically demonstrate this leads to a reduction in the notion of forgetting defined below.

Original: Local client forgetting We formalize the notion of local client forgetting discussed in Sec 1 for aclassification problem.
Revised: Local client forgetting We formalize the notion of local client forgetting discussed in Sec 1 for a classification problem.

Original: Denoting the accuracy on a client k ’s local test data Acc k ( w ) , with w themodel parameters.
Revised: Denoting the accuracy on a client k ’s local test data Acc k ( w ) , with w the model parameters.

Original: We can consider the local client forgetting F ki
Revised: We can consider the local client forgetting F ki

Original: =
Revised: =

Original: Acc k ( w it ) − Acc k ( w t − 1 ) .
Revised: Acc k ( w it ) − Acc k ( w t − 1 ) .

Original: Herew it refers to the model of client i at round t (before the aggregation step) and w t − 1 the global modelat the end of round t − 1 .
Revised: Here w it refers to the model of client i at round t (before the aggregation step) and w t − 1 the global model at the end of round t − 1 .

Original: Furthermore, we can define an average forgetting for a client k ’s model
Revised: Furthermore, we can define an average forgetting for a client k ’s model

Original: F k = 1 K − 1 (cid:80)
Revised: F k = 1 K − 1 (cid:80)

Original: i ̸
Revised: i ̸

Original: = k F ki .
Revised: = k F ki .

Original: In the sequel we will study these quantities for a standard FL setting.
Revised: In the sequel we will study these quantities for a standard FL setting.

Original:  Experiments
Revised:  Experiments

Original: In this section, we present the empirical results for the TCE framework.
Revised: In this section, we present the empirical results for the TCE framework.

Original: We analyze the notionof forgetting in the context of standard FedAVG, and then show how the application of the TCEobjective can  resolve this.
Revised: We analyze the notion of forgetting in the context of FedAvg and then show how the application of the TCE to the FedAvg objective can help resolve local client forgetting.

Original: Subsequently, we study how TCE can enhance standard FL algorithms like [Table] mean per model 0. FedAvg Post Local Update 0.
Revised: Subsequently, we study how TCE can enhance standard FL algorithms like FedAvg, improving their overall performance over a range of key hyperparameters.

Original: [Figure] [Figure] FedAVG, improving their overall performance and making them more robust to key hyperparameters,namely the learning rate and choice of normalization technique. Datasets and Setting We utilize CIFAR-10, CIFAR-100 [14] and FEMNIST [15] datasets in theexperiments.
Revised: Datasets and Setting We utilize CIFAR-10, CIFAR-100 [14] and FEMNIST [15] datasets in our experiments, each of these datasets come pre-separated into training and testing sets.

Original: Primary evaluations consider 100 clients where each client possesses their own trainingand validation sets according to their own unique distribution.
Revised: Our primary evaluations consider 100 clients where each client requires their own training and validation sets according to their own unique distribution.

Original: To facilitate this, the  training set isseparated into equally sized non-i.i.d partitions using the Dirichlet distribution parameterized byα = 0 . 1 , similar to the method of [16].
Revised: To facilitate this, the entire training set is separated into equally sized non-iid partitions using the Dirichlet distribution parameterized by α = 0 . 1 , similar to the method of [16].

Original: These client partitions are then further separated into training( 90% ) and validation ( 10% ).
Revised: These client partitions are then further separated into training ( 90% ) and validation ( 10% ) sets for each client.

Original: Clients are sampled without replacement for each round but can beselected again in subsequent rounds.

Original: The fraction of clients sampled is 10% for CIFAR-10 and FEMNIST datasets and 1% for CIFAR-100.
Revised: For example, 100 clients being trained using CIFAR-10 which contains 50 000 training samples would each have 500 of these training samples.

Original: In each case our primary evaluations train a ResNet-18over 4000 communication rounds for3 local epochs, using a mini-batch size of 64. We use SGD asour optimizer, with weight decay of 1 × 10 − 4 following [17], [16]. Experiments done without TCE(FedAvg) replace BN with GN as BM performs poorly in a FL setting [18], experiments using TCEare run twice, once with BN and once with GN.
Revised: Of thosesamples, 450 would be used for local model updates and 50 would be used exclusively for validation.

Original: Validation Throughout the training process the global model is evaluated periodically on theaggregation of client validation sets to gauge overall training progress.
Revised: Validation Throughout the training process the global model is evaluated periodically on the aggregation of client validation sets to gauge overall training progress.

Revised: A model is evaluated on the training set only once, after the completion of the entire 4000 rounds of training.

Original: We focus our analysis on thehighly heterogeneous case α = 0 . 1 , which canalso lead to higher variance in the results, particularlyfor smaller datasets such as CIFAR-10 and CIFAR-100.
Revised: This value is the accuracy reported in the Table 1. Since we focus our analysis on the highly heterogeneous case α = 0 . 1 , this can lead to higher variance in the validation statistics [16], particularly for smaller [Figure]

Revised: [Figure]

Revised: datasets such as CIFAR-10 and CIFAR-100.

Original: We thus run each training 3 times to reducevariance of the results.
Revised: To combat this effect, we run each training three times to reduce variance of the final reported results [Figure]

Original: 3.1 Forgetting During a Federated Round
Revised: 3.1 Forgetting During a Federated Round

Original: We study the local client forgetting at different optimization rounds of FL.
Revised: We study the local client forgetting at different optimization rounds of FL.

Original: In Figure 1 we show F ki(the heatmap) and F k (the last column) for the set of participating clients for the respective round for
Revised: In Figure 1 we show F ki (the heatmap) and F k (the last column) for the set of participating clients for the respective round for

Original: FedAVG using both TCE and CE.
Revised: FedAvg using both TCE and CE.

Original: We observe forgetting is very high when using standard CE but
Revised: We observe forgetting is very high when using standard CE but

Original: TCE is able to control local client forgetting leading to better performance of the global model afteraggregation.
Revised: TCE is able to control local client forgetting leading to better performance of the global model post aggregation.

Original: The results in Figure 1 are  for round 2800 of training however, the observation persistsfor other rounds as shown in the Appendix Sec A.3.
Revised: The results in Figure 1 are shown for round 2800 of training however, the observation persists for other rounds as shown in the Appendix Sec A.3.

Original: Having observed that TCE can indeed reduce thelocal client forgetting we now study its effect on the aggregated models.
Revised: Having observed that TCE can indeed reduce the local client forgetting we now study its effect on the aggregated models.

Original: 3.2 Evaluations on CIFAR-10,CIFAR-100 and FEMNIST
Revised: 3.2 Evaluations on CIFAR-10,CIFAR-100 and FEMNIST

Original: Figure 2 shows the best performing models among FedAVG and FedAVG+TCE for CIFAR-10 and
Revised: Figure 2 shows the best performing models among FedAvg and FedAvg+TCE for CIFAR-10 and

Original: CIFAR-100.
Revised: CIFAR-100.

Original: From these results, we observe FedAvg+TCE is robust to the normalization layer unlike [Figure] [Figure] [Figure]
Revised: From these results, we observe FedAvg+TCE is robust to the normalization layer unlike vanilla FL methods which typically do poorly when using BN.

Original: [Figure]

Original: [Figure]

Original: u

Original: [Figure]

Original: c

Original: [Figure]

Original: Dirichlet Alpha (Heterogeneity) (CIFAR-10)

Original: [Figure]

Original: TCE

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: cc u r a c y

Original: Fraction of Clients Reporting (CIFAR-10)

Original: [Figure]

Original: vanilla FL methods which typically do poorly when using BN.

Original: In general we conclude that for both
Revised: In general we conclude that for both

Original: CIFAR-10 and CIFAR-100, TCE substantially improves performance for any training budget.
Revised: CIFAR-10 and CIFAR-100, TCE substantially improves performance for any training budget.

Original: We seea similar performance increase when using the FEMNIST dataset (Table 1) further supporting ourclaims.
Revised: We see a similar performance increase when using the FEMNIST dataset (Table 1) further supporting our claims.

Original: Robustness to learning rate and normalization In the Appendix Table 1 we summarize a detailedanalysis of model performance over a range of learning rates.
Revised: Robustness to learning rate and normalization In the Appendix Table 1 we summarize a detailed analysis of model performance over a range of learning rates.

Original: Additionally, we vary the normalizationmethod between BN and GN for TCE.
Revised: Additionally, we vary the normalization method between BN and GN for TCE.

Original: With regard to the normalization method, we observecomparable performance using either BN or GN indicating TCE helps to mitigate performancedegradation typically observed when using BN in heterogeneous FL [19].
Revised: With regard to the normalization method, we observe comparable performance using either BN or GN indicating TCE helps to mitigate performance degradation typically observed when using BN in heterogeneous FL [17].

Original: In general, we observe thatacross all client learning rates TCE provides performance improvement using either normalizationsetting.
Revised: In general, we observe that across all client learning rates TCE provides performance improvement using either normalization setting.

Original: We particularly notice improvements by TCE even at higher learning rates, where vanilla
Revised: We particularly notice improvements by TCE even at higher learning rates, where vanilla

Original: FedAVG can collapse or under-perform.
Revised: FedAvg can collapse or under-perform.

Original: We also remark that not only is overall performancehigher for FedAVG+TCE
Revised: We also remark that not only is overall performance higher for FedAvg+TCE

Original: but it’s performance deviates less from the case of it’s best performinghyperparameters as various settings are changed.
Revised: but it’s performance deviates less from the case of it’s best performing hyperparameters as various settings are changed.

Original: This analysis is further supported by Table 2 in the Appendix.
Revised: This analysis is further supported by Table 2 in the Appendix.

Original: 3.3 Ablations
Revised: 3.3 Ablations

Original: We further study the behavior of TCE in combination with FedAVG under different data distributionsand client participation settings.
Revised: We further study the behavior of TCE in combination with FedAvg under different data distributions and client participation settings.

Original: In Figure 3 ablations for the CIFAR-10 setting are shown where weablate one setting at a time.
Revised: In Figure 3 ablations for the CIFAR-10 setting are shown where we ablate one setting at a time.

Original: We observethere are particular settings of α and fraction of clients, forwhich TCE provides improvement.
Revised: We observe particular settings under which TCE provides vast improvement over vanilla FedAvg.

Original: The Dirichlet distribution is parameterized by α .
Revised: Parameter α of the Dirichlet Distribution

Original: As α approaches0 the client distribution will become increasingly likely to contain fewer labels and as α increases theclient distribution will be increasingly i.i.d..
Revised: The Dirichlet distribution is parameterized by α as α → 0 the client distributions will become increasingly heterogeneous and as α → ∞ , each client data edges closer towards the same i.i.d distribution.

Original: We investigate a range of α from 0 .
Revised: In our base case we use α = 0 .

Original: 01 to 100 ( α = 100is
Revised: and we now investigate how model performance is affected by changing the heterogenaity of the data partitions. Similar to the work of [16], we investigate α = { 0 . 01 , 0 . 1 , 0 . 2 , 0 . 5 , 1 , 10 , 100 } Figure 4 offers a practical illustration of how client partitions change as a function of α .

Original: considered i.i.d.)
Revised: From Figure 3 we observe

Original: and we observe as α increases, the gap between TCE and CE shrinks. Thisobservation is due to the fact that as α approaches 100 most clients will have most of the classesand thus the approaches become nearly equivalent.
Revised: TCE has a more significant effect as α decreases with the biggest margin of improvement over vanilla FedAvg occurring when α = 0 . As the data becomes increasingly homogeneous, the gap between cross entropy with and without TCE shrinks until the data is iid and the two methods are equivalent.

Original: As we reduce α to 0 .
Revised: From Figure 4 we see that by the time α = 0 .

Original: 01 , an heterogeneous case,the performance improvement gap widens significantly between FedAvg and FedAvg+TCE.
Revised: 5 most of the clients possess all of the classes, albeit in very skewed proportions. This is also the point at which the performance of FedAvg and FedAvg+TCE become very close, from this observation, we conclude TCE is most advantageous when clients do not possess all possible class labels.

Original:  For thefraction of participating clients, we observe the largest performance gap between FedAvg+TCE and FedAvg when the number of participating clients is low.
Revised: Fraction of Participating Clients For the fraction of participating clients C , we observe the largest performance gap between FedAvg+TCE and FedAvg when the number of participating clients is low.

Original: This result is significant since it mimicsmost closely potential real world settings.
Revised: This result is significant since it mimics potential real world settings in which communication constraints can impede a user’s ability to send its update to the central server.

Revised: We hypothesize the performance gap as a function of client fraction between the FedAvg and FedAvg+TCE is due to the larger impact of local client forgetting when we limit communication capability.

Original: We hypothesize this effect is because unless we controllocal client forgetting, non-participating clients are unable to contribute their updates to the globalmodel and will have their data distributions forgotten.
Revised: Unless we actively take steps to control forgetting, non-participating clients will have their data distributions forgotten because unlike participating clients, they will be unable to contribute their updates to the global model.

Revised: As the fraction of clients selected at each round increases, we observe the performance gap between the two methods narrow since more clients will have the opportunity to be selected at each round and "remind" the model of their data distributions.

Original:  Conclusion 
Revised:  Conclusion and Future Directions

Original: In this paper we took a deeper look at the local client forgetting problem, we show empirically thatwhen a client performs local updates during FL, it risks over optimizing its local objective, whichcan lead to forgetting on other subsets of data which degrades performance of the global model.
Revised: In this paper we took a deeper look at the local client forgetting problem, we show empirically that when a client performs local updates during FL, it risks over optimizing its local objective, which can lead to forgetting on other subsets of data which degrades performance of the global model.

Original: Wedemonstrate this phenomenon is especially severe in cases where there is a significant distributionmismatch across clients.
Revised: We demonstrate this phenomenon is especially severe in cases where there is a significant distribution mismatch across clients.

Original: We proposed a client level modification of the objective function whichwe call TCE, that allows us to mitigate client level forgetting.
Revised: We proposed a client level modification of the objective function which we call TCE, that allows us to mitigate client level forgetting.

Original: We demonstrate our method leads toimproved performance when combined with a standard FL algorithm, particularly in the regime ofhighly heterogeneous client datasets and/or when a small percentage of clients are selected at eachround.
Revised: We demonstrate our method leads to improved performance when combined with a standard FL algorithm across a range of learning rates. particularly in the regime of highly heterogeneous client datasets and/or when a small percentage of clients are selected at each round.

Original: We also demonstrate TCE is more robust to a wide range of hyperparameter settings thanvanilla FedAvg.
Revised: We also demonstrate TCE is more robust to a wide range of hyperparameter settings than vanilla FedAvg.

Original: Finally, we showed that addressing local client forgetting is particularly important incases of severe data heterogeneity.
Revised: In on-going and future work we are investigating soft versions of the truncation operation, which can effectively deal with cases where classes are present but in very low quantities.  Acknowledgements

Original: Future investigation can consider the combination of our modifiedobjective function with a broader range of algorithms designed for FL.
Revised: This research was partially funded by NSERC Discovery Grant RGPIN-2021-04104. We also would like to acknowledge the support from Digital Research Alliance and Calcul Quebec.

Original: A Appendix 
Revised: A Appendix A.1 Related Work

Original: A.1 Related Work

Original: Federated Learning The most commonly used baseline in FL is the FedAvg algorithm proposed by[1].
Revised: Federated Learning The most commonly used baseline in FL is the FedAvg algorithm proposed by [1].

Original: Communication efficiency in FL of upmost importance [2] and FedAvg reduces communicationcosts by allowing clients to train multiple iterations successively.
Revised: Communication efficiency in FL of upmost importance [2] and FedAvg reduces communication costs by allowing clients to train multiple iterations successively.

Original: Convergence of FedAvg has beenwidely studied for both i.i.d [20, 21] and non i.i.d settings [22].
Revised: Convergence of FedAvg has been widely studied for both i.i.d [18, 19] and non i.i.d settings [20].

Original: Under non i.i.d settings, convergencedeteriorates as a function of increasing heterogeneity [16].
Revised: Under non i.i.d settings, convergence deteriorates as a function of increasing heterogeneity [16].

Original: [19] studied approaches for introducingadaptive algorithms into the server updates to accelerate FL algorithms.
Revised: [17] studied approaches for introducing adaptive algorithms into the server updates to accelerate FL algorithms.

Original: Non-Heterogeneous Data Partitions and Client Drift
Revised: Non-Heterogeneous Data Partitions and Client Drift

Original: One significant challenge of decentralizeddata is client sample heterogeneity.
Revised: One significant challenge of decentralized data is client sample heterogeneity.

Original: Different local distributions create differing local objectives,when clients progress too far towards minimizing their own objectives, local models drift apart.
Revised: Different local distributions create differing local objectives, when clients progress too far towards minimizing their own objectives, local models drift apart.

Original: Thisdegrades performance of the shared global model and slows convergence[17, 23, 3].
Revised: This degrades performance of the shared global model and slows convergence[21, 22, 3].

Original: Attempts toalleviate client drift include approaches centered around knowledge distillation like [24] and [25]who ensemble information about the global data distribution and disseminate it to clients via modelstrained at the server.
Revised: Attempts to alleviate client drift include approaches centered around knowledge distillation like [23] and [24] who ensemble information about the global data distribution and disseminate it to clients via models trained at the server.

Original: These methods possess the added risk of privacy attacks and while [24] takesteps to mitigate this risk, their method requires the existence of an unlabeled dataset which maynot be practical in all settings.
Revised: These methods possess the added risk of privacy attacks and while [23] take steps to mitigate this risk, their method requires the existence of an unlabeled dataset which may not be practical in all settings.

Original: Other approaches attempt to regularize updates at the client level, [3]propose SCAFFOLD, an algorithm to control client drift by use of control norms modifying clientgradients.
Revised: Other approaches attempt to regularize updates at the client level, [3] propose SCAFFOLD, an algorithm to control client drift by use of control norms modifying client gradients.

Original: [22] add a proximal term to the local objective limiting variation in local updates and [26]propose a gradient masking technique that modifies server side update aggregation.
Revised: [20] add a proximal term to the local objective limiting variation in local updates and [25] propose a gradient masking technique that modifies server side update aggregation.

Original: Our methodon the other hand, modifies the objective functions locally, this approach is compatible with anyfederated optimization method in the literature.
Revised: Our method on the other hand, modifies the objective functions locally, this approach is compatible with any federated optimization method in the literature.

Original: Continual Learning CL is a process by which tasks are learned sequentially over a period of time,the learner retains knowledge of previous tasks and leverages that prior knowledge to learn new tasks[27].
Revised: Continual Learning CL is a process by which tasks are learned sequentially over a period of time, the learner retains knowledge of previous tasks and leverages that prior knowledge to learn new tasks [26].

Original: CL is made difficult by the fact that neural networks suffer from catastrophic forgetting, inwhich learning a new task overrides weights learned from past training, degrading performance onpreviously learned tasks [4].
Revised: CL is made difficult by the fact that neural networks suffer from catastrophic forgetting, in which learning a new task overrides weights learned from past training, degrading performance on previously learned tasks [4].

Original: Attempts to mitigate the catastrophic forgetting include architecturebased approaches [8] which attempt to grow or modify an architecture over time to expand knowledge,regularization based methods which attempt to use regularization to solve the problem [5] and anotherclass of methods which take the approach of storing some subset of old data for "rehearsal" [28, 7, 29].
Revised: Attempts to mitigate the catastrophic forgetting include architecture based approaches [8] which attempt to grow or modify an architecture over time to expand knowledge, regularization based methods which attempt to use regularization to solve the problem [5] and another class of methods which take the approach of storing some subset of old data for "rehearsal" [27, 7, 28].

Original: A federated CL setting in which each client continuously collects data was proposed by [30], thisdiffers from our work which considers the standard FL setting where each client maintains a fixed setof data and draws connections to a notion of forgetting across clients to motivate a modification ofthe loss function.
Revised: A federated CL setting in which each client continuously collects data was proposed by [29], this differs from our work which considers the standard FL setting where each client maintains a fixed set of data and draws connections to a notion of forgetting across clients to motivate a modification of the loss function.

Original: [31] have used ideas from CL to propose FedCurv based on the EWC algorithm [5]from CL, FedCurv requires sending additional information and is not compatible with all FL methods.
Revised: [30] have used ideas from CL to propose FedCurv based on the EWC algorithm [5] from CL, FedCurv requires sending additional information and is not compatible with all FL methods.

Original: Along this line [32] proposed an approach inspired by rehearsal methods, generating pseudo dataand adding an additional regularization term.
Revised: Along this line [31] proposed an approach inspired by rehearsal methods, generating pseudo data and adding an additional regularization term.

Original: This requires an expensive pseudo data generatingprocedure that increases local training time.
Revised: This requires an expensive pseudo data generating procedure that increases local training time.

Original: Normalization in FL Batch normalization (BN) is a commonly employed machine learning tacticthat normalizes features by the mean and variance computed across a mini batch.
Revised: Normalization in FL Batch normalization (BN) is a commonly employed machine learning tactic that normalizes features by the mean and variance computed across a mini batch.

Original: It has been shownto help with generalization and stabilize optimization [33].
Revised: It has been shown to help with generalization and stabilize optimization [32].

Original: A limitation of BN is that under certainconditions, the mean and standard deviation used at test time may differ significantly from those usedin training.
Revised: A limitation of BN is that under certain conditions, the mean and standard deviation used at test time may differ significantly from those used in training.

Original: Scenarios including small batch sizes or non-i.i.d batch distributions such as those inheterogeneous FL have been noted to suffer from significant performance degradation when using
Revised: Scenarios including small batch sizes or non-i.i.d batch distributions such as those in heterogeneous FL have been noted to suffer from significant performance degradation when using

Original: [Table]

Original: of FedAVG

Original: with BN are not included since this setting is known to yield poor performance [19], howerver, TCE combined with BN based models is competitive with group norm.

Original: NC indicates cases for which the algorithm did not converge

Original: BN [19].
Revised: BN [17].

Original: Group normalization (GN) is not dependent on batch statistics, and is effective at mitigatingthe effect of model performance degradation induced by skewed data partitions and small batch sizes[18, 34].
Revised: Group normalization (GN) is not dependent on batch statistics, and is effective at mitigating the effect of model performance degradation induced by skewed data partitions and small batch sizes [33, 34].

Original: Recent works using FedAvg and any of its variants typically avoid using BN [35].
Revised: Recent works using FedAvg and any of its variants typically avoid using BN [35].

Original: A.2 Evaluation of Hyperparameter Robustness
Revised: A.2 Evaluation of Hyperparameter Robustness

Original: In Table A.2 we further confirm the robustness of FedAvg+TCE to learning rate.
Revised: In Table A.2 we further confirm the robustness of FedAvg+TCE to learning rate.

Original: For each datasetused in the experiments we indicate accuracies of models trained using a different learning ratethat fall within 2% , 3% and 5% of the best accuracy obtained in training.
Revised: For each dataset used in the experiments we indicate accuracies of models trained using a different learning rate that fall within 2% , 3% and 5% of the best accuracy obtained in training.

Original: The best learning rate isindicated in column 2 for convenience.
Revised: The best learning rate is indicated in column 2 for convenience.

Original: [Table]
Revised: [Table]

Revised: of FedAvg with BN

Revised: are not included since this setting is known to yield poor performance [17], howerver

Revised: [Table]

Revised: TCE combined with BN based models is competitive with group norm.

Revised: NC indicates cases for which the algorithm did not converge

Original:  A.3 Additional Forgetting studies
Revised: [Table] A.3 Additional Forgetting studies

Original: In Figures4, 5, 6 we show additional results for other rounds of training for forgetting.
Revised: In Figures 5, 6, 7 we show additional results for other rounds of training for forgetting.

Original: Figure 4 isevaluated after the first round of training, Figure 6 is evaluated after the 4000 th round of trainingand Figure 5 is evaluated right in the middle of training, after the 2000 th round.
Revised: Figure 5 is evaluated after the first round of training, Figure 7 is evaluated after the 4000 th round of training and Figure 6 is evaluated right in the middle of training, after the 2000 th round.

Original: We observe invery early rounds since performance is still very low for many client datasets, there is not as muchaccuracy to destroy, however we still observe several cases where initial accuracy of the model issubstantial enough that forgetting is observably more extreme without TCE.
Revised: We observe in very early rounds since performance is still very low for many client datasets, there is not as much accuracy to destroy, however we still observe several cases where initial accuracy of the model is substantial enough that forgetting is observably more extreme without TCE.

Original: By the middle of trainingat round 2000, we see clear indications of forgetting, the bottom row of Figure 5 corresponding to
Revised: By the middle of training at round 2000, we see clear indications of forgetting, the bottom row of Figure 6 corresponding to

Original: FedAvg without TCE shows substantially better performance on its own dataset (indicated along thediagonal values).
Revised: FedAvg without TCE shows substantially better performance on its own dataset (indicated along the diagonal values).

Original: At the completion of training (Figure 6) we see FedAvg+TCE doing much betterthan FedAvg at overcoming the local client forgetting problem.
Revised: At the completion of training (Figure 7) we see FedAvg+TCE doing much better than FedAvg at overcoming the local client forgetting problem.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure][Table] [Figure] [Figure]
Revised: [Figure]