Original: F ED M IX : A PPROXIMATION OF M IXUP UNDER M EAN A UGMENTED F EDERATED L EARNING
Revised: F ED M IX : A PPROXIMATION OF M IXUP UNDER M EAN A UGMENTED F EDERATED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device, thus preserving privacy and eliminating the need to store data globally.
Revised: Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device, thus preserving privacy and eliminating the need to store data globally.

Original: While there are promising results under the assumption of independent and identically distributed (iid) local data, current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases.
Revised: While there are promising results under the assumption of independent and identically distributed (iid) local data, current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases.

Original: To resolve this issue, we propose a simple framework, Mean Augmented Federated Learning (MAFL) , where clients send and receive averaged local data, subject to the privacy requirements of target applications.
Revised: To resolve this issue, we propose a simple framework, Mean Augmented Federated Learning (MAFL) , where clients send and receive averaged local data, subject to the privacy requirements of target applications.

Original: Under our framework, we propose a new augmentation algorithm, named FedMix , which is inspired by a phenomenal yet simple data augmentation method, Mixup, but does not require local raw data to be directly shared among devices.
Revised: Under our framework, we propose a new augmentation algorithm, named FedMix , which is inspired by a phenomenal yet simple data augmentation method, Mixup, but does not require local raw data to be directly shared among devices.

Original: Our method shows greatly improved performance in the standard benchmark datasets of FL, under highly non-iid federated settings, compared to conventional algorithms.
Revised: Our method shows greatly improved performance in the standard benchmark datasets of FL, under highly non-iid federated settings, compared to conventional algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: As we enter the era of edge computing, more data is being collected directly from edge devices such as mobile phones, vehicles, facilities, and so on.
Revised: As we enter the era of edge computing, more data is being collected directly from edge devices such as mobile phones, vehicles, facilities, and so on.

Original: By decoupling the ability to learn from the delicate process of merging sensitive personal data, Federated learning (FL) proposes a paradigm that allows a global neural network to learn to be trained collaboratively from individual clients without directly accessing the local data of other clients, thus preserving the privacy of each client (Kone ˇ cn ´ y et al., 2016; McMahan et al., 2017).
Revised: By decoupling the ability to learn from the delicate process of merging sensitive personal data, Federated learning (FL) proposes a paradigm that allows a global neural network to learn to be trained collaboratively from individual clients without directly accessing the local data of other clients, thus preserving the privacy of each client (Kone ˇ cn ´ y et al., 2016; McMahan et al., 2017).

Original: Federated learning lets clients do most of the computation using its local data, with the global server only aggregating and updating the model parameters based on those sent by clients.
Revised: Federated learning lets clients do most of the computation using its local data, with the global server only aggregating and updating the model parameters based on those sent by clients.

Original: One of the standard and most widely used algorithm for federated learning is FedAvg (McMahan et al., 2017), which simply averages model parameters trained by each client in an element-wise manner, weighted proportionately by the size of data used by clients.
Revised: One of the standard and most widely used algorithm for federated learning is FedAvg (McMahan et al., 2017), which simply averages model parameters trained by each client in an element-wise manner, weighted proportionately by the size of data used by clients.

Original: FedProx (Li et al., 2020b) is a variant of FedAvg that adds a proximal term to the objective function of clients, improving statistical stability of the training process.
Revised: FedProx (Li et al., 2020b) is a variant of FedAvg that adds a proximal term to the objective function of clients, improving statistical stability of the training process.

Original: While several other methods have been proposed until recently (Mohri et al., 2019; Yurochkin et al., 2019; Wang et al., 2020), they all build on the idea that updated model parameters from clients are averaged in certain manners.
Revised: While several other methods have been proposed until recently (Mohri et al., 2019; Yurochkin et al., 2019; Wang et al., 2020), they all build on the idea that updated model parameters from clients are averaged in certain manners.

Original: Although conceptually it provides an ideal learning environment for edge devices, the federated learning still has some practical challenges that prevent the widespread application of it (Li et al., 2020a; Kairouz et al., 2019).
Revised: Although conceptually it provides an ideal learning environment for edge devices, the federated learning still has some practical challenges that prevent the widespread application of it (Li et al., 2020a; Kairouz et al., 2019).

Original: Among such challenges, the one that we are interested in this paper is the heterogeneity of the data, as data is distributed non-iid across clients in many real-world settings; in other words, each local client data is not fairly drawn from identical underlying distribution.
Revised: Among such challenges, the one that we are interested in this paper is the heterogeneity of the data, as data is distributed non-iid across clients in many real-world settings; in other words, each local client data is not fairly drawn from identical underlying distribution.

Original: Since each client will learn from different data distributions, it becomes harder for the model to be trained efﬁciently, as reported in (McMahan et al., 2017).
Revised: Since each client will learn from different data distributions, it becomes harder for the model to be trained efﬁciently, as reported in (McMahan et al., 2017).

Original: While theoretical evidence on the convergence of
Revised: While theoretical evidence on the convergence of

Original: FedAvg with non-iid case has recently been shown in (Li et al., 2020c), efﬁcient algorithms suitable for this setting have not yet been developed or systematically examined despite some efforts (Zhao et al., 2018; Hsieh et al., 2020).
Revised: FedAvg with non-iid case has recently been shown in (Li et al., 2020c), efﬁcient algorithms suitable for this setting have not yet been developed or systematically examined despite some efforts (Zhao et al., 2018; Hsieh et al., 2020).

Original: In addition to non-iid problem, another important issue is that updating model parameters individually trained by each client is very costly and becomes even heavier as the model complexity increases.
Revised: In addition to non-iid problem, another important issue is that updating model parameters individually trained by each client is very costly and becomes even heavier as the model complexity increases.

Original: Some existing works Smith et al.
Revised: Some existing works Smith et al.

Original: (2016); Sattler et al.
Revised: (2016); Sattler et al.

Original: (2019) target this issue to decrease
Revised: (2019) target this issue to decrease

Original: [Figure]
Revised: [Figure]

Original: the amount of communications while maintaining the performance of FedAvg.
Revised: the amount of communications while maintaining the performance of FedAvg.

Original: A more practical approach to reduce communication cost is to selectively update individual models at each round, rather than having all clients participate in parameter updates.
Revised: A more practical approach to reduce communication cost is to selectively update individual models at each round, rather than having all clients participate in parameter updates.

Original: This partial participation of clients per round hardly affects test performance in ideal iid settings but it can exacerbate the heterogeneity of weight updates across clients and as a result, the issue of non-iid (McMahan et al., 2017).
Revised: This partial participation of clients per round hardly affects test performance in ideal iid settings but it can exacerbate the heterogeneity of weight updates across clients and as a result, the issue of non-iid (McMahan et al., 2017).

Original: In order to mitigate the heterogeneity across clients while protecting privacy, we provide a novel yet simple framework, mean augmented federated learning (MAFL), in which each client exchanges the updated model parameters as well as its mashed (or averaged) data.
Revised: In order to mitigate the heterogeneity across clients while protecting privacy, we provide a novel yet simple framework, mean augmented federated learning (MAFL), in which each client exchanges the updated model parameters as well as its mashed (or averaged) data.

Original: MAFL framework allows the trade-off between the amount of meaningful information exchanged and the privacy across clients, depending on several factors such as the number of data instances used in computing the average.
Revised: MAFL framework allows the trade-off between the amount of meaningful information exchanged and the privacy across clients, depending on several factors such as the number of data instances used in computing the average.

Original: We ﬁrst introduce a naive approach in our framework that simply applies Mixup (Zhang et al., 2018) between local data and averaged external data from other clients to reduce a myopic bias.
Revised: We ﬁrst introduce a naive approach in our framework that simply applies Mixup (Zhang et al., 2018) between local data and averaged external data from other clients to reduce a myopic bias.

Original: Here, we go further in our framework and ask the following seemingly impossible question: can only averaged data in our framework that has lost most of the discriminative information, bring the similar effect as a global Mixup in which clients directly access others’ private data without considering privacy issues?
Revised: Here, we go further in our framework and ask the following seemingly impossible question: can only averaged data in our framework that has lost most of the discriminative information, bring the similar effect as a global Mixup in which clients directly access others’ private data without considering privacy issues?

Original: Toward this, we introduce our second and more important approach in our framework, termed Federated Mixup (FedMix), that simply approximates the loss function of global Mixup via Taylor expansion (it turns out that such approximation only involves the averaged data from other clients!).
Revised: Toward this, we introduce our second and more important approach in our framework, termed Federated Mixup (FedMix), that simply approximates the loss function of global Mixup via Taylor expansion (it turns out that such approximation only involves the averaged data from other clients!).

Original: Figure 1 brieﬂy describes the concept of our methods.
Revised: Figure 1 brieﬂy describes the concept of our methods.

Original: We validate our method on standard benchmark datasets for federated learning, and show its effectiveness against the standard federated learning methods especially for non-iid settings.
Revised: We validate our method on standard benchmark datasets for federated learning, and show its effectiveness against the standard federated learning methods especially for non-iid settings.

Original: In particular, we claim that FedMix shows better performance and smaller drop in accuracy with more heterogeneity or fewer clients update per communication round, further increasing difﬁculty of federated learning.
Revised: In particular, we claim that FedMix shows better performance and smaller drop in accuracy with more heterogeneity or fewer clients update per communication round, further increasing difﬁculty of federated learning.

Original: Our contribution is threefold:
Revised: Our contribution is threefold:

Original: • We propose a simple framework for federated learning that averages and exchanges each local data.
Revised: • We propose a simple framework for federated learning that averages and exchanges each local data.

Original: Even naive approach in this framework performing Mixup with other clients’ mashed data shows performance improvement over existing baselines on several settings.
Revised: Even naive approach in this framework performing Mixup with other clients’ mashed data shows performance improvement over existing baselines on several settings.

Original: • We further develop a novel approximation for insecure global Mixup accessing other clients’ local data, and ﬁnd out that Taylor expansion of global Mixup only involves the averaged data from other clients.
Revised: • We further develop a novel approximation for insecure global Mixup accessing other clients’ local data, and ﬁnd out that Taylor expansion of global Mixup only involves the averaged data from other clients.

Original: Based on this observation, we propose FedMix in our framework approximating global Mixup without accessing others’ raw data.
Revised: Based on this observation, we propose FedMix in our framework approximating global Mixup without accessing others’ raw data.

Original: • We validate FedMix on several FL benchmark datasets especially focusing on non-iid data settings where our method signiﬁcantly outperforms existing baselines while still preserving privacy with minimal increases in communication cost.
Revised: • We validate FedMix on several FL benchmark datasets especially focusing on non-iid data settings where our method signiﬁcantly outperforms existing baselines while still preserving privacy with minimal increases in communication cost.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Federated learning Federated learning was ﬁrst proposed in Koneˇcn´y et al.
Revised: Federated learning Federated learning was ﬁrst proposed in Koneˇcn´y et al.

Original: (2016) where the prevalent asynchronous SGD (Dean et al., 2012) is used to update a global model in a distributed fashion.
Revised: (2016) where the prevalent asynchronous SGD (Dean et al., 2012) is used to update a global model in a distributed fashion.

Original: A pioneering work in this ﬁeld proposed the currently most widely used algorithm, FedAvg (McMahan et al., 2017), which is also the ﬁrst synchronous algorithm dedicated to federated setting.
Revised: A pioneering work in this ﬁeld proposed the currently most widely used algorithm, FedAvg (McMahan et al., 2017), which is also the ﬁrst synchronous algorithm dedicated to federated setting.

Original: Shortly after, Li et al.
Revised: Shortly after, Li et al.

Original: (2020b) proposed a variant of FedAvg, named FedProx, where the authors claimed to overcome statistical heterogeneity and increase stability in federated learning.
Revised: (2020b) proposed a variant of FedAvg, named FedProx, where the authors claimed to overcome statistical heterogeneity and increase stability in federated learning.

Original: Recent studies attempt to expand federated learning with the aim of providing learning in more diverse and practical environments such as multi-task learning (Smith et al., 2017), generative models (Augenstein et al., 2020), continual learning (Yoon et al., 2020), semi-supervised learning (Jeong et al., 2020), and data with noisy labels (Tuor et al., 2020).
Revised: Recent studies attempt to expand federated learning with the aim of providing learning in more diverse and practical environments such as multi-task learning (Smith et al., 2017), generative models (Augenstein et al., 2020), continual learning (Yoon et al., 2020), semi-supervised learning (Jeong et al., 2020), and data with noisy labels (Tuor et al., 2020).

Original: Our paper focuses on general federated settings, but it could be considered in such various situations.
Revised: Our paper focuses on general federated settings, but it could be considered in such various situations.

Original: However, these algorithms may obtain suboptimal performance when clients participating in FL have non-iid (Zhao et al., 2018; Hsieh et al., 2020) distributions.
Revised: However, these algorithms may obtain suboptimal performance when clients participating in FL have non-iid (Zhao et al., 2018; Hsieh et al., 2020) distributions.

Original: While the convergence of FedAvg on such settings was initially shown by experiments in McMahan et al.
Revised: While the convergence of FedAvg on such settings was initially shown by experiments in McMahan et al.

Original: (2017) and later proved in
Revised: (2017) and later proved in

Original: Li et al.
Revised: Li et al.

Original: (2020c)
Revised: (2020c)

Original: , it does not guarantee performance as good as it would have been for iid setting.
Revised: , it does not guarantee performance as good as it would have been for iid setting.

Original: Existing algorithms that pointed out this issue have major limitations, such as privacy violation by partial global sharing of local data (Zhao et al., 2018) or no indication of improvement over baseline algorithms such as FedAvg (Hsieh et al., 2020).
Revised: Existing algorithms that pointed out this issue have major limitations, such as privacy violation by partial global sharing of local data (Zhao et al., 2018) or no indication of improvement over baseline algorithms such as FedAvg (Hsieh et al., 2020).

Original: Our method aims to improve performance particularly on these non-iid situations, without compromising privacy.
Revised: Our method aims to improve performance particularly on these non-iid situations, without compromising privacy.

Original: Mixup Mixup (Zhang et al., 2018) is a popular data augmentation technique that generates additional data by linear interpolation between actual data instances.
Revised: Mixup Mixup (Zhang et al., 2018) is a popular data augmentation technique that generates additional data by linear interpolation between actual data instances.

Original: Mixup has been usually applied to image classiﬁcation tasks and shown to improve test accuracy on various datasets such as CIFAR and ImageNet-2012 (Russakovsky et al., 2015), and, on popular architectures such as ResNet (He et al., 2016) and ResNeXt (Xie et al., 2017), for various model complexity.
Revised: Mixup has been usually applied to image classiﬁcation tasks and shown to improve test accuracy on various datasets such as CIFAR and ImageNet-2012 (Russakovsky et al., 2015), and, on popular architectures such as ResNet (He et al., 2016) and ResNeXt (Xie et al., 2017), for various model complexity.

Original: It is also reported in
Revised: It is also reported in

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018) that Mixup helps with stability, adversarial robustness (Zhang et al., 2018), calibration, and predictive certainty (Thulasidasan et al., 2019).
Revised: (2018) that Mixup helps with stability, adversarial robustness (Zhang et al., 2018), calibration, and predictive certainty (Thulasidasan et al., 2019).

Original: Mixup is expanding from various angles due to its simplicity and popularity.
Revised: Mixup is expanding from various angles due to its simplicity and popularity.

Original: First, beyond image classiﬁcation tasks, its effectiveness has been proven in various domains such as image segmentation (Eaton-Rosen et al., 2020), speech recognition (Warden, 2018), and natural language processing (Guo et al., 2019).
Revised: First, beyond image classiﬁcation tasks, its effectiveness has been proven in various domains such as image segmentation (Eaton-Rosen et al., 2020), speech recognition (Warden, 2018), and natural language processing (Guo et al., 2019).

Original: Also, several extensions such as Manifold Mixup (Verma et al., 2018), which performs Mixup in latent space, or
Revised: Also, several extensions such as Manifold Mixup (Verma et al., 2018), which performs Mixup in latent space, or

Original: CutMix (Yun et al., 2019), which replaces speciﬁc regions with others patches, have been proposed.
Revised: CutMix (Yun et al., 2019), which replaces speciﬁc regions with others patches, have been proposed.

Original: In most of the previous studies on federated learning, Mixup was partially (or locally) used as a general data augmentation technique.
Revised: In most of the previous studies on federated learning, Mixup was partially (or locally) used as a general data augmentation technique.

Original: Some recent studies (Oh et al., 2020; Shin et al., 2020) proposed to send blended data to server using Mixup, but they require sending
Revised: Some recent studies (Oh et al., 2020; Shin et al., 2020) proposed to send blended data to server using Mixup, but they require sending

Original: locally- and linearly-mixed (mostly from two instances) data to server at every round, therefore being susceptible to privacy issues with huge communication costs.
Revised: locally- and linearly-mixed (mostly from two instances) data to server at every round, therefore being susceptible to privacy issues with huge communication costs.

Original: Our work properly modiﬁes Mixup under the restrictions of federated learning and mitigates the major challenges of federated learning such as non-iid clients.
Revised: Our work properly modiﬁes Mixup under the restrictions of federated learning and mitigates the major challenges of federated learning such as non-iid clients.

Original:  M EAN A UGMENTED F EDERATED L EARNING (MAFL) AND F ED M IX
Revised:  M EAN A UGMENTED F EDERATED L EARNING (MAFL) AND F ED M IX

Original: We now provide our framework exchanging averaged data for federated learning and main method approximating insecure global Mixup under our framework, after brieﬂy introducing the setup.
Revised: We now provide our framework exchanging averaged data for federated learning and main method approximating insecure global Mixup under our framework, after brieﬂy introducing the setup.

Original: 3. S ETUP AND B ACKGROUND
Revised: 3. S ETUP AND B ACKGROUND

Original: Federated learning and FedAvg Federated Averaging (FedAvg) (McMahan et al., 2017) has been the most popular algorithmic framework for federated learning.
Revised: Federated learning and FedAvg Federated Averaging (FedAvg) (McMahan et al., 2017) has been the most popular algorithmic framework for federated learning.

Original: For every communication round t = 0 , . . .
Revised: For every communication round t = 0 , . . .

Original: , T − 1 , a client k ∈ 1 , . . .
Revised: , T − 1 , a client k ∈ 1 , . . .

Original: , N selected for local training sends back its model w kt (or only difference to reduce communication cost) to a global server.
Revised: , N selected for local training sends back its model w kt (or only difference to reduce communication cost) to a global server.

Original: For every round, K number of clients are selected to locally update and send model parameters.
Revised: For every round, K number of clients are selected to locally update and send model parameters.

Original: The server simply averages parameters received, so that the global model w t after t rounds of communications becomes w t
Revised: The server simply averages parameters received, so that the global model w t after t rounds of communications becomes w t

Original: = (cid:80) nk =1 p k w kt where p k is the importance of client k based on the relative number of data in k among all selected clients at t .
Revised: = (cid:80) nk =1 p k w kt where p k is the importance of client k based on the relative number of data in k among all selected clients at t .

Original: The updated global model is sent back to clients for the next round, which undergoes the following E local updates via stochastic gradient descent (SGD): w kt +1 ,i +1 ← w kt
Revised: The updated global model is sent back to clients for the next round, which undergoes the following E local updates via stochastic gradient descent (SGD): w kt +1 ,i +1 ← w kt

Original: +1 ,i − η t +1 ∇ (cid:96) ( f ( x ki ; w t +1 ,i ) , y ki ) for i = 0 , 1 , . . .
Revised: +1 ,i − η t +1 ∇ (cid:96) ( f ( x ki ; w t +1 ,i ) , y ki ) for i = 0 , 1 , . . .

Original: , E − 1 , batch size B , and local learning rate η .
Revised: , E − 1 , batch size B , and local learning rate η .

Original: Here, (cid:96) is the loss function for learning and f ( x ; w t ) is the model output for input x given model weight w t .
Revised: Here, (cid:96) is the loss function for learning and f ( x ; w t ) is the model output for input x given model weight w t .

Original: Mixup Mixup (Zhang et al., 2018) is a simple data augmentation technique using a linear interpolation between two input-label pairs ( x i , y i ) and ( x j , y j ) to augment ˜ x
Revised: Mixup Mixup (Zhang et al., 2018) is a simple data augmentation technique using a linear interpolation between two input-label pairs ( x i , y i ) and ( x j , y j ) to augment ˜ x

Original: = λ x i + (1 − λ ) x j and ˜ y
Revised: = λ x i + (1 − λ ) x j and ˜ y

Original: = λy
Revised: = λy

Original: i + (1 − λ )
Revised: i + (1 − λ )

Original: y j .
Revised: y j .

Original: The variable λ ∈ [0 , 1] is a hyperparameter that is chosen from the beta distribution for each training step.
Revised: The variable λ ∈ [0 , 1] is a hyperparameter that is chosen from the beta distribution for each training step.

Original: 3. MAFL: M EAN A UGMENTED F EDERATED L EARNING
Revised: 3. MAFL: M EAN A UGMENTED F EDERATED L EARNING

Original: The most obvious and powerful way for local models to receive information about data from other clients is to simply receive raw individual data.
Revised: The most obvious and powerful way for local models to receive information about data from other clients is to simply receive raw individual data.

Original: However, under typical federated setting, each client does not have direct access to individual external data due to privacy constraints, leading to overall performance degradation.
Revised: However, under typical federated setting, each client does not have direct access to individual external data due to privacy constraints, leading to overall performance degradation.

Original: We propose a federated learning framework that relaxes the limitation of accessing others’ raw data and allows a more granular level of privacy depending on applications.
Revised: We propose a federated learning framework that relaxes the limitation of accessing others’ raw data and allows a more granular level of privacy depending on applications.

Original: In our new framework, termed mean augmented federated learning (MAFL), clients not only exchange model parameters but also its mashed (or averaged) data.
Revised: In our new framework, termed mean augmented federated learning (MAFL), clients not only exchange model parameters but also its mashed (or averaged) data.

Original: In MAFL, only the part that exchanges averaged data of each client has been added to the standard
Revised: In MAFL, only the part that exchanges averaged data of each client has been added to the standard

Original: FL paradigm (Algorithm 1).
Revised: FL paradigm (Algorithm 1).

Original: Here, the number of data instances used in computing the average,
Revised: Here, the number of data instances used in computing the average,

Original: M k , controls the key features of MAFL such as privacy and communication costs.
Revised: M k , controls the key features of MAFL such as privacy and communication costs.

Original: Lower M k value results in more relevant information passed over, but only in cost of less secure privacy and larger communication cost.
Revised: Lower M k value results in more relevant information passed over, but only in cost of less secure privacy and larger communication cost.

Original: In one extreme of M k = 1 , raw data is thoroughly exchanged and privacy is not protected at all, which is clearly inappropriate for FL.
Revised: In one extreme of M k = 1 , raw data is thoroughly exchanged and privacy is not protected at all, which is clearly inappropriate for FL.

Original: But, in the other extreme, all data of each client is averaged to ensure a considerable degree of privacy.
Revised: But, in the other extreme, all data of each client is averaged to ensure a considerable degree of privacy.

Original: In addition, it also has an advantage on communication cost; each client sends a set of n k /M
Revised: In addition, it also has an advantage on communication cost; each client sends a set of n k /M

Original: k averaged data where n k is local data size of client k .
Revised: k averaged data where n k is local data size of client k .

Original: The remaining question is whether it is possible to improve performance even when exchanging information that is averaged from all local data and loses discriminative characteristics.
Revised: The remaining question is whether it is possible to improve performance even when exchanging information that is averaged from all local data and loses discriminative characteristics.

Original: The most naive way we can consider in our MAFL framework is to directly use the mashed data from other clients, just like regular local data.
Revised: The most naive way we can consider in our MAFL framework is to directly use the mashed data from other clients, just like regular local data.

Original: However, since mashed data has a lot less usable
Revised: However, since mashed data has a lot less usable

Original: Input: D k = { X k , Y k } for k = 1 , . .
Revised: Input: D k = { X k , Y k } for k = 1 , . .

Original: N
Revised: N

Original: M k : number of data instances used for computing average ¯ x , ¯ y
Revised: M k : number of data instances used for computing average ¯ x , ¯ y

Original: Initialize w 0 for global server for t = 0 , . .
Revised: Initialize w 0 for global server for t = 0 , . .

Original: , T − 1 do for client k with updated local data do
Revised: , T − 1 do for client k with updated local data do

Original: Split local data into M k sized batches
Revised: Split local data into M k sized batches

Original: Compute ¯ x , ¯ y for each batch
Revised: Compute ¯ x , ¯ y for each batch

Original: Send all ¯ x , ¯ y to server end S t ← K clients selected at random
Revised: Send all ¯ x , ¯ y to server end S t ← K clients selected at random

Original: Send w t to clients k ∈ S t if updated then
Revised: Send w t to clients k ∈ S t if updated then

Original: Aggregate all ¯ x , ¯ y to X g , Y g
Revised: Aggregate all ¯ x , ¯ y to X g , Y g

Original: Send X g , Y g to clients k ∈ S t end for k ∈ S t do w kt +1 ← LocalUpdate ( k, w t ; X g , Y g ) end w t +1 ← 1 K (cid:80) k ∈ S t p k w kt + end
Revised: Send X g , Y g to clients k ∈ S t end for k ∈ S t do w kt +1 ← LocalUpdate ( k, w t ; X g , Y g ) end w t +1 ← 1 K (cid:80) k ∈ S t p k w kt + end

Original: LocalUpdate ( k, w t ; X g , Y g ) under
Revised: LocalUpdate ( k, w t ; X g , Y g ) under

Original: MAFL (Algorithm 1): w ← w t for e = 0 , . . .
Revised: MAFL (Algorithm 1): w ← w t for e = 0 , . . .

Original: , E − 1 do
Revised: , E − 1 do

Original: Split D k into batches of size B for batch ( X , Y ) do
Revised: Split D k into batches of size B for batch ( X , Y ) do

Original: Select an entry x g , y g from
Revised: Select an entry x g , y g from

Original: X g , Y g (cid:96) 1 = (
Revised: X g , Y g (cid:96) 1 = (

Original: [Equation]
Revised: [Equation]

Original: λ ) (cid:96) (cid:0) f
Revised: λ ) (cid:96) (cid:0) f

Original: ((1 − λ )
Revised: ((1 − λ )

Original: X ; w ) , Y (cid:1) (cid:96) 2 =
Revised: X ; w ) , Y (cid:1) (cid:96) 2 =

Original: [Equation]
Revised: [Equation]

Original: (cid:0) f
Revised: (cid:0) f

Original: [Equation]
Revised: [Equation]

Original: λ ) X ; w ) , y g (cid:1) (cid:96)
Revised: λ ) X ; w ) , y g (cid:1) (cid:96)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: · x
Revised: · x

Original: [Equation]
Revised: [Equation]

Original: (derivative calculated at x = (1 − λ )
Revised: (derivative calculated at x = (1 − λ )

Original: x i and y =
Revised: x i and y =

Original: y i for each of x
Revised: y i for each of x

Original: i , y i in X , Y ) (cid:96)
Revised: i , y i in X , Y ) (cid:96)

Original: [Equation]
Revised: [Equation]

Original: (cid:96)
Revised: (cid:96)

Original: [Equation]
Revised: [Equation]

Original: (cid:96)
Revised: (cid:96)

Original: [Equation]
Revised: [Equation]

Original: w ← w −
Revised: w ← w −

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: ∇ (cid:96) end endreturn w information than local data, we can think of a method of mixing it with local data:
Revised: ∇ (cid:96) end endreturn w information than local data, we can think of a method of mixing it with local data:

Original: [Equation]
Revised: [Equation]

Original: where ( x i , y i ) is an entry from local data and (¯ x j , ¯ y j ) corresponds to means of (inputs,labels) from other client j .
Revised: where ( x i , y i ) is an entry from local data and (¯ x j , ¯ y j ) corresponds to means of (inputs,labels) from other client j .

Original: Note that Eq.
Revised: Note that Eq.

Original: (1) can be understood as the generalization of the loss of directly using the mashed data mentioned above in the sense that such loss can be achieved if λ in Eq.
Revised: (1) can be understood as the generalization of the loss of directly using the mashed data mentioned above in the sense that such loss can be achieved if λ in Eq.

Original: (1) is set deterministically to 0 and 1 .
Revised: (1) is set deterministically to 0 and 1 .

Original: In the experimental section, we conﬁrm the effectiveness of MAFL using (cid:96) NaiveMix .
Revised: In the experimental section, we conﬁrm the effectiveness of MAFL using (cid:96) NaiveMix .

Original: However, in the next subsection, we will show how to achieve better performance by approximating the global
Revised: However, in the next subsection, we will show how to achieve better performance by approximating the global

Original: Mixup in a more systematical way in our MAFL framework.
Revised: Mixup in a more systematical way in our MAFL framework.

Original: 3. F ED M IX : A PPROXIMATING GLOBAL M IXUP VIA INPUT DERIVATIVE
Revised: 3. F ED M IX : A PPROXIMATING GLOBAL M IXUP VIA INPUT DERIVATIVE

Original: We now provide our main approach in the MAFL framework that aims to approximate the effect of global Mixup only using averaged data from other clients.
Revised: We now provide our main approach in the MAFL framework that aims to approximate the effect of global Mixup only using averaged data from other clients.

Original: Consider some client i with its local data ( x i , y i ) .
Revised: Consider some client i with its local data ( x i , y i ) .

Original: It is not allowed in federated learning, but let us assume that client i has access to client j ’s local data ( x j , y j ) .
Revised: It is not allowed in federated learning, but let us assume that client i has access to client j ’s local data ( x j , y j ) .

Original: Then, client i would leverage ( x j , y j ) to improve the performance of its local model especially in non-iid settings by augmenting additional data via Mixup:
Revised: Then, client i would leverage ( x j , y j ) to improve the performance of its local model especially in non-iid settings by augmenting additional data via Mixup:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: If Mixup rate λ is 1, ( x j , y j ) from client j is again directly used like a regular local data, and it would be much more efﬁcient than indirect update of local models through the server.
Revised: If Mixup rate λ is 1, ( x j , y j ) from client j is again directly used like a regular local data, and it would be much more efﬁcient than indirect update of local models through the server.

Original: The essence of our method is to approximate the loss function (cid:96) (cid:0) f
Revised: The essence of our method is to approximate the loss function (cid:96) (cid:0) f

Original: (˜ x ) , ˜ y (cid:1) for the augmented data from Eq.
Revised: (˜ x ) , ˜ y (cid:1) for the augmented data from Eq.

Original: (2), with Taylor expansion for the ﬁrst argument x .
Revised: (2), with Taylor expansion for the ﬁrst argument x .

Original: Speciﬁcally, we derive the following proposition:
Revised: Speciﬁcally, we derive the following proposition:

Original: Proposition 1 Consider the loss function of the global Mixup modulo the privacy issues,
Revised: Proposition 1 Consider the loss function of the global Mixup modulo the privacy issues,

Original: [Equation]
Revised: [Equation]

Original: for cross-entropy loss (cid:96) 1 .
Revised: for cross-entropy loss (cid:96) 1 .

Original: Suppose that Eq.
Revised: Suppose that Eq.

Original: (3) is approximated by applying Taylor series around the place where λ (cid:28) 1 .
Revised: (3) is approximated by applying Taylor series around the place where λ (cid:28) 1 .

Original: Then, if we ignore the second order term (i.e., O ( λ 2 ) ), we obtain the following approximated loss:
Revised: Then, if we ignore the second order term (i.e., O ( λ 2 ) ), we obtain the following approximated loss:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the derivative ∂(cid:96)∂
Revised: where the derivative ∂(cid:96)∂

Original: x is evaluated at x = (1 − λ )
Revised: x is evaluated at x = (1 − λ )

Original: x i and y =
Revised: x i and y =

Original: y i .
Revised: y i .

Original: While Eq.
Revised: While Eq.

Original: (4) still involves x j and y j , invading the privacy of client j , the core value of Propositiongets clearer when mixing up multiple data instances from other clients.
Revised: (4) still involves x j and y j , invading the privacy of client j , the core value of Propositiongets clearer when mixing up multiple data instances from other clients.

Original: Note that the vanilla Mixup is not mixing one speciﬁc instance with other data, but performing augmentations among several random selected data.
Revised: Note that the vanilla Mixup is not mixing one speciﬁc instance with other data, but performing augmentations among several random selected data.

Original: In a non-iid FL environment, we can also expect that the effect will be greater as we create Mixup data by accessing as much private data as possible from other clients.
Revised: In a non-iid FL environment, we can also expect that the effect will be greater as we create Mixup data by accessing as much private data as possible from other clients.

Original: From this point of view, let us assume that client i has received a set of M private instances, J , from client j .
Revised: From this point of view, let us assume that client i has received a set of M private instances, J , from client j .

Original: Then, the global Mixup loss in Eq.
Revised: Then, the global Mixup loss in Eq.

Original: (3) is
Revised: (3) is

Original: [Equation]
Revised: [Equation]

Original: and the approximated FedMix loss in Proposition 1 becomes
Revised: and the approximated FedMix loss in Proposition 1 becomes

Original: [Equation]
Revised: [Equation]

Original: where we utilize the linearity of Equation 4 in terms of x j and y j , and ¯ x j and ¯ y j correspond to mean of M inputs and labels in J , respectively.
Revised: where we utilize the linearity of Equation 4 in terms of x j and y j , and ¯ x j and ¯ y j correspond to mean of M inputs and labels in J , respectively.

Original: The algorithmic details are provided in the appendix due to the space constraint (see Algorithm 2 in Appendix A).
Revised: The algorithmic details are provided in the appendix due to the space constraint (see Algorithm 2 in Appendix A).

Original: 3. P RIVACY ISSUES AND ADDITIONAL COSTS OF MAFL
Revised: 3. P RIVACY ISSUES AND ADDITIONAL COSTS OF MAFL

Original: Privacy issues of MAFL MAFL requires exchanging averaged data by construction.
Revised: Privacy issues of MAFL MAFL requires exchanging averaged data by construction.

Original: Even though
Revised: Even though

Original: MAFL exchanges only the limited information allowed by the application, it may causes new types of privacy issues.
Revised: MAFL exchanges only the limited information allowed by the application, it may causes new types of privacy issues.

Original: The potential privacy risk of FL or MAFL is beyond the main scope of our study, but in this section, we brieﬂy discuss some basic privacy issues of MAFL and potential solutions.
Revised: The potential privacy risk of FL or MAFL is beyond the main scope of our study, but in this section, we brieﬂy discuss some basic privacy issues of MAFL and potential solutions.

Original: • There is possibility that local data distribution can be inferred relatively easily from averaged data.
Revised: • There is possibility that local data distribution can be inferred relatively easily from averaged data.

Original: This issue simply arises as M k is not large enough, so that individual data could be inferred from the averaged data easily.
Revised: This issue simply arises as M k is not large enough, so that individual data could be inferred from the averaged data easily.

Original: On the other hand, if n k is not big enough, each entry in X g , Y g could reveal too much about the whole local distribution of the client it came from.
Revised: On the other hand, if n k is not big enough, each entry in X g , Y g could reveal too much about the whole local distribution of the client it came from.

Original: • It could be easy to infer ownership of each entry in X g , Y g , if it contains client-id speciﬁc information.
Revised: • It could be easy to infer ownership of each entry in X g , Y g , if it contains client-id speciﬁc information.

Original: If clients could identify what other client each entry came from, information about local data of that client could be inferred.
Revised: If clients could identify what other client each entry came from, information about local data of that client could be inferred.

Original: • Additional concerns involve identiﬁcation of data by detecting change in exchanged averaged data, in case of continual learning, which involves local data change across time.
Revised: • Additional concerns involve identiﬁcation of data by detecting change in exchanged averaged data, in case of continual learning, which involves local data change across time.

Original: This issue is exacerbated as there is update of averaged data for every minute change on local data, which makes the client receiving X g , Y g easier to infer the changed portion.
Revised: This issue is exacerbated as there is update of averaged data for every minute change on local data, which makes the client receiving X g , Y g easier to infer the changed portion.

Original: One simple suggestion to alleviate this issue would be to only update X g , Y g when there is enough change in local data across enough number of clients, so that such changes are not easily exploitable.
Revised: One simple suggestion to alleviate this issue would be to only update X g , Y g when there is enough change in local data across enough number of clients, so that such changes are not easily exploitable.

Original: • As a way to strengthen privacy protection under MAFL (and possibly to help with issues mentioned above), we in the server can average within entries of X g , Y g .
Revised: • As a way to strengthen privacy protection under MAFL (and possibly to help with issues mentioned above), we in the server can average within entries of X g , Y g .

Original: If this additional average is done across every random m entries at the server, it would effectively provide averaged data across all local data of m clients, but would result in an m -fold decrease in the number of averaged data.
Revised: If this additional average is done across every random m entries at the server, it would effectively provide averaged data across all local data of m clients, but would result in an m -fold decrease in the number of averaged data.

Original: This variant is considered in Appendix J. • In case where the global server is not credential, the averaged data itself should ensure privacy as it is sent to the server.
Revised: This variant is considered in Appendix J. • In case where the global server is not credential, the averaged data itself should ensure privacy as it is sent to the server.

Original: A most obvious concern comes from when M k is not large enough, so that each entry of M k reveals more of information of each individual input.
Revised: A most obvious concern comes from when M k is not large enough, so that each entry of M k reveals more of information of each individual input.

Original: Simply using a sufﬁciently large value of M k can alleviate this issue, although this might result in worse performance.
Revised: Simply using a sufﬁciently large value of M k can alleviate this issue, although this might result in worse performance.

Original: However, for clients whose n k is quite small, there is a limit for M k to be large enough.
Revised: However, for clients whose n k is quite small, there is a limit for M k to be large enough.

Original: One way to alleviate this issue is to introduce a cut-off threshold for allowing clients to send averaged data to server.
Revised: One way to alleviate this issue is to introduce a cut-off threshold for allowing clients to send averaged data to server.

Original: We report the results in Appendix H.
Revised: We report the results in Appendix H.

Original: Communication cost Since MAFL requires sending averaged input data between server and clients, additional communication costs are incurred.
Revised: Communication cost Since MAFL requires sending averaged input data between server and clients, additional communication costs are incurred.

Original: However, it turns out that this additional cost is very small compared to communication cost required for exchanging model parameters.
Revised: However, it turns out that this additional cost is very small compared to communication cost required for exchanging model parameters.

Original: This is mainly due to the fact that input dimension is typically much smaller than number of model parameters.
Revised: This is mainly due to the fact that input dimension is typically much smaller than number of model parameters.

Original: Speciﬁcally, for input dimension d i , exchange of averaged data among N clients incurs 2 Nd i cost (factor of 2 for server receiving and sending the values).
Revised: Speciﬁcally, for input dimension d i , exchange of averaged data among N clients incurs 2 Nd i cost (factor of 2 for server receiving and sending the values).

Original: Meanwhile, the cost for exchange of model parameters is 2 Np m where p m is number of model parameters.
Revised: Meanwhile, the cost for exchange of model parameters is 2 Np m where p m is number of model parameters.

Original: Under typical circumstances, averaged data is only exchanged at the beginning of the ﬁrst communication round, while model parameters have to be exchanged every round.
Revised: Under typical circumstances, averaged data is only exchanged at the beginning of the ﬁrst communication round, while model parameters have to be exchanged every round.

Original: Thus the ratio between the two costs after T communication rounds is d i / ( Tp m ) .
Revised: Thus the ratio between the two costs after T communication rounds is d i / ( Tp m ) .

Original: Since d i (cid:28) p m in general, we consider extra communication burden to be negligible (even in the worst case where we update averaged data every round, the ratio is still d i / ( p m ) .
Revised: Since d i (cid:28) p m in general, we consider extra communication burden to be negligible (even in the worst case where we update averaged data every round, the ratio is still d i / ( p m ) .

Original: FedMix also requires calculation of input derivative term in its loss function, so potentially extra memory is required.
Revised: FedMix also requires calculation of input derivative term in its loss function, so potentially extra memory is required.

Original: We further provide additional computation costs of MAFL in Appendix G.
Revised: We further provide additional computation costs of MAFL in Appendix G.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We test our result on various benchmark datasets with NaiveMix (direct mixup between local data and averaged data) and FedMix, then compare the results with FedAvg (McMahan et al., 2017) and[Figure]
Revised: We test our result on various benchmark datasets with NaiveMix (direct mixup between local data and averaged data) and FedMix, then compare the results with FedAvg (McMahan et al., 2017) and

Original: [Table] FedProx (Li et al., 2020b), as well as other baseline Mixup scenarios.
Revised: FedProx (Li et al., 2020b), as well as other baseline Mixup scenarios.

Original: We create a highly non-iid environment to show our methods excel in such situations
Revised: We create a highly non-iid environment to show our methods excel in such situations.

Revised: [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. E XPERIMENTAL S ETUP
Revised: E XPERIMENTAL S ETUP

Original: Dataset We implement the typical federated setting where clients have their own local data and one centralized server that receives and sends information from/to the clients.
Revised: Dataset We implement the typical federated setting where clients have their own local data and one centralized server that receives and sends information from/to the clients.

Original: We utilize a large number of clients and only utilize partial set of clients chosen each round to locally update.
Revised: We utilize a large number of clients and only utilize partial set of clients chosen each round to locally update.

Original: We used three popular image classiﬁcation benchmark datasets: FEMNIST (Caldas et al., 2019), CIFAR10, and CIFAR100, as well as a popular natural language processing benchmark dataset, Shakespeare.
Revised: We used three popular image classiﬁcation benchmark datasets: FEMNIST (Caldas et al., 2019), CIFAR10, and CIFAR100, as well as a popular natural language processing benchmark dataset, Shakespeare.

Original: See Appendix B for more details about dataset, models, and hyperparameters used.
Revised: See Appendix B for more details about dataset, models, and hyperparameters used.

Original: We introduce data size heterogeneity for FEMNIST dataset: each client has different size of local data, each from a unique writer.
Revised: We introduce data size heterogeneity for FEMNIST dataset: each client has different size of local data, each from a unique writer.

Original: Meanwhile, we introduce label distribution heterogeneity for CIFAR datasets, with clients having data with only a limited number of classes.
Revised: Meanwhile, we introduce label distribution heterogeneity for CIFAR datasets, with clients having data with only a limited number of classes.

Original: Algorithms We study the performance of FedMix and NaiveMix and compare with FedAvg and
Revised: Algorithms We study the performance of FedMix and NaiveMix and compare with FedAvg and

Original: FedProx.
Revised: FedProx.

Original: We also compare our method against FedAvg with Mixup within local data (labeled LocalMix; see Figure 1(b)), to show whether Mixup within local data is sufﬁcient to allow the model to perform well on external data.
Revised: We also compare our method against FedAvg with Mixup within local data (labeled LocalMix; see Figure 1(b)), to show whether Mixup within local data is sufﬁcient to allow the model to perform well on external data.

Original: To show the effectiveness of FedMix, we also compare our method to the case where we perform direct Mixup with external data (and thus violating privacy, labeled
Revised: To show the effectiveness of FedMix, we also compare our method to the case where we perform direct Mixup with external data (and thus violating privacy, labeled

Original: Global Mixup; see Figure 1(a)).
Revised: Global Mixup; see Figure 1(a)).

Original: 4. P ERFORMANCE OF F ED M IX AND N AIVE M IX ON N ON -IID F EDERATED S ETTINGS
Revised: 4. P ERFORMANCE OF F ED M IX AND N AIVE M IX ON N ON -IID F EDERATED S ETTINGS

Original: We compare the learning curves of each method under the same federated settings, in terms of number of communication rounds conducted.
Revised: We compare the learning curves of each method under the same federated settings, in terms of number of communication rounds conducted.

Original: Comparing MAFL-based algorithms, NaiveMix shows slight performance increases than FedAvg, FedProx, and Localmix, while FedMix outperforms and [Table] [Table]
Revised: Comparing MAFL-based algorithms, NaiveMix shows slight performance increases than FedAvg, FedProx, and Localmix, while FedMix outperforms and shows faster convergence than all of FedAvg, FedProx, Localmix and NaiveMix for all datasets tested as in Figure 2.

Original: [Table]
Revised: While NaiveMix and FedMix is already superior to FedProx, they are parallel to FedProx modiﬁcation and can be applied in conjunction with FedProx.

Original: shows faster convergence than all of FedAvg
Revised: We compare performances across FedProx

Original: [Table]
Revised: [Table]

Original: FedProx, Localmix and NaiveMix for all datasets tested as in Figure 2. While NaiveMix and FedMix is already superior to FedProx, they are parallel to FedProx modiﬁcation and can be applied in conjunction with FedProx.
Revised: [Table]

Original: We compare performances across FedProx variants of various Mixup algorithms in Table 1.
Revised: [Table] variants of various Mixup algorithms in Table 1.

Original: FedMix outperforms vanilla FedProx for various datasets, although they do fall short of default version of FedMix used for the main experiment.
Revised: FedMix outperforms vanilla FedProx for various datasets, although they do fall short of default version of FedMix used for the main experiment

Revised: [Table]

Original: To conﬁrm whether received information is properly incorporated, we compare FedMix with possible Mixup scenarios under MAFL.
Revised: To conﬁrm whether received information is properly incorporated, we compare FedMix with possible Mixup scenarios under MAFL.

Original: We show the results in Appendix D.
Revised: We show the results in Appendix D.

Original: While Mixup is usually performed for image classiﬁcation tasks, it could be applied for language models.
Revised: While Mixup is usually performed for image classiﬁcation tasks, it could be applied for language models.

Original: For language datasets, since Mixup cannot be performed on input, we perform Mixup on embeddings (for a detailed explanation of Mixup between hidden states, see Appendix E).
Revised: For language datasets, since Mixup cannot be performed on input, we perform Mixup on embeddings (for a detailed explanation of Mixup between hidden states, see Appendix E).

Original: When tested on Shakespeare dataset, FedMix and NaiveMix both show better performance than baseline algorithms (Table 2).
Revised: When tested on Shakespeare dataset, FedMix and NaiveMix both show better performance than baseline algorithms (Table 2).

Original: Note that for this task, LocalMix has the lowest performance, and global Mixup does not result in the superior performance above federated algorithms as expected.
Revised: Note that for this task, LocalMix has the lowest performance, and global Mixup does not result in the superior performance above federated algorithms as expected.

Original: We think Mixup does not provide performance boost for this speciﬁc task, but claim that MAFL algorithms still result in better performance compared to FedAvg.
Revised: We think Mixup does not provide performance boost for this speciﬁc task, but claim that MAFL algorithms still result in better performance compared to FedAvg.

Original: We also claim that FedMix is superior compared to other methods under various settings, in terms of varying number of clients ( N ) and varying number of local data per clients.
Revised: We also claim that FedMix is superior compared to other methods under various settings, in terms of varying number of clients ( N ) and varying number of local data per clients.

Original: We observe superior performance of FedMix compared to other algorithms for all settings (see Tables 3 and 4).
Revised: We observe superior performance of FedMix compared to other algorithms for all settings (see Tables 3 and 4).

Original: We also vary the number of local epochs ( E ) between global updates, and still observe that FedMix outperforms other methods (see Appendix F).
Revised: We also vary the number of local epochs ( E ) between global updates, and still observe that FedMix outperforms other methods (see Appendix F).

Original: FedMix compared to global Mixup with ﬁxed mixup ratio Since FedMix approximates loss function of global Mixup for ﬁxed value of λ (cid:28) 1 , we can evaluate the efﬁciency of approximation by comparing between FedMix and a global Mixup scenario with ﬁxed λ value.
Revised: FedMix compared to global Mixup with ﬁxed mixup ratio Since FedMix approximates loss function of global Mixup for ﬁxed value of λ (cid:28) 1 , we can evaluate the efﬁciency of approximation by comparing between FedMix and a global Mixup scenario with ﬁxed λ value.

Original: Table 5 shows varying performance between global Mixup and FedMix under various values of λ .
Revised: Table 5 shows varying performance between global Mixup and FedMix under various values of λ .

Original: As λ increases,
Revised: As λ increases,

Original: Mixup data reﬂects more of the features of external data, resulting in better performance in case of global Mixup.
Revised: Mixup data reﬂects more of the features of external data, resulting in better performance in case of global Mixup.

Original: However, this also results in our approximation being much less accurate, and we indeed observe performance of FedMix decreasing instead.
Revised: However, this also results in our approximation being much less accurate, and we indeed observe performance of FedMix decreasing instead.

Original: The result shows that the hyperparameter [Table] [Figure] [Table]
Revised: The result shows that the hyperparameter λ should be chosen to balance between better Mixup and better approximation.

Original: [Figure] [Table] [Figure] [Table]
Revised: However, it seems that high λ results in signiﬁcant decrease in both methods, probably due to external data (which is out-of-distribution for local distribution) being overrepresented during local update.

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table][Figure] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: λ should be chosen to balance between better Mixup and better approximation.

Original: However, it seems that high λ results in signiﬁcant decrease in both methods, probably due to external data (which is out-of-distribution for local distribution) being overrepresented during local update.

Original: Effect of M k to compute mean In our algorithm, we chose to calculate X g , Y g with all local data for each client.
Revised: Effect of M k to compute mean In our algorithm, we chose to calculate X g , Y g with all local data for each client.

Original: To observe a potential effect of M k , we varied M k used to compute the averaged data that is sent from other clients.
Revised: To observe a potential effect of M k , we varied M k used to compute the averaged data that is sent from other clients.

Original: Inevitably, reducing M k will result in X g , Y g having much more rows, imposing additional computation burden and less preservation of privacy.
Revised: Inevitably, reducing M k will result in X g , Y g having much more rows, imposing additional computation burden and less preservation of privacy.

Original: In general, for both
Revised: In general, for both

Original: FEMNIST and CIFAR10, there is only small performance decline as privacy is enhanced, as can be seen in Figure 3.
Revised: FEMNIST and CIFAR10, there is only small performance decline as privacy is enhanced, as can be seen in Figure 3.

Original: We show that using all local data to calculate each mean is sufﬁcient to both preserve privacy and still have good performance.
Revised: We show that using all local data to calculate each mean is sufﬁcient to both preserve privacy and still have good performance.

Original: Mixup between hidden states Manifold Mixup (Verma et al., 2018) was proposed to show improvements over input Mixup (Zhang et al., 2018) in various image classiﬁcation tasks such as
Revised: Mixup between hidden states Manifold Mixup (Verma et al., 2018) was proposed to show improvements over input Mixup (Zhang et al., 2018) in various image classiﬁcation tasks such as

Original: CIFAR10, CIFAR100, and SVHN.
Revised: CIFAR10, CIFAR100, and SVHN.

Original: We discuss the possibilities and implications of applying Mixup between hidden states in Appendix E.
Revised: We discuss the possibilities and implications of applying Mixup between hidden states in Appendix E.

Original: In summary, we show that variants of using hidden states do not show meaningful advances over FedMix using input Mixup, suggesting that in general, it is relatively inefﬁcient since it imposes additional communication burden.
Revised: In summary, we show that variants of using hidden states do not show meaningful advances over FedMix using input Mixup, suggesting that in general, it is relatively inefﬁcient since it imposes additional communication burden.

Original: Effect of non-iid-ness and client participation We claim that our method is efﬁcient when faced with non-iid federated settings.
Revised: Effect of non-iid-ness and client participation We claim that our method is efﬁcient when faced with non-iid federated settings.

Original: For example, our setting of CIFAR10 having only data from 2 classes per client is very non-iid, as in average a pair of clients share only roughly 20% of data distribution.
Revised: For example, our setting of CIFAR10 having only data from 2 classes per client is very non-iid, as in average a pair of clients share only roughly 20% of data distribution.

Original: We test settings for CIFAR10 where clients have data from greater number of classes, and while there is little difference for iid (10 class/client) setting, we observe that FedMix outperform other methods and suffer less from increased heterogeneity from highly non-iid settings (Table 6).
Revised: We test settings for CIFAR10 where clients have data from greater number of classes, and while there is little difference for iid (10 class/client) setting, we observe that FedMix outperform other methods and suffer less from increased heterogeneity from highly non-iid settings (Table 6).

Original: In addition, we also observe less decline and better performance for MAFL-based algorithms, FedMix in particular, as we train less number of clients per round, reducing communication burden in cost of performance (Table 7).
Revised: In addition, we also observe less decline and better performance for MAFL-based algorithms, FedMix in particular, as we train less number of clients per round, reducing communication burden in cost of performance (Table 7).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed MAFL, a novel framework, that exchanges averaged local data, to gain relevant information while still ensuring privacy.
Revised: We proposed MAFL, a novel framework, that exchanges averaged local data, to gain relevant information while still ensuring privacy.

Original: Under the new framework, we ﬁrst suggested NaiveMix, which is a naive implementation of Mixup between local and received data.
Revised: Under the new framework, we ﬁrst suggested NaiveMix, which is a naive implementation of Mixup between local and received data.

Original: More interestingly, we proposed FedMix, which provides approximation of global Mixup only using averaged data.
Revised: More interestingly, we proposed FedMix, which provides approximation of global Mixup only using averaged data.

Original: MAFL, and FedMix in particular, showed improved performance over existing algorithms in various benchmarks, particularly in non-iid environments where each client has data distributed heterogeneously.
Revised: MAFL, and FedMix in particular, showed improved performance over existing algorithms in various benchmarks, particularly in non-iid environments where each client has data distributed heterogeneously.

Original: While our method is very effective and still preserving privacy, future work needs to be done to deal with various non-iid environments, desirably with better privacy and beyond image classiﬁcation tasks. 
Revised: While our method is very effective and still preserving privacy, future work needs to be done to deal with various non-iid environments, desirably with better privacy and beyond image classiﬁcation tasks. A A LGORITHMS

Revised: We present a brief depiction of FedAvg in Algorithm 3.

Original: B E XPERIMENTAL D ETAILS
Revised: Algorithm Input: N, T, K, E, B, p k , D k = X k , Y k } , k = 1 , . .

Revised: . , N, η t , t = 0 , . .

Revised: , T −

Revised: Initialize w 0 for global server for t = 0 , . .

Revised: , T − 1 do S t ← K clients selected at random

Revised: Send w t to clients k ∈ S t for k ∈ S t do w kt +1 ← LocalUpdate ( k, w t ) end w t +1 ← 1 K (cid:80) k ∈ S t p k w kt + end

Revised: LocalUpdate ( k, w t ) : w ← w t for e = 0 , . . .

Revised: , E − 1 do

Revised: Split D k into batches of size B for batch ( X , Y ) do w ← w − η t +1 ∇ (cid:96) ( f ( X ; w ) , Y ) end endreturn

Original:  FEMNIST FEMNIST is EMNIST (Cohen et al., 2017), a handwritten MNIST dataset, organized into federated setting, as in Caldas et al.
Revised: w B E XPERIMENTAL D ETAILS FEMNIST FEMNIST is EMNIST (Cohen et al., 2017), a handwritten MNIST dataset, organized into federated setting, as in Caldas et al.

Original: EMNIST is very similar to MNIST, but has several differences.
Revised: EMNIST is very similar to MNIST, but has several differences.

Original: It includes all 26 capital and small letters of alphabet as classes along with numbers, making it 62 classes in total to classify.
Revised: It includes all 26 capital and small letters of alphabet as classes along with numbers, making it 62 classes in total to classify.

Original: Also, each image contains information of the writer of the letter.
Revised: Also, each image contains information of the writer of the letter.

Original: In a realistic non-iid setting, each client has local data consists of only one writer, which is about 200 to 300 samples per client in average, with differing number of samples.
Revised: In a realistic non-iid setting, each client has local data consists of only one writer, which is about 200 to 300 samples per client in average, with differing number of samples.

Original: We use N =clients and trained only K = 10 clients per communication round.
Revised: We use N =clients and trained only K = 10 clients per communication round.

Original: We used LeNet-5 (Lecun et al., 1998) architecture for client training.
Revised: We used LeNet-5 (Lecun et al., 1998) architecture for client training.

Original: LeNet is consisted of 2 conv layers followed by 2x2 maxpool layer then 3 fc layers.
Revised: LeNet is consisted of 2 conv layers followed by 2x2 maxpool layer then 3 fc layers.

Original: We used 5x5 conv layers with 6 andchannels.
Revised: We used 5x5 conv layers with 6 andchannels.

Original: Following fc layers have exactly the same hidden dimension of original LeNet-5 model.
Revised: Following fc layers have exactly the same hidden dimension of original LeNet-5 model.

Original: CIFAR10 and CIFAR100 CIFAR10 and CIFAR100 are very popular and simple image classiﬁcation datasets for federated setting.
Revised: CIFAR10 and CIFAR100 CIFAR10 and CIFAR100 are very popular and simple image classiﬁcation datasets for federated setting.

Original: Both contain 50,000 training data and 10,000 test data.
Revised: Both contain 50,000 training data and 10,000 test data.

Original: We split the data into each client, N = 60 in case of CIFAR10 and N = 100 in case of CIFAR100.
Revised: We split the data into each client, N = 60 in case of CIFAR10 and N = 100 in case of CIFAR100.

Original: To create an artiﬁcial non-iid environment, we allocate data such that each client only has data from 2 (20 for
Revised: To create an artiﬁcial non-iid environment, we allocate data such that each client only has data from 2 (20 for

Original: CIFAR100) randomly chosen classes.
Revised: CIFAR100) randomly chosen classes.

Original: We use a modiﬁed version of VGG architecture (Simonyan &

Original: Zisserman, 2015), so that it has 6 convolutional layers (more details in Appendix B).

Original: We train only K = 15 clients per round for CIFAR10 and K = 10 for CIFAR100.
Revised: We train only K = 15 clients per round for CIFAR10 and K = 10 for CIFAR100.

Original: No validation data was split and we used all training data for local training.
Revised: No validation data was split and we used all training data for local training.

Original: We used modiﬁed version of VGG architecture (Simonyan & Zisserman, 2015).
Revised: We used modiﬁed version of VGG architecture (Simonyan & Zisserman, 2015).

Original: Modiﬁed VGGnet is consisted of 6 convolutional layers with 3 max pooling layers.
Revised: Modiﬁed VGGnet is consisted of 6 convolutional layers with 3 max pooling layers.

Original: 3x3 conv layers are stacked and 2x maxpool layer is stacked after every 2 conv layers.
Revised: 3x3 conv layers are stacked and 2x maxpool layer is stacked after every 2 conv layers.

Original: Conv layers have channel sizes of 32, 64, 128, 128, 256, 256.
Revised: Conv layers have channel sizes of 32, 64, 128, 128, 256, 256.

Original: Then 3 fc layers are stacked with hidden dimension 512.
Revised: Then 3 fc layers are stacked with hidden dimension 512.

Original: We use Dropout layer three times with probability 0.1 after the second, third maxpool layers and before the last fc layer.
Revised: We use Dropout layer three times with probability 0.1 after the second, third maxpool layers and before the last fc layer.

Original: We remove all batch normalization layers since it is reported that they hurt federated learning performance (Hsieh et al., 2020).
Revised: We remove all batch normalization layers since it is reported that they hurt federated learning performance (Hsieh et al., 2020).

Original: Shakespeare We use dataset from The Complete Works of William Shakespeare , which is a popular dataset for next-character prediction task.
Revised: Shakespeare We use dataset from The Complete Works of William Shakespeare , which is a popular dataset for next-character prediction task.

Original: We partition the dataset so that each client has conversations of one speaking role, as in Caldas et al.
Revised: We partition the dataset so that each client has conversations of one speaking role, as in Caldas et al.

Original: (2019), which naturally results in a heterogeneous setting, as in FEMNIST.
Revised: (2019), which naturally results in a heterogeneous setting, as in FEMNIST.

Original: We use N = K = 60 , each with different number of data (minimum is 200).
Revised: We use N = K = 60 , each with different number of data (minimum is 200).

Original: Since input-level Mixup cannot be performed for discrete character labels, we performed
Revised: Since input-level Mixup cannot be performed for discrete character labels, we performed

Original: Mixup on the embedding layer.
Revised: Mixup on the embedding layer.

Original: Additional concerns for this variation is considered at Appendix E.
Revised: Additional concerns for this variation is considered at Appendix E.

Original: We used 2-layer LSTM, both with hidden dimension of 256.
Revised: We used 2-layer LSTM, both with hidden dimension of 256.

Original: The recurret network is followed by an embedding layer.
Revised: The recurret network is followed by an embedding layer.

Original: The output of LSTM is passed to a fully-connected layer, with softmax output of one node per character.
Revised: The output of LSTM is passed to a fully-connected layer, with softmax output of one node per character.

Original: There are 84 characters used in the dataset.
Revised: There are 84 characters used in the dataset.

Original: [Table]

Original: Local clients are trained by SGD optimizer with learning rate 0.01 and learning decay rate per round 0.999.
Revised: Local clients are trained by SGD optimizer with learning rate 0.01 and learning decay rate per round 0.999.

Original: We set local batch size as 10 for training.
Revised: We set local batch size as 10 for training.

Original: Speciﬁc hyperparameter setting for each dataset is  explained in following Table 8.
Revised: Speciﬁc hyperparameter setting for each dataset is [Table] explained in following Table 8.

Original: Throughout the experiment, M k is ﬁxed to local client’s dataset size.
Revised: Throughout the experiment, M k is ﬁxed to local client’s dataset size.

Original: Changes in these parameters are indicated, if made, are stated for all experiments.
Revised: Changes in these parameters are indicated, if made, are stated for all experiments.

Original: Note that we use a ﬁxed small value of λ for MAFL-based algorithms to show superior performance.
Revised: Note that we use a ﬁxed small value of λ for MAFL-based algorithms to show superior performance.

Original: C P ROOF OF P ROPOSITION
Revised: C P ROOF OF P ROPOSITION

Original: We demonstrate mathematical proof of Proposition 1 for FedMix.
Revised: We demonstrate mathematical proof of Proposition 1 for FedMix.

Original: Starting from Eq.
Revised: Starting from Eq.

Original: (3), since the loss function is linear in y for cross-entropy loss (cid:96) 2 , we have (cid:96)(cid:0) f
Revised: [Table] since the loss function is linear in y for cross-entropy loss (cid:96) 2 , we have (cid:96)

Original: (˜ x ) , ˜ y (cid:1) =

Original: (1 − λ ) (cid:96) (cid:16) f

Original: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: Unlike the original paper

Original: [Equation]
Revised: [Table]

Original: + λ [Equation] j (cid:1) , y i (cid:17) + λ(cid:96) (cid:16) f (cid:0) (1 −
Revised: if we assume λ (cid:28)

Original: [Equation]
Revised: [Table]

Original: x i + λ x j (cid:1) , y j (cid:17) .

Original: Unlike the original paper, if we assume λ (cid:28) 1 , we can treat this loss as objective loss function for vicinal risk minimization (VRM).
Revised: we can treat this loss as objective loss function for vicinal risk minimization (VRM).

Original: Under this assumption, each term in Eq.
Revised: Under this assumption [Table] each term in Eq.

Original: (6) can be approximated by independent Taylor expansion on the ﬁrst and second argument of (cid:96) , so that we have
Revised: (6) can be approximated by independent Taylor expansion on the ﬁrst and second argument of (cid:96) , so that we have

Original: [Equation]
Revised: [Equation]

Original: Since λ (cid:28) 1 , we can ignore the last term in the second row of Eq.
Revised: Since λ (cid:28) 1 , we can ignore the last term in the second row of Eq.

Original: (7), which is O ( λ 2 ) .
Revised: (7), which is O ( λ 2 ) .

Original: We simplify this equation and switch the second term in the ﬁrst row and the ﬁrst term in the second row, to ﬁnally obtain
Revised: We simplify this equation and switch the second term in the ﬁrst row and the ﬁrst term in the second row, to ﬁnally obtain

Original: [Equation]
Revised: [Equation]

Original: The derivative ∂(cid:96)∂ x is calculated at x = (1 − λ )
Revised: The derivative ∂(cid:96) ∂ x is calculated at x = (1 − λ )

Original: x i and y =
Revised: x i and y =

Original: y i .
Revised: y i .

Original: The coefﬁcient of the last term is changed to λ from λ (1 − λ ) since we are ignoring O ( λ 2 ) terms.
Revised: The coefﬁcient of the last term is changed to λ from λ (1 − λ ) since we are ignoring O ( λ 2 ) terms.

Original: D C OMPARISON OF F ED M IX WITH B ASELINE M IXUP S CENARIOS
Revised: D C OMPARISON OF F ED M IX WITH B ASELINE M IXUP S CENARIOS

Original: Since our MAFL-based algorithms could be considered as VRM, it could be considered as a data augmentation (Chapelle et al., 2000).
Revised: Since our MAFL-based algorithms could be considered as VRM, it could be considered as a data augmentation (Chapelle et al., 2000).

Original: Thus, it is important to conﬁrm that increased performance from MAFL not only comes from data augmentation but also from relevant information received from other clients.
Revised: Thus, it is important to conﬁrm that increased performance from MAFL not only comes from data augmentation but also from relevant information received from other clients.

Original: To check whether this is true, we compare FedMix with algorithms where we use either randomly generated noises as averaged data for Mixup (labeled Mixup w/ random noise) and where we use only locally averaged data for Mixup (labeled Mixup w/ local means).
Revised: To check whether this is true, we compare FedMix with algorithms where we use either randomly generated noises as averaged data for Mixup (labeled Mixup w/ random noise) and where we use only locally averaged data for Mixup (labeled Mixup w/ local means).

Original: In Table 9, we observe that if averaged data for MAFL is substituted for randomly generated noise or locally generated images, it does not show the level of performance FedMix is able to show.
Revised: In Table 9, we observe that if averaged data for MAFL is substituted for randomly generated noise or locally generated images, it does not show the level of performance FedMix is able to show.

Original: Thus, we claim that FedMix properly incorporates relevant information received.
Revised: Thus, we claim that FedMix properly incorporates relevant information received.

Original: [Figure]
Revised: [Figure]

Original: E V ARIANT OF F ED M IX A ND N AIVE M IX WITH M IXUP BETWEEN H IDDEN S TATES
Revised: E V ARIANT OF F ED M IX A ND N AIVE M IX WITH M IXUP BETWEEN H IDDEN S TATES

Original: While input Mixup methods promise signiﬁcant enhancements, one can expect similar performance from hidden state Mixup, originally proposed by Verma et al.
Revised: While input Mixup methods promise signiﬁcant enhancements, one can expect similar performance from hidden state Mixup, originally proposed by Verma et al.

Original: The authors of this work suggest that Manifold Mixup demonstrates similar, if not greater, advantage in terms of performance and adversarial robustness.
Revised: The authors of this work suggest that Manifold Mixup demonstrates similar, if not greater, advantage in terms of performance and adversarial robustness.

Original: We can think of variants of FedMix and NaiveMix that implement hidden state Mixup, and test if this variant outperforms vanilla methods based on input Mixup.
Revised: We can think of variants of FedMix and NaiveMix that implement hidden state Mixup, and test if this variant outperforms vanilla methods based on input Mixup.

Original: Although the original paper proposed randomizing the layer k just before hidden states that undergo
Revised: Although the original paper proposed randomizing the layer k just before hidden states that undergo

Original: Mixup for each batch, we propose setting this layer k constant.
Revised: Mixup for each batch, we propose setting this layer k constant.

Original: This is to reduce communication cost signiﬁcantly, since selecting randomized layer for Mixup will require other clients having to send multiple hidden states (which usually have large dimensions), further imposing communication burden.
Revised: This is to reduce communication cost signiﬁcantly, since selecting randomized layer for Mixup will require other clients having to send multiple hidden states (which usually have large dimensions), further imposing communication burden.

Original: Another change is that while original Manifold Mixup (Verma et al., 2018) suggests backpropagating the entire computational graph through the whole network, including the encoder (part of model projecting input to designated hidden representation) and the decoder (rest of the model projecting hidden representation to output).
Revised: Another change is that while original Manifold Mixup (Verma et al., 2018) suggests backpropagating the entire computational graph through the whole network, including the encoder (part of model projecting input to designated hidden representation) and the decoder (rest of the model projecting hidden representation to output).

Original: Such thing is impossible to do in typical federated setting, since the computational graph to calculate hidden representation of local data and the graph to calculate hidden representation of other clients’ data is separated, and they cannot be updated simultaneously through local update (doing so requires communicating encoder weights across clients every local update, which is highly inefﬁcient in terms of communication).
Revised: Such thing is impossible to do in typical federated setting, since the computational graph to calculate hidden representation of local data and the graph to calculate hidden representation of other clients’ data is separated, and they cannot be updated simultaneously through local update (doing so requires communicating encoder weights across clients every local update, which is highly inefﬁcient in terms of communication).

Original: Thus, during Mixup between hidden representations, only the decoder weights can be updated, since only updating encoder of the selected local client will desynchronize encoder weight values for calculating hidden states of local data from those for calculating hidden states of other clients’ data every local update, so that the model does not learn properly.
Revised: Thus, during Mixup between hidden representations, only the decoder weights can be updated, since only updating encoder of the selected local client will desynchronize encoder weight values for calculating hidden states of local data from those for calculating hidden states of other clients’ data every local update, so that the model does not learn properly.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: To compensate for this downside, we propose performing vanilla SGD updates without Mixup after
Revised: To compensate for this downside, we propose performing vanilla SGD updates without Mixup after

Original: Manifold Mixup SGD (which updates weights only in decoder).
Revised: Manifold Mixup SGD (which updates weights only in decoder).

Original: The vanilla updates will have both local encoder and decoder weights to be updated, thus driving the model to have better hidden representations for Mixup.
Revised: The vanilla updates will have both local encoder and decoder weights to be updated, thus driving the model to have better hidden representations for Mixup.

Original: However, this difference not only imposes additional computation cost, but also does not guarantee that it will show better performance compared to input Mixup methods.
Revised: However, this difference not only imposes additional computation cost, but also does not guarantee that it will show better performance compared to input Mixup methods.

Original: While utilizing hidden representations from other clients sound like a safe idea, it does not ensure data privacy, primarily because the updating client has knowledge of the exact encoder weight values used to calculate hidden states received, and based on our modiﬁcation, the received hidden states are treated as constants during Mixup.
Revised: While utilizing hidden representations from other clients sound like a safe idea, it does not ensure data privacy, primarily because the updating client has knowledge of the exact encoder weight values used to calculate hidden states received, and based on our modiﬁcation, the received hidden states are treated as constants during Mixup.

Original: Model inversion attacks (Fredrikson et al., 2015) have been suggested to recover input images from hidden states or outputs, with access to weight values.
Revised: Model inversion attacks (Fredrikson et al., 2015) have been suggested to recover input images from hidden states or outputs, with access to weight values.

Original: Thus, direct Mixup between hidden states does not guarantee data privacy.
Revised: Thus, direct Mixup between hidden states does not guarantee data privacy.

Original: Variant of FedMix and NaiveMix can be applied during decoder training phase, so that privacy is ensured while we successfully approximate Mixup.
Revised: Variant of FedMix and NaiveMix can be applied during decoder training phase, so that privacy is ensured while we successfully approximate Mixup.

Original: The performance of proposed algorithms is shown in Figure 4 on CIFAR10 (same settings with main experiment for dataset, model, and training is used; see Appendix B).
Revised: The performance of proposed algorithms is shown in Figure 4 on CIFAR10 (same settings with main experiment for dataset, model, and training is used; see Appendix B).

Original: Comparison between methods in Figure 4(a) shows that while variants of FedMix and NaiveMix show improved performance over existing methods, they still do not outperform our method based on input Mixup (compare with dotted line).
Revised: Comparison between methods in Figure 4(a) shows that while variants of FedMix and NaiveMix show improved performance over existing methods, they still do not outperform our method based on input Mixup (compare with dotted line).

Original: Meanwhile, comparison between using different layers for Mixup is shown in Figure 4(b).
Revised: Meanwhile, comparison between using different layers for Mixup is shown in Figure 4(b).

Original: It is shown that k = 4 has fastest learning curve but converges similarly to case of k = 2 , both being slightly outperformed by case of input Mixup.
Revised: It is shown that k = 4 has fastest learning curve but converges similarly to case of k = 2 , both being slightly outperformed by case of input Mixup.

Original: Considering additional computation burden required to communicate hidden states (which often have larger dimensions than raw input) and necessity to communicate the hidden states every communication round (since hidden representations change with encoder weights), we propose that FedMix using input Mixup is superior, and use this method for our main analyses.
Revised: Considering additional computation burden required to communicate hidden states (which often have larger dimensions than raw input) and necessity to communicate the hidden states every communication round (since hidden representations change with encoder weights), we propose that FedMix using input Mixup is superior, and use this method for our main analyses.

Original: F E FFECT OF L OCAL E POCHS
Revised: F E FFECT OF L OCAL E POCHS

Original: Previous works (McMahan et al., 2017; Caldas et al., 2019) show that number of local epochs,
Revised: Previous works (McMahan et al., 2017; Caldas et al., 2019) show that number of local epochs,

Original: E , affects federated learning performance.
Revised: E , affects federated learning performance.

Original: We tested the effect of E on CIFAR10.
Revised: We tested the effect of E on CIFAR10.

Original: In general, we showed that test performance increases as E increases.
Revised: In general, we showed that test performance increases as E increases.

Original: In addition, we observed that under various values of E , FedMix shows the best performance compared to other algorithms (see Table 10), being a close second after NaiveMix for E = 10 .
Revised: In addition, we observed that under various values of E , FedMix shows the best performance compared to other algorithms (see Table 10), being a close second after NaiveMix for E = 10 .

Original: MAFL-based algorithms outperform existing algorithms for all values of E tested.
Revised: MAFL-based algorithms outperform existing algorithms for all values of E tested.

Original: G A DDITIONAL C OMPUTATION C OST I NCURRED BY MAFL
Revised: G A DDITIONAL C OMPUTATION C OST I NCURRED BY MAFL

Original: For FedMix, additional computation and memory are required on edge devices during model training for each communication round, since (cid:96) FedMix requires additional terms, including gradient by input,
Revised: For FedMix, additional computation and memory are required on edge devices during model training for each communication round, since (cid:96) FedMix requires additional terms, including gradient by input,

Original: ∂(cid:96)∂ x , compared to vanilla FedAvg.
Revised: ∂(cid:96)∂ x , compared to vanilla FedAvg.

Original: We claim that FedMix does not result in an additional computation burden.
Revised: We claim that FedMix does not result in an additional computation burden.

Original: Speciﬁcally, we trained FedMix on CIFAR10 with the same settings as the main experiment to 70% accuracy in 1.94 hours; FedAvg takes 1.95 hours.
Revised: Speciﬁcally, we trained FedMix on CIFAR10 with the same settings as the main experiment to 70% accuracy in 1.94 hours; FedAvg takes 1.95 hours.

Original: FedMix spends a comparable amount of time to reach a similar level of performance of FedAvg.
Revised: FedMix spends a comparable amount of time to reach a similar level of performance of FedAvg.

Original: While in the memory aspect, FedMix requires about twice more GPU memory allocation compared to FedAvg, this phenomenon is also observed on LocalMix and NaiveMix.
Revised: While in the memory aspect, FedMix requires about twice more GPU memory allocation compared to FedAvg, this phenomenon is also observed on LocalMix and NaiveMix.

Original: The extra memory burden comes from Mixup by enlarging the input dimension twice.
Revised: The extra memory burden comes from Mixup by enlarging the input dimension twice.

Original: For instance, FedAvg requires 46.00MB to allocate, LocalMix requires 94.00MB and 98.00MB for FedMix.
Revised: For instance, FedAvg requires 46.00MB to allocate, LocalMix requires 94.00MB and 98.00MB for FedMix.

Original: Calculating gradient of the input derivative gives only negligi- ble 2-3MB additional memory usage, which is reasonable concerning the substantial performance increase from LocalMix to FedMix.
Revised: Calculating gradient of the input derivative gives only negligi- ble 2-3MB additional memory usage, which is reasonable concerning the substantial performance increase from LocalMix to FedMix.

Original: H I NTRODUCTION OF C UT -O FF T HRESHOLD IN MAFL
Revised: H I NTRODUCTION OF C UT -O FF T HRESHOLD IN MAFL

Original: To better ensure privacy, a cut-off threshold that prevents clients with fewer data to send averaged data could be introduced.
Revised: To better ensure privacy, a cut-off threshold that prevents clients with fewer data to send averaged data could be introduced.

Original: We performed this in FEMNIST, since for such procedure to be effective, heterogeneous size of local client data is necessary.
Revised: We performed this in FEMNIST, since for such procedure to be effective, heterogeneous size of local client data is necessary.

Original: We test with N = 300 clients, and introduce different threshold levels to test its efﬁciency.
Revised: We test with N = 300 clients, and introduce different threshold levels to test its efﬁciency.

Original: In addition, we also test with multiple λ values, to see whether threshold level affects optimal value of λ for FedMix.
Revised: In addition, we also test with multiple λ values, to see whether threshold level affects optimal value of λ for FedMix.

Original: We present the results in Table 11.
Revised: We present the results in Table 11.

Original: While threshold does not hugely affect performance, we observe that a moderately small threshold level of 100 results in the best performance.
Revised: While threshold does not hugely affect performance, we observe that a moderately small threshold level of 100 results in the best performance.

Original: We suggest that as the threshold level is heightened, there is less overﬁtting to clients with small size local data, but it also results in a decrease in the number of averaged data received by each client.
Revised: We suggest that as the threshold level is heightened, there is less overﬁtting to clients with small size local data, but it also results in a decrease in the number of averaged data received by each client.

Original: We indeed ﬁnd an appropriate value of threshold that maximizes performance.
Revised: We indeed ﬁnd an appropriate value of threshold that maximizes performance.

Original: In case where there are a different number of data per client, the sensitivity of λ could also be different compared to when all clients have the same number of data.
Revised: In case where there are a different number of data per client, the sensitivity of λ could also be different compared to when all clients have the same number of data.

Original: Results in Table 13 show that there is little change in performance by change in λ , especially compared to Table 5.
Revised: Results in Table 13 show that there is little change in performance by change in λ , especially compared to Table 5.

Original: In addition, an inspection of the performance of a global model on individual test data of clients does not reveal any noticeable pattern by the size of local data (see Table 13).
Revised: In addition, an inspection of the performance of a global model on individual test data of clients does not reveal any noticeable pattern by the size of local data (see Table 13).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: I MAFL IN C ONJUNCTION WITH G AUSSIAN N OISE
Revised: I MAFL IN C ONJUNCTION WITH G AUSSIAN N OISE

Original: With results in Figure 3, we expressed concern with small values of M k causing privacy issues with only a small performance boost, if at all.
Revised: With results in Figure 3, we expressed concern with small values of M k causing privacy issues with only a small performance boost, if at all.

Original: A common practice of introducing additional privacy is adding Gaussian noise.
Revised: A common practice of introducing additional privacy is adding Gaussian noise.

Original: This is a popular method associated with differential privacy (McMahan et al., 2018), but adding noise alone does not guarantee differential privacy, since the noise level should be explicitly linked to differential privacy levels, (cid:15) and δ .
Revised: This is a popular method associated with differential privacy (McMahan et al., 2018), but adding noise alone does not guarantee differential privacy, since the noise level should be explicitly linked to differential privacy levels, (cid:15) and δ .

Original: Addition of artiﬁcial pixel-wise noise will enhance privacy but will result in a quality drop of averaged data.
Revised: Addition of artiﬁcial pixel-wise noise will enhance privacy but will result in a quality drop of averaged data.

Original: While privacy added by noise and privacy from averaging data cannot be directly compared, we can select a noise level which in conjunction with small M k , visually provides data privacy similar to that of maximum M k .
Revised: While privacy added by noise and privacy from averaging data cannot be directly compared, we can select a noise level which in conjunction with small M k , visually provides data privacy similar to that of maximum M k .

Original: Results show that the introduction of Gaussian noise does result in a decline in performance (Table 14), although the decline is very small.
Revised: Results show that the introduction of Gaussian noise does result in a decline in performance (Table 14),although the decline is very small.

Original: Interestingly as noise gets larger as σ = 0 .
Revised: Interestingly as noise gets larger as σ = 0 .

Original: 3 , random noise
Revised: 3 , random noise

Original: [Table] [Table]
Revised: [Table] [Table]

Original: provides an effect as data augmentation and results in a performance increase compared toclean data ( σ = 0 ).
Revised: provides an effect as data augmentation and results in a performance increase compared to σ = 0 .

Original: This experiment is in line with Appendix D. We conclude that introduction of noise in averaged data could provide us with a reasonable alternative to FedMix with large M k .
Revised: This experiment is in line with Appendix D. We conclude that introduction of noise in averaged data could provide us with a reasonable alternative to FedMix with large M k .

Original: While our method does not align directly with differential privacy, we leave as future work how FedMix could be smoothly combined with DP-related methods and how its privacy could be quantiﬁed in terms of differential privacy.
Revised: While our method does not align directly with differential privacy, we leave as future work how FedMix could be smoothly combined with DP-related methods and how its privacy could be quantiﬁed in terms of differential privacy.

Original: J A DDITIONAL E XPERIMENTS : V ARIATIONS OF F ED M IX Averaging within X g , Y g
Revised: J A DDITIONAL E XPERIMENTS : V ARIATIONS OF F ED M IX Averaging within X g , Y g

Original: Further averaging between entries of X g , Y g practically provides an extension of the range of viable M k such that it exceeds n k , in the sense that each averaged data is from multiple clients’ data.
Revised: Further averaging between entries of X g , Y g practically provides an extension of the range of viable M k such that it exceeds n k , in the sense that each averaged data is from multiple clients’ data.

Original: Such a process would also result in fewer data included in X g , Y g , so we tested effect of this procedure on model performance.
Revised: Such a process would also result in fewer data included in X g , Y g , so we tested effect of this procedure on model performance.

Original: Table 15 shows that for m -fold extra averaging, we even observe increase in performance, but it quickly declines as m gets too large.
Revised: Table 15 shows that for m -fold extra averaging, we even observe increase in performance, but it quickly declines as m gets too large.

Original: This method provides an improvement in privacy while even possibly resulting in better performance.
Revised: This method provides an improvement in privacy while even possibly resulting in better performance.

Original: Effect of same-class split for averaging We perform random split of local data for averaging, but an unbalanced split, such as only averaging data with the same class labels, could result in better performance.
Revised: Effect of same-class split for averaging We perform random split of local data for averaging, but an unbalanced split, such as only averaging data with the same class labels, could result in better performance.

Original: We compared between random split and same-class split while keeping M k = 0 .
Revised: We compared between random split and same-class split while keeping M k = 0 .

Original: 5 n k be equal for both methods.
Revised: 5 n k be equal for both methods.

Original: Same-class split resulted in a signiﬁcant decline in performance, and we conclude that there is no advantage of such split over random split that we are using for our main results.
Revised: Same-class split resulted in a signiﬁcant decline in performance, and we conclude that there is no advantage of such split over random split that we are using for our main results.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: NaiveMix with varying Mixup ratio λ
Revised: NaiveMix with varying Mixup ratio λ

Original: We varied Mixup ratio λ for NaiveMix as well.
Revised: We varied Mixup ratio λ for NaiveMix as well.

Original: Results in
Revised: Results in

Original: Table 17 shows that NaiveMix also has an intermediate optimal value of λ .
Revised: Table 17 shows that NaiveMix also has an intermediate optimal value of λ .

Original: The drop in performance for λ = 0 .
Revised: The drop in performance for λ = 0 .

Original: 5 is much more dramatic than for FedMix (see Table 5 for comparison with Global Mixup and FedMix).
Revised: 5 is much more dramatic than for FedMix (see Table 5 for comparison with Global Mixup and FedMix).

Original: We think that NaiveMix loss also suffers as it gives more weight to the averaged data, especially for large M k .
Revised: We think that NaiveMix loss also suffers as it gives more weight to the averaged data, especially for large M k .

Original: Heterogeneity from skewed label distribution Recent papers (Yurochkin et al., 2019; Wang et al., 2020) suggested an alternative heterogeneous environment, which does not limit the number of classes per client but skews label distribution in local data.
Revised: Heterogeneity from skewed label distribution Recent papers (Yurochkin et al., 2019; Wang et al., 2020) suggested an alternative heterogeneous environment, which does not limit the number of classes per client but skews label distribution in local data.

Original: We used a Dirichlet distribution of α = 0 . 2 , 0 .
Revised: We used a Dirichlet distribution of α = 0 . 2 , 0 .

Original: 5 as described by Yurochkin et al.
Revised: 5 as described by Yurochkin et al.

Original: (2019) and Wang et al.
Revised: (2019) and Wang et al.

Original: Results show that
Revised: Results show that

Original: FedMix still outperforms all other algorithms.
Revised: FedMix still outperforms all other algorithms.

Original: We think that such label skewing introduces less heterogeneity compared to our practice of limiting the number of classes per client, but nevertheless,
Revised: We think that such label skewing introduces less heterogeneity compared to our practice of limiting the number of classes per client, but nevertheless,

Original: FedMix is still the most powerful method in terms of performance.
Revised: FedMix is still the most powerful method in terms of performance.

Original: [Table]
Revised: [Table]