Paper: FLea: Improving federated learning on scarce and label-skewed data via  privacy-preserving feature augmentation (ICLR-2024)

Reviewer comment (Novelty):
The authors claim that the proposed method is novel, but many components come from the literature. For example, the authors claim that they have devised a novel feature augmentation method, but λ, a key parameter in the method, is drawn from existing literature.

Author response:
Firstly, we would like to clarify the method regarding feature augmentation. λ was proposed for data augmentation to improve the model generalization (Zhang et al, 2018),  while **we are the first to extend it for feature augmentation in the federated learning setup**. This effectively addresses the local overfitting problem caused by data scarcity. We apologise for the wrong reference regarding the mixup in the previous PDF. Now we have corrected it to:
Zhang, Hongyi, et al. "Mixup: Beyond Empirical Risk Minimization." International Conference on Learning Representations. 2018.
And we revised the context to: ‘‘Inspired by the data augmentation method in the centralized setting (Zhang et al, 2018), we sample the weight $\lambda$  for each data point from a symmetrical Beta distribution (Gupta & Nadarajah, 2004):  $\lambda \sim Beta(a,a)$.   $\lambda \in [0,1]$ controls the strength of interpolation between the local and global feature pairs: A smaller $\lambda$ basically makes the generated sample closer to the local feature while a larger one pushes that to the global feature.”

Although this augmentation method exists, **the novelty of our study lies in the observation that data scarcity will lead to local model over-fitting (refer to Section 3.2) and hence leveraging feature augmentation can effectively address this problem**. Data scarcity is a common issue in FL but its impact is not very understood. We provide a comprehensive analysis of this issue and propose the feature augmentation-based method FLea. As demonstrated by the extensive experiments, our method achieves a significant model accuracy boost compared to the start-of-the-art FL baselines. Moreover,  instead of sharing raw data and implementing data augmentation, we also demonstrate that feature augmentation can better trade off the feature utility and data privacy protection.  Given these results, we believe our contribution to the FL community is significant.

Reviewer comment (Reproducibility):
λ1 and λ2 in Equation 6 are critical parameters, but it is unclear how they are set.

Author response:
In this updated version, we supplement how we identify the hyper-parameters $\lambda_1$ and $\lambda_2$  in the loss function in *Section 5.2 Impact of pyper-parameters*. We present a heuristic search and identify the results considering the model accuracy and the data privacy simultaneously. Finally, it has been observed that the model achieves the best trade-off when $\lambda_1$ ranges from 1 to 2 (refer to Figure 7(a)) and when  $\lambda_2$ ranges from 1 to 4 (refer to Figure 7(b)). Finally, we set $\lambda_1=1$ and $\lambda_2=3$ for all experiments.

Reviewer comment (Experiments):
The paper does not discuss the impact of different model types on the proposed method or specify which layer (l-th) has been learned in the experiments. The evaluation solely relies on one AI model, MobileNet, and one dataset, CIFAR-10, which is not sufficient.

Author response:
As per our submitted code, we implemented our methods for various architectures and datasets, which we did not include before due to the limited spaces. Now, according to your suggestion, we add the results on another data modality-audio with a CNN model. The details can be found in Section 5, and Table 2 has been updated (including the results for UrbanSound8K in the right part), with extensive results and discussions presented in Appendix C.4 (page 17-18). The results show that FLea consistently outperforms the state-of-the-art baselines by 2.59~6.27%. For your convenience, we quote part of the results below:

Table 2 Overall performance comparison. The local data size $|\mathcal{D}_k|$ is as small as $100$ on average.
| Accuracy %       |     |          CIFAR10       |                | |        UrbanSound8K        |                |
| ------ | ----------- | -------------- | -------------- | ------------ | -------------- | -------------- |
|               | Quantity(3) | Dirichlet(0.5) | Dirichlet(0.1) |    Quantity(3)  | Dirichlet(0.5) | Dirichlet(0.1) |
| **FedMix**|   $44.04{\pm1.53}$ |$45.50{\pm 1.88}$ |$    38.13{\pm2.06}$   |$51.56{\pm 0.59}$ |$54.18{\pm 0.62}$  |$43.35{\pm 0.72}$
| **FLea**  |    $47.03{\pm1.01}$|$48.86{\pm1.43}$|$44.40{\pm1.23}$| $     57.73 {\pm 0.51}$ |$59.22 {\pm 0.78}$ |$45.94{\pm 0.77}$

Reviewer comment (Theory):
The statement, "we characterize the distribution of the low-dimensional features from the penultimate layer of local and global models," lacks justification regarding why the penultimate layer is suitable for characterizing low-dimensional features.

Author response:
Deep neural networks usually consist of multiple linear and non-linear layers, making their functioning difficult to explain. It is a common way to divide the model into two parts: the prototype extractor (from the input to the penultimate layer) and the linear classifier (the last fully connected layer) [1,2,3]. The prototypes have lower dimensionality than the features from other intermedia layers, which is more straightforward for visualisation (after dimension reduction). We follow the setup from previous work to carry out our analysis.  We have explained the suitability of this more clearly in the paper.

[1]Tan, Yue, et al. "Fedproto: Federated prototype learning across heterogeneous clients." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 8. 2022.

[2]Li, Xin-Chun, et al. "Fedrs: Federated learning with restricted softmax for label distribution non-iid data." Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.

[3]Guo, Yongxin, et al. "FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reduction." (2023).

Reviewer comment (Writing):
Section 2.2 mentions low-dimensional feature overlapping and classifier bias as main label skew problems, but it does not elaborate on low-dimensional feature overlapping or explicitly define the problem.

Author response:
Section 2.2 delves into insights gathered from prior research. In [3], it was discerned that owing to label skew, "local features, even for data from different classes, are too similar to one another to be accurately distinguished (cf. Figure 1(b))," thus giving rise to low-dimensional feature overlapping. As this is not the primary focus of our contribution, we have provided a concise discussion of it in  Section 2.2 in the interest of directing the reader's attention to our key insights in Sections 2.3 and 3, where we delve into the issue of data scarcity.

Reviewer comment (Theory):
The paper would benefit from a more comprehensive discussion explaining why sharing features can enhance model performance.

Author response:
Our approach enhances model performance in scenarios characterized by both data scarcity and label skew through two key aspects: 1) leveraging features from other clients, particularly those of locally absent classes, to alleviate local bias, and 2) employing feature augmentation across all classes, to enhance local model generalization. The effectiveness of the first aspect is validated by the outstanding performance FLea achieves in various non-iid settings, with a more pronounced improvement observed in the most skewed case, as illustrated in Table 2. The second aspect is demonstrated by consistently outperforming state-of-the-art methods designed for addressing label skew but not explicitly considering overfitting. We have also incorporated this summarization into the results section of the updated PDF.

Reviewer comment (Writing):
The use of 'λ ∼ Beta(a, a)' needs clarification. It would be helpful to explain what Beta() is and how it is relevant. The reference at the end of the sentence should specify which part of the information it pertains to.

Author response:
Beta distribution is a continuous probability distribution defined on the interval [0, 1]. It is characterized by two shape parameters, usually denoted as $\alpha$ and $\beta$, which determine the shape of the distribution. The probability density function (PDF) of the Beta distribution is given by,
 $ f(\lambda; \alpha,\beta) = \frac{\lambda^{\alpha-1} (1-\lambda)^{(\beta-1)} }{N}$, where $N$ is the normalizing factor and $\lambda\in[0,1]$. In our study, we choose $\alpha=\beta=a$ and herein,   $f(\lambda)= \frac{1}{N} (\lambda (1-\lambda)) ^{\alpha-1}$, which is symmetrical. We move the reference (zhang et al. 2018) to where it is mentioned to make the explanation clear.

Reviewer comment (Reproducibility):
Question — How are the values of λ1 and λ2 in Equation 6 set?

Author response:
Please refer to our response to Major Issue 2.

Reviewer comment (Writing):
Question — What is y^fi in Bf?

Author response:
It is the class label for the feature $f^{\mathcal{F}}_i$. We have improved the notations as mentioned above.

Reviewer comment (Experiments):
Question — What are the impacts of model types on the proposed method?

Author response:
Our method is effective for various data modalities and model architectures. Please refer to our response to Major Issue 3.

Reviewer comment (Writing):
Question — What is the Beta() function?

Author response:
Beta distribution is a continuous probability distribution defined on the interval [0, 1]. Please refer to our answers above.

Reviewer comment (Experiments):
The issue of data scarcity and label skew has previously been studied, particularly in Bayesian Federated Learning frameworks. For instance, reference [1] conducted experiments in the same setting. Given that Bayesian FL does not necessitate the additional communication of features, it appears to be a safer alternative to Flea. A comparison would be valuable to ascertain whether Flea can achieve superior accuracy.

Author response:
**W-ii. A comparison to [1]:**

[1] is relevant to our study but the primary goal is different. We aim to enhance the global model (GM) for global distribution while this paper adapts the local model in the local distribution via personalised federated learning, although this also benefits the global model. As per the results shown in [1], in the non-iid and data scarcity scenarios, the proposed GM achieved a very marginal performance gain compared to FedAvg. Specifically, for CIFAR10 with 100 clients, the GM of the proposed method in Table 1 yields an accuracy of 60.1%, against 59.4% of FeaAvg. On the contrary, on the same dataset and with the same number of clients, our method FLea outperforms FedAvg by 9.27~15.44%, as shown in Table 6 (page 17). This demonstrates our advantage in achieving a better global model in the scarce and non-iid setting. 

We thank the reviewer for pointing out this paper and we have included it in our related work, as blew.
"The above-mentioned methods are developed to prevent the global model from diverging under label
skew, while another way to cope with this issue is to learn a personalized model per client, with the
goal of enhancing performance within their local data distribution (Kulkarni et al., 2020; Tan et al.,
2022a; Kotelevskii et al., 2022). Recently, personalized FL methods based on variational Bayesian
inference have shown promising results, supported by theoretical guarantees (T Dinh et al., 2020;
Zhang et al., 2022b; Zhu et al., 2023). However, these methods still face challenges in learning an
optimal global model."

Reviewer comment (Evaluation):
The first two conclusions in the Section 3.1 are not very convincing. Specifically, "The performance of FL methods decreases remarkably as data scarcity and label skew increase." This argument is concluded from "FedAvg, its accuracy decreases from 75% to 56% when |Dk| reduces from 5000 to 100 in the IID setting. When local data is sufficient (|Dk| = 5000), its accuracy drops from 75% for the IID setting to 60% for the non-IID setting." However, it is unclear why "|Dk| reduces from 5000 to 100" can be compared with changing IID to non-IID setting. Additionally, "Loss-based methods can address label skew only with sufficient local data." This argument is too strong. As mentioned in the second point, Bayesian frameworks are more amenable to data scarcity without requiring more data.

Author response:
**W-iii. The first two conclusions in Section 3.1 are not very convincing:**

We agree with this opinion and we have revised the writing in the updated PDF to make the conclusions more precise.  
For your question ‘it is unclear why "|Dk| reduces from 5000 to 100" can be compared with changing IID to non-IID setting’: we don’t compare the reduction in data size with the changing to non-IID setting, instead, we show that both the reduction in the amount of data and the skewness of the class can lead to performance degradation. 
For your comment ‘Loss-based methods can address label skew only with sufficient local data, This argument is too strong’:  we revised it to ‘The compared loss-based methods can address label skew only with sufficient local data’.

Reviewer comment (Evaluation):
The plot of accuracy as a function of $\lambda_2$ can better clarify the privacy-utility trade-off.

Author response:
**Q-i. The plot of accuracy as a function of $\lambda_2$:** 

 We have now added this in Section 5.2 Impact of hyper-parameters. As shown in Figure 7(c),  with the increasing of $\lambda_2$,  the accuracy drops only marginally, while privacy is better protected. We formulate privacy protection by the reduced correlation between the feature and the original data ($1-\bar{c}$) . As demonstrated in Section 5.3, reducing the correlation can effectively prevent the data from being reconstructed from the features.  Thus, the privacy protection and the utility can be balanced with $\lambda_2$ ranging from 2 to 4.

Reviewer comment (Experiments):
It seems that differential privacy can also be employed to obfuscate the features. Since DP is more mathematically rigorous and widely accepted, it may be more interesting to replace $\ell_{dec}$ with DP.

Author response:
**Q-ii. To replace ℓ_dec with DP:**

FLea introduces feature sharing in addition to model parameter sharing. Existing methods have studied privacy concerns raised by sharing model parameters, for example, using differentially private SGD [3]. These methods can be trivially combined with Flea (for example, using DP-SGD instead of SGD in the local training) and thus this is not discussed in the paper. The less obvious case in this context is the privacy risk posed by the feature-sharing which we empirically analyse in this paper.

Moreover, defining differential privacy in this context is challenging (and thus is a separate research in our opinion). Suppose, we want to protect the presence of a particular sample in the local training data. A reasonable definition of DP could be the following: if the adversary finds the ‘exact’ feature corresponding to the target data sample (using the global model), no shared feature should be nearby. Thus the ‘value’ of the feature vectors needs to be protected. Here the definition of ‘nearby’ depends on the model, dataset, training process and stage, distribution of the data in the clients, etc. Further, making this independent of the strength of the adversary (the core promise of DP) is challenging as the feature needs to be translated to either data point or to presence/absence using certain computation methods. While such data obfuscation has been studied in plausible deniability [4], obfuscating high dimensional real-valued vectors that may transmit multiple times is more challenging.

[2]Yang, Zhiqin, et al. "FedFed: Feature Distillation against Data Heterogeneity in Federated Learning." Thirty-seventh Conference on Neural Information Processing Systems. 2023.

[3]Abadi, Martin, et al. "Deep learning with differential privacy." Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 2016.

[4] Bindschaedler, Vincent, Reza Shokri, and Carl A. Gunter. "Plausible deniability for privacy-preserving data synthesis." arXiv preprint arXiv:1708.07975 (2017).

For your convenience, we quote the revision in *Section Conclusion* as follows,

'Limitations. In reality, because of extra feature sharing, FLea can introduce some extra overheads like communication and storage. We leave the improvement of the efficiency for real-world applications for future work. To enhance privacy, FLea can be trivially combined with methods protecting model parameters (such as DP-SGD[3]), however, improving the statistical privacy risks posed by feature sharing is hard due to the challenges originating from the stochasticity in modelling, data distribution in the clients, the high dimensionality of real-valued features, etc. We thus leave this as a future work. We anticipate that our work will inspire further investigations to comprehensively study feature sharing in the low data regime.'

Reviewer comment (Novelty):
The idea of broadcasting features in the paper appears to make a relatively modest contribution. The proposed under-explored scenarios of Label-skewed and Data-skewed resemble another expression of Non-iid data, which is not an under-explored scenario itself. It is unclear whether the authors can provide an explanation for the distinctions between these scenarios and Non-iid data.

Author response:
We would like to clarify our definition of label skew and data scarcity. **Label skew** is a kind of non-IID where local class distribution is skewed. **Data scarcity** refers to the scenario where local data is very limited. We didn’t define **data skew**, but we suppose you mean the data quantity disparity, i.e.,  the amount of local data across clients varies. 

Label skew and data quantity disparity usually happen together. One example is that the widely adapted data split method in FL literature employs Dirichlet distribution, resulting in label and quantity skew simultaneously [1]. Specifically, if we distribute the training set of CIAFR10 (50,000 samples) to 10 clients using a Dirichlet distribution parameterized by 0.1, these clients will present different class distributions, and the total number of local samples ranges from 2853 to 8199. This is the commonly explored non-IID setting. In this paper, we further explore scarce non-IID data. Continuing with the abovementioned CIFAR10 examples, we now split the data into 100 and 500 clients, respectively. As a result,  the number of samples per client reduces significantly: the median number drops from 5685 to 90 when the number of clients increases from 10 to 500, as shown in Figure 10(a). This is the realistic scenario that we are interested in. It is also worth mentioning that data scarcity is independent of label skew and it can happen in the IID scenario. As shown in Figure 10(b), the local data covers 10 classes uniformly, but the data scarcity problem becomes severe when the number of clients increases. 

Overall, the label skew and quantity disparity have been studied, but to the best of our knowledge, we are the first to deeply look into the data scarcity issue in FL. To make the data scarcity senior more clever, we have supplied Figure 10 and the discussion in Appendix C.1.

[1] Zhang, Jie, et al. "Federated learning with label distribution skew via logits calibration." International Conference on Machine Learning. PMLR, 2022.

Reviewer comment (Novelty):
The concept of overfitting and Client-Drift resulting from Label-skewed and Data-skewed scenarios seems consistent and has been extensively investigated in prior research.

Author response:
They are different in our paper and overfitting caused by data scarcity in FL is under-explored.  Overfitting is a common issue in deep learning, and it occurs when a model learns the training data very well but it can hardly generalize to new, unseen data [2]. One primary reason leading to overfitting is that the amount of training data is limited, and thus the model might memorize the examples rather than learning the underlying patterns. A more detailed illustration can be found in Section 3.2 where we use an IID setting to explain overfittting. The concept of *client drift* was proposed to illustrate the phenomenon in federated learning [3,4]. With non-IID data, the local model is optimised by local distribution, which leads to a sub-optimal global model after aggregation.  Here, the local model could be overfitted (biased) to local distribution and thus cannot generalize to the global distribution. On the contrary, in our studied setting, because of the limited local data (small in data size), the model can be thoroughly overfitted and cannot generalize to the unseen data from the local distribution. Therefore, our identified overfitting and client drift are orthogonal in FL, rather than overlapping.  To make this clearer, we have revised the Introduction when we introduce the concepts in the updated submission: ‘...data scarcity, which refers to the situation where all clients have a limited number of samples.” and ‘‘Label skew alone can lead to model bias: local models are over-fitted by the local distribution and struggle to generalize to the global distribution. This is known as client drift, which consequently leads to a sub-optimal global model”.

[2] Brownlee, Jason. Better deep learning: train faster, reduce overfitting, and make better predictions. Machine Learning Mastery, 2018.
[3]Zhao, Yue, et al. "Federated learning with non-iid data." arXiv preprint arXiv:1806.00582 (2018).

[4]Karimireddy, Sai Praneeth, et al. "Scaffold: Stochastic controlled averaging for federated learning." International conference on machine learning. PMLR, 2020.

Reviewer comment (Experiments):
The paper's experimental dataset is limited to CIFAR10, without inclusion of other datasets, indicating a lack of domain generalization.

Author response:
**For a) and b) regarding experiments**, we have now supplemented the results for audio classification, using the UrbanSound8K data and a different CNN architecture. Table 2 has been updated to include the new results, with extensive results and discussions presented in Appendix C.4 (pages 17-18). The results demonstrate that FLea consistently outperforms the start-of-the-art baseline by 2.59%~6.27% across the two tasks.  We include the results for FLea compared to the SOTA FedMix blew, 

Table 2 Overall performance comparison. The local data size $|\mathcal{D}_k|$ is as small as $100$ on average.
| Accuracy %       |     |          CIFAR10       |                | |        UrbanSound8K        |                |
| ------ | ----------- | -------------- | -------------- | ------------ | -------------- | -------------- |
|               | Quantity(3) | Dirichlet(0.5) | Dirichlet(0.1) |    Quantity(3)  | Dirichlet(0.5) | Dirichlet(0.1) |
| **FedMix**|   $44.04{\pm1.53}$ |$45.50{\pm 1.88}$ |$    38.13{\pm2.06}$   |$51.56{\pm 0.59}$ |$54.18{\pm 0.62}$  |$43.35{\pm 0.72}$
| **FLea**  |    $47.03{\pm1.01}$|$48.86{\pm1.43}$|$44.40{\pm1.23}$| $     57.73 {\pm 0.51}$ |$59.22 {\pm 0.78}$ |$45.94{\pm 0.77}$

Reviewer comment (Experiments):
The paper employs feature broadcasting but lacks a comparative study of communication costs, and the two communication rounds increase overhead.

Author response:
**For c) regarding the communication cost**, we acknowledge that FLea achieves superior model accuracy at the cost of more communication overheads and computation burden compared with non-augmentation methods, but FLea is more advanced in the convergence speed. To validate this, we have now added model learning speed analysis in Figure 5.  It is evident that FLea consistently requires fewer communication rounds to attain a target model accuracy.  This suggests that FLea proves advantageous in scenarios where extensive communication with a large number of clients is not always feasible. 

Moreover, compared to the augmentation-based methods like FedData and FedMix, we also demonstrate that with similar communication and computation costs, FLea yields significant accuracy gain (refer to Section 5.2) and effectively reduces the privacy risk (refer to Section 5.3).

We have now added the discussion to Section 5.2 and Section 6 Conclusions.

Reviewer comment (Experiments):
Observing that three loss terms act on $\theta_k$ in Eq 6, the ablation experiments are not presented in the main text. It is recommended that the authors consider reducing the discussion of Label-skewed and Data-skewed scenarios to make the core arguments more concise.

Author response:
**For d) regarding the organization of the paper,** following your suggestion, we have now shortened the discussion for related work and given more space for experimental settings and results. More concretely, we have added a hyper-parameter study in Section 5.2 with a deep analysis of the weight in the loss function. As shown in  Figure 7, we demonstrate how we find the weight in the loss function to achieve the best performance improvement and privacy protection trade-off.

Reviewer comment (Novelty):
This paper focuses on scarce and label-skewed data in federated learning. This is a kind of non-iid case, and a lot of papers study the non-iid and heterogenous scenarios in FL; why do the authors say this is the first research?

Author response:
**Response:** Given our answers to W1 and W2 as above, to the best of our knowledge, we are the first to decouple data scarcity from label skew in the non-iid FL. We found that label skew leads to local model bias while data scarcity leads to local overfitting. Label skew has gained extensive attention but data scarcity is relatively under-explored. One piece of evidence is that most related publications split CIFAR10 into 10~100 clients, while we experiment on 500 clients, which leads to significant small local datasets. 
Addressing these two problems is equally important for real-world FL applications and our study pushes the envelope. To make this clearer, we have rephrased our first contribution in the INTRODUCTION to 'The first study on a common but under-explored scenario in FL, where all the clients possess limited and highly label-skewed data. We find that model over-fitting caused by data scarcity is under-looked by existing methods.'

Reviewer comment (Reproducibility):
More details needed on feature buffer, until very late it is not obvious what is the number of pairs and how it is collected.

Author response:
**W1.  More details on feature buffer:**

Feature buffer stores the feature-label pairs from the previous communication round. As shown in Figure 4, in each round, this buffer gathers the feature-target pairs from several selected clients and each client contributes the features from a fraction of its local data. To be more specific, for each selected client n, we first randomly hold out a fraction ($\alpha$) of its data $D_k$, and extract features from the held-out data. Finally, all the extracted feature-label pairs will be sent to the server to construct the feature buffer. To make this clearer, we improve the introduction of our method in Section 4 by introducing it earlier and explicitly. 

The number of pairs in the buffer is not a fixed value, it depends on the number of selected clients and their data size.  In our experiments,  10% of the clients participate in each round and features from 10% of the local data of the selected clients will be sent to the buffer. For example, in the CIFAR10 experiments where we set up 500 clients ($|D_n|$ is about 100 and the number of shared features is about 10) and 50 of them are selected in each round, the size of the feature buffer will be about $10\times50=500$.  Keeping the fraction small can reduce the communication burden and privacy risk. In addition,  we also explore the performance for sharing different fractions, as shown in Figure 6. It demonstrates that $\alpha=$10% is good enough to boost the performance.

Reviewer comment (Experiments):
The scale of the experiments (number of samples and clients) is good, but more datasets are needed for evaluation.

Author response:
**W2. More datasets:**

We had implemented and tested our methods for various architectures and datasets previously (as can be checked from the submitted code), but did not include them due to the limited spaces. Now, following your suggestion, we supply the results on another data modality-*audio* with a CNN model in the main paper. Section 5 and Table 2 have been updated (including the results for UrbanSound8K in the right part), with extended results presented in Appendix C.4 (page17-18). The results show that FLea consistently outperforms the start-of-the-art by 2.59%~6.27%. For your convenience, we quote part of the results below:

Table 2 Overall performance comparison. The local data size $|\mathcal{D}_k|$ is as small as $100$ on average.
| Accuracy %       |     |          CIFAR10       |                | |        UrbanSound8K        |                |
| ------ | ----------- | -------------- | -------------- | ------------ | -------------- | -------------- |
|               | Quantity(3) | Dirichlet(0.5) | Dirichlet(0.1) |    Quantity(3)  | Dirichlet(0.5) | Dirichlet(0.1) |
| **FedMix**|   $44.04{\pm1.53}$ |$45.50{\pm 1.88}$ |$    38.13{\pm2.06}$   |$51.56{\pm 0.59}$ |$54.18{\pm 0.62}$  |$43.35{\pm 0.72}$
| **FLea**  |    $47.03{\pm1.01}$|$48.86{\pm1.43}$|$44.40{\pm1.23}$| $     57.73 {\pm 0.51}$ |$59.22 {\pm 0.78}$ |$45.94{\pm 0.77}$

Reviewer comment (Experiments):
In section 3.1 authors report some performance changes due to data scarcity etc. but the setting is not clear (e.g. what is the communication frequency/local iterations).

Author response:
**W3.  Setting for Sec 3.1:**

 Section 3.1 involves IID and non-IID data splits with different local data sizes and local class distribution. To make it clearer, we have added a group index (a,b,c,d,e,f) in Figure 1 to ease the presentation and supplemented the experimental setup in Appendix A (page 13). Briefly, in (a,b,c) local data contains all 10 classes uniformly while in (d,e,f)  local data only covers randomly selected 3 of the 10 classes. For the local data size, (a)>(b)>(c) and (d )>(e)>(f). Please check our updated submission for more details.

Reviewer comment (Experiments):
The parameter $a$ is very critical, yet its resulted sensitivity is not thoroughly analyzed in experiments. Also introducing such a parameter is not ideal for FL settings.

Author response:
**W4. The sensitivity of parameter $a$:**  

The performance of FLea is not sensitive to the choice of a. As formulated in Eq.(2), for each data point, we sample the mixup weight λ from a beta distribution, i.e., $\lambda$\~Beta(a,a) with parameter a controlling the sharpness of the beta distribution. Λ ranges from 0 to 1: A smaller $\lambda$ basically makes the generated sample closer to the local feature while a larger one pushes that to the global feature. The Beta distribution is originally controlled by two parameters  $\lambda$\~Beta($\alpha$,$\beta$) [1], and we use  $\alpha=\beta=a$ to ensure that the distribution is symmetrical and expectation of $\lambda$ is 0.5, so that the global features and local features have equal opportunities to be used for training. We have also added an empirical study in Section 5.2 to illustrate the model performance with varying a. Figure 7(c) shows the pdf for Beta distribution with the corresponding accuracy, It is evident that the results are not sensitive to $a (a>1)$.

[1] Gupta, Arjun K., and Saralees Nadarajah, eds. Handbook of beta distribution and its applications. CRC press, 2004.

Reviewer comment (Experiments):
Communication and computation burden due to the added feature buffer is not examined adequately.

Author response:
**W5. Communication and computation burden:**

Firstly, we would like to acknowledge that FLea yields more communication overheads and computation burden compared with non-augmentation methods, as additional features (which can often be as large as the model) need to be transferred and used for training.  We leave a detailed investigation for the future as the sizes can be optimized in various ways like sharing features every several rounds, compressing the feature vectors, caching previously shared features at the clients, etc., each of which can constitute its own line of investigation. 


However, it is worth noting that our study focuses on showing the promise of sharing features against sharing other data augmentations like raw data or the average of raw data. We show that with similar communication and computation costs, FLea significantly outperforms FedMix (refer to Section 5.2) and effectively reduces the privacy risk of FedData and FedMix (refer to Section 5.3).

Moreover, compared to the non-augmentation methods, we also demonstrate that although requiring more communication overheads, FLea not only achieves significantly higher final accuracy but also exhibits faster learning. For this purpose, we supplement a convergence speed analysis in Figure 5.  From the figure, it is evident that FLea consistently requires fewer communication rounds to attain a target model accuracy.  This suggests that FLea proves advantageous in scenarios where extensive communication with a large number of clients is not always feasible.
We have now added the discussion to Section 5.2 and Section 6 Conclusions.

Reviewer comment (Theory):
By choosing $\lambda_2$ authors adjust how private the algorithm is but since there is no rigorous privacy utility tradeoff; it is hard to characterize the choice of $\lambda_2$. I think this lack of rigor is undesirable in FL.

Author response:
**W6. The choice of $\lambda_2$:** 

We have now added a privacy protection and feature utility trade-off analysis in Figure 6(c). We formulate the privacy protection by the reduced correlation between the feature and the original data ($1-\bar{c}$) and the utility by the final model accuracy. As $\lambda_2$ increases, $1-\bar{c}$ increases correspondingly while the accuracy presents an acceptable drop. Since reducing the correlation can effectively prevent the data from being reconstructed from the feature, as we demonstrated in Section 5.3, the privacy protection and the utility can be balanced with $\lambda_2$ ranging from 2 to 4.