Paper: Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors (ICLR-2024)

Reviewer comment (Evaluation):
The introduction shows that the proposed approach aims to generate minimal synthetic anchor data to enhance client-model generalization. However, this "minimal" property of generated data is not discussed in the experiments. The ablation study in subsection 5.4 shows that the size of synthetic data can significantly affect the inter-accuracy. Thus, there might exist a trade-off between the model performance and the size of synthetic data. More explanations can be provided here.

Author response:
Finding the minimal amount synthetic data is not the focus of this work. We have replaced “minimal” with “small” in our revision for clarification. Following your suggestion, we have performed a more detailed hyperparameter search on IPC. Specifically, we search IPC from {5, 10, 20, 50, 100, 200} and the experimental results are updated in our revision. 
Overall, blindly increasing the IPC does not guarantee to obtain optimal intra- and inter-accuracy. It will cause the loss function to be dominated by the last 2 terms of Eq. 8, \textit{i.e.,} by synthetic data rather than minimizing the empirical risk of classification on cross-entropy loss. However, synthesizing larger number of synthetic data may degrade its quality, and the sampled batch for $\mathcal{L}_{REG}$ may fail to capture the distribution.

Reviewer comment (Experiments):
Subsection 3.1 shows that synthetic anchor data is shared amongst the client’s neighbors. However, it is unclear how the neighbor information is defined in the experiments. In addition, it seems that the synthetic anchor data $D^{Syn}$ in Subsection 3.2 simply combines all the anchors $D^{Syn}_i$ within each client.

Author response:
Regarding how we merge D^{syn}: We perform simple interpolation (averaging) among clients as it is shown that using this mixup strategy can improve model fairness [1]. 

Regarding how we define our neighbors: We assume the client nodes form a complete graph in DIGITS and OFFICE experiments. For CIFAR10C experiments, we randomly sample neighboring clients for each round.
We have added the clarification in our revision.


[1]  Chuang CY, Mroueh Y. Fair Mixup: Fairness via Interpolation. InInternational Conference on Learning Representations 2020 Oct 2.

Reviewer comment (Theory):
In the derived generalization in Theorem 1, it assumes (i) real labeling and synthetic data labeling are similar, and (ii) real labeling and distillation data labeling are also similar. It is confusing how both assumptions can always be guaranteed in real scenarios.

Author response:
We thank the reviewer for his astute observation about the labelling functions.The only mathematical assumption that the theorem makes is that the encoded distributions of the source $P_s$ and the global data domain $P_t$ converge to each other. Practically, this is enforced by the anchor term in the loss function, $\mathcal{L}_{REG}$. 

Coming to the interpretation of the bound through the similarity of the labeling functions.

Firstly, it is important to note that the theorem does not require both of the error terms to be small. As long as any single one of the terms is low, we can change the component weights $\alpha$ to rely more on that term , so that we get a tighter generalization guarantee. These component weights alpha are directly tied to the regularization weights $\lambda_{REG}$ and $\lambda_{KD}$.
The explanation for either of the labelling functions being close to $f_T$ is given below, 

1. The real data and the synthetic data will have similar labeling functions because their distributions are enforced to be similar under an MMD metric during the data generation process. - (Eq. 3) . Therefore $f^{syn}$ will be close to $f_T$ 

2. The statement about the extended KD data is a bit more subtle. It is crucial to note that $D_{KD}^{syn}$ depends on the client models $M_j$  and **is not a static distribution.**  Towards the end of training, when the empirical KD loss consistently decreases and generalized models are learned, the labeling function of the extended KD data $D_{KD}^{syn}$, which represents the consensus knowledge of all the domains,  will correctly classify any example drawn from the global data  domain  $P_T$.
Therefore, we project that the labeling function $f_{KD}^{syn}$ will be close to $f_T$.

Reviewer comment (Experiments):
The experimental settings show that for heterogeneous model experiments, multiple baselines are compared, including FedMD, FedDF, FCCL, FedGen, and VHL. But Table 2 only lists the results of FedHe, FedDF, and FCCL.

Author response:
The comparison results with FedGen are presented in Homogeneous Model Experiments (Section 5.3). We did not include FedGen in Table 2 as it is not applicable to model heterogeneous scenarios discussed in table 2. FedMD[1] was not considered to be included due to its poor performance, consistent with the observation report in its similar method FedHe [3]. We have corrected the paragraph by removing FedGen and FedMD and adding FedHe in our revision.

[1] Li D, Wang J. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581. 2019 Oct 8.

[2] Zhu Z, Hong J, Zhou J. Data-free knowledge distillation for heterogeneous federated learning. InInternational conference on machine learning 2021 Jul 1 (pp. 12878-12889). PMLR.

[3] Chan YH, Ngai EC. Fedhe: Heterogeneous models and communication-efficient federated learning. In2021 17th International Conference on Mobility, Sensing and Networking (MSN) 2021 Dec 13 (pp. 207-214). IEEE.

Reviewer comment (Experiments):
The client structure information is not provided in the experiments. Does it imply that all the clients are connected with each other in all experiments?

Author response:
We assume the client nodes form a complete graph in DIGITS and OFFICE experiments. For CIFAR10C experiments, we randomly sample neighboring clients for each round.

Reviewer comment (Writing):
In subsection 3.3., "$P(\cdot)$ index class category" is confusing. Where is $P(\cdot)$ used in this section?

Author response:
We thank the reviewer for the careful review. The $P(\cdot)$ is unused and we have removed $P(\cdot)$ in our revision.

Reviewer comment (Evaluation):
The communication costs of DESA can be analyzed, because it might include additional anchor sharing and logits sharing compared to baselines.

Author response:
We thank the reviewer for the suggestion. As noted in our Section 3.1, DeSA only requires sharing logits w.r.t. Global synthetic data during model training. Thus it has a relatively low communication overhead compared to baseline methods which require sharing model parameters. For example, if we use ConvNet as our model and set IPC=50, the number of parameters for communication for DeSA will be 10(number of classes) x 50(images/class) x 10(logits/image) = 5k. In comparison, baseline methods need to share 320K parameters (the number of parameters of ConvNet model), which is much larger than DeSA. We also add the discussion in Appendix D in our revision.

Reviewer comment (Experiments):
Below Proposition 2, it is shown that when the local data heterogeneity is severe, the model learning should rely more on the centralized data, e.g., synthetic data and the extended KD data. This can be verified in the experiments, e.g., how the hyperparameters $\lambda_{REG}$ and $\lambda_{KD}$ can be changed with respect to the data heterogeneity.

Author response:
We clarify that the statement “when the local data heterogeneity is severe, the model learning should rely more on the centralized data” is on the assumption  that “ synthetic and the extended KD datasets are similar to the global ones” Thus the LHS of Eq(9) will be dominated by $ \epsilon_{{P}^T}(f^{Syn})$ and $ \epsilon_{{P}^T}(f_{\text{KD}}^{Syn})$. It is intuitive that if synthetic data is ideal aligned with global distribution and we fully rely on synthetic data in local training, the performance gain between training on synthetic data vs on local data is expected to be larger when local data is more heterogeneous. Also, as noted in Theorem 1, the performance is not solely related to hyperparameters $\lambda_{REG}$ and $\lambda_{KD}$. It will be also affected (or dominated) by the fourth term of Eq 8, when we rely on sufficient number of ideal synthetic data (aligned with global distribution).

Reviewer comment (Writing):
"FedSAB" in subsection 5.3 is undefined.

Author response:
Sorry for the typo, we have corrected it in our revision.

Reviewer comment (Theory):
The motivation of considering generalization ability of client models on inter-domain tasks is not clear. In the model heterogenous setting, each client may process a model with a different architecture, which is compatible with its own configuration. While the client can benefit from other clients’s data to train a personalized model, why does this model have to perform well on other clients’ tasks too? Other clients may not be able to acquire or deploy the model.

Author response:
We thank the reviewer for raising this concern. Let us think about an essential scenario: In 2020, hospitals from different regions want to collaboratively train a COVID-19 classification model. Due to quarantine, people cannot travel outside the local area, so the local hospital can only collect local cases with local strains of Covid viruses. If they train a model that only cares about intra-domain, this *personalized* model will suffer from a performance drop when the quarantine restriction is revoked and people from one region (with its local virus strains) will travel to other regions and use other hospitals’ models. Thus, we consider *intra* and *inter*-domain performances to be both important, and allowing each client model to generalize well on other client domains can potentially improve model robustness. Thus, In this work, we consider a different FL setting without sever and define it as  *decentralized federated mutual learning*, which shares the same motivation as the existing work [1].

[1] Huang W, Ye M, Du B. Learn from others and be yourself in heterogeneous federated learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022 (pp. 10143-10153).

Reviewer comment (Experiments):
In experiments, from Table 2 it seems that data heterogeneity and model heterogeneity are correlated. That is, each dataset is assigned one model architecture. The results would be more interesting if both different datasets and models are assigned independently (by dividing a dataset into multiple clients).

Author response:
We thank the reviewer for suggesting that we should split a dataset and assign different models to subsets. In fact, we implement this strategy in our CIFAR10C experiments, where we split Cifar10C into 57 clients (subsets) and randomly assigned different model architectures to them. The objective of Table 2 is to show that our method can obtain competitive inter and intra-domain performance under both data and model heterogeneous scenarios.

Reviewer comment (Experiments):
In DIGITS and OFFICE experiments, what is the number of clients? Is a client identified as a dataset?

Author response:
For DIGITS and OFFICE experiments, we assume each dataset represents one client following [1].  We further clarify the setting in our revision.

[1] Li X, JIANG M, Zhang X, Kamp M, Dou Q. FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. InInternational Conference on Learning Representations 2020 Oct 2.

Reviewer comment (Experiments):
How does DeSA perform in cases where each client has limited training data, e.g., a few samples per class? Can each model benefit more from the global synthetic dataset and KD?

Author response:
We thank the reviewer for the great question. In Eq. 8, if we have limited local data, the bound will be dominated by $\alpha^{Syn}$ since $\alpha$ is small. Thus, the performance depends on the quality of global synthetic data. However, we would also like to point out limited local training data may affect the generation of synthetic data and further decrease its utility.

Reviewer comment (Theory):
In Equation (3), the definition of D_i^{Syn} involves summation over i.

Author response:
We thank the reviewer for the careful review and pointing out the typos. We have addressed the typos as follows. We have also done more careful proofreading in our revision.
- There shouldn’t be a summation over i to N, and we have corrected it. 
- The L_{CE} description shouldn’t be there and we have removed it.
- We corrected it into DeSA.

Reviewer comment (Writing):
First line of Section 5.3: FedSAB?

Author response:
We thank the reviewer for the careful review and pointing out the typos. We have addressed the typos as follows. We have also done more careful proofreading in our revision.
- There shouldn’t be a summation over i to N, and we have corrected it. 
- The L_{CE} description shouldn’t be there and we have removed it.
- We corrected it into DeSA.

Reviewer comment (Theory):
Equation (2) introduces an objective that encompasses all clients for defining inter-client loss. However, the decentralized nature of the problem implies that each client can communicate only with adjacent nodes, raising questions about the feasibility of this objective.

Author response:
We thank the reviewer for pointing out the typo of Eq. 2. The summation should be from 1 to N(C_i), and we have corrected it in our revision. As shown in Algorithm 1, our method is designed to work only to receive neighbor node information, and DeSA is designed for peer-to-peer decentralized network. Although we are working on a peer-to-peer decentralized network but the loss requires all nodes’ information, we can leverage the FastMix algorithm to aggregate all nodes’ information [1,2]. This method can aggregate all nodes' information via adjacent nodes’ communication at a linear speed. It is very common in fully decentralized optimization. In fact, our method can also work if each node can only receive neighbor nodes’ information, and we empirically show the feasibility in our CIFAR10C experiments by sampling neighboring clients.

Reviewer comment (Writing):
Equation (3) suffers from unclear terminology; specifically, the meaning of the representation (x∣y) is not explained. Additionally, the methodology for generating "randomly sampled feature extractors" is also unclear.

Author response:
We thank the reviewer for the clarification question. $\psi(x|y)$ means the comparison is conditional on label y , which allows us to distill synthetic data based on each class c. 

We thank the reviewer for the clarification question. Using randomly sampled feature extractors allows us to perform empirical MMD loss as described in [3]. To implement it, we simply randomly initialize the model parameters for each round with PyTorch.

Reviewer comment (Writing):
Equation (4) contains undefined notations, requiring clarification.

Author response:
We appreciate reviewer’s careful review. . We have removed the unused notation $P$ and add description of $K$ in our revision.

Reviewer comment (Presentation):
The paper touches upon privacy concerns arising from the sharing of synthetic data but fails to delve deep enough into this critical issue. Given the importance of privacy in federated learning algorithms, the authors should offer a more comprehensive discussion, preferably in the main text rather than relegating it to the appendix.

Author response:
We thank the reviewer for the suggestion. Since our main focus is to solve the challenging issue - decentralized federated learning with data and model heterogeneities, we decided to put more focus on how our method can utilize data distillation to solve the problem in our manuscript. 

We agree that showing the privacy guarantee is important, so we have discussed the empirical privacy guarantee in Section 3.2 and pointed the readers to our Appendix B and C for our privacy-preservation results. Specifically, we provide a DP version for synthetic data in our Appendix C. In addition, we have shown our method can defend against Membership Inference Attack (MIA) [1] in Appendix B. We share the concern that incorporating a comprehensive discussion into the main text might limit the space available for elaborating on the technical details essential for understanding our innovative method. Our priority is to ensure that the core aspects of our approach are communicated effectively and clearly. Other reviewers seem to like our current presentation. In the revised manuscript, we underscored the sentence in the main text that directs readers to the appendix. 

[1] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. IEEE, 2022a.

Reviewer comment (Novelty):
The theoretical results are primarily based on Ben-David et al. (2010), a fact highlighted in the appendix but missing from the main text. This could potentially weaken the paper’s contribution.

Author response:
We would like to point out that Ben - David 2010 et al’s work was a general bound for learning on new domains. Our theory’s  contribution is not only applying Ben-David et al’s work to our scenario, but also connecting it to our main algorithm through the addition of terms like $f^{syn}$ and $f^{syn}_{KD}$.This addition introduces novel mathematical insights into the connection between labelling functions and the anchor and KD regularization losses.

We also provide a novel stronger version of our theorem in Proposition 2, that holds under the mild conditions enforced by our algorithm, and other novel and useful Lemmas in the appendix that theoretically motivate our algorithm.

Following your valuable suggestion, we have edited the reference for Ben-David et al. in our revision.

Reviewer comment (Theory):
The paper struggles to bridge the gap between theoretical claims and empirical results. The presented theorems are contingent upon strong assumptions, and their relevance to the experimental findings is not intuitively obvious.

Author response:
We thank the reviewer for his interesting remark. However, we strongly believe that our theorem gives remarkable insights into the empirical success of our algorithm.

- The only assumption that our theorem makes is the convergence in probability of the encoded target and source domains, which is empirically enforced by our anchor loss regularizer $L_{reg}$. This is a novel condition in theorem 1, that inspired our design of the empirical anchor regularization term in Equation (7).

- It is of paramount importance to note that the previous work of Ben-David 2010 et al, whose shoulders we build Theorem 1 upon, has no connections to any practical algorithm at all. It was our novel contribution to develop a bound that incorporated empirical connections like the labelling function of synthetic data $f^{syn} $ and the Extended KD data $f^{syn}_{KD}$. These labelling functions connect deeply with our empirical anchor and KD regularization losses . 
Theorem 1 helps us predict the global performance of models using the component weights $\alpha$ given to the empirical regularization terms.
We have further clarified these connections in the appendix of our revision.

- Our newly added ablation studies clearly show that improving the quality and quantity of synthetic data ($D^{syn}$) boosts both inter and intra accuracy. Remarkably, Theorem 1 accurately predicted this outcome. In simpler terms, the theorem suggests that better synthetic data reduces the generalization error of labeling functions ($f^{syn}$ and $f^{syn}_{KD}$), tightening the upper bound on global generalization error. This alignment between theory and experiment underscores the reliability of Theorem 1 in predicting empirical outcomes.

- The connection between the quality of synthetic data and the upper bound in Theorem 1 is further confirmed during experiments in Section C of the Appendix. The injection of noise reduces the quality of the global synthetic data $D_{syn}$, which thereby worsens the upper bound guarantee in Theorem 1. This conforms with the empirical observation of a drop in inter and intra-client accuracy.

Therefore, we strongly believe that our theory is connected with the empirical side of our problem, and helps in explaining it’s results.

Reviewer comment (Writing):
Could you please clarify the issues listed under "Cons"?

Author response:
Dear Reviewer VrA4,

As the rebuttal deadline is approaching, we would like to know if our rebuttals have addressed your concerns and questions. Also, we have tried our best to reflect your brilliant feedback to our revision. We appreciate the opportunity to discussing during this stage and are delighted to address your further question if there is any. If you are satisfied with our response and revision, we are grateful if you could kindly re-consider the rating for DeSA.

Best Regards,

DeSA authors

Reviewer comment (Theory):
The local synthetic anchor dataset is shared. Thus, the privacy of the synthesized anchor should be considered. Although the DP is used to protect synthetic anchor, could this defend against recovering the raw data?

Author response:
We thank the reviewer for the clarification question. We agree the privacy of the synthetic data is important, and we provide a DP version for synthetic data in our Appendix C. In addition, we have shown our method can defend against Membership Inference Attack (MIA) [1] in Appendix B. Regarding raw data recovery, since we don’t share gradients among clients (we only share logits w.r.t. global synthetic data), the commonly used gradient inversion attacks [2,3] may not be able to recover original raw data.

[1] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. IEEE, 2022a.

[2] Geiping J, Bauermeister H, Dröge H, Moeller M. Inverting gradients-how easy is it to break privacy in federated learning?. Advances in Neural Information Processing Systems. 2020;33:16937-47.

[3] Huang Y, Gupta S, Song Z, Li K, Arora S. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems. 2021 Dec 6;34:7232-41.

Reviewer comment (Experiments):
It is unclear how difficult the data synthesis process (i.e., equation 3) is when the data is highly non-iid across clients. Since it basically minimizes the discrepancy between representations of local data and global data, does this process always successfully generate satisfactory anchors regardless of how data is partitioned? There is some visualization of synthetic anchors in the appendix, but the quality of synthetic anchors still seems to be mysterious.

Author response:
We thank the reviewer for the constructive question. Let us think about an extreme non-iid scenario where clients lack data in class c, our Eq. 2 intuitively suggests that client i, lacking samples in class c, cannot reasonably generate data samples with label c. In such instances, we adjust the interpolation ratio according to the class prior, thus deweighting client i on synthetic data class c and adding weight to the clients with abundant data in class i. Consequently, this implies client i will have a greater reliance on other clients’ data in class c, thereby addressing heterogeneity and enhancing overall generalizability.

**Regarding the quality of synthetic data:**
We hope to clarify that the purpose of data distillation is not to generate hight-fidelity images that look similar to the original image, but rather the ones keeping the similar level of utilities. Our generated distilled data share similar patterns as the ones presented in the original distribution-based dataset distillation paper [1] (see its Fig. 2). 

[1] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514–6523, 2023.

Reviewer comment (Experiments):
The ablation studies for DESA are not comprehensive enough to clarify the subtlety of model components. The hyperparameters (e.g., $\lambda_{KD}$, $\lambda_{REG}$, and IPC) are not searched comprehensively. For example, the inter accuracy vs. $\lambda_{KD}$ is still monotonic with the three data points provided, and readers cannot grasp a full picture of the role of $\lambda_{KD}$ or the KD loss.

Author response:
We thank the reviewer for the constructive comment. Following your suggestion, we have performed a more fine-grained hyperparameter search on Reg loss and KD loss. Specifically, we search $\lambda_{KD}$ from {0, 0.1, 0.5, 1, 2} and $\lambda_{REG}$ from {0, 0.01, 0.02, 0.05, 0.1}, and the experimental results are updated in our revision. The findings are aligned with our original version that $\lambda_{KD}$ helps improve the inter-accuracy increases, and $\lambda_{REG}$ helps improve the intra-accuracy as well as the inter-accuracy within a certain range (observe the peak inter-accuracy at $\lambda_{REG}=0.01$).