Paper: FENDA-FL: Personalized Federated Learning on Heterogeneous Clinical Datasets (ICLR-2024)

Reviewer comment (Writing):
This paper addresses the challenge of personalized federated learning using heterogeneous local datasets; however, the mechanism is not thoroughly explained.

Author response:
We agree that such a discussion is important and was not adequately present in our original draft. A formal description of the FENDA-FL approach is now included in Appendix J. Therein, we also discuss the admissibility of the three most common types of distribution changes: label shift, covariate shift, and concept drift. Because FENDA-FL includes a locally trained feature extractor and classification head, each of these shifts can theoretically be handled by the approach. For label and covariate shift, these modules may adapt feature representations and predictions to lean more or less heavily on global representations or modify prediction distributions to match local statistics. For concept drift, the relationship between labels and global features may be de-emphasized in favor of local representations. In extreme cases, where global information is not relevant, the local prediction head may learn to ignore global features altogether. We hope that this section more thoroughly, and clearly, describes the FENDA-FL design and its flexibility in non-IID data settings.

Reviewer comment (Theory):
Please describe the mechanism of the proposed method in detail and elucidate its effectiveness in addressing different types of distribution shifts, particularly in the context of client dataset heterogeneity.

Author response:
We agree that such a discussion is important and was not adequately present in our original draft. A formal description of the FENDA-FL approach is now included in Appendix J. Therein, we also discuss the admissibility of the three most common types of distribution changes: label shift, covariate shift, and concept drift. Because FENDA-FL includes a locally trained feature extractor and classification head, each of these shifts can theoretically be handled by the approach. For label and covariate shift, these modules may adapt feature representations and predictions to lean more or less heavily on global representations or modify prediction distributions to match local statistics. For concept drift, the relationship between labels and global features may be de-emphasized in favor of local representations. In extreme cases, where global information is not relevant, the local prediction head may learn to ignore global features altogether. We hope that this section more thoroughly, and clearly, describes the FENDA-FL design and its flexibility in non-IID data settings.

Reviewer comment (Presentation):
It would be nicer to also see a table of the results, rather than a visual representation due to the number of settings.

Author response:
Thank you for your suggestion. Other reviewers also noted that the bar graphs did not present our results as clearly as we had hoped. Taking your suggestion, we have modified the presentation of the results to be in a tabular format. We hope that the representation of the results is improved. The bar graphs are still presented in Appendix K. However, if they should be removed, we are happy to do so.

Reviewer comment (Presentation):
Minor comment: The color choices for the bar graphs seem to represent goodness, with red being worse and blue being better; however, the different colors are a little jarring and could cause confusion.

Author response:
Thank you for your suggestion. Other reviewers also noted that the bar graphs did not present our results as clearly as we had hoped. Taking your suggestion, we have modified the presentation of the results to be in a tabular format. We hope that the representation of the results is improved. The bar graphs are still presented in Appendix K. However, if they should be removed, we are happy to do so.

Reviewer comment (Evaluation):
The performance benefit compared to APFL shown in Fig. 2 is not very clear, also in Fig. 3, a similar observation can be made for FedAVG and APFL.

Author response:
With the change of results presentation, we hope that the readability and ease of analysis has been measurably improved. The experimental results are now presented in Tables 2-4. FENDA-FL outperforms APFL on the Fed-Heart-Disease task by a significant margin. While FENDA-FL is the second best method for Fed-IXI, the new architecture studies in Appendix F show that the gap between the two methods can be closed through thoughtful network design. For Fed-ISIC2019, each method under-performs global approaches. For both GEMINI Mortality and Delirium, FENDA-FL outperforms APFL in all metrics.

Reviewer comment (Novelty):
The idea of using domain adaptation seems like a natural pairing with FL. Can the authors include a discussion on the inherent differences / relationships between these 2 fields for better context? Google scholar reveals some papers such as https://arxiv.org/abs/1911.02054 and https://arxiv.org/abs/1912.06733 which may be relevant.

Author response:
We agree that the pairing of techniques in domain adaptation and federated learning is quite natural. In the design of FENDA-FL, we have borrowed an approach from domain adaptation to improve personalized FL and aim to continue to do so in future work. Generally, domain adaptation (DA), including federated domain adaptation, considers the setting of training a model using labeled source datasets and unlabeled, or sparsely labeled, target datasets such that the model performs well in the target domain. As such, the space considers a distinct setting from that considered here but deals with similar data drift issues. For example, in [1], the server holds an unlabeled target dataset while labeled client-hosted datasets serve as the source.

In the revision, we have highlighted the differences and intersections between the disciplines in Section 2, including a discussion of the references provided and those suggested by Reviewer #4. Interestingly, [2] uses a slight variation of APFL and applies DP-SGD for privacy during client training and, as such, does not consider a true DA setting. We have included a reference to this work under settings not considered because the main difference is the use of DP-SGD in client-side training.

[1] X. Peng, Z. Huang, Y. Zhu, and K. Saenko. Federated adversarial domain adaptation. arXiv preprint arXiv:1911.02054, 2019.

[2] D. W. Peterson, P. Kanani, and V. J. Marathe. Private federated learning with domain adaptation. In the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Proceedings of NeurIPS, Vancouver, Canada, 2019.

Reviewer comment (Reproducibility):
Is it possible to release a version of the code that works for open-source datasets?

Author response:
This is very much possible and our intention! In releasing the experimental code associated with the paper, which works with the open-source datasets in FLamby, we will also be making the underlying library, which is not specific to the datasets studied, publicly available. The library is compatible with any open-source dataset that can be used with pytorch. Moreover, the library includes a number of examples to facilitate using the implemented methods for other experiments. Our hope is that this library will accelerate research in FL by making experimentation easier and the implementation of new techniques more straightforward.

Reviewer comment (Novelty):
While adapting FENDA's concept to the FL scenario required some modifications, the application of FENDA in this methodology appears overly incremental. The approach of combining features to utilize both global and local information, as opposed to APFL's method of aggregating logits, is not particularly novel.

Author response:
While the architectural differences between APFL and the FENDA-FL approach are not large, there are several significant benefits to the FENDA-FL design that we believe make the proposed modifications important. First and foremost, FENDA-FL outperforms APFL in many of the experiments and is quite competitive in the cases that it does not. Conceptually, the move from the aggregation of logits to the synthesis of feature representations is fairly straightforward. However, the change also implies that training can be done monolithically and that global and local module combinations can incorporate non-linear transformations rather than just weighted averaging. 

As noted below, the approach also provides significant flexibility in the design of the local and global feature extraction modules and does not require additional hyper-parameters. In APFL, the local and global modules are exact copies, which limits any inductive bias one might inject into the module designs. As pointed out in another comment, we did not demonstrate the advantage of this flexibility as well as we could have. Appendix F now contains results for studies considering the benefit of adding inductive bias to the FENDA-FL model design.

Finally, the move to the FENDA-FL design is well founded on the success of FENDA. In our revision, we have endeavored to make these improvements more clear, while also recognizing that the change in design is straightforward.

Reviewer comment (Novelty):
The methodological contribution to the ICLR community is unclear. The application of FENDA to federated learning is straightforward, "federated checkpointing" is conceptually a combination of early stopping with federated learning.

Author response:
Thank you for the opportunity to emphasize the contributions of this work. While the adaptation of FENDA is straightforward, we believe it is a useful conceptual bridge between personalized FL methods and domain adaptation. In future work, we hope to bring additional ideas from domain adaptation into the setting of personalized FL. With respect to checkpointing, the proposed characterization is certainly applicable.  However, early stopping considers terminating training early, while in the proposed approach federated training continues, with other clients potentially benefiting from further training. Most existing works simply perform a fixed number of server rounds before evaluation.

While the approach is fairly simple, it helps mitigate the risk of overfitting, especially when client data imbalance is an issue. The combination of federated checkpointing and our proposed evaluation criterion aim to bring FL evaluation more in line with standard ML practices. As suggested in another comment, we have conducted experiments without federated checkpointing to demonstrate the advantage of its use. These results are included in Appendix E in the revision.

When considered as a whole, we believe that the experimental framework and results presented here constitute an important representation of effective and fair benchmarking in FL studies. We offer several improvements to evaluation, including checkpointing and various lenses through which FL methods should be evaluated. Finally, the experiments on the FLamby benchmark establish significant improvements for many methods on the selected tasks.

Reviewer comment (Experiments):
The paper lacks a thorough ablation study on federated checkpointing methods. Additionally, while it benefits from not restricting the network architectures between global and local feature extractors, it would have been informative to show performance variations with the use of diverse network architectures.

Author response:
These are extremely valuable suggestions. Ablation studies of federated checkpointing for Fed-Heart-Disease and Fed-IXI are now reported in Appendix E, demonstrating the benefit of the method in several scenarios. Further, experiments studying the effects of architecture variations for the global and local feature extractors in FENDA-FL are discussed in Appendix F. These variations include modifying the size of the underlying feature extractors and the corresponding latent spaces. The experiments demonstrate the benefits that can be seen when injecting inductive bias into the FENDA-FL architecture design.