Paper: Benchmarking Algorithms for Federated Domain Generalization (ICLR-2024)

Reviewer comment (Experiments):
In the context of domain generalization, we are particularly concerned with whether models trained on limited source domains can generalize to new target domains. This paper also uses held-out domains (in Appendix C.2). I believe that this aspect should be more explicitly explained in the main body of the text; otherwise, readers might easily misconstrue that all domains were used to construct training clients, which is misleading.

Author response:
Good point. We agree that our setup of held-out validation domains is of great importance and need better visibility. We have added the corresponding part to the main paper section 4.3, page 7 per the reviewer’s suggestion, where we emphasize that we are using validation domain dataset: “In DG task, we cannot access the test domain data. However, we are particularly concerned about the model performance outside the training domains, thus we preserve a small portion of the domains we can access as held-out validation domains, and the held-out validation domains are used for model selection and early stopping. ”

Reviewer comment (Evaluation):
Error bars are not provided for experiment results, the conclusion may be influence by random fluctuation.

Author response:
Great suggestion. Following the Reviewer's suggestion, we have included in the revised manuscript error bar in Section 4.3. In particular, we have included the standard deviation of the repeated experiments on PACS, CelebA, Cameylon17. Due to the computational challenges of training many different models with different settings, we keep working on the results on Py150, CivilComments, IWildCam and FEMNIST datasets, and we will include them in the final version.

Reviewer comment (Experiments):
In Eq. (1), two kinds of DG is mentioned. Which kind of DG is mainly used in your benchmark?

Author response:
Great question. We used both formulations. We elaborate this both in terms of methods and datasets.

Regarding methods, since test domains are inaccessible for direct optimization, DG methods typically start from one of the two theoretical formulations and seek to find an approximate objective on training domains to work with. For instance, Fish starts with the average-case objective, and IRM, GroupDRO starts with the worst-case objective. We included diverse methods from both formulations.

Regarding datasets, PACS, FEMNIST, Camelyon17, and IWildCam focus on averaging test domain performance, aligning with the first DG objective for evaluation. Among them, IWildCam uses the F1 Score, which is a weighted average, placing greater emphasis on rarer species. In contrast, CelebA and CivilComments prioritize worst domain accuracy, aligning with the second DG objective. Further, Py150 is unique in targeting a specific sub-population accuracy, termed 'method-class accuracy.' This is particularly relevant for Py150's application in code completion, where the accuracy of predicting method and class names is critical.

Reviewer comment (Evaluation):
The performance for some algorithm, for example, FedSR, is very low, and consistently lower than FedAvg. Could you explain the reason behind?

Author response:
Thank you for your question. First, we would like to point out that we replicated the results in the experimental setting of the FedSR paper, where their evaluation on the PACS dataset is only based on 3 clients. In our benchmark, we observed two new phenomena for FedSR:
- the degradation in performance when the number of clients is large (increase up to $100$)
- the hyperparameters are also more sensitive compared to their 3 clients setting. 

As suggested by reviewer AnGM, we conducted repeated experiments (Sec. 4.3), where we provided each method with a larger hyperparameter tuning budget, thus, FedSR achieves better results than in the original experiments. Even with an increased budget, we observed that FedSR is still significantly worse than FedAvg ($40\%$ vs $90\%$ on PACS). Thus, it does not alter the conclusion of the original manuscript.

**A potential explanation of the phenomena**

The convergence analysis is well established for FedAvg in the literature, see [3] and subsequent works. However, there’s no theoretical guarantees for FedSR, where the objective function is constructed by the summation of local data loss and local non-smooth penalty. We suspect that the extra heterogeneous local nonsmooth penalty terms introduced in FedSR potentially diverts each client's learning trajectory, leading to harder convergence when client number increases. 

[3] Stich, Sebastian U. "Local SGD converges fast and communicates little." arXiv preprint arXiv:1805.09767 (2018).

Reviewer comment (Theory):
The paper mainly summarizes the experimental observations, but does not offer much theoretical analysis or explanation for why some methods perform better than others in certain scenarios, which would be helpful to gain more insights into the FDG problem and the design of effective methods.

Author response:
We appreciate the reviewer for the comments and feedback. While we cannot fully explain all performance differences, we suggest a few possible explanations and general insights that may help in designing more effective federated DG methods.

**Mismatch between method’s (implicit) assumptions vs real-world datasets** One potential issue is that each method may implicitly or explicitly assume certain conditions about the data generating process. For instance, IRM assumes the special structural equation model and FedDG assumes that the amplitude spectrum mainly consists of style information and the phase spectrum mainly consists of object information. Yet, the underlying data generating process may not match these assumptions. This mismatch may lead to varying relative performance across datasets. More careful analysis of the implicit or explicit assumptions of each method compared to real-world datasets could be a fruitful future research direction.

**Privacy Constraint** Centralized domain generalization methods usually assume that the data from each domain are accessible. Yet, in practice, each client may only have access to one domain (our domain separation case). Thus, centralized methods are not directly applicable in the domain separation case. A potential research direction for this could be designing a DG objective to be decomposable across domains so that it is amenable for FL.

**Convergence in FL setting** While in a centralized setting, it is often assumed that the method will naturally converge using an appropriate learning rate, the speed of convergence and solution at convergence may have a much higher impact on DG in the FL setting. For example, FedSR and FedGMA may generalize well but the convergence is very slow when the number of clients is large.  On the other hand, the convergence analyses are well established for FedAvg, FedProx, Scaffold in the literature, see [2] and subsequent works. However, there’s no theoretical guarantees for newly designed methods for Federated domain generalization. Because FedSR’s objective function is constructed by the summation of local data loss and local non-smooth penalty, we suspect that the heterogeneous local nonsmooth penalty terms introduced in FedSR potentially diverts each client's learning trajectory, leading to slow convergence when client number increases. For other methods that are designed to converge quickly, like FedProx and Scaffold, the solution they converge to does not generalize well. This is because they were designed to address client heterogeneity but were not designed for train-test heterogeneity (i.e., distribution shifts). Therefore, it is unsurprising that they do not perform well on DG tasks. Given all of this, a potential research direction would be to combine the strengths of provably convergent FL algorithms with DG objectives or regularizations. It seems that for good federated DG performance, both aspects need to be solved simultaneously.
Thus, how to design a method generalize better than simple FedAVG and maintaining the convergence property of FedAvg is of great importance. This is the message we wish to convey in the benchmark.  We will incorporate those explanations in the paper to provide intuition on solving domain generalization in the Federated context. 

[1] Koh, Pang Wei, et al. "Wilds: A benchmark of in-the-wild distribution shifts." International Conference on Machine Learning. PMLR, 2021.

[2] Stich, Sebastian U. "Local SGD converges fast and communicates little." arXiv preprint arXiv:1805.09767 (2018).