B ENCHMARKING A LGORITHMS FOR F EDERATED D OMAIN G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we present a unified platform to study domain generalization in the federated learning (FL) context and conduct extensive empirical evaluations of the current state-of-the-art domain generalization algorithms adapted to FL.
In particular, we perform a fair comparison of 11 existing algorithms in solving domain generalization either centralized domain generalization algorithms adapted to the FL context or existing FL domain generalization algorithms to comprehensively explore the challenges introduced by FL.
These challenges include statistical heterogeneity among clients, the number of clients, the number of communication rounds, etc.
The evaluations are conducted on five diverse datasets including PACS (image dataset covering photo, sketch, cartoon, and painting domains), FEMNIST (image dataset containing writing digits and characters written by more than 3500 users), iWildCam (image dataset with 323 domains), Py (natural language processing dataset with 8421 domains) and CivilComments (natural language processing dataset with 16 domains).
The experiments show that the challenges brought by federated learning stay unsolved in realistic experimental settings.
Furthermore, the code base supports fair and reproducible evaluation of new algorithms with little implementation overhead.
I NTRODUCTION
Federated learning (FL) Koneˇcn`y et al.
(2016) is a distributed machine learning approach that assumes each client or device owns a local dataset and this local dataset cannot be exchanged or centrally collected because of privacy or communication constraints.
Given this context, a natural paradigm for FL (e.g., FedAvg McMahan et al.
(2017)) is to alternate between two stages: clients locally update the model based on its local dataset and a central server aggregates client models.
Because the clients may be phones, network sensors, hospitals, or alternative local information sources, the local datasets are naturally heterogeneous between clients.
Specifically, there are at least two types of realistic statistical data heterogeneity in the FL context.
Client heterogeneity is the data heterogeneity between clients involved in training—e.g., hospitals may use different staining procedures or imaging equipment.
Train-test heterogeneity is the data heterogeneity between the training and testing data—e.g., the performance on a new client that was not involved in training or a natural shift in real-world test data due to changes over time, location, or context.
Client heterogeneity has long been considered a statistical challenge since federated learning was introduced.
FedAvg McMahan et al.
(2017) has experimentally shown that their methods effectively mitigate some client heterogeneity.
There are many other extensions based on the FedAvg framework tackling the heterogeneity among clients in FL Hsieh et al.
; Li et al.
Karimireddy et al.
There is an alternative setup in FL, known as the personalized setting, which aims to learn personalized models for different clients to tackle heterogeneity.
Numerous recent papers have proposed FL models and algorithms to accommodate personalization Smith et al.
(2017); Chen et al.
Hanzely et al.
T Dinh et al.
; Deng et al.
Acar et al.
However, these prior works only train the model on simple data sets such as MNIST,
EMNIST, and CIFAR10 and the client heterogeneity is constructed mainly through class imbalance , which assumes the ratio of data from each class is different for each client but the class conditional distributions are homogeneous across clients.
Class imbalance is a special kind of heterogeneity called prior probability shift.
In practice, due to the difference between the location of the local data collector (cameras, sensors, etc), real data heterogeneity is more complex than simple class
K
K
Train-test Heterogeneity M Training
Homogeneous (λ=1)
Heterogenous (λ=0.1)
Domain Separation (λ=0)
Client
Client Client
Client Client
Client
Domain Split
Validation, Test
Domains
Domain Split when K > M
Client
Client Client
Domain
[Figure]
imbalance as in many prior works.
Furthermore, most prior FL works do not consider train-test heterogeneity but rather assume the train and test data are i.i.d.
While most FL works have usually ignored train-test heterogeneity, the task of domain generalization (DG) Blanchard et al.
(2011) formalizes a special case of train-test heterogeneity in which the training algorithm has access to data from multiple source domains and the goal is to perform well on data from an unseen test domain.
There is an active line of research on domain generalization in the centralized setting Muandet et al.
; Saito et al.
(2018); Ganin & Lempitsky (2015);
Long et al.
Arjovsky et al.
; Sagawa et al.
; Shi et al.
; Li et al.
Current benchmark papers Koh et al.
(2021); Gulrajani & Lopez-Paz (2020) provide thorough comparisons between different algorithms for DG.
However, these centralized DG benchmarks do not consider the unique constraints of the FL context.
In particular, they fail to provide insights on how the client dataset heterogeneity, the number of clients, and the communication budget will influence the generalization ability.
From the FL side, to the best of our knowledge, there are currently only four published works, FedADG Zhang et al.
FedDG Liu et al.
, FedSR Nguyen et al.
and FedGMA Tenison et al.
(2022) that tackle DG in the FL context
but their evaluations are limited in the follwing senses: 1) The evaluation datasets are limited in the number and diversity of domains.
FedDG Liu et al.
(2021) only evaluates on retinal fundus images with 4 domains and prostate T2-weighted MRI images from 6 domains, and FedADG only focuses on PACS and VLCS, which share similarities in terms of number of domains (each with only 4 domains) and sample size.
FedGMA Tenison et al.
(2022) only focuses on ColorMNIST with a few domains.
However, we consider more realistic datasets with thousands of domains.
See Sec.
4.2 for details.
Their evaluations are restricted to the case when the number of clients is equal to the number of domains, which may be an unrealistic assumption (e.g., a hospital that has multiple imaging centers or a device that is used in multiple locations).
For example, FedSR Nguyen et al.
(2022) and FedADG
Zhang et al.
(2021) only evaluates on the case when the number of clients equals to the number of domains.
However, we show in this paper, FedSR Nguyen et al.
and
FedADG Zhang et al.
(2021) are sensitive to clients number, and they fail even on the simple dataset PACS when the clients number large than 20 .
See Sec.
4.3 massive number of clients for detail.
They fail to show the network effect of FL; in particular, neither work considers the influence of the number of clients on the performance, and FedADG Zhang et al.
(2021) does not consider the effect of the number of communication rounds.
In summary, the current DG benchmarks fail to consider challenges unique to FL, and the few FL methods for DG have limited evaluations.
Therefore, more systematic evaluation is needed both to aid in systematic progress at the intersection of FL and DG but also to aid in answering many open questions.
For example, how does client domain heterogeneity influence the performance of current algorithms?
What is the performance of a direct translation of centralized DG algorithms (if applicable) to the FL context?
How does the performance of current algorithms scale with the number of clients and communications rounds on complicated real-world datasets?
Major contributions: This work addresses the above questions, and our contributions can be summarized as follows.
We propose a standardized definition of client domain heterogeneity that is unique to the FL context and interpolates between domain homogeneity and domain separation (see Figure 1) while limiting the class imbalance.
In particular, we develop an experimental setup method to split dataset domain samples among any number of clients (see subsection 3.1).
We provide a fair comparison over multiple representative centralized DG methods adapted to the FL context as well as four prior works on federated domain generalization on five different benchmark datasets.
We also explore the impact on the generalization ability of client domain heterogeneity, the total client number, communication rounds, which are unique to the FL context.
From these results, we identify significant generalization gaps between centralized domain generalization and domain generalization in the federated Learning setting.
We release an extensible open-source code library for studying domain generalization in the FL context (see https://github.com/anonymous-lab-ml/benchmarking-dg-fed ).
B ACKGROUND AND R ELATED W ORK
Domain generalization.
Domain generalization (DG) task Blanchard et al.
(2011) tackles the classification problem where the domains of test and training dataset are heterogeneous, i.e., the test data comes from an unseen domain distribution.
Existing DG algorithms design alternative objectives seeking for a good approximation usually by utilizing training data from multiple domains.
Sch ¨ olkopf et al.
(2021) claims that current machine learning model only captures superficial statistical correlation rather than underlying causal relationship between data and their labels.
Inspired by causality Lopez-Paz et al.
; Peters et al.
; Heinze-Deml et al.
(2018), Arjovsky et al.
(2019) proposes IRM seeking invariant predictor on top of the representation.
Li et al.
(2018) instead, explicitly tries to align the feature representation distribution by minimizing the maximum mean discrepancy (MMD) Gretton et al.
Similarly, Sun & Saenko (2016) proposes CORAL which utilizes the second-order information.
Besides invariant representation, Shi et al.
(2021) solves the DG problem using regularization, the proposed FISH algorithm is a first-order method aligning the gradient directions generated by data from different domains.
Sagawa et al.
(2019) introduces the objective that tackles the worst-performed domain.
The proposed algorithm did not directly optimize the objective by solving the minimax objective, but to solve ERM while increasing the importance of domains with larger loss during training.
Zhang et al.
(2017) proposes a straightforward data augmentation principle called Mixup that trains a neural network on convex combinations of data from two distinct domains.
Federated Domain Generalization: Limited works seek to solve the DG task in the FL context.
Liu et al.
(2021) recently proposed FedDG, a federated learning paradigm specifically designed for medical image classification.
The proposed method requires sharing the amplitude spectrum of images among local clients.
Zhang et al.
(2021) applies generative adversarial network (GAN)
Goodfellow et al.
(2020) in the FL context, where each client contains four models: a featurizer Φ , a classifier w , a generator G and a discriminator F .
The featurizer Φ and classifier w solves the classification task, while the generator G learns the feature distribution Z .
During local training, FedADG
Zhang et al.
(2021)first trains the featurizer Φ and classifier w by minimizing the empirical loss, then trains the generator G and the discriminator F using the GAN Goodfellow et al.
(2020) approach.
After the local training, the central server aggregates the local featurizer, classifier and generator.
However, this method requires training 4 models together along with tens of hyperparameters to tune, which makes it hard to converge empirically.
Tenison et al.
(2022) proposes a new aggregation method called Federated Gradient Masking Averaging (FedGMA) with the goal of improving generalization across clients and of the global model.
Their gradient masking prioritizes gradient components that are aligned with the overall dominant direction across clients while the inconsistent components of the gradient are given less importance.
Yuan et al.
(2021) introduced the concept participation gap to identify dataset heterogeneity.
They train models using a set of participating clients and examine their performance on held-out data from these clients as well as an additional set of non-participating clients.
Therefore it is closely related to domain adaptation in FL context, which is different than the domain generalization setting considered in this benchmark, i.e., the training and test domains do not overlap.
Nguyen et al.
(2022) proposed FedSR where they enable domain generalization while still respecting the decentralized and privacy-preserving natures of FL context by enforcing ℓ 2 -norm and a conditional mutual information regularizer on the representation.
In
Table 1, we give an overview of domain heterogeneity across two dimensions: 1) between training and testing datasets (i.e., standard vs domain generalization task) and 2) among clients (i.e., domain imbalance between clients).
While some work has considered the standard supervised learning task (left column), a new fair evaluation is needed to understand the behaviour of domain generalization algorithms in the federated context including the influence of data heterogeneity, communication budget, and the number of clients.
[Table]
P ROBLEM S ETUP 3. D OMAIN HETEROGENEITY .
Algorithm 1 DomainSplit function where w.l.o.g. the domains are assumed to be in descending order, i.e., n 1 ≥ n 2 ≥ · · · ≥ n M .
Input K, n 1 , . .
. , n M if K ≤ M then ∀ k, P k ← ∅ for m = 1 , 2 , . .
, M do k ∗ ∈ arg min k
(cid:80) m ′
∈ P k n m ′
P k ∗ ← P k ∗ ∪ { m } end for else if K > M then ∀ k ∈ { 1 , 2 , . . .
, M } , P k ← { k } for k =
M + 1 , . .
, K do m ∗ ∈ arg max m n m (cid:80)
Kk ′
=1 1 [ m ∈ P k ′ ]
P k ← { m ∗ } end for end
if
Output P k
Implementation of domain heterogeneity We now provide a concrete procedure for implementing domain heterogeneity for the benchmark (see
Figure 1 for an illustration).
Given the number of training samples for all M domains, denoted by { n m } Mm =1 where n m is the number of samples for domain m , we first assign “primary” domains P k ⊆ { 1 , 2 , . . .
, M } to each client via the domain split function defined in Algorithm 1, i.e., { P k } Kk =1 =
DomainSplit ( K, { n m } Mm =1 ) .
Algorithm 1 carefully handles two cases: fewer clients than domains ( K ≤
M ) and more clients than domains ( K > M ).
In the first case, the domains are sorted in descending order and are iteratively assigned to the client k ∗ which has the smallest number of training samples (cid:80) m ′ ∈ P k ∗ n m ′ currently.
In this way, the algorithm outputs { P k } Kk =1 such that no client shares domains with the others but attempts to balance the total number of training samples between clients.
In the case K > M , we first assign the domains one by one to the first M clients.
Then, starting from client k = M + 1 , we iteratively split the largest domain m ∗ , where the samples are evenly split among all clients where m ∗ ∈ P k .
In this way, some clients may share one domain, but no client holds two domains simultaneously.
Again, this also attempts to balance the number of samples across clients as much as possible.
After selecting the primary domains P k for each client, we define the training sample counts, denoted n m,k ( λ ) , for domain m , client k , and domain balance parameter λ ∈ [0 , 1] :
[Equation]
where rounding to integers is carefully handled when not perfectly divisible and where 1 [ · ] is an indicator function.
This is simply a convex combination between a uniform splitting of domains among clients (i.e., the n m K term) and a splitting where each client has a disjoint set of domains (i.e., the 1 [ m ∈ P k ] · n m (cid:80) Kk ′
=1 1 [ m ∈ P k ] term)—unless K > M
and then we try to split domains evenly based on number of samples as defined in Algorithm 1.
After defining n m,k ( λ ) , we can denote the total training samples related to client k with domain balance parameter λ as n k ( λ ) =
(cid:80) Mm =1 n m,k ( λ ) .
3. F EDERATED D OMAIN G ENERALIZATION
In the federated domain generalization problem, we are interested in collaboratively training a model across the clients to perform well on unseen test domains D ∈ D test , which is different from training domains 1 , . .
, M. Therefore, we focus on minimizing not the average loss on the source domains constructed from each client f k ( θ ) =
E ( x,y )
∼D k [ ℓ ((( x, y ); θ ))]
≈ (1 /n k ) (cid:80) n k i =1 ℓ ( x ik , y ik ; θ ) , but on the unseen test domains, in either an average (or worst-case sense) as defined below
[Equation]
To solve equation 2 in the federated learning context, we need to consider client domain heterogeneity, as it can introduce unique challenges for domain generalization.
In particular, local client may not have access to all the domains, and their domains may only overlap partially; in the extreme case, local data included in one client may naturally form a unique domain.
Therefore, whether centralized generalization ability is achievable when local client has heterogeneous domains remains unknown.
Further, if some distributed algorithms can achieve the centralized generalization ability as if the server has information of all the domains, the communication cost remains unclear.
In addition, the impact on the generalization ability of the total clients number remains unknown.
Adapting Centralized DG Methods to FL Setting.
To adapt centralized methods, we simply run the centralized DG method at each client locally with their own local dataset (see subsection 3. for how the local datasets are created), and then compute an average of model parameters at each communication round (see next paragraph).
This approach is straightforward for the homogeneous ( λ = 1 ) and heterogeneous ( λ = 0 . 1 ) settings where each client has data from all domains— albeit quite imbalanced for λ = 0 .
This can be seen as biased updates at each client based on biased local data.
Similarly, this approach can be implemented in the domain separation case if at least one client holds multiple non-overlapping domains (i.e., ∃ k, | P k | ≥ 2 , which would happen if K < M ).
However, if all clients only have one primary domain, i.e., ∀ k, | P k | = 1 , which will happen if K ≥ M , this simple approach cannot be extended to the domain separation setting ( λ = 0 ) because centralized DG methods require data from at least two domains.
In fact, these centralized
DG methods degenerate to ERM if there is only one domain per client.
Extending these methods to the case where all clients only have one domain is an interesting direction for future work.
Synchronization Schedule and Batch Creation For all algorithms, we run E epochs locally on each client and then the server computes a weighted average of the resulting models, i.e., θ t +1 =
(cid:80)
Kk =1 n k n θ t +1 k .
Because each epoch runs through the whole dataset, the k -th client runs (cid:80) m n m,k /B batches, where B denotes the batch size.
For FedAvg (ERM) and FedGMA, we uniform randomly sample a batch from the local dataset without considering the domain labels.
For the FL adaptations of centralized algorithms, we use the sampling method from the WILDS benchmark
Koh et al.
(2021), namely each client uniform randomly samples two domains from its local dataset and then uniform randomly samples B/ 2 examples from each domain without replacement.
Beyond simple model averaging, FedGMA Tenison et al.
(2022) additionally adopts a global masking operation at the server for update changes.
We refer the readers to Appendix B.2 for the detailed hyperparameters including learning rate, batch size, and model selection.
M AIN R ESULTS
We adapt six representative centralized DG methods into FL context, include IRM Arjovsky et al.
(2019), Fish Shi et al.
MMD Li et al.
(2018), Coral Sun & Saenko (2016), GroupDRO
Sagawa et al.
Mixup Zhang et al.
(2017), and compare them with FedDG Liu et al.
FedADG Zhang et al.
, FedSR Nguyen et al.
and FedGMA Tenison et al.
(2022) which are naturally designed for solving domain generalization tasks in federated learning.
4. B ASELINE S ETTING (PACS AND FEMNIST-62)
In the baseline setting, we consider three domain heterogeneity regimes on image classification tasks PACS Li et al.
(2017) widely used in domain generalization, and on FEMNIST-62 (digits and characters)
Cohen et al.
(2017), a prototype dataset in the FL context.
PACS.
We evaluate PACS with K = 100 clients, M = 2 training domains (cartoon, sketch),validation domain (art painting) and 1 test domain (photo).
The maximum communication rounds is set to 50 with 1 local epoch per communication.
The domain balance parameter was varied λ ∈ { 1 , 0 . 1 , 0 } .
For the domain separation case λ = 0 , each client locally only has one training domain (i.e., ∀ k,
| P k | = 1 ); in this case, no centralized methods are suitable (see subsection 3.2) so we only compare the four federated domain generalization methods to ERM.
Figure 2 plots the held-out test accuracy versus communication rounds on PACS with increasing domain heterogeneity.
As seen from Figure 2, 1) most algorithms perform reasonably well in the homogeneous case ( λ = 1 ) except FedADG and FedSR. In fact those two fail in all three cases.
They are sensitive to the client number and work favorably when K is small, e.g. K = 2 , see subsection 4.2 for discussion on the effect of number of clients.
2) As domain heterogeneity increases, i.e., λ from 1 to 0 .
1 , the algorithms consistently converge slower and have worse test accuracy, which demonstrates that domain heterogeneity among clients is a unique extra challenge introduced by FL.
In particular, the centralized DG methods FISH, CORAL, IRM, and MMD extended to the FL setting have poor performance compared with that of ERM in the heterogeneous case, while GroupDRO outperforms ERM both in homogeneous and heterogeneous case.
In the domain separation case, because K > M for PACS, each client locally only holds one training domain, and thus no centralized methods are suitable to use.
FedDG requires sharing the amplitude spectrum of images among local clients, which causes privacy concerns.
Therefore, even for this dataset containing only four domains in total, prior works struggle to compete with ERM.
 
  
 311<2-$!;-32 W W W¤ W¥     ' 9 ;   $$ < 8 !
$ @ T,313+'2'3< 
   
 311<2-$!;-32 WT,';'83+'2'3< 

    311<2-$!;-32 T&31!-29'6!8!;-3 	   -9,  38!£ -?<
83<6
'&  '&  '&  '&
[Table]
[Table]
We also report the final test accuracy using held-out-domain validation in Table 2, similar results with in-domain validation is deferred to
Table.
13 in Sec.
Appendix C. It shows as the domain heterogeneity increases ( λ from 1 to 0 ), the test accuracy of current centralized methods (except GroupDRO) degrades dramatically using either validation principle.
For the current four natural FL algorithms, only FedDG works comparably to ERM, FedGMA performs worse than the above two on PACS, and because FedADG and FedSR are sensitive to the number of clients, they both fail to converge in this case as K = 100 .
FEMNIST-62 (digits and characters).
We consider the FL prototyping dataset FEMNIST62 with K = 100 clients and treat handwritten digits and characters each users created as a natural domains.
We split into M = 2586 training domains, 320 validation domains, and 331 test domains.
The maximum communication rounds is set to 50 with 1 local epoch per communication.
The domain balance parameter is varied λ ∈ { 1 , 0 . 1 , 0 } .
For the homogeneous and heterogeneous cases, each client locally holds M = 2586 training domains.
The data are divided according to Equation 1.
For the domain separation case, given that K < M, we use Algorithm 1 to split the domains to each client.
Therefore, each client holds multiple non-overlapping domains.
All of the methods are applicable on this dataset.
We observe that as the domain balance parameter λ decreases from 1 to 0 , FedGMA, FedSR are consistently comparable to ERM while the others fail.
The in-domain and held-out domain accuracy are reported in Table 7, Sec.
Appendix C.

  
311<2-$!;-32 W W W¤ W¥     ' 9 ;   $$ < 8 !
$ @ T,313+'2'3< 
   311<2-$!;-32 WT,';'83+'2'3<

  
311<2-$!;-32 T&31!-29'6!8!;-3 	   -9,  38!£ -?<
83<6
'&  '&

4. M ORE R EALISTIC D ATASETS (P Y 150, C IVIL C OMMENTS , I W ILD C AM )
Centralized
[Table]
[Figure]
[Table]
In this subsection, we tackle more realistic domain generalization tasks in the FL context.
We include three domain heterogeneity regimes on natural language processing dataset
Py150 (with 8421 domains), CivilComments (with 16 domains), and image dataset iWildCam (with 323 domains).
Note Mixup Zhang et al.
FedDG Liu et al.
(2021), and
FedADG Zhang et al.
(2021) are only suitable for image datasets, thus are excluded in the experiments on Py150 and CivilComments.
In the following experiments on these three datasets, we choose maximum communication round C = 50 for IWildCam, and C = 10 for
Py150 and CivilComments, 1 local epoch per communication.
For the homogeneous and heterogeneous case, respectively, each client locally holds all the training domains.
For the domain separation case, we use Algorithm 1 to split the domains to each client.
When K < M, client locally holds non-overlapping domains; all of the centralized methods are applicable in this case.
When K ≥ M, each client locally only holds one training domain; thus, no centralized method is applicable, only four natural federated domain generalization methods are applicable, we compare them with ERM.
Py150: We evaluate Py150 with 100 clients, 5477 training domains, 261 validation and 2471 test domains.
Given that K < M, we compare all the methods except Mixup, FedDG and FedADG in all three different domain heterogeneity regimes.
IWildCam: We evaluate IWildCam withclients, 243 training domains, 32 validation and 48 test domains.
We compare all the methods in homogeneous and heterogeneous regimes.
Given that K = M, we can only compare FedDG,
FedADG, FedSR, FedGMA with ERM in the domain separation regime.
CivilComments:
CivilComments is a special kind of DG where the test domain is a subpopulation of the training domain, and our goal is to perform well on the worst-case domain.
CivilComments contains 100 clients and 16 domains.
We compare all the methods except Mixup, FedDG and FedADG in homogeneous and heterogeneous regimes.
Given that K < M, we can only compare FedSR, FedGMA with ERM in the domain separation regime.
[Table]
    2<1#'83($£-'2; K T&31!-29'6!8!;-3 	   -9, -?<  38!£ 83<6
'&  '&  '&
We compare the final test accuracy using held-out validation in Table 4a for Py150, Table 4b for
CivilComments, and Table 3 for IWildCam.
(Using two validation criterion are summarized in
Table 8, Table 9 and Table 10 in Appendix C).
The results show that the performance of ERM dominates the other algorithms on these three datasets.
No algorithm achieves its centralized domain generalization ability, where centralized corresponds to training on the centralized dataset gathered from all the training clients.
We also plot accuracy versus communication figures in Figure 7 for
Py150, Figure 6 for CivilComments and Figure 8) for IWildCam in Sec.
C. 4. A DDITIONAL FL- SPECIFIC CHALLENGES FOR DOMAIN GENERALIZATION
Besides domain heterogeneity, we also investigate the challenges brought by FL, including massive number of clients number and communication constraints, which are unique to the FL setting.
i)
Massive number of clients: In this experiment, we explore the performance of different algorithms when the number of clients K increases on PACS.
We fix the communication rounds C
=and the local number of epoch is 1 (synchronizing the models every epoch).
Figure 4 plots the held-out DG test accuracy versus number of clients for different levels of data heterogeneity.
The following comments are in order: given communication budget, 1) current domain generalization methods all degrade a lot in particular after K ≥ 50 , while the performance ERM and FedDG maintain relatively unchanged as the clients number increases given communication budget.
FedADG and FedSR are are sensitive to the clients number, and they both fail after K ≥ 20 . 2)
Even in the simplest homogeneous setting λ
= 1 , where each local client has i.i.d training data, current domain generalization methods IRM, FISH, Mixup, MMD, Coral, GroupDRO
work poorly in the existence of large clients number, this means new methods are needed for DG in FL context when data are stored among massive number of clients.
ii) Communication constraint: To show the effect of communication rounds on convergence, we plot the test accuracy versus communication rounds in Figure 5.
We fix the number of clients K = 100 on PACS and decreases rounds of communication (together with increasing local epochs), that is, C = (50 , 10 , 5) (with E = (1 , 5 , 10) ).
That is, if the regime restricts the communication budget, then we increase its local computation E to have the same the total computations.
Therefore, the influence of communication on the performance is fair between algorithms because the total
data pass is fixed.
We observe that 1) the curves are relatively “flat” for most of algorithms, this is predictable because we vary the number of pass of the data E per round for a changing C , and locally the aggregation rules are the same θ t +1 =
(cid:80)
Kk =1 n k n θ t +1 k , where n k is the training sample size at client k. 2)
In particular, ERM, GroupDRO, and FedDG can achieve comparable good performance when communication budget is low ( C = 10 ) comparable to when there communication budget is high ( C = 50 ), showing their communication efficiency in the FL context.

  
[Figure]
W W W¤ W¥ 
[Figure]
9 ;   $$ < 8 !
$ @ T,313+'2'3< 
  
 2<1#'83($311<2-$!;-32 C WT,';'83+'2'3< 
  
 2<1#'83($311<2-$!;-32 C T&31!-29'6!8!;-3 	   -9, -?<  38!£ 83<6
'&  '&
C ONCLUSION AND D ISCUSSION
This work evaluates multiple algorithms to solve domain generalization task in the federated learning context.
We evaluate the influence of client domain heterogeneity, the total number clients, and communication rounds on the domain generalization ability.
We show that DG in FL context is an unsolved problem, and it brings abundant new challenges.
Specifically, the following aspect might be future directions of interest: 1) domain heterogeneity across the clients dramatically impacts the generalization ability on image dataset PACS and IWildCAM as well as NLP datasets Py150 and
CivilComments; designing new FL algorithm to recover centralized domain generalization ability remains open; 2) previous works (eg, FedSR, FedADG) evaluate the generalization ability built upon small number of clients K is not enough, massive clients setting needs to be take into consideration; 3) more realistic datasets need to be considered in the domain generalization in FL context; 4) For the domain separation case, few prior works are applicable to the case where each client only holds one domain—new DG algorithms for the FL setting are required for this case.
We list the gap table below for summarizing the current DG algorithms performance gap w.r.t ERM in the FL context, in particular, positive means it outperforms ERM, negative means it is worse than ERM.
It can be seen that in the on the simple dataset, the best DG migrated from centralized setting is better than ERM
[Table]
In the domain separation case, no centralized DG algorithms can be adapted to it
[Table]
and FedDG and
FedADG performs comparably good in this setting.
However, they fail in harder datasets.
Federated
DG algorithms that outperforming ERM, supporting NLP dataset
[Table]
and free of data sharing are still in need.
[Table]
A R EPRODUCIBILITY S TATEMENT
Code for reproduce the result is available at the anonymous link.
We include detailed documentation in using our code to reproduce the results throughout the paper.
We also provide documentation in adding new algorithm’s DG evaluation in the FL context.
B E XPERIMENTS SETTING SUPPLEMENTARY B. D ATASETS AND M ODELS
In this section, we introduce the datasets we used in our experiments, and the split method we used to build heterogeneous datasets in the training and testing phase as well as the heterogeneous local training datasets among clients in the FL. Datasets.
We contain three datasets as the benchmark:
PACS Li et al.
, IWildCam Koh et al.
(2021), and Py150 Koh et al.
These three datasets cover different levels of difficulty as well as different types of tasks.
PACS and IWildCam are both image classification datasets and Py150 is a natural language processing (NLP) dataset.
PACS is an image dataset for domain generalization.
It consists of four domains, namely Photo (1,670 images),
Art Painting (2,048 images), Cartoon (2,344 images), and Sketch (3,929 images).
This task requires learning the classification task on a set of objects by learning on totally different renditions.
The
Py150 is a natural language processing dataset containing 150 , 000 python source code dataset from 8 , 421 repository.
The goal is to predict the next token given the context of previous tokens.
This is a real-world NLP dataset that contains multiple repositories which naturally form multiple domains.
The IWildCam contains wild animals captured by multiple heats or motion-activated static cameras.
Due to the variation in camera model, position, color, background, and relative animal frequencies, the samples form multiple domains.
It contains 203 , 029 images from 323 different camera traps, the images contain 182 different animal species.
For Py150 and IWildcam datasets, we follow the same split method as the Wilds Koh et al.
For PACS, we use cartoon and sketch as the training domains, art-painting as the held-out-validation domain, and photo as the test domain, and we use 90% of the data from cartoon and sketch domains to be the training domains, and about 5% to be in-domain validation, and other 5% to be the in-domain test set.
During the sampling, we keep the class distribution the same among the training dataset, the in-domain validation dataset, and the in-domain test dataset.
Models.
For image classification datasets PACS and IWildCam, we use
ResNet50 model He et al.
(2016), and Py150 is a natural language processing (NLP) dataset where we use OpenAI GPT2 Radford et al.
(2019) to train.
B. H YPERPARAMETERS AND M ODEL S ELECTION
Hyperparameters To make fair comparisons, we allocate the same budget during training for each algorithm on each dataset.
The budget includes the times of allowed hyperparameter search, model architecture, local computation resources, and communication rounds.
For each dataset, we fix the model architecture and initialization to be the same.
We conduct eight times hyperparameter searching for each algorithm, and choose the set of hyperparameters that achieves the best performance.
For all the datasets and algorithms, we set 100% for clients’ participation in the training during each communication.
For PACS, we fix the number of clients K = 100 , 50 communications in total where each communication happens after one epoch of local training.
the batch size is 64 .
We use Adam optimizer and the learning rate is 1 × 10 − 3 except for FedSR algorithm, where we choose SGD with lr = 0 .
002 , momentum = 0 . 9 and weight decay = 5 × 10 − 4 .
For Py150, we set K = 100 , and the total number of communication is 3 in the centralized case and 10 in the distributed case.
The batch size is 96 .
We use AdamW optimizer, lr = 8 × 10 − 5 , and ϵ = 1 × 10 − 8 .
For IWildcam, the number of clients K = 243 , 12 communications in the centralized case, andcommunications in the federated learning case, and the batch size is 16 .
We use Adam optimizer where the learning rate is 3 × 10 − 5 .
For FEMNIST, the number of clients K = 100 , 20 communications in the centralized case, andcommunications in the federated learning case, and the batch size is 64 .
We use Adam optimizer where the learning rate is 1 × 10 − 3 .
For CivilComments, the number of clients K = 100 , 5 communications in the centralized case, and 10 communications in the federated learning case, and the batch size is 16 .
We use Adam optimizer where the learning rate is 1 × 10 − 5 .
For each dataset, we choose the hyperparameters starting from their default value consistent with the choice in previous benchmarks DomainBed Gulrajani & Lopez-Paz (2020)
Wilds Koh et al.
(2021), or their value proposed in the original paper .
Model Selection In DG, model selection could significantly affect the model performance.
During training, we evaluate the aggregated model by using the validation dataset after each communication round.
After performing C communication rounds, we select the model that achieves the best performance on the validation dataset.
This work adopts two model selection methods for the DG task: in-domain and held-out-domain model selection Gulrajani & Lopez-Paz (2020).
In-domain model selection method uses validation dataset which is independent and identically sampled from the training dataset.
The held-out-domain model selection method uses validation dataset that only contains examples from a set of domains that do not overlap with the training and testing domains.
C O THER EVALUATION
We put our extra evaluations here for reference.
The experiments are summarized in terms of dataset.
C. PACS and FEMNIST
[Table]
We report the final test accuracy using two validation criterion in Table 6.
It shows that the homogeneity case ( λ = 1 column) may even slightly outperform the its counterpart centralized domain generalization accuracy in the simple case (small client and domain numbers).
This could come from the natural regularization brought by the FL.
In the domain separation case ( λ = 0 column), although FedADG is not communication efficient as shown in Fig 2, it seems to be more robust across validation strategies whereas FedDG performs poorly using in-domain validation.
Overall, as expected, existing DG algorithms with enough communication rounds are able to perform reasonably well in this simple setting where the dataset is simple and the number of clients and domains are small.
[Table]
[Table]
C. M ORE REALISTIC DATASETS (P Y 150, C IVIL C OMMENTS , I W ILD C AM )
To observe the convergence of each algorithm
[Table]
we plot Figure 7 for Py150 (resp.
Figure 6 for CivilComments and Figure 8 for IWildCam).
It shows that with realistic dataset as well as with non-trivial number of clients, all of the algorithms tend to be more sensitive to domain heterogeneity.
Even in the heterogeneous case, where each client locally holds all the training domains, their generalization abilities on the unseen domains are much worse than its centralized counterpart; let alone to solve the even harder domain separation case.
We also reported accuracy using the in-domain and held-out domain validation for Py150 in Table 8,
CivilCommnets in Table
[Table]
and IWildCAM in Table 10.
[Table]
[Table]
[Table] FedGMA
N.A.
N.A.
  ¤ ¥ 
311<2-$!;-32 W W W W¤ 
 ' 9 ;   $$ < 8 !
$ @ T,313+'2'3<   ¤ ¥  311<2-$!;-32 WT,';'83+'2'3<   ¤ ¥

311<2-$!;-32 T&31!-29'6!8!;-3 	   -9,  38!£ -?< 83<6
'&  '&  '&  '&
  ¤ ¥

311<2-$!;-32 W W W W¤ W 
 ' 9 ;   $$ < 8 !
$ @ T,313+'2'3<   ¤ ¥  311<2-$!;-32 WT,';'83+'2'3<   ¤ ¥

311<2-$!;-32 T&31!-29'6!8!;-3 	   -9,  38!£ 83<6
'&
 
  
 311<2-$!;-32 W W
W W     ' 9 ;   $$ < 8 !
$ @ T,313+'2'3< 
   
 311<2-$!;-32 WT,';'83+'2'3< 

  
 311<2-$!;-32 T&31!-29'6!8!;-3 	   -9,  38!£ -?<
83<6
'&  '&
C. A DDITIONAL FL- SPECIFIC CHALLENGES FOR DOMAIN GENERALIZATION
In this subsection, we reported the additional results related to Sec 4.3.
Including Table 11, Table 12,
Table 13, Table 14 for test accuracy using the held-out and in-domain validation for PACS with
K = 20 , 50 , 100 , 200 , respectively.
[Table]
[Table]
[Table]
[Table]
[Table]