W HAT D O W E M EAN BY G ENERALIZATION IN F EDERATED L EARNING ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated learning data is drawn from a distribution of distributions: clients are drawn from a meta-distribution, and their data are drawn from local data distributions.
Thus generalization studies in federated learning should separate performance gaps from unseen client data ( out-of-sample gap ) from performance gaps from unseen client distributions ( participation gap ).
In this work, we propose a framework for disentangling these performance gaps.
Using this framework, we observe and explain differences in behavior across natural and synthetic federated datasets, indicating that dataset synthesis strategy can be important for realistic simulations of generalization in federated learning.
We propose a semantic synthesis strategy that enables realistic simulation without naturally-partitioned data.
Informed by our ﬁndings, we call out community suggestions for future federated learning works.
I NTRODUCTION
Federated learning (FL) enables distributed clients to train a machine learning model collaboratively via focused communication with a coordinating server.
In cross-device FL settings, clients are sampled from a population for participation in each round of training (Kairouz et al., 2019; Li et al., 2020a).
Each participating client possesses its own data distribution, from which ﬁnite samples are drawn for federated training.
Given this problem framing, deﬁning generalization in FL is not as obvious as in centralized learning.
Existing works generally characterize the difference between empirical and expected risk for clients participating in training (Mohri et al., 2019; Yagli et al., 2020; Reddi et al., 2021;
Karimireddy et al., 2020; Yuan et al., 2021).
However, in cross-device settings, which we focus on in this work, clients are sampled from a large population with unreliable availability.
Many or most clients may never participate in training (Kairouz et al., 2019; Singhal et al., 2021).
Thus it is crucial to better understand expected performance for non-participating clients.
In this work, we model clients’ data distributions as drawn from a meta population distribution (Wang et al., 2021), an assumption we argue is reasonable in real-world FL settings.
We use this framing to deﬁne two generalization gaps to study in FL: the out-of-sample gap , or the difference between empirical and expected risk for participating clients, and the participation gap , or the difference in expected risk between participating and non-participating clients.
Previous works generally ignore the participation gap or fail to disentangle it from the out-of-sample gap, but we observe signiﬁcant participation gaps in practice across six federated datasets (see Figure 1), indicating that the participation gap is an important but neglected feature of generalization in FL.
We present a systematic study of generalization in FL across six tasks.
We observe that focusing only on out-of-sample gaps misses important effects, including differences in generalization behavior across naturally-partitioned and synthetically-partitioned federated datasets.
We use our results to inform a series of recommendations for future works studying generalization in FL.
Our contributions:
• Propose a three-way split for measuring out-of-sample and participation gaps in centralized and FL settings where data is drawn from a distribution of distributions (see Figure 2).
• Observe signiﬁcant participation gaps across six different tasks (see Figure 1) and perform empirical studies on how various factors, e
g., number of clients and client diversity, affect generalization performance (see Section 5).
[Figure]
We conduct experiments on four image classiﬁcation tasks and two text prediction tasks.
As described in Section 3.1, the participation gap can be estimated as the difference in metrics between participating validation and unparticipating data (deﬁned in Figure 2).
Prior works either ignore the participation gap or fail to separate it from other generalization gaps, indicating the participation gap is a neglected feature of generalization in FL.
• Observe signiﬁcant differences in generalization behavior across naturally-partitioned and synthetically-partitioned federated datasets, and propose semantic partitioning , a dataset synthesis strategy that enables more realistic simulations of generalization behavior in FL without requiring naturally-partitioned data (see Section 4).
• Present a model to deﬁne the participation gap (Section 2), reveal its connection with data heterogeneity (Section 3.2), and explain differences in generalization behavior between labelbased partitioning and semantic partitioning (Section 4.2).
• Present recommendations for future FL works, informed by our ﬁndings (see Section 6).
• Release an extensible open-source code library for studying generalization in FL (see ?? ).
1. R ELATED WORK
We brieﬂy discuss primary related work here and provide a detailed review in Appendix A.
We refer readers to Kairouz et al.
; Wang et al.
for a more comprehensive introduction to federated learning in general.
Distributional heterogeneity in FL.
Distributional heterogeneity is one of the most important patterns in federated learning (Kairouz et al., 2019; Wang et al., 2021).
Existing literature on FL heterogeneity is mostly focused on the impact of heterogeneity on the training efﬁciency (convergence and communication) of federated optimizers (Karimireddy et al., 2020; Li et al., 2020b; Reddi et al., 2021).
In this work, we identify that the participation gap is another major outcome of the heterogeneity in FL, and recommend using the participation gap as a natural measurement for dataset heterogeneity.
Personalized FL.
In this work, we propose to evaluate and distinguish the generalization performance of clients participating and non-participating in training.
Throughout this work, we focus on the classic FL setting (Kairouz et al., 2019; Wang et al., 2021) in which a single global model is learned from and served to all clients.
In the personalized FL setting (Hanzely & Richt´arik, 2020; Fallah et al., 2020; Singhal et al., 2021), the goal is to learn and serve different models for different clients.
While related, our focus and contribution is orthogonal to personalization.
In fact, our three-way split framework can be readily applied in various personalized FL settings.
For example, for personalization via ﬁne-tuning (Wang et al., 2019; Yu et al., 2020), the participating clients can be deﬁned as the clients that contribute to the training of the base model.
The participation gap can then be deﬁned as the difference in post-ﬁne-tuned performance between participating clients and unparticipating clients.
Out-of-distribution generalization.
In this work, we propose to train models using a set of participating clients and examine their performance on heldout data from these clients as well as an additional set of non-participating clients.
Because each client has a different data distribution, unparticipating clients’ data exhibits distributional shift compared to the participating clients’ validation data.
Therefore, our work is related to the ﬁeld of domain adaptation (Daum´e III, 2009; Ben-David et al., 2007; Shimodaira, 2000; Patel et al., 2015), where a model is explicitly adapted to make predictions on a test set that is not identically distributed to the training set.
The participation gap that we observe is consistent with ﬁndings from the out-of-distribution research community (Ovadia et al., 2019;
Amodei et al., 2016; Lakshminarayanan et al., 2016), which shows on centrally trained (non-federated) models that even small deviations in the morphology of deployment examples can lead to systematic degradations in performance.
Our setting differs from these other settings in that our problem framing assumes data is drawn from a distribution of client distributions, meaning that the training and deployment distributions eventually converge as more clients participate in training.
In contrast, the typical OOD setup assumes that the distributions will never converge (since the deployment data is out-of-distribution, by deﬁnition it does not contribute to training).
Our meta-distribution assumption makes the problem of generalizing to unseen distributions potentially more tractable.
S ETUP FOR G ENERALIZATION IN FL
We model each FL client as a data source associated with a local distribution and the overall population as a meta-distribution over all possible clients.
Deﬁnition 2.1 (Federated Learning Problem) .
Let Ξ be the (possibly inﬁnite) collection of all the possible data elements, e.g., image-label pairs.
For any parameters w in parameter space
Θ , we use f ( w , ξ ) to denote the loss at element ξ ∈ Ξ with parameter w .
Let C be the (possibly inﬁnite) collection of all the possible clients.
Every client c ∈ C is associated with a local distribution D c supported on Ξ .
Further, we assume there is a meta-distribution P supported on client set C , and each client c is associated with a weight ρ c for aggregation.
The goal is to optimize the following two-level expected loss as follows:
[Equation]
Similar formulations as in Equation (1) have been proposed in the existing literature (Wang et al., 2021; Reisizadeh et al., 2020).
To understand Equation (1), consider a random procedure that repeatedly draws clients c from the meta-distribution P and then evaluates the loss on samples ξ drawn from the local data distribution D c .
Equation (1) then characterizes the weighted-average limit of the above process.
Remark.
The selection of client weights { ρ c : c ∈ C } depends on the desired aggregation pattern.
For example, setting ρ c ≡ 1 will equalize the performance share across all clients.
Another common example is setting ρ c to be proportional to the training dataset size contributed by client c .
Intuitive Justiﬁcation.
The formulation in Equation (1) is especially natural in cross-device FL settings, where the number of clients is generally large and modeling clients’ local distributions as sampled from a meta-distribution is reasonable.
This assumption also makes the problem of generalization to non-participating client distributions more tractable since samples from the metadistribution are seen during training.
Discretization.
While the ultimate goal is to optimize the expected loss over the entire metadistribution P and client local distributions {D c : c ∈ C } , only ﬁnite training data and a ﬁnite number of clients are accessible during training.
We call the subset of clients that contributes training data the participating clients , denoted as ˆ C .
We assume ˆ C is drawn from the meta-distribution P .
For each participating client c ∈ ˆ C , we denote ˆΞ c the training data contributed by client c .
We call these data participating training client data and assume ˆΞ c satisﬁes the local distribution D c .
Deﬁnition 2.2.
The empirical risk on the participating training client data is deﬁned by
[Equation]
Equation (2) characterizes the performance of the model (at parameter w ) on the observed data possessed by observed clients.
There are two levels of generalization between Equation (2) and Equation (1): (i) the generalization from ﬁnite training data to unseen data, and (ii) the generalization from ﬁnite participating clients to unseen clients.
To disentangle the effect of the two levels, a natural intermediate stage is to consider the performance on unseen data of participating (seen) clients.
Deﬁnition 2.3.
The semi-empirical risk on the participating validation client data is deﬁned by
[Equation]
[Equation]
Equation (3) differs from Equation (2) by replacing the intra-client empirical loss with the expected loss over D c .
We shall also call F ( w ) deﬁned in Equation (1) the unparticipating expected risk and denote it as F unpart ( w ) for consistency.
Now we are ready to deﬁne the two levels of generalization gaps formally.
Deﬁnition 2.4.
The out-of-sample gap is deﬁned as F part val ( w ) − F part train ( w ) .
Deﬁnition 2.5.
The participation gap is deﬁned as F unpart ( w ) − F part val ( w ) .
Note that these gaps are also meaningful in centralized learning settings where data is sampled from a distribution of distributions.
U NDERSTANDING G ENERALIZATION G APS 3. E STIMATING R ISKS AND G APS VIA THE T HREE -W AY S PLIT
Both F part val and F unpart take an expectation over the distribution of clients or data.
To estimate these two risks in practice, we propose splitting datasets into three blocks.
The procedure is demonstrated in Figure 2.
Given a dataset with client assignment, we ﬁrst hold out a percentage of clients ( e.g., 20%) as unparticipating clients, as shown in the rightmost two columns (in purple).
The remaining clients are participating clients.
We refer to this split as inter-client split .
Within each participating client, we hold out a percentage of data ( e.g., 20%) as participating validation data, as shown in the upper left block (in orange).
The remaining data is the participating training client data, as shown in the lower left block (in blue).
We refer to this second split as intra-client split .
Existing FL literature and benchmarks typically conduct either an inter-client or intra-client trainvalidation split.
However, neither inter-client nor intra-client split alone can reveal the participation gap.
1 To the best of our knowledge, this is the ﬁrst work that conducts both splits simultaneously .
3. W HY IS THE P ARTICIPATION G AP I NTERESTING ?
Participation gap is an intrinsic property of FL due to heterogeneity.
Heterogeneity across clients is one of the most important phenomena in FL.
We identify that the participation gap is another outcome of heterogeneity in FL, in that the gap will not exist if data is homogeneous.
Formally, we can establish the following proposition.
Proposition 3.1.
If D c ≡ D for any c ∈ C and ρ c ≡ ρ , then for any participating clients ˆ C ⊂ C and w in domain, the participation gap is always zero in that F unpart ( w ) ≡ F part val ( w ) .
Proposition 3.1 holds by deﬁnition as
[Equation]
Remark.
We assume unweighted risk with ρ c ≡ ρ for ease of exposition.
Even if ρ c are different, one can still show F unpart ( w ) F part val ( w ) is always equal to a constant independent of w .
Therefore the triviality of the participation gap for homogeneous data still holds in the logarithmitic sense.
Participation gap can quantify client diversity.
The participation gap can provide insight into a federated dataset since it provides a quantiﬁable measure of client diversity / heterogeneity.
With other aspects controlled, a federated dataset with larger participation gap tends to have greater heterogeneity.
For example, using the same model and hyperparameters, we observe in Section 5 that CIFARexhibits a larger participation gap than CIFAR-10.
Unlike other indirect measures (such as the degradation of federated performance relative to centralized performance), the participation gap is intrinsic in federated datasets and more consistent with respect to training hyperparameters.
Participation gap can measure overﬁtting on the population distribution.
Just as a generalization gap that increases over time in centralized training can indicate overﬁtting on training samples, a large or increasing participation gap can indicate a training process is overﬁtting on participating clients.
We observe this effect in Figure 1 for Shakespeare and Stack Overﬂow tasks.
Thus measuring this gap can be important for researchers developing models or algorithms to reduce overﬁtting.
Participation gap can quantify model robustness to unseen clients.
From a modeler’s perspective, the participation gap quantiﬁes the loss of performance incurred by switching from seen clients to unseen clients.
The smaller the participation gap is, the more robust the model might be when deployed.
Therefore, estimating participation gap may guide modelers to design more robust models, regularizers, and training algorithms.
Participation gap can quantify the incentive for clients to participate.
From a client’s perspective, the participation gap offers a measure of the performance gain realized by switching from unparticipating (not contributing training data) to participating (contributing training data).
This is a fair comparison since both F part val and F unpart are estimated on unseen data.
When the participation gap is large ( e.g., if only few clients participate), modelers might report the participation gap as a well-justiﬁed incentive to encourage more clients to join a federated learning process.
R EFLECTIONS ON C LIENT H ETEROGENEITY AND S YNTHETIC D ATASET P ARTITIONING
Since participation gaps can quantify client dataset heterogeneity, we study how participation gaps vary for different types of federated datasets.
Many prior works (McMahan et al., 2017; Zhao et al., 2018;
Hsu et al., 2019; Reddi et al., 2021) have created synthetic federated versions of centralized datasets.
These centralized datasets do not have naturally-occurring client partitions and thus need to be synthetically partitioned into clients.
Due to the importance of heterogeneity in FL, partitioning schemes generally ensure client datasets are heterogeneous in some respect.
Previous works typically impose heterogeneity at the label level.
For example, Hsu et al.
(2019) create heterogeneous federated datasets by assigning each client a distribution over labels, where each local distribution is drawn from a Dirichlet meta-distribution.
Once conditioned on labels, the drawing process is homogeneous.
We refer to these schemes as label-based partitioning .
While label heterogeneity is generally observed in natural federated datasets, it is not the only observed form of heterogeneity.
In particular, each client in a natural federated dataset has its own separate data generating process.
For example, for Federated EMNIST (Cohen et al., 2017), different clients write characters using different handwriting.
Label-based partitioning does not account for this form of heterogeneity.
To show this, in Figure 3 we visualize the clustering of client data between natural and label-based partitioning (Hsu et al., 2019) for Federated EMNIST.
We project clients from each partitioning into a 2D space using T-SNE (Van der Maaten & Hinton, 2008) applied to the raw pixel data.
Naturally partitioned examples clearly cluster more than label-based partitioned examples, which appear to be distributed similarly to the full data distribution. [Figure]
Interestingly [Figure] differences in heterogeneity also signiﬁcantly affect generalization behavior.
In Figure [Figure] we compare the training progress of the naturally-partitioned EMNIST dataset with a label-based partitioning following the scheme by Hsu et al.
Despite showing greater label heterogeneity (Fig. 4(a)) [Figure] the label-based partitioning does not recover any signiﬁcant participation gap, in sharp contrast to the natural partitioning (Fig. 4(d)).
In Figure 5, we also see minimal participation gap in label-based partitioning for CIFAR.
This motivates a client partitioning approach that better preserves the generalization behavior of naturally-partitioned datasets
[Figure]
4. S EMANTIC C LIENT P ARTITIONING AND THE P ARTICIPATION G AP
To explore and remediate differences in client heterogeneity across natural and synthetic datasets, we propose a semantics-based framework to assign semantically similar examples to clients during federated dataset partitioning.
We instantiate this framework via an example of an image classiﬁcation task.
Our goal is to reverse-engineer the federated dataset-generating process described in Equation (1) so that each client possesses semantically similar data.
For example, for the EMNIST dataset, we expect every client (writer) to (i) write in a consistent style for each digit ( intra-client intra-label similarity ) and (ii) use a consistent writing style across all digits ( intra-client inter-label similarity ).
A simple approach might be to cluster similar examples together and sample client data from clusters.
However, if one directly clusters the entire dataset, the resulting clusters may end up largely correlated to labels.
To disentangle the effect of label heterogeneity and semantic heterogeneity, we propose the following algorithm to enforce intra-client intra-label similarity and intra-client inter-label similarity in two separate stages.
• Stage 1: For each label, we embed examples using a pretrained neural network (extracting semantic features), and ﬁt a Gaussian Mixture Model to cluster pretrained embeddings into groups.
Note that this results in multiple groups per label.
This stage enforces intra-client intra-label consistency.
• Stage 2: To package the clusters from different labels into clients, we aim to compute an optimal multi-partite matching with cost-matrix deﬁned by KL-divergence between the Gaussian clusters.
To reduce complexity, we heuristically solve the optimal multi-partite matching by progressively solving the optimal bipartite matching at each time for randomly-chosen label pairs.
This stage enforces intra-client inter-label consistency.
We relegate the detailed setup to Appendix D. Using this procedure we can generate clients which have similar example semantics.
We show in Figure 5 that this method of partitioning preserves the participation gap.
In Appendix D, we visualize several examples of our semantic partitioning on various datasets, which can serve as benchmarks for future works.
[Figure]
4. E XPLAINING DIFFERENCES BETWEEN LABEL - BASED AND SEMANTIC PARTITIONING
To explain the above behavior, we revisit our mathematical setup and the deﬁnition of the participation gap.
Recall that the participation gap is deﬁned as (we omit the weights by setting ρ c ≡ 1 for simplicity):
[Equation]
In order to express the ideas without diving into excessive details of measure theory, we assume without loss of generality that the meta-distribution P is a continuous distribution supported on C with probability density function p P ( c ) .
We also assume that for each client c ∈ C , the local distribution D c is a continuous distribution supported on Ξ with probability density function p D c ( ξ ) .
Therefore, the participation gap becomes
[Equation]
Therefore the scale of participation gap could depend (negatively) on the concentration speed from
| ˆ C | (cid:80) c
∈ ˆ C p D c ( ξ ) to (cid:82) c ∈ C
p D c ( ξ ) p P ( c ) d c as | ˆ C | → ∞ .
3 We hypothesize that for label-based partitioning, the concentration is fast because each client has a large entropy as it can cover the entire distribution of a given label.
On the other hand, for natural or semantic partitioning, the concentration is slower as the local distribution of each client has lower entropy due to the (natural or synthetic) semantic clustering.
We validate our hypothesis with an empirical estimation of local dataset entropy, shown in Figure 6.
We observe that the clients generated by label-based partitioning demonstrate much higher entropy than the natural ones.
Notably, our proposed semantic partitioning has a very similar entropy distribution across clients as the natural partitioning.
This indicates that the heterogeneity in EMNIST is mostly attributed to semantic heterogeneity. [Figure] [Table]
E XPERIMENTAL E VALUATION
We conduct experiments on four image classiﬁcation tasks:
EMNIST-10 (digits only), EMNIST62 (digits and characters) (Cohen et al., 2017; Caldas et al., 2019), CIFAR-10 and CIFAR-
(Krizhevsky et al., 2009); and two next character/word prediction tasks: Shakespeare (Caldas et al., 2019) and
StackOverﬂow (Reddi et al., 2021).
We reported F ED A VG M for image classiﬁcation tasks and F ED A DAM for text-based tasks.
The detailed setups (including model, dataset preprocessing, hyperparameter tuning) are relegated to Appendix C. We summarize our main results in Figure 1 and
Table 1.
In the following subsections, we provide more detailed ablation studies regarding the effect of various aspects of the training process on generalization performance.
5. E FFECT OF THE NUMBER OF PARTICIPATING CLIENTS
In this subsection we study the effect of the number of participating clients on the generalization performance on various tasks.
To this end, we randomly sample subsets of clients of different scales as partcipating clients, and perform federated training with the same settings otherwise.
The results are shown in Figure 7.
As the number of participating clients increases, the unparticipating accuracy monotonically improves, and the participation gap tends to decrease.
This is consistent with our theoretical understanding, as the participating clients can be interpreted as a discretization of the overall clients population distribution.
5. E FFECT OF CLIENT DIVERSITY
In this subsection, we study the effect of client diversity on generalization performance.
Recall that in the previous subsection, we vary the number of participating clients while keeping the amount of training data per client unchanged.
As a result, the total amount of training data will grow proportionally with the number of participating clients.
To disentangle the effect of diversity and the growth of training data size, in the following experiment, we instead ﬁx the total amount of the training data, while varying the concentration across clients.
The experiment is conducted on the EMNIST digits dataset.
As shown in Figure 8, the training data from [Figure] [Figure] Observe that while the total amount of training data is identical, the more diverse settings exhibit better performance in terms of both participating validation and unparticipating accuracy.
a new participating client can be more valuable than those contributed by the existing participating clients.
The intuition can also be justiﬁed by our model in that the data from a new participating client is drawn from the overall population distribution (cid:82) c ∈ C p P ( c ) p D c ( ξ )
d c , whereas the data from existing clients are drawn from the distribution aggregated by existing clients 1 | ˆ C | (cid:80)
c
∈ ˆ C p D c ( ξ ) .
This reveals the importance of client diversity in federated training.
5. O VERVIEW OF A DDITIONAL E XPERIMENTS IN A PPENDICES
We provide more detailed analysis and further experiments in Appendices B and C, including:
• Training progress of centralized optimizers on six tasks, see Appendix B.1.
• Detailed analysis of metrics distributions across clients, see Appendices B.2 and B.3.
We observe that the unparticipating clients tend to exhibit longer tails on the lower side of accuracy.
• Results on alternative hyperparameter choices, see Appendices C.1.1 and C.2.1.
• Effect of multiple local epochs per communication round, see Appendix C.1.2.
• Effect of the strength of weight decay, see Appendix C.2.2.
• Effect of model depth, see Appendix C.2.3.
C OMMUNITY S UGGESTIONS
In this work we have used the three-way-split, dataset partitioning strategies, and distributions of metrics to systematically study generalization behavior in FL.
Our results inform the following suggestions for the FL community:
• Researchers can use the three-way split to disentangle out-of-sample and participation gaps in empirical studies of FL algorithms.
• When proposing new federated algorithms, researchers might prefer using naturally-partitioned or semantically-partitioned datasets for more realistic simulations of generalization behavior.
• Distributions of metrics across clients ( e.g., percentiles, variance) may vary across groups in the three-way split (see Table 2 and Figure 10).
We suggest researchers report the distribution of metrics across clients, instead of just the average, when reporting metrics for participating and non-participating clients.
We encourage researchers to pay attention to the difference of two distributions (participating validation and unparticipating) as it may have fairness implications.
R EPRODUCIBILITY S TATEMENT
We provide complete descriptions of experimental setups, including dataset preparation and preprocessing, model conﬁgurations and hyperparameter tuning in Appendix C. Appendix D describes the detailed procedure for semantic partitioning and Appendix E presents the detailed approach for estimating the entropy that generates Figure 6.
We also attach the source code for reproducing all experiments and partitioning datasets in supplementary materials.
We are releasing an extensible code framework for measuring out-of-sample and participation gaps and distributions of metrics ( e g., percentiles) for federated algorithms across several tasks.
We include all tasks reported in this work; the framework is easily extended with additional tasks.
We also include libraries for performing label-based and semantic dataset partitioning (enabling new benchmark datasets for future works, see Appendix D).
This framework enables easy reproduction of our results and facilitates future work.
The framework is implemented using TensorFlow Federated (Ingerman & Ostrowski, 2019).
The code is released under Apache License 2.0.
We hope that the release of this code encourages researchers to take up our suggestions presented in Section
Appendices L IST OF A PPENDICES A Additional Related Work
B Additional Experimental Results
B.1 Training progress of centralized optimizers . .
B.2 Percentiles of metrics across clients . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Federated training progress at the 25 th percentile acorss clients . .
C Additional Details on Experimental Setup and Task-Speciﬁc Experiments
C.1 EMNIST Hand-written Character Recognition Task . . . . . . . . . . . . . . . . .
C.1.
Consistency across various hyperparameters choices . . . . . . . . . . . .
C.1.
Effect of multiple local epochs per communication round . . . . . . . . . .
C.2 CIFAR 10/100 Image Classiﬁcation Task . . . . . . . . . . . . . . . . . . . . . .
C.2.
Consistency across various hyperparameters choice . . . . . . . . . . . . .
C.2.
Effect of Weight Decay Strength . . . . . . . . . . . . . . . . . . . . . . .
C.2.
Effect of Model Depth . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Shakespeare Next Character Prediction Task . . . . . . . . . . . . . . . . . . . . .
C.4 Stackoverﬂow Next Word Prediction Task . . . . . . . . . . . . . . . . . . . . . .
D Details of Semanticically Partitioned Federated Dataset
D.1 Details of the Semantic Partitioning Scheme . . . . . . . . . . . . . . . . . . . . .
D.
[Table]
Visualization of Semanticically Partitioned CIFAR-100 Dataset .
D.3 Visualization of Semantically Partitioned MNIST Dataset
[Table]
A A DDITIONAL R ELATED W ORK
Recent years have observed a booming interest in various aspects of Federated Learning, including communication-efﬁcient learning (McMahan et al., 2017; Koneˇcn´y et al., 2016; Zhou & Cong, 2018;
Haddadpour et al., 2019a; Wang & Joshi, 2018; Yu & Jin, 2019; Yu et al., 2019; Basu et al., 2019;
Stich, 2019; Khaled et al., 2020; Yuan & Ma, 2020; Woodworth et al., 2020; Yuan et al., 2021; Li et al., 2021; Huang et al., 2021), model ensemble (Bistritz et al., 2020; He et al., 2020; Lin et al., 2020; Chen & Chao, 2021), integration with compression (Faghri et al., 2020; Gorbunov et al., 2020;
Sohn et al., 2020; Beznosikov et al., 2020; Horv´ath & Richt´arik, 2020; Albasyoni et al., 2020; Jiang et al., 2020; Islamov et al., 2021), systems heterogeneity (Smith et al., 2017; Diao et al., 2020), data (distributional) heterogeneity (Haddadpour et al., 2019b; Khaled et al., 2020;
Li et al., 2020d;
Koloskova et al., 2020;
Woodworth et al., 2020; Mohri et al., 2019;
Zhang et al., 2020; Li et al., 2020b; Wang et al., 2020a; Karimireddy et al., 2020; Pathak & Wainwright, 2020; Al-Shedivat et al., 2021), fairness (Wang et al., 2020b; Li et al., 2020c; Mohri et al., 2019), personalization (Smith et al., 2017; Nichol et al., 2018; Khodak et al., 2019; Balcan et al., 2019; Jiang et al., 2019; Wang et al., 2019;
Chen et al., 2019; Fallah et al., 2020; Hanzely et al., 2020; London, 2020; T. Dinh et al., 2020;
Yu et al., 2020; Hanzely & Richt´arik, 2020; Agarwal et al., 2020; Deng et al., 2020;
Hao et al., 2020;
Liang et al., 2020), and privacy (Balle et al., 2020; Chen et al., 2020; Geiping et al., 2020; London, 2020; So et al., 2020; Zhu et al., 2020; Brown et al., 2020).
These works often study generalization and convergence for newly proposed algorithms.
Huang et al.
(2021) studied the generalization of
Federated Learning in Neural-tangent kernel regime.
But, to our knowledge, there is no existing work that disentangles out-of-sample and participation gaps in federated training.
We refer readers to (Kairouz et al., 2019; Wang et al., 2021) for a more comprehensive survey on the recent progress in
Federated Learning.
B A DDITIONAL E XPERIMENTAL R ESULTS
In this section, we present several experimental results omitted from the main body due to space constraints.
Additional task-speciﬁc ablation experiments can be found in Appendix C.
B. T RAINING PROGRESS OF CENTRALIZED OPTIMIZERS
In this subsection, we repeat the experiment in Figure 1 with centralized optimizers.
The results are shown in Figure 9.
Observe that participation gap still exists with centralized optimizers.
This is because participation gap is an intrinsic outcome of the heterogeneity of federated dataset.
[Figure]
B. P ERCENTILES OF METRICS ACROSS CLIENTS
In this subsection, we report the detailed statistics of metrics across clients.
Recall that in Table 1, we aggregated the metrics across clients by weighted averaging, where the weights are determined by the number of elements contributed by each client.
In the following Table 2, we report ﬁve percentiles of metrics across clients: 95 th , 75 th , 50 th (a.k.a. median), 25 th , and 5 th .
These statistics provide a detailed characterization on the metrics distribution across clients.
[Table]
Shakespeare 95 th 68.
75 th 60.
50 th 57.
25 th 54.
5 th 38.
StackOverﬂow
th 31.
75 th 27.
50 th 25.
25 th 23.
5 th 20.
B. F EDERATED TRAINING PROGRESS AT THE 25 TH PERCENTILE ACORSS CLIENTS
To further inspect the distribution of metrics across clients, we plot the 25 th percentile of accuracy across clients versus the communication rounds (training progress).
The results are shown in Figure 10.
[Figure]
C A DDITIONAL D ETAILS ON E XPERIMENTAL S ETUP AND T ASK -S PECIFIC E XPERIMENTS
In this section we provide the details of the experimental setup, including dataset preparation/preprocessing, model choice and hyperparameter tuning.
We also include task-speciﬁc experiments with ablation study.
For every setting, unless otherwise stated, we tune the learning rate(s) to achieve the best sum of participating validation accuracy and unparticipating accuracy (so that the result will not be biased towards one of the accuracy).
C. EMNIST H AND - WRITTEN C HARACTER R ECOGNITION T ASK
Federated Dataset Description and Proprocessing.
The EMNIST dataset (Cohen et al., 2017) is a hand-written character recognition dataset derived from the NIST Special Database 19 (Grother &
Flanagan, 1995).
We used the Federated version of EMNIST (Caldas et al., 2019) dataset, which is partitioned based on the writer identiﬁcation.
We consider both the full version (62 classes) as well as the numbers-only version (10 classes).
We adopt the federated EMNIST hosted by Tensorﬂow
Federated (TFF).
In TFF, the federated EMNIST was by default split intra-cliently, namely all the clients appeared in both the “training” and “validation” dataset.
To construct a three-way split, we hold out 20% of the total clients as unparticipating clients.
Within each participating client, we keep the original training/validation split, e.g., the original training data that are assigned to participating clients will become participating training data.
We tested the performance under various number of participating clients, as shown in Figure 7.
The results reported in Table 1 are for the case withparticipating clients.
Model, Optimizer, and Hyperparameters.
We train a shallow convolutional neural network with approximately one million trainable parameters as in (Reddi et al., 2021).
For centralized training, we run 200 epochs of SGD with momentum = 0.9 with constant learning rate with batch size 50.
The (centralized) learning rate is tuned from { 10 − 2 . 5 , 10 − 2 , . . .
, 10 − 0 . 5 } .
For federated training, we run 3000 rounds of F ED A VG M (Reddi et al., 2021) with server momentum = 0.9 and constant server and client learning rates.
For each communication round, we uniformly sample 20 clients to train for 1 epoch with client batch size 20.
The client and server learning rates are both tuned from { 10 − 2 , 10 − 1 . 5 , . . .
C.1. C ONSISTENCY ACROSS VARIOUS HYPERPARAMETERS CHOICES
In Table 1, we only presented the best hyperparameter choice (learning rate combinations).
In this subsubsection, we show that the pattern of generalization gap is consistent across various hyperparameter choices.
The result is shown in Figure 11.
[Figure]
participation gap
C.1. E FFECT OF MULTIPLE LOCAL EPOCHS PER COMMUNICATION ROUND
In the main experiments we by default let each sampled client run one (1) local epoch every communication round.
In this subsubsection, we evaluate the effect of multiple local epochs on the generalization performance.
The result is shown in Figure 12.
[Figure]
C. CIFAR 10/100 I MAGE C LASSIFICATION T ASK
Federated Dataset Preprocessing.
The CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009) are datasets of natural images distributed into 10 and 100 classes respectively.
Since the dataset does not come with user assignment, we ﬁrst shufﬂe the original dataset and assign to clients by applying our proposed semantic synthesized partitioning.
The CIFAR-10 and CIFAR-100 dataset are partitioned into 300 and 100 clients, respectively.
For three-way split, we hold out 20% (60 for
CIFAR-10, 20 for CIFAR-100) clients as unparticipating clients, and leave the remaining client as participating clients.
Within each participating client, we hold out 20% of data as (participating) validation data.
Model, Optimizer, and Hyperparameters We train a ResNet-
(He et al., 2016) in which the batch normalization is replaced by group normalization (Wu & He, 2018) for improved stability in federated setting, as recommended by Hsieh et al.
For centralized training, we run 200 epochs of SGD with momentum =
with batch size 50, and decay the learning rate by 5x every 60 epochs.
The initial learning rate is tuned from { 10 − 2 . 5 , 10 − 2 , . . .
, 10 − 0 . 5 } .
For federated training, we run 2,000 rounds of F ED A VG M (Reddi et al., 2021) with server momentum = 0.9, and decay the server learning rate by 5x every 600 communication rounds.
For each communication round, we uniformly sample 10 clients (for CIFAR-100) or 30 clients (for CIFAR-10), and let each client train for 1 local epoch with batch size 20.
The client learning rate is tuned from { 10 − 2 , 10 − 1 . 5 , . . .
, 1 } ; the server learning rate is tuned from { 10 − 1 . 5 , 10 − 1 , . . .
C.2. C ONSISTENCY ACROSS VARIOUS HYPERPARAMETERS CHOICE
In the main result Table 1 we only present the best hyperparameter choice (learning rate combinations).
In this subsubsection, we show that the pattern of generalization gap is consistent across hyperparameter choice.
The result is shown in Figures 13 and 14.
C.2. E FFECT OF W EIGHT D ECAY S TRENGTH
In the main experiments we by default set the weight decay of ResNet-18 to be 10 − 4 .
In this subsubsection, we experiment various other options of weight decay from 10 − 5 to 10 −
The result is shown in Figure 15.
[Figure]
[Figure] [Figure]
CIFAR-100 training.
We federated train the
ResNet-18 networks for CIFAR-100 with various levels of weight decay ranging from 10 −to 10 − 2 .
We observe that a moderate scale of weight decay might improve the unparticipating accuracy and therefore decrease the participation gap.
However, an overlarge weight decay might hurt both participating validation and unparticipating performance.
C.2. E FFECT OF M ODEL D EPTH
In the main experiments we by default train a ResNet-18 for the CIFAR task.
In this subsubsection, we experiment a deeper model (ResNet-50) for the CIFAR-100.
The result is shown in Figure 16.
[Figure]
C. S HAKESPEARE N EXT C HARACTER P REDICTION T ASK
Federated Dataset Description and Preprocessing.
The Shakespeare dataset (Caldas et al., 2019) is a next character prediction dataset containing lines from the Complete Works of William Shakespeare where each client is a different character from one of the plays.
We adopt the federated shakespeare dataset hosted by Tensorﬂow Federated (TFF).
In TFF, the federated shakespeare dataset was by default split intra-cliently, namely all the clients appeared in both the “training” and “validation” dataset.
To construct a three-way split, we hold out 20% of the total clients as unparticipating clients, and leave the remaining (80%) clients as participating clients (which gives the result reported in
Table 1).
Within each participating client, we keep the original training/validation split, e.g., the original training data that are assigned to these participating clients will become participating training data.
We also tested the performance under other numbers of participating clients, as shown in
Figure 7.
Model, Optimizer, and Hyperparameters We train the same recurrent neural network as in (Reddi et al., 2021).
For centralized training, we run 30 epochs of Adam (with (cid:15) =
10 − 4 ) with batch size 20.
We tune the centralized learning rate from { 10 − 3 , 10 − 2 . 5 , . . .
, 10 − 1 } .
For federated training, we run 3,000 rounds of F ED A DAM (Reddi et al., 2021) with server (cid:15) =
10 − 4 .
For each communictaion round, we uniformly sample 10 clients, and let each client train for 1 local epoch with batch size 10.
Both client and server learning rates are tuned from { 10 − 2 , 10 − 1 . 5 , . . .
C. S TACKOVERFLOW N EXT W ORD P REDICTION T ASK
Federated Dataset Description and Preprocessing.
The Stack Overﬂow dataset consists of questions and answers taken from the website Stack Overﬂow.
Each client is a different user of the website.
We adopt the stackoverﬂow dataset hosted by Tensorﬂow Federated (TFF).
In TFF, the federated stackoverﬂow dataset is splitted inter-cliently , namely the training data and validation data belong to two disjoint subsets of clients.
To construct a three-way split, we will treat the original “validation” clients as unparticipating clients.
Within each participating client, we randomly hold out the max of 20% or 1000 elements as (participating) validation data, and the max of 80% orelements as (participating) training data.
Due to the abundance of stackoverﬂow data, we randomly sample a subset of clients from the original “training” clients as participating clients.
The result shown in Table 1 is for the case with 3425 participating clients.
We also tested other various levels of participating clients, shown in Figure 7.
Model, Optimizer, and Hyperparameters We train the same recurent neural network as in (Reddi et al., 2021).
For centralized training, we run 30 epochs of Adam (with (cid:15) =
10 − 4 ) with batch size 200.
We tune the centralized learning rate from { 10 − 3 , 10 − 2 . 5 , . . .
, 10 − 1 . 5 } .
For federated training, we run 6,000 rounds of F ED A DAM (Reddi et al., 2021) with server (cid:15) =
10 − 4 .
For each communictaion round, we randomly sample 100 clients, and let each client train for 1 local epoch with batch size 50.
Both client and server learning rates are tuned from { 10 − 2 , 10 − 1 . 5 , . . .
The client learning rate is tuned from { 10 − 3 , 10 − 1 . 5 , . . .
, 10 − 1 } ; the server learning rate is tuned from { 10 − 2 , 10 − 1 . 5 , . . .
D D ETAILS OF S EMANTICICALLY P ARTITIONED F EDERATED D ATASET D. D ETAILS OF THE S EMANTIC P ARTITIONING S CHEME
In this section we provide the details of the proposed algorithm to semantically partition a federated dataset for CIFAR-10 and CIFAR-100.
For clarify, we use K to denote the number of classes, and C to denote the number of clients partitioned into.
The ﬁrst stage aims to cluster each label into C clusters.
Embed the original inputs of dataset using a pretrained EfﬁcientNetB3.
This gives a embedding of dimension 1280 for each input.
Reduce the dimension of the above embeddings to 256 dimensions via PCA.3.
For each label, ﬁt the corresponding input with a Gaussian mixture model with C clusters.
This step yields C gaussian distribution for each of the K labels.
Formally, we let D kc denote the (Gaussian) distribution of the cluster c of label k .
The second stage will package the clusters from different labels across clients.
We aim to compute an optimal multi-partite matching with cost-matrix deﬁned by KL-divergence between the Gaussian clusters.
To reduce complexity, we heuristically solve the optimal multi-partite matching by progressively solving the optimal bipartite matching at each time for some randomly-chosen label pairs.
Formally, we run the following procedure
1: Initialize S unmatched ← { 1 , . . .
, K } 2: Randomly sample a label k from S unmatched , and remove k from S unmatched .
while S unmatched (cid:54) = ∅ do 4: Randomly sample a label k (cid:48) from S unmatched , and remove k (cid:48) from S unmatched .
5: Compute a cost matrix A of dimension C × C , where A ij ← D KL ( D ki ||D k (cid:48) j ) .
6: Solve and record the optimal bipartite matching with cost matrix A . 7: Set k ← k (cid:48)
D. V ISUALIZATION OF S EMANTICICALLY P ARTITIONED CIFAR-100 D ATASET
D. V ISUALIZATION OF S EMANTICALLY P ARTITIONED MNIST D ATASET
E M ETHODOLOGY FOR C OMPUTING E NTROPY
We hypothesize that a participation gap exists for naturally partitioned datasets and not for synthetically partitioned datasets because the naturally partitioned datasets inherently contain correlated inputs not drawn IID from the full data generating distribution.
Put another way, the entropy of the input data for a given label from a naturally partitioned client is lower than the entropy for that same label from a synthetically partitioned client.
To evaluate this claim, we need to (approximately) infer the data generating distribution for each client, and then measure the entropy of this distribution, deﬁned as:
[Equation]
To infer the client data generating distribution, we used deep generative models.
Because our clients possess relatively few training examples ( O (10) for a particular class), many deep generative models such as Glow (Kingma & Dhariwal, 2018) or PixelCNN (Salimans et al., 2017) will not be able to learn a reasonable density model.
We instead used a Variational Autoencoder (Kingma & Welling, 2013) to approximate the deep generative process.
This model is signiﬁcantly easier to train compared to the much larger generative models, but does not have tractable log-evidence measurement.
Instead, models are trained by minimizing the negative Evidence Lower Bound (ELBO).
We ﬁltered each client to contain data only for a single label.
Because of the sparseness of the data after ﬁltering, we found that a 2 dimensional latent space was sufﬁcient to compress our data without signiﬁcant losses.
We used a Multivariate Normal distribution for our posterior and prior, and an
Independent Bernoulli distribution for our likelihood.
The posterior was given a full covariance matrix to account for correlations in the latent variable.
All models were trained for 10 4 training steps.
In order to evaluate our models, we used a stochastic approximation to the log-evidence, given by a 1000 sample IWAE (Burda et al., 2015).
IWAE is a lower bound on the Bayesian Evidence that becomes asymptotically tight when computed with a large number of samples.
We evaluated the entropy for 100 clients from naturally partitioned, syntactically partitioned, and synthetically partitioned datasets, and computed the average across clients as our estimate for the client data entropy.
We ﬁnd that synthetic partitioning results in an average client entropy of 50 Nats, while
Natural partitioning results in clients with only 40 Nats of entropy.
Syntactic partitioning falls in between these two, having 45 Nats of entropy.