Motley : Benchmarking Heterogeneity and Personalization in Federated Learning
Abstract
Personalized federated learning considers learning models unique to each client in a heterogeneous network.
The resulting client-speciﬁc models have been purported to improve metrics such as accuracy, fairness, and robustness in federated networks.
However, despite a plethora of work in this area, it remains unclear: (1) which personalization techniques are most effective in various settings, and (2) how important personalization truly is for realistic federated applications.
To better answer these questions, we propose Motley , a benchmark for personalized federated learning.
Motley consists of a suite of cross-device and cross-silo federated datasets from varied problem domains, as well as thorough evaluation metrics for better understanding the possible impacts of personalization.
We establish baselines on the benchmark by comparing a number of representative personalized federated learning methods.
These initial results highlight strengths and weaknesses of existing approaches, and raise several open questions for the community.
Motley aims to provide a reproducible means with which to advance developments in personalized and heterogeneity-aware federated learning, as well as the related areas of transfer learning, meta-learning, and multi-task learning.
Code for the benchmark is open-source and available at: https://github.com/google-research/ federated/tree/master/personalization_benchmark .
Introduction
Federated learning (FL) aims to share knowledge across disparate data silos in a privacy-preserving manner [36, 46, 57].
Relative to standard (e g., data center) distributed learning, a deﬁning trait of federated learning is the presence of heterogeneity 3 , as each of the clients (e g., a sensor, mobile phone, or entire organization) may generate data according to a distinct distribution.
In response to this key difference, a signiﬁcant amount of work has been devoted to understanding and addressing heterogeneity in federated learning.
For example, heterogeneity has been the focus of the analysis and development of numerous federated optimization methods [e.g., 1, 13, 29, 37, 47, 50, 72, 73, 74], and the impact of heterogeneity on FL has been explored more generally in connection with issues of fairness, robustness, and privacy in federated networks [e.g., 48, 49, 58, 79].
The presumed presence of heterogeneity has also directly driven a large body of work in personalized federated learning .
In personalized FL, the goal is to learn or adapt models to more closely reﬂect the distinct data distribution of each federated client.
For example, personalized FL approaches may consider ﬁne-tuning a global or meta-trained model to adapt to local client data [16, 19, 35, 38, 67, 82], learning a clustering structure amongst the clients [20, 59, 65], or using multi-task learning techniques to model possibly more complex relationships [17, 21, 22, 49, 68].
Despite a signiﬁcant amount of work in personalized FL, there remains a lack of consensus about which methods perform best in various federated settings, let alone the degree to which personalization itself is really necessary in practice.
For example, many works target a speciﬁc form of personalization (e.g., clustering) in their methodology, and then artiﬁcally create or modify data to match this assumption (e.g., manually clustering the data) before testing the efﬁcacy of their approach [74].
While this is a reasonable sanity check, it fails to validate how impactful such approaches are for real-world FL applications.
In particular, a major issue is that while ‘heterogeneity’ is believed to be a natural occurrence in federated networks, there is a lack of benchmarks that reﬂect real applications of FL, making it hard to understand the prevalence and magnitude of heterogeneity in practice.
To address these concerns, we propose Motley , a comprehensive benchmark for personalized and heterogeneity-aware FL.
Motley is designed with a focus on ease-of-use and reproducibility 1 .
The benchmark includes a suite of four cross-device and three cross-silo datasets, as well as baseline personalization methods evaluated on them.
The datasets themselves are drawn from real-world applications mirroring federated settings in an effort to better reﬂect naturally occurring forms of heterogeneity.
In developing baselines for the benchmark, we pay careful attention to the evaluation of personalization and heterogeneity—making concrete suggestions for future work in evaluating the impact of heterogeneity/personalization.
Finally, although we focus speciﬁcally on the application of
FL, this benchmark is also a useful tool for the areas of transfer learning, meta learning, and multi-task learning more generally, as techniques from these areas are commonly used for personalized FL.
Cross-device vs. Cross-silo.
Based on the size and characteristics of the network, there are two common FL settings: cross-device FL and cross-silo FL , and two types of algorithms: stateful and stateless [74].
We provide a brief description below (see Table 1), and defer readers to [36, 74] for more detailed discussion.
Cross-device applications typically involve learning across a large number (e.g., hundreds of millions) of mobile or IoT devices.
Given the network scale coupled with the unreliability of such devices, it is common for devices to only participate once (if at all) in training.
This characteristic motivates the development of methods for cross-device FL which are stateless 2 , in that model or variable state is not maintained on each client from one round to another, and there exists no unique identiﬁer for each client.
In contrast, cross-silo FL applications often consider learning across a handful of organizations such as hospitals or schools, where clients are almost always available at each training round.
These properties allow the silos to be more easily accessed and identiﬁed, permitting the use of possibly more complex stateful approaches.
[Table]
(Appendix A.2)
On the scale of 0.1%-1% (see
Table 3 for the exact value)
[Table]
round)
Train/valid/test split (Section 2.2)
Split clients (Fig. 1)
[Table]
(Fig. 1)
Stateful or stateless algorithm
Stateless 2 .
Both work.
Heterogeneity and Personalization.
As pointed out previously, a deﬁning trait of FL is that each client may generate according to a unique distribution.
To better account for this heterogeneity 3 , it is therefore common to consider techniques that learn personalized, client-speciﬁc models.
Existing personalized FL approaches can be categorized in three different ways: (1) stateful vs stateless (see Table 1 and the discussion above); (2) model-agnostic vs model-speciﬁc (whether it targets a speciﬁc model and requires domain-speciﬁc information [e.g., 34, 51, 67, 84]); (3) in terms of methodologies, e.g., meta-learning [2, 13, 19, 35, 38], clustering [20, 53], local memory [54], and multi-task learning (MTL)
[e.g., 4, 17, 21, 22, 23, 49, 52, 68].
We defer readers to Appendix D and the recent surveys [70], [74, §7.5] for more detailed discussion on existing literature.
In benchmarking
Motley , we consider ﬁve model-agnostic personalized FL methods (four stateless and one stateful) from the major methodology categories (meta-learning, clustering, local memory, and MTL). [Table]
Existing FL benchmarks.
Concurrent works benchmarking personalized FL [15, 55] do not take into account differences between cross-device and cross-silo federated settings (Table 1) [Table] which we ﬁnd to have a signiﬁcant impact on the baseline methods and resulting conclusions.
Beyond these works, other prior FL benchmarks (see, e.g, [8, 9, 10, 12, 26, 27, 31, 42, 44]) do not consider personalization baselines and focus instead on the standard FedAvg [57] algorithm and variations [62].
Motley : A Benchmark for Personalized Federated Learning
Table 2 gives an overview of the three components of Motley (methods, datasets, baselines).
We now brieﬂy describe the methods and datasets.
See Appendix A for more detailed description of Motley .
2.1 Personalization Methods
Motley includes ﬁve simple and model-agnostic algorithms for learning personalized models: (1)
Local training , refers to every client training a local model using their own data, without collaborating with others.
(2) FedAvg+Fine-tuning [35], in which we ﬁrst train a global model via FedAvg [62] 4 , and then, each client ﬁne-tunes the global model on their local data to get the personalized model.
This simple method has a natural connection to meta-learning [35].
(3) HypCluster [53] (also
IFCA [20]), which jointly clusters clients and learns a model for each cluster.
(4) FedAvg+kNNPer [54], a very recent personalization algorithm proposed to interpolate/ensemble the output of two models: a globally-trained FedAvg model and a local kNN model.
(5) Multi-Task Learning (MTL), a class of methods used to deliver personalized models for a set of tasks by learning the task relations (either explicitly or implicitly).
The ﬁrst four algorithms are stateless 2 , and hence, are appropriate in the cross-device FL setting (see Table 1 and [36, 74] for discussions on stateful vs stateless).
[Figure]
2.2 Datasets and Pre-processing
Table 2 lists the datasets carefully chosen to reﬂect real-world FL applications, including a health data 5 and a school data 6 in the cross-silo setting (see Appendix E for detailed descriptions).
All datasets have natural per-user partitions and distinct local statistics (Figure 1).
Motley provides data pre-processsing pipelines for all data 1 , and includes a critical and often overlooked distinction between pre-processing cross-device vs. cross-silo data.
As shown in Figure 1, for cross-device data, we ﬁrst randomly split the clients into three disjoint sets: train, validation (for hyperparameter tuning), and test (for ﬁnal evaluation).
This split reﬂects practical cross-device FL settings: given the population scale (e.g., millions of mobile devices [24]), devices participating in inference may never join training.
Then we split each validation and test client’s local examples into two equal-sized sets: a personalization set (for learning a personalized model) and an evaluation set (for evaluating the personalized model).
For cross-silo data, because the total number of silos is small (e.g., tens of hospitals), and the same silos usually participate in both training and inference, we split each silo’s local examples into three sets: a train, validation, and test set (Figure 1).
Cross-Device Experiments
The complete results on the cross-device datasets are in Appendix B (in particular, Table 4 has a detailed summary of the results).
Next we brieﬂy summarize the results and our ﬁndings.
As we focus on personalization, it is crucial to see how the per-client accuracy changes.
As shown in Figure 2, ﬁne-tuning improves the average per-client accuracy on EMNIST, StackOverﬂow, and Landmarks.
Besides accuracy, we also report a fairness metric 7 in Table 4, and found that personalization algrorithms usually improve fairness. [Figure] n
[Figure]
3.1 FedAvg + Fine-tuning (FT)
[Figure]
Observation 1: Fine-tuning can hurt clients.
As shown in Figure 3(b), some clients drop accuracy after ﬁne-tuning (i
[Figure]
e., fall below the red dashed line at zero).
There are two reasons: 1) The per-client accuracy metric is noisy due to small local data; 2)
The ﬁne-tuning hyperparameters are tune globally but different clients prefer different hyperparameters.
See Appendix B.1 for the full explanation.
Observation 2: Hyperparameter tuning can be difﬁcult.
First, different metrics may favor different hyperparameters, as shown in Figure 3(c).
Second, since the per-client accuracies are noisy, we need to compare the statistical signiﬁcance of two results.
Third, extra hyperparameters add another layer of complexity.
For example, it may be helpful to tune FedAvg together with ﬁne-tuning [35]. [Figure] [Figure]
Observation 3: Sensitivity to distribution shift.
Fine-tuning is able to quickly adapt a model to a client’s local data, but is also prone to overﬁtting (a.k.a.
catastrophic forgetting [56]).
To illustrate this, we evaluate each test client’s personalized model over the in-distribution (ID) test set (i.e., the client’s local evaluation set 8 ) and an out-of-distribution (OOD) test set.
This OOD test set is formed by randomly sampling local examples from the test clients.
This result is shown in Figure 4, which indicates an extra maintenance cost of personalized algorithms, i.e., decide when and how to continuously personalize the model when clients have new data.
Remark: Acceptance criteria and robust personalization.
Our experiments assume that each client always uses the ﬁne-tuned model for inference.
In practice, one can design appropriate criteria so that a client only accepts the ﬁne-tuned model under certain conditions [66].
Mitigating catastrophic forgetting [56] during model adaptation is an active research area, e.g., recent work [3, 33, 41, 77, 78], which is worth exploring in the personalized FL setting.
Observation 4: Performance drops when clients have smaller personalization set 8 .
As shown in Figure 6(c), where we reduce the number of examples in each test client’s persoanlization set, the performance of ﬁne-tuning and kNN-Per drops quickly.
By contrast, HypCluster is quite robust to this change, which makes sense because HypCluster only needs the personalization set for model evaluation and choosing the best model, while ﬁne-tuning and kNN-Per needs to train a new model.
3.2 HypCluster / IFCA
As shown in Figure 2, HypCluster performs worse than other two personalization algorithms.
Observation 1: “Mode collapse” can hurt training.
“Mode collapse” happens when all clients will always choose the same cluster (Figure 5(a)).
One way to mitigate this issue is to warm start
HypCluster with models trained by FedAvg.
However, mode collapse can still happen even with warm start.
How to effectively train HypCluster without mode collapse remains an open problem.
Observation 2: High communication cost per round.
A limitation of HypCluster is that in each round, the server needs to broadcast k models (where k is the number of learned clusters) to the clients, and hence, incurs k times the communication cost of FedAvg 9 .
This is shown in Figure 5(b-c).
Observation 3: Difﬁculty in interpreting the learned clusters.
While it is generally hard to interpret the learned clusters on an arbitrary dataset, we do know that TedMulti-EnEs has two natural clusters of clients, i.e., English and Spanish users.
Figure 5(d) shows that HypCluster indeed uncovers the two underlying clusters on TedMulti-EnEs. [Figure]
Observation 4: Sensitivity to distribution shift.
We follow the same ID and OOD evaluation procedure in Section 3.1.
As shown in Figure 4, HypCluster gives a worse ID-OOD tradeoff (i e., achieves a lower OOD accuracy at the same ID accuracy) compared to the other algorithms.
3.3 FedAvg + kNN-Per
Observation 1: Clients may be hurt.
This is illustrated in Figure 6(a-b).
Section 3.1 gives two reasons why clients may hurt after ﬁne-tuning, which also apply to kNN-Per.
Observation 2: Difﬁculty in hyperparameter tuning.
The interpolation coefﬁcient is a hyperparameter of kNN-Per.
It is a scalar in (0, 1), where 0 means that no personalization and 1 means that the local prediction is formally entirely based on nearest neighbors.
Figure 6(a-b) shows that different metrics may prefer different hyperparameter values.
Section 3.1 points out two other reasons why hyperparameter tuning is difﬁcult for ﬁne-tuning.
Both of them apply to kNN-Per as well.
Observation 3: Sensitivity to distribution shift.
We follow the same ID and OOD evaluation procedure in Section 3.1.
As shown in Figure 4, kNN-Per gives a similar or better ID-OOD tradeoff (i.e., achieves a higher OOD accuracy at the same ID accuracy) compared to the other algorithms.
Observation 4: Performance drops when clients have smaller personalization set 8 .
This has been discussed previously in Section 3.1.
Cross-Silo Experiments
We now consider baselines for the cross-silo personalization methods and datasets in Motley .
The complete results are in Appendix C (see Table 5 for a detailed summary of the results).
[Figure]
Results.
Most of the practical concerns of FedAvg+Fine-tuning and HypCluster discussed in Section 3 are still applicable here, e.g.,
HypCluster is still difﬁcult to train due to the mode collapse issue (although it may happen less frequently when training simple linear models on Vehicle and School).
In addition to the shared practical concerns, we observe four key trends speciﬁc to the cross-silo setting.
Observation 1: Effectiveness of local training.
Local training may be a strong baseline in cross-silo settings.
If we consider client(silo)- level joint differential privacy [34, 45], it has an extra beneﬁt of no privacy cost.
Observation 2: Importance of personalization.
The four personalization algorithms all achieve better mean accuracy (or MSE) and fairness 7 than that of vanilla FedAvg.
Observation 3: Performance of MTL.
We see that (stateful) MTL methods could yield competitive performance with ﬁne-tuning on the three datasets, with potentially less hyperparameter tuning.
Observation 4: Effectiveness of client-speciﬁc hyperparameter tuning.
For FedAvg+Fine-tuning, we compared tuning the ﬁne-tuning hyperparameters (i.e., ﬁne-tuning learning rate and the number of ﬁne-tuning epochs) in a per-client (every client chooses their own hyperparameter) vs global (all clients share the same hyperparameter) manner.
Using client-speciﬁc ﬁne-tuning hyperparameters can be better than tuning globally when each client’s local data is large (see Figure 7).
Discussion and Open Directions
In this work we present Motley , a large-scale benchmark for personalized FL covering both crossdevice and cross-silo settings.
Motley provides a reproducible, end-to-end experimental pipeline including data preprocessing, algorithms, evaluation metrics, and tuned hyperparameters.
Beyond these baselines, our experiments provide several new insights about personalized FL (see the detailed summary in Appendix H). These insights suggest several directions of future work, such as:
• The notion of the "best" method (or "best" hyperparameter of the same method, e
g., Figure 3) can change depending on the evaluation metric or setting (Table 4 and 5). A critical direction is thus to develop systematic evaluation schemes for personalized FL (i.e., mean accuracy alone is not enough). • Existing literature often overlook or obfuscate the practical complexities of deploying personalized FL algorithms in real-world settings (e.g, Section 3.1 discusses the hyperparameter tuning difﬁculties due to local data scarcity and heterogeneity in cross-device FL). Designing new practical personalized FL methods that take these considerations into account is an important direction to democratize
FL. • Developing techniques to train and interpret clustering methods such as HypCluster without the "mode collapse" issue (see Figure 5 and Section 3.2) is a necessary step to make these approaches more effective in practice.
• Tradeoffs exists between adapting a client’s personalized model to the current local distribution and generalizing to future distributions (Figure 4), which is worth exploring in greater detail (see the "Remark: Acceptance criteria and robust personalization" in Section 3.1).
• Given the observed beneﬁts of per-client hyperparameter tuning in
cross-silo FL
(Figure 7), it may be beneﬁcial to develop similar, scalable approaches for hyperparameter tuning in cross-device FL.
Finally, we note that the area of benchmarking itself can be improved in future iterations.
For example, we hope that Motley can inspire benchmarking of additional evaluation metrics such as privacy, other notions of fairness 7 , and robustness, and additional datasets and applications.
Acknowledgements
We are grateful to Zachary Garrett, Jakub Kone ˇ cn ` y, H. Brendan McMahan, Sewoong Oh, Daniel
Ramage, Keith Rush and Ananda Theertha Suresh for helpful discussions and comments.
A Motley : A Benchmark for Personalized Federated Learning
Table 2 gives an overview of the three components of Motley (methods, datasets, baselines).
We describe the included methods and datasets in this section.
In Sections B and C, we provide baseline results and discuss the practical concerns of performing personalization in real-world FL applications.
A.1 Personalization Methods
Motley includes ﬁve simple and model-agnostic algorithms for learning personalized models.
The ﬁrst four algorithms are stateless , and hence, are appropriate in the cross-device FL setting (see
Section 1 and [36, 74] on why stateless algorithms are typically necessary for cross-device FL).
• Local training refers to every client training a local model using their own data, without collaborating with others.
While it is not an FL algorithm, we include it in Motley as it provides a good baseline for FL algorithms, and it may be competitive when clients have sufﬁcient local data [79]. • FedAvg+Fine-tuning 35] is a simple method for stateless, model-agnostic personalized FL, operating as follows:
First train a global model via FedAvg [62] 4 ; then, each client ﬁne-tunes the global model on their local data and uses the ﬁne-tuned model for inference.
It has a natural connection to meta-learning [35, 38] 10 , and has been reported to work well in real-world on-device applications [66, 75].
In Motley , we explore two variations: ﬁne-tuning the entire model or ﬁne-tuning only the last layer (the latter can be viewed as a form of hard parameter sharing MTL). • HypCluster [53] (also IFCA [20]) is a stateless method that jointly clusters clients and learns a model for each cluster.
Both HypCluster and IFCA work as follows: in each training round, the server sends all models (one per cluster) to the participating clients; each client receives the models and picks the cluster associated to the model with the lowest loss on its local data.
It then computes an update for the selected model and sends the update and cluster identity to the server.
The server aggregates the model updates for each cluster in the same way as in FedAvg [62] 4 .
In Motley , we explore two initialization strategies : random and warm start with FedAvg 4 .
• FedAvg+kNN-Per [54] is a very recent personalization algorithm.
The idea is to interpolate/ensemble the output of two models: a globally-trained FedAvg model and a local kNN model.
For each client’s local example, a representation vector is obtained from the FedAvg model (e.g., the input to the last softmax layer, or the states of an LSTM model).
The Euclidean distance between the representations are used to learn a kNN model.
• Multi-Task Learning (MTL) is a class of methods used to deliver personalized models for a set of tasks by learning the task relations (either explicitly or implicitly).
Each ‘task’ corresponds to a client in the FL setting.
MTL approaches usually require the clients to be stateful [e.g., 21, 68], and hence, are more appropriate for cross-silo settings (see Section 1 on stateless vs stateful algorithms).
Many existing personalized FL methods can be viewed as a form of MTL [e g., 21, 22, 49, 68, 69].
In Motley , we consider two MTL algorithms: Mocha [ the ﬁrst work that proposes to personalize federated models
in convex settings
, and
2) Ditto a recent work that regularizes personalized models towards optimal global models for both convex and non-convex problems
A.2 Datasets and Pre-processing
Table 2 lists the datasets carefully chosen to reﬂect real-world FL applications (see Appendix E for detailed descriptions).
All datasets have natural per-user partitions and distinct local statistics (Fig. 1).
Motley provides data pre-processsing pipelines for all data 1 , and includes a critical and often overlooked distinction between pre-processing cross-device vs. cross-silo data (described below).
Pre-processing cross-device datasets.
Two steps are necessary to evaluate personalization algorithms in the cross-device setting.
First, we randomly split the clients into three disjoint sets: train, validation (for hyperparameter tuning), and test (for ﬁnal evaluation).
This split reﬂects practical cross-device FL settings: given the population scale (e.g., millions of mobile devices [24]), devices participating in training are usually different from those in inference.
Second, we split each validation and test client’s local examples into two equal-sized sets: a personalization set and an evaluation set.
See Section B for how they are used in each algorithm.
Speciﬁcally, the personalization set is used to train a local model, ﬁne-tune a FedAvg-trained model, or select the best HypCluster model.
On
StackOverﬂow, we sort the local examples by time before splitting 11 .
We perform random split on other datasets because they do not save time information 12 .
Pre-processing cross-silo datasets.
Unlike cross-device FL, in the cross-silo FL setting the total number of silos is small (e.g., tens of hospitals), and the same silos usually participate in both training and inference.
To evaluate this setting, we split each silo’s local examples into three sets: a train, validation, and test set.
See Section C for details on how they are used in training and evaluation.
Clients sampling.
Besides the distinction in the pre-processing steps, another distinction is the clients sampling rate during training.
As mentioned in Section 1, the clients sampling rate is usually very low [36] in the cross-device setting, e.g., the total population is hundreds of millions devices but only a few thousands participate at every training round.
Table 3 lists the number of clients sampled per round used in our cross-device experiments.
In the cross-silo experiments, we assume that all clients are always available at each training round.
[Table]
B Cross-Device Experiments
We now provide cross-device personalization baselines (Table 4) by evaluating the stateless methods discussed in Section A on the cross-device datasets included in Motley , and discuss our ﬁndings.
Training and evaluation process.
As discussed in Section A.2, each cross-device dataset has train, validation, and test clients.
We use the train clients to train a single model via FedAvg or a model per cluster via HypCluster, then evaluate the trained model(s) on the validation clients.
For FedAvg + Fine-tuning, this evaluation is done on each client by ﬁrst ﬁne-tuning the FedAvg-trained model on the local personalization set, then evaluating the ﬁne-tuned model on the local evaluation set.
For HypCluster, this evaluation is performed on each client by using the local personalization set to ﬁnd the model with the lowest loss, and then evaluating the selected model on the evaluation set.
For FedAvg+kNN-Per, each client ﬁrst uses the FedAvg-trained model to extract the example representations on the local personalization set, and use them to get a kNN model, and after that, evaluates the personalized model (interpolated between the kNN model and FedAvg-trained model) on the evaluation set.
We use the validation metrics (i.e., the evaluation metrics over the validation clients) to select the best hyperparameters (see Appendix F) and report the test metrics (i.e., evaluation metrics on the test clients) in Table 4.
We discuss additional details and results for each stateless FL method below.
B.1 FedAvg + Fine-tuning (FT)
As we focus on personalization, it is crucial to see how the per-client accuracy changes with and without personalization.
This is shown by the "Per-client acc before/after FT" metrics in Table 4, where we list the mean ± standard deviation of the test clients’ per-client accuracy before and after ﬁne-tuning.
The mean and standard deviation metrics here are averaged over 5 runs 13 .
As
[Table]
564 ± .
0.163 ± .
Practical concerns : hard to train due to mode collapse; high communication cost; hard to interpret the clusters; sensitive to distribution shift (see Section B.2)
FedAvg + kNN-Per
Per-client acc 0.876 ± .06 0.275 ± .
0.735 ± .
0.162 ± .
% clients "hurt" 19.8% 23.6% 5.6% 34.4%
Practical concerns : difﬁcult to tune hyperparameters; may hurt clients; sensitive to distribution shift; performance drops with fewer examples (see Section B.3) shown in Table 4 and Figure 2, ﬁne-tuning improves the average per-client accuracy on EMNIST,
StackOverﬂow, and Landmarks, and reduces the standard deviation across the clients’ local model accuracies on EMNIST and Landmarks (i.e., improves fairness 7 ).
We explored ﬁne-tuning the entire model or only last layer and found that ﬁne-tuning all layers perform better, as shown by the metric "FT all layers vs last layer" in Table 4.
[Figure]
Observation 1: Fine-tuning can hurt clients.
The metric "% clients hurt after FT" in Table 4 is the percentage of test clients whose local accuracy drops after ﬁne-tuning.
This phenomenon can be better visualized in Figure 8: while Figure 8(a) shows that ﬁne-tuning increases the average per-client accuracy on the StackOverﬂow dataset, the scatter plot (each dot is a client) of the per-client accuracy deltas (i.e., accuracy after ﬁne-tuning – before ﬁne-tuning) in Figure 8(b) shows that some clients drop accuracy after ﬁne-tuning (i.e., fall below the red dashed line at zero).
We identify two reasons: a. The per-client accuracy is noisy for clients with small local datasets.
This effect occurs in two distinct ways: 1) Clients use a small number of examples to ﬁne-tune the model 14 .
In Figure 8(c), we show the same scatter plot (each dot is a client) as in Figure 8(b) with the x-axis being the number of examples used to ﬁne-tune the model.
The clients hurt by ﬁne-tuning tend to have a small local personalization set 8 . 2)
Clients use a small number of examples to evaluate the ﬁne-tuned model.
In Figure 8(d), we ﬁne-tune the same model on the same set of clients.
The only difference is the size of the evaluation set 8 .
The evaluation metrics are noisier (i.e., we see a larger range in Figure 8(d)) for smaller evaluation sets.
b. Heterogeneity among clients and the fact that the ﬁne-tuning hyperparameters are chosen globally.
Since each client’s local dataset is typically small, instead of choosing the ﬁne-tuning hyperparameters (i.e., the ﬁne-tuning learning rate, the number of ﬁne-tuning epochs, and which layer to ﬁne-tune) in a per-client manner 15 , we tune them globally and apply the same hyperparameters to all test clients.
The global ﬁne-tuning hyperparameters work for most clients, but can adversely affect others.
This also illustrates why hyperparameter tuning is difﬁcult (see below).
Observation 2: Hyperparameter tuning can be difﬁcult.
At least three hyperparameters are speciﬁc to ﬁne-tuning: which layers to ﬁne-tune, the ﬁne-tuning learning rate 16 , and the number of epochs to ﬁne-tune the model.
As pointed out in Observation 1b), due the small local dataset on each client, instead of choosing the best ﬁne-tuning hyperparameters in a per-client manner, we typically tune them globally and apply the same hyperparameters to all the test clients.
Choosing a good set of ﬁne-tuning hyperparameters can be particularly difﬁcult due of the following reasons
: a. Different metrics may favor different hyperparameters 17 .
For example, if we look at Figure 9(a), we may want to choose 0.1 as the ﬁne-tuning learning rate because it gives higher average perclient accuracy; however, if we look at Figure 9(b), we may want to choose 0.25 because fewer clients drop their accuracies after ﬁne-tuning (i
e., per-client acc delta < 0).
In Figure 9(c), we plot the two metrics across different ﬁne-tuning learning rates, and it is clear
that different metrics may prefer different learning rates.
b.
Since the per-client accuracies are noisy, we need to be careful when determining whether the difference between two sets of hyperparameters is statistically signiﬁcant.
c. Extra hyperparameters add another layer of complexity.
For example, it may be helpful to tune FedAvg together with ﬁne-tuning so that the model produced by FedAvg is a good initial model for ﬁne-tuning [35].
Existing FL hyperparameter optimization methods (e
g., [39]) often overlook the practical issues (e g., clients may be hurt by ﬁne-tuning, inconsistent and noisy metrics) mentioned above.
Another example of an extra hyperparameter is that one may want to learn a thresholdsuch that only clients whose local dataset size is larger than this threshold can perform ﬁne-tuning (see also the remark on "acceptance criteria and robust personalization" after Observation
Observation 3: Sensitivity to distribution shift.
Fine-tuning is able to quickly adapt a model to a client’s local data, and hence, can achieve good test performance as shown in Table 2, especially on the two image datasets EMNIST and Landmarks.
However, a potential drawback of this quick adaption is that the ﬁne-tuned model may perform poorly if a client’s future data distribution differs from the existing data (a.k.a.
catastrophic forgetting [56]).
To illustrate this, we evaluate each test client’s personalized model over the in-distribution (ID) test set (i.e., the client’s local evaluation set 8 ) and an out-of-distribution (OOD) test set.
This OOD test set is formed by randomly sampling local examples from the test clients.
The averaged ID and OOD test accuracies are shown in Figure 4.
We plot the results for different ﬁne-tuning epochs.
Larger ﬁne-tuning epoch gives higher ID accuracy but lower OOD accuracy.
This trade-off indicates an extra maintenance cost of personalized algorithms, i.e., decide when and how to continuously personalize the model when clients have new data.
Remark: Acceptance criteria and robust personalization.
Our experiments assume that each client always performs ﬁne-tuning and uses the ﬁne-tuned model for inference.
In practice, one can design appropriate criteria so that a client only performs ﬁne-tuning and accepts the ﬁne-tuned model under certain conditions.
For example, a client can perform ﬁne-tuning only when its local dataset
[Figure]
size is large enough as mentioned previously 14 ; a client can only accepts the ﬁne-tuned model when its evaluation metric is better than that of the baseline model.
See Section 2.2 of [66] for an example of acceptance criteria developed for a real-world on-device personalization application.
Designing a good set of acceptance criteria can mitigate the negative effects due to Observation 1 and 3 above, and potentially prevent clients from accepting ﬁne-tuned models that do not generalize well.
Mitigating catastrophic forgetting [56] during model adaptation is an active research area, e.g., some recent work [3, 33, 41, 77, 78], which would be worth exploring in the setting of personalized federated learning.
Observation 4: Performance drops when clients have smaller personalization set 8 .
This is already indicated previously in Figure 8(c), which shows that clients with smaller personalization set (i.e., fewer examples used to ﬁne-tuned the model) tend to be hurt by ﬁne-tuning.
In Figure 6(c), we reduce the number of examples in each test client’s persoanlization set to 50% and 25% of its original size, and report the accuracy averaged over the test clients.
We see that the performance of ﬁne-tuning and kNN-Per drops quickly when we decrease the number of local examples to personalize the model.
On the other hand, HypCluster is quite robust to this change, which makes sense because HypCluster only needs the personalization set for model evaluation and use the evaluation metrics to choose the best model, while ﬁne-tuning and kNN-Per needs the local examples to train a new model.
B.2 HypCluster / IFCA
We now discuss results of HypCluster/IFCA algorithm.
As shown in Table 2, the average per-client accuracy of HypCluster is slightly better than that of FedAvg 18 , but is much worse than that of
FedAvg+Fine-tuning on the ﬁrst three datasets.
We tuned 19 the number of clusters between {2,3,4}; however, due to the mode collapse issue (described below), we found that it difﬁcult to learn k models with k > 2 .
A natural baseline to compare with HypCluster is to run FedAvg k times, and ensemblethe k models.
We also include this baseline in Table 2, and discuss it in Observation 3 below.
Observation 1: “Mode collapse” can hurt training.
As shown in Figure 5(a), we monitor the percentage of clients that choose the ﬁrst cluster when running HypCluster with random initialization on EMNIST.
After a few rounds, all clients will always choose the same cluster.
The same thing happens to all the other datasets.
One way to mitigate this “mode collapse” issue is to warm start
HypCluster with models trained by FedAvg.
In our experiments, we train a few rounds of FedAvg for k times, and use the k models to warm start HypCluster (see Appendix F for the hyperparameters.).
Figure 5(a) shows that warm start can help mitigate the mode collapse issue on EMNIST.
However, mode collapse can still happen even when warm start is used, especially for k > 2 .
How to effectively train HypCluster without mode collapse in real-world cross-device settings remains an open problem
Observation 2: High communication cost per round.
A limitation of HypCluster is that in each round, the server needs to broadcast k models (where k is the number of learned clusters) to the clients, and hence, incurs k times the communication cost of FedAvg 21 .
In Figure 5(b) and (c), we plot the average per-client accuracies with respect to the total number of parameters communicated (including those used for training the warm start models) for EMNIST and TedMulti.
While TedMulti eventually achieves a slightly better average accuracy than FedAvg (Table 2), it is arguably worse than
FedAvg from a accuracy-communication ratio perspective.
While one can reduce communication costs via weight-sharing [20], we found that this increased the likelihood of mode collapse 22 .
In future work it would be interesting to explore connections between mode collapse and weight sharing, and consider how to best train HypCluster in constrained networks.
Observation 3: Difﬁculty in interpreting the learned clusters.
As discussed in the beginning of
Section B.2, a natural baseline to compare with HypCluster is to run FedAvg k times, and ensemblethe k models.
Speciﬁcally, we train HypCluster with k clusters for y rounds, and compare it with training FedAvg k times and each for y rounds, so both have the similar communication cost.
As shown in Table 2, the average per-client accuracy of ensembling k FedAvg models is similar to that of HypCluster for 3/4 datasets, so a natural question is: does HypCluster actually learn a meaningful cluster structure of the underlying data?
While it is generally hard to answer this question on an arbitrary dataset, we do know that TedMulti-EnEs has two natural clusters of clients, i.e., English and
Spanish users.
Figure 5(d) lists the number of English and Spanish clients (total 117 test clients) in the two learned clusters.
HypCluster indeed uncovers the two underlying clusters on TedMulti-EnEs.
Observation 4: Sensitive to distribution shift.
We follow the same ID and OOD evaluation procedure in Section B.1.
As shown in Figure 4, HypCluster seems to give a worse ID-OOD tradeoff (i.e., achieves a lower OOD accuracy at the same ID accuracy) compared to the other personalization algorithms.
B.3 FedAvg + kNN-Per
We now discuss the results of kNN-Per, which are shown in Table 4 as well as in Figure 2.
Observation 1: Clients may be hurt.
In Table 4, besides the per-client accuracy, we also list the percentage of clients "hurt" after personalization (i.e., test clients whose accuracy on the local evaluation set 8 after kNN-Per is lower than that of FedAvg).
As shown in Table 4, compared to ﬁne-tuning, kNN-Per has a lower "per-client acc" but a higher "% clients hurt" for 3/4 datasets.
The two reasons why clients may hurt after ﬁne-tuning given in Section B.1 also applies to kNN-Per: 1)
The per-client accuracy metric is noisy due to local data scarsity; 2)
The hyperparameter is chosen globally but each client has heterogeneous local distribution.
Similar to ﬁne-tuning, so far we assume all test clients will perform kNN-Per.
Designing a good set of conditions as to when a client should perform kNN-Per may help reduce the fraction of client hurt (see "Remark: Acceptance criteria and robust personalization" in Section B.1).
Observation 2: Difﬁculty in hyperparameter tuning.
Following [54], we set k (the number of nearest neighbors) to 10 for all kNN-Per experiments ([54] shows that kNN-Per is robust to the choice of k ).
We tune the interpolation coefﬁcient globally, i.e., all clients use the same coefﬁcient 23 .
The interpolation coefﬁcient is a scalar in (0, 1) (see Eq.(7) of [54]), where 0 means that no personalization and 1 means that the local prediction is formally entirely based on nearest neighbors.
Figure 6(a-b) plot the two metrics "average per-client accuracy" and "% client hurt" for different interpolation coefﬁcients.
We see that the two metrics may prefer different hyperparameter values.
Section B. points out two other reasons why hyperparameter tuning is difﬁcult for ﬁne-tuning.
Both of them apply to kNN-Per as well, including the noisy local metrics and potentially extra hyperparameters.
Observation 3: Sensitive to distribution shift.
We follow the same ID and OOD evaluation procedure in Section B.1.
As shown in Figure 4, kNN-Per seems to give a similar or better ID-OOD tradeoff (i.e., achieves a higher OOD accuracy at the same ID accuracy) compared to the other
[Table]
Practical concerns: Similar to those in the cross-device setting (Table 4)
[Table]
Tuning per-client hyperparameter may outperform global tuning (Fig. 7).
[Table]
FedAvg
[Table]
Practical concerns: Similar to the cross-device setting (Table 4); on Vehicle and School, mode collapse occurs less potentially from using a linear model.
[Table]
Practical concerns: Similar to those in the cross-device setting (Table 4).
[Table]
personalization algorithms.
Nevertheless, it is still worth exploring methods that improve the OOD robustness during personalization, as discussed in the "Remark: Acceptance criteria and robust personalization" in Section B.1.
Observation 4: Performance drops when clients have smaller personalization set 8 .
This has been discussed previously in Section B.1.
As shown in Figure 6, the performance of kNN-Per and ﬁne-tuning drops quickly when the test clients have fewer examples to personalize the model (in the case of kNN-Per, fewer examples to ﬁnd the neareast neighbors).
This is in contrast to HypCluster, which only uses the local personalization set for identifying the best model instead of training a new model as in ﬁne-tuning and kNN-Per.
C Cross-Silo Experiments
We now consider baselines for the cross-silo personalization methods and datasets in Motley .
Training and evaluation setup.
As mentioned in Section A.2, in the cross-silo setting, the same silos appear in both training and inference; to evaluate this setting, we split each client’s local examples into three sets: train/validation/test.
We train on the training sets across all clients, tune hyperparameters on the validation sets (see Appendix F.3), and report metrics on the test sets.
Unlike the cross-device setting, in the cross-silo setting, each client typically has sufﬁcient compute power and a large number of local examples.
This allows us to tune ﬁne-tuning hyperparameters in a per-client manner (e.g., every client chooses their own hyperparameters based on their validation metric), as opposed to tuning globally in the cross-device setting in Section B.1.
Results.
Table 5 reports the cross-silo experimental results in the same format as in Table 4.
Most of the practical concerns of FedAvg+Fine-tuning and HypCluster discussed in Section B are still applicable here, e.g., HypCluster is still difﬁcult to train due to the mode collapse issue (although it may happen less frequently when training simple linear models on Vehicle and School).
In addition to the shared practical concerns, we observe four key trends speciﬁc to the cross-silo setting.
Observation 1: Effectiveness of local training.
Local training may be a strong baseline in cross-silo settings.
If we consider client(silo)-level differential privacy, it has an extra beneﬁt of no privacy cost.
Observation 2: Importance of personalization.
The three personalization algorithms (FedAvg +
Fine-tuning, HypCluster, and MTL) all achieve better mean accuracy (or MSE) and fairness 7 (i.e., smaller per-client metric standard deviation in Table 5) than that of FedAvg.
Observation 3: Performance of MTL.
We see that (stateful) MTL methods could yield competitive performance with ﬁnetuning on the three datasets, with potentially less hyperparameter tuning.
Observation 4: Effectiveness of client-speciﬁc ﬁne-tuning.
For FedAvg+Fine-tuning, we compared tuning the ﬁne-tuning hyperparameters (i.e., ﬁne-tuning learning rate and the number of ﬁne-tuning epochs) in a per-client (every client chooses their own hyperparameter) vs global (all clients share the same hyperparameter) manner.
Using client-speciﬁc ﬁne-tuning hyperparameters can be better than tuning globally when each client’s local data is large (see the ADNI result in
Figure 7).
D Background on Personalized Federated Learning
Traditionally, federated learning objectives consider ﬁtting a single global model, w , across all local data in the network.
The aim is to solve:
[Equation]
where F k ( w ) is the local objective for client k , and G ( · ) is a function that aggregates the local objectives { F k ( w ) } k ∈
[ K ] from each client.
For example, in FedAvg [57], G ( · ) is typically set to be a weighted average of local losses, i.e., (cid:80)
Kk =1 p k F k ( w ) , where p k is a pre-deﬁned non-negative weight such that (cid:80) k p k
= 1 .
However, in general, each client may generate data x k via a distinct distribution D k , i.e., F k ( w ) :
=
E x k ∼D k
[ f k ( w ; x k )] .
To better account for this heterogeneity, it is therefore increasingly common to consider techniques (described below) that learn personalized, client-speciﬁc models, { w k } k ∈
[ K ] across the network.
A distinguishing factor of personalized FL methods is whether the approach requires any variables to be maintained on participating clients from one round to another [74].
As discussed in Section 1, whereas stateless approaches may be applicable to either cross-device or cross-silo FL, stateful approaches are more appropriate for cross-silo settings given the typical size and conﬁguration of the network.
Besides stateful vs stateless, the personalized FL algorithms can be categorized into modelagnostic and model-speciﬁc.
Model-speciﬁc approaches target a speciﬁc model and usually require domain-speciﬁc information [34, 67, 84].
The current version of Motley focuses on benchmarking personalization algorithms that can work without assumptions on the model or application scenario.
We discuss major model-agnostic approaches in stateful vs. stateless personalized FL below, and defer readers to the recent surveys [70][74, §7.5] for more related work on personalized FL.
Stateful Approaches.
A common class of stateful personalized FL methods are multi-task learning (MTL) methods.
These methods view each client (or group of clients) as a separate ‘task’, and aim to jointly learn task-speciﬁc models while exploiting similarities/differences between tasks.
The idea of solving multiple learning tasks simultaneously was ﬁrst popularized by Caruana [11] in the 90’s, who described multi-task learning as a technique in which related tasks act as a form of inductive bias to improve generalization.
Many approaches can be captured in the following general and widely-used formulation, known as multi-task relationship learning [68, 80, 81]:
[Equation]
Here W :=
[ w 1 , . . .
, w K ] ∈ R d × K is a matrix whose k -th column is the weight vector for the k -th task (in this case, the model on client k ).
The matrix Ω ∈ R K × K models relationships amongst tasks, and is either known a priori or estimated while simultaneously learning task models.
MTL problems differ based on their assumptions on R , which takes Ω as input and promotes some suitable structure amongst the tasks.
In federated learning, a number of speciﬁc MTL instantiations have been proposed, including variants of general relationship learning [32, 68], cluster-regularized
MTL [65, 68], global-regularized MTL [49, 79], and mean-regularized MTL [17, 21, 22].
Beyond these approaches, another common variant of MTL, particularly for deep learning problems, is ‘hard parameter’ sharing approaches [63].
These methods consider splitting the model architecture itself into two components: a shared part that is jointly learned by all clients, and a local part that is personalized to each client.
The local portion can be simply ﬁne-tuned (as discussed below), or may be learned in conjunction with the shared portion by saving the local state on each client at every round, in which case they would also fall under the category of stateful FL [4, 23, 52].
Stateless Approaches.
In the stateless category, one of the most common forms of personalization is simple ﬁne-tuning.
With ﬁne-tuning, a shared model is trained and deployed on each client, and the model is then ﬁne-tuned or adapted locally to the client’s data.
In its simplest form the deployed model could be a global model trained via a typical procedure such as FedAvg [57, 62], and additional iterations of a stochastic optimizer such as mini-batch SGD can be run locally after deployment.
However, it is also natural to consider meta-learning approaches as part of this workﬂow [13, 19, 35, 38], which are speciﬁcally designed to learn an algorithm that can solve a new task with a small number of training samples.
These approaches can be particularly useful in cross-device FL settings, where each client may generate only a small number of training points, and it may be necessary to deploy and adapt models to clients that did not participate in training.
Finally, unlike the stateful MTL-based clustering approaches discussed above, it is also possible to consider clustering variants that don’t require state to be maintained.
In particular, a common approach is to maintain multiple global models and to have participating clients determine which of the models is best suited to their local data, thus forming a natural clustering amongst the clients [20, 53].
In developing baselines for Motley , we explore a simple ﬁne-tuning after FedAvg approach (ﬁnetuning different layers), and a stateless clustering algorithm
[20, 53] (different initialization schemes) for both cross-silo and cross-device settings.
We also include a baseline of training local models without federation.
Furthermore, we explore a stateful baseline of MTL (both simpler and more complex approaches) for the cross-silo setting.
See Section A.1 for more details.
E Datasets
[Figure]
All the datasets used in Motley are listed in Table 2.
They do not contain personally identiﬁable information or offensive content.
They all have natural per-client data partitions, which will be discussed in more details below.
Figure 10 shows that each client’s local data distribution is distinct from each other, a common property of real-world federated datasets (see the discussions on "Heterogeneity" in
Section 1 and Appendix D).
E.1 Cross-device
We describe the cross-device datasets in Table 2.
We will also mention how we split the clients into train, validation, and test clients (a key preprocessing step described in Section A.2).
• EMNIST is ﬁrst processed by the LEAF benchmark [10].
We use the version from [6], which contains 3400 writers/clients and a total of 671,585 images 24 .
Each client has on average 200 images (28-by-28 pixels) of hand-written digits and characters (62 classes).
We use a CNN model [43] with 1M parameters (same as [14]).
The 3400 clients are randomly split into 2500/400/500 as train/validation/test.
• StackOverﬂow is a large-scale federated language dataset.
Each client is a user in the Stack Overﬂow online forum.
The examples are the questions and answers posted by this user.
Each example is a sentence.
The task is next word prediction.
We use an LSTM model [28] with 4M parameters and 10k vocabulary size (same as [14]).
The version from [7] contains 342k train clients and 38k held-out clients.
We randomly split the held-out clients into 10k/28k for validation/test. • Landmarks is a federated image classiﬁcation dataset processed by Hsu et al.
[30] (from theLandmark Recognition Challenge [76]).
Each client is a Wikipedia contributor.
The images are the landmark (e.g., famous monuments and mountains) photos uploaded by the photographers.
The per-user data distribution naturally varies based on the photographer’s geographic location.
The dataset contains 164k images, 2028 landmark labels, and 1262 clients.
We use a MobileNetV model [64] with 4M parameters (same as [30]).
We randomly split the clients into 1112/50/100 as train/validation/test. • TedMulti-EnEs is a subset of the TedMultiTranslate dataset.
The original TedMultiTranslate is a multilingual (60 languages) dataset derived from the TED Talk transrcipts [61].
We only use two languages English and Spanish (and hence, "EnEs" in the name).
This dataset has not been used in any federated learning experiment before, so we need to do more preprocessing work.
We ﬁrst partition the clients by the TED Talk author and language (a client contains either English data or Spanish data but not both 25 ).
Each example is a sentence from the talk.
The task is next word prediction.
We build a vocabulary of size 15k containing both English and Spanish words (these words appear in at least 20 clients).
We use a transformer model [71] with 4M parameters.
Clients with less than 20 examples are removed.
We randomly split the 4184 clients into 3969/98/117 as train/validation/test.
E.2 Cross-silo
• Vehicle dataset, originally collected by Duarte and Hu [18], is a binary classiﬁcation dataset containing measurements of road segments from a distributed network of 23 vehicle sensors with a total of 43,695 feature vectors for classifying the type of the passing vehicles.
Following Smith et al.
[68], we treat each sensor as a task (data silo) and use 50 acoustic and 50 seismic features as inputs to a linear SVM model.
It is suitable for cross-silo FL because of the small number of silos and sufﬁciently large local datasets for local training.
We split each client(silo)’s local dataset into 70%/15%/15% train/validation/test sets. • School dataset, originally collected by the now-defunct Inner London Education Authority,is a regression dataset for predicting the exam scores of 15,362 students distributed acrosssecondary schools.
Each school has records up to 251 students with each student described by a 28-dimensional feature vector capturing attributions such as the school ranking, student birth year, and whether the school provided free meals.
Simple linear regression models sufﬁce for reasonable performance.
We split each client(silo)’s local dataset into 70%/15%/15% train/validation/test sets. • ADNI (Alzheimer’s Disease Neuroimaging Initiative) 27 is a public medical dataset containing various formats of data to help advance the study of Alzheimer’s disease.
The task is to predict Standardized Uptake Value Ratio (SUVR) from PET scans of human brains.
We treat each scanner vendor as a silo, and there are 9 silos in total.
We speciﬁcally use a subset of PET scans (with AV45 and preprocessing step ‘Coreg, Avg, Std Img and Vox Siz, Uniform
Resolution’ ) that have existing labels obtained from UC Berkeley study in the database.
We convert the data into .png
format, normalize each pixel value to (0 , 1) , and rescale each image into size 32 × 32.
We split each client(silo)’s local dataset into 80%/10%/10% train/validation/test sets.
E.3 License/Usage Information
• EMNIST.
The Extended MNIST dataset 29 is a variant of the original NIST Special Database 19 30 .
In this paper, we use a federated version of this Extended MNIST dataset, which is processed and provided by the LEAF 31 benchmark [10] under the BSD-2-Clause license.
• StackOverﬂow.
This dataset is derived from the Stack Overﬂow Data hosted by Kaggle 32 , under the CC BY-SA 3.0 license.
In this paper, we use the federated version of this dataset processed by Tensorﬂow Federated [7], which is open-sourced under the Apache-2.0 license.
• Landmarks.
The original Google Landmarks Dataset v2 [76] can be accessed from GitHub 33 : "The annotations are licensed by Google under CC BY 4.0 license.
The images listed in this dataset are publicly available on the web, and may have different licenses".
In this paper, we use a subset of the full dataset processed by Hsu et al.
[30]: the data split function is available on GitHubunder the Apache-2.0 license.
• TedMulti-EnEs.
The original TedMultiTranslate dataset is derived from the TED Talk transcripts [61].
The TED Talk usage policy can be found from their website 35 .
The TedMultiTranslate dataset can be downloaded from GitHub 36 .
We were unable to ﬁnd the license information.
In this paper, we extract a subset of this dataset and group the examples by talk author and language 25 . This preprocessing pipeline is available from our benchmark code 1 under the Apache-2.0 license. • Vehicle.
The Vehicle dataset is made publicly available by the original authors as a research dataset [18].
It has been subsequently used in recent work (e
g. [68]).
License information was unavailable online.
A copy of the dataset may be obtained from these URLs.
37, • School.
The School dataset was collected by a now-defunct entity and we were unable to ﬁnd license information online.
The dataset has been made freely available online and can be obtained from this URL.
Alternatively, a copy may also be obtained from [83] under the GNU GPL v license.
• ADNI.
The term of use of ADNI can be found on their website.
F Hyperparameters and Implementation Details
In this section, we describe the computation resources, the hyperparameter grids and the best hyperparameters for all the experiments described in Section B and Section C.
F.1 Cross-device
The deﬁnitions of all hyperparameters can be found in finetuning_trainer.py and hypcluster_trainer.py in our code, which will be open-sourced after the paper is accepted.
Common hyparameters.
The following hyperparameters are ﬁxed across all cross-device experiments.
Note that we focus on FedAdam here, i.e., a generalized version of FedAvg [62], where the server optimizer is Adam optimizer [40], because [14] shows FedAdam gives good performance across different datasets.
• client_optimizer=[‘sgd’] • server_optimizer=[‘adam’] • server_adam_beta_1=[0.9] • server_adam_beta_2=[0.99] • train_epochs=[1] (This is the number of local training epochs performed by a client during a round of training.)
Computation resources.
We summarize the computation resources allocated to run one experiment (i.e., one point in the the hyperparameter grids) on each dataset: EMNIST (80 CPU cores);
StackOverﬂow (400 GPU cores); Landmarks (16 GPUs); TedMulti-EnEs (2 GPUs).
Note that the actual usage may be smaller than the allocated resources.
F.1.1 FedAvg (i
e., FedAdam in our case) + Fine-tuning
The deﬁnitions of all the hyperparaemeters for running this algorithm can be found in finetuning_trainer.py .
Since FedAvg + Fine-tuning is a two step process, the hyperparameters contain FedAvg (i.e., FedAdam in our case) hyperparameters and ﬁne-tuning hyperparameters.
For EMNIST and StackOverﬂow, we use the best FedAdam hyperparameters from [14].
For Landmarks, we use the best FedAdam hyperparameters from [74].
For TedMulti-EnEs, we tune the
FedAdam hyperparameters from scratch.
We have four ﬁne-tuning hyperparameters.
Their names are all started with finetune_ , including the optimizer used to ﬁne-tune the model (where we focus on SGD 16 ), the ﬁne-tuning learning rate, and whether to only ﬁne-tune the last layer.
We also need to tune the number of local epochs used to ﬁne-tune the model - this value is automatically found by finetuning_trainer.py by postprocessing the validation metrics.
Speciﬁcally, we compute the average validation accuracy of the ﬁne-tuned models after every ﬁne-tuning epoch (until finetune_max_epochs ), and then ﬁnd the best ﬁne-tuning epoch that gives the highest average validation accuracy (see, e.g., Figure 9(a) 41 ).
The best ﬁne-tuned epoch will be in the range [0, finetune_max_epochs] , so all we need is to set a proper value for finetune_max_epochs .
EMNIST
Fixed hyperparameters (we use the best FedAdam hyperparameters from [14]):
• client_learning_rate=[0.1] • server_learning_rate=[0.001] • server_adam_epsilon=[0.001] • clients_per_train_round=[50] • train_batch_size=[20] • total_rounds=[1500] • valid_clients_per_round=[100] • test_clients_per_round=[100]
• rounds_per_evaluation=[100] • rounds_per_checkpoint=[100] • finetune_optimzier=[‘sgd’] • finetune_max_epochs=[20]
Tuned hyperparameters (best values are highlighted in **value** ):
• finetune_learning_rate=[0.001, 0.003, **0.005**, 0.01, 0.05] • finetune_last_layer=[True, **False**]
StackOverﬂow
Fixed hyperparameters (we use the best FedAdam hyperparameters from [14]):
• client_learning_rate=[1.0] • server_learning_rate=[0.1] • server_adam_epsilon=[0.001] • clients_per_train_round=[200] • train_batch_size=[16] • total_rounds=[1500] • valid_clients_per_round=[200] • test_clients_per_round=[200] • rounds_per_evaluation=[100] • rounds_per_checkpoint=[100] • finetune_optimzier=[‘sgd’] • finetune_max_epochs=[20]
Tuned hyperparameters (best values are
[Table]
in ** [Table]
• finetune_learning_rate=[**10^(-1.0)**, 10^(-0.6), 10^(-0.2), 10^(0.2), 10^(0.6), 10^(1.0)] • finetune_last_layer=[True, **False**]
Landmarks
Fixed hyperparameters (we use the best FedAdam hyperparameters from [74]):
• client_learning_rate=[0.01] • server_learning_rate=[10^(-2.5)] • server_adam_epsilon=[10^(-5)] • clients_per_train_round=[64] • train_batch_size=[16] • total_rounds=[30000] • valid_clients_per_round=[32] • test_clients_per_round=[96] • rounds_per_evaluation=[1000] • rounds_per_checkpoint=[1000] • finetune_optimzier=[‘sgd’] • finetune_max_epochs=[10]
Tuned hyperparameters (best values are highlighted in **value** ):
• finetune_learning_rate=[0.0001, 0.001, 0.005, **0.007**, 0.01, 0.03, 0.05] • finetune_last_layer=[True, **False**] TedMulti-EnEs
Fixed hyperparameters:
• clients_per_train_round=[32] • train_batch_size=[16] • total_rounds=[1500] • valid_clients_per_round=[98] • test_clients_per_round=[117] • rounds_per_evaluation=[30] • rounds_per_checkpoint=[50] • finetune_optimzier=[‘sgd’] • finetune_max_epochs=[20]
Tuned hyperparameters (best values are highlighted in **value** ):
• client_learning_rate=[10^(-2.5), 10^(-2), 10^(-1.5), **10^(-1)**, 10^(-0.5)] • server_learning_rate=[10^(-2.5), **10^(-2)**, 10^(-1.5), 10^(-1), [-10^(-0.5)]• finetune_learning_rate=[**0.
 **,-]10^(-0.5)] • finetune_learning_rate=[**0.0005**, 0.0007, 0.001, 0.002, 0.003] • finetune_last_layer=[True, **False**]
F.1.2 HypCluster
Deﬁnitions of all the hyperparameters for this algorithm can be found in hypcluster_trainer.py .
Because HypCluster with random initialization usually ends up with all clients choosing the same model (i.e., the mode collapse issue shown in Figure 5), we will use models learned by FedAvg to warmstart HypCluster.
Speciﬁcally, we will run FedAvg (with the hyperparameters in Appendix F.1. above) for num_warmstart_fedavg_rounds ; repeat this for num_clusters times, and use the models to warmstart HypCluster.
EMNIST
Fixed hyperparameters:
• clients_per_train_round=[50] • train_batch_size=[20] • total_rounds=[1500] • valid_clients_per_round=[100] • test_clients_per_round=[100] • rounds_per_evaluation=[100] • rounds_per_checkpoint=[100] • num_warmstart_fedavg_rounds=[100]
Tuned hyperparameters
• client_learning_rate=[0.01, 0.05, **0.1**, 0.2] • server_learning_rate=[0.0001, 0.0005, **0.001**, [-0.002]• server_adam_epsilon=[**0.
 **,-]0.002] • server_adam_epsilon=[**0.0001**, 0.0005, 0.001, 0.002] • num_clusters=[**2**, 3, 4, 5]
StackOverﬂow
Fixed hyperparameters:
• clients_per_train_round=[200] • train_batch_size=[16] • total_rounds=[1500] • valid_clients_per_round=[200] • test_clients_per_round=[200] • rounds_per_evaluation=[100] • rounds_per_checkpoint=[100] • num_warmstart_fedavg_rounds=[100]
Tuned hyperparameters
highlighted
• client_learning_rate=[0.1, **0.5**, 1.0, 2.0] • server_learning_rate=[**0.01**, 0.05, 0.1, 0.2] • server_adam_epsilon=[10^(-5), **10^(-4)**, 10^(-3), 10^(-2)] • num_clusters=[**2**, 3, 4, 5] Landmarks
Fixed hyperparameters:
• clients_per_train_round=[64] • train_batch_size=[16] • total_rounds=[30000] • valid_clients_per_round=[32] • test_clients_per_round=[96] • rounds_per_evaluation=[1000] • rounds_per_checkpoint=[1000] • num_warmstart_fedavg_rounds=[8000]
Tuned
hyperparameters (best
are highlighted in *
• client_learning_rate=[10^(-3), **10^(-2.5)**, 10^(-2), 10^(-1.5)] • server_learning_rate=[10^(-3.5), **10^(-3)**, 10^(-2.5), 10^(-2)] • server_adam_epsilon=[10^(-6), 10^(-5), **10^(-4)**, 10^(-3)] •
num_clusters=[**2**, 3, 4] TedMulti-EnEs
Fixed hyperparameters:
• clients_per_train_round=[32] • train_batch_size=[16] • total_rounds=[1500] • valid_clients_per_round=[98] • test_clients_per_round=[117] • rounds_per_evaluation=[30]
• rounds_per_checkpoint=[50] • num_clusters=[2] • num_warmstart_fedavg_rounds=[100]
Tuned hyperparameters (best values are highlighted in **value** ):
• client_learning_rate=[10^(-2.5), 10^(-2), 10^(-1.5), **10^(-1)**, 10^(-0.5)] • server_learning_rate=[10^(-2.5), **10^(-2)**, 10^(-1.5), 10^(-1), 10^(-0.5)] • server_adam_epsilon=[**0.
F.2 FedAvg + kNN-Per
We use the model trained by FedAvg (same as "FedAvg + Fine-tuning") as the global model for personalization.
Following [54], we set the number of nearest neighbors k to 10 for all experiments (the paper shows that the performance of kNN-Per is robust to this value).
As mentioned in Section B.3, we tune the interpolation coefﬁcient globally for all clients (i.e., every client use the same coefﬁcient), because tuning per-client speciﬁc coefﬁcient performed worse.
The tuned interpolation hyperparameters are shown below (best values are highlighted in **value** ):
EMNIST coefficient=[0, 0.1, 0.2, 0.3, **0.4**, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
StackOverﬂow coefficient=[0, 0.1, **0.2**, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
Landmarks coefficient=[0, 0.1, 0.2, 0.3, 0.4, **0.5**, 0.6, 0.7, 0.8, 0.9, 1.0]
TedMulti-EnEs coefficient=[0, **0.1**, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]
F.2.1 Local training
Traning a local model at each client can be done by running finetuning_trainer.py with total_rounds=0 .
Note that what happens is that every client ﬁne-tunes a random model (sent by the server) locally.
As long as we set a large enough finetune_max_epochs (the best number of local epochs will be found after postprocessing the validation metrics), this will give the desired metrics where each client learns a local model without federation.
EMNIST
Fixed hyperparameters:
• total_rounds=[0] • valid_clients_per_round=[100] • test_clients_per_round=[100] • finetune_optimzier=[‘sgd’] • finetune_max_epochs=[200]
Tuned hyperparameters (best values are highlighted in **value** ):
• finetune_last_layer=[True, **False**] • finetune_learning_rate=[0.001, 0.01, **0.1**, 0.5, 1.0]
StackOverﬂow
Fixed hyperparameters:
• total_rounds=[0] • valid_clients_per_round=[200] • test_clients_per_round=[200] • finetune_optimzier=[‘sgd’] • finetune_max_epochs=[200]
Tuned hyperparameters (best values are highlighted in **value** )
• finetune_last_layer=[True, **False**] • finetune_learning_rate=[0.1, **0.5**, 1.0]
Landmarks
Fixed hyperparameters:
• total_rounds=[0] • valid_clients_per_round=[32] • test_clients_per_round=[96] • finetune_optimzier=[‘sgd’] • finetune_max_epochs=[50]
Tuned
hyperparameters (best values are highlighted in **value** ): • finetune_last_layer=[True, **False**] • finetune_learning_rate=[0.0001, 0.001, **0.01**, 0.1]
TedMulti-EnEs Fixed hyperparameters:
• total_rounds=[0] • valid_clients_per_round=[98] • test_clients_per_round=[117] • finetune_optimzier=[‘sgd’] • finetune_max_epochs=[50]
Tuned hyperparameters (best values are highlighted in value
• finetune_last_layer=[**True**, False] • finetune_learning_rate=[0.5, **1.0**, 2.0, 3.0]
F.3 Cross-silo
The deﬁnitions of hyperparameters can be found in main.py in the implementation folders for the
Vehicle, School, and ADNI datasets respectively in our code, which will be open-sourced after the paper is accepted.
Common hyperparameters.
The following hyperparameters are ﬁxed across all cross-silo experiments:
• client_optimizer=[‘sgd’] • server_optimizer=[‘fedavgm’] • fedavgm_momentum=[0.9] • inner_epochs=[1] (This is the number of local training epochs performed by a client during a round of training.
For methods that train a global model, this is the number of local training epochs before returning the model update to the server.
We focus on FedAvgM [29] for simplicity of hyperparameter tuning.
Note also that the precise variable names of the hyperparameters (typefaced with monospaced font ) may vary depending on the dataset-speciﬁc implementation.
Computational resources.
For the Vehicle and School datasets, all experiments (including hyperparameter search) are done on 88 commodity CPU cores.
For the ADNI dataset, each run is done on one GPU.
F.3.1 Local training Vehicle
Fixed hyperparameters:
• num_rounds=[500] • clients_per_round=[23] • batch_size=[64] (client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ):
• client_lrs=[0.003, 0.01, **0.03**, 0.1, 0.3]
School
Fixed hyperparameters:
• num_rounds=[500] • clients_per_round=[139] • batch_size=[32] (client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ):
• client_lrs=[0.001, 0.003, **0.01**, 0.03, 0.1]
ADNI Fixed hyperparameters:
• num_rounds=[70] • clients_per_round=[9]
• batch_size=[64] (client local training batch size)
Tuned hyperparameters (best values are highlighted in *value
• client_lrs=[0.001, **0.01**, 0.1]
F.3.2 FedAvg+Fine-tuning
Vehicle
Fixed hyperparameters:
• num_rounds=[500] • finetune_epochs=[100] (max number of local epochs for ﬁne-tuning) • finetune_every=[50] (run ﬁne-tuning every number of rounds) • finetune_optimizer=[‘sgd’]
• clients_per_round=[23] • batch_size=[64] (client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ):
• client_lrs=[**0.
*, 0.01, 0.03, 0.1, 0.3] • server_lrs=[0.1, 0.3, 1, 3, **10**] • finetune_lrs=[0.003, 0.01, 0.03, 0.1, 0.3]
(The ﬁne-tuning client learning rates are tuned separately for each client.)
School
Fixed hyperparameters:
• num_rounds=[500] • finetune_epochs=[100] (max number of local epochs for ﬁne-tuning) • finetune_every=[50] (run ﬁne-tuning every number of rounds) • finetune_optimizer=[‘sgd’] • clients_per_round=[139] • batch_size=[32] (client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ):
• client_lrs=[**0.001**, 0.003, 0.01, 0.03, 0.1] • server_lrs=[0.1, 0.3, 1, **3**, 10] • finetune_lrs=[0.001, 0. [Table]
(The ﬁne-tuning client learning rates are tuned separately for each client.) ADNI
Fixed hyperparameters:
• num_rounds=[70] • finetune_optimzier=[‘sgd’] • clients_per_round=[9] • batch_size=[64]
(client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ):
• finetune_last_layer=[True, False] • server_lrs=[1, 3, **10**, 20] • client_lrs=[0.0001, 0.001, **0.01**, 0.1] • finetune_lrs=[0.0001, 0.001, 0.01, 0.1] (When running FedAvg, the optimal client-side learning rate is 0.01.
During ﬁne-tuning, we tune client learning rates (i.e., ﬁne-tuning learning rates) separately for each client.)
[Table]
Fixed hyperparameters:
• num_rounds=[500] • clients_per_round=[23] • batch_size=[64] (client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ):
• client_lrs=[0.003, **0.01**, 0.03, 0.1, 0.3] • server_lrs=[0.1, 0.3, 1, 3, **10**] • num_clusters=[2, 3, **4**] • warmstart_fracs=[**0**, 0.2] (The fraction of rounds for warm starting; the rest runs clustering training.) School
Fixed hyperparameters:
• num_rounds=[500] • clients_per_round=[139] • batch_size=[32] (client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ):
• client_lrs=[0.001, 0.003, **0.01**, 0.03, 0.1] • server_lrs=[0.1, 0.3, 1, 3, **10**] • num_clusters=[2, **3**, 4] • warmstart_fracs=[0, **0. **] (The fraction of rounds for warm starting; the rest runs clustering training.)
ADNI
Fixed hyperparameters:
• num_rounds=[70] • warm_start_rounds=[20] • clustering_training_rounds=[50] • clients_per_round=[9] • batch_size=[64]
(client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ):
• server_lrs=[1, 3, **10**, 20] • client_lrs=[0.0001, 0.001, **0.01**, 0.1] • num_clusters=[**2**, 3, 4] F.4 FedAvg + kNN-Per
We use the model trained by FedAvg (same as "FedAvg + Fine-tuning") as the global model for personalization.
Following [54], we set the number of nearest neighbors k to 10 for all experiments (the paper shows that the performance of kNN-Per is robust to this value).
Similar to the cross-device setting, we also tune the interpolation coefﬁcient globally for all clients (i.e., every client use the same coefﬁcient).
The tuned interpolation coefﬁcients are shown below (best values are highlighted in **value** ):
Vehicle coefficient=[0, 0.1, 0.3, 0.5, 0.7, **0.9**, 1.0]
School coefficient=[0, 0.1, 0.3, **0.5**, 0.7, 0.9, 1.0]
ADNI coefficient=[0, 0.1, 0.3, **0.5**, 0.7, 0.9, 1.0]
F.4. 1 MOCHA
Note that MOCHA [68] was implemented in its primal form (i.e. gradient descent update for the local personalized models, with task-relationship learning regularization) instead of the dual form since the dual was not derived for regression in the original paper.
We were able to reproduce the results reported in the original paper in the primal (with an average error rate of 6.29 in Tables 5 and 7, smaller than 6.59 that was reported).
In the primal, MOCHA has optimization hyperparameters such as client optimizer, client learning rates, batch sizes, etc.
as with other methods.
Vehicle
Fixed hyperparameters:
• num_rounds=[500] • clients_per_round=[23] • batch_size=[64] (client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ; an alternative combination that gives very similar results are highlighted in _values_ ):
• client_lrs=[0.003, _0.01_, **0.03**, 0.1, 0.3] • lambdas=[**0.0001**, _0.001_, 0.01, 0.1, 0.3, 1, 3] (MTL regularization strength.)
• mocha_outers=[**1**, _2_, 5]
(The number of local epochs every server update of the task-relationship matrix.
This nests with inner_epochs and may thus increase the total number of epochs over the local datasets
School
Fixed hyperparameters:
• num_rounds=[500] • clients_per_round=[139] • batch_size=[32] (client local training batch size)
Tuned hyperparameters (best values are highlighted in **value** ; an alternative combination that gives very similar results are highlighted in _values_ ):
• client_lrs=[0.001, **0.003**, _0.01_, 0.03, 0.1] • lambdas=[**0.0001**, 0.001, _0.01_, 0.1, 0.3, 1, 3] (MTL regularization strength.)
• mocha_outers=[_1_, **2**, 5] (See Vehicle hyperparameters above for description.
F.4.2 Ditto
Vehicle
Fixed hyperparameters:
• num_rounds=[500] • personalized_model_inner_epochs=[1] • clients_per_round=[23] • batch_size=[64] (client local training batch size)
Tuned hyperparameters (best values
are highlighted
in value*
• client_lrs=[0.003, 0.01, 0.03, 0.1, **0.3**] • server_lrs=[0.1, 0.3, **1**, 3, 10] • lambdas=[0.0001, 0.001, **0.01**, 0.1, 0.3, 1, 3] (MTL regularization strength.)
• personalized_model_lrs=[0.003, 0.01, 0.03, **0.1**, 0.3] (The client learning rate for Ditto’s personalized models.
This may be different from the client learning rates used to update the global model.)
School
Fixed hyperparameters:
• num_rounds=[500] • personalized_model_inner_epochs=[1] • clients_per_round=[139] • batch_size=[32] (client local
Tuned hyperparameters (best values are highlighted in **value** ):
• client_lrs=[**0.001**, 0.003, 0.01, 0.03, 0.1] • server_lrs=[0.1, 0.3, 1, 3, **10**] • lambdas=[0.0001, 0.001, 0.01, 0.1, 0.3, **1**, 3] (MTL regularization strength.)
• personalized_model_lrs=[0.001, 0.003, **0.01**, 0.03, 0.1] (See Vehicle hyperparameters above for description.)
ADNI
Fixed hyperparameters:
• num_rounds=[100] • clients_per_round=[9] • local_batch_size=[64] • personalized_model_local_epoch=[1]
Tuned hyperparameters (best values are highlighted in **value** ):
• server_lrs=[1, 3, **10**, 20] • client_lrs=[0.0001, 0.001, **0.01**, 0.1] • personalized_model_lrs=[0.0001, **0.001**, 0.01] (See Vehicle hyperparameters above for description.)
• lambdas=[0.01, 0.1, **1**] (MTL regularization strength.)
G Additional Experimental Results and Discussions
In this section, we provide a few more experimental results that are omitted from the main paper due to space limitation.
Cross-device experiments.
Table 6 extends Table 4 with standard deviations across 5 different runs for each metric.
Note that there are two standard deviations here: 1) the standard deviation of per-client accuracy across the test clients, which is used as a fairness metric 7 and is already included in Table 4; 2) the standard deviation of each metric value (e.g., the standard deviation of the fairness metric mentioned in 1)) across 5 different runs (each run uses a different seed), which is omitted from the main paper and is included in Table 6.
Cross-silo experiments.
Similar to the cross-device counterpart, Table 7 extends Table 5 with standard deviation across 5 different runs for each metric.
One small caveat around the use of client-speciﬁc ﬁne-tuning hyperparameters, as discussed in Appendix C of the main paper, is that it may improve performance only when the clients’ local datasets are large.
Here, we perform an additional analysis on the School dataset, where the local datasets are relatively small (recall Table 2).
Figure 11 visualizes the statistics of clients hurt when training on School, akin to Figure 8 (note that we use MSE here, so a positive delta means this client gets hurt by ﬁne-tuning).
Observe that while many clients beneﬁted from the client-speciﬁc ﬁne-tuning (those with negative metric deltas), some clients do not observe an improvement and may even see a slight degradation in their utility compared to the client-agnostic ﬁne-tuning setting.
We argue that this is due to clients having small local datasets such that the selected client-speciﬁc FT hyperparameters may overﬁt to the small local validation sets and may not reﬂect general improvement on the local test sets.
Note also that while Table 5 and Table 7 suggests a seemingly considerable fraction (33%) of clients is hurt after ﬁne-tuning, we can observe from Figure 11 that most ‘hurt’ clients have borderline metric changes.
Overall, these observations suggest that the effect of client-speciﬁc ﬁne-tuning can be mixed if the client local datasets are relatively small, as is usually the case with cross-device federated learning.
"per-client acc std", the std of per-client accuracy across the test clients, which is considered in [ is already included in Table across different omitted from Table shown
[Table]
difﬁcult to tune hyperparameters
[Table]
may hurt clients
[Table]
sensitive to distribution shift
[Table]
performance drops with fewer local examples (see Section B.1)
[Table]
002 0.273 ± .
0.573 ± .005 0.163 ± .
Per-client acc std 0.08 ± .
0.03 ± .
0.16 ± .
0.04 ± .
No. tuned clusters ( k )    % clients largest cluster 52.6% ± 0.8%85.1% ± 1.6% 92.1% ± 3.6%54.7% ±
Warmstart from FedAvg
Yes
Yes Yes
Yes
Per-client acc mean by ensembling k FedAvg models 0.860 ± .005 0.271 ± .
0.564 ± .016 0.163 ± .
Per-client acc std
by ensembling k FedAvg models 0.08 ± .
0.03 ± .
0.16 ± .
0.04 ± .
Practical concerns : difﬁcult to train due to mode collapse; high communication cost; difﬁcult to interpret the learned clusters; sensitive to distribution shift (see Section B.2)
FedAvg + kNN-Per
Per-client acc mean 0.876 ± .003 0.275 ± .
0.735 ± .007 0.162 ± .
Per-client acc std 0.06 ± .
0.03 ± .
0.13 ± .
0.05 ± .
% clients "hurt" 19.8% ± 4.2% 23.6% ± 3.4% 5.6% ± 2.3% 34.4% ± 4.5%
Practical concerns : difﬁcult to tune hyperparameters; may hurt clients; sensitive to distribution shift; performance drops with fewer local examples (see Section B.3)
negative the per-client metric delta, the better.
Unlike Figure 7, tuning the ﬁne-tuning hyperparameters globally and tuning in a per-client manner performs the same on the School dataset.
This is potentially due to the fact that the client’s local dataset is relatively small for School (as shown in Table 2).
[Table]
Practical concerns: Similar to those in the cross-device setting (Table 4)
[Table]
Tuning per-client FT hyperparameter may outperform tuning globally (Figure 7)
[Table]
Practical concerns: Similar to those in the cross-device setting (Table 4); on Vehicle and School, mode collapse may occur less potentially because of linear model
[Table]
Practical concerns [Table] Similar to those in the cross
[Table]
device setting (Table 4)
[Table]
H Conclusion
We present Motley , the ﬁrst large-scale benchmark of personalized federated learning covering both cross-device and cross-silo settings.
Motley provides an end-to-end experimental pipeline including data preprocessing, algorithms, evaluation metrics and tuned hyperparameters, which ensures reproducibility.
Beyond these baselines, our experiments provide new insights about personalized FL and suggest several directions of future work.
• The notion of the "best" method (or even the "best" hyperparameter of the same method) can change when we change the evaluation metrics or settings.
In the cross-device setting, FedAvg+Fine-tuning achieves the best average per-client accuracy for 3/4 datasets, and at the same time also improves fairness 7 (Table 4 and Figure 2).
The best hyperparameters of FedAvg+Fine-tuning can change if we look at different metrics (Figure 9).
On the TedMulti-EnEs dataset, it is difﬁcult to determine which personalization algorithm is the "best": while HypCluster achieves the best average per-client accuracy, it is worse than FedAvg+Fine-tuning from the perspective of accuracy-communication ratio (Figure 5(c)); on the other hand, the ﬁne-tuned models of 40% clients on TedMulti-EnEs are worse than that of FedAvg (Table 4).
In the cross-silo setting, both FedAvg+Fine-tuning and MTL achieve the best average per-client accuracy over the three datasets (Table 5).
MTL may have an additional advantage of having less hyperparameters than FedAvg+Fine-tuning.
On the Vehicle dataset, local training seems the "best" because it achieves a similar accuracy as other methods but is more private.
Given that the notion of "best" method can change depending on the metrics or settings, a critical future direction is thus to develop systematic evaluation schemes for personalized FL (i.e., mean accuracy alone is not enough).
• Existing literature often overlook or obfuscate the practical complexities of deploying personalized FL algorithms in real-world settings.
For example, in Section B.1, we discuss that local data scarcity and heterogeneity create a fundamental challenge to tuning FedAvg+Fine-tuning hyperparameters in the cross-device setting.
Because each client has a very small local dataset in the cross-device setting, the ﬁne-tuning hyperparameters are tuned globally (instead of in a per-client manner as in the cross-silo setting).
The globally tuned hyperparameters may hurt some clients (Figure 8). Ideally, we want to ﬁnd a good set of ﬁne-tuning hyperparameters such that the overall improvement is large and no clients are hurt after ﬁne-tuning, which can be fundamentally difﬁcult (Figure 9).
• Improving HypCluster requires solving the mode collapse issue and rethinking the difference between clustering and ensembling.
How to effectively train HypCluster in the real-world federated learning systems is an interesting open problem (Figure 5(a)).
Although warmstart can mitigate the mode collapse issue, Table 4 and Table 5 show that the performance of HypCluster is similar to that of ensembling 20 multiple models learned by FedAvg.
Does HypCluster really capture the underlying clustering structure (Figure 5(d))?
Answering this question requires interpreting the learned clusters in the federated learning setting.
• Tradeoffs exists between adapting a client’s personalized model to the current local distribution and generalizing to future distributions.
As shown in Figure 4, compared to FedAvg, the ﬁne-tuned models are more sensitive to the distribution shift between the examples used in ﬁne-tuning and testing.
As a result, it may be necessary to continuously ﬁne-tune the model when clients have new data.
Besides the ﬁne-tuning method, this tradeoff between personalization and generalization exists for other personalized FL algorithms as well, which is worth exploring in greater detail. • Given the observed beneﬁts of per-client hyperparameter tuning in
cross-silo FL
(Figure 7), it may be beneﬁcial to develop similar, scalable approaches for hyperparameter tuning in cross-device FL.
Limitations and future work.
Motley can be expanded in the following ways.
First, add more evaluation metrics.
Our benchmark results contains the following metrics (see Table 4 and Table 5): the average per-client accuracy, fairness 7 (clients should have similar local accuracies), and algorithmspeciﬁc metrics such as the percentage of clients hurt by ﬁne-tuning.
When comparing algorithms on speciﬁc datasets, we also use metrics such as the communication cost (Figure 5) and robustness to distribution shift (Figure 4).
Motley can be expanded by adding more evaluation metrics such as privacy, other notions of fairness 7 , and robustness to attacks.
Second, add more datasets, especially those with known clustering structure (similar to TedMulti-EnEs) and datasets representing realworld cross-silo applications.
Third, add more algorithms.
Motley currently has ﬁve model-agnostic federated learning personalization algorithms.
Expanding Motley to include more algorithms (see
Appendix D) would be another important future work.