F I T: P ARAMETER E FFICIENT F EW - SHOT T RANSFER L EARNING FOR P ERSONALIZED AND F EDERATED I MAGE C LASSIFICATION
A BSTRACT
Modern deep learning systems are increasingly deployed in situations such as personalization and federated learning where it is necessary to support i) learning on small amounts of data, and ii) communication efficient distributed training protocols.
In this work, we develop FiLM Transfer (F I T) which fulfills these requirements in the image classification setting by combining ideas from transfer learning (fixed pretrained backbones and fine-tuned FiLM adapter layers) and metalearning (automatically configured Naive Bayes classifiers and episodic training) to yield parameter efficient models with superior classification accuracy at low-shot.
The resulting parameter efficiency is key for enabling few-shot learning, inexpensive model updates for personalization, and communication efficient federated learning.
We experiment with F I T on a wide range of downstream datasets and show that it achieves better classification accuracy than the leading Big Transfer (BiT) algorithm at low-shot and achieves state-of-the art accuracy on the challenging VTAB-1k benchmark, with fewer than 1 % of the updateable parameters.
Finally, we demonstrate the parameter efficiency and superior accuracy of F I T in distributed low-shot applications including model personalization and federated learning where model update size is an important performance metric.
I NTRODUCTION
With the success of the commercial application of deep learning in many fields such as computer vision (Schroff et al., 2015), natural language processing (Brown et al., 2020), speech recognition (Xiong et al., 2018), and language translation (Wu et al., 2016), an increasing number of models are being trained on central servers and then deployed on remote devices, often to personalize a model to a specific user’s needs.
Personalization requires models that can be updated inexpensively by minimizing the number of parameters that need to be stored and / or transmitted and frequently calls for few-shot learning methods as the amount of training data from an individual user may be small (Massiceti et al., 2021).
At the same time, for privacy, security, and performance reasons, it can be advantageous to use federated learning where a model is trained on an array of remote devices, each with different data, and share gradient or parameter updates instead of training data with a central server (McMahan et al., 2017).
In the federated learning setting, in order to minimize communication cost with the server, it is also beneficial to have models with a small number of parameters that need to be updated for each training round conducted by remote clients.
The amount of training data available to the clients is often small, again necessitating few-shot learning approaches.
In order to develop data-efficient and parameter-efficient learning systems, we draw on ideas developed by the few-shot learning community.
Few-shot learning approaches can be characterized in {+[Figure] Parameters 
 Classification (VTAB-1k)+}
{+[Figure] Parameters 
 Personalization [Figure]+}
ORBIT)
[Figure]
BiT FiT
Commun
[Figure]
{+Cost 
+}
Federated Learning (CIFAR100) terms of shared and updateable parameters.
From a statistical perspective, shared parameters capture similarities between datasets, while updateable parameters capture the differences.
Updateable parameters are those that are either recomputed or learned as the model is updated or retrained, whereas shared parameters are fixed.
In personalized or federated settings, it is key to minimize the number of updateable parameters, while still retaining the capacity to adapt.
Broadly, there are two different approaches to few-shot learning: meta-learning (Hospedales et al., 2020) and transfer learning (fine-tuning)
(Yosinski et al., 2014).
Meta-learning approaches provide methods that have a small number of updatable parameters (Requeima et al., 2019).
However, while meta-learners can perform strongly on datasets that are similar to those they are meta-trained on, their accuracy suffers when tested on datasets that are significantly different (Dumoulin et al., 2021).
Transfer learning algorithms often outperform meta-learners, especially on diverse datasets and even at low-shot (Dumoulin et al., 2021; Tian et al., 2020).
However, the leading Big Transfer (BiT) (Dumoulin et al., 2021; Kolesnikov et al., 2019) algorithm requires every parameter in a large network to be updated.
In summary, performant transfer learners are parameter-inefficient, and parameter-efficient few-shot learners perform relatively poorly.
In this work we propose FiLM Transfer or FiT, a novel method that synthesizes ideas from both the transfer learning and meta-learning communities in order to achieve the best of both worlds – parameter efficiency without sacrificing accuracy, even when there are only a small number of training examples available.
From transfer learning, we take advantage of backbones pretrained on large image datasets and the use of fine-tuned parameter efficient adapters.
From meta-learning, we take advantage of metric learning based final layer classifiers trained with episodic protocols that we show are more effective than the conventional linear layer classifier.
We experiment with F I T on a wide range of downstream datasets and show that it achieves better classification accuracy at low-shot with two orders of magnitude fewer updateable parameters when compared to BiT (Kolesnikov et al., 2019) and competitive accuracy when more data are available.
We also demonstrate the benefits of F I T on a low-shot real-world model personalization application and in a demanding few-shot federated learning scenario.
A summary of our results is shown in
Fig.
1, where we see that F I T has superior parameter efficiency and classification accuracy compared to BiT in multiple settings.
Our contributions:
• A parameter and data efficient network architecture for low-shot transfer learning that (i) utilizes frozen backbones pretrained on large image datasets; (ii) augments the backbone with parameter efficient FiLM (Perez et al., 2018) layers in order to adapt to a new task; and (iii) makes novel use of an automatically configured Naive Bayes final layer classifier instead of the usual linear layer, saving a large number of updateable parameters, yet improving classification performance; • A meta-learning inspired episodic training protocol for low-shot fine-tuning requiring no data augmentation, no regularization, and a minimal set of hyper-parameters; • Superior classification accuracy at low-shot on standard downstream datasets and state-of-the-art results on the challenging VTAB-1k benchmark (74.9% for backbones pretrained on ImageNet-21k) while using ≈ 1% of the updateable parameters when compared to the leading method BiT; • Demonstration of superior parameter efficiency and classification accuracy in distributed low-shot personalization and federated learning applications where model update size is a key performance metric.
We show that the F I T communication cost is more than 3 orders of magnitude lower than BiT (7M versus 14B parameters transmitted) in our CIFAR100 federated learning experiment.
F I LM T RANSFER (F I T)
In this section we detail the F I T algorithm focusing on the few-shot image classification scenario.
Preliminaries We denote input images x ∈ R ch × W × H where W is the width, H the height, ch the number of channels and image labels y ∈ { 1 , . . .
, C } where C is the number of image classes indexed by c .
Assume that we have access to a model f ( x ) =
h ϕ ( b θ ( x )) that outputs class-probabilities for an image p ( y = c
| x , θ , ϕ ) for c = 1 , . . .
, C and is comprised of a feature extractor backbone b θ ( x ) ∈ R d b with parameters θ that has been pretrained on a large upstream dataset such as Imagenet where d b is the output feature dimension and a final layer classifier or head h ϕ ( · ) ∈ R C with weights ϕ .
Let D = { ( x n , y n ) } Nn =1 be the downstream dataset that we wish to fine-tune the model f to.
F I T Backbone For the network backbone, we freeze the parameters θ to the values learned during upstream pretraining.
To enable parameter-efficient and flexible adaptation of the backbone, we add
Feature-wise Linear Modulation (FiLM)
(Perez et al., 2018) layers with parameters ψ at strategic points within b θ .
A FiLM layer scales and shifts the activations a ij arising from the j th channel of a convolutional layer in the i th block of the backbone as FiLM ( a ij , γ ij , β ij ) =
γ ij a ij + β ij , where γ ij and β ij are scalars.
The set of FiLM parameters ψ = { γ ij , β ij } is learned during fine-tuning.
We add FiLM layers following the middle 3 × 3 convolutional layer in each ResNetV2 (He et al., 2016b) block and also one at the end of the backbone prior to the head.
Fig.
A.1a illustrates a FiLM layer operating on a convolutional layer, and Fig.
A.1b illustrates how a FiLM layer can be added to a ResNetV2 network block.
FiLM layers can be similarly added to other backbones.
An advantage of FiLM layers is that they enable expressive feature adaptation while adding only a small number of parameters (Perez et al., 2018).
For example, in a ResNet50 with a FiLM layer in every block, the set of FiLM parameters ψ account for only 11648 parameters which is fewer than 0.05% of the parameters in b θ .
We show in Section 4 that FiLM layers allow the model to adapt to a broad class of datasets.
Fig.
A.2a and Fig.
A.2b show the magnitude of the FiLM parameters as a function of layer for FiT-LDA on CIFAR100 and SHVN, respectively.
Refer to Appendix A.7 for additional detail.
F I T Head For the head of the network, we use a specially tailored Gaussian Naive Bayes classifier.
Unlike a linear head, this head can be automatically configured directly from data and has only a small number of free parameters which must be learned, ideal for few-shot, personalization and federated learning.
We will also show that this head is often more accurate than a standard linear head.
The class probability for a test point x ∗ is:
[Equation]
) where π c = N c N , µ c = 1 N c (cid:80) N c i =1 b θ , ψ ( x i ) , Σ c = 1 N c (cid:80) N c i =1 ( b θ , ψ ( x i ) − µ c )( b θ , ψ ( x i ) − µ c ) T are the maximum likelihood estimates, N c is the number of examples of class c in D , and N ( z | µ, Σ) is a multivariate Gaussian over z with mean µ and covariance Σ .
Estimating the mean µ c for each class c is straightforward and incurs a total storage cost of Cd b .
However, estimating the covariance Σ c for each class c is challenging when the number of examples per class N c is small and the embedding dimension of the backbone d b is large.
In addition, the storage cost for the covariance matrices may be prohibitively high if d b is large.
Here, we use three different approximations to the covariance in place of Σ c in Eq.
(1) (Fisher, 1936; Duda et al., 2012):
• Quadratic Discriminant Analysis (QDA) : Σ QDA = e 1 Σ class
+ e 2 Σ task + e 3 I • Linear Discriminant Analysis (LDA) :
Σ LDA = e 2 Σ task + e 3 I • ProtoNets
(Snell et al., 2017): Σ PN
=
I ; i e. there is no covariance and the class representation is parameterized only by µ c and the classifier logits are formed by computing the squared Euclidean distance between the feature representation of a test point b θ , ψ ( x ∗ ) and each of the class means.
In the above, Σ class is the computed covariance of the examples in class c in D , Σ task is the computed covariance of all the examples in D assuming they arise from a single Gaussian with a single mean, e = { e 1 , e 2 , e 3 } are weights learned during training, and the identity matrix I is used as a regularizer.
QDA mainly serves as a baseline since it has a very large set of updateable parameters arising from the fact that it stores a covariance matrix for each class in the dataset.
LDA is far more parameter efficient than QDA, sharing a single covariance matrix across all classes.
We show that LDA leads to
[Figure]
Updateable
Example
BiT
[Equation]
F I [Equation] - QDA
[Equation]
+21,013,
[Equation]
[Figure]
I T -
[Figure]
[Equation]
|
[Equation]
[Figure]
very similar performance to QDA.
The number of model shared and updateable parameters for the three F I T variants as well as the BiT algorithm are detailed in Table 1.
Refer to Appendix A.2 for details on the parameter calculations.
An empirical justification for the use of the F I T-LDA head is shown in Fig.
2a where it outperforms a linear head in the case of FiLM and when all the backbone parameters are learned.
Refer to Table A. for details.
In Fig.
2b, we see for both datasets, F I T-LDA converges significantly faster than BiT, which uses a linear head.
The primary limitation of the Naive Bayes head is the higher (versus linear) computational cost due to having to invert a d b × d b covariance matrix on each training iteration.
F I T Training We learn the FiLM parameters ψ and the covariance weights e via fine-tuning (parameters θ are fixed from pretraining).
One approach would be to apply standard batch training on the downstream dataset, however it was hard to balance under- and over-fitting using this setup.
Instead, we found that an approach inspired by episodic training (Vinyals et al., 2016) that is often used in meta-learning yielded better performance.
We refer to this approach as episodic fine tuning and it works as follows.
Note that we require ‘training’ data to compute π , µ , Σ to configure the head, and a ‘test’ set to optimize ψ and e via gradient ascent.
Thus, from the downstream dataset D , we derive two sets – D train and D test .
If D is sufficiently large ( |D| ≈ 1000), such that overfitting is not an issue, we set D train =
D test =
D .
Otherwise, in the few-shot scenario, we randomly split D into D train and D test such that the number of examples or shots in each class c are roughy equal in both partitions and that there is at least one example of each class in both.
Refer to Algorithm A. for details.
For each training iteration, we sample a task τ consisting of a support set D τS drawn from D train with S τ examples and a query D τQ set drawn from D test with Q τ examples.
First, D τS is formed by randomly choosing a subset of classes selected from the range of available classes in
D train .
Second, the number of shots to use for each selected class is randomly selected from the range of available examples in each class of D train with the goal of keeping the examples per class as equal as possible.
Third, D τQ is formed by using the classes selected for D τS and all available examples from D test in those classes up to a limit of 2000 examples.
Refer to Algorithm A.2 for details.
Episodic fine-tuning is crucial to achieving the best classification accuracy with the Naive Bayes head.
If all of D train and D test are used for every iteration, overfitting occurs, limiting accuracy (see
Table A.2 and Table A.4).
The support set D τS is then used to compute π , µ , and Σ
and we then use
D Q = {{ x τ ∗ q , y τ [Equation]
q } Q τ q [Equation] 1 } Tτ =1 to train ψ and e with maximum likelihood.
We optimize the following:
[Equation]
F I T training hyper-parameters include a learning rate, |D τS | , and the number of training iterations.
For the transfer learning experiments in Section 4 these are set to constant values across all datasets and do not need to be tuned based on a validation set.
We do not augment the training data.
In the 1-shot case, we do not perform episodic fine-tuning and leave the FiLM parameters at their initial value of γ = 1 , β = 0 and e = (0 . 5 , 0 . 5 , 1 . 0) and predict as described next.
In Section 4 we show this can yield results that better those when augmentation and training steps are taken on 1-shot data.
F I T Prediction Once the FiLM parameters ψ and covariance weights e have been learned, we use
D for the support set to compute π c , µ c , and Σ c for each class c
and then Eq.
(1) can be used to make a prediction for any unseen test input.
R ELATED W ORK
We take inspiration from residual adapters (Rebuffi et al., 2017; 2018) where parameter efficient adapters are inserted into a ResNet with frozen pretrained weights.
The adapter parameters and the final layer linear classifier are then learned via fine-tuning.
More recently, a myriad of additional parameter efficient adapters have been proposed including FiLM, Adapter (Houlsby et al., 2019),
LoRA (Hu et al., 2021), VPT (Jia et al., 2022), AdaptFormer (Chen et al., 2022), NOAH (Zhang et al., 2022), Convpass (Jie & Deng, 2022), (Mudrakarta et al., 2019), and CaSE (Patacchiola et al., 2022).
For F I T we use FiLM as it is the most parameter efficient adapter, yet it allows for expressive adaptation, and can be used in various backbone architectures including ConvNets and Transformers.
To date, transfer learning systems that employ adapters use a linear head for the final classification layer.
In meta-learning systems it is common to use metric learning heads (e.g. ProtoNets (Snell et al., 2017)), which have no or few learnable parameters.
Meta-Learning systems that employ a metric learning head are normally trained with an episodic training regime (Vinyals et al., 2016).
Some of these approaches (e.g. TADAM (Oreshkin et al., 2018), FLUTE (Triantafillou et al., 2021), and
Simple CNAPs (Bateni et al., 2020) use both a metric head and FiLM layers to adapt the backbone.
F I T differs from all of the preceding approaches by using a powerful Naive Bayes metric head that uses episodic fine-tuning in the context of transfer learning, as opposed to the usual meta-learning.
We show in Fig.
2a and Section 4 that the episodically fine-tuned Naive Bayes head consistently outperforms a conventional batch trained linear head in the low-shot transfer learning setting.
E XPERIMENTS
In this section, we evaluate the classification accuracy and updateable parameter efficiency of F I T in a series of challenging benchmarks and application scenarios.
In all experiments, we use Big Transfer (BiT) (Kolesnikov et al., 2019), a leading, scalable, general purpose transfer learning algorithm as a point of comparison.
First, we compare different variations of F I T to BiT on several standard downstream datasets in the few-shot regime.
Second, we evaluate F I T against BiT on VTAB-1k (Zhai et al., 2019), which is arguably the most challenging transfer learning benchmark.
Additionally, we compare F I T to the latest vision transformer based methods that have reported the highest accuracies on VTAB-1k to date.
Third, we show how F I T can be used in a personalization scenario on the ORBIT (Massiceti et al., 2021) dataset, where a smaller updateable model is an important evaluation metric.
Finally, we apply F I T to a few-shot federated learning scenario where minimizing the number of parameter updates and their size is a key requirement.
Training and evaluation details are in Appendix A.11.
Source code for experiments can be found at: https://github.com/cambridge-mlg/fit
. 4. F EW - SHOT R ESULTS
Fig.
3 shows the classification accuracy as a function of updateable parameters for F I T-LDA and BiT on four downstream datasets (CIFAR10, CIFAR100 (Krizhevsky et al., 2009), Pets (Parkhi et al., 2012), and Flowers (Nilsback & Zisserman, 2008)) that were used to evaluate the performance of BiT (Kolesnikov et al., 2019).
Table A.1 contains complete tabular results with additional variants of F I T and BiT. All methods use the BiT-M-R50x1 (Kolesnikov et al., 2019) backbone that is pretrained on the ImageNet-21K (Russakovsky et al., 2015) dataset.
The key observations from Fig.
3 are:
• For ≤ 10 shots (except 1-shot on CIFAR100), F I T-LDA outperforms BiT, often by a large margin. [Figure] u r a c
y [Figure] • On 3 out of 4 datasets, F I T-LDA outperforms BiT even when all of D is used for fine-tuning.
• F I T-LDA outperforms BiT despite BiT having more than 100 times as many updateable parameters.
• To avoid overfitting when D is small, Table A.2 indicates that it is better to split D into two disjoint partitions D train and D test and that D τS and D τQ should be randomly sub-sampled as opposed to using all of the data in each training iteration.
The datasets used in this section were similar in content to the dataset used for pretraining and the performance of F I T-QDA and F I T-LDA was similar, indicating that the covariance per class was not that useful for these datasets.
In the next section, we test on a wider variety of datasets, many of which differ greatly from the upstream data.
4. VTAB-1 K R ESULTS
The VTAB-1k benchmark (Zhai et al., 2019) is a low to medium-shot transfer learning benchmark that consists of 19 datasets grouped into three distinct categories (natural, specialized, and structured).
From each dataset, 1000 examples are drawn at random from the training split to use for the downstream dataset D .
After fine-tuning, the entire test split is used to evaluate classification performance.
Table 2 shows the classification accuracy and updateable parameter count for the three variants of F I T and BiT (see Table A.3 for error bars).
The key observations from our results are:
• Both F I T-QDA and F I T-LDA outperform BiT on VTAB-1k.
• The F I T-QDA variant has the best overall performance, showing that the class covariance is important to achieve superior results on datasets that differ from those used in upstream pretraining (e.g. the structured category of datasets).
However, the updateable parameter cost is high.
• F I T-LDA utilizes two orders of magnitude fewer updateable parameters compared to BiT, making it the preferred approach.
Table
A.4 shows that it is best to use all of D for each of D train and D test (i e. no split) and that D τS and D τQ should be episodically sub-sampled as opposed to using all of the data in each iteration. [Table] [Table]
O
[Table]
ll
[Table]
cc ↑
C
[Table]
lt ec h
C
I F
A R
F l o w e r s
P e t s
[Table]
un
[Table]
VHN
D
[Table]
D
[Table]
u r o
[Table]
A
T
R e s i c
[Table]
C a m e l yon
R e ti nop a t hy
C l e v r-
c oun t
C l e v r-
d i s t d S p r it e sl o c d S p r it e so r
i s NO RB - az i s NO [Table] Table 3 shows that F I T-LDA achieves state-of-the-art classification accuracy when compared to leading transfer learning methods pretrained on ImageNet-21k
[Table]
while requiring the smallest number of updateable parameters and using the smallest backbone.
All competing methods use a linear head
[Table]
[Table]
[Table]
P ERSONALIZATION
In the personalization experiments, we use ORBIT (Massiceti et al., 2021), a real-world few-shot video dataset recorded by people who are blind/low-vision.
A user collects a series of short videos of objects that they would like to recognize.
The collected videos and associated labels are then uploaded to a central service to train a personalized classification model for that user.
Once trained, the personalized model is downloaded to the user’s smartphone.
In this setting, models with a smaller number of updateable parameters are preferred in order to save model storage space on the central server and in transmitting any updated models to a user.
The personalization is performed by taking a large pretrained model and adapting it using user’s individual data.
We follow the object recognition benchmark task proposed by the authors, which tests a personalized model on two different video types: clean where only a single object is present and clutter where that object appears within a realistic, multi-object scene.
In Table 4, we compare F I T-LDA to several competitive transfer learning and meta-learning methods that benchmark on the ORBIT dataset.
We use the LDA variant of F I T, as it achieves higher accuracy in comparison to the ProtoNets variant, while using far fewer updateable parameters than QDA.
As baselines, for transfer learning, we include FineTuner (Yosinski et al., 2014), which freezes the weights in the backbone and fine-tunes only the linear classifier head on an individual’s data.
For meta-learning approaches, we include ProtoNets (Snell et al., 2017) and Simple CNAPs (Bateni et al., 2020), which are meta-trained on Meta-Dataset (Dumoulin et al., 2021).
In the lower part of Table 4, we compare F I T-LDA and BiT.
Both models use the same BiT-M-R50x1 backbone pretrained on
ImageNet-21K. Training and evaluation details are in Appendix A.11.2.
For this comparison, we show frame and video accuracy, averaged over all the videos from all tasks across all test users ( 17 test users, 85 tasks in total).
We also report the number of shared and individual updateable parameters required to be stored or transmitted.
The key observations from our results are:
• F I T-LDA outperforms competitive meta-learning methods, Simple CNAPs and ProtoNets.
• F I T-LDA also outperforms FineTuner in terms of the video accuracy and performs within error bars of it in terms of the frame accuracy.
• The number of individual parameters for F I T-LDA is far fewer than in Simple CNAPs and BiT, and is of the same order of magnitude as FineTuner and ProtoNets.
• F I T-LDA pretrained on ImageNet-21K performs on par with BiT, while having orders of magnitude fewer updatable parameters.
4. F EW - SHOT F EDERATED L EARNING
We now show how F I T can be used in the few-shot federated learning setting, where training data are split between client nodes, e.g. mobile phones or personal laptops, and each client has only a handful of samples.
Model training is performed via numerous communication rounds between a server and clients.
In each round, the server selects a fraction of clients making updates and then sends the current model parameters to these clients.
Clients update models locally using only their personal data and then send their parameter updates back to the server.
Finally, the server aggregates information from all clients, updates the shared model parameters, and proceeds to the next round until
[Table]
convergence. In this setting, models with a smaller number of updateable parameters are preferred in order to reduce the client-server communication cost which is typically bandwidth-limited.
The configuration and training protocol of F I T help significantly reduce communication cost compared to standard fine-tuning protocols, as i) only FiLM layers are transferred at each communication round, ii) in contrast to standard linear head, the Naive Bayes head is not transferred and is constructed locally for each client.
Table 5 shows communication parameters savings of F I T compared to BiT.
Experiments We use CIFAR100 (Krizhevsky et al., 2009), a relatively large-scale dataset compared to those commonly used to benchmark federated learning methods (Reddi et al., 2021; Shamsian et al., 2021).
We employ the basic FedAvg (McMahan et al., 2016) algorithm, although other algorithms can also be used (Table A.7).
We train all models for 60 communication rounds, with 5 clients per round and 10 update steps per client.
Each client has 10 classes, which are sampled randomly before the start of training.
We use a ProtoNets head for simplicity, although QDA and LDA heads could also be used.
More specific training and evaluation details are in Appendix A.11.3.
We evaluate F I T in two scenarios, global and personalized.
In the global setting, the aim is to construct a global classifier and report accuracy on the CIFAR100 test set.
We assume that the server knows which classes belong to each client, and constructs a shared classifier by taking a mean over prototypes produced by clients for a particular class.
In the personalized scenario, we test a personalized model on test classes present in the individual’s training set and then report the mean accuracy over all clients.
As opposed to the personalization experiments on ORBIT, where a personalized model is trained using only the client’s local data, in this experiment we initialize a personalized model with the learned global FiLM parameters and then construct a ProtoNets classifier with an individual’s data.
Thus, the goal of the personalized setting is to estimate how advantageous distributed learning can be for training FiLM layers to build personalized few-shot models.
To test the global scenario, we compare F I T to BiT. As the training protocol used in BiT cannot be directly applied in the federated learning context, we trained BiT with a constant learning rate.
We do not provide comparison on the personalized scenario, as BiT uses a linear classification head trained globally, as opposed to using only client’s local data.
As, to the best of our knowledge, there are no suitable federated learning systems to compare to, we define baselines which form an upper and lower bounds on the model performance.
For the global scenario, we take a F I T model trained centrally on all available data as the upper bound baseline.
To get the lower bound baseline, we train a F I T model for each client with their local data, then average FiLM parameters of these individual models and construct a global ProtoNets classifier using the resulting FiLM parameters.
The upper bound is therefore standard batch training, the performance of which we hope federated learning can approach.
The lower bound is a simplistic version of federated learning with a single communication round which federated averaging should improve over.
For the personalized setting, the upper bound baseline is as in the global scenario from which we form a personalized classifier by taking a subset of classes belonging to a client from a global 100 -way classifier.
The lower bound baseline is set to a F I T model trained for each client individually.
The upper bound is again standard batch training and the lower bound is derived from locally trained models which do not share information and therefore should be improved upon by federated learning.
Results Table 5 shows the comparison of F I T to BiT. Fig.
4 shows global and personalized classification accuracy as a function of communication cost for different numbers of clients and shots per
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
client.
By communication cost we mean the number of parameters transmitted during training.
The key observations from our results are:
• F I T and BiT show comparable performance in terms of test accuracy, however F I T is much more parameter efficient and requires transmission of only 7 M parameters through training, in contrast to 14 B parameters for BiT.
This makes F I T highly suitable for federated learning applications.
• In the global setting, the federated learning model is only slightly worse ( 3 − 5% ) than the upper bound baseline, while outperforming the lower bound model, often by a large margin.
This shows that F I T can be efficiently used in federated learning settings with different configurations.
• In the personalized scenario, for a sufficient number of clients ( ≥ 50 ) the gap between the federated learning model and the upper bound model is significantly reduced with the increase in number of shots.
Federated training strongly outperforms the lower bound baseline, surprisingly even in the case of 10 clients with disjoint classes.
This provides empirical evidence that collaborative distributed training can be helpful for improving personalized models in the few-shot data regime.
In Appendix A.8, we show that distributed training of a F I T model can be efficiently used to learn from more extreme, non-natural image datasets like Quickdraw (Jongejan et al., 2016).
D ISCUSSION
In this work, we proposed F I T, a parameter and data efficient few-shot transfer learning system that allows image classification models to be updated with only a small subset of the total model parameters.
We demonstrated that F I T can outperform BiT using fewer than 1% of the updateable parameters and achieve state-of-the-art accuracy on VTAB-1k.
We also showed the parameter efficiency benefits of employing F I T in model personalization and federated learning applications. A CKNOWLEDGMENTS
Aliaksandra Shysheya, John Bronskill, Massimiliano Patacchiola and Richard E. Turner are supported by an EPSRC Prosperity Partnership EP/ T005386/1 between the EPSRC, Microsoft Research and the University of Cambridge. This work has been performed using resources provided by the Cambridge
Tier-2 system operated by the University of Cambridge Research Computing Service https: //www.hpc.cam.ac.uk funded by EPSRC Tier-2 capital grant EP/P020259/1.
We thank the anonymous reviewers for key suggestions and insightful questions that significantly improved the quality of the paper.
We also thank Aristeidis Panos and Siddharth Swaroop for providing helpful comments and suggestions.
Reproducibility Statement Source code for experiments can be found at: https://github. com/cambridge-mlg/fit . The README file details the data preparation steps and includes the command lines to configure and run the experiments.
Appendix A.11 details the training and evaluation procedures and hyperparameter settings for all of the experiments including few-shot and VTAB-1k transfer learning experiments, personalization on ORBIT experiments, and the federated learning experiments.
A. M ODEL P ARAMETERS
In this section, we provide details on the updateable parameter calculations for the LDA and QDA variants of BiT. Refer to Table 1.
For F I T-QDA and F I T-LDA, the means and covariances contribute to the updateable parameter count.
We use a mean for every class which contributes Cd b updateable parameters.
A covariance matrix has d b × d b values, however it can be represented in Cholesky factorized form which results in a lower triangular matrix and thus can be represented with d b ( d b + 1) / 2 values, with the rest being zeros.
In the case of F I T-LDA, where the covariance matrix is shared across all classes, a more compact representation is possible, resulting in considerable savings in updateable parameters: p ( [Equation]
θ , [Equation] [Equation] ) , π , µ , Σ LDA )
[Equation]
From Eq.
(A.1)
[Equation]
it follows that to compute the probability of classifying a test point x ∗
[Equation]
we need to store µ Tc Σ − 1 LDA which has dimensionality d b and µ Tc Σ − 1 LDA µ c which has dimensionality 1 for each class c , resulting in only C ( d b + 1) parameters required for the F I T-LDA head.
Since the covariance matrix is not shared in the case of F I T-QDA, no additional savings are possible in that case.
A. VTAB-1 K D ATASETS
The VTAB-1k benchmark (Zhai et al., 2019) is a low to medium-shot transfer learning benchmark that consists of 19 datasets grouped into three distinct categories (natural, specialized, and structured).
The natural datasets are: Caltech101 (Fei-Fei et al., 2006), CIFAR100 (Krizhevsky et al., 2009),
Flowers102 (Nilsback & Zisserman, 2008), Pets (Parkhi et al., 2012), Sun397 (Xiao et al., 2010),
SVHN (Netzer et al., 2011), and DTD (Cimpoi et al., 2014).
The specialized datasets are: EuroSAT (Helber et al., 2019), Resics45 (Cheng et al., 2017),Patch
Camelyon (Veeling et al., 2018), and Retinopathy (Kaggle & EyePacs, 2015).
The structured datasets are: CLEVR-count (Johnson et al., 2017), CLEVR-dist (Johnson et al., 2017), dSprites-loc (Matthey et al., 2017),
dSprites-ori
(Matthey et al., 2017), SmallNORB-azi (LeCun et al., 2004), SmallNORB-elev (LeCun et al., 2004), DMLab (Beattie et al., 2016), KITTI-dist (Geiger et al., 2013).
A. A DDITIONAL R ESULTS
This section contains additional results that would not fit into the main paper, including tabular versions of figures.
Note, in some results, we use a new metric Relative Model Update Size or RMUS, which is the ratio of the number of updateable parameters in one model to another.
In our experiments, we measure RMUS relative to the number of updatable parameters in BiT model.
A.4. A DDITIONAL F EW - SHOT R ESULTS
Table A.1 shows the tabular version of Fig.
In addition, Table A.1 includes results for an additional variant of BiT (BiT-FiLM), and two additional variants of F I T (F I T-QDA and F I T-ProtoNets).
BiT-FiLM is a variant of BiT that uses the same training protocol as the standard version of BiT, but the backbone weights θ are frozen and FiLM layers are added in the same manner as F
I T.
The FiLM parameters ψ and the linear head weights ϕ are learned during training.
The results are shown in
Table A.1 and the key observations are:
• In general, at low-shot, the standard version of BiT outperforms BiT-FiLM.
However, as the shot increases, especially when training on all of D , BiT-FiLM is equal in classification accuracy.
• The above implies that FiLM layers have sufficient capacity to accurately fine-tune to downstream datasets, but the F I T head and training protocol are needed to achieve superior results.
• While the accuracy of F I T-QDA and F I T-LDA is similar, the storage requirements for a covariance matrix for each class makes QDA impractical if model update size is an important consideration.
• The accuracy of F I T-ProtoNets is slightly lower than F I T-LDA, but often betters BiT, despite BiT having more than 100 times as many updateable parameters. [Table]
[Table]
A.4. F EW - SHOT T ASK A BLATIONS
Table A.2 shows the few-shot results for all three variants of F I T with different ablations on how the downstream dataset D is allocated during training.
No Split indicates that D train is not split into two disjoint partitions and D train =
D test =
D .
However, D train and D test are sampled to form episodic fine-tuning tasks as detailed in Algorithm A.2.
Split indicates that D train is split into two disjoint partitions as detailed in Algorithm A.1 and then sampled into tasks as described in
Algorithm A.2.
Use All indicates that D train =
D test =
D (i.e. D is not split) and that D train and
D test are not sampled and that D τ S
= D τ Q = D for all tasks τ .
Table A.2 shows that Use All is consistently the worst option.
In general, in the few-shot case, Split either outperforms No Split (CIFAR10, Pets) or achieves the same level of performance (CIFAR100,
Flowers102).
As a result, we use the Split option when reporting the few-shot results.
[Table]
A. A DDITIONAL VTAB-1 K R ESULTS
Table A.3 depicts a more comprehensive version of Table 2 including error bars and RMUS for each dataset.
A.5. VTAB-1 K T ASK A BLATIONS
Table A.4 shows the VTAB-1k results for all three variants of F I T with different ablations on how the downstream dataset D is allocated during training.
Refer to Appendix A.4.2 for the meanings of No Split , Split , and Use All .
With some minor exceptions, the Use All case performs the worst.
The performance of the No Split and Split options is very close, with No Split being slightly better when averaged over all of the datasets.
As a result, we use the No Split option when reporting the
VTAB-1k results.
Table A.5 compares the classification accuracy of the LDA and linear heads as a function of the learnable parameters in the backbone.
When all parameters in the backbone are learnable and when
FiLM layers are employed, the Naive Bayes LDA head outperforms the linear head.
In the case when all of the parameters in the backbone are frozen, the linear head is superior since in that case the LDA head has only 2 learnable parameters (the covariance weights e ).
[Table]
sign indicates the 95% confidence interval over 3 runs.
Bold type indicates the highest scores
[Table] [Table]
A. D ISCUSSION OF THE EMPIRICAL RESULTS
FiLM works well for low shots per class because it has a small number of parameters to adapt – this gets you a long way with large pre-trained backbones – and it also prevents over-fitting as compared to adapting all the parameters in the backbone.
Evidence: eventually as dataset size increases, full body adaptation is at least as good as FiLM.
For larger models, this transition happens at a larger number of data points and so we expect adapters to be especially useful as the field transitions to using large foundation models.
Fig.
3 provides an empirical justification of this argument, showing that FiT-LDA outperforms BiT at low-shot.
The linear head is more flexible and works best of all methods in high data, but in the low- to medium-shot setting the linear head appears to lead to increased overfitting and the meta-trained LDA head performs best in this region.
See Fig.
3 for empirical justification.
The F I T approach suffers when there are a fairly large number of data points and the dataset is very far from the pretraining data (ImageNet) e.g. see the DSprites results in Table 3 where BiT beats
FiT-LDA by a large margin.
In addition, the LDA and QDA variants of the Na¨ıve Bayes head are more computationally expensive compared to a linear head since LDA and QDA require inverting d b × d b matrices for each training iteration.
A. F I LM L AYER P ARAMETER V ARIATION
Fig.
A.2a and Fig.
A.2b show the magnitude of the FiLM parameters as a function of layer for
FiT-LDA on CIFAR100 and SHVN, respectively, in the VTAB-1k setting.
We see that for a dataset that differs from the pretraining data (SHVN), the FiLM layers are required to learn a greater degree of adjustment compared to a dataset that is similar to the pretraining data (CIFAR100). https://eprint.iacr.org/2017/715.pdf
A. A DDITIONAL F EW - SHOT F EDERATED L EARNING R ESULTS
[Table]
Table A.6 shows the tabular version of Fig. 4.
For the Federated Learning results, it includes only the resulting accuracy after training for 60 communication rounds.
Refer to Section 4.4 for analysis.
In general, F I T can operate well with other FL aggregation methods, as they are agnostic to the neural network architectures used to learn the parameters.
However, as we are constructing a Na¨ıve Bayes head instead of training a linear head via SGD, some of the theoretical convergence results of the methods mentioned may not hold.
Despite this fact, F I T has shown to have quite fast convergence in practice.
In Table A.7 we provide the results of training F I T with Federated Averaging (FedAvg) and
FedProx(Li et al., 2020).
[Figure] [Figure]
[Figure] [Figure]
To test distributed training of a F I T model on a more extreme, non-natural image dataset, we also include the results for federated training of F I T on the Quickdraw dataset.
As there is no pre-defined train/test split for the Quickdraw dataset [Figure] we randomly choose 100 samples from each of theclasses and use them for testing.
We train all federated training models for 120 communication rounds, with 5 clients per round
[Figure]
and 10 update steps per client.
Since Quickdraw is a more difficult dataset than CIFAR100, it requires more communication rounds for training.
Each client has 35 classes, which are sampled randomly at the start of training.
In our experiments, we omit the 10 -clients case, as the overall amount of data in the system is not enough to even train a robust global upper bound baseline model [Figure]
Fig.
A.3 shows global and personalized classification accuracy as a function of communication cost for different numbers of clients and shots per client for Quickdraw, while Table A.9 shows the tabular version of this figure.
Similarly to CIFAR100 experiments, we compare F I T to BiT on the Quickdraw dataset.
We train both models for 60 communication rounds.
The results are presented in Table A.8.
For 5 andshots per class, F I T performs similarly to BiT for all numbers of clients tested.
However, for 2 -shot,
BiT outperforms F I T for all numbers of clients tested, but the performance gap is reduced as the number of clients is increased.
We attribute this behavior to the use of the ProtoNets classifier.
During training at each client’s optimization step, we construct a N-way local ProtoNets classifier using only one image per class, which may result in a classifier not robust enough for reasonable optimization of the FiLM layers.
This hypothesis is also supported by Fig.
A.3, where we see a huge gap between
[Table] [Table]
the performance of Federated Learning and centralized learning (the upper bound) for 2 -shot.
In contrast, the linear head in BiT is not local and is trained using all clients’ data, thus avoiding this pitfall.
This leads us to the following observation – if there is enough local client data to construct a robust metric-based classifier, then Na¨ıve Bayes head helps to significantly reduce communication cost without sacrificing the final model quality.
However, if there is not enough local client data, then the use of a linear classification head may be more appropriate.
[Figure] [Figure]
[Table]
A. C ONNECTIONS TO P ERSONALIZED F EDERATED L EARNING
Partial model based personalized federated learning (PFL)
(Collins et al., 2021; Arivazhagan et al., 2019; Liang et al., 2020; Pillutla et al., 2022) is related to the personalized setting in our federated learning experiments, but it is different in important regards.
With regard to the similarities, the local
Na¨ıve Bayes head parameters in our approach may be considered as “personal” parameters (there are personalized parameters for each client), while the FiLM parameters could be viewed as “shared” parameters.
However, there are a few major differences between the ideas:
• The personalized parameters in our setting, i.e. the ProtoNets head, does not require an optimization loop to be learned.
This simplifies deployment significantly.
• The partial model personalization literature (Collins et al., 2021; Arivazhagan et al., 2019; Liang et al., 2020; Pillutla et al., 2022) is more concerned with proposing stable federated training algorithms that would work within heterogeneous settings, where clients have diverse data and standard FedAvg algorithm would fail, thus necessitating a need to introduce ‘personal’ parameters.
In contrast, personalized heads are required in our setup as each user has a different classification task to perform.
Moreover, we propose a particular architecture that would be highly suitable for federated learning applications, as the number of parameters required to be transmitted is small, while most methods in PFL propose architecture agnostic optimization algorithms.
Our model can then be trained with an arbitrary Federated Learning algorithm.
• Another distinctive difference between the model personalization federated learning literature and our work is that in the former most of the methods train all model’s parameters, while in our work we used a deep pretrained network and fine-tune only the FiLM layers.
A. F I T T RAINING A LGORITHMS
Algorithm A.1 and Algorithm A.2 detail how episodic fine-tuning tasks are split and sampled, respectively, for use in the F I T training protocol.
Algorithm A.
Splitting the downstream dataset D
Require:
D = { ( x n , y n ) } Nn =1 = { x , y } : downstream dataset
Require: unique() ≡ function that returns a list of unique classes and list of counts of each class
Require: select by class () ≡ function that extracts samples of a specified class from a dataset 1: procedure SPLIT ( D ) 2:
D train ← [ ] ▷ Create an empty list to hold D train 3:
D test ← [ ] ▷ Create an empty list to hold D test 4: classes, class counts ← unique ( y ) 5: for all c ∈ classes do 6: assert ( class counts ( c ) > 1)
▷ Require a minimum of 2 shots per class.
7: train count ← ceil ( class counts ( c ) / 2) 8:
D c ← select by class ( c ) ▷ Select examples of class c from D 9:
D train ← D train + D c [: train count ]
▷ Add train count examples to D train 10:
D test ← D test + D c [ train count :]
▷ Add remaining examples to D test 11: end for 12: return D train , D test 13: end procedure
A. T RAINING AND E VALUATION D ETAILS
In this section, we provide implementation details for all of the experiments in Section 4.
A.11. F EW - SHOT AND VTAB-1 K T RANSFER L EARNING E XPERIMENTS F I T
All of the F I T few-shot and VTAB-1k transfer learning experiments were carried out on a single NVIDIA A100 GPU with 80GB of memory.
The Adam optimizer (Kingma & Ba, 2015) with a constant learning rate of 0.0035, for 400 iterations, and |D τS | =100 was used throughout.
No data augmentation was used and images were scaled to 384 × 384 pixels unless the image size was 32 × 32 pixels or less, in which case the images were scaled to 224 × 224 pixels.
In each iteration of episodic fine tuning, we perform a single step of gradient ascent on D τQ .
These hyper-parameters were empirically derived from a small number of runs.
F I T-QDA, F I T-LDA, and F I T-ProtoNets take approximately 12, 10, and 9 hours, respectively, to fine-tune on all 19 VTAB datasets and 5, 3, and 3 hours, respectively, to fine tune all shots on thelow-shot datasets.
For the F I T-LDA results using the EfficientNetV2-M backbone in Table 3, we used 1000 iterations instead of 400 and ran the experiments on 4 NVIDIA A100 GPUs, each with 80GB of memory.
Require: D train = { ( x s , y s ) } S
τ s =1 = { x S , y S } : train portion of downstream dataset
Require: D test = { ( x q , y q ) } Q τ q =1 = { x Q , y Q } : test portion of downstream dataset
Require: support set size : size of the support set |D τS |
Require: unique() ≡ function that returns a list of unique classes and list of counts of each class
Require: randint( min, max ) ≡ function that returns a random integer between min and max
Require: choice ( range, count ) ≡ function that returns a random list of count integers from range
: procedure SAMPLE TASK
( D train , D test , support set size ) 2:
D τS ← [ ] ▷ Create an empty list to hold D τS 3:
D τQ ← [ ] ▷ Create an empty list to hold D τQ 4: train classes, train class counts ← unique ( y S ) 5: test classes, test class counts ← unique ( y Q ) 6: min way ← min ( len ( train classes ) , 5) 7: max way ← min ( len ( train classes ) , support set size ) 8: way ← randint ( min way , max way )
▷ Classification way to use for this task 9: selected classes ← choice ( train classes , way )
▷ List of classes to use in this task 10: balanced shots = max ( round ( support set size / len ( selected classes )), 1) 11: max test shots ← max (1 , floor (2000 / way )) 12: for all c ∈ selected classes do 13: class shots ← train class counts ( c ) 14: shots to use ← min ( class shots , balanced shots ) 15: selected shots ← choice ( class shots , shots to use )
▷ Support shot list 16:
D τS ← D τS + D train [ selected shots ] ▷ Add examples to D τS 17: class shots ← test class counts ( c ) 18: shots to use ← min ( class shots , max test shots ) 19: selected shots ← choice use
▷ shot list 20:
D τQ ← D τQ + D test [ selected shots ] ▷ Add examples to D τQ 21: end for 22: return D τS , D τQ 23: end procedure
BiT For the BiT few-shot experiments, we used the code supplied by the authors (Kolesnikov et al., 2020) with minor augmentations to read additional datasets.
The BiT few-shot experiments were run on a single NVIDIA V100 GPU with 16GB.
For the BiT VTAB-1k experiments, we used the three fine-tuned models for each of the datasets that were provided by the authors (Kolesnikov et al., 2020).
We evaluated all of the models on the respective test splits for each dataset and averaged the results of the three models.
The BiT-HyperRule (Kolesnikov et al., 2019) was respected in all runs.
These experiments were executed on a single
NVIDIA GeForce RTX 3090 with 24GB of memory.
A.11. P ERSONALIZATION ON ORBIT E XPERIMENTS
The personalization experiments were carried out on a single NVIDIA GeForce RTX 3090 with 24GB of memory.
It takes approximately 5 hours to train F I T-LDA personalization models for all the
ORBIT (Massiceti et al., 2021) test tasks.
We derived all hyperparameters empirically from a small number of runs.
We used the ORBIT codebase 1 in our experiments, only adding the code for splitting test user tasks and slightly modifying the main training loop to make it suitable for F I T training.
In the comparison of F I T to standard benchmarks on ORBIT (upper part of Table 4), all methods use an EfficientNet-B0 ( d b = 1280 ) as the feature extractor and an image size of 224 × 224 .
F I T-LDA,
FineTuner (Yosinski et al., 2014) and Simple CNAPs (Bateni et al., 2020) use a backbone pretrained on ImageNet (Deng et al., 2009), while ProtoNets (Snell et al., 2017) meta-trained the weights of the feature extractor on Meta-Dataset (Dumoulin et al., 2021).
The task encoder in Simple CNAPs (Bateni et al., 2020) is meta-trained on Meta-Dataset.
The results for all models (FineTuner (Yosinski et al., 2014), Simple CNAPs (Bateni et al., 2020) and ProtoNets (Snell et al., 2017)) are from (Bronskill et al., 2021).
FiLM layers in F I T-LDA are added to the feature extractor as described in Section 2, resulting in | ψ | = 20544 .
For the comparison of F I T and BiT (lower part of Table 4), we use the BiT-M-R50x1 backbone pretrained on ImageNet-21K. BiT was trained using the BiT-HyperRule (Kolesnikov et al., 2019) on every user task.
For experiments with BiT, we used the code supplied by the authors (Kolesnikov et al., 2019) with minor augmentations to read the ORBIT dataset.
We follow the task sampling protocols described in (Massiceti et al., 2021), and train the F I T model for 50 optimization steps using the Adam optimizer with a learning rate of 0 .
007 for EfficientNet-B and 0 .
003 for BiT-M-R50x1.
The ORBIT test tasks have a slightly different structure in comparison to standard few-shot classification tasks, so in Algorithm A.3 we provide a modified version of the data splitting for the classifier head construction.
In particular, each test user has a number of objects (classes) they want to recognize, with several videos recorded per object.
Each video is split into clips, consecutive 8 -frame parts of the video.
A user test task is comprised of these clips, randomly sampled from different videos of the user’s objects, and associated labels.
Since clips sampled from the same video can be semantically similar, we split the test task so that clips from the same video can only be in either the support or query set, except for the cases when there is only one video of an object available.
A.11. F EDERATED L EARNING E XPERIMENTS
For each local update a new Adam optimizer is initialized.
In each communication round, 5 clients are randomly chosen for making model updates.
All of the federated learning experiments were carried out on a single NVIDIA A100 GPU with 80GB of memory.
In all experiments we use F I T with the
BiT-M-R50x1 (Kolesnikov et al., 2019) backbone pretrained on the ImageNet-21K (Russakovsky et al., 2015) dataset and a ProtoNets head.
We derive all hyperparameters empirically from a small number of runs.
BiT As the training protocol proposed in BiT cannot be directly applied to the federated learning setting, we simplify it and train the BiT model for 60 communication rounds with a constant learning rate of 0 . 003 .
Require: D : downstream dataset; D = {D c } Cc =1 , where C is the number of classes in test task, D c is data of class c ; D c = { V ci } n c i =1 , where n c is the number of videos in class c , V ci is the set of clips from i th video of class c ; V ci = { ( x cij , c ) } n ci j =1 , where n ci is the number of clips in i th video of class c , x cij is the j th clip from video V ci
Require: batch size : size of context split
Require: choose( n, m ) ≡ function that randomly samples m different integers from a set { i } ni =
Require: select by index ( D , i ) ≡ function that extracts samples of indices i from a dataset D
Require: diff ( a, b ) ≡ function that computes set difference between sets a and b
Require: range ( n ) ≡ function that returns a set of values { i } ni = 1: procedure SPLIT ORBIT TASK ( D ) 2:
D train ← [ ] 3:
D test ← [ ] 4: num clips ← floor ( batch size /C ) 5: for c ← 1 to C do 6: num context videos ← ceil ( n c / 2) 7: context videos indices ← choose ( n c , num context videos ) 8: num clips per video ← floor ( num clips / num context videos ) 9: if n c = 1 then 10: context clips indices ← choose ( n c 1 , num clips per video ) 11: target clips indices ← diff ( range ( n c 1 ) , context clips indices ) 12:
D train ← D train + select by index ( V c 1 , context clips indices ) 13:
D test ← D test + select by index ( V c 1 , target clips indices ) 14: else 15: for j ← context videos indices do 16: context clips indices ← choose ( n cj , num clips per video ) 17:
D train ← D train + select by index
18: end for 19: for j ← do 20:
D ← D V cj 21: end for 22: end if 23: end for 24: return D
D
25: end procedure CIFAR100 We train all federated learning models with different number of clients and shots per client for 60 communication rounds, with 5 clients per round and 10 update steps per client.
Each client has 10 classes, which are sampled uniformly before the start of training.
In our experiments, we used 2 , 5 or 10 shot per class.
For the global setting, we used the CIFAR100 test split for the global model evaluation.
For the personalized setting, we test each client using the CIFAR100 test split, but using only the classes that a particular client owns.
This results in 10 -class classification task.
We then report the average accuracy across all clients.
We use a learning rate of 0 .
003 at the start of the training, decaying it by 0 .
3 every 20 communication rounds.
Upper and lower bound baselines for both the global and personalized scenarios were trained for 400 epochs using the Adam optimizer with a constant learning rate of 0 .
It takes around 20 minutes to train federated learning models, with slightly more training time required for the models with a larger number of shots.
Quickdraw Each client has 35 classes, which are sampled uniformly at the start of training.
As there is no pre-defined train/test split for the Quickdraw dataset, we randomly choose 100 samples from each of the 345 classes and use them for testing.
This test data is then used to test both the personalized and global settings as described in the paragraph above.
We train all federated training models for 120 communication rounds, with 5 clients per round, and 10 update steps per client.
We use a constant learning rate of 0 .
006 for training all federated learning models.
Upper bound baseline models, which require training a global model using all available data, were trained for 3000 steps using the Adam optimizer with a constant learning rate of 0 . 006 .
Lower baseline models, requiring training a personalized model for each individual, were trained for 400 steps using the
Adam optimizer with a learning rate of 0 . 006 .
It takes around 2 hours to train federated learning models, with slightly more training time required for the models with a larger number of shots.