Original: F EW -R OUND L EARNING FOR F EDERATED L EARNING
Revised: F EW -R OUND L EARNING FOR F EDERATED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning (FL) presents an appealing opportunity for individuals who are willing to make their private data available for building a communal model without revealing their data contents to anyone else.
Revised: Federated learning (FL) presents an appealing opportunity for individuals who are willing to make their private data available for building a communal model without revealing their data contents to anyone else.

Original: Of central issues that may limit a widespread adoption of FL is the signiﬁcant communication resources required in the exchange of updated model parameters between the server and individual clients over many communication rounds.
Revised: Of central issues that may limit a widespread adoption of FL is the signiﬁcant communication resources required in the exchange of updated model parameters between the server and individual clients over many communication rounds.

Original: In this work, we focus on limiting the number of model exchange rounds in FL to some small ﬁxed number R , to control the communication burden.
Revised: In this work, we focus on preparing an initial model that can limit the number of model exchange rounds in FL to some small ﬁxed number R .

Revised: We assume that the tasks of the clients participating in FL are not known in the preparing stage.

Original: Following the spirit of meta-learning for few-shot learning, we take a meta-learning strategy to train the  model so that once the metatraining phase is over, only R rounds of FL would produce a model that will satisfy the needs of all participating clients.
Revised: Following the spirit of meta-learning for few-shot learning, we take a meta-learning strategy to prepare the initial model so that once this meta-training phase is over, only R rounds of FL would produce a model that will satisfy the needs of all participating clients.

Original: A key advantage of employing meta-training is that the main labeled dataset used in training could differ signiﬁcantly (e.g., different classes of images) from the actual data sample presented at inference time.

Original: Compared to the meta-training approaches to optimize personalized local models at distributed devices, our method better handles the potential lack of data variability at individual nodes.
Revised: Compared to the metatraining approaches to optimize personalized local models at distributed devices, our method better handles the potential lack of data variability at individual nodes.

Original: Extensive experimental results indicate that metatraining geared to few-round learning provide large performance improvements compared to various baselines.
Revised: Extensive experimental results indicate that meta-training geared to few-round learning provides large performance improvements compared to various baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Major machine learning applications including computer vision and natural language processing are currently supported by central data centers equipped with massive computing resources and ample training data.
Revised: Major machine learning applications including computer vision and natural language processing are currently supported by central data centers equipped with massive computing resources and ample training data.

Original: At the same time, growing amounts of valuable data are also being collected at distributed edge nodes such as mobile phones, wearable client devices and smart vehicles/drones.
Revised: At the same time, growing amounts of valuable data are also being collected at distributed edge nodes such as mobile phones, wearable client devices and smart vehicles/drones.

Original: Directly sending these local data to the central server for model training raises signiﬁcant privacy concerns.
Revised: Directly sending these local data to the central server for model training raises signiﬁcant privacy concerns.

Original: To address this issue, an emerging trend known as federated learning (McMahan et al., 2017; Konecny et al., 2016;
Revised: To address this issue, an emerging trend known as federated learning (McMahan et al., 2017; Konecny et al., 2016;

Original: Bonawitz et al., 2019; Li et al., 2019; Zhao et al., 2018; Sattler et al., 2019; Reisizadeh et al., 2019), where server uploading of local data is not necessary, has been actively researched.
Revised: Bonawitz et al., 2019; Li et al., 2019; Zhao et al., 2018; Sattler et al., 2019; Reisizadeh et al., 2019), where server uploading of local data is not necessary, has been actively researched.

Original: Unfortunately, federated learning (FL) generally requires numerous communication rounds between the server and the distributed nodes (or clients) for model exchange, to achieve a desired level of prediction performance.
Revised: Unfortunately, federated learning (FL) generally requires numerous communication rounds between the server and the distributed nodes (or clients) for model exchange, to achieve a desired level of prediction performance.

Original: This makes the deployment of FL a signiﬁcant challenge in bandwidthlimited or time-sensitive applications.
Revised: This makes the deployment of FL a signiﬁcant challenge in bandwidthlimited or time-sensitive applications.

Original: Especially in real-time applications (e.g., connected vehicles or drones), where the model should quickly adapt to dynamically evolving environments, the requirement on many communication rounds becomes a major bottleneck.
Revised: Especially in real-time applications (e.g., connected vehicles or drones), where the model should quickly adapt to dynamically evolving environments, the requirement on many communication rounds becomes a major bottleneck.

Original: Moreover, the considerable amounts of time and computational resources required for training place a high burden on individual clients wishing to participate in FL.
Revised: Moreover, the considerable amounts of time and computational resources required for training place a high burden on individual clients wishing to participate in FL.

Original: Excessive communication rounds in FL are a major concern especially in light of the increased communication burden for guaranteeing full privacy via secure aggregation (Bonawitz et al., 2017).
Revised: Excessive communication rounds in FL are a major concern especially in light of the increased communication burden for guaranteeing full privacy via secure aggregation (Bonawitz et al., 2017).

Original: To combat this limitation, we propose few-round learning to obtain a high-accuracy global model within only a few communication rounds between the server and the clients.
Revised: To combat this limitation, we focus on preparing an initial model that can quickly obtain a highaccuracy global model within only a few communication rounds between the server and the clients.

Original: Following the spirit of meta-learning for few-shot learning, we meta-train the model via episodic training to mimic and tee up for few-round FL.
Revised: Following the spirit of meta-learning for few-shot learning, we meta-train the model via episodic training to mimic and tee up for few-round FL .

Original: Meta-training enables reliable prediction even when the data sample at hand does not share the same characteristics with the dataset the given model was trained with.
Revised: Meta-training enables reliable prediction even when the data sample at hand does not share the same characteristics with the dataset the given model was trained with.

Original: In contrast to existing meta-training attempts to initiate a model for further personalized optimizations at local devices, our approach takes advantage of FL’s ability to exploit varying data distributions across clients.
Revised: In contrast to existing meta-training attempts to initiate a model for further personalized optimizations at local devices, our approach takes advantage of FL’s ability to exploit varying data distributions across clients.

Original: A high-level description of our idea is depicted in Fig.
Revised: A high-level description of our idea is depicted in Fig.

Original: Given a small target value 
Revised: Given a small target value R , our goal is to create an initial model that can quickly adapt, within R rounds of FL, to a set of clients with tasks not seen during meta-training. As long as the tasks are different

Original: [Figure]
Revised: [Figure]

Original: R , our goal is to create an initial model that can quickly adapt, within R rounds of FL, to a set of clients with tasks not seen during meta-training.

Original: As long as the tasks are different between metatraining and deployment, it is immaterial whether a node that participated in the meta-training also partakes in few-round federated learning (followed by inference).
Revised: between meta-training and deployment, it is immaterial whether a node that participated in the metatraining also partakes in few-round federated learning (followed by inference).

Original: In the context of image classiﬁcation, different tasks mean classiﬁcation involving different sets of image categories or classes (e.g., different categories of lung diseases to be diagnosed using chest X-ray images).
Revised: In the context of image classiﬁcation, different tasks mean classiﬁcation involving different sets of image categories or classes (e.g., different categories of lung diseases to be diagnosed using chest X-ray images).

Original: The prospect of meta-training also raises an intriguing possibility that meta-training can actually be done using proxy data at the server simply mimicking the federated optimization process, although we will not be concerned with this approach in the present work.
Revised: The prospect of meta-training also raises an intriguing possibility that meta-training can actually be done using proxy data at the server simply mimicking the federated optimization process, although we will not be concerned with this approach in the present work.

Original: Extensive experimental results show that our few-round learning algorithm outperforms various baselines in both IID (independent, identically distributed) and non-IID data distribution setups.
Revised: Extensive experimental results show that our few-round learning algorithm outperforms various baselines in both IID (independent, identically distributed) and non-IID data distribution setups.

Original: In an IID setup, for example, our algorithm achieves a 75.32% the accuracy on tiered ImageNet within only R = 3 rounds of FL, which surpasses ﬁne-tuned federated averaging (FedAvg) by 14.63% and ﬁne-tuned one-shot federated learning (Guha et al., 2019) by 12.88%.
Revised: In an IID setup, for example, our algorithm achieves a 75.32% the accuracy on tiered ImageNet within only R = 3 rounds of FL, which surpasses ﬁne-tuned federated averaging (FedAvg) by 14.63% and ﬁne-tuned one-shot federated learning (Guha et al., 2019) by 12.88%.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Few-shot learning.
Revised: Few-shot learning.

Original: Few-shot learning is an instantiation of meta-learning.
Revised: Few-shot learning is an instantiation of meta-learning.

Original: In the context of image classiﬁcation, few-shot learning typically involves episodic training where each episode of training data is arranged into a few training (support) sample images and validation (query) samples to mimic inference that uses only a few examples (Vinyals et al., 2016).
Revised: In the context of image classiﬁcation, few-shot learning typically involves episodic training where each episode of training data is arranged into a few training (support) sample images and validation (query) samples to mimic inference that uses only a few examples (Vinyals et al., 2016).

Original: Through a repetitive exposure to a series of varying episodes with different sets of image classes, the model learns to handle new tasks (classiﬁcation against unseen classes) each time.
Revised: Through a repetitive exposure to a series of varying episodes with different sets of image classes, the model learns to handle new tasks (classiﬁcation against unseen classes) each time.

Original: Two widely-known few-shot learning methods with different philosophical twists, which are also conceptually relevant to the present work, are model-agnostic meta-learning (MAML) of (Finn et al., 2017) and Prototypical Networks of (Snell et al., 2017).
Revised: Two widely-known few-shot learning methods with different philosophical twists, which are also conceptually relevant to the present work, are model-agnostic meta-learning (MAML) of (Finn et al., 2017) and Prototypical Networks of (Snell et al., 2017).

Original: MAML attempts to generate an initial model from which different models targeting different tasks can be obtained quickly via just a few gradient updates.
Revised: MAML attempts to generate an initial model from which different models targeting different tasks can be obtained quickly via just a few gradient updates.

Original: The idea is that the initial model is learned via meta-training to develop an internal representation that is close in some sense to a variety of unseen tasks.
Revised: The idea is that the initial model is learned via meta-training to develop an internal representation that is close in some sense to a variety of unseen tasks.

Original: Prototypical Networks, on the other hand, do not rely on such ﬁnetuning using few examples but rather learn embedding space such that model outputs cluster around class prototypes, the class-speciﬁc centroids of the embedder outputs.
Revised: Prototypical Networks, on the other hand, do not rely on such ﬁnetuning using few examples but rather learn embedding space such that model outputs cluster around class prototypes, the class-speciﬁc centroids of the embedder outputs.

Original: With episodic training, simple
Revised: With episodic training, simple

Original: Prototypical Networks appear to be effective in learning inductive bias for successful generalization to new tasks.
Revised: Prototypical Networks appear to be effective in learning inductive bias for successful generalization to new tasks.

Original: Our work takes from both concepts: we utilize prototype representation and we also adopt ﬁne-tuning during R -round federated learning in adapting to new tasks.
Revised: Our work takes from both concepts: we utilize prototype representation and we also adopt ﬁne-tuning during R -round federated learning in adapting to new tasks.

Original: Federated meta-learning.
Revised: Federated meta-learning.

Original: Recent research activity has focused on improving model personalization via federated meta-learning (Lin et al., 2020;
Revised: Recent research activity has focused on improving model personalization via federated meta-learning (Lin et al., 2020;

Original: Chen et al., 2018; Fallah et al., 2020; Jiang et al., 2019).
Revised: Chen et al., 2018; Fallah et al., 2020; Jiang et al., 2019).

Original: The common goal of these works is to generate an initial model based on which each new client can ﬁnd its own optimized model via a few local gradient steps and using only its own data.
Revised: The common goal of these works is to generate an initial model based on which each new client can ﬁnd its own optimized model via a few local gradient steps and using only its own data.

Original: In these works, meta-learning employed during federated learning intends to enable each client to handle previously unseen tasks, in the spirit of MAML of (Finn et al., 2017).
Revised: In these works, meta-learning employed during federated learning intends to enable each client to handle previously unseen tasks, in the spirit of MAML of (Finn et al., 2017).

Original: User-speciﬁc nextword prediction at individual smartphones, for example, could be an application of approaches along this direction.
Revised: User-speciﬁc nextword prediction at individual smartphones, for example, could be an application of approaches along this direction.

Original: Compared to this line of work, we focus on creating an initial model that leads to a high-accuracy global model, rather than personalized models, within only a few rounds of federated learning .
Revised: Compared to this line of work, we focus on creating an initial model that leads to a high-accuracy global model, rather than personalized models, within only a few rounds of federated learning .

Original: In this way, we seek to take advantage of a higher variety of data as well as the larger data volume that would be made available through collaborative learning of many distributed nodes.
Revised: In this way, we seek to take advantage of a higher variety of data as well as the larger data volume that would be made available through collaborative learning of many distributed nodes.

Original: A clear example is the diagnosis of a broader class of diseases that would be possible through collective training across numerous examples contributed by a larger group of individuals.
Revised: A clear example is the diagnosis of a broader class of diseases that would be possible through collec- tive training across numerous examples contributed by a larger group of individuals.

Original: Personalized optimization would be especially at disadvantage in non-IID setting, where each client necessarily lacks a sufﬁcient variety of training data.
Revised: Personalized optimization would be especially at disadvantage in non-IID setting, where each client necessarily lacks a sufﬁcient variety of training data.

Original: One-shot federated learning.
Revised: One-shot federated learning.

Original: Another line of work recently focused on one-shot federated learning, where the goal is to train a global model with just one communication round between the server and the clients.
Revised: Another line of work recently focused on one-shot federated learning, where the goal is to train a global model with just one communication round between the server and the clients.

Original: The authors of (Guha et al., 2019) proposed an ensemble method to choose reliable client-speciﬁc models from given clients.
Revised: The authors of (Guha et al., 2019) proposed an ensemble method to choose reliable client-speciﬁc models from given clients.

Original: To effectively capture global information, they proposed various criteria by which participating clients are chosen.
Revised: To effectively capture global information, they proposed various criteria by which participating clients are chosen.

Original: In the work of (Shin et al., 2020), local clients send XOR-encoded MNIST image data to the server, and the server decodes it to train the global model.
Revised: In the work of (Shin et al., 2020), local clients send XOR-encoded MNIST image data to the server, and the server decodes it to train the global model.

Original: While the server would need certain data in advance to decode the received results,
Revised: While the server would need certain data in advance to decode the received results,

Original: XOR operation can serve as data augmentation while preserving privacy.
Revised: XOR operation can serve as data augmentation while preserving privacy.

Original: In the fusion learning of (Kasturi et al., 2020), each local client uploads both the model parameters and the distribution parameters to the server.
Revised: In the fusion learning of (Kasturi et al., 2020), each local client uploads both the model parameters and the distribution parameters to the server.

Original: The server generates artiﬁcial data samples from the distribution parameters to train a global model.
Revised: The server generates artiﬁcial data samples from the distribution parameters to train a global model.

Original: When the data gets complex, however, it is not clear whether conversion into a simple distribution would be reliable.
Revised: When the data gets complex, however, it is not clear whether conversion into a simple distribution would be reliable.

Original: Compared to the existing works on one-shot federated learning that employ some randomly initialized model, the key difference of our method is the use of meta-learning to obtain an initial model which can adapt to the unseen tasks of individual clients within R rounds of FL.
Revised: Compared to the existing works on one-shot federated learning that employ some randomly initialized model, the key difference of our method is the use of meta-learning to obtain an initial model which can adapt to the unseen tasks of individual clients within R rounds of FL.

Original: It is shown in Section 4 that the suggested approach yields remarkable performance gains compared to one-shot federated learning methods.
Revised: It is shown in Section 4 that the suggested approach yields remarkable performance gains compared to one-shot federated learning methods.

Original:  P ROPOSED F EW -R OUND L EARNING A LGORITHM 3. P ROBLEM S ETUP
Revised:  P ROPOSED F EW -R OUND L EARNING A LGORITHM 3. P ROBLEM S ETUP

Original: Federated learning.
Revised: Federated learning.

Original: Federated learning allows each distributed node k with a dataset D k to participate in iterative learning of a global model θ without having to reveal its data to anyone else including the central server.
Revised: Federated learning allows each distributed node k with a dataset D k to participate in iterative learning of a global model θ without having to reveal its data to anyone else including the central server.

Original: As a given round r starts, each of K participating nodes downloads a global model θ r from the server and updates it using its own local data D k .
Revised: As a given round r starts, each of K participating nodes downloads a global model θ r from the server and updates it using its own local data D k .

Original: The updated local models θ r +1 ( k ) get all uploaded to the server, which then aggregates them to get a new model θ r +

Revised: The updated local models θ r +1 ( k ) get all uploaded to the server, which then aggregates them to get a new model θ r +

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =

Revised: Kk =

Original: ρ k θ r +

Revised: ρ k θ r +

Original: ( k )
Revised: ( k )

Original: according to the relative dataset sizes ρ k = | D k | (cid:80)
Revised: according to the relative dataset sizes ρ k = | D k | (cid:80)

Original: Kk =1 | D k | .
Revised: Kk =1 | D k | .

Original: As the next rounds starts, this new global model gets downloaded to all participating nodes and the same process is repeated.
Revised: As the next rounds starts, this new global model gets downloaded to all participating nodes and the same process is repeated.

Original: Federated learning generally requires a signiﬁcant number of such global rounds to achieve the desired accuracy, with each round taking up substantial computing and communication resources.
Revised: Federated learning generally requires a signiﬁcant number of such global rounds to achieve the desired accuracy, with each round taking up substantial computing and communication resources.

Original: Problem formulation.
Revised: Problem formulation.

Original: To enable fast federated learning, we would like to create an initial model φ in which a group of clients can quickly obtain a high-performance global model within R communication rounds.
Revised: To enable fast federated learning, we would like to prepare an initial model φ in which a group of clients can quickly obtain a high-performance global model within R communication rounds.

Original: In applying meta-learning, we wish to satisfy clients who hope to predict data that were unseen during meta-training.
Revised: In applying meta-learning, we wish to satisfy clients who hope to predict data that were unseen during meta-training.

Original: During R rounds of FL at deployment time, each client is also expected to make available some small number of labeled examples associated with the data she hopes to predict.
Revised: During R rounds of FL at deployment time, each client is also expected to make available some small number of labeled examples associated with the data she hopes to predict.

Original: To create a training environment matching the actual R -rounds of FL followed by inference at deployment, in each episode, our meta-training updates the model over R federated rounds using the support set ( R global rounds) and then makes a ﬁnal adjustment (meta-update) using the query set.
Revised: To create a training environment matching the actual R -rounds of FL followed by inference at deployment, in each episode, our meta-training updates the model over R federated rounds using the support set ( R global rounds) and then makes a ﬁnal adjustment (meta-update) using the query set.

Original: This process is repeated as the model is exposed to a series of episodes.
Revised: This process is repeated as the model is exposed to a series of episodes.

Original: 3. M ETA -T RAINING
Revised: 3. M ETA -T RAINING

Original: Our meta-training procedure is given in Algorithm 1.
Revised: Our meta-training procedure is given in Algorithm 1.

Original: Let φ 0 be the initial model and φ T the model generated after all T episodic stages.
Revised: Let φ 0 be the initial model and φ T the model generated after all T episodic stages.

Original: To construct each episode t , the server selects a new set of K clients participating for training 1 .
Revised: To construct each episode t , the server selects a new set of K clients participating for training 1 .

Original: The model φ t has been updated through the end of episode stage t − 1 .
Revised: The model φ t has been updated through the end of episode stage t − 1 .

Original: The server lets θ 0 =
Revised: The server lets θ 0 =

Original: φ t and sends this initial model to the selected K clients.
Revised: φ t and sends this initial model to the selected K clients.

Original: Through R rounds of local updates and global aggregations guided by the support sets, the model θ 0 evolves to θ R .
Revised: Through R rounds of local updates and global aggregations guided by the support sets, the model θ 0 evolves to θ R .

Original: Before moving to the next episode, a loss based on the updated model θ R and the local query set is minimized with respect to φ t to get each meta-updated local model φ t +1 ( k ) .
Revised: Before moving to the next episode, a loss based on the updated model θ R and the local query set is minimized with respect to φ t to get each meta-updated local model φ t +1 ( k ) .

Original: In the ﬁnal server aggregation of the episode stage, the global meta-updated model φ t +1 is obtained from the local models, which then becomes the initial model for the next episode.
Revised: In the ﬁnal server aggregation of the episode stage, the global meta-updated model φ t +1 is obtained from the local models, which then becomes the initial model for the next episode.

Revised: Here, φ is a parameter that is updated by meta-update process with the query sets, and θ is a prameter that is updated during R rounds of federated learning with the support sets.

Original: Speciﬁcally, given K clients in each episode, we ﬁrst split D k (the local dataset of client k ) into support set S k and query set Q k such that they are disjoint.
Revised: Speciﬁcally, given K clients in each episode, we ﬁrst split D k (the local dataset of client k ) into support set S k and query set Q k such that they are disjoint.

Original: Starting from the initial model θ 0 , each client k utilizes its support set S k to update the local model during R global rounds.
Revised: Starting from the initial model θ 0 , each client k utilizes its support set S k to update the local model during R global rounds.

Original: The local update in Line 16 involves a loss function that depends on class prototypes, as to be clariﬁed shortly.
Revised: The local update in Line 16 involves a loss function that depends on class prototypes, as to be clariﬁed shortly.

Original: In our method, the server aggregates not only the local models (Line 18) but also the local prototypes (Line 20).
Revised: In our method, the server aggregates not only the local models (Line 18) but also the local prototypes (Line 20).

Original: After R rounds of local updates and global aggregations, the server obtains the global model θ R and sends it to the K clients while still in the current episode stage.
Revised: After R rounds of local updates and global aggregations, the server obtains the global model θ R and sends it to the K clients while still in the current episode stage.

Original: Based on the downloaded θ R and prototypes, each client k utilizes its query set Q k to compute the query loss L Q k local and metaupdates φ t to obtain φ t +1 ( k ) in Line 24.
Revised: Based on the downloaded θ R and prototypes, each client k utilizes its query set Q k to compute the query loss L Q k local and metaupdates φ t to obtain φ t +1 ( k ) in Line 24.

Original: Note that the local loss L Q k local obviously depends on θ R , which in turn depends on θ 0 =
Revised: Note that the local loss L Q k local obviously depends on θ R , which in turn depends on θ 0 =

Original: φ t ; in principle L Q k local is minimized with respect to φ t in this step.
Revised: φ t ; in principle L Q k local is minimized with respect to φ t in this step.

Original: At last, the server ﬁnds the ﬁnal output φ t +1 of the episode by aggregating the meta-updated local models φ t
Revised: At last, the server ﬁnds the ﬁnal output φ t +1 of the episode by aggregating the meta-updated local models φ t

Original: +

Revised: +

Original: ( k ) for k ∈ { 1 , 2 , ..., K } .
Revised: ( k ) for k ∈ { 1 , 2 , ..., K } .

Original: In the meta-learning point of view, the initial model φ ﬁrst adapts to the local support sets for R global rounds in the ﬁrst step, and then is meta-updated based on the losses computed from the adapted model θ R and the local query sets, in the second step.
Revised: In the meta-learning point of view, the initial model φ ﬁrst adapts to the local support sets for R global rounds in the ﬁrst step, and then is meta-updated based on the losses computed from the adapted model θ R and the local query sets, in the second step.

Original: This two-step training procedure mimics the deployment phase where the initial model is updated for R communication rounds before making predictions.
Revised: This two-step training procedure mimics the deployment phase where the initial model is updated for R communication rounds before making predictions.

Original: In the following, we provide some missing details on the description of loss functions and prototype processing.
Revised: In the following, we provide some missing details on the description of loss functions and prototype processing.

Original: ..K } splits D k into support set S k and query set Q k 4: θ 0 ← φ t 5: for each communication round r = 0 , 1 , ..., R − 1 do 6: for each client k in parallel do 7: if r = 0 then 8:
Revised: ..K } splits D k into support set S k and query set Q k 4: θ 0 ← φ t 5: for each communication round r = 0 , 1 , ..., R − 1 do 6: for each client k in parallel do 7: if r = 0 then 8:

Original: Download θ r from the server 9: else 10:
Revised: Download θ r from the server 9: else 10:

Original: Download θ r and P r − 1 from the server 11: end if 12: for each class c ∈ C k do 13:
Revised: Download θ r and P r − 1 from the server 11: end if 12: for each class c ∈ C k do 13:

Original: P cr ( k ) = 1 | S k ( c ) | (cid:80) x
Revised: P cr ( k ) = 1 | S k ( c ) | (cid:80) x

Original: ∈ S k ( c ) f θ r ( x ) (cid:46)
Revised: ∈ S k ( c ) f θ r ( x ) (cid:46)

Original: Local prototype calculation 14: with support set S k 15: end for 16: θ r +1 ( k ) ← θ r − α ∇ θ r F S k ( θ r ) (cid:46)
Revised: Local prototype calculation 14: with support set S k 15: end for 16: θ r +1 ( k ) ← θ r − α ∇ θ r F S k ( θ r ) (cid:46)

Original: Local update of θ with S k 17: end for 18: θ r +1 =
Revised: Local update of θ with S k 17: end for 18: θ r +1 =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 λ k θ r +1 ( k ) (cid:46) Server aggregation of local models θ ; 19: λ k is relative support set size 20:
Revised: Kk =1 λ k θ r +1 ( k ) (cid:46) Server aggregation of local models θ ; 19: λ k is relative support set size 20:

Original: P r = (cid:110) (cid:80)
Revised: P r = (cid:110) (cid:80)

Original: Kk =1 λ k P cr
Revised: Kk =1 λ k P cr

Original: k ) | c = 1 , 2 , ..., N c (cid:111)
Revised: k ) | c = 1 , 2 , ..., N c (cid:111)

Original: (cid:46) Server aggregation of local prototypes 21: end for 22: for each client k in parallel do 23:
Revised: (cid:46) Server aggregation of local prototypes 21: end for 22: for each client k in parallel do 23:

Original: Download θ R and P R − 1 from the server 24: φ t +1 ( k ) ← φ t − β ∇ φ t L Q k local ( θ R , P R − 1 ) .
Revised: Download θ R and P R − 1 from the server 24: φ t +1 ( k ) ← φ t − β ∇ φ t L Q k local ( θ R , P R − 1 ) .

Original: (cid:46)
Revised: (cid:46)

Original: Local meta-update of φ with query set Q k 25: end for 26: φ t +1 =
Revised: Local meta-update of φ with query set Q k 25: end for 26: φ t +1 =

Original: (cid:80)
Revised: (cid:80)

Original: Kk
Revised: Kk

Original: =

Revised: =

Original: ρ k φ t +1 ( k ) (cid:46) Server aggregation of meta-updated models; ρ k is relative data size 27: end for
Revised: ρ k φ t +1 ( k ) (cid:46) Server aggregation of meta-updated models; ρ k is relative data size 27: end for

Original: 3.2. R R OUNDS OF L OCAL U PDATES AND A GGREGATIONS
Revised: 3.2. R R OUNDS OF L OCAL U PDATES AND A GGREGATIONS

Original: In deﬁning the loss function, we utilize the class prototypes and associated distance metric of (Snell et al., 2017).
Revised: In deﬁning the loss function, we utilize the class prototypes and associated distance metric of (Snell et al., 2017).

Original: For each communication round r = 0 , 1 , ..., R − 1 , we not only aggregate the global model θ r +

Revised: For each communication round r = 0 , 1 , ..., R − 1 , we not only aggregate the global model θ r +

Original: but also the set of global prototypes P r
Revised: but also the set of global prototypes P r

Original: = { P cr | c = 1 , 2 , ..., N c } for all classes c ∈ { 1 , 2 , ...
Revised: = { P cr | c = 1 , 2 , ..., N c } for all classes c ∈ { 1 , 2 , ...

Original: N c } , where N c is the number of classes in the current episode over all clients.
Revised: N c } , where N c is the number of classes in the current episode over all clients.

Original: Model and global prototype download.
Revised: Model and global prototype download.

Original: In the beginning of round r ≥ 1 , the server has the global model θ r and the set of global prototypes P r − 

Revised: In the beginning of round r ≥ 1 , the server has the global model θ r and the set of global prototypes P r −

Original: =
Revised: =

Original: { P c r − 1 | c = 1 , 2 , ..., N c } which are the outputs of the previous round r − 1 .
Revised: { P c r − 1 | c = 1 , 2 , ..., N c } which are the outputs of the previous round r − 1 .

Original: Each client k ∈ { 1 , 2 , ..., K } ﬁrst downloads θ r and P r − 1 from the server.
Revised: Each client k ∈ { 1 , 2 , ..., K } ﬁrst downloads θ r and P r − 1 from the server.

Original: Since there is no global prototype in the ﬁrst round, the clients only download the model θ 0 from the server when r = 0 .
Revised: Since there is no global prototype in the ﬁrst round, the clients only download the model θ 0 from the server when r = 0 .

Original: Local prototype calculation.
Revised: Local prototype calculation.

Original: Based on the downloaded model θ r , each client k computes the local prototype P cr ( k ) for each class c ∈ C k using its support set S k as
Revised: Based on the downloaded model θ r , each client k computes the local prototype P cr ( k ) for each class c ∈ C k using its support set S k as

Original: [Equation]
Revised: [Equation]

Original: where C k is a set that contains all classes in client k and S k ( c ) is a set of all data samples in S k labeled with class c .
Revised: where C k is a set that contains all classes in client k and S k ( c ) is a set of all data samples in S k labeled with class c .

Original: For a given input data sample x , f θ ( x ) represents the output of function f with model parameters θ .
Revised: For a given input data sample x , f θ ( x ) represents the output of function f with model parameters θ .

Original: This local prototype P cr ( k ) can be viewed as a representative of class c calculated based on the local data (support set) of client k .
Revised: This local prototype P cr ( k ) can be viewed as a representative of class c calculated based on the local data (support set) of client k .

Original: Loss calculation from local prototype.
Revised: Loss calculation from local prototype.

Original: Let P r ( k ) be the set that contains all types of prototypes in client k , deﬁned as P r ( k )
Revised: Let P r ( k ) be the set that contains all types of prototypes in client k , deﬁned as P r ( k )

Original: = { P cr ( k )
Revised: = { P cr ( k )

Original: | c ∈ C k } .
Revised: | c ∈ C k } .

Original: Now using S k , θ r and P r ( k ) , each client k computes the local loss L S k local ( θ r , P r ( k )) according to
Revised: Now using S k , θ r and P r ( k ) , each client k computes the local loss L S k local ( θ r , P r ( k )) according to

Original: L S k local ( θ, P r ( k )
Revised: L S k local ( θ, P r ( k )

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: which is based on the squared euclidean distance d ( · ) between P cr ( k ) and f θ ( x ) for x ∈ S k ( c ) .
Revised: which is based on the squared euclidean distance d ( · ) between P cr ( k ) and f θ ( x ) for x ∈ S k ( c ) .

Original: We would like to minimize this local loss during local update at each client.
Revised: We would like to minimize this local loss during local update at each client.

Original: Auxiliary loss from global prototype.
Revised: Auxiliary loss from global prototype.

Original: If we only consider the loss function of (2), each client would have biased models after local updates, especially when the data distributions across different clients are non-IID.
Revised: If we only consider the loss function of (2), each client would have biased models after local updates, especially when the data distributions across different clients are non-IID.

Original: This generally leads to a performance degradation of the global model.
Revised: This generally leads to a performance degradation of the global model.

Original: Hence, we propose a global prototype-assisted learning (GPAL) strategy where a global prototype helps to train the local models in the right direction; the set of global prototypes P r − 1 can encourage local models to maintain stability during optimization because the global prototype contains all the information in local data from the entire set of clients.
Revised: Hence, we propose a global prototype-assisted learning (GPAL) strategy where a global prototype helps to train the local models in the right direction; the set of global prototypes P r − 1 can encourage local models to maintain stability during optimization because the global prototype contains all the information in local data from the entire set of clients.

Original: We adopt the dense classiﬁcation loss of (Lifchitz et al., 2019) to construct the auxiliary loss.
Revised: We adopt the dense classiﬁcation loss of (Lifchitz et al., 2019) to construct the auxiliary loss.

Original: Let g θ ( x ) be the output of the last convolutional neural network (CNN) layer of model θ , immediately before average pooling.
Revised: Let g θ ( x ) be the output of the last convolutional neural network (CNN) layer of model θ , immediately before average pooling.

Original: Let g θ ( x ) i be its feature vector at position i .
Revised: Let g θ ( x ) i be its feature vector at position i .

Original: For each local update, client k compares global prototype P r − 1 =
Revised: For each local update, client k compares global prototype P r − 1 =

Original: { P cr − 1 | c = 1 , 2 , ..., N c } with g θ ( x ) i and calculates auxiliary loss L S k aux
Revised: { P cr − 1 | c = 1 , 2 , ..., N c } with g θ ( x ) i and calculates auxiliary loss L S k aux

Original: ( θ r , P r − 1 ) similar to (2):
Revised: ( θ r , P r − 1 ) similar to (2):

Original: L S k
Revised: L S k

Original: [Equation]
Revised: [Equation]

Original: We also wish to minimize this auxiliary loss, to make the local clients not overly biased and to learn general embedding space.
Revised: We also wish to minimize this auxiliary loss, to make the local clients not overly biased and to learn general embedding space.

Original: Local model update.
Revised: Local model update.

Original: Based on the loss functions (2) and (3), for r ≥ 1 , the objective function for local optimization becomes
Revised: Based on the loss functions (2) and (3), for r ≥ 1 , the objective function for local optimization becomes

Original: [Equation]
Revised: [Equation]

Original: where γ is a balancing factor.
Revised: where γ is a balancing factor.

Original: For r = 0 , we have F S k
Revised: For r = 0 , we have F S k

Original: ( θ r ) =
Revised: ( θ r ) =

Original: L S k local ( θ r , P r ( k ))
Revised: L S k local ( θ r , P r ( k ))

Original: since the global prototype is not deﬁned in the ﬁrst global round.
Revised: since the global prototype is not deﬁned in the ﬁrst global round.

Original: Now based on the the objective function F S k ( θ r ) , each client k performs local update according to Line 16 of Algorithm 1, where α is the learning rate.
Revised: Now based on the the objective function F S k ( θ r ) , each client k performs local update according to Line 16 of Algorithm 1, where α is the learning rate.

Original: In federated learning, the clients generally performs multiple local updates, say E times.
Revised: In federated learning, the clients generally performs multiple local updates, say E times.

Original: Hence, the process of local prototype computation (1), loss computations (2), (3) and local model update of
Revised: Hence, the process of local prototype computation (1), loss computations (2), (3) and local model update of

Original: Line 16 of Algorithm 1 is repeated E times to obtain θ r +1 ( k ) .
Revised: Line 16 of Algorithm 1 is repeated E times to obtain θ r +1 ( k ) .

Original: Model and prototype aggregation at the server.
Revised: Model and prototype aggregation at the server.

Original: After performing local updates at the clients, each client k sends its updated local model θ r +1 ( k ) and the computed local prototypes P r ( k ) to the server.
Revised: After performing local updates at the clients, each client k sends its updated local model θ r +1 ( k ) and the computed local prototypes P r ( k ) to the server.

Original: Then the server aggregates the local models and the local prototypes according to Linesand 20 in Algorithm 1, respectively, where the weighting factor λ k =
Revised: Then the server aggregates the local models and the local prototypes according to Linesand 20 in Algorithm 1, respectively, where the weighting factor λ k =

Original: [Equation]
Revised: [Equation]

Original: | reﬂects the relative support set sizes.
Revised: | reﬂects the relative support set sizes.

Original: The above local update and global aggregation processes are repeated for R global rounds ( r = 0 , 1 , ..., R − 1 ), and the server ﬁnally obtains θ R and P R − 1 .
Revised: The above local update and global aggregation processes are repeated for R global rounds ( r = 0 , 1 , ..., R − 1 ), and the server ﬁnally obtains θ R and P R − 1 .

Original: 3.2. O NE -R OUND L OCAL M ETA -U PDATE AND A GGREGATION
Revised: 3.2. O NE -R OUND L OCAL M ETA -U PDATE AND A GGREGATION

Original: Towards the end of each episode processing stage, the clients download θ R and P R − 1 from the server.
Revised: Towards the end of each episode processing stage, the clients download θ R and P R − 1 from the server.

Original: At each client k , the local query loss L Q k local ( θ R , P R − 1 ) is calculated similar to (2) based on Q k , θ R , and P R − 1 .
Revised: At each client k , the local query loss L Q k local ( θ R , P R − 1 ) is calculated similar to (2) based on Q k , θ R , and P R − 1 .

Original: Taking a derivative of L Q k local with respect to φ t in Line 24 can easily be done through the chain rule, yielding (dropping super/subscripts and terms not critical for now): 
Revised: Taking a derivative of L Q k local with respect to φ t in Line 24 can easily be done through the chain rule, yielding (dropping super/subscripts and terms not critical for now): ∇

Original: [Equation]
Revised: [Equation]

Original: where θ 0 = φ t and the loss function F S k is the local loss seen in Line 16.
Revised: where θ 0 = φ t and the loss function F S k is the local loss seen in Line 16.

Original: We notice that the product term here involves double derivatives, which can be safely ignored according to our empirical observation (which is consistent with what was observed in a parallel setting in MAML of (Finn et al., 2017)).
Revised: We notice that the product term here involves double derivatives, which can be safely ignored according to our empirical observation (which is consistent with what was observed in a parallel setting in MAML of (Finn et al., 2017)).

Original: Thus, we simply use ﬁrst-order approximation ∇ φ t L ( θ R )
Revised: Thus, we simply use ﬁrst-order approximation ∇ φ t L ( θ R )

Original: ≈ ∇ θ R L ( θ R ) to perform the local meta-update in Line 24.
Revised: ≈ ∇ θ R L ( θ R ) to perform the local meta-update in Line 24.

Original: The server ﬁnally aggregates the meta-updated models from all clients and moves on to the next episode.
Revised: The server ﬁnally aggregates the meta-updated models from all clients and moves on to the next episode.

Original: 3. D EPLOYMENT (T ESTING )
Revised: 3. D EPLOYMENT (T ESTING )

Original: In the actual deployment or test phase, given a set of clients with unseen classes, the server sets θ 0 =
Revised: In the actual deployment or test phase, given a set of clients with unseen classes, the server sets θ 0 =

Original: φ T and then performs R rounds of federated learning to obtain θ R and P R − 1 .
Revised: φ T and then performs R rounds of federated learning to obtain θ R and P R − 1 .

Original: Now, given a test sample, we make prediction based on θ R and P R − 1 : we ﬁrst compute the output of model θ R with the new test sample, and then compare the distances with all global prototypes in P R − 1 to make the decision.
Revised: Now, given a test sample, we make prediction based on θ R and P R − 1 : we ﬁrst compute the output of model θ R with the new test sample, and then compare the distances with all global prototypes in P R − 1 to make the decision.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: We validate our proposed algorithm on two benchmark datasets for meta-learning, mini ImageNet (Vinyals et al., 2016) and tiered ImageNet (Ren et al., 2018), which have signiﬁcantly larger numbers of classes than MNIST or CIFAR-10 commonly used in federated learning studies.
Revised: We validate our proposed algorithm on two benchmark datasets for meta-learning, mini ImageNet (Vinyals et al., 2016) and tiered ImageNet (Ren et al., 2018), which have signiﬁcantly larger numbers of classes than MNIST or CIFAR-10 commonly used in federated learning studies.

Original: Following the data splits introduced in (Ravi & Larochelle, 2017), 100 classes are divided into 64 training,test and 16 validation classes for mini ImageNet.
Revised: Following the data splits introduced in (Ravi & Larochelle, 2017), 100 classes are divided into 64 training, 20 test and 16 validation classes for mini ImageNet.

Original: For tiered ImageNet, the dataset is divided into 351/160/97 classes for train/test/validation, respectively.
Revised: For tiered ImageNet, the dataset is divided into 351/160/97 classes for train/test/validation, respectively.

Revised: Experimental results on CIFARdataset can be found in the Supplementary Materials.

Original: Comparison schemes.
Revised: Comparison schemes.

Original: We compare our algorithm with the following schemes.
Revised: We compare our algorithm with the following schemes.

Original: First, as a simplest baseline, we consider FedAvg (McMahan et al., 2017), where a randomly initialized model is trained for R communication rounds.
Revised: First, as a simplest baseline, we consider FedAvg (McMahan et al., 2017), where a randomly initialized model is trained for R communication rounds.

Revised: Preparing stage is not utilized for this scheme.

Original: Second, we consider a FedAvg-based ﬁne-tuning method, where the overall model is ﬁrst pre-trained on the training set, and then ﬁne-tuned to the unseen test set for R communication rounds via FedAvg.
Revised: Second, we consider a FedAvg-based ﬁne-tuning method, where the overall model is ﬁrst pre-trained on the training set in the preparing stage, and then ﬁne-tuned to the unseen test set for R communication rounds via FedAvg in the deployment phase.

Original: For example, in mini ImageNet, a 64-way classiﬁer model is pre-trained ﬁrst.
Revised: For example, in mini ImageNet, a 64-way classiﬁer model is pre-trained ﬁrst.

Original: Next, the last linear layer is replaced and newly initialized, and then the overall model is ﬁne-tuned during testing.
Revised: Next, the last linear layer is replaced and newly initialized, and then the overall model is ﬁne-tuned during testing.

Original: Finally, we consider ﬁne-tuning based on one-shot FL (Guha et al., 2019), where the local models are sampled and aggregated by the ensemble cross-validation (CV) method.
Revised: Finally, we consider ﬁne-tuning based on one-shot FL (Guha et al., 2019), where the local models are sampled and aggregated by the ensemble cross-validation (CV) method.

Original: We allow a larger number of available clients for this scheme to accommodate user sampling.
Revised: We allow a larger number of available clients for this scheme to accommodate user sampling.

Original: The model is ﬁrst pre-trained on the training set, and then ﬁne-tuned to the unseen classes with the scheme of (Guha et al., 2019) for R rounds.
Revised: The model is ﬁrst pre-trained on the training set in the preparing stage, and then ﬁnetuned to the unseen classes with the scheme of (Guha et al., 2019) for R rounds.

Original: For our few-round learning (FRL) algorithm, we utilize both linear classiﬁer and distance-based classiﬁer (Snell et al., 2017) for comparison.
Revised: For our few-round learning (FRL) algorithm, we utilize both linear classiﬁer and distance-based classiﬁer (Snell et al., 2017) for comparison.

Original: For the linear classiﬁer case, we connect an additional linear layer behind CNN layers, as in other baselines.
Revised: For the linear classiﬁer case, we connect an additional linear layer behind CNN layers, as in other baselines.

Original: The distance-based classiﬁer utilizes the output of CNN layers, e.g., to build prototypes, instead of using the linear layer.
Revised: The distance-based classiﬁer utilizes the output of CNN layers, e.g., to build prototypes, instead of using the linear layer.

Original: For the distance-based classiﬁer, we observe the effect of global prototype-assisted learning (GPAL) using the auxiliary loss.
Revised: For the distance-based classiﬁer, we observe the effect of global prototype-assisted learning (GPAL) using the auxiliary loss.

Original: Experimental setup.
Revised: Experimental setup.

Original: We meta-train as well as test our model in federated learning setups.
Revised: We meta-train as well as test our model in federated learning setups.

Original: We typically set the target number of communication rounds to R = 3 .
Revised: We typically set the target number of communication rounds to R = 3 .

Original: Hence, in the meta-training phase, each episode of our scheme requires 4 global rounds: 3 rounds of local updates and aggregation, and 1 round of local meta-update and aggregation.
Revised: Hence, in the meta-training phase, each episode of our scheme requires 4 global rounds: 3 rounds of local updates and aggregation, and 1 round of local meta-update and aggregation.

Original: For a fair comparison, we let all baselines to consume the same amount of communication resources in the meta-training or pre-training phase: 3 . 2 × 10 4 communication rounds between the server and clients.
Revised: For a fair comparison, we let all baselines to consume the same amount of communication resources in the meta-training or pre-training phase: 3 . 2 × 10 4 communication rounds between the server and clients.

Original: Hence, our scheme is meta-trained over 8 . 

Revised: Hence, our scheme is meta-trained over 8 .

Original: × 10 3 episodes, taking 4 rounds in each episode.
Revised: × 10 3 episodes, taking 4 rounds in each episode.

Original: To construct each episode, the server selects K = 10 clients  for participation.
Revised: To construct each episode, the server selects K = 10 clients in the system for participation.

Original: We ﬁrst consider a 5-way setup where 5 classes are randomly sampled from the train/test sets to construct  each episode.
Revised: We ﬁrst consider a 5-way setup whereclasses are randomly sampled from the train/test sets to construct the datasets of clients in each episode.

Original:  120 images are sampled from each classand distributed to K = 10 clients depending on two data distribution setups: IID and non-IID. In the IID setup, the data samples from each class are equally distributed to K = 10 clients; each client has 12 samples for 5 classes, a total of 60 samples. In the non-IID setup, following the procedure
Revised: This type of modeling reﬂects the practical federated learning setup where a large number of clients having different local data coexist in the system. 120 images are sampled from each class

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: and distributed to K = 10 clients depending on two data distribution setups: IID and non-IID.

Revised: In the IID setup, the data samples from each class are equally distributed to K = 10 clients; each client has 12 samples for 5 classes, a total of 60 samples.

Original:  of (McMahan et al., 2017), we ﬁrst sort the 600 training samples of each episode by its category.
Revised: In the non-IID setup, following the procedure of (McMahan et al., 2017), we ﬁrst sort the 600 training samples of each episode by its category.

Original: We then divide the sorted data into 20 shards of size 30 and randomly allocate 2 shards to each client.
Revised: We then divide the sorted data into 20 shards of size 30 and randomly allocate 2 shards to each client.

Original: Hence, no client would be given data corresponding to more than 2 distinct classes.
Revised: Hence, no client would be given data corresponding to more than 2 distinct classes.

Original: For both
Revised: For both

Original: IID and non-IID setups, each client uses one half of its data from each class as support samples, and the remaining half as query samples.
Revised: IID and non-IID setups, each client uses one half of its data from each class as support samples, and the remaining half as query samples.

Original: In other words, in an IID setup, each client has 6 support samples and 6 query samples for each class.
Revised: In other words, in an IID setup, each client has 6 support samples and 6 query samples for each class.

Original: In the non-IID setup, the clients with 2 classes havesupport samples and 15 query samples for each class, while the clients with 1 class have 30 support samples and 30 query samples for that class.
Revised: In the non-IID setup, the clients with 2 classes havesupport samples and 15 query samples for each class, while the clients with 1 class have 30 support samples and 30 query samples for that class.

Original: For the one-shot FL method described above, we allow 20 clients and the server samples K = 10 of them to aggregate.
Revised: For the one-shot FL method described above, we allow 20 clients and the server samples K = 10 of them to aggregate.

Original: In the actual deployment (testing) phase, the support set is utilized for R = 3 rounds of local updates and the server calculates the test accuracy with the global model/prototype and the gathered query sets of all clients.
Revised: In the actual deployment (testing) phase, the support set is utilized for R = 3 rounds of local updates and the server calculates the test accuracy with the global model/prototype and the gathered query sets of all clients.

Original: We averaged the test accuracies over 5 . 

Revised: We averaged the test accuracies over 5 .

Original: × 10 3 randomly sampled episodes in the testing phase.
Revised: × 10 3 randomly sampled episodes in the testing phase.

Original: Implementation details.
Revised: Implementation details.

Original: For a fair comparison, our method and other baselines share the same neural network architecture.
Revised: For a fair comparison, our method and other baselines share the same neural network architecture.

Original: The model follows the settings of (Vinyals et al., 2016), which contains 4 consecutive 3 × 3 convolutional layers with 32 ﬁlters.
Revised: The model follows the settings of (Vinyals et al., 2016), which contains 4 consecutive 3 × 3 convolutional layers with 32 ﬁlters.

Original: Successively, each CNN output goes through
Revised: Successively, each CNN output goes through

Original: ReLU activation, batch normalization and 2 × 2 max pooling.
Revised: ReLU activation, batch normalization and 2 × 2 max pooling.

Original: We adopt the Adam optimizer as the meta-learner with a learning rate of β = 0 .
Revised: We adopt the Adam optimizer as the meta-learner with a learning rate of β = 0 .

Original: 01 and an SGD optimizer as the learner with a learning rate of α = 0 .
Revised: 01 and an SGD optimizer as the learner with a learning rate of α = 0 .

Original: The meta-learning rate reduces by one-tenth after 2 . 

Revised: The meta-learning rate reduces by one-tenth after 2 .

Original: × 10 4 rounds.
Revised: × 10 4 rounds.

Original: We set the number of local epochs to E = 1 .
Revised: We set the number of local epochs to E = 1 .

Original: As mentioned, we do not see advantages in using the second derivatives in computing the gradient in Line 24 of Algorithm 1 in the meta-updating process.
Revised: As mentioned, we do not see advantages in using the second derivatives in computing the gradient in Line 24 of Algorithm 1 in the meta-updating process.

Original: Given the memory and computational overhead issues as well, we naturally disregard the second derivatives and apply ﬁrst-order approximation.
Revised: Given the memory and computational overhead issues as well, we naturally disregard the second derivatives and apply ﬁrst-order approximation.

Original: Experimental results in a 5-way setup.
Revised: Experimental results in a 5-way setup.

Original: Table 1 shows the test accuracies in a 5-way setup.
Revised: Table 1 shows the test accuracies in a 5-way setup.

Original: The model is meta-trained targeting R = 3 and tested after R = 3 rounds in the test phase.
Revised: The model is meta-trained targeting R = 3 and tested after R = 3 rounds in the test phase.

Original: First, for both IID and non-IID scenarios, it can be seen that FedAvg yields signiﬁcantly lower accuracy compared to others, since it uses a randomly initialized model for training.
Revised: First, for both IID and non-IID scenarios, it can be seen that FedAvg yields signiﬁcantly lower accuracy compared to others, since it uses a randomly initialized model for training.

Original: By pre-training the model for 3 . 2 × 10 4 communication rounds, we observe that FedAvg-based ﬁne-tuning gives signiﬁcant performance gains compared to naive application of FedAvg.
Revised: By pre-training the model for 3 . 2 × 10 4 communication rounds, we observe that FedAvg-based ﬁne-tuning gives signiﬁcant performance gains compared to naive application of FedAvg.

Original: The ﬁne-tuning scheme based on one-shot federated learning shows further performance improvements in the IID setup.
Revised: The ﬁne-tuning scheme based on one-shot federated learning shows further performance improvements in the IID setup.

Original: However, since K = 10 clients are sampled from 20 clients for this ensemble method, there possibly exist some unseen classes when building the global model/prototype in the non-IID setup, which lowers the performance compared to ﬁne-tuned FedAvg.
Revised: However, since K = 10 clients are sampled from 20 clients for this ensemble method, there possibly exist some unseen classes when building the global model/prototype in the non-IID setup, which lowers the performance compared to ﬁne-tuned FedAvg.

Original: Our few-round learning algorithm performs the best, with the distance-based classiﬁer showing better accuracy compared to the linear classiﬁer.
Revised: Our few-round learning algorithm performs the best, with the distance-based classiﬁer showing better accuracy compared to the linear classiﬁer.

Original: It can be also seen that the performance of the global model can be further improved by global  prototype-assisted learning with the auxiliary loss.
Revised: It can be also seen that the performance of the global model can be further improved by global [Table] [Table] prototype-assisted learning with the auxiliary loss.

Original: Fig.
Revised: Fig.

Original: 2 shows how the ﬁnal test accuracy (after 3 ﬁxed rounds) in the deployment phase increases as the number of communication rounds in the meta-training (or pre-training) phase grows.
Revised: 2 shows how the ﬁnal test accuracy (after 3 ﬁxed rounds) in the deployment phase increases as the number of communication rounds in the meta-training (or pre-training) phase grows.

Original: The overall results in Table 1 and Fig.
Revised: The overall results in Table 1 and Fig.

Original: 2 conﬁrm the[Table] [Table] advantage of exploiting meta-learning and global prototype-assisted learning ideas to facilitate fewround federated learning.
Revised: 2 conﬁrm the advantage of exploiting meta-learning and global prototype-assisted learning ideas to facilitate fewround federated learning.

Original: Experimental results in a random-way setup.
Revised: Experimental results in a random-way setup.

Original: In Table 2, we provide test accuracies in a more general random-way setup.
Revised: In Table 2, we provide test accuracies in a more general random-way setup.

Original: To construct each episode in the meta-training phase, τ classes are randomly and independently sampled from the train sets, where τ follows a discrete uniform distribution in the range [3 , 7] .
Revised: To construct each episode in the meta-training phase, τ classes are randomly and independently sampled from the train sets, where τ follows a discrete uniform distribution in the range [3 , 7] .

Original: The same procedure of random τ sampling is also applied during the test phase.
Revised: The same procedure of random τ sampling is also applied during the test phase.

Original: As in the 5-way setup, we sample 120 images from each class to construct an episode, having a total of 120 τ samples.
Revised: As in the 5-way setup, we sample 120 images from each class to construct an episode, having a total of 120 τ samples.

Original: In the IID setup, each client has 12 samples for τ classes.
Revised: In the IID setup, each client has 12 samples for τ classes.

Original: In the non-IID setup, we again divide the sorted data into 20 shards of size 6 τ and randomly allocate 2 shards to each client.
Revised: In the non-IID setup, we again divide the sorted data into 20 shards of size 6 τ and randomly allocate 2 shards to each client.

Original: To meta-train the linear classiﬁers in this random-way setup, we ﬁx the output size of last linear layer to 7 and exploit perceptron from the beginning, according to the selected τ value.
Revised: To meta-train the linear classiﬁers in this random-way setup, we ﬁx the output size of last linear layer to 7 and exploit perceptron from the beginning, according to the selected τ value.

Original: The results are consistent with the one in the 5-way setup, conﬁrming signiﬁcant advantages of our few-round learning algorithm even in a more general setup.
Revised: The results are consistent with the one in the 5-way setup, conﬁrming signiﬁcant advantages of our few-round learning algorithm even in a more general setup.

Original: Experimental results with mismatch on R .
Revised: Experimental results with mismatch on R .

Original: In Table 3, we show the results for mismatched R .
Revised: In Table 3, we show the results for mismatched R .

Original: The model is meta-trained targeting R = 3 as in Tables 1 and 2, but the actual test accuracies are evaluated after R = 1 communication round in the test phase.
Revised: The model is meta-trained targeting R = 3 as in Tables 1 and 2, but the actual test accuracies are evaluated after R = 1 communication round in the test phase.

Original: Since the global model is obtained after only one global round, the accuracies are lower than in Table 1.
Revised: Since the global model is obtained after only one global round, the accuracies are lower than in Table 1.

Original: Overall, it can be seen that our few-round learning algorithm is still powerful in this mismatch scenario.
Revised: Overall, it can be seen that our few-round learning algorithm is still powerful in this mismatch scenario.

Original: Testing on mismatch with
Revised: Testing on mismatch with

Original: R = 2 , as provided in Supplementary Materials, also shows consistent results.
Revised: R = 2 , as provided in Supplementary Materials, also shows consistent results.

Original: Comparison with personalization scheme.
Revised: Comparison with personalization scheme.

Original: We note that our formulation targets creating a global model while the previous works on federated meta-learning (Lin et al., 2020; Chen et al., 2018) aim at personalized local models.
Revised: We note that our formulation targets creating a global model while the previous works on federated meta-learning (Lin et al., 2020; Chen et al., 2018) aim at personalized local models.

Original: Given these different goals, in a non-IID setup, our method can generally handle broader classes of data than existing personalization approaches.
Revised: Given these different goals, in a non-IID setup, our method can generally handle broader classes of data than existing personalization approaches.

Original: However, there is a simple way to obtain a global model based on these personalization models, in case a need arises for a globalized model after-the-fact: just run rounds of local updates and aggregations starting from the local models but using data now collected across interested clients.
Revised: However, there is a simple way to obtain a global model based on these personalization models, in case a need arises for a globalized model after-the-fact: just run rounds of local updates and aggregations starting from the local models but using data now collected across interested clients.

Original: We were curious about how this globalized model would fare.
Revised: We were curious about how this globalized model would fare.

Original: In Supplementary Materials, we added experimental results for this scheme.
Revised: In Supplementary Materials, we added experimental results for this scheme.

Original: A quick conclusion is that the performance is comparable to the best ﬁne-tuning methods but lags well behind our methods targeting a global model from the get-go.
Revised: A quick conclusion is that the performance is comparable to the best ﬁne-tuning methods but lags well behind our methods targeting a global model from the get-go.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposedfew-round learning for federated learning, which enables construction of a highaccuracy global model within only a few communication rounds between the server and the clients.
Revised: We proposed a meta-learning strategy that enables few-round learning.

Original: Given a set of clients with new tasks, our meta-trained model generalizes well, within only a few  rounds of federated learning.
Revised: Given a set of clients with new tasks, our meta-trained model generalizes well, within only a few communication rounds be-

Original:  Extensive experimental results conﬁrm the signiﬁcant advantagesof our method over different baselines in both IID and non-IID datadistribution setups.
Revised: tween the server the clients. Extensive experimental results conﬁrm the signiﬁcant advantages data

Original: A A DDITIONAL E XPERIMENTS U NDER M ISMATCH ON R
Revised: A A DDITIONAL E XPERIMENTS U NDER M ISMATCH ON R

Original: In Table A.
Revised: In Table A.

Original: [Table]
Revised: [Table]

Original: we show additional experimental results on a mismatch scenario in the 5-way setup, where the model is meta-trained targeting R = 3 but tested after only R = 2 communication rounds in the deployment phase.
Revised: we show additional experimental results on a mismatch scenario in the 5-way setup, where the model is meta-trained targeting R = 3 but tested after only R = 2 communication rounds in the deployment phase.

Original: Our few-round learning algorithm consistently achieves higher accuracies compared to other baselines this mismatch scenario
Revised: Our few-round learning algorithm consistently achieves higher accuracies compared to other baselines this mismatch scenario

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B A DDITIONAL E XPERIMENTS WITH T ARGET R =
Revised: B A DDITIONAL E XPERIMENTS WITH T ARGET R =

Original: While the results shown in the main manuscript utilized the model targeting R = 3 , here we show results with target R = 2 , in the same 5-way setup.
Revised: While the results shown in the main manuscript utilized the model targeting R = 3 , here we show results with target R = 2 , in the same 5-way setup.

Original: The model is meta-trained with R = 2 .
Revised: The model is meta-trained with R = 2 .

Original: The accuracies are evaluated after R = 2 communication rounds in Table B.
Revised: The accuracies are evaluated after R = 2 communication rounds in Table B.

Original: [Table]
Revised: [Table]

Original: and evaluated after R =communication round in Table B.2.
Revised: and evaluated after R =communication round in Table B.2.

Original: Overall results show that our few-round learning algorithm outperforms other baselines in this R = 2 case as well.
Revised: Overall results show that our few-round learning algorithm outperforms other baselines in this R = 2 case as well.

Original: Note that Table B.1 below and Table A.1 above evaluate accuracies after R = 2 global rounds, while the models are meta-trained targeting R = 2 and R = 3 , respectively.
Revised: Note that Table B.1 below and Table A.1 above evaluate accuracies after R = 2 global rounds, while the models are meta-trained targeting R = 2 and R = 3 , respectively.

Original: Comparing these results, we can say that meta-training using a higher R does not provide performance advantage in the deployment (testing) phase.
Revised: Comparing these results, we can say that meta-training using a higher R does not provide performance advantage in the deployment (testing) phase.

Original: Hence, given a constraint on R in the deployment time, there is no need to perform “higher-round training”, which saves the communication resources utilized in the meta-training phase
Revised: Hence, given a constraint on R in the deployment time, there is no need to perform “higher-round training”, which saves the communication resources utilized in the meta-training phase

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: C C OMPARISON WITH THE P ERSONALIZATION SCHEME
Revised: C C OMPARISON WITH THE P ERSONALIZATION SCHEME

Original: In this section, we provide additional experimental results to compare our few-round learning algorithm with the personalization schemes (Lin et al., 2020; Chen et al., 2018).
Revised: In this section, we provide additional experimental results to compare our few-round learning algorithm with the personalization schemes (Lin et al., 2020; Chen et al., 2018).

Original: For the personalization scheme, the clients perform local meta-updates with their query sets right after the local updates with the support sets, in the meta-training phase.
Revised: For the personalization scheme, the clients perform local meta-updates with their query sets right after the local updates with the support sets, in the meta-training phase.

Original: For testing, local updates and aggregations are repeated for R global rounds to construct a global model after-the-fact.
Revised: For testing, local updates and aggregations are repeated for R global rounds to construct a global model after-the-fact.

Original: Tables C.1 and C.2 compare our scheme with the personalization scheme combined with global aggregation, in a 5-way setup and a random-way (3 ∼ 7) setup, respectively.
Revised: Tables C.1 and C.2 compare our scheme with the personalization scheme combined with global aggregation, in a 5-way setup and a random-way (3 ∼ 7) setup, respectively.

Original: The detailed settings are exactly the same as in the main manuscript.
Revised: The detailed settings are exactly the same as in the main manuscript.

Original: The personalization scheme combined with global aggregation achieves comparable performance with best ﬁne-tuning methods in the main manuscript, but falls well below our schemes geared to few-round FL.
Revised: The personalization scheme combined with global aggregation achieves comparable performance with best ﬁne-tuning methods in the main manuscript, but falls well below our schemes geared to few-round FL.

Original: D A DDITIONAL E XPERIMENTS WITH LESS DATA SAMPLES AT EACH C LIENT
Revised: D A DDITIONAL E XPERIMENTS WITH LESS DATA SAMPLES AT EACH C LIENT

Original: So far, we sampled 120 images from each class to construct an episode.
Revised: So far, we sampled 120 images from each class to construct an episode.

Original: In this section, we observe the effect of reducing the number of data samples in each client.
Revised: In this section, we observe the effect of reducing the number of data samples in each client.

Original: Instead of sampling 120 images, we now sample 60 images from each class to construct an episode.
Revised: Instead of sampling 120 images, we now sample 60 images from each class to construct an episode.

Original: In a 5-way setup, 60 · 5/10 =samples are allocated to each client.
Revised: In a 5-way setup, 60 · 5/10 =samples are allocated to each client.

Original: In the IID setup, the data samples from each class are equally distributed to K = 10 clients; each client has 6 samples for each of 5 classes, having a total ofsamples.
Revised: In the IID setup, the data samples from each class are equally distributed to K = 10 clients; each client has 6 samples for each of 5 classes, having a total ofsamples.

Original: For each class, 3 samples are utilized as support samples and other 3 samples are utilized as query samples.
Revised: For each class, 3 samples are utilized as support samples and other 3 samples are utilized as query samples.

Original: In the non-IID setup, we again divide the sorted data into 20 shards of size 15 and randomly allocate 2 shards to each client.
Revised: In the non-IID setup, we again divide the sorted data into 20 shards of size 15 and randomly allocate 2 shards to each client.

Original: Table D.1 shows the test accuracies in this setup.
Revised: Table D.1 shows the test accuracies in this setup.

Original: Again we consider a 5-way setup, and the model is meta-trained targeting R = 3 .
Revised: Again we consider a 5-way setup, and the model is meta-trained targeting R = 3 .

Original: The accuracies are evaluated after R = 3 rounds of federated learning.
Revised: The accuracies are evaluated after R = 3 rounds of federated learning.

Original: Since we utilize less data samples for training, the performances are degraded compared to the results in the main manuscript.
Revised: Since we utilize less data samples for training, the performances are degraded compared to the results in the main manuscript.

Original: But, overall, the trend is consistent with the results we observed so far, conﬁrming the advantage of our few-round learning algorithm utilizing meta-learning.
Revised: But, overall, the trend is consistent with the results we observed so far, conﬁrming the advantage of our few-round learning algorithm utilizing meta-learning.

Revised: E A DDITIONAL E XPERIMENTS ON CIFAR-

Revised: To conﬁrm the applicability of our proposed scheme further, we provide additional experiments on CIFAR-100 in Table.

Revised: E.1.

Revised: Similar to mini ImageNet, we split 100 classes in CIFAR-100 into

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: training, 16 validation and 20 test classes.

Revised: The results are consistent with the tables on other datasets, conﬁrming the advantage of our approach

Revised: [Table]

Revised: F A DDITIONAL E XPERIMENTS WITH A LARGER NUMBER OF CLIENTS

Revised: To demonstrate the scalability of our proposed method

Revised: [Table]

Revised: we performed additional experiments with

Revised: K = 50 clients.

Revised: Table F.1 shows the details, which indicates that our scheme outperforms other baselines in a larger scale federated learning system

Revised: [Table]

Revised: [Table]