Paper: Rethinking the Starting Point: Enhancing Performance and Fairness of Federated Learning via Collaborative Pre-Training (ICLR-2024)

Reviewer comment (Evaluation):
The observation of centralized pretraining causes side effects like performance biases are not well analyzed/explained.

Author response:
>The centralized dataset is constructed by aggregating datasets from all clients during the pre-training phase. As evidenced in Tables 5, 38, 39, 40, and additional experiments provided below, centralized pre-training can introduce side effects such as performance biases and large variance. This phenomenon arises from the lack of generalizability in the model's design. When a model undergoes pre-training in a centralized manner based on SGD, it becomes rigidly bound to the knowledge in the pre-training dataset. This fixity poses a challenge in adapting the model to the diverse clients that may contain new or unseen data in downstream tasks, which arise due to the time-varying environment or new clients as described in our first response above. As mentioned, the key clarification is that the datasets used in the pre-training and downstream FL phases are disjoint, and the set of clients involved in these two phases is also not the same. 
>
>&nbsp;
>
>Additionally, since our downstream tasks involve FL, the absence of learning dynamic distributed setups during the pre-training phase can further magnify this issue. Consequently, the model exhibits performance inconsistencies across different client datasets, leading to heightened performance variance. Our CoPreFL tackles these challenges by meta-updating the temporary global model to mimic FL scenarios that clients may encounter during downstream tasks. This enables our initial model to have generalization capability to arbitrary downstream FL scenarios. We have made these points clearer in Section I of the revised manuscript.

Reviewer comment (Theory):
The idea of using meta-learning to simulate distributed scenarios may not make sense for non-iid settings. If the clients' data are non-iid distributed, using local data for pretraining may not help. The local simulation cannot represent the global distributions.

Author response:
### **Weakness 2: Pre-training in non-IID setups**

In our CoPreFL, each client’s local dataset does not need to necessarily represent the global distribution. As the reviewer understands correctly, our key idea is to use meta-learning to tackle downstream FL. Since the clients conduct meta-update on the global model using their own local dataset (which could be non-IID) and the server aggregates all the meta-updated models, our model is actually **learning to perform well across all  clients’ local datasets in the system**. This enables our pre-trained model to become a good initialization for downstream tasks; the constructed global model (starting from our initial model) performs well across all clients in the system during downstream. This is also confirmed via our extensive experimental results throughout the manuscript.

Reviewer comment (Experiments):
The meta-learning method part is not clear. The data split needs to be clarified.

Author response:
> We appreciate the reviewer for this comment. We would like to first make the following two points clearer:  
>  
> 1. In our setting, clients do not intentionally split their local dataset into pre-training dataset and downstream dataset: The clients participating during pre-training actually use all their local data samples to construct a pre-trained model that can adapt to any downstream FL tasks. This pre-training stage can be viewed as a “preparation phase” where clients use all their current data to *get prepared for various circumstances they may encounter in the future* during the downstream task. From the service provider’s perspective, as depicted in Fig. 1 of the manuscript, the pre-training stage can be viewed as a process to prepare a good product (i.e., a generalized initial model) that can be provided to any potential customers in the future (i.e., any group of clients in future downstream tasks).  
>   
> 2. The pre-training stage and the downstream stage share the same task domain (e.g., image classification for self-driving cars, disease predictions for patients), but their specific objective/task could be potentially different due to the **time-varying environment** or **new clients (that share the same task domain but with different objectives) joining the system**, e.g., classifying different objects depending on the self-driving car’s environment, classifying different diseases depending on the patient’s interest, or face/speech recognition for new phone users. Here, the clients participating during the downstream task could be either the one who participated during pre-training or the one who didn’t attend the pre-training stage but have become interested in the related task. For example, a self-driving car who participated during pre-training might become interested in another classification task depending on their location, weather condition, or other time-varying environments, which will make them to get interested in unseen classes. Here, even when clients observe unseen classes, they can still benefit from the pre-trained model that is pre-trained with the same task domain, rather than starting from a randomly initialized model. As another example, a new patient who didn’t participate during pre-training may want to take advantage of the pre-trained model for solving its own task with new diseases, instead of starting from a randomly initialized model.  
>   
> Under this scenario, our main question was: Can we design a pre-training methodology that can generalize well to any downstream FL tasks? Here, we would like to highlight that the downstream tasks are unknown during pre-training, and the downstream tasks may contain unseen classes, due to the time-varying scenarios and new clients joining the system as mentioned above. Note that if a specific downstream task is known during pre-training (i.e., if the set of classes for the downstream task is known), and if all those classes are available during pre-training, it is definitely possible to construct a specific pre-trained model tailored to that task by only utilizing the data samples that belong to those classes. However, this (i) requires the service provider to prepare different pre-trained models for every downstream task, which could be potentially infinite, and also (ii) requires the clients in the pre-training stage to contain all classes that will appear in any future downstream tasks, which is again impractical. 
>
>&nbsp;
>
>Due to these challenges, given the current set of clients, the goal of the service provider would be to construct a “generalized pre-trained model” that can **quickly adapt to any, and possibly unseen future downstream tasks**. This is the goal of this paper, and we propose a meta-learning-based pre-training algorithm to mimic and generalize well to any unseen future downstream tasks. During the meta-learning process of our scheme, each client splits its local dataset into support set and query set to “mimic” downstream scenarios with potentially unseen classes, but we note that the clients are still taking advantage of all data samples for pre-training: The support sets are used to construct a temporary global model while the query sets are utilized for meta-update, which enables the clients to obtain a generalized initial model when pre-training is completed.   (to be continued in the next response)

Reviewer comment (Novelty):
The technical contribution of applying meta-learning for a new distributed pretraining setting might be marginal.

Author response:
### **Weakness 4: Contribution over other meta-learning based methods**

Some personalized FL works (e.g., FedMeta) also adopt meta-learning. We clearly differentiate our work with existing strategies in Sections I and II of our manuscript. Here, we provide a more detailed response.


 The distinction between our CoPreFL and meta-learning based FL works (e.g., FedMeta) lies in the application, which requires us to conduct meta-update in a totally different way. FedMeta is designed to offer personalization to individual clients. In contrast, our emphasis is on creating an initial model that can construct a good “global model” during downstream instead of “personalized models”. This is the reason why we need to update the temporary global model instead of the local models, which is the key technical difference with FedMeta. Another technical difference is the linear combination between fairness and average performance term, as the reviewer understands correctly. These two key techniques enables CoPreFL to construct a robust initial model that can quickly adapt to “any group of clients” (instead of individual clients) to construct a global model during downstream tasks. The advantages of CoPreFL can be also confirmed via extensive experiments throughout the manuscript.

Again, thank you for your time and efforts. We would appreciate further opportunities to answer any remaining concerns you might have.

Reviewer comment (Evaluation):
Compared with using public datasets for local pretraining, this method suffers further communication costs.

Author response:
### **Weakness 5: Communication cost**

Thanks for this comment. As mentioned in our response to the “public dataset above”, even when the public dataset is given, the central server can still adopt our CoPreFL to further enhance the performance (ImageNet1K - Centralized vs. ImageNet1K - CoPreFL).  Since the public dataset is centralized, CoPreFL can be also conducted at the server using multiple machines. 	The communication load between these machines in the server is the cost for achieving a better performance compared to centralized SGD.

Conducting pre-training in a FL setup would become more important for the tasks in which the centralized public dataset is not large enough due to privacy issues. In healthcare applications, e.g., disease diagnosis, the available public datasets are not large enough to construct a strong pre-trained model that can generalize well, due to the privacy issues of collecting clients’ datasets. In such applications, different patients or hospitals can collaboratively pre-train the model based on their local datasets via FL, to construct a more robust, generalized initial model that can quickly adapt to any unseen downstream FL tasks. The communication cost in this process is a must  to construct a strong pre-trained model.

Reviewer comment (Experiments):
The meta-learning part is not clear. How to split the unseen task data and the client data for pretraining? The assumption that the downstream tasks are unseen may not be proper enough. For example, if a client joins FL, it usually has specific goals, and the client data will be task-specific. The clients have no motivation to hold unrelated data for pretraining. And it is also weird to keep the task-specific (downstream) data unseen.

Author response:
> We appreciate the reviewer for this comment. We would like to first make the following two points clearer:  
>  
> 1. In our setting, clients do not intentionally split their local dataset into pre-training dataset and downstream dataset: The clients participating during pre-training actually use all their local data samples to construct a pre-trained model that can adapt to any downstream FL tasks. This pre-training stage can be viewed as a “preparation phase” where clients use all their current data to *get prepared for various circumstances they may encounter in the future* during the downstream task. From the service provider’s perspective, as depicted in Fig. 1 of the manuscript, the pre-training stage can be viewed as a process to prepare a good product (i.e., a generalized initial model) that can be provided to any potential customers in the future (i.e., any group of clients in future downstream tasks).  
>   
> 2. The pre-training stage and the downstream stage share the same task domain (e.g., image classification for self-driving cars, disease predictions for patients), but their specific objective/task could be potentially different due to the **time-varying environment** or **new clients (that share the same task domain but with different objectives) joining the system**, e.g., classifying different objects depending on the self-driving car’s environment, classifying different diseases depending on the patient’s interest, or face/speech recognition for new phone users. Here, the clients participating during the downstream task could be either the one who participated during pre-training or the one who didn’t attend the pre-training stage but have become interested in the related task. For example, a self-driving car who participated during pre-training might become interested in another classification task depending on their location, weather condition, or other time-varying environments, which will make them to get interested in unseen classes. Here, even when clients observe unseen classes, they can still benefit from the pre-trained model that is pre-trained with the same task domain, rather than starting from a randomly initialized model. As another example, a new patient who didn’t participate during pre-training may want to take advantage of the pre-trained model for solving its own task with new diseases, instead of starting from a randomly initialized model.  
>   
> Under this scenario, our main question was: Can we design a pre-training methodology that can generalize well to any downstream FL tasks? Here, we would like to highlight that the downstream tasks are unknown during pre-training, and the downstream tasks may contain unseen classes, due to the time-varying scenarios and new clients joining the system as mentioned above. Note that if a specific downstream task is known during pre-training (i.e., if the set of classes for the downstream task is known), and if all those classes are available during pre-training, it is definitely possible to construct a specific pre-trained model tailored to that task by only utilizing the data samples that belong to those classes. However, this (i) requires the service provider to prepare different pre-trained models for every downstream task, which could be potentially infinite, and also (ii) requires the clients in the pre-training stage to contain all classes that will appear in any future downstream tasks, which is again impractical. 
>
>&nbsp;
>
>Due to these challenges, given the current set of clients, the goal of the service provider would be to construct a “generalized pre-trained model” that can **quickly adapt to any, and possibly unseen future downstream tasks**. This is the goal of this paper, and we propose a meta-learning-based pre-training algorithm to mimic and generalize well to any unseen future downstream tasks. During the meta-learning process of our scheme, each client splits its local dataset into support set and query set to “mimic” downstream scenarios with potentially unseen classes, but we note that the clients are still taking advantage of all data samples for pre-training: The support sets are used to construct a temporary global model while the query sets are utilized for meta-update, which enables the clients to obtain a generalized initial model when pre-training is completed.   (to be continued in the next response)

Reviewer comment (Experiments):
The experimental settings are also not practical. First, clients in FL usually may not hold 'new classes' for pretraining tasks; the client data are strongly related to the task in general. Second, for real-world FL, client local data are valuable and cost a lot to annotate. It does not make sense to split out part of classes for pre-training; instead, using as much data as possible to promote local performance is important.

Author response:
### **Comments regarding dataset splitting and unseen classes**

We also appreciate this comment. We agree with the reviewer that from the client’s viewpoint,  the main goal would  be to prepare a model for current classes, rather than "future classes". Our problem setup is more important from the **service-provider’s perspective**. From the service provider's viewpoint, once a robust pre-trained model is constructed, the service provider can sell/provide this model as an initial model to **any group of clients that aims to construct a global model via FL**, in future downstream tasks (Fig. 1 in the manuscript). Our main question was: Can we construct this kind of robust pre-trained model using the currently available clients? The service provider will use the current clients as a tool to construct a pre-trained model, and will give incentives to the clients to participate in the pre-training stage.

The service provider is willing to prepare this  especially considering  new clients joining the system (e.g., new self-driving cars joining a specific region [Badue et al]), which can introduce unseen classes.  Moreover, the clients’ needs  can also vary over time (e.g, as in continual FL settings [ICML’21], [CVPR’22]), and the clients may want to construct  a global FL model at a specific timestep.  These scenarios can be viewed as our key applications, and our work is suggesting the clients in these future downstream tasks to  conduct  FL  starting from our pre-trained model, instead of starting from a randomly initialized model or other pre-trained models.  **Overall, the ultimate goal of the service provider is to construct a strong pre-trained model that can adapt to any of these kinds of future downstream scenarios**. We have added the service provider's viewpoint in Section I of the manuscript.

 
[Badue et al.,] ”Self-driving cars: A survey." Expert Systems with Applications 165 (2021): 113816.

[ICML’21] Federated Continual Learning with Weighted Inter-client Transfer

[CVPR’22] Federated Class-Incremental Learning

Reviewer comment (Evaluation):
Why does the Non-IID setting present higher accuracy than the IID setting? When data becomes Non-IID, the model should suffer a performance drop.

Author response:
### **Better results in Non-IID scenarios**  
>Intriguingly, we observed this notable phenomenon in our experiments, when applying initializations pre-trained on different data distribution, the non-IID downstream tasks exhibited higher accuracy than the IID setting in some cases. This results prompted us to delve deeper into the implications of using an initialization pre-trained on a diverse distribution. One plausible explanation for the higher accuracy in the non-IID setting could be attributed to the fact that pre-training in a non-IID manner allows the model to capture heterogeneity information within the data. This learned heterogeneity information may exhibit a certain degree of similarity with the distribution of the downstream tasks during the subsequent phases, thereby contributing to the improved performance observed in some non-IID scenarios. This finding raises interesting questions about the interplay between pre-training strategies and downstream task distributions, suggesting that non-IID pre-training might facilitate better adaptation to certain non-IID downstream tasks.

Reviewer comment (Experiments):
In Table 5, the centralized pretraining is based on a centralized dataset from all the clients; why does this introduce significant performance variance? If the centralized data are from all clients, the pretraining should present less variance than random initialization. Also, it needs to be clarified how to build the centralized dataset.

Author response:
>The centralized dataset is constructed by aggregating datasets from all clients during the pre-training phase. As evidenced in Tables 5, 38, 39, 40, and additional experiments provided below, centralized pre-training can introduce side effects such as performance biases and large variance. This phenomenon arises from the lack of generalizability in the model's design. When a model undergoes pre-training in a centralized manner based on SGD, it becomes rigidly bound to the knowledge in the pre-training dataset. This fixity poses a challenge in adapting the model to the diverse clients that may contain new or unseen data in downstream tasks, which arise due to the time-varying environment or new clients as described in our first response above. As mentioned, the key clarification is that the datasets used in the pre-training and downstream FL phases are disjoint, and the set of clients involved in these two phases is also not the same. 
>
>&nbsp;
>
>Additionally, since our downstream tasks involve FL, the absence of learning dynamic distributed setups during the pre-training phase can further magnify this issue. Consequently, the model exhibits performance inconsistencies across different client datasets, leading to heightened performance variance. Our CoPreFL tackles these challenges by meta-updating the temporary global model to mimic FL scenarios that clients may encounter during downstream tasks. This enables our initial model to have generalization capability to arbitrary downstream FL scenarios. We have made these points clearer in Section I of the revised manuscript.

Reviewer comment (Theory):
The paper lacks theoretical performance/fairness guarantees or analyses of the proposed algorithm.

Author response:
### **Performance Guarantee** 

>We agree with the reviewer that a theoretical guarantee of the algorithm would make the paper stronger. We actually found this extremely challenging since it requires us to formally show “how the initial model constructed by each pre-trained model” affects the performance of the “global model constructed during downstream FL”. The authors of [1], which studies centralized pre-training for downstream FL, show that pre-trained model can help since it can potentially reduce the initial loss value in the beginning of downstream FL. However, when considering a general scenario involving arbitrary unseen downstream tasks, this might not always be the truth. Due to the challenge of providing exact theory, we aimed to consider as many baselines as possible to strengthen the empirical results. Since there were not many baselines we could use for pre-training, we explored various federated learning algorithms (including those utilizing meta-learning differently from ours) and centralized training for pre-training baselines. We evaluated the performance of these schemes over different data distribution setups for pre-training and downstream, and validated the superiority of our scheme. Nevertheless, we can still explain the superior performance of our scheme as follows: (i) We are adopting meta-learning, which is known to effectively handle unseen tasks, and (ii) we are taking advantage of meta-learning tailored to downstream FL by meta-updating the temporary global model considering both average performance and fairness, which enables us to construct a generalized initial model that leads to a strong global model. We believe that further investigating FL pertaining with theoretical analysis is a promising future research direction. 
>   
>&nbsp;   
> 
>[1] Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning, Nguyen et al. 
 

 
 

We again thank the reviewer for reviewing our paper and providing very helpful comments. We would appreciate further opportunities to answer any remaining concerns you might have.

Reviewer comment (Experiments):
The practicability of the method requires further justification, in terms of the number of tasks and amount of data required.

Author response:
### **Number of tasks during downstream in CoPreFL** 

> We appreciate the reviewer for pointing this out. Yes, we considered 5-way classification *during downstream FL to model various unseen downstream scenarios*. As mentioned in our second response above, our goal is to design a “generalized initial model” that can adapt to arbitrary downstream tasks that potentially contain unseen classes. To model unseen downstream scenarios, during experiments, we split the dataset into a pre-training dataset and a downstream dataset, allocating 80 and 20 classes for CIFAR-100 dataset, respectively. Here, **to prove that our scheme works well for “arbitrary” downstream scenarios, we had to generate different FL scenarios during downstream (i.e., by adopting 5-way classification) and measure the averaged performance over different downstream scenarios.** This allows us to assess whether the pre-trained model, generated using CoPreFL or other baselines, can offer effective initialization for a diverse range of downstream tasks. The goal is to evaluate the versatility of the pre-trained model in providing a robust starting point for various scenarios. However, this does not indicate that the number of downstream should be large in CoPreFL. To prove this, we now consider a “single downstream scenario“ with 20-way classification using all downstream classes (that have not appeared during pre-training). The results are provided below, indicating that CoPreFL still performs better than the baselines: 
> - Performance in a single-downstream scenario (Pre-training: Non-IID; Downstream: 20-way classification Non-IID FedAvg): 
> |Pre-training Method | Acc | Var | worst 10% | worst 20% | worst 30% 
>|--|--|--|--|--|--| 
>|FedAvg | 81.35 | 59.69| 70.35 | 70.91 | 71.63 
>|FedMeta| 82.69|46.38 |72.74|73.51|74.39 
>|q-FFL| 83.27|51.48|73.01|73.82|75.02 
>|CoPreFL | **84.71**|**39.26**|**73.28**|**74.99**|**75.35** 
>
> Here, it is also important to mention that **CoPreFL targets scenarios where the target downstream tasks are “unknown” during pre-training**. If we have prior information on the downstream task (100-way CIFAR-100 classification) during pre-training, and if these classes are available during pre-training, there is no reason to design the initial model to perform well on arbitrary downstream tasks by adopting meta-learning. In such cases, it might be more beneficial to adopt simple FedAvg during pre-training to capture a good data representation for downstream FL, as pre-training and downstream tasks share the same classes. The strength of our CoPreFL lies in the robustness to capture arbitrary and potentially unseen downstream scenarios (as depicted in Fig. 1 of the manuscript) which happen due to the time-varying local datasets of clients or new clients (that share the same task domain but with different objectives) joining the system.

Reviewer comment (Experiments):
Why is it the case that the existing works on federated meta learning for personalization are inapplicable to the problem setting of this paper? For the three papers (Jiang et al., 2019; Fallah et al., 2020; Collins et al., 2021) cited in the introduction, is it possible to simply replace the personalization stage of their algorithms with a federated global model training? If possible, can they be compared to as baselines in the experiment section?

Author response:
### **Comparison with other federated meta-learning algorithms** 
> Yes, as the reviewer pointed out, it possible to replace the personalization stage (of existing federated meta learning methods) with a federated global model training and adopt as a baseline. We have done this for FedMeta in our original submission. Based on the reviewer’s comment, during the rebuttal period, we have further implemented another personalized FL method (Fallah et al.). The final global model from this approach is employed as an initialization in our downstream scheme. Specifically, we implement the personalized baseline under a non-IID pre-training data distribution and subsequently use the global model of this method to initialize the downstream non-IID FL tasks. The experiment is conducted on CIFAR-100 dataset, following the pre-training/downstream partition in our paper. The results are provided below: 
> - Comparison with modification of PerFedAvg (Fallah et al.): 
>|Pre-training Method (Scenario I) | Acc | Var | worst 10% | worst 20% | worst 30% 
>|--|--|--|--|--|--| 
>NonIID-PerFedAvg | 81.58 |49.73 |67.21|71.05|72.39 
>NonIID-CoPreFL | **83.29** | **34.69** | **71.58** | **73.20** | **74.59**
> 
>The result further confirms the superiority of our method that meta-updates the temporary global model by balancing average performance and fairness. This table is now included in Section C.4 of the Appendix of the revised paper.

Reviewer comment (Experiments):
In scenario II, why is it reasonable to use the server’s data as the unseen dataset for downstream tasks? How to make sure the artificial tasks generated from the server data are a good representation of potential downstream tasks? The final goal of the algorithm is quick adaptation to the downstream tasks of clients, rather than the server. Is there a mismatch here for the optimization objective? Please justify this point since Scenario II is also an important claimed contribution of the paper.

Author response:
### **The utilization of server’s data to mimic unseen scenario, and partitioning server’s data** 

> We appreciate this comment. We would like to first highlight that we are not aiming to capture   representations of datasets in downstream FL. This is especially impossible when arbitrary and potentially new classes appear during downstream FL, which is the setup that we consider. In our setup, the pre-training stage and the downstream stage share the same task domain (e.g., image classification for self-driving cars, disease predictions for patients), but their specific objective/task could be potentially different due to the time-varying environment or new clients (that share the same task domain but with different objectives) joining the system, e.g., classifying different objects depending on the self-driving car’s environment, classifying different diseases depending on the patient’s interest, or face/speech recognition for new phone users. Here, the clients participating during the downstream task could be either the one who participated during pre-training or the one who didn’t attend the pre-training stage but have become interested in the related task. For example, a self-driving car who participated during pre-training might become interested in another classification task depending on their location, weather condition, or other time-varying environments, which will make them to get interested in unseen classes. Here, even when clients observe unseen classes, they can still benefit from the pre-trained model that is pre-trained with the same task domain, rather than starting from a randomly initialized model. To summarize, arbitrary “new classes” that have not appeared during pre-training may appear in downstream scenarios. 
>    
>&nbsp;    
> 
>Due to the above setting with arbitrary and potentially unseen classes, even the client’s local dataset may not be a good representation of the datasets of clients in downstream tasks. Meta-learning plays a key role here to tackle this challenge. Instead of directly capturing data representations for downstream, our initial model aims to learn “how to quickly adapt” to arbitrary unseen downstream FL scenarios, which is achieved by distinguishing between the support and query sets during pre-training. To facilitate this in Scenario II, we aimed to construct |m| different query sets based on the server’s data to “mimic downstream FL scenarios”, which is done by arbitrary partitioning. This is a natural choice since we are not able to know any patterns/information of downstream FL. What we have shown in Scenario II is that, even this arbitrary |m| partitioning of the server’s data can lead to significant performance improvements by mimicking a scenario that clients have different objectives/datasets in downstream scenarios. This can even improve the performance of the model compared to the one in Scenario I, as the clients can utilize all of its local dataset as the support in Scenario II.  These results further confirm the applicability and effectiveness of the proposed CoPreFL in Scenario II.

Reviewer comment (Experiments):
The artificial partition of the server’s dataset into $|m|$ equal partitions also seems to be lacking justification. Also, empirically speaking, how well does your method work as compared to using (all local support data + all server data) for normal FedAvg training as the pre-trained model?

Author response:
Thank you for your response and keeping the positive score. We would like to briefly make the point regarding equal partition clearer. Instead of equal partitioning, one can also divide the server-side dataset into partitions with unequal sizes and use them for  meta-update. However, during pre-training, the server does not know the dataset sizes of clients in future downstream tasks. In this case, one intuitive way is to treat all clients equally/fairly, by meta-updating the model with the same query set sizes. We show that this equal partitioning provides significant performance improvements as can be seen in our experiments.

Regarding the baselines, there were not many baselines we could consider due to the lack of works on pre-training for/in FL, and thus we tried our best to include most of the variations we can think of (e.g., FedMeta, q-FFL, centralized pre-training). Thanks again for your time and efforts.

Best, Authors

Reviewer comment (Experiments):
Can the authors briefly introduce FedMeta in the main text since it is an important baseline? Also, you probably need to highlight the difference between FedMeta and CoPreFL.

Author response:
### **Difference compared with FedMeta** 
>Thanks for reviewer’s suggestion. We now mention FedMeta in the related work section of the revised manuscript. We clarified the distinction between our method and FedMeta. A more detailed difference can be also found in the below answer: 
>   
>&nbsp;   
> 
> The distinction between our CoPreFL and FedMeta lies in the application, which required us to make the meta-update strategy different. Consider a healthcare application where each client, such as a hospital or an individual patient, aims to build a comprehensive global model capable of classifying a wide range of diseases. However, individual clients may possess limited types of diseases in their local datasets – for instance, one client may have data on diseases A and B but lacks information on diseases C and D. In this context, federated learning becomes essential even in the downstream tasks. Clients need to collaborate to construct a global model that not only reflects the diseases available locally but also incorporates information about diseases not present in their individual datasets, ensuring a more robust and universally applicable healthcare model. FedMeta is designed to offer personalization to individual clients, e.g., when a specific client is interested in predicting only diseases A and B. In contrast, our emphasis is on creating an initial model that can construct a good “global model” during downstream instead of “personalized models”, targeting the aforementioned applications. This made us to directly meta-update the temporary global model, which enables our CoPreFL to construct a robust initial model that can quickly adapt to “any group of clients” (instead of individual clients) to construct a global model during downstream tasks.

Reviewer comment (Experiments):
In reality, does the CoPreFL method require access to a large number of tasks to work well? For example, in the experiments of this paper, we only perform 5‑way classification for CIFAR‑100 data, which comprises 100 classes. I am guessing that this is done so that a variety of tasks could be generated. Let’s say that my final goal is to train a good and fair model on CIFAR‑100 now, how should I go about training the meta pre‑trained model? Could the authors clarify this point?

Author response:
### **Number of tasks during downstream in CoPreFL** 

> We appreciate the reviewer for pointing this out. Yes, we considered 5-way classification *during downstream FL to model various unseen downstream scenarios*. As mentioned in our second response above, our goal is to design a “generalized initial model” that can adapt to arbitrary downstream tasks that potentially contain unseen classes. To model unseen downstream scenarios, during experiments, we split the dataset into a pre-training dataset and a downstream dataset, allocating 80 and 20 classes for CIFAR-100 dataset, respectively. Here, **to prove that our scheme works well for “arbitrary” downstream scenarios, we had to generate different FL scenarios during downstream (i.e., by adopting 5-way classification) and measure the averaged performance over different downstream scenarios.** This allows us to assess whether the pre-trained model, generated using CoPreFL or other baselines, can offer effective initialization for a diverse range of downstream tasks. The goal is to evaluate the versatility of the pre-trained model in providing a robust starting point for various scenarios. However, this does not indicate that the number of downstream should be large in CoPreFL. To prove this, we now consider a “single downstream scenario“ with 20-way classification using all downstream classes (that have not appeared during pre-training). The results are provided below, indicating that CoPreFL still performs better than the baselines: 
> - Performance in a single-downstream scenario (Pre-training: Non-IID; Downstream: 20-way classification Non-IID FedAvg): 
> |Pre-training Method | Acc | Var | worst 10% | worst 20% | worst 30% 
>|--|--|--|--|--|--| 
>|FedAvg | 81.35 | 59.69| 70.35 | 70.91 | 71.63 
>|FedMeta| 82.69|46.38 |72.74|73.51|74.39 
>|q-FFL| 83.27|51.48|73.01|73.82|75.02 
>|CoPreFL | **84.71**|**39.26**|**73.28**|**74.99**|**75.35** 
>
> Here, it is also important to mention that **CoPreFL targets scenarios where the target downstream tasks are “unknown” during pre-training**. If we have prior information on the downstream task (100-way CIFAR-100 classification) during pre-training, and if these classes are available during pre-training, there is no reason to design the initial model to perform well on arbitrary downstream tasks by adopting meta-learning. In such cases, it might be more beneficial to adopt simple FedAvg during pre-training to capture a good data representation for downstream FL, as pre-training and downstream tasks share the same classes. The strength of our CoPreFL lies in the robustness to capture arbitrary and potentially unseen downstream scenarios (as depicted in Fig. 1 of the manuscript) which happen due to the time-varying local datasets of clients or new clients (that share the same task domain but with different objectives) joining the system.

Reviewer comment (Novelty):
This paper has focused on a setting where the pre-training dataset is not centralized. However, I wonder whether such a setting exists in any real-world application given so many open-sourced pre-trained models (e.g., Bert, GPT2, ViT, etc.) and large-scale public datasets (Common Crawl, Wikipedia, ImageNet, etc.) in different areas. Some real-world examples are needed to clarify the necessity of the FL pre-training stage.

Author response:
We appreciate the comment. Our responses can be summarized as follows:  
1. **Pre-training in a FL setup would be especially important for the tasks in which the centralized public dataset is not large enough due to privacy issues:** In healthcare applications, e.g., disease diagnosis, the available public datasets are not large enough to construct a strong pre-trained model that can generalize well, due to the privacy issues of collecting clients’ datasets. In such applications, different patients or hospitals can collaboratively pre-train the model based on their local datasets via FL, to construct a more robust, generalized initial model that can quickly adapt to any unseen downstream FL tasks.  

2. **More importantly, as mentioned in Remark 2 of our manuscript, our scheme is directly applicable to centralized public datasets to further improve the performance:** As mentioned in Remark 2,  the server can intentionally split the centralized dataset and apply our scheme (using multiple computing units available at the server) to obtain a pre-trained model. Here the question is, what is the advantage of doing this compared to simple centralized training? As can be seen from Table 5, 38, 39, 40, centralized pre-training can introduce side effects such as performance biases and large variance. This phenomenon arises from the lack of generalizability in the model's design. When a model undergoes pre-training in a centralized manner based on SGD, it becomes rigidly bound to the knowledge in the pre-training dataset. This fixity poses a challenge in adapting the model to the diverse clients that may contain new or unseen data in downstream tasks, which arise due to the time-varying environment or new clients joining the system, e.g., classifying different objects depending on the self-driving car’s environment, classifying different diseases depending on the patient’s interest, or face/speech recognition for new phone users. Additionally, since our downstream tasks involve FL, the absence of learning dynamic distributed setups during the pre-training phase can further magnify this issue. Consequently, the model exhibits performance inconsistencies across different client datasets, leading to heightened performance variance. Our CoPreFL tackles these challenges by meta-updating the temporary global model to mimic FL scenarios that clients may encounter during downstream tasks. This enables our initial model to have generalization capability to arbitrary downstream FL scenarios. We have made these points clearer in Section I of the revised manuscript.  

To confirm the advantages of our CoPreFL over centralized pre-training in the large-scale public dataset, we have included an additional experiment considering a scenario where a public dataset is available. We conduct pre-training using the ImageNet_1K dataset following Nguyen et al.'s idea. We sample 200 images for each of the 1000 classes in ImageNet_1K. During the pre-training phase, we trained the model using the SGD optimizer with a learning rate of 1e-3 for 50 epochs, and considered this centralized training result as a baseline. For our method, we distribute all the data across 100 nodes, sampling 20 nodes in each round, and conducted non-IID CoPreFL for 50 rounds. For the downstream FL phase, we apply ten non-IID FedAvg tasks to the 20-classes downstream dataset we used in CIFAR-100 dataset. As a result, all classes observed during downstream tasks are the seen classes that have already appeared during pre-training.

- Results using **public large-scale dataset** for pre-training (downstream on CIFAR-100):
|Dataset & Pre-training Method | Acc | Var| 
|--|--|--|
|CIFAR100 – Centralized| 81.30 | 69.44 |
|CIFAR100 – FedAvg| 78.96 | 64.80 |
| CIFAR100 – FedMeta | 82.45 | 48.72
| CIFAR100 – qFFL | 80.01 | 88.92
| CIFAR100 – CoPreFL | 83.29 | 34.69
| ImageNet1K - Centralized | 86.75 | 67.34
| ImageNet1K - CoPreFL| **87.96** | **30.79**

As we can expect, models pre-trained on ImageNet1K achieve higher accuracy compared to those pre-trained on CIFAR-100 since there is no unseen classes when pre-trained on ImageNet. More importantly, as mentioned, we can still apply our method by intentionally splitting the dataset and mimicking the distributed nature of downstream FL to achieve further performance improvements: The centrally pre-trained model on ImageNet achieves lower accuracy and higher variance compared to our CoPreFL. This advantage of CoPreFL is achieved by initializing the model to get higher accuracy and better fairness in federated settings based on meta-learning. The overall results further confirm the advantage and applicability of our approach. These new results are now included in Section C.4 and Table 41 of Appendix.

Reviewer comment (Experiments):
Even though the empirical results show that the proposed methods outperform the "Centralized" baseline, the centralized model tested in Table 5 is actually personally trained with a subset of data. There is no public large-scale pre-trained model tested in the experiments.

Author response:
We appreciate your response. We pre-trained the model for “ImageNet1K - Centralized” by ourselves to make a *fair comparison with “ImageNet1K - CoPreFL” under the same setting*. The sentence in the Appendix needs to be removed. We have corrected it accordingly. Our goal here is to highlight that even when the public dataset is given, we can always apply our CoPreFL to mimic downstream FL scenarios, to construct a more robust initial model that can quickly adapt to arbitrary and possibly unseen downstream tasks. If a large public dataset is not available due to privacy issues (e.g., medical applications) as mentioned above, we must rely on pre-training in distributed settings.

Please let us know if you have any further questions. Thank you.

Best, Authors

Reviewer comment (Experiments):
In Section 4, FedAvg is used in the fine-tuning stage in all the experiments. However, I think the comparison between CoPreFL+FedAvg and Centralized+FedAvg is not enough to show the superiority of the proposed method. Why must the fairness issue be tackled in the pre-training stage instead of the fine-tuning stage? How if I start from a public pre-trained model and do fine-tuning with any fairness-aware FL method (e.g., Centralized+FedProx)? Note that model pre-training is usually much more expensive than fine-tuning. Then, isn't it more sensible to consider the additional steps for optimizing (2) in the fine-tuning stage?

Author response:
We appreciate this comment. Based on the reviewer’s comment, we additionally consider FedProx, a more advanced FL algorithm that addresses heterogeneity compared to  FedAvg, to examine the robustness and generalizability of our pre-trained method. The results are provided in the table below. The additional experiments are conducted using the CIFAR-100 dataset. The FL methods in the pre-training phase are implemented under a non-IID data distribution, and we carry out 10 non-IID FedProx tasks for downstream evaluation. Here, we note that not only centralized pre-training but also other pre-training methods (including ours) can take advantage of FedProx during downstream tasks to further improve the performance. The results demonstrate that our pre-trained method maintains superiority in downstream FedProx compared to other pre-training methods. We would also like to note that the choice of FedAvg as our downstream task is made to minimize the varying impact introduced by other FL algorithms. Comparing the pre-training + downstream pairs, the improvement of CorPreFL+FedAvg over Centralized+FedProx shows that a better initialization, which considers the distributed scenario and balances fairness/performance in the pre-training phase, could potentially benefit the inferior downstream FL algorithm. These new results are now included in Appendix C.7. 

- Experiments with **applying FedProx in downstream FL**: 
|Pre-training Method + Fien-tuning Method | Acc | Var | worst 10% | worst 20% | worst 30% 
|--|--|--|--|--|--| 
|Centralized + FedProx  | 82.39 | 51.46 | 70.33 | 71.28 | 73.52 
|FedAvg + FedProx | 79.53 | 46.15 | 63.17 | 69.74 | 71.59 
|FedMeta + FedProx| 81.77 | 63.12 | 63.58 | 68.19 | 70.28 
|q-FFL + FedProx | 83.19 | 52.12 | 67.41 | 70.59 | 72.33 
|CoPreFL + FedProx | 84.31 | 30.55 |  70.19 |  73.88 | 75.13 
|CoPreFL + FedAvg | 83.29 | 34.69 | 71.58 | 73.20 | 74.59

Reviewer comment (Theory):
In Eqs. (1) and (2), what does the $\mathcal{G}$ denote? Does $p(\mathcal{G})$ represent the distribution of a single downstream FL task or all the downstream tasks? If $p(\mathcal{G})$ is the distribution of a single task $\mathcal{G}$, there should be an additional expectation over $\mathcal{G}$ for both (1) and (2). Otherwise, the variance equation in (2) is weird. Why did you compute the variance of loss over clients in different FL tasks? For different downstream tasks, the loss function $f$ might be even not comparable.

Author response:
In Eqs. (1) and (2), $p(\mathcal{G})$ represents the distribution of all possible sets of clients that appear in downstream. Hence, $\mathcal{G}$ is a random variable while G is a specific downstream task sampled from distribution $p(\mathcal{G})$. Based on these definitions, equations (1) and (2) can be also clearly defined without any issue.

Reviewer comment (Reproducibility):
In Algorithm 1 Line 17, how is the gradient computed on the server side with distributed data?

Author response:
The gradient is computed on the client side. The communication process involves the server acquiring a temporary aggregated model from the local model trained using the support set (line 10). The server then broadcasts this model to each client for meta-evaluation, where gradients are computed locally on each client (lines 11-14). Subsequently, clients send these gradients back to the server, which utilizes them to update the temporary aggregated model (line 17). We have made these points clearer in Section 3.2 of the revised manuscript.

Reviewer comment (Evaluation):
In Appendix B, the authors mentioned that they assessed various balancer values $\gamma$ in all scenarios and reported the best-performing value. How did you define the "best" here given the accuracy-fairness tradeoff?

Author response:
We report the results of our method, which achieved the best accuracy among all $\gamma$-configurations. Moreover, for fair selection/comparison, we also report the results of other baselines with their own best accuracy when searching for hyperparameters.

Reviewer comment (Novelty):
Algorithm 1 seems very similar to FedMeta [1] except for the objective which is a linear combination between a fairness term and a utility term.

Author response:
>**Key applications**: Consider a healthcare application where each client, such as a hospital or an individual patient, aims to build a comprehensive global model capable of classifying a wide range of diseases. However, individual clients may possess limited types of diseases in their local datasets – for instance, one client may have data on diseases A and B but lacks information on diseases C and D. In this context, federated learning becomes essential. Clients need to collaborate to construct a global model that not only reflects the diseases available locally but also incorporates information about diseases not present in their individual datasets, ensuring a more robust and universally applicable healthcare model. Similarly, in the domain of autonomous vehicles, each self-driving car may strive to develop a global model for scenario detection in various weather conditions. However, individual cars might encounter limited weather scenarios locally – one car might navigate through a desert environment, while another faces challenges in a snowy storm. Through federated learning, these cars can collectively construct a global model that accounts for a broad spectrum of weather conditions, ensuring robust scenario detection capabilities for all vehicles involved. The wide availability of datasets open to public also makes our Scenario II practical, as the server can hold these public datasets and meta-update the global model learned by clients. 
>   
>&nbsp;   
> 
> **Contributions beyond FedMeta**: The distinction between our CoPreFL and FedMeta lies in the application, which requires us to conduct meta-update in a totally different way.  FedMeta is designed to offer personalization to individual clients, e.g., when a specific client is interested in predicting only diseases A and B, or when a specific self-driving car is interested in the model tailored to a specific weather. In contrast, our emphasis is on creating an initial model that can construct a good “global model” during downstream instead of “personalized models”,  targeting the aforementioned applications. This is the reason why we need to update the temporary global model instead of the local models, which is the key technical difference with FedMeta. Another technical difference is the linear combination between fairness and average performance term, as the reviewer understands correctly. These two key techniques enables  CoPreFL to construct a robust initial model that can quickly adapt to “any group of clients” (instead of individual clients) to construct a global model during downstream tasks. The advantages of CoPreFL can be also confirmed via extensive experiments throughout the manuscript.
>   
>&nbsp;   
> 
>We have clarified the distinction with FedMeta in the related work section of the revised manuscript.

Reviewer comment (Theory):
CoPreFL currently lacks formal performance guarantees. This work would significantly benefit from a formal analysis of the proposed method(s) to provide a better understanding and confidence in their behaviour and their performance. Currently, evaluating the empirical results is challenging due to the nature of the baselines, which (to my understanding) have been adapted to accommodate pretraining and are not actual baselines (as the authors also identified) since they are optimized for entirely different scenarios.

Author response:
>We agree with the reviewer that a formal analysis would make the paper stronger. We actually found this extremely challenging since it requires us to formally show “how the initial model constructed by each pre-trained model” affects the performance of the “global model constructed during downstream FL”. The authors of [1], which studies centralized pre-training for downstream FL, show that pre-trained model can help since it can potentially reduce the initial loss value in the beginning of downstream FL. However, when considering a general scenario involving arbitrary unseen downstream tasks, this might not always be the truth.  Due to the challenge of providing exact theory, we aimed to consider as many baselines as possible to strengthen the empirical results. Since there were not many baselines we could use for pre-training, we explored various federated learning algorithms (including those utilizing meta-learning differently from ours) and centralized training for pre-training baselines.  We evaluated the performance of these schemes over different data distribution setups for pre-training and downstream, and validated the superiority of our scheme. Nevertheless, we can still explain the superior performance of our scheme as follows: (i) We are adopting meta-learning, which is known to effectively handle unseen tasks, and (ii) we are taking advantage of meta-learning tailored to downstream FL by meta-updating the temporary global model considering both average performance and fairness, which enables us to construct a generalized initial model that leads to a strong global model. We believe that further  investigating FL pertaining with theoretical analysis is a promising future research direction. 
>   
>&nbsp;   
> 
> [1] Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning, Nguyen et al.

Reviewer comment (Writing):
What are the algorithmic differences between FedMeta and algorithm 1?

Author response:
>**Key applications**: Consider a healthcare application where each client, such as a hospital or an individual patient, aims to build a comprehensive global model capable of classifying a wide range of diseases. However, individual clients may possess limited types of diseases in their local datasets – for instance, one client may have data on diseases A and B but lacks information on diseases C and D. In this context, federated learning becomes essential. Clients need to collaborate to construct a global model that not only reflects the diseases available locally but also incorporates information about diseases not present in their individual datasets, ensuring a more robust and universally applicable healthcare model. Similarly, in the domain of autonomous vehicles, each self-driving car may strive to develop a global model for scenario detection in various weather conditions. However, individual cars might encounter limited weather scenarios locally – one car might navigate through a desert environment, while another faces challenges in a snowy storm. Through federated learning, these cars can collectively construct a global model that accounts for a broad spectrum of weather conditions, ensuring robust scenario detection capabilities for all vehicles involved. The wide availability of datasets open to public also makes our Scenario II practical, as the server can hold these public datasets and meta-update the global model learned by clients. 
>   
>&nbsp;   
> 
> **Contributions beyond FedMeta**: The distinction between our CoPreFL and FedMeta lies in the application, which requires us to conduct meta-update in a totally different way.  FedMeta is designed to offer personalization to individual clients, e.g., when a specific client is interested in predicting only diseases A and B, or when a specific self-driving car is interested in the model tailored to a specific weather. In contrast, our emphasis is on creating an initial model that can construct a good “global model” during downstream instead of “personalized models”,  targeting the aforementioned applications. This is the reason why we need to update the temporary global model instead of the local models, which is the key technical difference with FedMeta. Another technical difference is the linear combination between fairness and average performance term, as the reviewer understands correctly. These two key techniques enables  CoPreFL to construct a robust initial model that can quickly adapt to “any group of clients” (instead of individual clients) to construct a global model during downstream tasks. The advantages of CoPreFL can be also confirmed via extensive experiments throughout the manuscript.
>   
>&nbsp;   
> 
>We have clarified the distinction with FedMeta in the related work section of the revised manuscript.

Reviewer comment (Theory):
For Algorithm 2, what is the motivation behind applying the meta update to the temporary global model? Wouldn't applying the meta-update to the local models (akin to personalized FL) make more sense in the context of FL? Can you provide a real example of scenario 2?

Author response:
>**Key applications**: Consider a healthcare application where each client, such as a hospital or an individual patient, aims to build a comprehensive global model capable of classifying a wide range of diseases. However, individual clients may possess limited types of diseases in their local datasets – for instance, one client may have data on diseases A and B but lacks information on diseases C and D. In this context, federated learning becomes essential. Clients need to collaborate to construct a global model that not only reflects the diseases available locally but also incorporates information about diseases not present in their individual datasets, ensuring a more robust and universally applicable healthcare model. Similarly, in the domain of autonomous vehicles, each self-driving car may strive to develop a global model for scenario detection in various weather conditions. However, individual cars might encounter limited weather scenarios locally – one car might navigate through a desert environment, while another faces challenges in a snowy storm. Through federated learning, these cars can collectively construct a global model that accounts for a broad spectrum of weather conditions, ensuring robust scenario detection capabilities for all vehicles involved. The wide availability of datasets open to public also makes our Scenario II practical, as the server can hold these public datasets and meta-update the global model learned by clients. 
>   
>&nbsp;   
> 
> **Contributions beyond FedMeta**: The distinction between our CoPreFL and FedMeta lies in the application, which requires us to conduct meta-update in a totally different way.  FedMeta is designed to offer personalization to individual clients, e.g., when a specific client is interested in predicting only diseases A and B, or when a specific self-driving car is interested in the model tailored to a specific weather. In contrast, our emphasis is on creating an initial model that can construct a good “global model” during downstream instead of “personalized models”,  targeting the aforementioned applications. This is the reason why we need to update the temporary global model instead of the local models, which is the key technical difference with FedMeta. Another technical difference is the linear combination between fairness and average performance term, as the reviewer understands correctly. These two key techniques enables  CoPreFL to construct a robust initial model that can quickly adapt to “any group of clients” (instead of individual clients) to construct a global model during downstream tasks. The advantages of CoPreFL can be also confirmed via extensive experiments throughout the manuscript.
>   
>&nbsp;   
> 
>We have clarified the distinction with FedMeta in the related work section of the revised manuscript.

Reviewer comment (Experiments):
How should the support and query sets be selected in practice by each client?

Author response:
> - How should the support and query sets be selected in practice by each client? 
>>For each client, we randomly select 80% of their local data as support set and the remaining 20% as query. 
>   
>&nbsp;

Reviewer comment (Evaluation):
What is the difference between the performance variance across clients during pretraining and downstream tasks for high $\gamma$ in the noniid settings?

Author response:
> - What is the difference between the performance variance across clients during pretraining and downstream tasks for high gamma in the noniid  settings? 
>> Performance variance is assessed similarly during both pretraining and downstream tasks, involving the evaluation of the global model's performance on each participant's query/testing samples. However, the context in which variance is employed differs. During pretraining, performance variance serves as a factor to control the pretraining objective, utilizing a parameter, $\gamma$, to modulate its influence. A higher $\gamma$ encourages the pre-trained model to lean towards achieving better averaged accuracy performance, prioritizing accuracy over variance. Conversely, in downstream tasks, performance variance is merely a measure of the quality of the tasks and is independent of IIDness. 
>   
>&nbsp;

Reviewer comment (Experiments):
Could you provide additional details regarding your selection process for the number of communication rounds and local epochs in the experiments? Additionally, considering that you've reported performance on the worst-performing clients, is there a reason you didn't consider AFL or qFFL for larger values of q, such as q=20, which optimize for worst-case fairness across clients?

Author response:
>### **Parameter selection detail and q-value for qFFL** 
>For all FL pre-training methods, we consistently use 50 global rounds with each round of local training comprising 5 iterations per participant. This configuration has proven to be stable in our setup and is consistent with findings in prior work [1].  Regarding the choice of the q value in q-FFL, we selected values from the range [1, 3, 5] based on previous  research findings [2]. The authors have demonstrated that smaller q values (q > 0, i.e., 0.1 or 1) tend to result in better performance. On the contrary, higher q values might compromise performance and slow down the convergence speed. The selected range aimed to strike a balance between fairness considerations and optimization for better overall performance. 
>   
>&nbsp;   
> 
>[1] On the Importance and Applicability of Pre-Training for Federated Learning, Chen et al., ICLR ‘22. 
> 
>[2] Fair Resource Allocation in Federated Learning, Li et al., ICLR ‘20.

Reviewer comment (Writing):
Does the phrase "All of these schemes are adopted during the pre-training phase to construct initial models." in "Baselines for pre-training" mean that qFFL and FedAvg algorithms were modified to consider pretraining?

Author response:
> - Does the phrase "All of these schemes are adopted during the pre-training phase to construct initial models." in "Baselines for pre-training" mean that qFFL and FedAvg algorithms were modified to consider pertaining? 
>> Yes, the phrase means that we consider different baselines, including FedAvg  and q-FFL, during the pre-training phase using the pre-training dataset. The goal is to compare their ability to initialize the downstream FL tasks, which is the main focus of this work. We have included these schemes as baselines to strengthen the empirical results as there were not many baselines available for pre-training for FL. 
>   
>&nbsp;

Reviewer comment (Writing):
Page 4 prior to Eq. 2: "achieving performance gains any unseen FL task" should be corrected to "achieving performance gains on any unseen FL task".

Author response:
> - Other suggestions & typos 
>> All typos have been corrected in our revised paper.

Reviewer comment (Writing):
Fix the rephrasing in the abstract and conclusions to clarify that both scenarios are addressed: where clients own the data exclusively, and where both clients and the server own the data.

Author response:
> - Other suggestions & typos 
>> All typos have been corrected in our revised paper.

Reviewer comment (Writing):
Page 5: "The objectives of our pre-trained model are to strike a balance" should be corrected to "The objective of our pre-trained model is to strike a balance".

Author response:
> - Other suggestions & typos 
>> All typos have been corrected in our revised paper.