Paper: ZOOPFL: EXPLORING BLACK-BOX FOUNDATION MODELS FOR PERSONALIZED FEDERATED LEARNING (ICLR-2024)

Reviewer comment (Experiments):
The motivation for deploying zeroth-order optimization methods based on the local encrypted black-box model setup is not well-motivated. This setting implies that it is entirely possible to train a white-box emulator [1] as a proxy for the black-box model and directly perform first-order optimization based on the white-box emulator. However, the authors do not provide relevant discussions and experimental comparisons.

Author response:
It is important to note that we are in a black-box setting while [1] uses a white-box setting. Thus, *the comparison with [1] is neither fair nor meaningful.* But we will cite [1] in the future.

Specifically, [1] involves compressing or substituting large models, requiring a deeper understanding, which differs from our current framework. We make the assumption that *the internals of large models are entirely unknown*. While training substitute models through obtaining outputs based on given inputs could be another avenue, as mentioned in our future work, 'Foundation models in ZOOPFL can be enhanced by other ways, e.g., auxiliary models, to serve as a complement to foundation models.' Currently, no one in the field has pursued this direction. Additionally, in our setup, each client has a limited amount of training data, potentially challenging the construction of robust substitute models ('For COVID-19, each client only has about 50 samples.').

[1] Xiao, Guangxuan, Ji Lin, and Song Han. "Offsite-tuning: Transfer learning without full model." arXiv preprint arXiv:2302.04870 (2023).

Reviewer comment (Experiments):
In terms of model privacy, the privacy leakage of a black-box model is closely related to the number of queries [2], but the authors do not provide theoretical or empirical studies on this.

Author response:
A good question! However, our paper *does not* have such an issue. Why? The black-box models are in each client, and there is no data exchange between clients and servers, or between clients. 

Regarding the privacy protection of the model, it's not within the scope of consideration for our current paper. Our focus lies in exploring how to better leverage a black-box model provided by the model suppliers—enhancing the alignment between local task inputs and outputs with the model.

Reviewer comment (Experiments):
The experimental section lacks ablation experiments with varying levels of noise added on the transformed data and visualizations of transformed data.

Author response:
- First, we do have the experiments on varying levels of noise, as illustrated in Figure 6(b).
- As for visualization, we add such extra experiments in the revised version, as illustrated in Appendix F.5.

Reviewer comment (Novelty):
The idea seems very similar to the soft-prompt training [1], which is also working on the input surgery without directly inference the foundational model. What is the benefit of the auto-encoder pre-training in your paper?

Author response:
We appreciate your recognition of the significance of [1] and we have cited this work in the revision. Where is the benefits of auto-encoder pre-training?

We use auto-encoders to alleviate the computational complexity associated with zeroth-order optimization. The rationale behind pre-training the auto-encoder stems from the need to adapt it to samples and tasks effectively. The pre-training step, as depicted in Figure 4 of the original manuscript, serves as a crucial aspect of our methodology, as illustrated by the ablation experiments.

In summary, we agree with your assessment and appreciate the opportunity to clarify the unique contributions of our work in addressing the where, how, and why aspects of input manipulation, particularly in the context of personalized federated learning with large black-box models. We will certainly emphasize the distinctions and innovations of our approach in the revised manuscript, giving due credit to the relevant literature, including Reference [1].

[1] Wang, Zifeng et al. “Learning to Prompt for Continual Learning.” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 139-149.

Reviewer comment (Evaluation):
What are the benefits of personalization? In the experiment part, it mainly focused on the overall accuracy boost compared to ZS, which does not reflect anything regarding to personalization.

Author response:
Indeed, personalized learning stands out as one of the **most crucial and practical challenges** in FL [2], and our approach naturally tackles this issue. While we do furnish personalized outcomes for each client (as depicted in Figure 3 in the main text and Tables 4-6 in the appendix), constraints posed by page limitations compel us to emphasize average results in our report. It's worth noting that we explicitly stated in the original text, '**Our method achieves the best accuracy in most clients.**' 

Without Step 3 or implementing Step 3 via FedAVG, our method seamlessly adapts to the common federated learning (FL) setting. However, the incorporation of Step 3 not only **enhances semantic matching** but also leads to **superior overall outcomes and personalized effects**.

[2] Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. In The Eleventh International Conference on Learning Representations, 2023

Reviewer comment (Writing):
What does ZS stand for in the paper? Does it stand for zero-shot training?

Author response:
The meaning of ZS is introduced in the paper: ZS denotes "zero-shot pre-trained models", i.e., zero-shot inference using the pre-trained models. And with all due respect, the term "zero-shot training" in your question is incorrect since we cannot "zero-shot" train a model if there is no sample available.

Reviewer comment (Writing):
I am not very clear why ZooPFL needs Semantic re-mapping. Could the author elaborate more on this?

Author response:
The benefit of semantic remapping is introduced in the main paper. Here we explain it more. Short answer: the model outputs do not match the target task, so a re-mapping is needed.

Long answer: As highlighted in our main text, "Some popular language models such as BERT and GPT-2 in Huggingface utilize a random projection between extracted features and logits," and "This step endeavors to re-map logits into meaningful semantic spaces with a simple linear projection." The intentional inclusion of a random mapping followed by a specific semantic mapping is justified in practical terms. Moreover, when meeting data not covered by foundation models, foundation models can be able to extract meaningful features but be unable to map it to expected target labels. Semantic re-mapping eliminates these gaps. The ablation experiments further substantiate the significance of this component.

Reviewer comment (Reproducibility):
While the idea of using FM as black box is interesting, there might be some privacy implications. It is unclear if the input to the FM reveals any information about the private input that is used both for training and during inference. I assume that the FM --being a black box and too big to be hosted on-device-- is run externally). As a result, this method might limit the ability of FL to offer privacy-preserving training. In other words, if we assume that the "black box" in figure 2.b runs externally, what are the privacy implications with respect to its input and output crossing the device boundary. If it runs on-device then what are the assumptions with respect to its size and the fact that it is a black box.

Author response:
1. If we assume that the "black box" in figure 2.b runs externally, what are the privacy implications wrt to its input and output crossing the device boundary. If it runs on-device then what are the assumptions wrt to its size and the fact that it is a black box.

- First, we currently do not assume cross-device data exchange following existing work on PFL. So, the privacy issue does not exist.
- Second, for on-device training of our model, the sizes of models and data highly depend on the local computation hardware, but not our algorithm. By design, our algorithm supports all kinds of models and datasets.

Reviewer comment (Novelty):
The authors assume that the FM is a black box. With more and more FM being open-sourced, it would be great if the authors can further motivate their approach and what might be the main advantages of incorporating a black box.

Author response:
2. The authors assume that the FM is a black box. With more and more FM being open-sourced, it would be great if the authors can further motivate their approach and what might be the main advantages of incorporating a black box.

Please refer to the general response for the detailed answer.

Short answer: even if you can have full access to the model locally, it is still expensive to actually train them. "Black-box" does not equal to close-source, even open-source models are expensive to train on ordinary devices.

Reviewer comment (Experiments):
The evaluation is mostly done on rather simple benchmarks. I was wondering if the proposed approach (to train just parts of the model) would carry enough capacity to tackle larger tasks. Maybe some discussion or even evaluation on a more complex task would be great.

Author response:
3. The evaluation is mostly done on rather simple benchmarks. I was wondering if the proposed approach (to train just parts of the model) would carry enough capacity to tackle larger tasks. Maybe some discussion or even evaluation on a more complex task would be great.

In actuality, our study encompassed extensive experimentation across **eight** datasets that include **two modalities**: COVID-19, APTOS, Terra100, Terra46, SST2, COLA, Financial-phrasebank (Financial), and Flipkart. The results from these experiments underline the superiority of the framework proposed in our study. 

Please be aware that the tasks we have selected are currently **not adequately addressed or poorly performed** by large models. Our aim is to align these large models with specific tasks, **rather than undergoing retraining**. Consequently, we do not require an abundance of data or a larger dataset.

Reviewer comment (Evaluation):
The paper might benefit from some understanding of the memory footprint and computation complexity of this method. Overall, the main target of this method is to make FM training possible with FL (on-device). As a result, we should have a good understanding on the memory/computation overhead.

Author response:
4. The paper might benefit from some understanding of the memory footprint and computation complexity of this method. Overall, the main target of this method is to make FM training possible with FL (on-device). As a result, we should have a good understanding on the memory/computation overhead.

The answer is simple: we only introduce *negligible* memory and computation burden to the foundation models.

As depicted in Figure 4(d) of the original paper, it's evident that the parameters adjusted by our method are significantly smaller in scale compared to the large foundation models. This aspect reinforces the notion that our approach hones in on a smaller subset, effectively **tailoring the model's adjustments**, rather than necessitating the manipulation or direct handling of the entire foundation model. Our emphasis remains on the utilization of these large models as protected tools within federated learning, ensuring their functionality while mitigating the challenges associated with their training costs.

Reviewer comment (Theory):
The paper assumes that foundation models are located at FL clients, but the clients cannot perform back-propagation on these models. It is not clear in what practical scenario such an assumption would hold. It is worth noting that most large language models (LLMs) nowadays are hosted in the cloud. Obviously, transmitting data to the cloud, even for inference, violates the privacy promise provided by FL. It seems that the authors of this paper try to overcome this privacy violation by assuming that the foundation model is hosted on each client. However, this has several issues. First, many types of LLMs are not feasible to run on mobile devices, which means that the proposed approach may only be possible in the case of cross-silo FL but not cross-device FL. Second, and more importantly, if the foundation model is hosted at the client, it is unclear why gradients cannot be computed, since each client has full access to its model in this case.

Author response:
1. In what practical scenario would a FL client host a foundation model, but does not have full access to it?

Please refer to the general response for a better understanding of our problem setting.

Short answer: *The understanding of the word "black-box" could be broad.* Black-box does not only deal with the case where one has no full access to the model; even if you have the model access, it is still extremely expensive to run model updates (Backpropagation) on the local devices given the increasing larger sizes of the models. As an example used in the paper, 'training GPT-2-small (Radford et al., 2019) requires at least two A100 GPUs for 16 hours, a resource unavailable to many'. We are a solution of using inference for BP to save computation and communication costs.

Reviewer comment (Novelty):
Overall, the proposed approach is a combination of several known techniques, including zeroth-order optimization, so the novelty seems limited.

Author response:
2. What are the key technical challenges and novel solution in this work?

Key technical challenges:
- efficiently update the foundation models locally
- distribution divergence in different clients

Our innovations to deal with them:
- A first pioneering exploration of the problem where models become too large to update, while FL clients still want to preserve privacy by not sharing data with the central server.
- A new framework to handle the problem using zeroth order optimization.
- We offer theoretical analysis along with extensive experiments to show the effectiveness of our work.