Paper: FedBiOT: a solution for federated large language model fine-tuning with intellectual property protection (ICLR-2024)

Reviewer comment (Experiments):
The number of clients is very small. In section 4.1, the number of clients is 4, which is relatively very small compared with that in real FL settings.

Author response:
Thanks for your question. There are two settings using more than four clients, i.e., 8 for Dolly (Question Answering) and 9 for Rosetta (Code Generation). Our experiments follow a well-developed benchmark, i.e., FederatedScope LLM [1]. Also, we notice some state-of-the-art works [2, 3, 4] discussing LLM in FL following the same setting as well. 

**References:** 

[1] FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning

[2] Towards Building the Federated GPT: Federated Instruction Tuning

[3] Federated Tuning for Black Box Large Models

[4] FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models

Reviewer comment (Experiments):
The selection of dropout rate is not well elaborated.

Author response:
Thanks for your question. Removing fewer layers will lead to significant intellectual property concerns while removing more layers will lead to poor performance. To find the trade-off between these two aspects, we discuss the setting of 0.2 and 0.5. 

There are 32 decoders in a LLaMA-7B model. If the dropout rate is 0.2 and the adapter size is 2, then the emulator consists of 24 decoders, and the total size of the compressed model is 26. Similarly, the sizes under different adapter size and dropout rate settings are given in the following table: 

|Dropout rate|Adapter size|Emulator size|Total size|
|:--------:|:-------:|:-------:|:------:|
|0.2|2|24|26|
|0.2|4|22|26|
|0.5|2|15|17|
|0.5|4|14|18|

Reviewer comment (Presentation):
Tables 1 and 2 are not clear. The first line is not explained. The unit can be added, and the meaning of the numbers can be explained.

Author response:
Thanks for your question. 0.2 means we remove 20% of layers, while 50% means we remove 50% of layers (as mentioned in **[W3]**). We will make it clear in the caption. In Table 2/3, the numbers in the first row refer to the indices of the communication rounds.

Reviewer comment (Evaluation):
The experimentation show that the performance of FedBiOT may be inferior than baselines.

Author response:
Thanks for your question. We have noticed that FedBiOT may be inferior to baselines and explained why it happens in experiments. We again highlight those fall-behinds and reiterate the reasons: 

- **Code Generation (Table 2/3).** We capture the checkpoints every 100 communication rounds. Generally speaking, FedBiOT performs better. However, since the results are reported at certain checkpoints, FedBiOT may suffer from a sudden drop such that other baselines surpass our proposed method. 
- **Question Answering (Table 4/5; now Figure 2/3 in revision).** The proposed method constantly outperforms other baselines except for NarrativeQA. In detail, AdapEmu in FedBiOT outperforms that in other baselines, but AdapFu in FedBiOT has poor performance. Possibly, the emulator cannot imitate the raw model in terms of NarrativeQA because the public dataset does not possess the relevant data. As a result, even if the adapter can answer NarrativeQA based on the given emulator, the ability cannot plug in the full model.

Reviewer comment (Experiments):
The classic FL approaches can be added as baselines.

Author response:
Thanks for your suggestion. The classic FL approaches will utilize the entire full model for fine-tuning, which conflicts with our setting to protect intellectual properties. We add these results in Figure 4, where our proposed work utilizes much fewer computation costs and achieves the comparable performance of FedLoRA, a classic FL approach fine-tuning LLM. 

**Appendix:** The result shows FedLoRA, and we fix the rank to be 8 and try the scaling factor $\alpha=32, 64, 128$

- GSM8k: 

|Method|Accuracy|
|---|---|
|Full Model (LoRA)|14.25% (188/1319)|

- Rosetta: 

|Method|Accuracy|
|---|---|
|Full Model (LoRA)|13.29%|

- Dolly: 

|Method|MMLU|HellaSwag|OpenbookQA|Natural Questions (open-book)|IMDB|Narrative QA|
|---|:-:|:-:|:-:|:-:|:-:|:-:|
|Full Model (LoRA)| 36.5 | 82 | 52 | 71.3 | 98 | 55.6 |

Reviewer comment (Presentation):
Tables 1 and 2 are not clear. The first line is not explained. What is the unit can be added?

Author response:
Thanks for your question. 0.2 means we remove 20% of layers, while 50% means we remove 50% of layers (as mentioned in **[W3]**). We will make it clear in the caption. In Table 2/3, the numbers in the first row refer to the indices of the communication rounds.

Reviewer comment (Experiments):
The classic FL approaches can be added as baselines. I wonder if the authors can compare FedBiOT with classic approaches.

Author response:
Thanks for your suggestion. The classic FL approaches will utilize the entire full model for fine-tuning, which conflicts with our setting to protect intellectual properties. We add these results in Figure 4, where our proposed work utilizes much fewer computation costs and achieves the comparable performance of FedLoRA, a classic FL approach fine-tuning LLM. 

**Appendix:** The result shows FedLoRA, and we fix the rank to be 8 and try the scaling factor $\alpha=32, 64, 128$

- GSM8k: 

|Method|Accuracy|
|---|---|
|Full Model (LoRA)|14.25% (188/1319)|

- Rosetta: 

|Method|Accuracy|
|---|---|
|Full Model (LoRA)|13.29%|

- Dolly: 

|Method|MMLU|HellaSwag|OpenbookQA|Natural Questions (open-book)|IMDB|Narrative QA|
|---|:-:|:-:|:-:|:-:|:-:|:-:|
|Full Model (LoRA)| 36.5 | 82 | 52 | 71.3 | 98 | 55.6 |

Reviewer comment (Novelty):
The contributions need to be clarified. The topic of this paper is interesting and worth exploring. However, it is not so clear what are the main contributions of this paper since previous work [1] has considered such setting and proposed FedOT (federated learning with offsite-tuning). Are the main contributions lying on improving FedOT via a bi-level optimization approach?

Author response:
Thank you for your comments. First, let us explain the relationship among Offsite Tuning, FedOT, and FedBiOT (our proposed work): 

- Offsite Tuning is the first work that points out the bottleneck of LLM fine-tuning, i.e., the LLM owner cannot transmit the model to the clients, while the data holder cannot share the data with the server. 
- FedOT considers there are multiple data holders for a specific task and extends the proposed offsite tuning algorithm to a federated learning setting. 
- FedBiOT (ours) highlights their expensive computation costs and explicit model and data privacy risks. To address these challenges, we introduce a more efficient bi-level optimization approach, FedBiOT, that guarantees intellectual properties.

Reviewer comment (Novelty):
The motivations need to be further clarified. This paper claims that the clients cannot obtain the full model due to intellectual property of LLM. However, it is questioned whether such claim still holds after the release of Llama2.

Author response:
Thanks for your question. Looking back at the development of GPT, we know that OpenAI released the checkpoints of GPT-2 and GPT-3 but kept mysterious for GPT-3.5 and GPT-4. Thus, there is no guarantee that those companies will still be willing to open-source more powerful models in the future. Our proposed technique can be an alternative besides the binary open-source v.s. closed-source options.

As a resilient and flexible solution, the proposed method can build a win-win community: The LLM holders release a compressed model, which the clients train with their local data. The server can aggregate the updated models from the clients and utilize them to enhance the model performance on the downstream task.

Reviewer comment (Experiments):
Some experiments for reference are missing. It would be more helpful if the authors can provide the results when clients can obtain the full model, such that it would be possible to see how large the gap is.

Author response:
Thank you for your suggestions. We provide you with concrete results when fine-tuning with the full model as follows. (Note: The result here shows LoRA, and we fix the rank to be 8 and try the scaling factor $\alpha=32, 64, 128$) 

- GSM8k: 

|Method|Accuracy|
|---|---|
|Full Model (LoRA)|14.25% (188/1319)|

- Rosetta: 

|Method|Accuracy|
|---|---|
|Full Model (LoRA)|13.29%|

- Dolly: 

|Method|MMLU|HellaSwag|OpenbookQA|Natural Questions (open-book)|IMDB|Narrative QA|
|---|:-:|:-:|:-:|:-:|:-:|:-:|
|Full Model (LoRA)| 36.5 | 82 | 52 | 71.3 | 98 | 55.6 | 


For computation comparison, We prepare a scatter figure in the appendix to show the difference in terms of the calculation in Figure 4. As we can see, the full model consumes a large amount of computation effort.

Reviewer comment (Experiments):
Computation resources comparisons are missing. This method requires more training resources (e.g., more training steps) compared to baselines. However, this paper does not show such comparisons, which would promote readers' understanding.

Author response:
Thank you for your suggestions. We provide you with concrete results when fine-tuning with the full model as follows. (Note: The result here shows LoRA, and we fix the rank to be 8 and try the scaling factor $\alpha=32, 64, 128$) 

- GSM8k: 

|Method|Accuracy|
|---|---|
|Full Model (LoRA)|14.25% (188/1319)|

- Rosetta: 

|Method|Accuracy|
|---|---|
|Full Model (LoRA)|13.29%|

- Dolly: 

|Method|MMLU|HellaSwag|OpenbookQA|Natural Questions (open-book)|IMDB|Narrative QA|
|---|:-:|:-:|:-:|:-:|:-:|:-:|
|Full Model (LoRA)| 36.5 | 82 | 52 | 71.3 | 98 | 55.6 | 


For computation comparison, We prepare a scatter figure in the appendix to show the difference in terms of the calculation in Figure 4. As we can see, the full model consumes a large amount of computation effort.

Reviewer comment (Writing):
There is confusion regarding "Improvement 1" at page 5. The definitions of bottom / first / last layers are unclear. Consistent expressions like first / last are suggested.

Author response:
Thanks for your suggestion. We now add a footnote to clarify these three words. In detail, The bottom/last layers refer to the transformer decoders near the output, while the top/first layers refer to the part close to the input.

Reviewer comment (Reproducibility):
According to the sentence "We apply offsite-tuning with one single client, where all data are loaded to the client.", it seems like the Offsite-tuning is training over the gathered dataset of all clients. But why is its performance quite low? More details are needed on how offsite-tuning is implemented.

Author response:
Thanks for your question. As Kuang et al. [1] have implemented Offsite-Tuning [2] in their FederatedScope framework, we utilize their codes and report the results of Offsite-Tuning on our designated tasks. In detail, they reproduce the algorithm with the following steps: 
- Step 1: Choose the adapter and the emulator from the full model and form a compressed model. In detail, the adapter is the first two and the last two decoders. 
- Step 2: Use the public dataset to align the knowledge between the full model and the compressed model while fixing the adapter 
- Step 3: Fine-tune the adapter until the model converges 

We owe its poor performance to two reasons. First, the emulator is yet to correctly align with the raw model. As offsite-tuning ignores the effect of the final two decoders (i.e., the latter part of the adapter), they can lead to a significant discrepancy even if their embeddings at the middle layers are very similar. Second, since the emulator never updates during the fine-tuning stage, the emulator fails to imitate the functionality of the raw model as the training rounds fly. Therefore, training the adapter of AdapEmu cannot boost the performance of the plug-in full model (i.e., AdapFu). 

**Reference:**

[1] FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning

[2] Offsite-Tuning: Transfer Learning without Full Model

Reviewer comment (Experiments):
The experimental evaluation was only implemented in LLaMA-7B. How does it work on other mainstream models such as ChatGPT2?

Author response:
We conduct our experiment on GPT-2 pipeline, and the results on GSM-8K are given as follows: 

- Dropout rate is 0.2: 

|Methods|AdapEmu Accuracy|AdapFu Accuracy|
|:-----:|:------:|:---------:|
|Offsite Tuning|2.30%|2.06%|
|FedOT|2.06%|2.30%|
|FedBiOT (Adapter 2)|2.22%|2.14%|
|FedBiOT (Adapter 4)|2.94%|2.54%|

- Dropout rate is 0.5: 

|Methods|AdapEmu Accuracy|AdapFu Accuracy|
|:-----:|:------:|:---------:|
|Offsite Tuning|2.70%|1.59%|
|FedOT|2.53%|1.51%|
|FedBiOT (Adapter 2)|1.83%|2.06%|
|FedBiOT (Adapter 4)|2.46%|2.54%|

Although our proposed method outperforms the baselines in most settings, it is noticed that the performance of our proposed work is inferior to the baselines when the dropout rate is 0.5. In light that AdapFu falls behind AdapEmu in most cases, we hypothesize that GPT-2 is poor in math problem solving, and therefore, fine-tuning the model on the training data can perform better than aligning with the pre-trained full model.

Reviewer comment (Experiments):
In the experiment, federated learning only considered 8 clients. There is a lack of experiments that vary the number of clients and the number of training samples each client own.

Author response:
Please refer to Table 4 in Appendix B. We consider three different settings for the number of clients, i.e., three clients for math problem solving, nine clients for code generation, and eight clients for question answering. At the same time, the training samples vary among the clients when we fine-tune the model on code generation and question-answering tasks. Take a question-answering task (Dolly) as an example. The client with the OpenQA dataset holds 3611 samples, while the client with the creative writing dataset holds 711 samples.

Reviewer comment (Novelty):
From a technical perspective, the two improvements proposed in the article may be incremental. One involves changing the index of the fine-tuning layers (based on observation), and the other relies on the traditional distillation method. Both methods are essentially at the level of tricks and are insufficient to serve as contributions to the paper.

Author response:
In our tasks, we want to overcome clients' computation constraints and protect both clients' data privacy and LLMs' intellectual property. Compared with the traditional distillation method, the data distribution used for alignment on the server differs from the downstream task; meanwhile, the clients fine-tune the trainable parameters on their end with heterogeneous task data. Thus, our task is significantly more challenging than the traditional knowledge distillation.

To overcome such challenges, our key contribution to this paper is how we solve the challenges of LLM training in federated learning, i.e., formulating a bi-level optimization problem (Eq. 1 and 2). In detail, we expect AdapFu and AdapEmu to realize the outcomes that
- **AdapEmu** is trainable for the clients and protects the original LLMs' privacy (i.e., model structure and parameters)
- **AdapFu** boosts the performance on the downstream tasks by adopting some layers from **AdapEmu**

Then, the problem is formulated accordingly, i.e., we disjoint an LLM into two parts (i.e., adapter and emulator) and align the knowledge between **AdapEmu** and **AdapFu** from round to round.

To solve the formulated problem efficiently, we tried different layer selection strategies and noticed that the proposed one resulted in the best performance and the lowest computation costs.

To the best of our knowledge, this is the **first** work that
- trains a surrogate light model to boost the original full model using bi-level optimization.
- makes LLM trainable on the clients without the data exposure to any third parties, in comparison with some recent works [1, 2, 3] focusing on LLM training in FL.

**References:**

[1] Towards Building the Federated GPT: Federated Instruction Tuning

[2] Federated Tuning for Black Box Large Models

[3] FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models

Reviewer comment (Reproducibility):
The paper should provide a detailed algorithm to help readers follow.

Author response:
Thank you for your suggestions. We make it available in Algorithm 1 of Appendix A, and a visualized demonstration is given in Figure 1.