Paper: LIFT: Efficient Layer-wise Fine-tuning for Large Model Models (ICLR-2024)

Reviewer comment (Novelty):
While LIFT is a novel approach, the paper should discuss how it relates to and differs from the concept of greedy layer-wise unsupervised learning, which also involves layer-by-layer training.

Author response:
### **Q1: Difference between LIFT and Greedy Layer-Wise Training**

Hinton[1] and Bengio's[2] work primarily focuses on **unsupervised learning**, **forward and backward only one layer**,  trained with **gradient estimators**, aiming to **find good initialization** for deep belief nets.

While we draw inspiration from this previous work, LIFT is fundamentally different. In LIFT, we focus on **supervised fine-tuning**, **forward the whole model and backward to the selected layer**, trained with **back-propagations**, targeting to **deliver high-performance weights** for large language models.

[1] A fast learning algorithm for deep belief nets.

[2] Greedy Layer-Wise Training of Deep Networks

Reviewer comment (Experiments):
A more detailed comparison with state-of-the-art methods, including those that may not focus on parameter efficiency but achieve high throughput, would be valuable.

Author response:
### **Q3: Comparison with fine-tuning efficient PEFT**

We would like to include more experiments but very few of PEFT methods can save backward computations. As shown in survey[1],  only **3 of 26** PEFT methods are capable of saving backward computations and they all introduce extra inference overhead. Therefore, we design LIFT in order to address this limitation and raise awareness within the community.

We include a comparison with LST as suggested by reviewer #zNBj, which adds learnable layers in a highway structure, and compares its performance with llama-7b on PIQA. The results are provided below.

| Methods | PIQA (%) | Inference Latency (ms) | Memory (GB) | Speed (ms/token) |
| --- | --- | --- | --- | --- |
| FT-Full | 82.4 | 182.6 | 91 | 284 |
| LST | 80.7 (-1.7) | 201.5 (+10.3%) | 28 | 102 |
| LIFT | 81.6 | 182.7 | 32 | 122 |

Though LST shows less peak memory usage, there is a performance drop (**1.7% on PIQA**) and an additional **10.3% inference latency**. Instead, LIFT introduces **no inference overhead** while demonstrating **comparable performance** with FT-Full., showing its advantages when fine-tuning llama.

[1]: Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

Reviewer comment (Experiments):
How does the selection of layers to fine-tune affect the final model performance? Is there a heuristic for selecting layers that could lead to better results?

Author response:
### **Q2: Layers selection heuristics**

In fact, we have already discussed the **impact of different layers in Figure 4 from Appendix B** and analyzed **different training schedules in Table 7 from Appendix C.**

For layer-wise impact, we found that

- Attention and the first-FFN contribute more to the downstream performance.
- Middle blocks are more important in evaluating all GLUE tasks.

For training schedules, we would point out that

- There are a total of **N! schemes**, and it is **impossible to exhaustively test them all**. Therefore, we chose the most representative methods (end2front, front2end, and random) for comparison.
- In BERT level experiments, random yields the best performance, while end2front delivers the worst precision.
- For llama-7b, the performance of random and front2end is roughly the same, and both outperform end2front.

To ensure LIFT's generalizability and simplicity, we set the training schedule to be *front2end* for all experiments in the main paper. The results show that LIFT with the front2end heuristic achieves comparable accuracy to full fine-tuning.

Reviewer comment (Experiments):
Combination with Other Methods: The paper mentions that LIFT is orthogonal to methods like LoRA. Could you provide more insights into how LIFT interacts with these methods and any potential limitations of such combinations?

Author response:
### **Q4: Combination of LIFT with other PEFTs**

LIFT can be combined with most PEFTs. Taking LoRA as an example, we first insert low-rank learnable parameters to blocks (LoRA). In each iteration, only one LoRA branch is updated (LIFT). It is worth noting that LIFT does not alter the architecture or introduce any new learnable parameters. Therefore, the **additional storage per task for LIFT + LoRA is exactly the same as LoRA-only**. This combination allows us to improve fine-tuning throughput while maintaining parameter-efficiency for multiple downstream tasks.

Shown in Table 3, taking the llama experiments with WebQS as an example, LIFT + LoRA achieves higher accuracy than LoRA (39.7 v.s. 33.1) while achieving a much faster speed (106ms v.s 213ms). To further prove the effectiveness, we also evaluate LIFT + LoRA on GLUE and Alpaca dataset:

| BERT-BASE (input 32x512) | Avg score on GLUE | Speed | Learnable Params | Peak Memory |
| --- | --- | --- | --- | --- |
| FT-Full | 80.2% | 118.1ms | 125M | 11.3GB |
| LoRA | 79.0% | 93.1ms | 0.1M | 9.6GB |
| LIFT | 79.5% | 74.5ms | 8.3M | 9.5GB |
| LIFT + LoRA | 79.1% | 73.2ms | 0.1M | 9.4GB |

| Llama-7B (input 2x512) | Alpaca Loss | Speed | Peak Learnable Params | Peak Memory |
| --- | --- | --- | --- | --- |
| FT-Full | 0.34 | 514ms | 6.7B | 91GB |
| LoRA | 0.41 | 403ms | 21M | 41GB |
| LIFT | 0.36 | 221ms | 201M | 32GB |
| LIFT + LoRA | 0.38 | 194ms | 21M | 27GB |

On both experiments, LIFT + LoRA achieves better performance and speed than LoRA, while using the same number of learnable parameters.

Reviewer comment (Experiments):
Additionally, could the authors elaborate on whether there is potential for the combined use of LIFT with other fine-tuning methods to outperform the application of either approach in isolation?

Author response:
### **Q4: Combination of LIFT with other PEFTs**

LIFT can be combined with most PEFTs. Taking LoRA as an example, we first insert low-rank learnable parameters to blocks (LoRA). In each iteration, only one LoRA branch is updated (LIFT). It is worth noting that LIFT does not alter the architecture or introduce any new learnable parameters. Therefore, the **additional storage per task for LIFT + LoRA is exactly the same as LoRA-only**. This combination allows us to improve fine-tuning throughput while maintaining parameter-efficiency for multiple downstream tasks.

Shown in Table 3, taking the llama experiments with WebQS as an example, LIFT + LoRA achieves higher accuracy than LoRA (39.7 v.s. 33.1) while achieving a much faster speed (106ms v.s 213ms). To further prove the effectiveness, we also evaluate LIFT + LoRA on GLUE and Alpaca dataset:

| BERT-BASE (input 32x512) | Avg score on GLUE | Speed | Learnable Params | Peak Memory |
| --- | --- | --- | --- | --- |
| FT-Full | 80.2% | 118.1ms | 125M | 11.3GB |
| LoRA | 79.0% | 93.1ms | 0.1M | 9.6GB |
| LIFT | 79.5% | 74.5ms | 8.3M | 9.5GB |
| LIFT + LoRA | 79.1% | 73.2ms | 0.1M | 9.4GB |

| Llama-7B (input 2x512) | Alpaca Loss | Speed | Peak Learnable Params | Peak Memory |
| --- | --- | --- | --- | --- |
| FT-Full | 0.34 | 514ms | 6.7B | 91GB |
| LoRA | 0.41 | 403ms | 21M | 41GB |
| LIFT | 0.36 | 221ms | 201M | 32GB |
| LIFT + LoRA | 0.38 | 194ms | 21M | 27GB |

On both experiments, LIFT + LoRA achieves better performance and speed than LoRA, while using the same number of learnable parameters.

Reviewer comment (Reproducibility):
The paper mentions that, in order for LIFT to save memory, proper implementations might be needed. Does that mean LIFT cannot be efficient on popular frameworks, like PyTorch?

Author response:
### **PyTorch Implementation and Transformers Integration.**

The core part to implement LIFT is to ensure **only the optimizer state of (currently) learnable parameters is stored in GPU memory** (and offload others to CPU). This is a common technique used in algorithms such as  paged optimizer[2], which has been already merged in the Transformer library.

We implemented LIFT **similarly in PyTorch and transformer** and **achieved real savings** (all memory and latency numbers in submission are real-measured). We will open source the codebase and make sure every pytorch user can easily adapt for their need.

[1]: https://github.com/ylsung/Ladder-Side-Tuning

[2]: [https://github.com/huggingface/transformers/pull/23217](https://github.com/huggingface/transformers/pull/23217)