Paper: Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt (ICLR-2024)

Reviewer comment (Novelty):
The approach is simple and the novelty is not obvious. The soft prompt method is already proposed in multiple existing papers.

Author response:
### **[W1 - The approach is simple and the novelty is not obvious: Our paper uncovers new properties of prompt tuning and they offer new opportunities for model compression.]**

While novelty is a multifaceted concept in academic research, we believe it can be roughly viewed from two fronts: ***empirical novelty***, which involves uncovering new properties and behaviors unknown to the wider community (e.g., the emergent ability [1], lottery ticket hypothesis [2]), and ***technical novelty***, which pertains to the development of new solutions or methodologies (e.g., LoRA [3]).

We argue our work is rich in empirical novelty in at least two aspects:

1. **Transformation of Prompt Tuning from Task-Specific to Transferable:** Prior to our study, only a few works studied the transferability of learned prompts between different tasks [4, 5, 6]. Specifically, [4] finds it is possible to transfer learnable prompts with **additional fine-tuning on downstream task**. However, as mentioned in the experiment section, all our reported results are **zero-shot**, i.e., no fine-tuning is needed to obtain transferability. Furthermore, [5] finds that the learned prompt can only be transferred among similar tasks. However, as verified by our experiments, our learned prompts are transferable between datasets, tasks, and compressed models.

2. **New avenue to enhance Compressed Model Accuracy:** We show that prompt tuning can effectively recover the accuracy drop of compressed models. Traditional model compression methods often demand extensive engineering efforts and intricate designs, as seen in popular model compression papers like GPTQ [7], SPQR [8], and AWQ [8]. However, our method simplifies this process significantly. We leverage prompt tuning to effectively recover the accuracy drop in compressed models, opening a new avenue to optimize the trade-off between accuracy and efficiency.

These findings are not only valid but also generalizable across various model families, datasets, and tasks, underscoring the broad applicability and impact of our work.

Regarding technical contribution, we agree that our method is a direct application of prompt tuning. However, we emphasize that **we intensionally kept our method as simple as possible**. This simplicity, is in our opinon more of a strength than a weakness, as it just underscores the unknown properties of straightforward prompt tuning methods; where the uncovering of these properties and the new pathways they opened for model compression constitute the true novelty of our paper. Thereby, we argue that we contribute substantially to the field's understanding of prompt tuning and model compression.




[1] Emergent Abilities of Large Language Models

[2] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

[3] LoRA: Low-Rank Adaptation of Large Language Models

[4] On Transferability of Prompt Tuning for Natural Language Processing

[5] SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer

[6] Reducing Retraining by Recycling Parameter-Efficient Prompts

[7] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

[8] SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

[9] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Reviewer comment (Evaluation):
The evaluation is only conducted with PPL. However, the evaluation of LLM should be well designed and other methods should be exploited to further validate the proposed method.

Author response:
### **[W2,Q1,Q2 - The evaluation is only conducted with PPL: We have in fact evaluated on other tasks]**

**We kindly direct the reviewer's attention to [Appendix A.3](https://openreview.net/pdf?id=Gdm87rRjep#page=15)** , where we have explored QA (OpenBookQA, PIQA), commonsense NLI (Hellaswag), language understanding (high school European history from MMLU) tasks reported with non-PPL metrics. We highlight that all these reported results are zero-shot. Namely, the prompt is learned on C4, and then we directly stitch the prompt to the compressed model and test the performance on these downstream tasks. We hope the reviewer may find them helpful.


| Models |                  | OpenbookQA      | Hellaswag       | PIQA            | High School European History |
|--------|------------------|-----------------|-----------------|-----------------|------------------------------|
| Full   |                  | 0.410±0.022     | 0.497±0.005     | 0.702±0.011     | 0.364±0.038                  |
| 50%    | w./o. Prompt     | 0.412±0.022     | 0.449±0.005     | 0.682±0.011     | **0.364±0.038**              |
|        | + Learned Prompt | 0.400±0.022     | **0.469±0.005** | **0.689±0.011** | 0.358±0.037                  |
| 62.5%  | w./o. Prompt     | 0.396±0.022     | 0.380±0.005     | 0.638±0.011     | 0.345±0.037                  |
|        | + Learned Prompt | **0.402±0.022** | **0.433±0.005** | **0.668±0.011** | 0.345±0.037                  |
| 75%    | w./o. Prompt     | 0.366±0.022     | 0.280±0.004     | 0.549±0.012     | 0.315±0.036                  |
|        | + Learned Prompt | 0.358±0.021     | **0.344±0.005** | **0.614±0.011** | **0.358±0.037**              |
| 4-bit  | w./o. Prompt     | 0.410±0.022     | 0.487±0.005     | 0.690±0.011     | 0.358±0.037                  |
|        | + Learned Prompt | **0.418±0.022** | 0.487±0.005     | **0.692±0.011** | 0.352±0.037                  |
| 3-bit  | w./o. Prompt     | 0.378±0.022     | 0.446±0.005     | 0.674±0.011     | 0.358±0.037                  |
|        | + Learned Prompt | **0.404±0.022** | **0.459±0.005** | **0.688±0.011** | 0.358±0.037                  |
| 2-bit  | w./o. Prompt     | 0.354±0.021     | 0.240±0.004     | 0.491±0.012     | 0.315±0.036                  |
|        | + Learned Prompt | 0.350±0.021     | **0.294±0.005** | **0.563±0.012** | **0.333±0.037**              |

Reviewer comment (Experiments):
More tasks should be examined to show the effectiveness of the proposed approach.

Author response:
### **[W3 - The difference between the proposed methods and conventional soft prompt tuning: Transformation of Prompt Tuning from Task-Specific to Transferable]**


As we mentioned in our response to W1 above, the key difference between our prompt tuning and conventional soft prompt tuning is that our learned soft prompts are transferable between datasets, tasks, and compression levels. The transferring of prompts saves the prompt tuning time and effort originally required for each specific setups (according to our experience with 4 GPUs, that means 5-ish hours per task per compressed model).

We also already compared our methods with conventional soft prompt tuning in [experiment](https://openreview.net/pdf?id=Gdm87rRjep#page=8) (**NOTE**: it was originally in Table 4 of Appendix A.3, we move it to the main text in the updated version). For your convenience, we post our results and summarize our observations here:


> Perplexity comparison between full model and quantized models with different prompts. Where we report test perplexity on PTB and Wikitext-2 dataset. "w./o. prompt" refers to the quantized model without soft prompts. "w./ direct prompt" means the soft prompts are directly trained on the target dataset. "w./ transferred prompt" means the prompt is trained on C4 dataset and then transferred to the target dataset.

| Model                       | PTB     | Wikitext2 |
|-----------------------------|---------|-----------|
| Full Model                  | 11.02   | 6.33      |
| Full Model w./ direct prompt| 6.86    | 5.57      |
| 4-bit w./o. prompt          | 11.65   | 6.92      |
| 4-bit w./ direct prompt     | 7.04    | 5.88      |
| 4-bit w./ transferred prompt| 9.25    | 6.26      |
| 3-bit w./o. prompt          | 15.74   | 9.45      |
| 3-bit w./ direct prompt     | 7.76    | 6.33      |
| 3-bit w./ transferred prompt| 10.81   | 6.90      |
| 2-bit w./o. prompt          | 5883.13 | 2692.81   |
| 2-bit w./ direct prompt     | 14.98   | 16.67     |
| 2-bit w./ transferred prompt| 29.82   | 20.56     |

Given direct prompt receives a task-specific loss, our transferred prompt is, as expected, not as competitive as the direct one. However, such transferred prompt may significantly bridge the gap between a compressed and full model — e.g., our 3-bit & 4-bit quantized LLaMA-7B with transferred prompt can deliver on-par or better PPL than the full model on PTB and Wikitext2. We'd say this is an especially worthy contribution in practice, as one may possibly download the open-sourced transferable prompt to help on a compressed model with little effort.

Reviewer comment (Evaluation):
I wonder if there are other LLM evaluation methods besides PPL.

Author response:
### **[W2,Q1,Q2 - The evaluation is only conducted with PPL: We have in fact evaluated on other tasks]**

**We kindly direct the reviewer's attention to [Appendix A.3](https://openreview.net/pdf?id=Gdm87rRjep#page=15)** , where we have explored QA (OpenBookQA, PIQA), commonsense NLI (Hellaswag), language understanding (high school European history from MMLU) tasks reported with non-PPL metrics. We highlight that all these reported results are zero-shot. Namely, the prompt is learned on C4, and then we directly stitch the prompt to the compressed model and test the performance on these downstream tasks. We hope the reviewer may find them helpful.


| Models |                  | OpenbookQA      | Hellaswag       | PIQA            | High School European History |
|--------|------------------|-----------------|-----------------|-----------------|------------------------------|
| Full   |                  | 0.410±0.022     | 0.497±0.005     | 0.702±0.011     | 0.364±0.038                  |
| 50%    | w./o. Prompt     | 0.412±0.022     | 0.449±0.005     | 0.682±0.011     | **0.364±0.038**              |
|        | + Learned Prompt | 0.400±0.022     | **0.469±0.005** | **0.689±0.011** | 0.358±0.037                  |
| 62.5%  | w./o. Prompt     | 0.396±0.022     | 0.380±0.005     | 0.638±0.011     | 0.345±0.037                  |
|        | + Learned Prompt | **0.402±0.022** | **0.433±0.005** | **0.668±0.011** | 0.345±0.037                  |
| 75%    | w./o. Prompt     | 0.366±0.022     | 0.280±0.004     | 0.549±0.012     | 0.315±0.036                  |
|        | + Learned Prompt | 0.358±0.021     | **0.344±0.005** | **0.614±0.011** | **0.358±0.037**              |
| 4-bit  | w./o. Prompt     | 0.410±0.022     | 0.487±0.005     | 0.690±0.011     | 0.358±0.037                  |
|        | + Learned Prompt | **0.418±0.022** | 0.487±0.005     | **0.692±0.011** | 0.352±0.037                  |
| 3-bit  | w./o. Prompt     | 0.378±0.022     | 0.446±0.005     | 0.674±0.011     | 0.358±0.037                  |
|        | + Learned Prompt | **0.404±0.022** | **0.459±0.005** | **0.688±0.011** | 0.358±0.037                  |
| 2-bit  | w./o. Prompt     | 0.354±0.021     | 0.240±0.004     | 0.491±0.012     | 0.315±0.036                  |
|        | + Learned Prompt | 0.350±0.021     | **0.294±0.005** | **0.563±0.012** | **0.333±0.037**              |

Reviewer comment (Novelty):
I wonder what would be the difference between the proposed methods and conventional soft prompt tuning except using different LLMs, i.e., one with quantized or pruned model while the other one with the original model.

Author response:
### **[W3 - The difference between the proposed methods and conventional soft prompt tuning: Transformation of Prompt Tuning from Task-Specific to Transferable]**


As we mentioned in our response to W1 above, the key difference between our prompt tuning and conventional soft prompt tuning is that our learned soft prompts are transferable between datasets, tasks, and compression levels. The transferring of prompts saves the prompt tuning time and effort originally required for each specific setups (according to our experience with 4 GPUs, that means 5-ish hours per task per compressed model).

We also already compared our methods with conventional soft prompt tuning in [experiment](https://openreview.net/pdf?id=Gdm87rRjep#page=8) (**NOTE**: it was originally in Table 4 of Appendix A.3, we move it to the main text in the updated version). For your convenience, we post our results and summarize our observations here:


> Perplexity comparison between full model and quantized models with different prompts. Where we report test perplexity on PTB and Wikitext-2 dataset. "w./o. prompt" refers to the quantized model without soft prompts. "w./ direct prompt" means the soft prompts are directly trained on the target dataset. "w./ transferred prompt" means the prompt is trained on C4 dataset and then transferred to the target dataset.

| Model                       | PTB     | Wikitext2 |
|-----------------------------|---------|-----------|
| Full Model                  | 11.02   | 6.33      |
| Full Model w./ direct prompt| 6.86    | 5.57      |
| 4-bit w./o. prompt          | 11.65   | 6.92      |
| 4-bit w./ direct prompt     | 7.04    | 5.88      |
| 4-bit w./ transferred prompt| 9.25    | 6.26      |
| 3-bit w./o. prompt          | 15.74   | 9.45      |
| 3-bit w./ direct prompt     | 7.76    | 6.33      |
| 3-bit w./ transferred prompt| 10.81   | 6.90      |
| 2-bit w./o. prompt          | 5883.13 | 2692.81   |
| 2-bit w./ direct prompt     | 14.98   | 16.67     |
| 2-bit w./ transferred prompt| 29.82   | 20.56     |

Given direct prompt receives a task-specific loss, our transferred prompt is, as expected, not as competitive as the direct one. However, such transferred prompt may significantly bridge the gap between a compressed and full model — e.g., our 3-bit & 4-bit quantized LLaMA-7B with transferred prompt can deliver on-par or better PPL than the full model on PTB and Wikitext2. We'd say this is an especially worthy contribution in practice, as one may possibly download the open-sourced transferable prompt to help on a compressed model with little effort.

Reviewer comment (Novelty):
Novelty is limited. The literature has applied the soft prefix tuning method to uncompressed models, and this paper applies the soft prefix tuning method to a compressed model. It's like a verification scenario of the prefix tuning method.

Author response:
While novelty is a multifaceted concept in academic research, we believe it can be roughly viewed from two fronts: ***empirical novelty***, which involves uncovering new properties and behaviors unknown to the wider community (e.g., the emergent ability [1], lottery ticket hypothesis [2]), and ***technical novelty***, which pertains to the development of new solutions or methodologies (e.g., LoRA [3]).

We argue our work is rich in empirical novelty in at least two aspects:

1. **Transformation of Prompt Tuning from Task-Specific to Transferable:** Prior to our study, only a few works studied the transferability of learned prompts between different tasks [4, 5, 6]. Specifically, [4] finds it is possible to transfer learnable prompts with **additional fine-tuning on downstream task**. However, as mentioned in the experiment section, all our reported results are **zero-shot**, i.e., no fine-tuning is needed to obtain transferability. Furthermore, [5] finds that the learned prompt can only be transferred among similar tasks. However, as verified by our experiments, our learned prompts are transferable between datasets, tasks, and compressed models.

2. **New avenue to enhance Compressed Model Accuracy:** We show that prompt tuning can effectively recover the accuracy drop of compressed models. Traditional model compression methods often demand extensive engineering efforts and intricate designs, as seen in popular model compression papers like GPTQ [7], SPQR [8], and AWQ [8]. However, our method simplifies this process significantly. We leverage prompt tuning to effectively recover the accuracy drop in compressed models, opening a new avenue to optimize the trade-off between accuracy and efficiency.

These findings are not only valid but also generalizable across various model families, datasets, and tasks, underscoring the broad applicability and impact of our work.

Regarding technical contribution, we agree that our method is a direct application of prompt tuning. However, we emphasize that **we intensionally kept our method as simple as possible**. This simplicity, is in our opinon more of a strength than a weakness, as it just underscores the unknown properties of straightforward prompt tuning methods; where the uncovering of these properties and the new pathways they opened for model compression constitute the true novelty of our paper. Thereby, we argue that we contribute substantially to the field's understanding of prompt tuning and model compression.




[1] Emergent Abilities of Large Language Models

[2] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

[3] LoRA: Low-Rank Adaptation of Large Language Models

[4] On Transferability of Prompt Tuning for Natural Language Processing

[5] SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer

[6] Reducing Retraining by Recycling Parameter-Efficient Prompts

[7] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

[8] SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

[9] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Reviewer comment (Experiments):
Experiments are only conducted on small and relatively old models (4 OPT and LLaMA-v1, all <7B).

Author response:
Below, we compare our approach and LoRA tuning on LLaMA-2-13B and BLOOM-7B models with 3-bit quantization using GPTQ. 

> Results suggest that our soft prompt approach vastly outperforms LoRA. In fact, the 3-bit quantized LLaMA-2-13B and BLOOM-7B equipped with our approach may even exceed their FP16 counterparts.

| Dataset | Model                       | Precision | Recover method | Trainable Params (M) | ppl   |
|---------|-----------------------------|-------|--------|----------------------|-----------|
| C4      | LLaMA-2-13B                 | fp16  | NA     | NA                    | 6.96      |
| C4      | LLaMA-2-13B           | 3bit  | NA     | NA                    | 9.24      |
| C4      | LLaMA-2-13B           | 3bit  | Ours   | 0.5                | **6.75**      |
| C4      | LLaMA-2-13B           | 3bit  | LoRA   | 26               | 8.15      |
| C4      | BLOOM-7B                     | fp16  | NA     | NA                    | 15.87     |
| C4      | BLOOM-7B                     | 3bit  | NA     | NA                    | 18.40     |
| C4      | BLOOM-7B                     | 3bit  |  Ours  | 0.4               | **13.54**     |
| C4      | BLOOM-7B                     | 3bit  | LoRA   | 15.7             | 17.26     |


---

Given that both LoRA weights and soft prompts are technically transferrable, we also compare the transferability between the two methods by first training them on the C4 dataset, then transferring and testing on the wikitext2 dataset. 

> Results suggest our soft prompt approach still outperforms LoRA in this transferred setting.

| Dataset   | Model       | Precision | Method | Transferred Params (M) | ppl       |
|-----------|-------------|-------|--------|------------------------|---------------|
| wikitext2 | LLaMA-2-13B | fp16  | NA     | NA                      | 5.58          |
| wikitext2 | LLaMA-2-13B | 3bit  | NA     | NA                      | 7.88          |
| wikitext2 | LLaMA-2-13B | 3bit  | Ours   | 0.5                  | **5.89**      |
| wikitext2 | LLaMA-2-13B | 3bit  | LoRA   | 26                 | 7.07     |
| wikitext2 | BLOOM-7B     | fp16  | NA     | NA                      | 13.26         |
| wikitext2 | BLOOM-7B     | 3bit  | NA     | NA                      | 16.06         |
| wikitext2 | BLOOM-7B     | 3bit  | Ours   | 0.4                 | **12.42**     |
| wikitext2 | BLOOM-7B     | 3bit  | LoRA   | 15.7              | 15.65    |

Reviewer comment (Experiments):
Do not compare with other strategies of finetuning in model compression, e.g., how about we apply LoRA tuning to restore the performance of model compression?

Author response:
Below, we compare our approach and LoRA tuning on LLaMA-2-13B and BLOOM-7B models with 3-bit quantization using GPTQ. 

> Results suggest that our soft prompt approach vastly outperforms LoRA. In fact, the 3-bit quantized LLaMA-2-13B and BLOOM-7B equipped with our approach may even exceed their FP16 counterparts.

| Dataset | Model                       | Precision | Recover method | Trainable Params (M) | ppl   |
|---------|-----------------------------|-------|--------|----------------------|-----------|
| C4      | LLaMA-2-13B                 | fp16  | NA     | NA                    | 6.96      |
| C4      | LLaMA-2-13B           | 3bit  | NA     | NA                    | 9.24      |
| C4      | LLaMA-2-13B           | 3bit  | Ours   | 0.5                | **6.75**      |
| C4      | LLaMA-2-13B           | 3bit  | LoRA   | 26               | 8.15      |
| C4      | BLOOM-7B                     | fp16  | NA     | NA                    | 15.87     |
| C4      | BLOOM-7B                     | 3bit  | NA     | NA                    | 18.40     |
| C4      | BLOOM-7B                     | 3bit  |  Ours  | 0.4               | **13.54**     |
| C4      | BLOOM-7B                     | 3bit  | LoRA   | 15.7             | 17.26     |


---

Given that both LoRA weights and soft prompts are technically transferrable, we also compare the transferability between the two methods by first training them on the C4 dataset, then transferring and testing on the wikitext2 dataset. 

> Results suggest our soft prompt approach still outperforms LoRA in this transferred setting.

| Dataset   | Model       | Precision | Method | Transferred Params (M) | ppl       |
|-----------|-------------|-------|--------|------------------------|---------------|
| wikitext2 | LLaMA-2-13B | fp16  | NA     | NA                      | 5.58          |
| wikitext2 | LLaMA-2-13B | 3bit  | NA     | NA                      | 7.88          |
| wikitext2 | LLaMA-2-13B | 3bit  | Ours   | 0.5                  | **5.89**      |
| wikitext2 | LLaMA-2-13B | 3bit  | LoRA   | 26                 | 7.07     |
| wikitext2 | BLOOM-7B     | fp16  | NA     | NA                      | 13.26         |
| wikitext2 | BLOOM-7B     | 3bit  | NA     | NA                      | 16.06         |
| wikitext2 | BLOOM-7B     | 3bit  | Ours   | 0.4                 | **12.42**     |
| wikitext2 | BLOOM-7B     | 3bit  | LoRA   | 15.7              | 15.65    |

Reviewer comment (Experiments):
Can the prompt prefix generalize across different models?

Author response:
A learned prompt can be generalized within the models that share the same tokenizer & embedding dim (e.g., LLaMA 1 and 2, or simply any model with different compression levels). While projecting soft prompt into different dimensions is certainly doable (e.g., project LLaMA-7B trained soft prompt with `4096` dim into `8192` to work with LLaMA-65B's), we leave it to future work. Yet, models with different tokenizers will require extra steps to form alignment, as different tokenizers will interpret the same soft prompt differently.

Reviewer comment (Presentation):
Very minor but presentation of some figures could be improved. Consider including baseline value (i.e., value of green line in Figures 2, 3, 4).

Author response:
We thank the reviewer for the strong support and meticulous reading!


### **[W1 - Including baseline value in Figures: Sure! Thank you for suggesting it]**
We have updated the paper by printing out the value for green lines in [Figures 2,3,4](https://openreview.net/pdf?id=Gdm87rRjep#page=4). Please see the revised submission for a better presentation.