Paper: DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR-2024)

Reviewer comment (Novelty):
The motivation is weak: soft prompts are already very parameter-efficient; it is not necessary to reduce the number of parameters in prompt tuning.

Author response:
We thank the reviewer for the feedback. We agree that prompt tuning is very parameter-efficient. However, this weakness is not applicable to our work, because we do **NOT** reduce the number of parameters compared to prompt tuning.

Reviewer comment (Novelty):
The motivation is weak: usually, the length of soft prompts is small, e.g., 8, 16, 32; the additional inference cost is minor compared with the current LLM, which can accept thousands of tokens as input.

Author response:
We would like to express our appreciation for the reviewer's feedback. However, we respectfully present a different perspective:

- Previous works [1,2,3,4,5,6,7,8] typically use soft prompts with a default length exceeding 100 virtual tokens. To support this point, we can directly quote previous work [5], which states that "***for hard sequence tasks, usually, a longer prompt than 100 would be helpful***". Similar claims can be found throughout the literature. To the best of our knowledge, there is no existing work that utilises soft prompts with a length of fewer than 32 virtual tokens while accepting thousands of tokens as the input.
- Let's take a step back and consider the situation more broadly. Assuming that we can only add at most 32 virtual tokens to the soft prompt, our proposed method allows us to achieve a highly competitive or even better performance than using around 100 virtual tokens, with minor operations and costs (*i.e.,* add the multiplication of low-rank matrices element-wisely to frozen word embeddings at the input sequence). We believe that this should be regarded as a notable strength of our work.

Reviewer comment (Evaluation):
For the method, the idea of using a LoRA matrix to approximate part of soft prompts is odd. Can the authors visualize the learned A and B matrices to explain why DePT is better than Prompt tuning?

Author response:
We thank the reviewer for this suggestion. The intuition of our method is that (1) given the same number of trainable parameters, allowing some updates for word embeddings will improve the performance; and (2) shorter soft prompt will improve the efficiency. To illustrate, the previous study [9] has shown that a soft prompt can interpolate between many token embeddings, enabling the representation of more abstract concepts compared to relying on a single discrete token. However, the soft prompt in the Prompt Tuning is consistently added at the beginning of the frozen word embedding. In contrast, we propose DePT, which decomposes the long soft prompt into a short soft prompt and a pair of low-rank matrices. This approach can (1) reduce the length of the soft prompt for better efficiency; and (2) permit representation updates within the frozen word embedding, thereby increasing the adaptability of input representations that were previously unavailable.

Reviewer comment (Experiments):
The tasks used in experiments are too simple; better to try more challenging tasks, like GeoQuery (Zelle & Mooney, 1996), NL2Bash (Lin et al., 2018), WebQS (Berant et al., 2013).

Author response:
We thank the reviewer for this suggestion. However, we respectfully disagree with the reviewer:

- All other reviewers (`mz3i`,`BZux`,`yLsP`) have commented positively on the thoroughness of our experiments, suggesting that they find our current evaluation to be adequate.
- Our work has already included a comprehensive evaluation of 23 datasets. These tasks range from natural language inference and sentiment analysis to open-domain question answering, common-sense reasoning, and even visual-language tasks. We believe that these tasks collectively represent a diverse and challenging set, and we do not consider them to be simple.
- Our experiments adhere to the settings and tasks used in previous works [1,2,3,4,5,6,7,8], strictly following the two most recent works [7,8].
- WebQs is a question-answering task, while GeoQuery and NL2Bash are generation tasks. Both of these task types have already been covered by the existing datasets in our work.

Reviewer comment (Reproducibility):
Datasets in GLUE and SuperGLUE are sensitive to hyperparameters. What are the hyperparameters for each dataset?

Author response:
We appreciate the reviewer's question. For the soft prompt, we search the learning rate within the set {3e-1, 4e-1, 5e-1}. For the low-rank matrice pairs, we search the learning rate within the set {1e-04, 5e-4, 1e-03}. For more details, please refer to our Section D: Implementation details in the Appendix and our anonymous code repository (https://anonymous.4open.science/r/DePT-8F43/README.md).

Reviewer comment (Evaluation):
In Table 3, DePT is much worse than FT on VQA, but can perform better on MSCOCO; why?

Author response:
We appreciate the reviewer's observation regarding the performance discrepancy between DePT and FT on VQA in Table 3. Our intention is to illustrate that DePT may not perform well in certain tasks. As discussed in Section 3.2 of our paper, this variation suggests that in specific tasks, the use of a greater number of trainable parameters could be beneficial. In essence, our work aims to show the versatility of our proposed method across a range of tasks while acknowledging that the effectiveness of PEFT approaches can be task-dependent. We believe that this insight contributes to a better understanding of PEFT approaches.

Reviewer comment (Reproducibility):
How to initialize the soft prompt in prompt tuning? Usually, they can be initialized randomly or use the embeddings of label tokens.

Author response:
We appreciate the reviewer's question. For the task with a large number of training examples (Table 1 and 2), we initiate the soft prompt using word embedding from the vocabulary. In the few-shot learning experiments (Table 4 and 5), we initiate the soft prompts from the pre-trained soft prompts on one of the source tasks (MNLI, QQP, SST-2, SQUAD, and ReCoRD), as we mentioned in the Implementation details of Section 3.1 and Appendix D.

Reviewer comment (Experiments):
Experiments are conducted only with smaller models (<1B model parameters). All experiments in the main paper are conducted on smaller models which hurt the technical contribution of the work. The paper presents improved results over several parameter-efficient methods, by a small margin, on models that have less than 1 billion parameters. The authors have included results of Llama2 in Appendix B but show improvements over prompt tuning on only one dataset (SST-2). It would be of interest to the community to know if the work is applicable to larger models such as T5-XL, T5-XXL, or Llama2 on more datasets.

Author response:
`Experiments with only smaller models (<1B model parameters).`

We respectfully disagree with the reviewer. While we acknowledge the importance of exploring this avenue, we are currently constrained by limited computing resources. This is a common issue faced by many researchers and practitioners. Therefore, the findings of this paper are relevant not just to us but also to others in a similar situation. However, also in response to this complain, we have conducted additional experiments on T5-3B. We report the average performance on the Glue dataset as well as inference speed, measured in inference samples per second. Our findings indicate that DePT (m=60, r=30) outperforms PT in terms of inference speed by 37%. This suggests the advantage of DePT increases as the model size increases. 

|  | Average Glue Performance | Inference Speed |
| --- | --- | --- |
| DePT (m=60, r=30) | 86.4 | 8.9 |
| PT (m=100) | 85.6 | 6.5 |

(*m indicates the length of the soft prompt*, *r indicates the rank of low rank matrices*)

We will do our best to get the necessary resources to investigate this important direction, and we leave this to future work.

Reviewer comment (Novelty):
Related work citation suggestion. The title of the work is similar to the following papers: [a] Decomposed prompting: A modular approach for solving complex tasks. ICLR 2023, and [b] Learning to compose soft prompts for compositional zero-shot learning. ICLR 2023. The proposed work differs from these, and the authors could choose to cite them and clarify the differences in the next version of the paper.

Author response:
`Related work: The title of your work is similar to the below papers [a,b] but the proposed work differs. You could choose to cite them and clarify the differences in the next version of the paper.`

We thank the reviewer for the insightful suggestion. We appreciate the opportunity to clarify that: the meaning of “compose” and the method are fundamentally different between previous works [5,6] and our work. Specifically, Decomposed Prompting [5] focuses on in-context learning, without the need to update parameters. Decomposed Prompting aligns closely with the work of chain-of-thought and self-consistency. In addition, CSP [6] treats the attributes and objects that are composed to define classes as learnable tokens within the vocabulary. In contrast, our proposed method DePT does not train soft prompts associated with any vocabulary token, nor does it add additional tokens to the vocabulary. The main goal of DePT is to improve the efficiency of Prompt Tuning (PT) due to the increased input sequence issue. We have mentioned these two works in the Appendix of our revised version.

Reviewer comment (Experiments):
Completeness suggestion for additional experimental results. For completeness, the authors could include the GPT-2 results in the Appendix on GLUE and SuperGLUE. It would be interesting to see if there are any changes in the performance.

Author response:
`For completeness, it would be great if the authors could include the GPT-2 results in the Appendix on GLUE and SuperGLUE. I’d be curious to see if there are any changes in the performance.`

We thank the reviewer for the insightful suggestion. In response, we have added this part of the results to Figure 6 in Appendix A.

Reviewer comment (Reproducibility):
Clarification question on initialization. In the implementation details, the authors state that the soft prompts and low rank matrices are initialized from soft prompts derived from one of the source tasks. It is unclear why this is necessary, and clarification is requested.

Author response:
`In implementation details, you have included that the soft prompts and low-rank matrices are initialized from soft prompts derived from one of the source tasks. Could you clarify this detail? It is unclear why this is necessary.`

We thank the reviewer for the question. This is due to the limitation of Prompt Tuning, which is hard to initialise soft prompts especially when the number of training data is not large, which is discussed in Section 1 of our paper. To mitigate this issue, previous works (e.g., [1,2]) have proposed Parameter-efficient Transfer Learning (PETL). PETL initially trains soft prompts on source tasks and then fine-tunes them on target tasks. PETL can effectively improve the performance of the PT and make it easier to use. In our work, we have demonstrated that our approach is compatible with PEFL approaches in the few-shot learning experiments.

Reviewer comment (Experiments):
Question on scaling factor for $\mathbf{BA}$. The reviewer asks if $\mathbf{BA}$ could use a scaling factor like $\frac{\alpha}{r}$. It is possible that the lack of such a scaling factor could be causing the low performance when only the decomposed matrices are trained.

Author response:
`Do you think BA could use a scaling factor like a/r? It is possible that this could be causing the low performance due to when only the decomposed matrices are trained.`

We thank the reviewer for the insightful suggestion. We have explored the possibility of incorporating a scaling factor, such as a/r, within the BA. However, our experiments indicate that this approach did not lead to an improvement in performance.  We have discussed them in the Appendix of our revised version.

Reviewer comment (Reproducibility):
Question on setting sequence length $s$. In the background, it is mentioned that $s$ is the maximum sequence length. The reviewer asks how $s$ would be set for Llama2 or models that do not have a fixed maximum sequence length.

Author response:
`In the background, you have mentioned that s is the maximum sequence length. How would you set s for Llama2 or models that do not have a fixed maximum sequence length?`

We thank the reviewer for the insightful suggestion. We appreciate the opportunity to clarify: the max sequence length is for truncation and padding, and it does not represent the maximum input length for the models.

Reviewer comment (Experiments):
Some of the important baseline methods like IA3 are missing. See questions below.

Author response:
We thank the reviewer for the insightful suggestion. In response, we have performed additional experiments as follows:

(1) Firstly, we have added LoRA and IA3 as baselines and filled in the missing numbers for Table 1. Specifically, we train LoRA and IA3 with 30k steps with a batch size of 16. We use the rank as 35 for LoRA. We perform a learning rate search among 3e-3, 5e-4, 1e-4, and 5e-5, and report the test results of the best model on the development set. Our experiment shows that DePT obtains better mean performance on Glue and SuperGlue benchmarks.

|                 | MNLI | QQP | QNLI | SST-2 | STS-B | MRPC | RTE | CoLA | Mean |
|----------|-------|-------|----|----|------|------|------|------|------|
|LoRA        | 86.3  | 89.0  | 93.2  | 94.3  | 90.9  | 90.1  | 75.5  | 63.3  | 85.3 | 
|IA3           |  85.7|  90.3 | 93.6 | 94.2 |   90.5 | 86.8 | 82.0  | 58.3 | 85.2 |

|                                | Multirc | Bool   | WiC   | WSC | CB    | Mean |
|----------------    |-------  |-------|----    |----    |------|------|
|LoRA                       | 72.6     | 81.3    |  68.3 | 67.3   | 89.3  | 75.8 |
|IA3                           | 73.3    | 80.9    |  67.7 | 73.1    | 85.7  | 76.1  | 
|Fine-tuning(m)       |  74.4     | 81.1     | 70.0  | 71.2    |  85.7 | 76.1  |
|Adapter(m)             | 72.6  | 82.3     |  66.5 | 67.3   |  89.3  |  75.6 |
|HyperFormer(m)    | 72.9  | 82.5      |  69.0 | 67.3   | 85.7  | 75.4 |
|HyperDecoder(m)  | 70.4 |  78.8     | 67.1    | 61.5   | 82.1  | 72.0 |
*m indicates multi-task learning*

(2) We also add LoRA as a baseline in Table 2. While it is worth noting that LoRA performs highly competitively on WinoGrande, YelpPolarity, SciTail, and PAWS datasets, we have encountered challenges when training LoRA on certain MRQA tasks, such as HotpotQA. Our experiment shows that the advantage of DePT persists.

|                 | NQ        | HP      | SQA  | News | Mean |
|----------|-------  |-------|----    |----   |------  |
|LoRA        | 69.7      |  61.6  |71.9   | 56.6  |    65.0 |
|DePT   | 73.2 | 76.8 | 77.6 | 64.4 | 73.0 |

| |  WG | Yelp      | SciTail | PAWS | Mean |
|----------|-------  |-------|----      |----   |----   |
|LoRA        | 58.2 | 97.1       | 94.7   | 94.0    | 86.0 |
|DePT   | 59.0 | 96.8 | 95.6 | 93.7 | 86.3 |

(3) In addition, we have added IA3 as a baseline to Table 4 in the few-shot learning setting (4, 16, or 32 training examples). We report average results across three seeds. Specifically, we find that DePT, along with other PT variants (ATTEMPT and MPT), can outperform IA3, highlighting the effectiveness of DePT with transfer learning approaches.

|                 | 4  | 16  | 32  | 
|-------     |-------|----|----   |
| Boolq      | 56.7  | 62.0 | 67.2 |
| CB          | 65.5 | 71.4 | 75.0 |
| SciTail      | 65.4 | 74.4 | 80.4 |

Reviewer comment (Theory):
The idea is interesting, however some more intuition on why this works might strengthen the paper.

Author response:
We acknowledge the reviewer's inquiry regarding the intuition of our method. The intuition of our method is that (1) **given the same number of trainable parameters, allowing some updates for word embeddings will improve the performance**; and (2) **shorter soft prompt will improve the efficiency.** To illustrate, the previous study [1] has shown that a soft prompt can interpolate between many token embeddings, enabling the representation of more abstract concepts compared to relying on a single discrete token. However, the soft prompt in the Prompt Tuning is consistently added at the beginning of the frozen word embedding. In contrast, we propose DePT, which decomposes the long soft prompt into a short soft prompt and a pair of low-rank matrices. This approach can (1) reduce the length of the soft prompt for better efficiency; and (2) permit representation updates within the frozen word embedding, thereby increasing the adaptability of input representations that were previously unavailable. 

[1] Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. Findings of EMNLP 2022.

Reviewer comment (Experiments):
Many of the tables have inconsistent baseline and incomplete results. All the superglue results should be filled in Table 1. I understand that the numbers are not available in past papers but it is not hard to obtain these numbers. This is important because the LORA method is the closest PEFT method on the glue benchmark, however it is missing when looking at Superglue. Similarly, Table 2 should have LoRA as a baseline.

Author response:
(1) Firstly, we have added LoRA and IA3 as baselines and filled in the missing numbers for Table 1. Specifically, we train LoRA and IA3 with 30k steps with a batch size of 16. We use the rank as 35 for LoRA. We perform a learning rate search among 3e-3, 5e-4, 1e-4, and 5e-5, and report the test results of the best model on the development set. Our experiment shows that DePT obtains better mean performance on Glue and SuperGlue benchmarks.

|                 | MNLI | QQP | QNLI | SST-2 | STS-B | MRPC | RTE | CoLA | Mean |
|----------|-------|-------|----|----|------|------|------|------|------|
|LoRA        | 86.3  | 89.0  | 93.2  | 94.3  | 90.9  | 90.1  | 75.5  | 63.3  | 85.3 | 
|IA3           |  85.7|  90.3 | 93.6 | 94.2 |   90.5 | 86.8 | 82.0  | 58.3 | 85.2 |

|                                | Multirc | Bool   | WiC   | WSC | CB    | Mean |
|----------------    |-------  |-------|----    |----    |------|------|
|LoRA                       | 72.6     | 81.3    |  68.3 | 67.3   | 89.3  | 75.8 |
|IA3                           | 73.3    | 80.9    |  67.7 | 73.1    | 85.7  | 76.1  | 
|Fine-tuning(m)       |  74.4     | 81.1     | 70.0  | 71.2    |  85.7 | 76.1  |
|Adapter(m)             | 72.6  | 82.3     |  66.5 | 67.3   |  89.3  |  75.6 |
|HyperFormer(m)    | 72.9  | 82.5      |  69.0 | 67.3   | 85.7  | 75.4 |
|HyperDecoder(m)  | 70.4 |  78.8     | 67.1    | 61.5   | 82.1  | 72.0 |
*m indicates multi-task learning*

(2) We also add LoRA as a baseline in Table 2. While it is worth noting that LoRA performs highly competitively on WinoGrande, YelpPolarity, SciTail, and PAWS datasets, we have encountered challenges when training LoRA on certain MRQA tasks, such as HotpotQA. Our experiment shows that the advantage of DePT persists.

|                 | NQ        | HP      | SQA  | News | Mean |
|----------|-------  |-------|----    |----   |------  |
|LoRA        | 69.7      |  61.6  |71.9   | 56.6  |    65.0 |
|DePT   | 73.2 | 76.8 | 77.6 | 64.4 | 73.0 |

| |  WG | Yelp      | SciTail | PAWS | Mean |
|----------|-------  |-------|----      |----   |----   |
|LoRA        | 58.2 | 97.1       | 94.7   | 94.0    | 86.0 |
|DePT   | 59.0 | 96.8 | 95.6 | 93.7 | 86.3 |

Reviewer comment (Experiments):
IA3 is a recent PEFT method that is very parameter efficient and is missing from the baseline comparison. It is a crucial baseline, especially in the case of the few-shot adaptation. It should be added at least to Tables 4 and 5, and adding it to Tables 1-2 would also be good.

Author response:
We thank the reviewer for the insightful suggestion. In response, we have performed additional experiments as follows:

(1) Firstly, we have added LoRA and IA3 as baselines and filled in the missing numbers for Table 1. Specifically, we train LoRA and IA3 with 30k steps with a batch size of 16. We use the rank as 35 for LoRA. We perform a learning rate search among 3e-3, 5e-4, 1e-4, and 5e-5, and report the test results of the best model on the development set. Our experiment shows that DePT obtains better mean performance on Glue and SuperGlue benchmarks.

|                 | MNLI | QQP | QNLI | SST-2 | STS-B | MRPC | RTE | CoLA | Mean |
|----------|-------|-------|----|----|------|------|------|------|------|
|LoRA        | 86.3  | 89.0  | 93.2  | 94.3  | 90.9  | 90.1  | 75.5  | 63.3  | 85.3 | 
|IA3           |  85.7|  90.3 | 93.6 | 94.2 |   90.5 | 86.8 | 82.0  | 58.3 | 85.2 |

|                                | Multirc | Bool   | WiC   | WSC | CB    | Mean |
|----------------    |-------  |-------|----    |----    |------|------|
|LoRA                       | 72.6     | 81.3    |  68.3 | 67.3   | 89.3  | 75.8 |
|IA3                           | 73.3    | 80.9    |  67.7 | 73.1    | 85.7  | 76.1  | 
|Fine-tuning(m)       |  74.4     | 81.1     | 70.0  | 71.2    |  85.7 | 76.1  |
|Adapter(m)             | 72.6  | 82.3     |  66.5 | 67.3   |  89.3  |  75.6 |
|HyperFormer(m)    | 72.9  | 82.5      |  69.0 | 67.3   | 85.7  | 75.4 |
|HyperDecoder(m)  | 70.4 |  78.8     | 67.1    | 61.5   | 82.1  | 72.0 |
*m indicates multi-task learning*

(3) In addition, we have added IA3 as a baseline to Table 4 in the few-shot learning setting (4, 16, or 32 training examples). We report average results across three seeds. Specifically, we find that DePT, along with other PT variants (ATTEMPT and MPT), can outperform IA3, highlighting the effectiveness of DePT with transfer learning approaches.

|                 | 4  | 16  | 32  | 
|-------     |-------|----|----   |
| Boolq      | 56.7  | 62.0 | 67.2 |
| CB          | 65.5 | 71.4 | 75.0 |
| SciTail      | 65.4 | 74.4 | 80.4 |