Paper: When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations (ICLR-2024)

Reviewer comment (Presentation):
The paper has some clarity issues; specifically, the attention figures in Figures 1, 3, etc. are difficult to interpret in terms of the patterns the authors are trying to highlight. The captions could include interpretation guidance for readers (e.g., indicating that the figures should be read left-to-right, where …).

Author response:
Thank you for highlighting these clarity issues. 
We marked more clearly the part of the attention patterns that we refer to and elaborated further in the figure captions and the text in the revised version*. In Fig. 1 that would be the attention when the first response $Y_1$ is being generated, as this determines which task it going to be solved. For sorting in ascending order, when generating $Y_1$ we need to attend to the smallest values. And when sorting in descending order when generating $Y_1$ one should attend to the largest values. However, despite the prefix-tuning on the descending task, the model still attends to the smallest values as prefix-tuning cannot overcome the pre-trained attention pattern for sorting in ascending order. We also spotted a mistake in the labels in the "Model response" part of Figure 1 that we have now fixed. We believe that thanks to your feedback our paper is now much easier to read.

Reviewer comment (Theory):
For Theorems 1 and 2, the provided proofs are not fully convincing. Although the reviewer can see why the theorems might be true, a more detailed proof sketch would improve clarity and provide stronger justification for the claim that soft prompting and prefix-tuning are more expressive in output generation than prompting.

Author response:
For both of these theorems, we provide proofs by construction. That is, we construct an example of a transformer model that attains the expressiveness described in the theorems. For Theorem 1, that is a model that can generate any target text unconditionally, and for Theorem 2, a model that can generate a target conditional token response to any token the user provides. In the Appendix, we extend this to a model that can generate any length conditional response to user input of any length. The Appendix provides the explicit constructions but also explanations as to how we came up with them and what the role of each individual component is.

We also edited Section 3 and hope that it is much more clear now.

Reviewer comment (Theory):
In equation 7, it is unclear what the equivalence is between the pre-trained (\( t_i^{ft} \)) and prefix-tuned (\( t_i^{pt} \)) model outputs, or whether such an equivalence exists. While the reviewer generally accepts the argument that changes to \( W_v \) produce more output changes than adding the \( W_V s_1 \) term, it is not convincing that this is always true or under what conditions equivalence might be achieved. The reviewer suggests that to firmly support the claim that pre-training has more potential for output variation than prefix-tuning, it would be useful to identify scenarios where adding the bias term from prefix-tuning can or cannot be equivalent to adding an update to \( W_V \).

Author response:
This is an excellent question and one that also interests us so much that we are currently working on a follow-up work! 

Let us give an example of why adding the bias term from prefix-tuning can never be equivalent to adding a general update to $W_V$.
Consider that all inputs to the layer are on the surface of a sphere centered at the origin and that $W_V$ is a rotation matrix. Applying this rotation matrix would still keep all the points on the surface. Modifying $W_V$ to be $W_V'=W_V+\Delta W_V$, a different rotation matrix, will have the same property. However, there is no non-zero vector that one can add to a set of points on the surface of a sphere that will keep them all on the surface of the sphere. Therefore, this is a transformation that can be expressed with a matrix update (full fine-tuning) but not with a bias (prefix-tuning).

An open question is whether there can exist particular pretrained transformers that can interpret this bias in a way that makes it much more expressive (though for a given transformer size it would still be less expressive than full fine-tuning). However, even if that would be possible, it would likely be very sample-inefficient and difficult to converge. Therefore, for all practical reasons, one would likely still be better using other fine-tuning approaches, e.g., LoRA. We added experiments that show that in Section 6.2 and Appendix C. We appreciate that this is a very interesting question so we extended the discussion on the deeper effects in the new Section 6. However, the full formal analysis is quite involved and of a very different flavor than the approach in this work, so we believe it is better suited as a separate follow-up work.

Reviewer comment (Novelty):
Prompting is not only used in linear data like text but also applied to non-linear data recently like graphs. It would be more solid to discuss them in the related work section (e.g. X Sun, et al. "All in One: Multi-task Prompting for Graph Neural Networks". KDD2023).

Author response:
Unfortunately, we are not very familiar with the graph prompting area. 
After reading the suggested paper, our impression is that it incorporates the prompts in an additive way (adding the prompt embeddings to the content embeddings or taking a dot product), rather than in the concatenation way typically used in language models. As a result, the graph prompt would affect the attention layers in a very different way.
For example, we would not be able to separate the attention over the prompt positions and the attention over the content positions because the positions contain both prompts and content. For this reason, this approach falls out of the scope of our work.

We tried to keep our analysis focused on the most general architecture to keep the results as broadly applicable as possible. That being said, as long as the attention mechanism works in a similar way, the limitations should also hold. For example, our results immediately translate to Visual Prompt Tuning (Jia et al., 2022). 

Regardless, given our limited expertise in graph transformers and graph prompting, it is difficult to provide more detailed answer. However, as graph problems admit a larger variety of attention architectures, we believe that this could be an interesting avenue for future work.

Jia, M. et al. (2022). Visual Prompt Tuning. ECCV 2022.

Reviewer comment (Theory):
It would be even better if the author could further confirm whether their theoretical analysis applies to the graph prompting area.

Author response:
Unfortunately, we are not very familiar with the graph prompting area. 
After reading the suggested paper, our impression is that it incorporates the prompts in an additive way (adding the prompt embeddings to the content embeddings or taking a dot product), rather than in the concatenation way typically used in language models. As a result, the graph prompt would affect the attention layers in a very different way.
For example, we would not be able to separate the attention over the prompt positions and the attention over the content positions because the positions contain both prompts and content. For this reason, this approach falls out of the scope of our work.

We tried to keep our analysis focused on the most general architecture to keep the results as broadly applicable as possible. That being said, as long as the attention mechanism works in a similar way, the limitations should also hold. For example, our results immediately translate to Visual Prompt Tuning (Jia et al., 2022). 

Regardless, given our limited expertise in graph transformers and graph prompting, it is difficult to provide more detailed answer. However, as graph problems admit a larger variety of attention architectures, we believe that this could be an interesting avenue for future work.

Jia, M. et al. (2022). Visual Prompt Tuning. ECCV 2022.

Reviewer comment (Experiments):
It would be better to further test models larger than LLaMA-7B as models with more parameters may exhibit different properties.

Author response:
Our contributions are primarily theoretical and our experiments are simply a means to validating them. We use LLaMA-7B only to validate that a prefix cannot change the relative attention and only acts as a bias. We only look at this effect at the first layer as the non-linear effects kick in for the latter layers (we have elaborated further on this in Section 6 in the revised manuscript*). As our theory characterizes the effect of the prefix exactly, there would be no difference to this validation experiment if we look at larger models.

Reviewer comment (Experiments):
An additional ablation experiment may be required in subsection 'Prefix-tuning can combine knowledge from pretraining tasks to solve new tasks'. The authors utilize a 4-layer 4-head model to validate the point that prefix-tuning can learn a new task as long as the “skill” required to solve the new task is a combination of “skills” the pretrained model has seen. However, in a 4-layer 4-head model, a prefix-induced bias can have non-linear behavior when passed through non-linear MLPs and attention blocks as mentioned in Section 5.2. It would be better to further test a 1-layer 4-head model for further clarification.

Author response:
This is a great suggestion but we actually performed an even more illustrative experiment.
The claim of that subsection is that prefix-tuning can combine pre-training skills but cannot learn completely new tasks.
The "sort and increment" task is the one that can be solved via combining pre-training skills.
We added a new task, "double histogram" (mapping each element to the number of elements in the sequence with the same value) which cannot be solved compositionally from the pre-training tasks.
Prefix-tuning can reach very high accuracy on "sort and increment" but cannot learn "double histogram" at all.
Therefore, this demonstrates that the ability of prefix-tuning to learn one but not the other task is not a matter of scale of the model but only due to the mismatch of the pre-training data and the fine-tuning task.

Reviewer comment (Experiments):
Is the cross-layer effect enough to make the prefix fine-tuning learn new tasks when the transformer consists of more attention layers like many current language models?

Author response:
We argue that, while it may potentially be possible for a very deep model with a lot of training to be able to learn some simple novel tasks, the sample complexity for this would be extremely high. Therefore, for all practical purposes, the depth does not help much. We think this is a really important question so we dedicated a new Section 6 in the revised manuscript on this question.
We are also currently working on a followup work that formally examines the deeper effects and whether under some conditions prefix-tuning can have universal approximation properties.

However, in Section 6.2 we added an experiment that shows that even if prefix-tuning has universal approximation properties, the sample complexity would be impracticably high, especially compared to other fine-tuning approaches. In particular, we show that prefix-tuning fails to learn the novel task "double histogram" but rank-1 LoRA with the exact same number of learnable parameters as the prefix easily reaches 92% accuracy. Therefore, even if prefix-tuning for deep models does have expressive capacity, it would be impractically difficult to optimize for and one would still be better using, e.g., LoRA. Fundamentally, learning is not only about density-type approximation results but also about the rate of convergence.

Reviewer comment (Experiments):
Is there any potential improvements for tuning methods based on the analysis?

Author response:
A potential improvement could be combining prefix-tuning with LoRA. Combining the two an studying whether prefix-tuning + LoRA is empirically better than LoRA. If that is the case, then this indicates that prefix-tuning could potentially have more value beyond just efficiency. This would be an interesting new direction to explore.