IDPG: An Instance-Dependent Prompt Generation Method
Abstract
Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage.
It freezes the pre-trained language model and only optimizes a few taskspecific prompts.
In this paper, we propose a conditional prompt generation method to generate prompts for each input instance, referred to as the Instance-Dependent Prompt Generation (IDPG).
Unlike traditional prompt tuning methods that use a fixed prompt, IDPG introduces a lightweight and trainable component to generate prompts based on each input sentence.
Extensive experiments on ten natural language understanding (NLU) tasks show that the proposed strategy consistently outperforms various prompt tuning baselines and is on par with other efficient transfer learning methods such as Compacter while tuning far fewer model parameters.
Introduction
Recently, pre-training a transformer model on a large corpus with language modeling tasks and finetuning it on different downstream tasks has become the primary transfer learning paradigm in natural language processing (Devlin et al., 2019).
Notably, this paradigm requires updating and storing all the model parameters for every downstream task.
As the model size proliferates (e.g., 330M parameters for BERT (Devlin et al., 2019) and 175B for GPT3 (Brown et al., 2020)), it becomes computationally expensive and challenging to fine-tune the entire pre-trained language model (LM).
Thus, it is natural to ask whether we can transfer the knowledge of a pre-trained LM into downstream tasks by tuning only a tiny portion of its parameters, with most of them freezing.
[Figure]
prompt tuning
P
[Figure]
[Table]
CompacterAdapterIDPG
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
e
Our method approaches RoBERTa-FT’s performance and uses fewer parameters than Adapter-based methods.
Studies have attempted to address this question from different perspectives.
One line of research (Li and Liang, 2021) suggests augmenting the model with a few small trainable modules and freezing the original transformer weight.
Take
Adapters (Houlsby et al., 2019; Pfeiffer et al., 2021, 2020) for example, they insert a small set of additional modules between each transformer layer.
Only these additional and task-specific modules are trained during fine-tuning, reducing the trainable parameters to ∼ 1–3% of the original transformer model per task.
Compacter (Mahabadi et al., 2021) optimizes the training parameters further by designing a lightweight module to replace the bottleneck structure in Adapters.
Another line of work focuses on prompting.
The GPT-3 models (Brown et al., 2020; Schick and Schütze, 2021) find that with proper manual prompts, a pre-trained LM can successfully match the fine-tuning performance of BERT models.
LMBFF (Gao et al., 2021a), EFL (Wang et al., 2021), and AutoPrompt (Shin et al., 2020) further this direction by insert prompts in the input embedding layer.
However, these methods rely on grid-search for a natural language-based prompt from an ample search space, resulting in difficulties to optimize.
To tackle this issue, prompt tuning (Lester et al., 2021), prefix tuning (Li and Liang, 2021), and Ptuning (Liu et al., 2021a,b) are proposed to prepend trainable prefix tokens to the input layer and train these soft prompts only during the fine-tuning stage.
In doing so, the problem of searching discrete prompts is converted into a continuous optimization task, which can be solved by a variety of optimization techniques such as SGD and thus significantly reduced the number of trainable parameters to only a few thousand.
However, all existing prompt-tuning methods have thus far focused on task-specific prompts, which creates a large gap between the pre-training and fine-tuning objectives.
Specifically, it is unlikely to see many different sentences with the same prefix in the pre-training corpus.
Thus, a unified prompt may disturb the prediction and lead to a performance drop.
In light of these limitations, we instead ask the following question: Can we generate input-dependent prompts to smooth the domain difference?
This paper presents the instance-dependent prompt generation (IDPG) strategy for efficiently tuning large-scale LMs.
Unlike traditional prompttuning methods that rely on a fixed prompt for each task, IDPG instead develops a conditional prompt generation model to generate prompts for each instance.
Formally, the IDPG generator can be denoted as f ( x ; W ) , where x is the instance representation and W represents the trainable parameters.
Note that by setting W to a zero matrix and only training the bias, IDPG would degenerate into the traditional prompt tuning process (Lester et al., 2021).
To further reduce the number of parameters in the generator f ( x ; W ) , we propose to apply a lightweight bottleneck architecture (i.e., a two-layer perceptron) and then decompose it by a parameterized hypercomplex multiplication (PHM) layer (Zhang et al., 2021).
To summarize, this work makes the following contributions:
• We introduce an input-dependent prompt generation method—IDPG—that only requires training 134K parameters per task, corresponding to ∼ 0.04% of a pre-trained LM such as RoBERTa-Large (Liu et al., 2019).
• Extensive evaluations on ten natural language understanding (NLU) tasks show that IDPG consistently outperforms task-specific prompt tuning methods by 1.6–3.1 points.
Additionally, it offers comparable performance to Adapter-based methods while using fewer parameters.
• We conduct substantial intrinsic studies, revealing how and why each component of the proposed model and the generated prompts could help the downstream tasks.
Preliminary 2. Manual Prompt
Manual prompt learning (Brown et al., 2020;
Schick and Schütze, 2021) inserts a pre-defined label words in each input sentence.
For example, it reformulates a sentence sentiment classification task with an input sentence S 1 as x in =
[CLS] P [SEP] S 1 [EOS] , where P is the prompt such as “indicating the positive user sentiment”.
Using the pre-trained language model M , we can obtain the sentence representation
h
[CLS] =
M ( x in ) , and train a task-specific head softmax ( Wh [CLS] ) to maximize the logprobability of the correct label.
LM-BFF (Gao et al., 2021a) shows that adding a specifically designed prompt during fine-tuning can benefit the few-shot scenario.
EFL (Wang et al., 2021) further suggests that reformulating the task as entailment can further improve the performance in both lowresource and high-resource scenarios.
2. Prompt Tuning
Prompt tuning (Lester et al., 2021), prefix tuning (Li and Liang, 2021), and P-tuning (Liu et al., 2021a,b) methods propose to insert a trainable prefix in front of the input sequence.
Specifically, they reformulate the input for single sentence tasks as
[Equation]
and for sentence pair tasks as
[Equation]
where W p is the embedding table of the inserted prompt, S 2 and S 3 are input sentences, and E denotes the operation of tokenization and extraction of embeddings.
Apart from LM-BFF and EFL, there is no corresponding real text for the prompt as W p is a set of random-initialized tensors to represent the soft prompt.
[Figure]
Instance-Dependent Prompt Generation (IDPG)
We now introduce our proposed method, IDPG, along with various model optimizations.
The main procedure is illustrated in Figure
[Figure]
3. Instance-Dependent Generation
Let us assume a task T with training data D train = { ( x i , y i ) }
Ki = 1 .
Following prompt tuning, we define the input x i =
E
( [SEP] S 1 [SEP] S 2 [EOS] ) for sentence-pair task or x i =
E ( [SEP] S 1 [EOS] ) for single-sentence task, where E ( · ) is the token embedding for input sentences.
Different from all previous works that only define a task-specific prompt
W p ( T ) ∈ R d × t , where t is the number of tokens in prompt representation and d is the hidden dimension, we propose a instance-dependent prompt generation method.
Specifically, we suppose that the generation of prompt should not only depend on the task T , but also be affected by input sequence x i .
If M ( x i ) ∈ R
d is a representation of the input sequence x i from same pre-trained LM M , we design a lightweight model G to generate the prompt,
[Equation]
Then, we insert a prompt W p ( T ) together with input sequence x i to infer y i during fine-tuning.
In this way, we have a unified template
[Equation]
where W is the trainable LM classification head.
To reduce the number of trainable parameters in G , we apply a lightweight bottleneck architecture (i.e., a two-layer perceptron) for generation.
As illustrated in Figure 2 (c), the generator G first projects the original d -dimensional sentence representation h i into m dimensions.
After passing through a nonlinear function, generator G projects the hidden representation back to a d dimensions with t timestamps.
The total number of parameters for generator G is m ( d + 1 ) + td ( m + 1 ) (bias term included).
This model can be regarded as the general version of prompt tuning: in the second layer of G , the bias term td is a task-specific prompt, with preceding parts td × m generating an instance-dependent prompt.
The final prompt our method generated is a combination of both.
In short, what we discussed here is to generate a t -length prompt for one Transformer layer.
An optimization of multi-layer prompt generation will be introduced in Section 3.2.2.
We can control the added number of trainable parameters by setting m ≪ d , but it is still expensive since hidden dimension d is usually large ( in BERT/RoBERTa-Large).
In the sequel, we will introduce a parameter squeezing method to further reduce trainable parameters without sacrificing performance.
Note that our proposed method relies on the input sentence representation M ( x i ) to generate prompts.
One caveat is that this method will have two forward passes of the pre-trained LM during inference time – first to generate M ( x i ) and then to generate classification results.
However, the sentence representation M ( x i ) used in our method is task-agnostic.
In practice, we can cache the prediction M ( x i ) and use it in various downstream tasks or rely on a lightweight sentence representation such as GloVe (Pennington et al., 2014)
(Cf.
Section 4.5.1).
3. Optimization
We propose two optimization techniques to further improve our proposed method.
3.2. Parameterized Hypercomplex Multiplication (PHM) Layers
Inspired by the recent application of parameterized hypercomplex multiplication (PHM) layers (Zhang et al., 2021) in Compacter (Mahabadi et al., 2021), we leverage PHM layers to optimize our prompt generator, G .
Generally, the PHM layer is a fullyconnected layer with form y
=
W x + b , where x ∈ R d is the input feature, y ∈ R m is the output feature, and W ∈ R m × d and b ∈ R m are the trainable parameters.
When m and d are large, the cost of learning W becomes the main bottleneck.
PHM replaces the matrix W by a sum of Kronecker products of several small matrices.
Given a user-defined hyperparameter n ∈ Z + that divides m and d , W can be calculated as follows:
W = n ∑ i =
A i (cid:79) B i (4)
where A i ∈ R n × n , B i ∈
R mn × dn , and (cid:78) is Kronecker product.
In this way, the number of trainable parameters is reduced to n × ( n × n + mn × dn ) =
n 3 + m × d n .
As n is usually much smaller than m and d , PHM reduces the amount of parameters by a factor of n .
Suppose that we have a two layer perceptron with down-sample projection W 1 ∈ R m × d
and upsample projection W 2 ∈ R t × d × m , where d is the input embedding dimension, m is the hidden layer dimension, and t is the number of tokens we generate.
For example, we use RoBERTa-Large with hidden size d = 1024, generator hidden size m = 256, n = 16, prompt length t = 5.
By substituting the
W 1 and W 2 by two PHM layers and letting A i shared by both layers, we can reduce the number of parameters from 1.5M to 105K.
3.2. Multi-layer Prompt Tuning
Prompt tuning (Lester et al., 2021) and Ptuning (Liu et al., 2021b)
both insert continuous prompts into the first transformer layer (cf.
Figure 2(b)).
While proven efficient in some specific settings, single layer prompt tuning has two main limitations: (i)
Capturing deep contextual information: the impact of the first-layer prompts on final prediction is low when transformer goes deeper.
(ii) Generalizing to long sequence tasks: it is unclear that prompt tuning can perform well in tasks with long input when only a limited number of parameters can be inserted in single layer.
Following Prefix tuning (Li and Liang, 2021) and P-tuning v2 (Liu et al., 2021a), we prepend our generated prompts at each transformer layer to address the above issues.
However, simply generalizing our model (IDPG) to a multi-layer version (M-IDPG), will significantly increase the number of training parameters, since each layer requires an independent generator G .
Instead, we explore different architectures in Section 4.5.3 to balance the number of tuned parameters against model performance.
In short, assuming each layer generator G i has form y
=
W x + b i , we share the weight matrix
W across generators and set the bias term b i ∈ R m to be layer-specific, where i = 1 ,..., N is the layer index and N is the number of transformer layers.
[Table]
Experiment Results 4. Experimental Setup
We evaluate on ten standard natural language understanding (NLU) datasets – MPQA (Wiebe et al., 2005), Subj (Pang and Lee, 2004), CR (Hu and
Liu, 2004), MR (Pang and Lee, 2005), and six tasks from GLUE (Wang et al., 2019), viz.
SST-2,
QNLI, RTE, MRPC, STS-B (Cer et al., 2017) and
QQP.
We compare our proposed method with a wide range of methods, as follows:
Transformer fine-tuning: We instantiated two versions – a vanilla transformer fine-tuning (Liu et al., 2019) and the entailment-based finetuning (Wang et al., 2021).
Prompt tuning: We implemented two versions – standard prompt tuning (Lester et al., 2021) and multi-layer prompt tuning (Li and Liang, 2021; Liu et al., 2021a).
Adapter-based fine-tuning: This efficient transfer learning method inserts an adaptation module inside each transformer layer including Compactor (Mahabadi et al., 2021) and Adapter (Houlsby et al., 2019).
We compare these against two versions of singlelayer instance-dependent generation methods: SIDPG-DNN and S-IDPG-PHM.
The first version is based on a 2-layer perceptron generator, which contains 1.5M parameters.
The second one uses the PHM layer and only contains 105K parameters.
We also explore three versions of multilayer instance-dependent generation methods:
M-IDPG-DNN, M-IDPG-PHM, M-IDPG-PHMGloVe.
Again, the difference between the first two is in the prompt generator, while M-IDPG-PHMGloVe uses GloVe to encode input sequences.
For a fair comparison, all the pre-trained LMs are 24-layer 16-head RoBERTa-Large models (Liu et al., 2019).
Additional training details can be found in Appendix A.1.
Notably, Prompt-tuning134 uses 134 prompt lengths in Table 1, and it is set so to match the training parameters of the proposed method, M-IDPG-PHM.
4. Performance in high-resource scenario
Table 1 shows the results of all the methods on full datasets across 10 NLU tasks.
We observe that: (i) Our proposed method M-IDPG-PHM consistently outperforms the prompt tuning method and
Ptuning v2 by average 3.1pt and 1.6pt, respectively (except on the RTE dataset).
(ii)
Compared with other efficient transfer learning methods, IDPG performs slightly worse than the Compacter (Mahabadi et al., 2021) and Adapter (Houlsby et al., 2019), across the ten tasks.
However, the gap is mostly from RTE and QQP.
Note that IDPG uses 15K fewer parameters than the Compacter.
M-IDPG-PHM is better than Compacter on four tasks and has the same performance on three tasks.
(iii)
The improvement of our method is more prominent in the single-sentence classification task.
The four best results (MPQA, Subj, CR, MR) among all competing methods in single-sentence classification tasks are made by IDPG models.
Specifically,
M-IDPG-PHM performs 0.84pt and 0.36pt better than RoBERTa and EFL, respectively.
(iv) PHM- based generator performs on par with the DNNbased generator while having a significantly lower number of trainable parameters.
(v) GloVe-based sentence encoder also performs similar to LMbased sentence encoder, indicating the advancement of instance-dependent prompt generation does not rely on a robust contextual sentence encoder.
(vi) When we fix the training parameters to be the same, the comparison between Prompttuning-134 and M-IDPG-PHM illustrates that our approach works better than prompt tuning not just because of using more parameters.
4. Efficiency
Table 2 lists the number of trainable parameters for different methods excluding the classification head.
The general goal for efficient transfer learning is to train models with fewer parameters while achieving better performance.
Traditional prompt-tuning method only requires training a token embedding table with a few thousand parameters.
However, its performance is worse than a lightweight adapter model (e.g., Compacter with 149K parameters).
Our proposed method, especially the M-IDPGPHM, falls in the gap between prompt-tuning and adapter model, since it only requires training 134K parameters and performs on par with Compacter.
[Table]
4. Performance in low-resource scenario
We further evaluate our proposed method in the low-resource scenario.
Following the existing evaluation protocols in the few-shot setting (He et al., 2021), we sample a subset of the training data for each task with size K ∈ { 100 , 500 , 1000 } as our training data and another subset with sizeas a development set.
We compare our proposed methods with all prompt tuning methods, one finetuning model (EFL), and one adapter tuning model (Compacter).
In the extreme low-resource case when K =100,
M-IDPG-PHM performs 2.5pt better than the traditional prompt tuning method and 0.5pt better than the multi-layer P-Tuning v2 method.
This improvement illustrates that our method has better generalization in few-shot settings.
When K becomes larger, IDPG-PHM still maintains good results with 1.9pt, 0.2pt ( K =500) and 2.0pt, 0.2pt ( K =1000) better accuracy than traditional prompt tuning, P-tuning v2, respectively.
We also observe that when K is small, our method sometimes has a high variance (e.g., 4.6 on MPQA
when K = 100).
We suspect that this may be due to bad initialization that leads the model to non-optimal parameters.
We also notice that other state-of-the-art work (such as LM-BFF (Gao et al., 2021a)) attempt to address the few-shot learning problem from different perspectives.
We want to highlight that we are exploring a solution for the problem by training as few parameters as possible while maintaining good performance.
Testing the limitation of our model without freezing any parameters would be an interesting point but is not considered the main focus of this paper.
4. Intrinsic Study
We conduct several ablation studies including exploration of different generator architectures and impact of selecting different prompt positions.
4.5. Sentence Encoder: GloVe or LMs?
The proposed IDPG method relies on pre-trained
LM to extract sentence representation, i.e., [CLS] token embedding.
Obtaining contextualized transformer sentence embedding is often expensive if it is not pre-computed.
One open question is to explore reliability on lightweight sentence representations such as GloVe embedding (Pennington et al., 2014) or token embedding of pre-trained language models.
To answer this question, we apply the pre-trained
GloVe word vectors 2 to extract the sentence representation.
Specifically, we take the average of word vectors as the sentence embeddings:
[Equation]
[Table]
where x i is the input sequence with k tokens t 1 ,..., t k .
According to Table 1, using GloVe as sentence encoder to generate prompts doesn’t sacrifice much performance over the ten tasks and outperforms prompt tuning and P-tuning v2.
It indicates that our model does not benefit a lot from a strong contextual pre-trained LM.
Instead, a light sentence encoder such as GloVe can also help the tasks.
Also, instance-dependent prompt tuning shows promising improvement over non-instancedependent prompt tuning models.
Compared with
Compacter, one drawback of our method is that its FLOPS is roughly twice, even though it uses slightly fewer parameters.
Adopting GloVe as sentence encoder makes it unnecessary to go through
LM twice and thus effectively reduces IDPG’s
FLOPS by half.
4.5. Prompt Generator: PHM or DNN?
To reduce the tuning parameters, we substitute the
DNN layers with PHM layers.
An open question we seek to answer is what is the best generation model for prompt regardless of training parameters.
Hence, we compare the PHM-based prompt generator with the DNN-based prompt generator, as shown in Table 1.
We observe that including DNN as a generator doesn’t improve performance signif-
[Figure]
icantly, with +0.1pt gain on average, while adding 87K parameters (with hidden size m=16).
On the other hand, this ablation study further verifies PHM layers’ efficiency in the generation model.
4.5. Multi-layer Architecture Exploration
When applying the instance-dependent generation model G into a multi-layer case, the first challenge we face is the considerable increase in training parameters.
If each transformer layer requires an independent generator G i , the number of training parameters increases N times, where N is the number of transformer layers (24 in RoBERTa-Large).
Assuming G has the form
y = W x + b , there are three alternatives: (i)
Smallest version (S version): sharing both W and b ; (ii) Middle version (M version): sharing W and making b layer-specific; and (iii) Largest version (L version): making both W and b layer-specific.
Another way to reduce the training parameters is by adjusting the hidden size m of the generator.
We compare two models with m = 16 and m = 256.
Surprisingly, we find that generator with a hidden size 16 is not far from the large model (92.0 vs.
92.1, respectively, in M version).
We hypothesize that the smaller hidden size of 16 is already enough to store useful instance information, and setting m too large may be less efficient.
Besides, in single-layer prompt generation model, the input to G is M ( x i ) - the representation of input sequence x i .
In a multi-layer case, the input to each layer generator has another option, i.e., the previous layer’s output.
However, as shown in
Figure 3, the experiment results suggest no significant difference between the two input ways.
As for the generator selection, the three models perform as expected (S version <
M version < L version).
In
Table 1, M-IDPG-PHM uses the previous layer’s output as input, M version as the generator, andas the generator hidden size.
Detailed information for all models’ performance on each task can be found in Appendix A.3.
4.5. Prompt Insertion: Single-layer or Multi-layer?
P-tuning v2 (Liu et al., 2021a) conducted substantial ablation studies on the influence of inserting prompt into different transformer layers.
To boost single-layer IDPG performance, we add supplementary training (cf.
Appendix A.4)
and conduct ablation studies in Appendix A.5.
We come to a similar conclusion that multi-layer instancedependent prompt tuning model (M-IDPG) is significantly better than the single-layer method (SIDPG) in both evaluation settings.
An interesting finding is that the impact of supplementary training on S-IDPG is high while it is limited for M-IDPG.
4.5. How Prompts Help?
Given two sentences, we encode each of them by one of the comparison models and compute the cosine similarity.
We sort all sentence pairs in
STS-B dev set in descending order by the cosine
[Figure]
similarity scores and get a distribution for number of pairs in each group that is included in Top-k ranking.
We compare a vanilla model without any prompts with M-IDPG-PHM.
Both models are finetuned on STS-B training set.
As shown in Figure 4, prompts bring the similar sentences closer while pushing the dissimilar ones apart.
4.5. IDPG Scalability
We study our proposed model’s scalability in this section.
In general, the performance of IDPG in downstream tasks improves gradually when using a larger prompt length (Cf.
Appendix A.7).
Related Work
Supplementary Training:
Existing works (Phang et al., 2018; Liu et al., 2019) have observed that starting from the fine-tuned MNLI model results in a better performance than directly from the vanilla pre-trained models for RTE, STS, and MRPC tasks.
A series of work (SentenceBERT (Reimers and
Gurevych, 2019), BERT-flow (Li et al., 2020), SimCSE (Gao et al., 2021b))
explored intermediate training to improve STS tasks.
All of them applied pre-fine tuning on NLI datasets.
More recently,
EFL (Wang et al., 2021) proposed a task transformation paradigm, improving single sentence tasks with less labels using rich sentence-pair datasets.
Adapter Tuning: Adapter tuning has emerged as a novel parameter-efficient transfer learning paradigm (Houlsby et al., 2019; Pfeiffer et al., 2020), in which adapter layers – small bottleneck layers – are inserted and trained between frozen pre-trained transformer layers.
On the GLUE
benchmark, adapters attain within 0 .
4% of the performance of full fine-tuning by only training 3 . 6% parameters per task.
Compactor (Mahabadi et al., 2021) substitutes the down-projector and upprojector matrices by a sum of Kronecker products, reducing the parameters by a large margin while maintaining the overall performance.
Prompting: Hand-crafted prompts were shown to be helpful to adapt generation in GPT-3 (Brown et al., 2020).
Existing works including LMBFF (Gao et al., 2021a; Wang et al., 2021) explored the prompt searching in a few-shot setting.
Recently, several researchers have proposed continuous prompts training to overcome the challenges in discrete prompt searching.
Prefix tuning (Li and Liang, 2021) and P-tuningv2 (Liu et al., 2021a) prepend a sequence of trainable embeddings at each transformer layer and optimizes them.
Two contemporaneous works – prompt tuning (Lester et al., 2021) and P-tuning (Liu et al., 2021b), interleave the training parameters in the input embedding layer instead of each transformer layer.
All these methods focus on task-specific prompt optimization.
Our proposed method, IDPG, is the first prompt generator that is not only taskspecific but also instance-specific.
Conclusion and Discussion
We have introduced IDPG, an instance-dependent prompt generation model that generalizes better than the existing prompt tuning methods.
Our method first factors in an instance-dependent prompt, which is robust to data variance.
Parameterized Hypercomplex Multiplication (PHM) is applied to shrink the training parameters in our prompt generator, which helps us build an extreme lightweight generation model.
Despite adding fewer parameters than prompt tuning, IDPG shows consistent improvement.
It is also on par with the lightweight adapter tuning methods such as Compacter while using a similar amount of trainable parameters.
This work provided a new research angle for prompt-tuning of a pre-trained language model.
Acknowledgment
We thank Brian Lester, Jonas Pfeiffer, Tianyu Gao,
Qinyuan Ye, Zhenhao Zhang, Fei Sun, Umich
NLP4Health research group members, and all anonymous reviewers for helpful discussion and valuable feedback.
A Appendix A. Experimental Settings A.1. Training hyperparameters
We use RoBERTa-Large (Liu et al., 2019) model implemented by Fairseq (Ott et al., 2019) as our basic model.
The detailed model hyperparameters are listed in Table 4.
Note that for both transformer fine-tuning methods, including RoBERTa (Liu et al., 2019) and
EFL (Wang et al., 2021), we follow their official training instructions, i.e., using a polynomial learning rate scheduler with 6% steps to warm up and tuning for ten epochs.
For all adapter-based and prompt-based methods, we train them more sufficiently (with fifty epochs) on small datasets (i
[Table]
e.,
MPQA, Subj, CR, MR, RTE, MRPC, STS-B).
A.1. Model hyperparameters
We report the detailed model hyperparameters for each method in Table 1 and illustrate how numbers in Table 2 are computed.
Compacter: hidden size d = 1024, adapter hidden size m = 16, user defined n = 4, each transformer layer inserts 2 compacters.
Down-project s i matrix takes 1024 / 4 × 4 × 24 × 2 = 48 K , downproject t i matrix takes 16 / 4 × 4 × 24 × 2 = 0 .
75 K , hidden bias takes 16 × 24 × 2 = 0 .
75 K , up-project
s i and t i matrix takes the same number of parameters as down-projector, the output bias takes 1024 × 24 × 2 = 48 K , the shared matrix A i takes 4 3 × 24 × 2 = 3 K .
Total parameters : 48 + 0 .
75 + 0 .
75 + 48 + 0 .
75 + 48 + 3 = 149 .
25 K .
Adapter: hidden size d = 1024, adapter hidden size m = 16.
Total parameters : ( 1024 × 16 + 16 + 16 × 1024 + 1024 ) × 24 × 2 = 1 . 55 M .
Prompt-tuning: prompt length t = 5.
Total parameters : 5 × 1024 = 5 K .
Prompt-tuning-134: prompt length t = 134.
Total parameters : 134 × 1024 = 134 K .
P-tuning v2: prompt length t = 5, inserted layers 24.
Total parameters : 5 × 24 × 1024 = 120 K .
S-IDPG-PHM: hidden size d = 1024, generator hidden size m = 256, prompt length t = 5, user defined n =
(Cf. Equation 4).
First PHM layer
W 1 takes 1024 / 16 × 256 / 16 × 16 + 256 = 16 .
25 K parameters, second PHM layer W 2 takes 256 / 16 × 5 × 1024 / 16 × 16 + 5 × 1024 = 85 K parameters, the shared matrix A i takes 16 3 = 4 K
(Note we use one shared matrix in single version IDPG).
Total parameters : 105 K .
S-IDPG-DNN: hidden size d = 1024, generator hidden size m = 256, prompt length t = 5.
Total parameters : 1024 × 256 + 256 + 256 × 5 × 1024 + 5 × 1024 = 1 .
5 M .
M-IDPG-PHM-GloVe: input vector size 300, generator hidden size m = 16,
prompt length t = 5, user defined n = 4 (Cf.
Equation 4).
First PHM layer W 1 takes 300 / 4 × 16 / 4 × 4 + 16 = 1216 parameters, second PHM layer W 2 takes 16 / 4 × 5 × 1024 / 4 × 4 + 5 × 1024 ×
24 = 140 K parameters, the shared matrix A i takes 4 3 × 2 = 128.
Total
[Table] [Table]
83.5 ± 0 .92.3 ± 0 .91.4 ± 0 .87.1 ± 0 .92.
parameters : 141 K .
M-IDPG-PHM: hidden size d = 1024, generator hidden size m = 16, prompt length t
= 5, user defined n =
(Cf. Equation 4).
First PHM layer
W 1 takes 1024 / 16 × 16 / 16 × 16 + 16 = 1 K parameters, second PHM layer W 2 takes 16 / 16 × 5 × 1024 / 16 × 16 + 5 × 1024 × 24 = 125 K parameters, the shared matrix A i takes 16 3 16 × 2 = 8 K .
Total parameters : 134 K .
M-IDPG-DNN: hidden size d = 1024, generator hidden size m = 16, prompt length t = 5.
Total parameters : 1024 × 16 + 16 + 16 × 5 × 1024 + 5 × 1024 × 24 = 216 K .
A. Datasets
We provide a detailed information in Table 5 for
NLU datasets we used.
A. Detailed results for Multi-layer Architecture Exploration
We provide a detailed result table for all compared methods in Section 4.5.3.
Note that the M version model with m = 16 and previous layer as input one is slightly higher than the results shown in Table 1(Cf.
M-IDPG-PHM)
, this is because we tune the learning rate more carefully in Table 6 ( lr ∈ { 1 e − 2 , 7 e − 3 , 5 e − 3 , 3 e − 3 , 1 e − 3 , 7 e − 4 , 5 e − 4 , 3 e − 4 , 1 e − 4 } ) to seek the best performance each model can reach.
While in Table 1, we tune the learning rate from { 5 e − 3 , 1 e − 3 , 5 e − 4 , 1 e − 4 } to make the fair comparison with other models.
A. Supplementary Training for Single-layer IDPG
According to previous works (Phang et al., 2018;
Wang et al., 2021), supplementing pre-trained LMs with rich data helps tasks with limited labels and stabilizes downstream fine-tuning.
Following this idea, we also conduct intermediate training for single-layer IDPG.
However, a drawback of supplementary training is that if the data distribution of the downstream tasks is quite different from the supplementary training task, i.e., MRPC vs. MNLI (Wang et al., 2019), it may harm the downstream performance.
Figure 5 provides a comprehensive statistic among all sentence pair tasks in GLUE benchmark.
For example, the length of the first sentence in MNLI is 9 .
8 longer than the second sentence on average, while this length difference in MRPC is only 0 .
One natural solution to smooth the length distribution difference between tasks is to insert prompt in both supplementary training and downstream finetuning stage.
For example, assuming that we are adding a prompt with a length t = 5 after the second sentence in the supplementary training stage on
MNLI.
Then, when fine-tuning downstream tasks such as MRPC, we concatenate the prompt after the first sentence.
In this way, the length difference in MNLI and MRPC becomes more balanced: 4 . 8 vs. 0 .
6 + 5 = 5 . 6.
As shown in Figure 6, we test five different insertion positions (Pos 0–4) for sentence pair tasks and three different positions (Pos 0, 1, 4) for single sentence tasks.
We further reduce the distribution difference by reconstructing the supplementary training data.
We double the MNLI dataset by reordering the two sentences on one shard, and use the doubled dataset during intermediate training.
[Figure]
Architecture
Avg
Voting
PHM 86.
+residual 85.
+LayerNorm 86.
+residual+LayerNorm 77.
A.
Ablation study for single-layer IDPG A.5. Generator Architecture Exploration
We explore three different architectures for the proposed PHM-based generator: (i) Residual: a residual structure (He et al., 2016) is applied to add the sentence representation to each generated tokens; (ii) LayerNorm: layer normalization (Ba et al., 2016) is also added to normalize the generated token embedding; (iii) residual + layerNorm: a mixed model that uses both the residual component and LayerNorm.
Note that, to balance the token embedding and sentence embedding, we apply
LayerNorm to each embedding first, then after the add-up, use LayerNorm again to control the generated tokens.
We observe that adding LayerNorm slightly improves the voting results, while residual performs slightly worse.
One surprising result is that the mixed model of Residual and LayerNorm has significantly poorer performance.
A.5. Prompt Position
As we discussed in Section A.4, the prompt position has a direct impact on the prediction results.
We conduct a comprehensive study of the prompt position for our proposed method in both supplementary training and downstream fine-tuning phases.
Looking at the prompt position in downstream tasks first, Figure 7(a) shows that for both standard prompt tuning and our proposed method, the best position is 0 for single-sentence tasks andfor sentence-pair tasks.
This result is intuitive for single-sentence tasks since prompt in positioncan be regarded as the premise and original input sentence as the hypothesis.
For sentence-pair tasks, we hypothesize that inserting prompt into position 1 can better align the two input sentences.
Figure 7(b) illustrates the effect of prompt position on the supplementary training phase.
It is interesting that IDPG achieves best results in positionwhile the standard prompt-tuning achieves the best results in position 4 for both single-sentence and [Figure] i [Figure]
i
[Figure]
i
[Figure]
i
[Figure]
i
[Figure]
sentence-pair tasks.
A. Cosine Similarity Distributions in STS-B
We present the cosine similarity distributions when k = 100 and k = 300 in Figure 8a and in Figure 8b, respectively.
A. Ablation Study on Prompt Length
We present the impact of prompt length among several prompt tuning methods in Figure 9.
IDPG shows its stability when scaling to larger models with longer prompts.
A. Potential Risks
Our proposed model IDPG is a novel efficient transfer learning method.
It tunes small portion parameters while directly employs backbone model parameters without any changing.
However, if the backbone model stored online is attacked, whether
IDPG could still work well remains unknown.
One should be careful to apply our proposed model and all other prompt tuning methods in high-stakes areas without a comprehensive test.
[Figure]