IDPG : An Instance-Dependent Prompt Generation Method (2022)

Abstract: Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. It freezes the pre-trained language model and only optimizes a few task-specific prompts. In this paper, we propose a conditional prompt generation method to generate prompts for each input instance, referred to as the Instance-Dependent Prompt Generation (IDPG). Unlike traditional prompt tuning methods that use a fixed prompt, IDPG introduces a lightweight and trainable component to generate prompts based on each input sentence. Extensive experiments on ten natural language understanding (NLU) tasks show that the proposed strategy consistently outperforms various prompt tuning baselines and is on par with other efficient transfer learning methods such as Compacter while tuning far fewer model parameters.

## introduction in recent years, pre-training a transformer model on a large corpus with language modeling tasks and fine-tuning it on different downstream tasks has become the primary transfer learning paradigm in natural language processing @xcite . notably, this paradigm requires updating and storing all the model parameters for each downstream task. as the model size proliferates (e.g., 330m parameters for bert @xcite and 175b for gpt-3 (brown et al., 2020)), it becomes computationally expensive and challenging to fine-tune the entire pre-trained language model (lm). thus, it is natural to ask whether we can transfer the knowledge of a pre-trained lm to downstream tasks by keeping most of the parameters fixed and tuning only a small fraction of them. previous studies have attempted to address this question from different perspectives. one line of research @xcite suggests augmenting the model with smaller, trainable modules and freezing the original transformer weights. adapters @xcite @xcite , for example, insert a small set of additional modules between each transformer layer. only these additional and task-specific modules are trained during fine-tuning, reducing the number of trainable parameters to ∼ 1-3% of the original transformer model per task. compacter @xcite optimizes the training parameters further by designing a lightweight module to replace the bottleneck architecture in adapters. another line of work focuses on prompting. the gpt-3 models @xcite find that, with proper manual prompts, a pre-trained lm can successfully match the fine-tuning performance of bert models. lm-bff @xcite , efl @xcite , and autoprompt @xcite extend this direction by inserting prompts in the input embedding layer. however, these methods rely on grid-search for a natural language-based prompt from an ample search space, leading to optimization challenges. to tackle this issue, prompt tuning @xcite , prefix tuning @xcite , and ptuning @xcite approaches propose to prepend trainable prefix tokens to the input layer and train these soft prompts only during the finetuning stage. in doing so, the problem of searching discrete prompts is converted to a continuous optimization task, which can be solved by a variety of optimization techniques such as sgd. this significantly reduced the number of trainable parameters to just a few thousand. however, all existing prompt-tuning methods have thus far focused on task-specific prompts, which are inadequate to address the gap between pre-training and fine-tuning objectives. specifically, it is unlikely to see many different sentences with the same prefix in the pretraining corpus. thus, a unified prompt may disturb the prediction and lead to a performance drop. in light of these limitations, we instead ask the following question: can we generate input-dependent prompts to smooth the domain difference? this paper presents the instance-dependent prompt generation (idpg) strategy for efficiently tuning large-scale lms. unlike traditional prompttuning methods that rely on a fixed prompt for each task, idpg instead develops a conditional prompt generation model to generate prompts for each instance. formally, the idpg generator can be denoted as f (x; w), where x is the instance representation and w represents the trainable parameters. note that by setting w to a zero matrix and only training the bias, idpg would degenerate into the traditional prompt tuning process @xcite . to further reduce the number of parameters in the generator f (x; w), we propose to apply a lightweight bottleneck architecture (i.e., a two-layer perceptron) and then decompose it by a parameterized hypercomplex multiplication (phm) layer @xcite . to summarize, this work makes the following contributions: ## instance-dependent prompt generation (idpg) we now introduce our proposed method, idpg, along with various model optimizations. the main procedure is illustrated in figure 2 . ## conclusion and discussion we have introduced idpg, an instance-dependent prompt generation model that generalizes better than the existing prompt tuning methods. our method first factors in an instance-dependent prompt, which is robust to data variance. parameterized hypercomplex multiplication (phm) is applied to shrink the training parameters in our prompt generator, which helps us build an extreme lightweight generation model. despite adding fewer parameters than prompt tuning, idpg shows consistent improvement. it is also on par with the lightweight adapter tuning methods such as compacter while using a similar amount of trainable parameters. this work provided a new research angle for prompt-tuning of a pre-trained language model.