D yna M a R : Dynamic Prompt with Mask Token Representation (2022)

Abstract: Recent research has shown that large language models pretrained using unsupervised approaches can achieve significant performance improvement on many downstream tasks. Typically when adapting these language models to downstream tasks, like a classification or regression task, we employ a fine-tuning paradigm in which the sentence representation from the language model is input to a task-specific head; the model is then fine-tuned end-to-end. However, with the emergence of models like GPT-3, prompt-based fine-tuning has been proven to be a successful approach for few-shot tasks. Inspired by this work, we study discrete prompt technologies in practice. There are two issues that arise with the standard prompt approach. First, it can overfit on the prompt template. Second, it requires manual effort to formulate the downstream task as a language model problem. In this paper, we propose an improvement to prompt-based fine-tuning that addresses these two issues. We refer to our approach as DynaMaR – Dynamic Prompt with Mask Token Representation. Results show that DynaMaR can achieve an average improvement of 10% in few-shot settings and improvement of 3.7% in data-rich settings over the standard fine-tuning approach on four e-commerce applications.

## introduction unsupervised pre-trained language models (lms) such as bert @xcite and roberta @xcite have achieved state-of-the-art performance on many natural language understanding tasks. in general, these models are fine-tuned for different tasks through the addition of a taskspecific head on top of the [cls] token representation @xcite . an alternative method to applying lms on downstream tasks is through discrete prompts. a discrete prompt is an additional text phrase inserted along with the original input text that encapsulates the task of interest. by adding the prompt, we convert the downstream task into a masked language (mlm) problem. for example, to classify the sentiment of a movie review, "i hate this movie.", we can append a prompt to the input to get "i hate this movie. it was [mask]". the pre-trained language model is thus prompted to identify the sentiment of the input statement and classify the [mask] token as "terrible" instead of "great" @xcite . in this paper, we call a function that includes a prompt and its position information a prompt template. prompt-based approaches have shown success in low-data regimes @xcite @xcite @xcite . prompt-based fine-tuning is beneficial in few-shot learning, because it provides extra task information to the model through the prompt text @xcite . however, when we explore this technique in practice, two issues have arisen. first, the trained model can overfit on words or phrases within the prompt and on the position of the @xcite token in the prompt @xcite . for example, in movie review sentiment analysis, when we append the prompt, "does the user like the movie? [mask]", to a negative review, "this is a bad movie.", the trained model is inclined to predict the positive class, because the word "like" frequently appears in positive reviews and the masked language model has greater attention on the words/phrases that are closer to the mask token as shown in figure 1 . we call this issue prompt-related overfitting in this work. we tackle prompt-related overfitting by introducing a dynamic prompt approach. in this approach, we create a prompt pool consisting of multiple prompt templates. to construct this pool, we generate a set of prompt candidates and filter by a similarity score we propose, called the pairwise prompt dissimilarity score (detailed in section 3). we then introduce the dynamic component of the algorithm by randomly selecting a prompt template from the pool and applying to the input for each training step. for example, in the movie review sentiment analysis task, the trained model will randomly see either "does the user like the movie? [mask]" or "does the user dislike the movie? [mask]" appended to the original input. this prevents the model to overfit on spurious correlations between words in the prompt and the class label. in addition, as previously mentioned, the standard prompt-based fine-tuning setup can be inefficient. it requires significant input and answer engineering to reformulate the downstream tasks as mlm problems @xcite . this process is time-consuming especially for tasks with large numbers of classes. besides, another disadvantage of the standard setup is that it cannot be directly applied to regression problems, as they cannot be easily converted to mlm problems. to simplify this process, we fine-tune the model by feeding the mask token representation into a task-specific classifier/predictor head instead of the pre-trained mlm head to avoid the answer engineering process, as shown in figure 2 . we refer to our promptbased approach with these two improvements as dynamic prompt with mask token representation (dynamar). we apply dynamar to both few-shot and data-rich settings and, for the first time, show improvement gains across four tasks not only in few-shot settings but also in data-rich settings. our contributions include: (1) proposing dyna-mar, which can be applied without reformulating downstream tasks into language problems and is robust to prompt-related overfitting, (2) showing dynamar can achieve improvements in both fewshot and data-rich settings, (3) proposing a prompt dissimilarity score to evaluate the degree of dissimilarity between two prompt templates and to help construct a diverse dynamic prompt pool, (4) demonstrating that a larger dynamic prompt pool achieves better performance on downstream tasks. ## related work our work can be divided into three components: language model fine-tuning, prompt generation, and the design of the prompt template. language model fine-tuning is the main focus of our work. recently, a large amount of research has focused on improved language model finetuning methods @xcite @xcite . these works mainly focus on optimization and regularization techniques to stabilize finetuning. in contrast to these works, @xcite describe the concept of prompt-based fine-tuning for language models. we adapt and simplify the core ideas from this work to create a simple yet efficient prompt-based fine-tuning approach. prompt generation is a key process in promptbased fine-tuning. the choice of prompt significantly influences performance. the most natural way to generate prompts is through manual design. @xcite employ manually generated prompts with elmo @xcite and bert @xcite ) models. they evaluate on the lama (language model analysis) benchmark @xcite without fine-tuning and conclude that the model is able to recall knowledge learned from the pre-training tasks. while manually crafting prompts is intuitive, creating effective prompts through manual effort requires time, experience, and expertise. to address this issue, a number of automatic prompt searching methods have been proposed. for example, @xcite propose a data mining-based method that searches for a prompt based on the shortest path between the original inputs and answers. they also propose paraphrasing-based methods that take a seed prompt and paraphrase it into several semantically similar expressions. @xcite treat prompt generation as a text generation task and utilize t5, a sequence-to-sequence pretrained model, in the template search process. they generate templates by specifying the position to insert a prompt template and then inputting samples into t5 to decode the templates. these automatic approaches achieve comparable performance to manually designed prompts. besides, @xcite propose the null prompt method. instead of generating prompts, they concatenate a [mask] token with original inputs and it performs competitively to manually designed prompts. in our experiments, we utilize the prompt generation methods to create candidates for the dynamic prompt pool, while also including the null prompt approach as one of the baselines. prompt template design factors are the factors that we take into consideration to create a metric that informs how prompts are selected for the dynamic prompt pool. numerous previous works analyze prompt template design factors and the impact of prompt design on performance. @xcite summarize the factors that influence the application of prompt-related technologies in language models. logan iv et al. (2021) note that the order in which the original input and the [mask] token are concatenated is an important consideration. @xcite propose to unify the prompts into a question-answering format. these previous works indicate that prompt construction impacts performance. to this end, we hypothesize that diversity in the set of prompt templates is an important factor in the performance of the model and propose a prompt dissimilarity score for measuring diversity. ## our method: dynamar in this section, we describe details of our approach, dynamar. before explaining the training process, we define two concepts: the dynamic prompt pool and the inference prompt. dynamic prompt pool is a pool of prompt templates from which a prompt template will be randomly selected and applied to the input during training. inference prompt is the prompt template used during inference. it is selected from the set of templates in the dynamic prompt pool. in general, it is the prompt template among those in the dynamic prompt pool that can achieve the highest performance in a fixed prompt setting. we generate the candidates for the dynamic prompt pool and inference prompt through manual generation and paraphrasing-based methods proposed by @xcite . however, we do not include all candidates in the dynamic prompt pool. we want to ensure the prompts within a pool are sufficiently diverse so that the model will not overfit on any of them. therefore, we introduce a prompt dissimilarity score to measure the level of dissimilarity between these candidates. we consider three factors in developing this metric: (1) prompt position, or whether to append or prepend the prompt to the input or even insert into the middle of pairwise inputs, (2) prompt wording or the prompt word selection, and (3) prompt format, or whether to create prompts in statement format or in the question-answering format proposed by @xcite . to define the prompt dissimilarity score, we first introduce the normalized hamming distance and the normalized levenshtein distance. normalized hamming distance is equal to the number of different bits between two binary representations divided by the length of the binary representations @xcite . let hd(b i , b j ) be the hamming distance between binary representations b i and b j with equal length k. the equation of normalized hamming distance n hd(b i , b j ) then follows: normalized levenshtein distance is equal to the minimum number of operations (substitution, insertion and deletion) required to transform a given string into another string divided by the length of the longer string and is calculated in a recursive fashion @xcite . let ld(s i , s j ) be the levenshtein distance between string s i and s j . let |s i | and |s j | be the length of prompt string s i and s j , respectively. let t(x) be a function that keeps a string of all but the first character of x. the equation of the normalized levenshtein distance n ld(s i , s j ) follows: suppose we generate n prompt templates. let p i and p j be two prompt templates with s i , s j as prompt strings, respectively, where @xmath0, @xmath1, 2, . . . , n }. we treat the prompt position and format information as categorical variables and convert them into binary representations, b i , b j . let p ds(p i , p j ) denote the prompt dissimilarity score between prompt templates p i and p j . the prompt dissimilarity score equation can be found below: ## conclusion in this work, we discuss methods for generating prompts and propose a way to select prompt templates to include in the dynamic prompt pool. also, we show that using the mask representation of a prompt either equals or improves upon the performance of standard fine-tuning on four e-commerce applications in both few-shot and data-rich settings. in addition, we find dynamar outperforms the fixed prompt approach in both settings. furthermore, we show that a larger dynamic prompt pool leads to improved model performance when employing dynamar. ruiqi zhong, kristy lee, zheng zhang, and dan klein.