P rompt G en: Automatically Generate Prompts using Generative Models (2022)

Abstract: Recently, prompt learning has received significant attention, where the downstream tasks are reformulated to the mask-filling task with the help of a textual prompt. The key point of prompt learning is finding the most appropriate prompt. This paper proposes a novel model PromptGen, which can automatically generate prompts conditional on the input sentence. PromptGen is the first work considering dynamic prompt generation for knowledge probing, based on a pre-trained generative model. To mitigate any label information leaking from the pre-trained generative model, when given a generated prompt, we replace the query input with “None”. We pursue that this perturbed context-free prompt cannot trigger the correct label. We evaluate our model on the knowledge probing LAMA benchmark, and show that PromptGen significantly outperforms other baselines.

## introduction prompt learning @xcite is a new learning paradigm for utilizing pretrained language models (lm), where downstream tasks are reformulated as a mask filling task with the help of a textual prompt in the original pretrained lm. recently, prompt learning has been used in applications such as knowledge probing @xcite @xcite ), text classification @xcite @xcite , natural language inference @xcite . furthermore, prompt learning has shown its utility in solving few-shot learning problems @xcite . the essence of prompt learning is designing the most appropriate prompts to trigger the correct target text for downstream tasks from an lm. the latest methods to construct prompts include: i) handwritten prompts @xcite , where users manually create intuitive templates based on human introspection, and ii) automatically searched prompts @xcite @xcite , where researchers search over the space of input tokens or embeddings for prompts that elicit correct predictions in the dev set. although manually written prompts are interpretable, they are limited by the manual effort, and might not be optimal for eliciting correct predictions. the automated approaches @xcite @xcite can overcome the limitations of manual prompts by training a model, but they learn a universal prompt for each task (e.g., factual probing for one relation), regardless of different inputs. but such a setting may result in sub-optimal prompts. for example in factual probing, different subjects might have a different context when describing the same relation in an open-domain corpus. similarly, for sentiment analysis, different query sentences might have different syntax or semantics. we hypothesize that learning different prompts conditioned on inputs can benefit the overall masked filling accuracy in prompt learning. towards that end, we propose a dynamic prompt generation model, named as promptgen, to automatically generate prompts based on inputs by leveraging the pre-trained generative model bart @xcite . generally, promptgen consists of an encoder and an autoregressive decoder based on transformer @xcite . we show the overall architecture of promptgen applied on factual probing task in figure 1 . a knowledge fact is defined as a triplet: <sub, rel, obj>. the encoder produces a latent representation from @xmath0, rel>, and the decoder autoregressively generates prompt in the form of @xcite [d 1 ]...[mask]..., [d m+n ]. generated prompts are then passed to a fixed pretrained lm (e.g., bert) to @xmath1. a crossentropy loss will be calculated based on the pre-encoder [sub] was born in [mask]. autoregressive @xmath2 ## related work factual probing the factual probing setting was introduced by the lama benchmark @xcite @xcite , where given subject and relation, we want to infer the object by querying a pre-trained mlm. in contrast to previous knowledge graph completion models @xcite @xcite @xcite and information extraction models @xcite , where they need to fine-tune a pre-trained mlm. here, we convert the knowledge graph completion task into a mask filling task, without mlm fine-tuning. pre-trained generative models. our work is based on generative models, hence recent pretrained generative models are related, including @xcite , bart @xcite and t5 @xcite , all of which are capable of filling in missing spans in the input. among all prompt search methods, @xcite is the most similar to ours since they used t5 to construct prompts. compared with our work, @xcite uses t5 without fine-tuning, and they learn one prompt for all inputs. in our work, we learn dynamic prompts conditional on the given input and fine-tune on the generative model. instance-level prompt learning. concurrently, couple instance-level prompt learning methods are developed, where given different query input, they utilize different prompts. @xcite learns instance-level prompts through calculating the relevance scores between token embedding in a universal prompt and token embedding in a given query, then the relevance scores are used to map the universal prompt into an instance-level prompt. idpg @xcite leans a light-weight generator to generate prompts, which are similar to our prompt-gen. however, for downstream tasks, idpg extracts the representation of @xcite token to make the final predictions. so, idpg has to fine-tune the pre-trained mlm, while we keep the downstream mlm frozen. ## methodology we elaborate our method on the application of the lama task, in which the downstream mlm is bert @xcite . our generative model adopts pre-trained bart @xcite . given a subject s, relation r, a generated prompt @xmath3,@xmath4, and an mlm, we can identify the word ô ∈ v to which the mlm assigns the highest probability of p ([mask] = ô|@xmath5,@xmath6, where @xmath7,@xmath8; v represents the predefined vocabulary. if the mlm can fill in the mask with the correct object, we conclude that the mlm encodes information about the fact. in this work, we will fine-tune bart using our novel approach to generate the optimal prompts. ## conclusion in this work, we propose promptgen for knowledge probing, which can automatically generate prompts conditional on the given query (i.e., subject, relation). our promptgen leverages a pre-trained generative model, e.g., bart. promptgen is end-to-end trainable, where we fine-tune the parameters of the generative model, while keeping the downstream pre-trained mlm frozen. we evaluate promptgen on the benchmark lama dataset. we observe the significant improvement of the performance on the down-stream mlm by finding more appropriate dynamic prompts without label information leaking.