Learning How to Ask: Querying LM s with Mixtures of Soft Prompts (2021)

Abstract: Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks, using a fill-in-the-blank paradigm (Petroni et al., 2019) or a few-shot extrapolation paradigm (Brown et al., 2020). For example, language models retain factual knowledge from their training corpora that can be extracted by asking them to “fill in the blank” in a sentential prompt. However, where does this prompt come from? We explore the idea of learning prompts by gradient descent—either fine-tuning prompts taken from previous work, or starting from random initialization. Our prompts consist of “soft words,” i.e., continuous vectors that are not necessarily word type embeddings from the language model. Furthermore, for each task, we optimize a mixture of prompts, learning which prompts are most effective and how to ensemble them. Across multiple English LMs and tasks, our approach hugely outperforms previous methods, showing that the implicit factual knowledge in language models was previously underestimated. Moreover, this knowledge is cheap to elicit: random initialization is nearly as good as informed initialization.

## introduction pretrained language models, such as elmo @xcite , bert @xcite , and bart @xcite , have proved to provide useful representations for other nlp tasks. recently, @xcite and @xcite demonstrated that language models (lms) also contain factual and commonsense knowledge that can be elicited with a prompt. for example, to query the date-of-birth of mozart, we can use the prompt "mozart mozart mozart mozart mozart mozart mozart mozart mozart mozart mozart mozart mozart mozart mozart mozart mozart was born in ," where we have filled the first blank with "mozart," and ask a cloze language model to fill in the second blank. the prompts used by @xcite are manually created, while @xcite use mining and paraphrasing based methods to automatically augment the prompt sets. finding out what young children know is difficult because they can be very sensitive to the form of the question @xcite . opinion polling is also sensitive to question design @xcite . we observe that when we are querying an lm rather than a human, we have the opportunity to tune prompts using gradient descent-the workhorse of modern nlp-so that they better elicit the desired type of knowledge. a neural lm sees the prompt as a sequence of continuous word vectors @xcite . we tune in this continuous space, relaxing the constraint that the vectors be the embeddings of actual english words. allowing "soft prompts" consisting of "soft words" is not only convenient for optimization, but is also more expressive. soft prompts can emphasize particular words (by lengthening their vectors) or particular dimensions of those words. they can also adjust words that are misleading, ambiguous, or overly specific. consider the following prompt for the relation date-of-death: this prompt may work for the male singer cab calloway, but if we want it to also work for the female painter mary cassatt, it might help to soften "performed" and "his" so that they do not insist on the wrong occupation and gender, and perhaps to soften "until" into a weaker connective (as cassatt was in fact too blind to paint in her final years). another way to bridge between these cases is to have one prompt using "performed" and another using "painted." in general, there may be many varied lexical patterns that signal a particular relation, and having more patterns will get better coverage @xcite . we therefore propose to learn a mixture of soft prompts. we test the idea on several cloze language models, training prompts to complete factual and com-mon sense relations from 3 datasets. comparing on held-out examples, our method dramatically outperforms previous work, even when initialized randomly. so when regarded as approximate knowledge bases, language models know more than we realized. we just had to find the right ways to ask. ## related work factual knowledge is traditionally extracted from large corpora using a pipeline of nlp tools @xcite , including entity extraction @xcite , entity linking @xcite and relation extraction @xcite . however, recent work has shown that simply training a system to complete sentences-language modeling-causes it to implicitly acquire nonlinguistic abilities from its training corpora @xcite , including factual knowledge @xcite , common sense @xcite , reasoning @xcite , summarization @xcite , and even arithmetic @xcite . most of the previous work manually creates prompts to extract answers from the trained language model. we use lama @xcite as a baseline. building on lama, the lm prompt and query archive (lpaqa) method @xcite searches for new prompts by either mining a corpus or paraphrasing existing prompts. autoprompt @xcite searches for improved prompts using a gradient signal, although its prompts are limited to sequences of actual ("hard") english words, unlike our method. we compare our novel soft prompts against all of these systems. after we submitted the present paper in november 2020, two still unpublished manuscripts appeared on arxiv that also investigated soft prompts. li and liang (2021) considered the setting of generating text from a pretrained language model (gpt-2 or bart) conditioned on a textual prompt. to improve the results, they prepended a few taskspecific "soft tokens" to the prompt and tuned the embeddings of only these tokens (at all embedding layers). @xcite adopted a strategy similar to ours by tuning fill-in-the-blank prompts in a continuous space, testing on gpt-2 and bert models, although they did not use the enhancements we proposed in § §3.2-3.4 below. like our work, both these papers achieved strong gains. in other work, @xcite mine prompts from a corpus, then fine-tune the whole language model so that it more accurately completes the prompts. @xcite are similar but fine-tune the language model differently for each prompt. our method complements these by tuning the prompts themselves. "probing" systems that ask what language models know about particular sentences (e.g., @xcite usually use feedforward networks rather than further natural-language prompts. yet @xcite show how to use naturallanguage prompts to ask about particular sentences. our method could potentially be applied to those prompts, or to "few-shot learning" prompts that include input-output examples @xcite . ## method our experiments will specifically aim at extracting relational knowledge from language models. we are given a fixed pretrained lm, a specific binary relation r such as date-of-death, and a training dataset e r consisting of known (x, y) pairs in r, such as @xcite . we will then train a system to predict y from x, and evaluate it on held-out (x, y) pairs of the same relation. a prompt t is a sentence or phrase that includes two blanks, as illustrated in §1. to pose the query, we fill the x blank with x: mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt mary cassatt performed until his death in y . we can ask the lm for its probability distribution p lm (y | t, x) over single words that can now fill y . the correct answer would be 1926. ## conclusion well-crafted natural language prompts are a powerful way to extract information from pretrained language models. in the case of cloze prompts used to query bert and bart models for single-word answers, we have demonstrated startlingly large and consistent improvements from rapidly learning prompts that work-even though the resulting "soft prompts" are no longer natural language. our code and data are available at https:// github.com/hiaoxui/soft-prompts . how about few-shot prediction with pretrained generative lms? here, @xcite show how to assemble a natural language prompt for input x from relevant input-output pairs (x i , y i ) selected by a trained retrieval model. allowing fine-tuned soft string pairs is an intriguing future possibility for improving such methods without needing to fine-tune the entire language model.