Few-Shot Text Generation with Natural Language Instructions (2021)

Abstract: Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion. Moreover, when combined with regular learning from examples, this idea yields impressive few-shot results for a wide range of text classification tasks. It is also a promising direction to improve data efficiency in generative settings, but there are several challenges to using a combination of task descriptions and example-based learning for text generation. In particular, it is crucial to find task descriptions that are easy to understand for the pretrained model and to ensure that it actually makes good use of them; furthermore, effective measures against overfitting have to be implemented. In this paper, we show how these challenges can be tackled: We introduce GenPET, a method for text generation that is based on pattern-exploiting training, a recent approach for combining textual instructions with supervised learning that only works for classification tasks. On several summarization and headline generation datasets, GenPET gives consistent improvements over strong baselines in few-shot settings.

## introduction pretraining large neural networks with a language modeling objective has led to significant improvements throughout nlp @xcite @xcite @xcite . further improvements are often possible by choosing a different pretraining objective that more closely matches the downstream task of interest. examples include casing prediction for named entity recognition @xcite , gap sentence generation for summarization (zhang et al., 1 our implementation of genpet and code to recreate our few-shot training datasets is publicly available at https: //github.com/timoschick/pet . ## related work masked language modeling was proposed as a pretraining objective by @xcite . several variants of this objective that involve generating sequences of text have been proposed, including t5 @xcite , bart @xcite and pegasus @xcite , of which we make use in this work. the idea to rephrase tasks as cloze questions is commonly used to probe the knowledge contained within masked language models (e.g., @xcite @xcite @xcite @xcite . @xcite propose pet, which combines this idea with gradient-based learning for efficient few-shot text classification. jiang et al. (2020) and schick et al. (2020) consider the problem of finding the best way to rephrase a given task as a cloze question. @xcite 's version of pet can generate multiple tokens, but still requires a text classification objective and does not scale to long output sequences. @xcite consider task descriptions for text generation tasks, but do so only in a zero-shot setting. in a similar spirit, @xcite investigate the ability of pretrained language models to leverage task descriptions and examples without any gradientbased optimization. other approaches to few-shot learning in nlp commonly require large sets of examples from related tasks @xcite @xcite , parallel data for consistency training @xcite , or highly specialized methods tailored towards a specific task @xcite ). in contrast, genpet requires no additional labeled data and provides an intuitive interface to leveraging task-specific human knowledge. our work is also related to prefix-constrained decoding in interactive machine translation for making suggestions on how to complete a partial translation @xcite . keskar et al. (2019) and he et al. (2020) similarly use prompts and keywords for controllable text generation, but require specific pretraining procedures and do so only in high-resource settings. ## pegasus pretraining we briefly summarize the pretraining procedure of pegasus @xcite , the model to which we apply genpet. pegasus is a standard transformer encoder-decoder architecture @xcite that is pretrained using gapsentence generation, an objective tailored to text summarization tasks. this pretraining objective requires a set of documents consisting of multi- p (x) american duo wins opening beach volleyball match x news: __ 2 1 3 y world business sports v(y) p(y | x) ∝ pm (v(y) | p (x)) figure 2: application of a pattern-verbalizer pair (p, v) in pet: the input x is converted into a cloze question p (x). the probability p(y | x) of each label y is derived from the probability that a pretrained model m assigns to its verbalization v(y) at the masked position. figure adapted from schick et al. (2020). ple sentences. the key idea is to preprocess each document by (i) picking a subset of m informative sentences, foot_0 (ii) replacing each of these sentences by a mask token, and (iii) concatenating all removed sentences into a pseudo-summary. the transformer model is then trained to generate this pseudo-summary given the partially masked document. similar to prior work (e.g., raffel et al., 2020; lewis et al., 2020), this is done by having the encoder process the entire masked document and the decoder generate the output autoregressively. zhang et al. (2020) train two variants of pe-gasus: pegasus-base, a 12-layer model with approximately 223m parameters, and pegasuslarge, a 16-layer model with 568m parameters. as only the latter version is publicly available in a variant that is not finetuned on any downstream task, all our experiments are based on pegasus-large. ## pattern-exploiting training pattern-exploiting training (pet, @xcite ) is a finetuning method for text classification tasks. that is, pet can be applied to problems where a text sequence @xmath0. as shown in figure 2 , pet enables data-efficient text classification by converting inputs into cloze questions; this drastically reduces the number of examples required @xcite . let m be a masked language model, v its vocabulary of tokens and __ ∈ v the mask token; we denote the set of all token sequences as v * . given an input sequence @xmath1, let p m (t | z) denote the probability assigned to @xmath2. as illustrated in figure 2 , pet requires: the probability of y given x is then derived from the probability that m assigns to v(y) at the masked position in p (x): for finetuning, the cross-entropy between p(y | x) and the true label of x is used as training objective. ## generation with instructions we now introduce genpet, our method for finetuning language models with instructions for text generation. similar to pet, we provide instructions by means of patterns p : @xmath3. however, we do not require a verbalizer as our output space already consists of natural language sentences, i.e., y ⊆ v * . in designing genpet, we tackle three key challenges for few-shot text generation with instructions: we consider an encoder-decoder model m pretrained by masked language modeling. that is, the model must be able to compute a probability p m (y | z) that measures to what extent y is a plausible substitute for the mask in z. we further require that this is done by decomposing the joint probability of y as follows: 3 if we happen to already know some prefix y 1:k-1 of y, we denote with ## experiments tasks we evaluate pegasus with and without genpet on a subset of the tasks in @xcite . as our computing resources are limited, we only choose those tasks for which the maximum output length in @xcite is at most 128 tokens. we include the following tasks: for each task, we use the entire test set for evaluation. 4 we create two types of training sets containing either 10 or 100 training examples; in addition, we provide 1,000 unlabeled examples per task decoder prefixes aeslc @xmath4: @xmath5: gigaword @xmath6: @xmath7: cnn/dm @xmath8: @xmath9: others @xmath10: @xmath11: instructions we use the same set of patterns across all tasks, but we combine them with different decoder prefixes. the patterns we use are: all decoder prefixes are shown in setup for all our experiments with genpet, we use pegasus-large @xcite as underlying language model and perform greedy decoding; our implementation is based on the transformers library @xcite and pytorch @xcite . unless stated differently, all experiments are performed using the same setup as schick and schütze (2021a) and a single gpu with 11gb ram (nvidia geforce gtx 1080 ti). for optimizing hyperparameters, much previous few-shot work uses development sets that are larger than the training sets by multiple orders of magnitude (e.g., @xcite @xcite ; however, assuming the existence of such large development sets is inconsistent with real-world few-shot settings. in contrast, schick and schütze (2021a) assume no development data at all and determine hyperparameters based only on previous work and practical considerations. we choose a middle course and create a small development set of 100 examples for only one of the six tasks, xsum. we use this development set in combination with a single training set of 10 examples to determine hyperparameters for all tasks and training sets. however, we do so only for hyperparameters for which no consistent value can be derived from previous work. following @xcite , we use a maximum input length of 512 tokens, the adafactor optimizer (shazeer and stern, 2018) with square root learning rate decay, a dropout rate of 0.1 and label smoothing setting ε = 0.1 @xcite ; we also adopt zhang et al. (2020)'s maximum output lengths for each task. as recommended by @xcite , we train all models for 250 steps using a batch size of 8. we also tried training for 500 and 1,000 steps on our development set but found no major differences in performance. for the learning rate, we tried values of α • 10 -5 with α ∈ {1, 10, 50} as schick and schütze (2021a) use α = 1 and zhang et al. (2020) use α = 50; we found α = 10 to perform best for all models. for unsupervised scoring ( §5.3), we use a threshold of τ = 0.2, i.e., we discard the 20% of examples that are least likely according to an untrained model. we chose this value by looking at texts generated by pegasus trained on 10 examples from the xsum development set, where we found the bottom 20% to contain texts of poor quality, including random telephone numbers and repetitions of the same word. for evaluation, we follow @xcite and report rouge1, rouge2 and rougel (r1/r2/rl) f1 scores @xcite after stemming using the porter algorithm @xcite . ## conclusion we investigated the ability of pretrained language models to make use of simple instructions with the aim of enabling more data-efficient text generation. we identified three major challenges: enabling language models to make good use of the instructions provided, ensuring that the instructions are useful and preventing overfitting. we tackle these in our proposed approach, genpet, by (i) introducing the concept of decoder prefixes, (ii) combining instructions through knowledge distillation where target sequences are generated with probabilistically sampled instructions and (iii) making use of unsupervised scoring and joint training. a pretrained pegasus model finetuned with genpet clearly outperforms regular finetuning in few-shot settings.