The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank (2022)

Abstract: The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do fill in the blank, a more useful model is one that can effectively perform _both_ FitB and continuation tasks. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how these models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.

## introduction natural language generation systems are increasingly being incorporated into applications where a human writer and an ai jointly collaborate to construct text. these range from creative domains such as collaborative story writing @xcite to more practical ones such as email composition and code synthesis @xcite @xcite . these applications are often limited to generating text at the end of what has been written so far. this is because language models (lms) are typically designed to produce text by repeatedly predicting the next word in a sequence given the previous words. however, there is a need for more powerful interactive tools which enable writers to solicit insertions at any chosen position within the existing text, a task referred to as fill in the blank (fitb) or infilling. for example, a creative writer might want a tool which can insert a description of a place or character, and a programmer might want a system that can fill in the contents of a function located in the middle of their code. fill in the blank with about 16 words and include the phrase "old dog": "the boy took the ____ for a walk." fill in the blank with about 4 words: "the boy took the ____ for a walk." continue the text with about 2 words: "the boy took the lonely old dog ____" continue the text with about 8 words and include the phrase "rocky path": "the boy took the lonely old dog ____" leash off the hook. his old dog still acted like a puppy when it came time two dalmatians to the beach inside the house. up the rocky path. it was slow going. ## supporting fitb and continuation definitions. we define filling in the blank as the task of predicting text to replace a single missing span, usually demarcated with a special token, in an input text passage. using a model pre-trained for fitb. t5 is a model pre-trained with a "span corruption" objective very similar to fitb; the model is asked to reconstruct the missing text after random subsequences of the input are replaced with special identifiers. thus, a pre-trained t5 model can be used without any further training to do both continuation and infilling by appropriately choosing text to mask out. the encoder-decoder architecture of t5 is also more conducive to fitb than the decoder-only architectures that are typically used for continuation-based generation, such as gpt-2 @xcite . this is because the attention mechanism in encoder-decoder architectures allows the context on the left side of the blank to attend to the context on the right, while decoder-only architectures only support masked attention (each token can only attend to the positions to its left). even though t5's pre-training objective was a form of fitb, finetuning is still advantageous. for one, our definition of fitb involves only a single masked out substring, not multiple, so finetuning improves alignment with the goal task. finetuning also allows us to incorporate additional conditioning signals not supported by the pre-trained t5, such as being able to specify the desired length of the generated text or specify words that ought to be included in the blank, a task we refer to as "goal conditioning." length control, which comes by default in a traditional language model by simply sampling more or fewer tokens, is particularly necessary for fitb, where the end of the generation must fit seamlessly with the text to its right. using a model pre-trained for continuation. the biggest language models available today were largely trained in the continuation rather than the fitb paradigm @xcite . since our primary goal is to have a single model for both tasks, we also address the question: if a continuation-trained model is big enough, can it handle fitb without the need for finetuning? few-shot learning with large language models, as popularized by @xcite , has had success on many tasks in nlp. we try out this approach for fitb by designing a few-shot prompt containing several demonstrations of the fitb task, formulated in a similar "infilling by language modelling" template to that proposed by @xcite . further details on our approach to selecting a few-shot prompt are in appendix a.1. ## experiments model. for all experiments with t5, we use the 800m parameter v1.1 'large' model ( appendix a.4 gives additional results from the 3b parameter 'xl' model). to finetune t5 for fitb, we construct training examples from documents by first partitioning the document text into a left context, gap, and right context. the input to the model is then the left and right contexts concatenated with textual representations of the additional conditioning signals. the target sequence is the true text for the blank. this formulation easily supports continuation, as the blank can be deliberately placed at the end (i.e., providing no right context). finetuning examples are drawn from c4, the same dataset t5 was pre-trained on. documents are split into word sequences, and these are then randomly truncated to be between 256 and 512 words long. a substring of between 1 and 64 words is selected to be blanked out. for half of the finetuning examples, the location of the blank is randomly selected, and for the other half, it is always placed at the end. to support length conditioning, we follow @xcite and include a bucketed version of the target length as part of the blank. to support goal conditioning, for half the examples, a random substring of up to half the words of the target is appended to the end of the input. examples are shown in baselines we compare t5 against @xcite 's 137b parameter decoder-only language model (referred to in this paper as llm). trained explicitly for continuation, this model has been used successfully for few-shot learning in other domains @xcite . we use the llm in two ways: (1) as a standard continuation model, prompting with only the left context of an example; and (2) in a few-shot learning paradigm. evaluation datasets we evaluate continuation and fitb on c4 as well as two story writing datasets. we chose this domain because creative writing assistant applications are one of the key areas we expect to benefit from multi-task models @xcite . reddit writing prompts (rwp) is a corpus of stories from the 'r/writingprompts' sub-reddit @xcite , and we construct validation sets rwpfillblank and rwpfillend using the same method described in the previous section. we cap the c4 and rwp validation sets to 5,000 examples each. roc stories (roc) is a crowd-sourced dataset of five-sentence commonsense stories @xcite . for roc stories, the 2018 validation set is used to construct rocfillmiddle, where the middle sentence of each story is blanked out, and rocfillend, where the last sentence is blanked out. unless otherwise noted, all evaluation is done without goal conditioning and uses random sampling with top-@xmath0. example generations for all evaluation sets can be found at https://bit.ly/2u0ixxa . ## findings automatic evaluation we measure the fluency of proposed generations by evaluating the perplexity of each dataset's examples when the predicted text is placed in the blank @xcite . we use the llm to measure perplexity foot_0 . the results are shown in ## conclusion in this work, we make the case for starting with a model capable of filling in the blank when attempting to build a system that can perform both fitb figure 3 : for each of the fitb validation sets, a histogram of the distribution of sequence lengths (measured in words) of the ground-truth blanked out text and the proposed infills from t5 (after and before finetuning). we see that pre-trained t5 tends to produce text that is shorter than the groundtruth. and continuation. as lms become bigger, it will be unsustainable to have separately trained models for each generation task. multi-task, domaintransferable models, such as the ones we propose, require less total training and are more efficient to store and use at inference time. while pre-trained t5 by itself is capable of both infilling and continuation, additional conditioning signals such as desired length and goal text can be successfully incorporated into fine-tuning in order to support an even greater diversity of model interactions. we focused our experiments on the t5 model; however, we expect that other model families and architectures can be trained similarly to support a variety of generation tasks. for example, gpt-3 @xcite ) recently began supporting "insertion" and "edit" interactions in addition to continuation. finally, we present a negative result that while fewshot learning is a promising method for building multi-task support without any finetuning, it is challenging to make work for the fitb task.