Enabling Language Models to Fill in the Blanks (2020)

Abstract: We present a simple approach for text infilling , the task of predicting missing spans of text at any position in a document. While infilling could enable rich functionality especially for writing assistance tools, more attention has been devoted to language modeling—a special case of infilling where text is predicted at the end of a document. In this paper, we aim to extend the capabilities of language models (LMs) to the more general task of infilling. To this end, we train (or fine tune) off-the-shelf LMs on sequences containing the concatenation of artificially-masked text and the text which was masked. We show that this approach, which we call infilling by language modeling , can enable LMs to infill entire sentences effectively on three different domains: short stories, scientific abstracts, and lyrics. Furthermore, we show that humans have difficulty identifying sentences infilled by our approach as machine-generated in the domain of short stories.

## introduction text infilling is the task of predicting missing spans of text which are consistent with the preceding and subsequent text. 1 systems capable of infilling have the potential to enable rich applications such as assisting humans in editing or revising text @xcite , connecting fragmented ideas @xcite , and restoring ancient documents @xcite . rather than targeting a particular application, our goal here is to provide a general, flexible, and simple infilling framework which can convincingly infill in a variety of domains. a special case of infilling is language modeling: predicting text given preceding but not subsequent text. 2 language models are (1) capable of generat- she ate leftover pasta for lunch. she ate [blank] for [blank]. leftover pasta [answer] lunch [answer] data input target our infilling framework she ate [blank] for [blank]. she ate leftover pasta for lunch. infilling task input output train language model infilling input [sep] target data input [sep] target output figure 1 : we consider the task of infilling, which takes incomplete text as input and outputs completed text. to tackle this task, our framework constructs training examples by masking random spans to generate pairs of inputs (text with blanks) and targets (answers for each blank). we then train unidirectional language models on the concatenation of each pair. once trained, a model takes text input with blanks, predicts the answers, and then combines them to produce the output. in this work, we present infilling by language modeling (ilm), a simple framework which en-ables lms to infill variable-length spans while preserving their aforementioned benefits: generation quality, efficient sampling, and conceptual simplicity. our framework involves a straightforward formulation of the infilling task which, as we demonstrate, can be learned effectively by existing lm architectures. as shown in figure 1 , our approach concatenates artificially-masked text with the text which was masked, and adopts a standard lm training (or fine-tuning) procedure on such examples. once trained, infilling can be performed for a document with blanks by using the lm to generate text and then replacing the blanks with this text. in addition to its conceptual simplicity, our experiments show that ilm enables off-the-shelf lms to infill effectively. furthermore, we find that infilling performance improves when starting from a large-scale pre-trained lm (as opposed to training from scratch), suggesting an additional benefit of using our model-agnostic framework compared to approaches which require specialized architectures. we provide an interactive web demo of models trained under our framework. this demo can infill multiple variable-length spans with different granularities (e.g. words, n-grams, and sentences) on the domains of short stories, scientific abstracts, and song lyrics: https://chrisdonahue.com/ilm . all code, data, and trained models are available at https://github.com/chrisdonahue/ilm and also on the codalab platform at https: //worksheets.codalab.org/worksheets/ 0x9987b5d9cce74cf4b2a5f84b54ee447b. ## problem statement the task of infilling is to take incomplete text x, containing one or more missing spans, and return completed text x. let [blank] be a placeholder for a contiguous sequence (span) of one or more missing tokens. then, incomplete text x is a sequence of tokens some of which are @xcite . in order to map x to x, an infilling strategy must specify both how many and which tokens to generate for each @xcite . note that there may be many reasonable x for a given x. hence, we are interested in learning a distribution p(x | x). ## infilling by language modeling in this section, we describe our ilm framework. we first outline a simple reparametrization of the infilling task. then, we define a procedure for automatically generating suitable training examples which can be fed to an off-the-shelf lm. fedus et al. (2018 ) explore an infilling framework where lms are trained on concatenations of x and x, i.e., they use lms to directly predict x given x. while their approach is effective at infilling individual words, it is somewhat redundant as the model must "predict" the unmasked text in x. additionally, a model is not guaranteed to exactly reproduce the unmasked text. ## experimental setup we design our experiments to determine if training an off-the-shelf lm architecture with our ilm framework can produce effective infilling models for a variety of datasets. specifically, we train on three datasets of different sizes and semantics (details in appendix a): short sto-ries @xcite , cs paper ab-stracts, and song lyrics. ## quantitative evaluation we evaluate the quantitative performance of our models on the sentence infilling task by measuring ppl on test data. 3 in this setting, a sentence is selected at random and masked out, and we measure the likelihood assigned by a model to the masked sentence in the context of the rest of the document. regardless of differences in the ordering and number of tokens that each strategy uses to represent a test example, ppl is always computed only for the span of tokens comprising the original sentence (e.g. green tokens in figure 2 ). we also observe that models trained via ilm perform similarly on the special case of language mod-6 human evaluation in addition to our quantitative evaluation, we seek to evaluate the qualitative performance of ilm. to this end, we sample a story from the stories test set and randomly replace one of its five humanwritten sentences with a model output. then, we task human annotators on amazon mechanical turk with identifying which of the sentences in a story was machine-generated (details in appendix e). we compare our ilm model to three baseline infilling strategies: an lm (context beyond the replaced sentence was discarded), the best model (self-attention; sa) from @xcite , and the pre-trained bert (base) model @xcite . all approaches except for bert were first fine-tuned on the stories dataset. to infill using bert, we replace the tokens representing the original sentence with mask tokens, and then generate text by replacing mask tokens one at a time (conditioning on previously-generated tokens). while vocabulary differences make it is less useful to compare ppl for the sa and bert baselines to our gpt-2-based strategies, we can still meaningfully compare them in this human evaluation setting. for each approach we compute a score, which we define as the percentage of examples where the annotator did not correctly identify the machinegenerated sentence. therefore, a higher score implies a better (more natural, human-like) model. we collect 100 responses for each model and report the scores in of the four strategies, ilm achieves the highest score, implying that sentences infilled by ilm are harder for humans to recognize as fake than those produced by other strategies. somewhat surprisingly, we observed that despite only observing past context the lm model performed better than bert and sa. bert may have performed poorly due to the intrinsic difficulty of finding convincing infills with a precise length in tokens. sa may have performed poorly because, unlike lm and ilm, it was not initialized from a large-scaled pre-trained lm. ## related work methodology. a number of systems have the capability to infill but have practical drawbacks. many systems are unable to automatically determine span length, and thus, can only infill fixedlength spans @xcite @xcite @xcite . methods such as bert present additional challenges during inference @xcite . @xcite frame narrative cloze as a generation task and employ language models, but they only consider one infill of a fixed length. @xcite ; @xcite infill multiple variable-length sequences, but these approaches require the masked context to be iteratively updated and reprocessed to fill in blanks one a time. in contrast, our approach appends infilled text to the context and does not require reprocessing the entire input sequence for each blank. @xcite train an lm which can fill in the middle of a paragraph given the first and last sentences-our work generalizes to such capabilities. task. the cloze task @xcite evaluates language proficiency by asking systems to fill in randomly-deleted words by examining context. cloze has been extended in the forms of discourse @xcite and narrative cloze @xcite , which remove phrases and narrative events respectively. recently, cloze has been used not only for evaluation, but also to improve text generation quality @xcite and transfer learning @xcite (under the name "masked language modeling"). text infilling can be thought of as generalizing the cloze task from single words to spans of unknown length. @xcite explore infilling as a pre-training objective to improve downstream performance on inference tasks; our work focuses on generation. ## conclusion we presented a simple strategy for the task of infilling which leverages language models. our approach is capable of infilling sentences which humans have difficulty recognizing as machinegenerated. furthermore, we demonstrated that our infilling framework is effective when starting from large-scale pre-trained lms, which may be useful in limited data settings. in future work, we plan to incorporate these features into co-creation systems which assist humans in the writing process. we hope that our work encourages more investigation of infilling, which may be a key missing element of current writing assistance tools.