Blank Language Models (2020)

Abstract: We propose Blank Language Model (BLM), a model that generates sequences by dynamically creating and filling in blanks. The blanks control which part of the sequence to expand, making BLM ideal for a variety of text editing and rewriting tasks. The model can start from a single blank or partially completed text with blanks at specified locations. It iteratively determines which word to place in a blank and whether to insert new blanks, and stops generating when no blanks are left to fill. BLM can be efficiently trained using a lower bound of the marginal data likelihood. On the task of filling missing text snippets, BLM significantly outperforms all other baselines in terms of both accuracy and fluency. Experiments on style transfer and damaged ancient text restoration demonstrate the potential of this framework for a wide range of applications.

## introduction neural language models have shown impressive performance across many applications such as machine translation and summarization where the text is generated from scratch @xcite . however, a broader set of text generation tasks -including text editing, information fusion, and ancient text restoration -requires the model to start with partially specified text and generate the missing fragments. in the general setup, the input document may have any number of missing spans, and each span may have an unknown number of missing tokens. to perform this text infilling task @xcite , a model should: they also have which . they also have ice cream which is really good . existing approaches focus on adapting left-toright language models for text infilling. intricate inference algorithms leveraging dynamic programming or gradient search are proposed to find the filling content that has a high likelihood within the surrounding context @xcite @xcite . these methods make simplified markov assumptions, require high decoding time complexity, and cannot adapt to variable infilling length. alternatively, @xcite predict the concatenation of the infilling content, but do not guarantee that the output will match the number of missing spans in the input. in this work, we introduce the blank language model (blm), which uses a special " " symbol to control where tokens can be placed. the generation of blm follows the grammar of replacing a blank with a word and possibly adjoining blanks. by jointly modeling context and missing content, blm supports the control of generation location and produces consistent infilling of variable length. our model can start from a single blank or partial text with blanks in specified locations. it maps the entire input into a sequence of vector representations, and further processes the representations in blank positions to determine the generation action. generation actions are performed iteratively until there are no blanks. since multiple trajectories of blm actions can produce the same final text, we train the model by maximizing a lower bound of the log-likelihood marginalized over trajectories. at test time, we can use simple greedy decoding or beam search to fill in the blanks in the input text. ## related work recent work has explored various sequence models for non-autoregressive machine translation @xcite . the insertion transformer supports dynamic canvas with word insertion @xcite , but does not allow users to specify where to insert. the model is unaware of which parts of the canvas are contiguous text spans that should remain intact, and which (potentially scattered) parts need to be filled in. directly forcing the insertion transformer to perform text infilling can therefore lead to suboptimal solutions. the levenshtein transformer combines insertion and deletion through complex policy learning @xcite . its insertion mechanism is a two-stage process in which placeholders are first predicted and then filled-in in a masked language model (mlm) manner. in text infilling where the blanks/placeholders are given, it reduces to an mlm. mlms are commonly used in representation learning @xcite . to use them in rewriting tasks, one needs to specify the insertion length in advance and heuristically determine the generation order among the masks @xcite @xcite . similarly, xl-net requires absolute positional embedding and thus does not support unknown-length text infilling @xcite . blm provides a natural formulation for generative modeling that can dynamically accommodate insertions of various length. another line of work focuses on finding an optimal language generation order, such as syntaxbased generation @xcite and learning adaptive generation order @xcite . these approaches are tailored to generation from scratch in a specific order. our model instead is attuned for text rewriting, where the missing parts can be located anywhere in the input text, and the algorithm must flexibly complete them. ## blank language models a blank language model (blm) generates sequences by creating and filling in blanks. generation starts with a single blank and ends when there is no blank. in each step, the model selects a blank " ", predicts a word w, and fills the blank with "w", " w", "w ", or " w ". this way, a blank can be expanded to any number of words. we define a canvas as a sequence of words interspersed with special " " tokens. the subsequent action is conditioned on this intermediate stage of generation. suppose the current canvas is blm maps this canvas to a distribution over actions specifying how the canvas is to be revised: and the terminal symbols come from the vocabulary v . the production rules are restricted to be of the form " " → " ?w ?" for @xmath0, where "?" indicates that the preceding symbol is optional. in contrast to context-free grammars, the probability distribution over production rules in blm is conditioned on the entire canvas generated so far. ## experiments we test blm's capacity to rewrite specified portions of text on three tasks: text infilling @xcite , ancient text restoration @xcite and style transfer @xcite . fig. 4 displays example inputs and outputs for these tasks. we also measure the perplexity of blm on language modeling benchmarks and compare with traditional left-to-right language models. ## conclusion in this paper, we proposed the blank language model for flexible text generation. given partially specified text with one or more blanks, blm will fill in the blanks with a variable number of tokens consistent with the context. we demonstrate the effectiveness of our model on various text rewriting tasks, including text infilling, ancient text restoration and style transfer. the action of blm consists of selecting a blank and replacing it with a word and possibly adjoining blanks. we train blm by optimizing a lower bound on the marginal data likelihood that sums over all possible generation trajectories. in this way, we encourage the model to realize a sentence equally well in all orders, which is suitable for filling arbitrary blanks. appendix c shows examples generated by blm along with their trajectories. depending on the application, we could also train the model to generate in specific orders by placing higher weights on the corresponding trajectories. blm has plenty of future applications, including template filling, information fusion, assisting human writing, etc. moreover, we can extend our formulation to a conditional generative model. such models can be used in machine translation to support editing and refining translation, as well as in dialogue systems to compose a complete sentence with given elements. while we proposed blm for language generation, it would also be interesting to compare the representations learned by blm with those produced by other pre-training methods.