Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order (2020)

Abstract: Masked language model and autoregressive language model are two types of language models. While pretrained masked language models such as BERT overwhelm the line of natural language understanding (NLU) tasks, autoregressive language models such as GPT are especially capable in natural language generation (NLG). In this paper, we propose a probabilistic masking scheme for the masked language model, which we call probabilistically masked language model (PMLM). We implement a specific PMLM with a uniform prior distribution on the masking ratio named u-PMLM. We prove that u-PMLM is equivalent to an autoregressive permutated language model. One main advantage of the model is that it supports text generation in arbitrary order with surprisingly good quality, which could potentially enable new applications over traditional unidirectional generation. Besides, the pretrained u-PMLM also outperforms BERT on a bunch of downstream NLU tasks.

## introduction large-scale pretrained language models @xcite @xcite @xcite have drawn lots of research attention as these models have brought significant improvements to many nlu and nlg tasks. as a major category of pretrained language models, masked language model (mlm) @xcite is trained using a denoising autoencoding objective. in a typical mlm, some tokens in a sentence are replaced by a special token @xcite . the training objective is to predict the original tokens that are masked in the sentence. as the first large-scale pretrained masked language model, bert chooses to mask 15% of the tokens in sentences randomly. following @xcite the wolf has an extraordinary speed , and it can often jump from a spot quick enough to escape a spot already occupied by an adult wolf . unlike the brown and black bear , where it is easily distracted by wolves , the gray fox does not run over a wolf , and is often driven mad . having jumps with high speed that breaks the wolf ' s legs before it is run over , a grey wolf could defend itself against an adult of other species as the best predator at any time . the black bear may kill packs of four lazy , though the gray fox can inflict significant wounds on a dog . figure 1 : a piece of text generated by a pmlm in random order. the bolded words, which compose the input sentence "the quick brown fox jumps over the lazy dog", are distributed across the paragraph with a predefined length. the blank spaces are filled by the model in a random order to form the complete paragraph. while the pretrained masked language models achieve state-of-the-art performances in a line of downstream nlu tasks, researchers pay more attention to autoregressive language model when it comes to text generation. unlike predicting the masked tokens, the autoregressive language model learns a sequential generative process of text sequences. hence it naturally performs better for natural language generation. for example, gpt-2 @xcite as well as transformer-xl @xcite , is able to generate fluent and coherent paragraphs of text that highly resembles human writings. in this paper, we propose a probabilistically masked language model (pmlm) to bridge the gap between masked and autoregressive language models. the basic idea behind the connection of two categories of models is similar to @xcite . pmlm is a masked language model with a probabilistic masking scheme, which defines the way sequences are masked by following a probabilistic distribution. while the existing work proposes masking strategies aiming at improving the nlu abilities, pmlm addresses the generation capability in particular. besides, as a masked language model, pmlm maintains its strong ability in natural language understanding. in addition to the traditional unidirectional (e.g., left-to-right) generation, a unique ability for pmlm is to autoregressively generate sequences in arbitrary order, and the generated sequences are still of high quality. in contrast to traditional left-to-right generation, arbitrarily ordered text generation has two main characteristics. first, the next token to be predicted could be in any position that is masked. second, the next token to be predicted depends on all the previous observed/generated tokens. arbitrarily ordered generation enables more interesting applications than unidirectional generation. for example, figure 1 shows an example of cloze test, where the prompted text "the quick brown fox jumps over the lazy dog" is distributed across a paragraph with a predefined length, and the task is to predict all the surrounding words and complete the paragraph. this is actually very challenging for conventional generation models since when predicting each word, the fluency and coherence of text are hard to be guaranteed given the contextual constraints on both sides. more applications may include acrostic poetry generation, news generation based on given facts, machine translation with lexical constraints, etc. we employ a simple uniform distribution of the masking ratio and name the model as u-pmlm. we prove that u-pmlm actually learns an autoregressive language model on random permutations of training sequences. the experiments show that the quality of text generated by u-pmlm in arbitrary order is as good as that generated by gpt in sequential order. besides, u-pmlm outperforms bert significantly on the glue benchmark for natural language understanding. ## probabilistically masked language model different masking schemes have been proposed for pretraining the masked language model. the most straightforward masking scheme is to randomly mask tokens in sentences in a fixed ratio, e.g., 15% in bert. following bert, various models have proposed modifying the masking scheme to improve its nlu capability. ernie @xcite proposes the entity-level masking and phrase-level masking, where the words composing an entity or phrase are masked as a whole. spanbert @xcite proposes to mask a continuous random span of text rather than random tokens. these masking strategies have shown to be effective for certain classes of nlu tasks. in contrast to the existing work, we propose a probabilistic masking scheme that tries to improve the text generation ability of the masked language model. probabilistically masked language mode (pmlm) is a natural generalization of the mlm with a probabilistic masking ratio. it assumes that the masking ratio is drawn from a probabilistic distribution. therefore, each training instance is associated with a different masking ratio sampled from the given distribution. ## experiments we evaluate both the natural language generation ability and natural language understanding ability of u-pmlm trained in the settings described in section 3.4. ## conclusion we have proposed a probabilistically masked language model for autoregressive generation in arbitrary word order. the experiments show that the text generated in arbitrary order has comparable quality with gpt. besides, the proposed probabilistic masking scheme also improves the nlu capability of a masked language model. permutations for π σ t+1 and -π σ t are factorials of |π σ t+1 | and | -π σ t |, denoted as: hence the duplication factor for log p(x σt |x -π σ t ) is computed as (n -k)!(k -1)!, which is the coefficient of the expected log-likelihood function of u-pmlm denoted in equation 10. thus we conclude that equation 10 is equivalent to equation 8, where the constant @xmath0.