Masked Language Model Scoring (2020)

Abstract: Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one. We show that PLLs outperform scores from autoregressive language models like GPT-2 in a variety of tasks. By rescoring ASR and NMT hypotheses, RoBERTa reduces an end-to-end LibriSpeech model’s WER by 30% relative and adds up to +1.7 BLEU on state-of-the-art baselines for low-resource translation pairs, with further gains from domain adaptation. We attribute this success to PLL’s unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 (+10 points on island effects, NPI licensing in BLiMP). One can finetune MLMs to give scores without masking, enabling computation in a single inference pass. In all, PLLs and their associated pseudo-perplexities (PPPLs) enable plug-and-play use of the growing number of pretrained MLMs; e.g., we use a single cross-lingual model to rescore translations in multiple languages. We release our library for language model scoring at https://github.com/awslabs/mlm-scoring .

## introduction bert @xcite and its improvements to natural language understanding have spurred a rapid succession of contextual language representations @xcite ; inter alia) which use larger datasets and more involved training schemes. their success is attributed to their use of bidirectional context, often via their masked language model (mlm) objectives. here, a token w t is replaced with @xcite and predicted using all past and future tokens w \t := (w 1 , . . . , w t-1 , w t+1 , . . . , w |w | ). ## sequence-to-sequence rescoring let x denote audio features or source text tokens, and let @xmath0, . . . , w |w | ) denote target text tokens. for non-end-to-end asr and mt systems, having separate acoustic/translation models p am/tm (x | w ) and language models p lm (w ) is motivated by the bayes rule decomposition used to select the best hypothesis ŵ @xcite : ## analysis we recall the log-linear model from section 3.1: although end-to-end models @xmath1, interpolation with the unconditional @xmath2. one explanation comes from cold and simple fusion @xcite , which further improve on shallow fusion (section 3.1) by learning g(w ) first. they argue g expresses fluency; fixing g early allows f (w , x) to focus its capacity on adequacy in encoding the source, and thus specializing the two models. with this perspective in mind, we compare log p lm and pll as candidates for log g. ## related work our work extends the closest previous works @xcite with regards to experiments and tasks, as outlined in section 2.1. furthermore, neither work considers the inference cost of masked rescoring, which we address with our maskless scoring approach, or analyze pll's numerical properties. future context. log probabilities conditioned on past and future context have been used in mt @xcite and perennially in asr @xcite @xcite to positive effect. however, these are not "deep bidirectional" as they model interactions between @xmath3, while mlms model all pairwise interactions w s and w s via dotproduct attention (compare elmo versus bert). their plls would have different properties from ours (e.g., their cross-entropies in figure 4 may be convex instead of flat). discriminative language modeling. previous works @xcite have explored training language models that directly optimize for a downstream metric (wer, bleu). while we also eschew using log probabilities from conventional lms, our approach remains generative. log probabilities model the joint distribution; pll does so as well, albeit implicitly (appendix b). pll's summands (conditional probabilities) remain accessible for gibbs sampling and are not tailored to any metric. the two approaches are complementary; for example, one could use pll as a "prior" or regularizer for scores given by discriminatively-finetuned bert models in tasks like passage re-ranking @xcite . language model integration. beyond finetuning pretrained lms and mlms, monolingual pretraining has also improved nmt performance @xcite . however, modular integration of language representation models remains prevalent for various pragmatic reasons, similar to fusion in asr. contemporary examples are the use of finetuned bert scores in a question-answering pipeline @xcite , or "as-is" cosine similarity scores from bert to evaluate generated text @xcite . for example, one might have no pretrained multilingual lms for decoder initialization or fusion, as such models are difficult to train @xcite . however, one may have an m-bert or xlm for the target language/domain. finally, n -best rescoring and pretraining are not mutually exclusive, though pretraining may already go partway to improve fluency. ## conclusion we studied scoring with mlm pseudo-loglikelihood scores in a variety of settings. we showed the effectiveness of n -best rescoring with plls from pretrained mlms in modern sequenceto-sequence models, for both asr and low-to medium-resource nmt. we found rescoring with plls can match or outperform comparable scores from large unidirectional language models (gpt-2). we attributed this to pll's promotion of fluency via self-consistency, as demonstrated by improvement on unsupervised acceptability judgements and by qualitative analysis. we examined the numerical properties of plls, proposed maskless scoring for speed, and proposed pseudo-perplexities for intrinsic evaluation of mlms, releasing a codebase implementing our work. future work could find additional modular uses of mlms, simplify maskless pll computations, and use plls to devise better sentence-or document-level scoring metrics.