A Better Way to Do Masked Language Model Scoring (2023)

Abstract: Estimating the log-likelihood of a given sentence under an autoregressive language model is straightforward: one can simply apply the chain rule and sum the log-likelihood values for each successive token. However, for masked language models (MLMs), there is no direct way to estimate the log-likelihood of a sentence. To address this issue, Salazar et al. (2020) propose to estimate sentence pseudo-log-likelihood (PLL) scores, computed by successively masking each sentence token, retrieving its score using the rest of the sentence as context, and summing the resulting values. Here, we demonstrate that the original PLL method yields inflated scores for out-of-vocabulary words and propose an adapted metric, in which we mask not only the target token, but also all within-word tokens to the right of the target. We show that our adapted metric (PLL-word-l2r) outperforms both the original PLL metric and a PLL metric in which all within-word tokens are masked. In particular, it better satisfies theoretical desiderata and better correlates with scores from autoregressive models. Finally, we show that the choice of metric affects even tightly controlled, minimal pair evaluation benchmarks (such as BLiMP), underscoring the importance of selecting an appropriate scoring metric for evaluating MLM properties.

## introduction most state-of-the-art transformer-based large language models (llms) fall into two classes: unidirectional (or autoregressive) models, where each token is generated based on its left context (e.g., gpt models; @xcite , and bidirectional models, where a token is predicted from both left and right context tokens, some of which may be masked (e.g., bert; @xcite . often, it is beneficial to compare these models' performance on controlled sentence generation benchmarks. whereas unidirectional architectures offer a figure 1 : three different ways to compute the pll score of a multi-token word (e.g., souvenir) during masked language modeling. purple: target token, pink: within-word tokens that are available during inference, turquoise: within-word tokens that are masked during inference. sentence tokens that do not belong to the current word are always available during inference. so far, the best available method to score a sentence under a bidirectional llm has been the pseudo-log-likelihood (pll) scoring approach described by @xcite (and initially used by @xcite . the pll of a sentence is calculated as the sum of pll scores for each token given all other sentence tokens, thus providing a comparable metric to unidirectional models' log-likelihood (ll) sentence scoring. the pll metric is extremely popular; it is used extensively in llm studies tackling topics as diverse as effects of training data @xcite , model fluency @xcite , syntactic and conceptual knowledge @xcite , social biases @xcite , and others. some of these studies have already accrued dozens of citations. here, we show that the metric proposed by @xcite has important shortcomings that limit its utility. specifically, pll-original overestimates the pll of outof-vocabulary (oov) words, which llm tokenizers split into multiple tokens. as a result, pll-original scores fail on several theoretically desired property tests: a robust inverse relationship between sentence length and sentence pll (section 4.1), a robust positive correlation between a word's frequency and its pll score (4.2), and a positive correlation between unidirectional and bidirectional model scores for the same sentences (section 5). to remedy these issues, we propose an adjusted pll metric, pll-word-l2r (l2r: leftto-right), which estimates token pll when future within-word tokens are also masked (figure 1 ). we show that the pll-word-l2r metric outperforms both pll-original and alternative pllbased metrics. we therefore recommend to use the pll-word-l2r metric when estimating sentence pll under a bidirectional llm. ## motivation: score inflation for multi-token words the pll-original metric grossly overestimates the probability of oov lexical items, such as souvenir (figure 2 ). this is because oov words are tokenized into subword tokens (e.g., so ##uven ##ir), and each subword token is predicted using the token's bidirectional context, which crucially includes the remaining tokens that make up the oov word. thus, even though the oov word itself may be surprising given the sentence context, the individual parts of the oov word are not surprising to a bidirectional model given a sentence context that includes all other subtokens of that word (e.g., it is easy to predict so given ##uven ##ir; see appendix a for additional examples). to mitigate this bias, we adjust the pll sentence scoring algorithm such that the model cannot access future within-word tokens (pll-word-l2r) or any within-word tokens (pll-whole-word) when predicting the target. below, we conduct a rigorous investigation of our modified metrics to determine whether this intuitive benefit holds quantitatively. ## methods for our analysis, we adapt the scorer module of the minicons library @xcite , an open-source wrapper library around huggingface transformers @xcite ) that enables efficient extraction of word-and sentence-level probabilities from llms. the mlm scoring procedure of the minicons library follows the procedure originally proposed by @xcite . for details on sentence preprocessing, see appendix b. ## correlation with gpt-2 scores we expect that pll scores for bidirectional models should be at least somewhat consistent with ll scores for unidirectional models: both metrics are designed to serve are a proxy for sentence probability. here, we show that the gpt-2/bert score correlation for the pll-original metric is very low, whereas correlation scores for pll-word-l2r and pll-whole-word are much higher (figure 4 ), indicating the validity of this metric for cross-model comparison. as in section 4.2, pll-word-l2r slightly outperforms pll-whole-word, likely because it does not penalize oov words as severely. see appendices d-f for evidence that all three trends hold for larger models and for other datasets (although the effects in other datasets are attenuated due to a lower oov ratio). ## effects on benchmarking here, we show that the choice of pll metric affects benchmarking results for a popular, highly controlled, minimal pair linguistic benchmark: blimp. despite the fact that the comparisons are highly controlled, different metrics yield different blimp scores. for all four tested models, pll-word-l2r achieves the best overall blimp score ( ## conclusion we have shown that pll-word-l2r is the preferred metric for evaluating sentence pll under a masked language model, such as bert. although the results from studies using the pll-original metric can still be informative, they become harder to interpret if the proportion of oov words in their test set is high. therefore, we recommend using pll-word-l2r in future works.