CONTESTS : a Framework for Consistency Testing of Span Probabilities in Language Models (2024)

Abstract: Although language model scores are often treated as probabilities, their reliability as probability estimators has mainly been studied through calibration, overlooking other aspects. In particular, it is unclear whether language models produce the same value for different ways of assigning joint probabilities to word spans. Our work introduces a novel framework, ConTestS (Consistency Testing over Spans), involving statistical tests to assess score consistency across interchangeable completion and conditioning orders. We conduct experiments on post-release real and synthetic data to eliminate training effects. Our findings reveal that both Masked Language Models (MLMs) and autoregressive models exhibit inconsistent predictions, with autoregressive models showing larger discrepancies. Larger MLMs tend to produce more consistent predictions, while autoregressive models show the opposite trend. Moreover, for both model types, prediction entropies offer insights into the true word span likelihood and therefore can aid in selecting optimal decoding strategies. The inconsistencies revealed by our analysis, as well their connection to prediction entropies and differences between model types, can serve as useful guides for future research on addressing these limitations.

## introduction pretrained large language models (llms) emerged as high-performance predictors for diverse tasks @xcite . tuning based on instructions and human feedback has further advanced their performance @xcite . in various applications, the model's success often depends solely on assigning high scores to the correct options. yet, their interpretation as probabilities is beneficial in several applications, such as 1 code is provided at https://github.com/ eitanwagner/contests figure 1 : experimental design -joint prediction estimation with masked language modeling. the middle white row displays the original unmasked tokens. below, in blue, the joint probability is calculated by first estimating the probability of the correct token in mask 1 and then of mask 2 (after revealing the correct token in mask 1 ). in the top rows, in green, the calculation is in the reversed order -first estimating the probability of the correct token in mask 2 and then in mask 1 (after revealing mask 2 ). indeed, llms are commonly trained using the cross-entropy objective, which due to gibbs' inequality is minimized by the true distribution. however, this global minima is hard to achieve @xcite , raising the question of whether the model's outputs can still be interpreted as estimated probabilities. while calibration of produced scores has been extensively studied @xcite , multiple other aspects remain unexplored. measuring calibration for string density estimation requires a ground-truth measure for the real distribution, which is challenging to obtain (see §3). therefore, alternative methods to validate the assumption that produced scores correspond to es-timated probabilities, are needed. for llm scores to be interpreted as probabilities, consistency across estimation methods is essential. however, regardless of probabilistic interpretation, detecting and understanding inconsistencies among estimation methods is crucial, especially in the identification of preferable estimation methods. considering completion of word spans, consistency implies that filling masks in different orders (first filling one word and then the other, or vice versa) produces the same joint probability (see figure 1 ). in this work, we investigate whether this requirement is fulfilled. various factorizations of a joint distribution into conditional probabilities are possible, for instance by applying the chain rule from left-to-right or right-to-left. however, an unrestricted set of conditional probabilities does not guarantee a unique joint probability (see example in @xcite . although restricting the conditional probabilities, such as by disallowing cycles in bayesian networks, can ensure a unique joint probability, it is not necessary (see example in @xcite . masked language modeling (mlm) training lacks mechanisms to ensure that a set of conditional probabilities will form a unique joint distribution. however, since language modeling is based on the assumption that a distribution over strings generates the samples, estimated conditional distributions are expected to align with joint conditioning. we investigate the consistency of both mlm and autoregressive models (which include decoderonly and encoder-decoder models), considering that mlm can function as a missing token classification task or a generative task with a specific instruction prompt (see §4). when treating autoregressive models, we account for task comprehension as a contributing factor. we introduce a novel framework that employs statistical tests -contests, for consistency testing over spans, to analyze discrepancies between different estimation methods, and their behaviors across various model types. our findings show that all examined llms fail to produce consistent probabilities for joint span modeling. however, we observe notable distinctions among model types and sizes: autoregressive models show increasing discrepancies as model parameters increase, while mlms tend to provide more consistent predictions, with larger models offering further improvements. additionally, we show that prediction entropies are indicative of the true likelihood for both model types, suggesting their usefulness in selecting optimal decoding strategies. ## preliminaries and notation the task of single-mask probabilistic masked language modeling is to estimate the probability of a masked location, given the rest of the sequence. for a model m and a sequence of tokens @xmath0, x 2 ...x n ), we denote the estimation by when multiple masks are considered, that is, when in addition to i, the positions j 1 , .., j k are masked as well, the predicted distribution for the i-th position given all unmasked ones is given by and the joint distribution of two masked positions i, j, given unmasked ones, is denoted by masked language modeling can be performed with autoregressive models, predicting a span given an appropriate instruction prompt. these models predict p m (x @xmath1, where s is a span of arbitrary length (i.e., len(s) ≥ 1). 2 the predicted length is determined by the prediction of an end-ofsentence (eos) token. as autoregressive models are trained to estimate the probability of completions, the probability of the token followed by eos should match the mlm probability. 3 although we define the masked language modeling task regardless of the prediction model, following common convention, we will use the simple term mlm for models that were pretrained with the mlm objective. we will explicitly mention the use of autoregressive models. for simplicity, we focus on the estimation of joint probabilities of two tokens. to neutralize the effect of word distances, we analyze probabilities of adjacent tokens p (x i , x i+1 ), which can be expressed in two forms: when the identity of x i and x i+1 is implicitly clear we denote these expressions simply as p i,i+1 and p i+1,i . although for the true distribution we have that p i,i+1 = p i+1,i , this may not hold for estimated probabilities since each direction involves two inference steps, each with slightly different inputs. 4 when analyzing discrepancies, we at times consider the pointwise mutual information (pmi) between two random variables at ## previous work consistency. few works have directly addressed the assessment of consistency. among those that have, many focused on testing whether outputs adhere to predefined constraints. li et al. (2019) and ribeiro et al. (2019) demonstrated violations of logical constraints in question-answering. elazar et al. (2021) evaluated the internal knowledge consistency of language models (lms) comparing outputs for paraphrases of semantically identical questions. pezeshkpour and hruschka (2023) demonstrated sensitivity to answer ordering in multiplechoice questions. qiu et al. (2023) illustrated inconsistent temporal reasoning in llms across various time-related tasks. in our work, we focus on the consistency of probabilities rather than outputs. while identical probabilities imply identical outputs, the reverse is not necessarily true in language modeling, making our approach more sensitive to inconsistencies. calibration. a common approach to assessing the quality of predicted probabilities is through calibration, which evaluates how well predicted probability scores align with membership probabilities in some reference data. in fully calibrated multiclass classifiers, calibration is considered for every class in every prediction. however, evaluating calibration, even in binned probabilities, becomes challenging with a large number of classes, making meaningful binning for every class with representative data difficult. to address this, many studies opt for top-class calibration @xcite , which focuses solely on calibrating the predicted class. although top-class calibration is sufficient to assess the confidence of the prediction, and therefore is frequently used @xcite , full calibration is an essential requirement for a model to be used as a density estimator in multi-class classification, in structured predictions such as sequence predictions in autoregressive text generation, and complex probabilistic generative models with textual components. while measuring full calibration directly is challenging, our approach, which compares the consistency of assigned probabilities to the same expression using different methods, offers alternative means to identify uncalibrated models -inconsistent estimations across different methods imply that at least one of them is miscalibrated. many works measured the calibration of neural models @xcite , generally finding that neural models are poorly calibrated @xcite . in lms, most prior work on calibration has focused on downstream tasks, such as classification and question answering @xcite . studies that specifically addressed language modeling typically restricted their evaluations to top predictions @xcite , top-prediction sets @xcite , or aggregate measures like entropy rates @xcite . these evaluations have primarily examined autoregressive models, consistently finding them to be miscalibrated. while some research on masked language models (mlms) has suggested that they tend to be relatively well-calibrated @xcite , to the best of our knowledge, full-distribution calibration in language modeling was never addressed. language models as density estimators. several studies have interpreted language models as density estimators and explored their probabilis-tic properties from a theoretical standpoint. hahn (2020) proved that some cases cannot be efficiently modeled by transformers. @xcite showed that the requirement that infinite length strings will have zero probability might not be held for all models. however, they defined a theoretical notion of tightness that is satisfied by most common models and guarantees the requirement. @xcite showed that mlms can be interpreted as markov random fields, thus providing probabilities for entire sentences. in contrast, @xcite showed theoretically that decoding based on softmax yields low-rank approximations that are inadequate for capturing the complexity of language distribution. additionally, @xcite presented findings indicating that decoding based on a single embedding vector cannot generate arbitrarily rich distributions. research investigating sampling-based text generation includes the work of @xcite that showed that sampling from an lm distribution results in low-quality text, and that the use of temperature scaling provides a tradeoff between quality and diversity. similarly, @xcite proposed nucleus sampling, avoiding the tail of the distribution, and @xcite presented a sampling scheme based on the expected entropy of natural language. several studies found conflicts between calibration and zero-shot capabilities. @xcite showed that instruction tuning significantly hurts calibration. kalai and vempala (2023) proved that strict calibration, with respect to the training data, must lead to hallucinations. @xcite showed a discrepancy between the cross-entropy loss, used for language modeling, and task-specific losses. other work disregarded the probabilistic nature altogether, filling masks with spans from a reference document @xcite . this comes with the price of losing qualities of probabilistic estimation. ## desired properties of a consistency testing framework our goal is to evaluate whether llms maintain consistency across different estimation orders when calculating the joint probability of a word span. to ensure the robustness and reliability of this evaluation, it must meet the following requirements. versatility. for the designed framework to apply to various models, it should address both mlms and autoregressive models. it should account for task comprehension in autoregressive models, which are not specifically built for filling masks. ## consistency testing framework we present the contests framework, designed to meet the requirements outlined in §4. here, we detail how each requirement is addressed. ## task comprehension in autoregressive models our results show that autoregressive models tend to be less consistent, prompting the question of how much this inconsistency can be attributed to the more challenging task setting. to investigate this, we compare token ranks between mlms and autoregressive models, where lower ranks indicate higher scores. ideally, both models should show similar rank distributions, indicating similar performance. our analysis, shown in figure 3 , indicates that while mlms assign lower ranks to the true first token, flan-t5, and llama 2 models often assign even lower ranks to the second token, suggesting improved predictions. however, llama 2-chat models exhibit significantly higher ranks, indicating poorer performance. 8 in appendix e, we provide a detailed analysis of probabilities assigned to an eos token following the predicted missing word, as an additional measure of task comprehension. we find a positive correlation between models excelling in word rank predictions and those showing good task comprehension (indicated by lower eos ranks). however, the poor performance of llama 2-chat models cannot be solely attributed to a lack of task understanding, as they assign low ranks to eos but high ranks to actual words. across all model types, larger models in each category generally exhibit better task comprehension. ## are there preferable decoding orders? we showed that all examined models exhibit inconsistencies between different orders of estimation of joint probabilities of word spans. this raises the question of whether any of the examined completion orders yields higher scores for the true tokens. to address this question, we analyze the correlation between d i,i+1 and the entropies of the estimated probabilities involved in the joint probability estimations, which we denote with h i , h i+1|i , h i+1 , h i|i+1 . a summary of the correlations is shown in figure 4 . the analysis reveals that: (1) for entropies of predictions when two tokens are masked (i.e., h i , h i+1 ) the correlation with the discrepancy in the corresponding order (d i,i+1 , -d i,i+1 ) is negative; these results suggest that selecting the direction with higher entropy for one-mask prediction and lower entropy for two masks is likely to increase the likelihood of true tokens. in appendix f we provide examples of the effect of decoding order. ## discussion in this work, we investigated the probabilistic consistency of language models (lms) and introduced a novel framework to quantify and explain discrepancies between equivalent estimation orders. our findings indicate statistically significant inconsistencies among 16 out of 18 models on the news dataset and all 18 models on the wikitext dataset. these results highlight significant differences between mlms and autoregressive models, with the latter showing considerably larger inconsistencies and discrepancy variances. the comparable discrepancy distributions between the news and wikitext datasets suggest that exposure in training has little effect on consistency. in contrast, results from the synthetic dataset (see appendix d.2) reveal a different pattern, with 0.3 correlation generative mlm 0.3 0.2 0.1 0.0 0.1 0.2 (a) 0.3 0.2 0.1 0.0 0.1 0.2 0.3 (b) figure 4: entropy and decoding order: correlations between four prediction entropies and the discrepancy d i,i+1 are presented. (a) presents the two entropies associated with predicting the i-th token first. (b) illus- a positive correlation between real data likelihood and overall entropy is termed overconfidence, while a negative correlation is called underconfidence @xcite . our experiments suggest that choosing a decoding direction with higher entropy for single-mask predictions and lower entropy for two masks is expected to yield higher estimated probabilities for the correct word pair. this suggests that overconfidence and underconfidence are influenced by the model type and the number of masked tokens, challenging the anticipated strong link between average likelihood and entropy. consequently, this calls for cautious application of methods to address overconfidence and underconfidence, as models can exhibit both. in conclusion, our investigation into the probabilistic consistency of language models has revealed significant inconsistencies across various model types and datasets. our findings highlight the need for careful interpretation and application of the predicted scores. prior research @xcite argued that modeling the "distribution of language" is a complex task. our findings provide further support for this claim, showing that even high-performing models struggle to generate consistent estimations. robust ranking of structured predictions and similar applications often require consistency in joint probability estimation. however, our research shows that achieving high performance does not necessarily depend on consistency, and vice versa: a model can be perfectly consistent but make inaccurate predictions. therefore, a combination of consistency testing with other performance metrics is essential for thorough evaluation. our analysis exposes inconsistencies and their links to prediction entropies and model-type disparities, offering valuable insights for future research to tackle these limitations effectively.