What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length (2025)

Abstract: When comparing the linguistic capabilities of language models (LMs) with humans using LM probabilities, factors such as the length of the sequence and the unigram frequency of lexical items have a significant effect on LM probabilities in ways that humans are largely robust to. Prior works in comparing LM and human acceptability judgments treat these effects uniformly across models, making a strong assumption that models require the same degree of adjustment to control for length and unigram frequency effects. We propose MORCELA, a new linking theory between LM scores and acceptability judgments where the optimal level of adjustment for these effects is estimated from data via learned parameters for length and unigram frequency. We first show that MORCELA outperforms a commonly used linking theory for acceptability—SLOR (Pauls and Klein, 2012; Lau et al., 2017)—across two families of transformer LMs (Pythia and OPT). Furthermore, we demonstrate that the assumed degrees of adjustment in SLOR for length and unigram frequency overcorrect for these confounds, and that larger models require a lower relative degree of adjustment for unigram frequency, though a significant amount of adjustment is still necessary for all models. Finally, our subsequent analysis shows that larger LMs’ lower susceptibility to frequency effects can be explained by an ability to better predict rarer words in context.

## introduction are the probabilities provided by language models (lms) compatible with theories of linguistics and human language processing? this is a fundamental question that has implications in fields from psycholinguistics to natural language processing applications, and requires understanding of how to relate lm probabilities with quantities associated 1 our code is available at https://github.com/ lindiatjuatja/morcela . it is silly for one to sing in the shower. figure 1 : english sentences with linguistic acceptability scores reported by @xcite . participants where asked to rate sentences on a scale from 1 (least acceptable) to 7 (most acceptable), whose scores were then normalized by participant to a mean of 0 and variance of 1. scores shown are averaged across participants. with human language processing. in this work, we consider the relationship between lm probabilities and human judgments of linguistic acceptability, and investigate how lm probabilities should be treated when comparing them to human acceptability judgments. acceptability judgments are speakers' reported perceptions about the well-formedness of utterances, which are often elicited by asking questions such as "how natural/acceptable/grammatical is this utterance?" @xcite . these judgments are typically reported in binary or numerical form @xcite @xcite , and they are collected through a variety of annotation tasks, such as binary classification, likert scale scoring, or ranking @xcite . examples of acceptability judgments, from @xcite , are provided in figure 1 . judgments such as these play a central role in linguistics, where they are used to motivate and evaluate theories of natural language syntax @xcite . in order to relate lm probabilities with any human behavioral measure, we need a linking theory between them to make the two quantities comparable. although the existence of a relationship between probability and acceptability has been subject to debate @xcite @xcite , an influential proposal by @xcite hypothesizes that sentence-level lm probabilities largely reflect linguistic acceptability, but are influenced by word frequency and sentence length in ways that humans are largely robust to. thus, a linking theory between lm probabilities and human acceptability scores should somehow control for these factors. out of the various functions they used to control for length and frequency, @xcite find that the syntactic log-odds ratio (slor, @xcite served as the best linking theory between acceptability and probabilities from n-gram lms and simple recurrent lms @xcite . slor controls for unigram frequency and length in a uniform manner across lms by dividing the probability of the sentence under the lm by the joint unigram frequency, then averaging over all tokens to control for length. however, it is not clear a priori that these model-agnostic transformations are the appropriate ones to link lm probabilities and acceptability judgments, nor that these transformations should be held constant across different lms. in this work, we first show that the modelagnostic transformations in slor may severely underestimate lm probability correlations with human acceptability judgments. we propose a new linking theory, magnitude-optimized regression for controlling effects on linguistic acceptability (morcela), a parameterized linking theory where the effect sizes of length and unigram frequency are automatically estimated from human acceptability judgment data. our experiments first show that morcela significantly outperforms slor in predicting human acceptability judgments from probabilities calculated by transformer lms @xcite from the pythia @xcite and opt @xcite families. our results show a relationship with scale, where larger models exhibit greater correlation with human judgments compared to smaller models in the same family, using the same linking theory. examining the estimated optimal parameter values of morcela reveals that larger models are more robust to length and unigram frequency effects, and thus their probabilities require a lower degree of adjustment when comparing them to human acceptability judgments. we show in particular that larger models' lower reliance on unigram frequency is driven by their improved ability to predict rare words given appropriate context. these results demonstrate that when comparing probabilitybased lm acceptability judgments to those of humans, controls for factors like length and unigram frequency should be made on a per-model basis. ## morcela: acceptability judgments from lm probabilities to evaluate lms according to their ability to predict human judgments of linguistic acceptability, we need a linking function that takes as input the probability of the sentence under a lm and outputs an acceptability score, which we then correlate with human judgments. this linking function should account for the effects of length and unigram frequency, as noted by @xcite , which impact lm probabilities in predictable ways that may cause them to deviate from human judgments. specifically, longer sentences will be assigned a lower probability than to any strictly smaller prefix of the sentence and a sentence containing a rare token will likely have a lower probability compared to one containing a more frequent one, all else being equal. we propose morcela (magnitude-optimized regression for controlling effects on linguistic acceptability), a parameterized linking function given by and taking β = -b/a and γ = c/a. morcela improves upon the syntactic log-odds ratio (slor, @xcite , widely used as a linking function for predicting acceptability judgments @xcite @xcite @xcite @xcite , by allowing for arbitrary linear relationships between the variables p and u via the parameters β and γ. we argue here that morcela mitigates overcorrections for length and frequency effects from slor, and our main experiment shows that morcela scores are significantly more correlated with z-normalized human acceptability ratings than slor scores. ## main experiment how much does optimizing the relative effect of length and unigram frequency via morcela impact fit of lm acceptability scores to human judgments? to investigate this, we correlate the lm acceptability scores from morcela to gradient human judgments across lms of varying sizes and compare the resulting correlation to two baseline linking functions: log probabilities and slor. ## results we first compare morcela to two baseline linking functions (raw log probabilities and slor), then assess the impact that parameterizing either unigram frequency or length has on correlation with human judgments. ## ability to predict infrequent tokens explains impact of unigram frequency as we have just shown, as models get larger and better at predicting human acceptability judgments, the smaller the relative importance of unigram frequency becomes. one possible explanation for this is that models that are better predictors of acceptability are so because they are better at predicting more infrequent tokens in context, and as a result are more robust to the effect of unigram frequency. the intuition behind this is that while some tokens may be very rare within the distribution of the entire corpus (e.g. names of chemical compounds), they may be relatively frequent given a specific context (e.g. within a scientific article). thus, if a model is better able to predict such cases of tokens by utilizing context, they should no longer need to be controlled as heavily for unigram frequency. to test this hypothesis, we first need a way to quantify the ability of a lm to predict rarer tokens in context. we operationalize this by correlating the lm's conditional log likelihood over instances of tokens with the unigram log-probability of those tokens. as our corpus to calculate conditional log likelihood over, we use a portion (∼ 100 million tokens) of the test set of the pile @xcite , the training corpus of the pythia models. for unigram log-probabilities, we use counts from the entire training split (as in our calculations of p u (s)). we do this for all sizes of models in the pythia suite. to calculate conditional log likelihood, we use a sliding window of the max sequence length of the pythia models (2048 tokens) with a stride of 1024. as before, since pythia was trained without a bos token, the log likelihood of the first token in a document is not considered. if our hypothesis-that lms better at predicting rarer tokens in context are more robust to unigram frequency effects-holds, we would expect that models that are worse at predicting human acceptability judgments have lower conditional log likelihood for more infrequent tokens compared to models that are better fits to judgments, and vice versa. in other words, if we were to plot conditional figure 5 : the slope of lm conditional log likelihood per token vs. token unigram frequency plotted against the correlation (average correlation using 5-fold cv) and optimal unigram coefficient β values of morcela (fit using all the data). each point represents a single pythia model, numbered from smallest to largest. models that are better at predicting rarer tokens in context should have a lower slope value, which we find is correlated with higher correlation with human judgments and generally lower β. we find that this prediction largely holds, as shown in figure 5 . in general, as models get better at predicting rarer tokens in context, i.e. a higher log likelihood on aggregate for rarer tokens and thus a less positive slope between log likelihood and token unigram frequency, they show greater correlation with human judgments as well as increasingly lower values of β. ## discussion our method morcela lies within the context of a large body of work that examines how well (neural) language models reflect human-like language processing. we now discuss our work's relation to previous studies evaluating lms as psycholinguistic subjects and the application of parameterized linking theories in this setting, and compare our results to related findings from comparisons of lm surprisal to reading times. ## conclusion in this work, we reexamine the assumptions made by commonly used linking theories such as slor in evaluating lms' fit to human acceptability judgments. we introduce a new linking theory, morcela, which parameterizes controls for length and unigram frequency, and learns the optimal values for these controls from acceptability data via linear regression. by adding two simple, interpretable parameters, morcela drastically improves correlation with human judgments compared to slor, showing that slor greatly underestimates correlation between lm and human acceptability scores. an inspection of the optimal values of these parameters shows that the magnitude of correction for confounds in slor overestimate the impact of frequency and length in transformer lms, and that this overestimation is greater as models grow larger. finally, we show that lms' robustness to unigram frequency effects can be explained by their ability to predict rarer words in context. our findings suggest that evaluations of probability-based lm acceptability judgments should account for model-specific qualities with respect to factors like frequency and length, and that doing so reveals that lms may be better correlated with human judgments than previously thought. however, there is still a sizable gap between the maximum correlation between lms and human judgments and the correlation between annotators. future work could investigate what additional factors/transformations could lead to closer correspondence between lms and humans, and further integrate these insights into training more cognitively plausible models.