Linking language model predictions to human behaviour on scalar implicatures (2025)

Abstract: We explore the behaviour of language models on adjectival scales in connection with negation when prompted with material used in human experiments. We propose several metrics extracted from the model predictions and analyze those metrics in relation to human data as well as use them to propose new items to be tested in human experiments.

## outline in this paper, we describe various experiments that explore the relationship between scalar implicatures and language modeling scalar implicatures are inferences such as the one in (1). here, difficult and impossible form a scale. the inference is such that using the weaker item on that scale (1-a) leads to the negation of the stronger item (1-b). in the first part of the paper, we aim to elicit an implicature (1-b) from the language model as the next token prediction. to do this, we prompt the model with the original sentence (containing a weak scalar item) followed by the repetition of the initial portion of the same sentence and a negation (2). we base this experiment on the material presented in @xcite and explore the output as well as the underlying processing of prompts that include negation. the project is difficult. this means that the project is not (prompt) next, we introduce metrics that are based on the model behaviour. these metrics prioritize lexical items that are likely to co-occur in the top predictions of the language model. we use these metrics to automatically extract new pairs of adjectives. from the obtained list of pairs we then select such pairs where the adjectives are on a scale and repeat the negation experiment using corpus data and both established and new adjective pairs. we analyze the model behaviour for both sets of pairs, focusing on the desirable crossing pattern of adjectival activation when the model encounters the negation. in the last part of the paper, we use the proposed model-related metrics in connection with human experiments. on one hand, with the help of one of the metric variants we can explain some part of the variability on the human ratings for scalar implicatures and negative strengthening. on the other hand, we discover that almost all the scales used in human experiments receive low values according to our metrics. we then extract new adjective pairs from the model and propose a set of scalar pairs to use in future human experiments that would be more evenly distributed from the perspective of the language model. foot_0 2 language models and negation ## model metrics and human behaviour in order to evaluate the obtained metrics on human data, we extract all the values as described above for the scalar adjectival pairs that were used in human experiments @xcite . we then compute the correlations between our metrics and human data. this reveals that the most helpful metric is scale by strong: that of the high ranking of the weak item in those sentences where the strong item is ranked high (e.g. how often is attractive present in the top ten prediction of those sentences that have stunning in top ten predictions). this metric strongly negatively correlates with human scalar implicature rating (-0, 51) and positively correlates with human rating for negative strengthening (0, 34). it must be noted, however, that our results are limited by the range of scales for which human data is available: as described above, for the metrics we have calculated how often the two adjectives of the pair co-occur within the top ten predictions of the language model. we have discovered that within this metric, although values between zero and one are possible (see appendix), the actual values for the original experimental items (see figure 4 ) are very low (less than 0.6). figure 4 illustrates that both linear and quadratic correlations are plausible (@xmath0.51 for linear and @xmath1.47 for quadratic correlation) given the data from the previous experiments due to the limited range of values of the scale by strong metric. the extension of the two correlation curves demonstrates that obtaining the experimental results for items that lie on the right spectrum of the scale by strong metric is essential for making a decision about the validity of either correlation. one parameter to take into account is the upper boundedness of the scale. it has been shown to be the main predictor for the human scalar implicature and negative strengthening ratings from a collection of linguistic features @xcite ; another relevant but less strong predictor is semantic similarity). although statistical evaluation of non-upperbounded scales is not possible due to the insufficient amount of data, the two categories are shown on figure 5 . it can be observed that all the human ratings of bounded scales are higher than these of non-bounded scales and at the same time almost all the language model scores for those scales are very low. the question whether this accidental or systematic can be studied by exploring the linguistic properties of the scales that receive high scores by the language model metrics. in relation to this it is also worth exploring fig. 5 that depicts the possible correlations of human scalar ratings with the model scores scale by weak. although the statistical analysis produces very low correlation values (@xmath2.05 for linear and @xmath3.08 for quadratic correlation), the visual inspection reveals that most of the values for the model scores are below 0.12, so the correlation analysis can not be reliably performed on the basis of this data. since scale by weak score is very low for most of the scales, the harmonic mean score that takes into account both scale by strong and scale by weak also does not provide a significant correlation for the available set of data. similarly to the case depicted on fig. 5 the set of data does not exclude the possibility of discovering such a correlation given a different set of data. since the value of the proposed metrics are very low for most of the items found in the experimental literature, we suggest that more experiments should be performed with different scales that are better distributed according to those metrics. as described above, we propose a list of pairs that satisfies these criteria from the model perspective and leave if to the linguists to pick the best experimental items from it. this list is provided in the appendix and the suggestions are marked in bold. ## discussion in this paper we described several experiments related to scalar adjectives. first of all, we could establish that despite the overt copying behaviour of the analyzed language model the underlying activation exhibits desirable trends. second, we have proposed model-based metrics to evaluate the scales and experimented with scales that receive high ratings according to these metrics. we could achieve a slight increase in desirable behaviour (crossing pattern between the activations of the weak and the strong items), although these results provided a lesser increase than we had expected. we believe that the observed underlying behaviour of the language model while processing the negation opens new perspectives in adjusting the model predictions by magnifying the desired trends. finally, we have explored the connection between the human experimental results and proposed metrics. we could observe that one of the metrics (scale by strong) can be used to partially explain the variability in human ratings of the scalar implicatures associated with different scales. at the same time time we could see that the scales used in human experiments have very low scores on all the proposed language model metrics. in this light we suggest new material for further experiments. this type of work opens a field of automatic generation of possible experimental material as well as running the experiments using the language model before transferring them to the lab. this can lead to a significant decrease in time needed for experimental design as well as to lowering the cost of running various versions of the same experiment since some relevant design problems can be already observed and corrected on the level of the language model experiments.