Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events (2025)

Abstract: Can language models reliably predict that possible events are more likely than merely improbable ones? By teasing apart possibility, typicality, and contextual relatedness, we show that despite the results of previous work, language models’ ability to do this is far from robust. In fact, under certain conditions, all models tested—including Llama 3, Gemma 2, and Mistral NeMo—perform at worse-than-chance level, assigning higher probabilities to impossible sentences such as ‘the car was given a parking ticket by the brake’ than to merely unlikely sentences such as ‘the car was given a parking ticket by the explorer’.

## introduction consider the following scenario: clothes is the obvious best answer here, and that most preferred by humans when presented with the options shown above (see @xcite . nonetheless, leaves is also a possible continuation. in fact, @xcite find that experimental participants rate leaves to be significantly more sensible than water, despite the fact that the sentence as a substitute for her pillow, she filled up an old sweater with leaves describes a highly unusual event. this example @xcite hinges specifically on the physical properties of objects and their 'affordances' (i.e., what they can be used to do; @xcite . but it exemplifies a broader behavior that humans exhibit largely unconsciously, namely, using knowledge of the world to distinguish between the impossible and the merely atypical. in this study, we focus on whether language models also have the capability do so. a growing body of work on world knowledge and event understanding investigates whether language models can select the most likely or plausible of a set of possibilities (e.g., @xcite @xcite ; see subsection 2.1). but this work often conflates different ways that the 'incorrect' options can be incorrect; some are impossible, but others are simply less typical. while finding that models are better or worse at identifying likely events is in itself important, it is only one component of robust event understanding. in fact, any use of language models in practice, whether as part of larger systems or directly as chat assistants, is likely to involve novel and unexpected situations. thus being able to tell the difference between the impossible and the merely unlikely is crucial, especially in critical domains where the use of language models has been suggested, such as medicine (see, e.g. @xcite @xcite . for this reason, we argue that it is important to also assess the extent to which language models are able to differentiate between possible but improbable events and impossible ones. we call this the sherlock holmes task after the famous epigraphic quote @xcite . in this study, we evaluate language models on a specific version of the sherlock holmes task, asking whether they are able to reliably assign a higher probability to sentences describing possible (but in some cases, atypical or unlikely) events (e.g., the car was given a parking ticket by the delinquent; @xcite than their impossible equivalents (e.g., the car was given a parking ticket by the stamp). we draw on experimental stimuli from previous research investigating how humans process such sentences @xcite . prior work shows that language models assign impossible sentences with words semantically related to their context higher probabilities than equivalent sentences with unrelated words @xcite . thus, we also include a further adversarial component, specifically considering cases where impossible sentences include related words (e.g., the car was given a parking ticket by the brake) and possible sentences include unrelated words (e.g., the car was given a parking ticket by the explorer). in contrast to previous work suggesting that language models have relatively good world knowledge and event understanding capabilities (e.g., @xcite , we find that both event atypicality and semantic relatedness lead to significant drops in performance. in the most highly adversarial case, where we compare the probabilities assigned to possible but atypical sentences with unrelated words to those assigned to impossible sentences with related words (e.g., the car was given a parking ticket by the explorer vs. brake), we find that all language models tested perform at or below chance-that is, they assign the impossible sentences a higher probability half or more of the time. we further find that this effect does not disappear with scale-in fact, smaller models often do better than their larger counterparts. together, these results suggest that language models may rely on typicality and semantic relatedness cues when making predictions rather than actual world knowledge, and thus, that any previously-ascribed event understanding capabilities are far from robust. ## the present study as has been discussed, the effects of possibility, typicality, and contextual semantic relatedness on language model probability have each been investigated individually. however, these are all intertwined-all else being equal, we should expect that typical events are also possible, and that words describing typical and possible events are more likely to be related to their semantic contexts than words describing atypical and impossible events. in this study, therefore, we aim to disentangle the effect of each by investigating the effects on prediction of cases where these cues conflict. to do this, our version of the sherlock holmes task uses a minimal pairs paradigm (see, e.g., @xcite @xcite where the language model is presented with two sentences that differ by a critical word that makes the sentence either possible or impossible. in order to allow any language model to carry out our task (not just large or finetuned models, see @xcite , we follow previous work such as blimp @xcite and directly compare the probability that a language model assigns to each sentence to determine the more probable one. as shown in in this study, as in previous work comparing possible and impossible sentences in language models @xcite , we look at impossibility arising from animacy violations. an animacy violation occurs when an inanimate entity has an event role that requires animacy. for example, in the cure for the disease was discovered by the..., the discoverer needs to be animate, and since stamps are inanimate, completing the sentence with stamp is an animacy violation. thus, when a sentence with an animacy violation is interpreted literally, it refers to an impossible state of affairs. all english stimuli were drawn from a human study carried out by @xcite , and all mandarin stimuli from a study carried out by @xcite . english minimal pairs are exactly as shown in ## experiment 3: statistical reliability 6.1 introduction in addition to replicating @xcite , we also make several novel findings in experiments 1-2. performance drops across the board when the possible critical word is atypical rather than typical, as well as when it is semantically unrelated rather than related. additionally, performance drops when the impossible critical word is related rather than unrelated. while it may be unsurprising that language models appear to predict typical words to be more probable than atypical ones and semantically related words to be more probable than semantically unrelated words (especially given, e.g., @xcite , we show for the first time that this has a significant impact on their ability to assign higher probabilities to possible sentences than impossible ones. ## general discussion previous work has shown that the predictions of language models are highly correlated with event typicality @xcite . in our study, we similarly find that language models are sensitive to large differences in typicality, for example when distinguishing between impossible events and typical events. however, we also find that they perform significantly worse when distinguishing between impossible events and lesstypical but still possible events. thus, our results suggest that there is ample room for potential improvement at this end of the scale. a larger and perhaps more striking result is the effect of semantic relatedness. a language model will become worse at distinguishing between a possible and impossible event when the possible event involves a critical word that is semantically unrelated to its preceding context, when the impossible critical word is semantically related, and especially when both of these are the case. combined with previous work showing that all else being equal, language models will often tend to predict that words that are semantically related to the context are more likely than unrelated ones @xcite , these results suggest that language models at least in part rely on this relatedness (in addition to typicality) as a cue for which words are more probable. the presence of such a heuristic would be in line with other work showing that the strong performance of language models and other artificial intelligence systems can often be explained by them learning simpler 'shortcuts' or other heuristics that correlate (often but not always) with the task at hand (see, e.g., @xcite @xcite @xcite @xcite @xcite @xcite @xcite . as previously noted, such a surface-level heuristic is likely adaptive. generally, words that are semantically related to their context are likely to be more appropriate sentence continuations than unrelated words-indeed, all the critical words in the typical sentences are semantically related to their context. this may at least in part explain why such a heuristic exists, not just in language models, as our study suggests, but also in humans (see, e.g., @xcite @xcite @xcite . crucially, this effect of contextual semantic relatedness does not disappear with increased model scale-in fact, while models trained on more data appear to become better at detecting larger differences in typicality, they also appear to increasingly rely on semantic relatedness to the context as a cue. thus, this issue is not likely to disappear as language models continue to be trained on ever more data-in fact, it may get worse. ## conclusions while contemporary language models can tell apart possible and impossible events @xcite , we find that this breaks down when the possible events are not highly typical, and when semantic relatedness is not a cue to event possibility. this suggests that much of language models' reported performance at reflecting event probability in previous work may be explained by language models' sensitivity to differences in typicality and contextual semantic relatedness.