A mbi S tory: A Challenging Dataset of Lexically Ambiguous Short Stories (2025)

Abstract: Word sense disambiguation is the task of selecting a word’s applicable word sense in a given context. However, ambiguous texts may lack the information necessary to disambiguate words completely, resulting in multiple word senses with varying degrees of plausibility. We design a dataset around this premise: Our samples consist of 4–5 sentence short stories, where the sentence with the word to be disambiguated is itself ambiguous and surrounding sentences only contain indirect clues towards the more plausible word sense. We collect annotations from humans who rate the plausibility of a given word sense on a scale from 1–5. In total, our dataset contains 19,701 human word sense annotations on 1,899 stories. We investigate the performance of large language models on our data and find that many poorly correlate with human judgments. We also find that fine-tuning on our data can increase performance.

## introduction lexical ambiguity describes the presence of multiple senses being applicable to the same word and has been argued to be a functional property of any efficient communicative system @xcite . indeed, psycholinguistic studies have shown that humans rapidly process such ambiguities in context (see e.g. @xcite . computational approaches to resolving lexical ambiguity in context, commonly referred to as the task of word sense disambiguation (wsd), have been studied at least since the 80s @xcite . in the past decade, research has moved from fixed sense inventories to graded assignments of senses (see §2), following the predominant psycholinguistic view that senses are not (strictly) categorical (for a discussion, see @xcite . in parallel with developments in theory formation and corresponding benchmarks, there have been immense technical developments: specifically, large language models (llms) such as @xcite , @xcite and llama-3 @xcite have not only dominated the nlp community but also shown "superhuman" performance in wsd tasks @xcite ). yet, lexical ambiguities still pose difficulties in tasks such as question answering, natural language inference, and machine translation @xcite . reasons for this include underspecified language, meaning that lexical ambiguities are not always resolvable (see @xcite , as well as differences in background knowledge that can lead to divergent interpretations of context (see e.g. @xcite . in this paper, we lay the foundations for investigating these difficulties. specifically, we build a dataset of lexically ambiguous word usages (see figure 1 for an example) and collect multiple judg-ments on the plausibility of different sentence readings under varying discourse-level contexts. our dataset consists of 4-5 sentence short stories where the fourth sentence contains a homonym foot_1 that is used in such a manner that the sentence is ambiguous when read in isolation. hints towards the preferred reading are provided in surrounding sentences that manipulate the plausibility of word senses by introducing additional details. a correct disambiguation of the homonym thus requires a higher-level understanding of the story. furthermore, as different contexts will change the plausibility of word senses to varying extents, our dataset contains both stories that are perceived as non-ambiguous and various degrees of ambiguity. based on the collected data, we examine the following research questions: rq1 how does context affect human plausibility judgments, and when do annotators disagree? rq2 how well do judgments by llms align with humans, and when are they different or fail? rq3 can an llm be trained in order to increase agreement with human judgments? rq4 how does additional context affect llm performance and human agreements? while numerous wsd tasks operate on the sentence-level, a considerable body of research has also been dedicated to document-level wsd. for instance, datasets for 'all-words' wsd tasks, where the objective is to classify the word sense of every word in the text, commonly consist of longer documents @xcite . other datasets similarly consist of large tagged corpora @xcite . comparatively little research has been dedicated to the idea of multiple word senses having varying levels of applicability. @xcite treat wsd as a ranking task, where word senses are given percentages based on their applicability. other research tackles wsd as a multilabel classification task @xcite or introduces an ordinal scale to the wic task @xcite . we combine ideas from these tasks and create the first dataset in which the plausibility of different word senses are judged independently given varying amounts of context beyond the sentence of a target word. ## data collection stories in our dataset consist of 4-5 sentences: three describing the situation ('precontext'), one ambiguous sentence, and optionally an ending. our collection of stories is a multi-step process in which we collect the precontexts and ambiguous sentences (forming a 'setup'), as well as the endings and the plausibility annotations separately. 3 we present our data collection approach in the following. first, we extract a set of homonyms to use in our short stories ( §3.1). then, we let humans write ambiguous sentences and prepend a precontext to them to form the setup ( §3.2). we then collect ending sentences written by humans ( §3.3), and finally collect human plausibility ratings for each 'sample', i.e. the combination of word sense and story ( §3.4). for each step, we collect data via prolific and require users to pass a qualification test before proceeding to the study to ensure data quality. in addition, we manually check each story for adherence to our guidelines and offensive content, and remove all low-quality submissions. during the collection of plausibility ratings, we include a question from the qualification test as an attention check. we filter annotators who fail this check. ## data analysis based on the data collection described in section 3 we want to analyze under which circumstances people view multiple readings as plausible and when disagreements occur (rq1). before that, we discuss statistical properties of the collected data ( §4.1) and present a preliminary analysis of the effects of context and word senses on plausibility judgments ( §4.2). ## computational experiments following the analysis of human annotations in section 4, we next conduct preliminary experiments on our dataset to answer the remaining research questions outlined in the introduction. we first describe the task setup, models and evaluation metrics used across experiments ( §5.1) and then address our research questions regarding the alignment between lms and human judgments ( §5.2) and on the possibility of training llms to increase this alignment ( §5.3). in context of these experiments, we also analyze the effect of endings as additional context on llm predictions. ## conclusion we introduced ambistory, a collection of 1,899 short stories with human plausibility ratings for word senses. each story contains a lexical ambiguity with multiple alternate endings designed to imply different word senses. we investigated how human plausibility judgments are influenced by story context, and found inter-annotator agreement to be affected by the frequency of word senses as well as relations between them, among other reasons. in computational experiments, we found llm predictions often do not match human-perceived plausibilities, for instance, because of model biases towards high-frequency senses and distracting key words in the story context. in general, we found models to perform worse when no ending to a story is provided, whereas human agreement remains stable. this may indicate that llms lack the common sense to 'fill in the gaps' in stories with only minimal disambiguating information. we believe that our dataset provides a useful basis for model development and testing, as exemplified by the possibility of improving the performance of two llms via fine-tuning. in fact, ambistory also serves as the benchmark for a shared task at semeval 2026 (task 5). we encourage the community to use ambistory to study and model human perception of lexical ambiguity beyond individual sentences.