Evaluating Contextualized Representations of ( S panish) Ambiguous Words: A New Lexical Resource and Empirical Analysis (2025)

Abstract: Lexical ambiguity—where a single wordform takes on distinct, context-dependent meanings–serves as a useful tool to compare across different language models’ (LMs’) ability to form distinct, contextualized representations of the same stimulus. Few studies have systematically compared LMs’ contextualized word embeddings for languages beyond English. Here, we evaluate semantic representations of Spanish ambiguous nouns in context in a suite of Spanish-language monolingual and multilingual BERT-based models. We develop a novel dataset of minimal-pair sentences evoking the same or different sense for a target ambiguous noun. In a pre-registered study, we collect contextualized human relatedness judgments for each sentence pair. We find that various BERT-based LMs’ contextualized semantic representations capture some variance in human judgments but fall short of the human benchmark. In exploratory work, we find that performance scales with model size. We also identify stereotyped trajectories of target noun disambiguation as a proportion of traversal through a given LM family’s architecture, which we partially replicate in English. We contribute (1) a dataset of controlled, Spanish sentence stimuli with human relatedness norms, and (2) to our evolving understanding of the impact that LM specification (architectures, training protocols) exerts on contextualized embeddings.

## introduction state-of-the-art language models (lms) display a remarkable level of formal linguistic competence @xcite . to date, however, we currently lack a precise accounting of the mechanisms underlying lms' fundamental linguistic capabilities. the opacity of model internals has motivated work probing the transformations that inputs undergo as they are processed through various model components @xcite @xcite . this work has focused on lms trained with english-language corpora, with a smaller subset of studies investigating cross-linguistic representations in multilingual models @xcite @xcite or comparing representations across multiple monolingual models @xcite . as others have noted @xcite , an over-reliance on english as a "model language" limits the generalizability of findings, as well as potential applications @xcite . here, we extend interpretability work to spanish, a language spoken by almost 600m people (with almost 500m native speakers) 1 . specifically, we: (1) evaluate the trajectory of ambiguous spanish words' semantic representations within mono-and multilingual lms and (2) identify layers along a model's architecture whose semantic representations best capture human judgments of semantic relatedness. lexical ambiguity-where a given wordform evokes multiple related or unrelated meaningsoffers a unique opportunity to dissociate a word's form from the contextualized, semantic representations that it can take on as it interacts with a given model's architecture. specifically, we can evaluate whether and how lms integrate the surrounding lexical items in a sentence to produce flexible, context-dependent representations. the representation and processing of ambiguous words is also well-studied in humans @xcite @xcite , offering a convenient comparison group. finally, ambiguity appears to pervade language, with some estimates in english positing that more than 80% of words have multiple meanings @xcite ; more frequent words are also more likely to evoke multiple senses @xcite . ambiguity, then, is an important phenomenon to contend with in lms and a useful tool to understand lm representations. we first provide a survey of related work on ambiguity and interpretability in lms (section 2), then present a novel dataset of human relatedness judgments about spanish ambiguous words-in context (saw-c) (section 3). section 4 documents our use of this dataset to empirically probe the representation of ambiguous words in pre-trained spanish lms, focusing first on pre-registered analyses foot_1 of the monolingual beto @xcite . we next systematically compare multiple monolingual spanish and multilingual bert-based models (section 5). we conclude with a discussion of the implications of the current results (section 6), as well as limitations and directions for future work (section 7). ## related work to date, lexical ambiguity has been largely explored within english monolingual models @xcite @xcite . less work is available with models trained on other languages, such as spanish @xcite , and crucial distinctions emerge across studies within this literature: (1) the operationalization of same and different sense conditions, (2) the degree to which sentential context is controlled around the target word, and (3) the extent to which human semantic judgments are collected to set usagebased expectations for the context-dependence and graded quality of ambiguous word meanings @xcite . in spanish, mono-and multilingual bert-based models can capture information about semantic relationships between homonyms and their synonyms @xcite and can approximate words' degree of polysemy @xcite . however, available studies tend to leverage naturalistic sentence stimuli from sense-annotated corpora. while valuable, the variability in token sequence length and target word position within naturalistic sentential contexts may make it challenging to isolate the precise effect of the context's semantic content from the uncontrolled effects of sentence frame @xcite . we thus follow as closely as possible the experimental structure leveraged (for english) in @xcite , creating spanish-language sentence pairs that vary along a single context cue evoking either the same or different sense of the target ambiguous noun (see section 3.1). for our dataset, we document the extent to which context cues presented true minimal pairs (e.g. zero-token-differences across sentence pairs) for the bert-based models we tested (see appendix a.1; using more controlled sentence stimulus design, coupled with empirically collected human benchmarks, prior studies in english have shown that bert-based models' contextualized embeddings capture some-though not all-variance in human similarity @xcite and relatedness @xcite judgments for ambiguous english words. some of this work has also argued that the continuous nature of lm contextualized representations makes them well-suited as models of human word meanings, which are likely graded to some extent @xcite @xcite @xcite @xcite though importantly, may also exhibit marked categoriality compared to lm representational spaces @xcite . finally, another important line of work has used techniques like classifier probes @xcite and activation patching @xcite @xcite to decode the putative functional role of different model components (e.g., layers, attention heads, etc.) in producing observed behavior. to our knowledge, there is little to no work combining these strands of research: i.e., making use of graded human judgments about ambiguous spanish words to trace the dynamics of contextualized representations in pre-trained spanish language models. this is the gap we aim to address. ## human annotation study we created a dataset containing graded human judgments about spanish ambiguous words-in context (saw-c). this process involved first producing and curating materials (i.e., target words and sentences), collecting judgments from native spanish speakers, and validating those judgments for quality and reliability. this study was pre-registered on the open science framework (osf) platform. ## analysis of beto, a pre-trained spanish llm using saw-c as a probe, we conducted multiple pre-registered analyses on how beto-a pretrained monolingual spanish bert-based model @xcite )-represents ambiguous words in context, whether and to what extent these representations are predictive of human semantic representations, and which layers of the model contained the most information @xcite . figure 2 : density plot representing the distribution of mean relatedness judgments for sentence pairs. as expected, word meanings were rated as more related when used in same sense than different sense contexts. ## comparing pre-trained spanish language models section 4 tested several pre-registered hypotheses ( ## discussion we introduced a novel dataset (saw-c) containing human relatedness judgments about ambiguous spanish words in controlled, minimal pair contexts. using this dataset, we probed pre-trained bertbased models' representations of ambiguous words, finding that: 1) lm representations correlate with human judgments but do not match inter-annotator agreement, and exhibit systematic errors; 2) performance varies across layers, with model families showing distinct trajectories of performance; and 3) performance scaled with model size. the systematic underestimation and overestimation errors observed with respect to same vs. dif-ferent sense contexts (figure 5 ) is consistent with past work conducted in english @xcite . one explanation for this is that the initial (static) embedding for an ambiguous wordform might entangle all of its multiple meanings @xcite , which must then be "teased apart" in context-but which might nevertheless persist as "attractors" in subsequent layers. disambiguation could be made even more difficult by the presence of minimal pair contexts @xcite . psycholinguistic research suggests that humans also activate uncued, dominant meanings in certain tasks @xcite ; here, however, humans appeared to distinguish target meanings with relative ease. it is possible that humans represent distinct homonymous (though not necessarily polysemous) meanings along clearly differentiable regions of meaning space @xcite @xcite , which would be consistent with the fact that categorical sense boundaries (same vs. different sense conditions) explained an overwhelming share of human relatedness judgments (section 4.5, trott and bergen (2023)). lastly, this work contributes to expanding the linguistic diversity of both human-annotated benchmarks and interpretability research. although spanish ranks among the most widely spoken lan-guages, it suffers from a surprising dearth of resources (with notable exceptions, baldissin et al. ( 2022 )), pre-trained models, and interpretability research-particularly when compared to english. in one study, a sample of 550 corpora (spanning 22 languages) @xmath0, @xmath1. a wider research perspective-considering varied languages-is critical for ensuring the generalizability of findings. another limitation concerns stimulus generation. the ambiguous nouns included in our dataset were spontaneously produced by native (caribbean) spanish speakers, or selected from previously published lists @xcite , rather than via automated searches. spontaneous production of ambiguous words may (1) overrepresent homonymous words @xcite , and (2) underrepresent words whose multiple meanings have large dominance asymmetries @xcite . in future work, we intend to collect dominance norms for these items. differences in spanish varieties may have also led some items to appear awkward to annotators from different nationalities @xcite . we address this concern by showing strong correlations in relatedness judgments from the three heavily represented groups (chile, spain, and méxico, section 3.4), but other groups' relatedness judgments may not be represented in this sample. finally, our sentence pairs are not naturalistic. our controlled minimal pair design was intentional, enabling us to identify key differences between lm and human representations, e.g., lms display less sensitivity than humans to the manipulation of word meaning across minimal pair contexts (figure 5 ). at the same time, it is important to know whether and to what extent the current results replicate with naturalistic stimuli. thus, we aim to augment saw-c with naturalistic examples of the target ambiguous words. ## ethical considerations this research was conducted with irb approval. all data from human participants has been fully anonymized before analysis and publication. andy zou, long phan, sarah chen, james campbell, phillip guo, richard ren, alexander pan, xuwang yin, mantas mazeika, ann-kathrin dombrowski, et al. 2023. representation engineering: a topdown approach to ai transparency. arxiv preprint arxiv:2310.01405.