Polysemy through the lens of psycholinguistic variables: a dataset and an evaluation of static and contextualized language models (2024)

Abstract: Polysemes are words that can have different senses depending on the context of utterance: for instance, ‘newspaper’ can refer to an organization (as in ‘manage the newspaper’) or to an object (as in ‘open the newspaper’). Contrary to a large body of evidence coming from psycholinguistics, polysemy has been traditionally modelled in NLP by assuming that each sense should be given a separate representation in a lexicon (e.g. WordNet). This led to the current situation, where datasets used to evaluate the ability of computational models of semantics miss crucial details about the representation of polysemes, thus limiting the amount of evidence that can be gained from their use. In this paper we propose a framework to approach polysemy as a continuous variation in psycholinguistic properties of a word in context. This approach accommodates different sense interpretations, without postulating clear-cut jumps between senses. First we describe a publicly available English dataset that we collected, where polysemes in context (verb-noun phrases) are annotated for their concreteness and body sensory strength. Then, we evaluate static and contextualized language models in their ability to predict the ratings of each polyseme in context, as well as in their ability to capture the distinction among senses, revealing and characterizing in an interpretable way the models’ flaws.

## introduction the meaning of individual words taken in isolation can look unambiguous. take for instance the word book. if encountered on its own, it evokes the image of an object made of sheets of paper bound together. however, when put in context, such as in the phrase 'explain the book', it clearly does not refer to that same concrete object -rather, it denotes its immaterial, abstract content. a word like book is called a polyseme @xcite @xcite . polysemes are easily understood when contrasted with monosemes (words with only one possible interpretation, like leaf ) and homonyms (words that can that can take two completely unrelated interpretations, like bat): polysemes can take different interpretations -also called senses -which are related among them and that follow patterns that also apply to other words (so-called regular polysemy; @xcite . in the case of book, for instance, the pattern is an alternation between a concrete object and an abstract meaning, which also characterizes other words like newspaper or painting. in computational linguistics and natural language processing (nlp), a large body of work has looked at polysemy. mainly, the aim is that of finding out to what extent the distinctions between different senses can be captured by current models -either with a theoretical focus @xcite @xcite @xcite @xcite or in applied tasks (word sense disambiguation @xcite @xcite @xcite @xcite . however, as pointed out in @xcite ; @xcite , a fundamental conceptual limitation has characterized approaches to polysemy in nlp so far. namely, they have (almost) exclusively assumed a traditional view of polysemy, the so-called sense enumeration view @xcite , which has been shown to afford only limited explanatory power. according to this theory, each sense of a polysemous word like book should be given a separate, dedicated representation -like the meanings of distinct words like leaf and curtain. this is the way in which knowledge graphs like wordnet @xcite or babelnet @xcite , the resources that are most typ-ically used as the golden standard for polysemy in nlp, are structured: for book, we find multiple entries -e.g. <noun.@xmath0.artifact>. however, this view is challenged from a large body of work in cognitive psychology and psycholinguistics. experimental approaches have rather proposed the so-called one representation view of polysemous nouns: different senses are not assumed not to be represented differently, but just to be different aspects or facets of the same semantic representation (among others, @xcite @xcite @xcite for comprehensive reviews). as a reflection of this theoretical gap, the datasets typically used for the evaluation of computational models of language at capturing polysemy are built according the sense enumeration view. lack of diverse evaluation approaches not only leaves a large amount of potential evidence untapped, but also obscures important insights that could emerge by taking a different perspective. we concur with @xcite ; haber and poesio (2023) that, to investigate in depth the ability of current computational models of semantic to capture polysemy, it is necessary to go beyond the sense enumeration view. to this aim, we propose to take a hybrid approach. we break down regularized patterns of polysemy -from the sense enumeration view -in terms of psycholinguistic variables like concreteness -inspired by the one representation view. in this framework, the variation happening when varying the interpretation of book @xmath1.@xmath2.@xmath3, possibly accompanied by an increase in its audibility (hearing). we build on previous work showing how hard distinctions between senses emerge from (and are contained by) complex representations of words @xcite @xcite . what we add is an explicit specification (i.e. in terms of psycholinguistic variables) of how sense alternations in polysemy take place. from previous approaches in nlp that rely on similarities in latent vector spaces @xcite @xcite , we retain the notion of using continuous measures of similarity/distance -i.e. a 'soft' approach to senses: however, while dimensions of language are not interpretable from a cognitive point of view, ours are. importantly, this framework has been previously successfully applied to model how the brain processes fine-grained lexical meaning variations @xcite . since our framework revolves around cognitively motivated semantic features, it aims at fostering research connecting computational and cognitive models of language -with the broader goal of allowing to gain insights on how similar the two are, which is a fundamental open question in the field @xcite @xcite . starting from this theoretical approach, in the current work we present two main contributions. first, we describe how we created an original dataset of examples of lexical polysemy. for each polyseme, the dataset provides ratings provided by human subjects in terms of concreteness and of sensory strength (with separate ratings for sight, hearing, touch, smell, taste) for phrases where the different senses are evoked. our dataset is carefully crafted by controlling for psycholinguistic variables, with the aim of allowing its use both for in silico and cognitive experiments. secondly, we evaluate static and contextualized language models on their ability to predict the ratings provided by humans and to distinguish among different senses of polysemous words. we hypothesized that contextualized language models would consistently outperform static language models. our results confirm our prediction, but they also show that there is large room for improvement in overall accuracy for contextualized language models too -indicating that polysemy is still a challenging semantic phenomenon for language models to capture. we publish the dataset together with the code foot_0 . ## models a fine-grained semantic phenomenon like polysemy has proven particularly challenging to capture for language models. older approaches (so-called static language models; bommasani et al., 2020), were particularly unsuited to face its subtleties @xcite . static language models learn fixed semantic representations for words, abstracted from specific contexts of usage. this made it hard to successfully model meaning of words in context -and consequently context-dependent phenomena such as polysemy @xcite . the more recent language models, called contextualized language models @xcite ), should be in principle better equipped to face the challenge of polysemy. they are trained to create semantic representations of words which are context-specific. when focusing broadly on nlp tasks requiring to consider contextual semantic knowledge (e.g. natural language generation, inference, relation classification), contexualized models are clearly able to reach impressive performance, outperforming static models @xcite . however, when zooming in through the lens of extremely specific semantic knowledge such as polysemy, synonymy, hypernymy and categorization, the picture changes: contextualized models appear to capture such phenomena only to a modest extent, leaving much room for improvement @xcite @xcite . to provide a better picture with regards to this, we use four models, including both static and contextualized language models @xcite . in the following we will briefly describe each model, and how the vectors for the polysemous nouns in context were extracted from each one of them. in appendix a we report an analysis measuring how similar the representations are across the models: the phrases that compose our dataset make notable differences emerge across different types of models, converging with our prediction and sense discrimination results (see sections 5.1, 3, 4). ## evaluation having obtained the vectors for each verb-noun phrase, we measure to what extent it is possible to learn to predict the ratings obtained from human subjects. we use a cross-validated procedure, with a ridge regression model (α is cross-validated within the train set among 0.01, 0.1, 1, 10, 100, 1000). we employ a linear model, an efficient choice given the low number of data points (100; @xcite . for crossvalidation, we use monte carlo cross-validation @xcite -which entails randomly sampling train and test sets many times (in our case, 20), in order to obtain a reliable average statistics. for the evaluation, we use two measures, explained below. ## limitations and future directions the main limitation of our study is the size of the dataset, and the fact that we focus on only one case of regular polysemy. future work could expand this dataset by considering more, and more specific types of polysemy that can be modelled within a similar framework -cases like chicken where another variable, taste, can explain sense alternations (animal vs taste; @xcite . another interesting direction could be investigating to what extent language models and human cognition align while processing these polysemes (e.g. using brain data; cf. @xcite .