Patterns of Polysemy and Homonymy in Contextualised Language Models (2021)

Abstract: One of the central aspects of contextualised language models is that they should be able to distinguish the meaning of lexically ambiguous words by their contexts. In this paper we investigate the extent to which the contextualised embeddings of word forms that display multiplicity of sense reflect traditional distinctions of polysemy and homonymy. To this end, we introduce an extended, human-annotated dataset of graded word sense similarity and co-predication acceptability, and evaluate how well the similarity of embeddings predicts similarity in meaning. Both types of human judgements indicate that the similarity of polysemic interpretations falls in a continuum between identity of meaning and homonymy. However, we also observe significant differences within the similarity ratings of polysemes, forming consistent patterns for different types of polysemic sense alternation. Our dataset thus appears to capture a substantial part of the complexity of lexical ambiguity, and can provide a realistic test bed for contextualised embeddings. Among the tested models, BERT Large shows the strongest correlation with the collected word sense similarity ratings, but struggles to consistently replicate the observed similarity patterns. When clustering ambiguous word forms based on their embeddings, the model displays high confidence in discerning homonyms and some types of polysemic alternations, but consistently fails for others.

## introduction capturing lexical ambiguity has been a driving factor in the development of contextualised language models (e.g. @xcite . evaluating their performance, much of the focus has been on homonymy, a variety of multiplicity of meaning exemplified by word forms such match in (1), whose different meanings are entirely unrelated. and indeed, research such as @xcite @xcite ) has achieved promising results on using contextualised language models to disambiguate homonyms. but homonymy is not the only form lexical ambiguity can take @xcite @xcite : in polysemy, word forms like school in (2) can elicit different distinct but related senses @xcite . polysemy is in fact much more common than homonymy, and most words can be considered polysemous to some degree @xcite @xcite -however, the ability of contextualised language models to capture this phenomenon has been studied much less. in this paper, we shift the focus to polysemy proper, and investigate how well contextualised language models capture graded word sense similarity as observed in human annotations. it is important to carefully distinguish polysemy from homonymy, as multiplicity of meaning and multiplicity of sense have almost opposing semantic effects: while a homonym needs to be interpreted correctly in order to arrive at the correct meaning of an utterance, polysemes refer to different aspects or facets of the same concept, and might not even need to be completely specified to elicit a good-enough interpretation of what is meant @xcite @xcite @xcite . evidence from psycholinguistic studies supports this distinction, indicating that polysemes are processed very differently than homonyms @xcite @xcite . a growing body of work recently also has started to challenge the uniform treatment of polysemic sense postulated by traditional theories such as the generative lexicon @xcite @xcite , and put forward proposals of a more structured mental representation of polysemic sense @xcite . using co-predication tests, studies such as @xcite ; @xcite ; @xcite show that not all polysemic interpretations can be co-predicated, and that some sense interpretations lead to zeugma: 1 (3) # they took the door off its hinges and walked through it. joining a range of recent work seeking to provide empirical data of graded word use similarity, in @xcite we recently released an experimental small-scale dataset of graded word sense similarity judgements for a highly controlled set of polysemic targets to investigate the notion of structured sense representations. analysing results for ten seminal english polysemes, we observed significant differences among polysemic sense interpretations @xcite @xcite and found first evidence of a distance-based grouping of word senses in some of the targets. in this paper we present a modified and extended version of this initial dataset to i) provide additional annotated data to validate previous observations, and ii) include new targets allowing for the same alternations as the initial set. this expansion enables us to carry out analyses not possible with the original dataset, including iii) investigating similarity patterns and polysemy types, iii) performing a detailed analysis the correlation between human judgements and sense similarities predicted by contextualised language models, and iv) obtaining preliminary insights on how well their 'off-the-shelf' representations of word sense can be used to cluster different sense interpretations of polysemic targets. the new data confirms previous observations of varying distances between polysemic word sense interpretations, and provides tentative evidence for similarity patterns within targets of the same type of polysemy. these patterns can to some degree be replicated by the similarities of embeddings extracted from bert large, opening up potential avenues of research utilising contextualised embeddings to proxy costly human annotations for 1 example from @xcite the collection of a large-scale repository of finegrained word sense similarity. the collected data is publicly available online. foot_0 ## methods the data for this study was created by revising and extending the dataset of contextualised word sense similarity presented in @xcite , and contains annotated sample contexts for different sense interpretations of 28 english polysemic nouns. ## results in our analyses we focused on three different aspects. first, we computed graded similarity and acceptability ratings based on the collected annotations, and investigated how these ratings relate to traditional distinctions of lexical ambiguity and recent proposals of a more structured representation of polysemic senses, especially considering the patterns of word sense similarities observed across different target words allowing for the same set of sense alternations. we then analysed how the different contextualised language models' target embeddings correlate with either of the human annotations, and to what degree they replicate the patterns of word sense similarity observed in the human annotations. lastly, we analysed the contextualised embbedings themselves, for a preliminary assessment of how well these 'off-the-shelf' word sense encodings fare in clustering samples based on their sense interpretation. ## related work most work focused on the word sense disambiguation capabilities of contextualised language models investigates the classification of homonyms with clear-cut evaluation criteria (see @xcite for a recent summary). polysemy proper adds another dimension of difficulty, since related senses can be perceived to be very similar to one another, and some form of graded relatedness criterion in necessary to properly evaluate model predictions @xcite . datasets that capture graded similarity judgements usually do so for word pairs in isolation -often intended to evaluate static word sense embeddings @xcite , or are conducted on a small number of items @xcite . notable exceptions are the word in context (wic) dataset by pilehvar and camacho-collados (2019), which contains over 7,000 sentence pairs with an overlapping english word, but was annotated based on a binary classification task. the cosimlex dataset @xcite on the other hand collects graded similarity judgements -but does so for different, related targets. in parallel to our work, @xcite recently conducted an investigation of 32 polysemic and homonymic word types extracted from the semcor corpus @xcite by comparing the distances between a selection of cross-sense samples as determined by participants arranging them in a 2d spatial arrangement task. in line with our results, they reported polysemic senses to be rated significantly more similar to one another in both the human annotations and bert base embeddings, and found a strong correlation between the cosine distance of bert sense centroids and aggregated relatedness judgements. in a similar approach, @xcite recently presented a novel dataset of 112 polysemes and homonyms, for each of which a number of highly controlled sentence pairs were annotated for similarity of use. while their data is very similar to ours, one noticeable difference can be found in the distribution of cross-sense polyseme ratings. based on our samples, different polyseme interpretations were rated to be mostly quite similar still, but their data displays an almost even distribution of similarity scores assigned to them. a closer inspection of the targets used in their study revealed two main factors that are likely to have contributed to this difference. firstly, while all of our targets were specifically chosen to be regular, metonymic polysemes, a large part of trott and bergen's polysemic targets are examples of metaphoric polysemy. re-analysing their data after distinguishing these different branches of polysemy might help to further investigate their respective effects. and secondly, we noticed the use of compound nouns (i.e. ice cone vs traffic cone) to disambiguate target words. considering polysemy as a form of under-specified language use, we argue that these expressions might undermine the raison d'être of polysemy proper as they over-specify the ambiguous target -but highlight an interesting additional facet of this research. ## conclusion we present a revised and extended dataset of graded word sense similarity for 28 seminal, lexically ambiguous word forms. the collected data supports previous observations of significant similarity differences between polysemic interpretations and led to the discovery of tentative patterns of word sense similarity for certain types of alternations. while more work on this matter will be needed before definite conclusions can be drawn, both of these observations can be taken as additional evidence against linguistic models proposing a uniform treatment of polysemic senses. we also used the collected data to test how well different 'off-the-shelf' contextualised language models can predict human word sense similarity ratings. among the tested models, especially bert large seems to capture nuanced word sense distinctions in a similar way to human annotators, and to some degree is capable of grouping sense interpretations by their contextualised embeddings. we hope to further expand the dataset presented in this paper to create a novel, more complex benchmark for the word sense disambiguation (wsd) task. in this endeavour, contextualised language models could be used to automatically detect relevant target word forms, and to collect corpus samples exhibiting specific targets and interpretations to be rated by human annotators for a more realistic, real-world test bed.