Assessing Polyseme Sense Similarity through Co-predication Acceptability and Contextualised Embedding Distance (2020)

Abstract: Co-predication is one of the most frequently used linguistic tests to tell apart shifts in polysemic sense from changes in homonymic meaning. It is increasingly coming under criticism as evidence is accumulating that it tends to mis-classify specific cases of polysemic sense alteration as homonymy. In this paper, we collect empirical data to investigate these accusations. We asses how co-predication acceptability relates to explicit ratings of polyseme word sense similarity, and how well either measure can be predicted through the distance between target words’ contextualised word embeddings. We find that sense similarity appears to be a major contributor in determining co-predication acceptability, but that co-predication judgements tend to rate especially less similar sense interpretations equally as unacceptable as homonym pairs, effectively mis-classifying these instances. The tested contextualised word embeddings fail to predict word sense similarity consistently, but the similarities between BERT embeddings show a significant correlation with co-predication ratings. We take this finding as evidence that BERT embeddings might be better representations of context than encodings of word meaning.

## introduction polysemy is a form of lexical ambiguity which occupies a unique middle ground between monosemy -word forms with exactly one interpretationand homonymy -word forms associated with two or more completely unrelated interpretations. unlike monosemes, polysemes can evoke different interpretations, but unlike homonyms, polysemic sense interpretations are thought to be closely related to each other @xcite . it is commonly this work is licensed under a creative commons attribution 4.0 international licence. licence details: http: //creativecommons.org/licenses/by/4.0/ . although the distinction is clear in theory, distinguishing monosemy, polysemy and homonymy in practice proves exceedingly difficult: at what point are interpretation nuances pronounced enough to speak of two different word senses? is the coercion of word sense a manifestation of polysemic sense alteration or a context effect on a monosemic word form? do word senses related through metaphor qualify as polysemes or are their interpretations a form or homonymic ambiguity? traditionally, co-predication tests are used to provide a linguistic means to answer these questions and attempt a classification of word sense interpretations into one of the three categories. in co-predication tests, two interpretations of a word form are simultaneously invoked by the context. if this renders a felicitous construction (see example 1), the two interpretations are considered to evoke the same sense or meaning of the word; if the reading is infelicitous (example 2) they are considered to be derived from two different word meanings. based on a range of experiments finding that homonyms seem to be processed differently than polysemes @xcite @xcite , the prevailing understanding of co-predication is that it is rendered felicitous if the different sense interpretations are activated simultaneously and can be shifted between without additional processing costs. co-predication is thought to lead to infelicitous sentences if the different activations are not activated automatically, and cognitive effort is involved in updating the assumed meaning of a word. these hypotheses informed a number of linguistic models to define different mental representations of homonymic meaning and polysemic sense, respectively. the generative lexicon @xcite @xcite for example postulates individual lexicon entries for different interpretations of a homonym, while all sense interpretations of a polyseme are represented by a single under-specified entry and therefore do not require any processing cost for sense switching. recently, a growing body of work however came to challenge a unified, under-specified representation of polysemic sense (see @xcite @xcite . @xcite for example report that their experiments indicate that the processing of irregular polysemes resembles homonymic meaning alterations more than the sense alterations in regular polysemes, while an ongoing series of co-predication studies (e.g. @xcite @xcite @xcite show that not all polysemic senses can be co-predicated either, and that the co-predication of some polya recent model of polyseme sense clustering proposed by @xcite tries to explain why certain polyseme senses lead to infelicitous co-predication by suggesting that polyseme senses might be grouped based on their similarity. according to their grouping, closely related senses are thought to form co-activation packages that remain active for a while, allowing for cost-free sense shifting and therefore felicitous co-predication. distantly related sense interpretations on the other hand would not co-activate and therefore require cognitive effort to be changed, much like homonynic meaning alterations. the difficulty in assessing this hypothesis is the unavailability of a ready reference of contextualised word sense similarity for polysemes. to mitigate this, we collected human annotated data on a number of different measures of polysemic sense similarity to empirically investigate the correlation between sense similarity ratings and co-predication acceptability judgements. specifically, we use crowdsourcing to collect i) graded co-predication acceptability judgements, ii) explicit (meta-linguistic) word sense similarity judgements, iii) word class similarity ratings, and iv) determine the similarity in a target word's contextualised embeddings derived from different models. if word sense similarity indeed governs co-activation and therefore co-predication acceptability, we expect similarity judgements to be a strong predictor for acceptability judgements. conversely, if copredication acceptability is a representative test of the mental processing of lexically ambiguous items, we expect acceptability judgements to be a strong predictor of similarity judgements and reliably tell apart homonyms from polysemes. we find that sense similarity appears to be a major contributor in determining co-predication acceptability, but that co-predication judgements tend to rate less similar sense interpretations equally as unacceptable as homonym pairs, effectively misclassifying these instances. we therefore argue that these findings provide both, a) support for a more hierarchical representation of polysemic sense based on sense similarity, and, b) an additional, empirically founded argument against copredication as a prevailing test for distinguishing polysemy and homonymy. finally, the tested contextualised word embeddings fail to predict word sense similarity consistently, but the similarities between bert embeddings show a significant correlation with co-predication ratings. ## method in order to evaluate both, i) the hypothesis that polysemic senses might form groupings based on their similarity, and ii) the prevalence of copredication as a linguistic test for the distinction between homonymy and polysemy, we collect three human annotated measures of word sense similarity together with five word sense similarity proxies derived from computational methods. we investigate how well these different metrics distinguish homonyms from polysemes, and to what degree they can predict one another. in order to achieve a fair comparison of the different measures, we defined a fixed set of target words, sense interpretations and contexts to be used in all experiments. ## results we report the collected data in four steps: firstly, we inspect to what degree the different metrics and combinations thereof can predict whether a pair of target sense interpretations is polysemic or homonymic. we then investigate the correlation between the three collected annotation metrics, and ## conclusion the data collected in this study allows for a number of observations about the role of word sense similarity in the processing of homonyms and polysemes. on the one hand, graded co-predication acceptability ratings are shown to be less able to tell apart samples of homonymic and polysemic sense pairs than explicit sense similarity ratings. this supports the growing collection of studies indicating that co-predication might not be as suited a tool to distinguish different types of lexical ambiguity as traditionally assumed. on the other hand, the collected judgements of word sense similarity indicate that polyseme sense pairs mis-classified by co-predication acceptability are overall less similar to each other than other sense pairs, and significantly so than same-sense interpretations. this to some degree vindicates co-predication as a linguistic test, suggesting that rather than distinguishing homonyms form polysemes per se, it might be a coarse indication of the underlying word sense similarity. our results also provide support for recent hypotheses suggesting that polyseme representation in the mental lexicon cannot be fully underspecified. during data collection, annotators rated some polysemic sense interpretations to be significantly less similar to each other than other sense pairs, and even rated some of the polyseme cross-sense co-predication samples as unacceptable. this indicates that the interpretations of polysemic words might be grouped based on their similarity, and only grouped interpretations are available for cost-free sense shifting and felicitous co-predication. because only a single target word per type of systematic polysemy was tested here, we cannot ascertain whether sense groupings are idiosyncratic or systematic across target words of a certain polysemy type. data for an analysis of this question can however easily be obtained by repeating our experiments with a larger set of target words. in a similar vain, we also recommend an in-depth analysis of irregular or metaphorical polysemes, which were omitted in this data collection effort. lastly, investigating the suitability of contextualised language models as proxies for human word sense similarity judgements, we find that the tested contextualised embeddings fail to predict word sense similarity consistently, but that the similarities between bert embeddings show a significant correlation with co-predication acceptability ratings. we take this finding as evidence that bert might create better encodings of complex contexts than encodings of actual word meaning, as it seems to perform well in determining whether contexts can be felicitously combined without consistently determining the similarity of word senses from these contexts first. we strongly encourage further research into determining the exact lexical semantic information available in bert encodings in order to shed more light on this issue.