Word Sense Distance in Human Similarity Judgements and Contextualised Word Embeddings (2020)

Abstract: Homonymy is often used to showcase one of the advantages of context-sensitive word embedding techniques such as ELMo and BERT. In this paper we want to shift the focus to the related but less exhaustively explored phenomenon of polysemy, where a word expresses various distinct but related senses in different contexts. Specifically, we aim to i) investigate a recent model of polyseme sense clustering proposed by Ortega-Andres & Vicente (2019) through analysing empirical evidence of word sense grouping in human similarity judgements, ii) extend the evaluation of context-sensitive word embedding systems by examining whether they encode differences in word sense similarity and iii) compare the word sense similarities of both methods to assess their correlation and gain some intuition as to how well contextualised word embeddings could be used as surrogate word sense similarity judgements in linguistic experiments.

## introduction homonymy, the linguistic phenomenon of a word taking on a different meaning based on its context, such as match in (1), is often used to showcase one of the advantages of context-sensitive word embedding techniques such as elmo @xcite , bert @xcite and gpt-2 @xcite over their traditional wordvector counterparts such as word2vec @xcite and glove @xcite , which are unable to encode context-dependent meaning. in this paper we want to shift the focus to the related but less exhaustively explored phenomenon of polysemy. we speak of polysemy when a word takes on different distinct but related senses given its context @xcite , such as school in the various contexts of (2). foot_0 specifically, we aim to investigate a recent model of polyseme sense clustering proposed by @xcite , suggesting that similarity differences in polysemic senses could lead to a grouping in their representation in the generative lexicon @xcite , addressing and attempting an explanation for processing differences observed within the seemingly homogeneous group of polysemes. through a range of surveys we collect word sense similarity judgements for a set of polysemes to provide empirical data for an investigation of word sense clustering as proposed by ortega-andrés and vicente. we then aim to extend the linguistic evaluation of context-sensitive word embeddings by examining whether their contextualised encodings of polysemes show signs of word sense grouping, and whether these groupings correlate with the patterns observed in the human judgements. if this is the case, contextualised word embeddings could be used as surrogate word sense indicators in linguistic experiments. ## method in order to generate the clearest results possible for investigating potential distances between different polysemic sense interpretations, we decided to use custom samples instead of resorting to corpus samples in this study. by creating the samples ourselves, we can construct contexts that invoke a certain polysemic sense as clearly as possible, and we can create sentence pairs that combine any of the different interpretations in order to have annotators judge their similarity directly. in addition, a preliminary investigation of contextsensitive polyseme representations obtained from elmo revealed that factors such as i) the position of the target expression in the sentence, ii) its syntactic function and iii) the overall sentence length all significantly influence the resulting embedding and might overshadow the differences in encoding stemming from interpretation differences. 3 designing custom samples helps us to control for these factors. ## conclusion while our results are difficult to collapse as they survey different types polysemes, there are some overarching conclusions to be drawn from the data collected in this study. first of all, we provide empirical evidence that readers are indeed sensitive to differences between polysemic word senses. polysemic target expressions in contexts designed to invoke the same sense interpretation are consistently rated as highly similar, while similarity ratings of cross-sense pairs receive ratings on a spectrum ranging from highly similar to significantly dissimilar. it thus seems that some sense interpretations are perceived to be more similar to each other than others, providing support for a similarity-based grouping of word senses like the one as proposed by @xcite . in some cases, distances in sense interpretations correspond to intuitive groupings of senses, but the collected judgements also reveal a notion of gradedness that usually is not assumed to be present in canonical samples (see e.g. lau et al. ( 2014 )). given these observations, we see merit in exploring a more structured representation of polysemic sense, since a fully under-specified, single-entry approach would be insufficient to fully account for them. having investigated only one target expression for a small set of systematic polysemes, we acknowledge that more empirical research is needed to investigate potential patterns within polysemy types or in the much larger set of irregular polysemes in order to determine whether there are any systematic effects -or whether each and every polysemic expression requires its own idiosyncratic representation structure. the graded word sense judgements obtained through our data collection however also indicate to us that a categorical approach to word sense disambiguation (wsd) such as implied by a number of recent models (e.g. levine et al. (2019); wiedemann et al. (2019); blevins and zettlemoyer (2020) ) might be geared more towards the distinction of homonymic ambiguity and fall short of capturing the full spectrum of polysemic sense alterations. current approaches focusing on graded word sense similarity and word sense shifting (see for example @xcite ) on the other hand might produce new insights in mapping out the intricacies of polysemic word sense interpretation and representation. concerning the encoding of polysemic sense in the contextualised word embeddings of elmo and bert, we do observe differences in the representation of polysemic expressions invoking different sense interpretations which could indicate the encoding of context-specific information, but similarities between word embeddings do not consistently correlate with the collected human similarity judgements. while the raw embeddings thus cannot directly be used to distinguish polysemic senses to the same degree as human judgements do, they still could contain word sense information that requires non-linear functions in order to be accessed. failing to provide conclusive answers in this respect, we hope that future work will help to determine to what extend -and how exactlypolysemic sense is represented in contextualised embeddings to shed more light into the black box processes that improve so many nlp systems.