Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge (2020)

Abstract: Understanding context-dependent variation in word meanings is a key aspect of human language comprehension supported by the lexicon. Lexicographic resources (e.g., WordNet) capture only some of this context-dependent variation; for example, they often do not encode how closely senses, or discretized word meanings, are related to one another. Our work investigates whether recent advances in NLP, specifically contextualized word embeddings, capture human-like distinctions between English word senses, such as polysemy and homonymy. We collect data from a behavioral, web-based experiment, in which participants provide judgments of the relatedness of multiple WordNet senses of a word in a two-dimensional spatial arrangement task. We find that participants’ judgments of the relatedness between senses are correlated with distances between senses in the BERT embedding space. Specifically, homonymous senses (e.g., bat as mammal vs. bat as sports equipment) are reliably more distant from one another in the embedding space than polysemous ones (e.g., chicken as animal vs. chicken as meat). Our findings point towards the potential utility of continuous-space representations of sense meanings.

## introduction a key challenge in natural language understanding is grasping the range of meanings that a word can take as a function of linguistic and non-linguistic context. successful linguistic comprehension involves constantly resolving lexical ambiguity of this nature @xcite . this re-use of word forms by speakers -relying on listeners to choose the appropriate meaning depending on context -confers language with higher communicative efficiency than it would otherwise have, and may pose a solution to the problem of limited memory @xcite . the re-use of word forms also allows speakers to extend the lexicon to new communicative situations, for example to refer to new objects, entities or processes using existing words in the language @xcite @xcite . how these context-specific meanings are understood and represented pose critical open questions regarding the lexicon. following the conventions of lexicographic resources like natural language dictionaries, variation in word meaning is often treated in a categorical fashion: lexical types contain clusters of related meanings, or word senses. under this treatment, word tokens (instances of a word type that are used in context) can be categorized into these word senses. different sub-types of lexical ambiguity are thus reflected in different relations between word senses: pairs of word senses are polysemous if they are semantically related @xcite , or homonymous if this is not the case @xcite @xcite . for example, using "bottle" to refer to a container as well as to the liquid it contains is an example of polysemy. by contrast, using "bank" to refer to a riverside or a financial institution -two semantically unrelated meanings -constitutes homonymy. polysemous relations are often regular @xcite , such that many word types exhibit the same alternation (e.g., the container-for-contents relation is also exemplified by "box" and "glass"); such patterns can also be generalized to new words @xcite . because of variation in the perceived relationship between senses across language users, theories in psycholinguistics assert that humans treat polysemy and homonymy as falling onto a continuous gradient @xcite . this is corroborated by psycholinguistic experiments which show that both adults and children represent and process polysemous and homonymous senses differently @xcite @xcite @xcite . however, these theories of word sense representation lie in stark contrast with many lexicographical approaches, most notably wordnet @xcite . although wordnet contains a vast store of word senses-and has historically been considered the gold standard ontology of word senses @xcite -it does not encode relations among those senses, and thus does not distinguish between polysemy and homonymy. models based on distributional semantics can help bridge the gap between psychological theories of word senses and existing lexicographical resources. contextualized word embeddings (cwes), especially bert @xcite , which is derived from a transformer-based architecture @xcite , have the potential to capture fine-grained distinctions in relatedness between word senses, because they offer a continuous measure of relatedness between individual uses of words in context (lake and murphy, 2020). while fine-tuned bert models are known to perform exceptionally well in word sense disambiguation (wsd) tasks @xcite @xcite , there are still open questions of whether bert representations capture fine-grained distinctions in sense relatedness. in the present work, we evaluate whether contextualized embeddings from bert capture relationships between word senses similar to english speakers. we collect human judgments of the relatedness among different senses of 32 english words through a web-based two-dimensional spatial arrangement task @xcite . we then compare the experimental data to bert vectors for the same set of words. this is done through extracting and analyzing bert embeddings for word tokens in the semcor corpus @xcite , which has been annotated with word sense identifiers from wordnet. assessing whether relatedness in bert representations corresponds with human judgments of sense relatedness is an important test of whether cwes can be used to develop more realistic computational models of word meanings. ## background the distributional hypothesis proposes that a word's meaning can be represented by the lexical context in which it occurs @xcite . this insight underlies the success of computational models that represent words as continuous-valued vectors representing their surrounding lexical context. latent semantic analysis @xcite , and neural models like word2vec @xcite encode information about all of a word's senses in a single vector. by contrast, newer models include contextualized embeddings: vector space representations of specific word uses (tokens), reflecting the context of their use. these richer word representations from contextualized embeddings have allowed researchers to ask which aspects of linguistic knowledge are incorporated in their vector spaces (see @xcite for a review; see also @xcite . for example, one of the most notable discoveries from this line of work suggests that a sentence's dependency parse tree can be reconstructed from bert embeddings @xcite @xcite . however, despite progress investigating how bert encodes syntactic information, relatively little work has explored how it encodes semantic information such as the relations between word senses. while older models like word2vec may encode some aspects of word sense @xcite , we focus here on transformer architectures @xcite , which may be able to find sensedistinguishing context words at larger distances, and thus perform better for wsd. bert-based approaches have led to state of the art performance on word sense disambiguation tasks @xcite @xcite , but approaches analyzing the model's representation of word senses remain exploratory @xcite . evidence from @xcite shows that bert embeddings for the same word in a large text corpus are largely clustered based on their meanings, but the authors do not investigate the model's ability to encode canon-ically homonymous or polysemous relationships. @xcite note that bert embeddings for the same word type vary as a function of the position in the sentences they occur in. lake and murphy (2020) assess bert's capacity for homonym resolution by comparing relatedness of words that are similar to a homonym in highly constraining contexts. ettinger (2020) points out the need for more psycholinguistic diagnostics of neural language models like bert, and compares bert to human data from tasks such as commonsense inference. our work falls under this framework, focusing on comparing human judgements of word sense relatedness to bert representations. ## methods we conducted a metalinguistic experiment (vs. a processing task testing implicit knowledge) where participants used a web interface to assess the relatedness of wordnet senses in a two-dimensional spatial arrangement task. we then obtained bert embeddings for word types in the semcor corpus @xcite and compared them to the experimental data, looking at both cosine distance in the embedding space as well as the accuracy of a sense classifier using bert's contextualized word embeddings as input. all stimuli, code and visualizations are available at https://osf.io/fm78w . ## results first, we verified that bert can discriminate between senses in semcor comparably to other test datasets. we then compared relatedness estimates derived from distances in the bert embedding space to the aggregate relatedness measures from the experiment. we then conduct two exploratory analyses, the first comparing the relatedness of homonymous and polysemous sense pairs, and the second examining the utility of pairwise sense confusion as a measure of relatedness. ## discussion we investigated the ability of an artificial neural network model that represents word tokens with contextualized word embeddings to capture human-like relations among english word senses. on a subset of word types from the semcor corpus, we reproduced previous word sense disambiguation results. we then showed that these same bert embeddings capture a significant amount of information regarding the relationship between word senses, and are able to at least partially reproduce human relatedness judgments. an exploratory analysis revealed that bert-based measures of the relatedness between pairs of homonymous senses are much lower than for pairs of polysemous senses, matching human intuitions for the same set of senses. pairwise confusion from the sense classifiers provided a slightly better predictor of human judgments of sense relatedness compared to distance in the embedding space, although this may be unsurprising given that pairwise confusion reflects some additional degree of supervision. analyzing the error rates of classifiers trained on bert embeddings of the stimuli, we also provided evidence that their results are more accurate for homonymous sense pairs compared to polysemous sense pairs. following in the approach of comparing bert vectors to experimental data from human participants @xcite , the present research addresses how bert represents word senses, bridging the gap between human and computational models of lexical ambiguity resolution. recent progress showing bert's performance on word sense disambiguation @xcite @xcite indicates that bert-based models perform better than past approaches, and our work specifies that the representations they use to accomplish this are relatively consistent with human intuitions. existing exploratory work aims to analyze how bert represents word senses @xcite ), but we systematically evaluate these claims with both wordnet senses and human data. our work corroborates claims by lake and murphy (2020) that bert captures relations between homonymous senses. indeed, poor performance in discriminating polysemous pairs of words senses suggests that more work needs to be done on capturing polysemous relations; one possibility is to explore how bert could be fine-tuned to capture relations exemplified by regular polysemy. one limitation from these findings is that the dataset only covers 32 word types, which we hope to address in future work. more generally, our findings suggest that a continuous measure of sense relatedness derived from neural language models could potentially be used to augment existing lexicographic resources like wordnet, in this case providing a measure of relatedness between senses. we encourage other researchers developing sense ontologies, especially those found through word sense induction, to consider representing the relationships among word senses in addition to discovering new sense inventories. ## conclusion through comparing results from a behavioral experiment with data from contextualized word embeddings, we demonstrate that new transformer-based neural network architectures may reflect human intuitions about the relationships between word senses, at least in english. exploratory analyses suggest that these measures of relatedness between sense pairs reflect a distinction between polysemous and homonymous relationships between word senses, but that these models are much more effective in discriminating homonymous sense pairs than polysemous ones. by demonstrating basic levels of consistency with human judgements, we hope to stimulate further research that combines contextualized word embedding techniques with discrete, symbolic ontologies to develop a more cognitively informed model of the lexicon.