The Importance of Context in the Evaluation of Word Embeddings: The Effects of Antonymy and Polysemy (2023)

Abstract: Word embeddings are widely used for diverse applications in natural language processing. Despite extensive research, it is unclear when they succeed or fail to capture human judgements of semantic relatedness and similarity. In this study, we examine a range of models and experimental datasets, showing that while current embeddings perform reasonably well overall, they are unable to account for human judgements of antonyms and polysemy. We suggest that word embeddings perform poorly in representing polysemy and antonymy because they do not consider the context in which humans make word similarity judgements. In support of this, we further show that incorporating additional context into transformer embeddings using general corpora and lexical dictionaries significantly improves the fit with human judgments. Our results provide insight into two key inadequacies of word embeddings, and highlight the importance of incorporating word context into representations of word meaning when accounting for context-free human similarity judgments.

## introduction lexical semantics seeks to provide a cognitive explanation of how word meaning is represented and how semantic relations such as hyponymy, antonymy and synonymy are encoded. vectorspace models are one of the dominant approaches 1 our code and processed datasets are available at https://github.com/bmmlab/lexicalsemantics-eval to studying lexical semantics. in vector-space models, a word is associated with a vector of real numbers called a word embedding, which captures information about word co-occurrences in a document or sentence. each component of this vector corresponds to an abstract feature in an underlying vector space @xcite . the meaning of each word is thus represented by the direction of its word embedding in semantic space. (in this paper we use 'word embeddings' loosely, referring to any vector representation of word meaning using real numbers). word embedding methods are widely used in natural language processing, where they are utilised by machine learning architectures that have achieved impressive performance on a range of applied language tasks @xcite @xcite . vector-space semantics models also have a natural synergy with neuroimaging techniques that measure patterns of voxel activities in response to linguistic stimuli, thus providing an interface between lexical semantics and cognitive neuroscience @xcite . it is therefore of considerable interest to evaluate the performance of these methods in modelling word meanings. vector-space approaches to semantics hypothesise that many aspects of word meaning, including semantic relationships such as synonymy, antonymy, hyponymy, and logical inference, can be efficiently represented by the relative direction of word embeddings in semantic space @xcite . one way to test this hypothesis is to compare the similarity relations between word embeddings with human judgements of word similarity and relatedness @xcite . a high correlation between the similarity structure of word embeddings and human similarity judgements is evidence that the embeddings successfully encode information about word meaning and semantic relationships between words. existing literature evaluating word embeddings against human similarity judgments, however, has typically ignored the implicit context humans use to make these judgements. we hypothesise that this omission is an important factor contributing to the relatively poor performance of word embedding models when evaluated against certain experimental datasets. in this study we focus on two specific semantic phenomena in which the effects of context are most likely to be apparent: antonymy and polysemy. in the case of antonymy, we hypothesise that humans judge the meaning of a word differently when it is presented in the context of a word opposite in meaning. likewise, we hypothesise that humans assess the meaning of polysemous words differently than non-polysemous words due to the need to use contextual information to select the relevant sense. we therefore anticipate an investigation into polysemy and antonymy will be important for understanding the limitations of word embeddings resulting from neglecting context. ## discussion this paper has highlighted significant differences between the manner in which humans and word embedding models represent the meaning of antonyms. while it has long been known that word embeddings perform poorly in predicting antonym similarity judgements @xcite , we have shown the reason for this is that antonyms are given consistently low similarity ratings by humans but moderate to high cosine similarities by embedding models. this effect is consistent across datasets and large in magnitude, reducing correlations by 0.10-0.15, even though antonym or near-antonym word pairs only account for about 10% of each dataset. previous research has sought to rectify the low accuracy of word embedding models on antonyms by adding constraints to artificially pull antonyms further apart in semantic space @xcite . however, we argue that this may be inappropriate, because when humans make similarity judgments between words, they may not be performing an analogous task to computing the cosine similarity between the corresponding embeddings. if this is the case, then the failure of word embedding cosine similarities to match human similarity judgments for antonyms should be interpreted as a limitation of the evaluation method, not a flaw of the word embeddings as a model of word meaning. relatedly, it has been argued that antonyms should have cosine similarities close to the smallest possible value of -1 @xcite . in practise, however, negative cosine similarities occur mostly between unrelated words rather than antonyms, with small absolute values (up to around -0.1 for conceptnet). this is likely because computing cosine similarity averages across all features whether salient or not, thereby computing 'property overlap' @xcite . since antonyms share most features in common, this results in a high cosine similarity. why then do humans rate antonyms as having very low semantic similarity? one potential explanation is that the salience of the semantic features of a word varies depending on the context in which the word is used. this has been observed for human judgements of noun combination tasks @xcite and feature verification tasks @xcite . such findings are consistent with our hypothesis that, when assessing the similarity of two antonyms, humans judge the dimension of meaning in which the two words differ as the most salient, and hence rate overall semantic similarity as low. this would also explain earlier findings that humans rate antonyms almost as similar as synonyms when asked to rate features separately, rather than providing an overall similarity score @xcite . these considerations highlight the need for a new method which enables more consistent and informative comparison between human similarity judgements and cosine similarities for antonyms. unfortunately, in this study we were unable to develop such a method. we experimented with simple methods such as providing both words to the ernie transformer and extracting the contextualised embeddings of each, but this yielded no useful results. further improvements will likely require identifying which particular features are most salient for assessment of antonyms, in line with several previous studies @xcite . in addition, our brief treatment of antonymy has not discussed important issues such as adjectival antonyms or the effects of discourse context on negation @xcite . we leave such considerations for future work. in this study we also found that polysemy significantly reduces the accuracy of word embeddings in describing the similarity of verbs. the dramatic increase in the correlation of ernie embeddings with human judgements when contextual information was incorporated (see figure 4 ) is evidence that the quality of the embeddings is significantly impaired by the inability to properly distinguish different word senses. our results are consistent with a strategy whereby humans assess the similarity of two words using an implicit context that maximises the aspects of meaning they share, ignoring any additional polysemous meanings. this would explain why providing ernie with additional information about context, like parts of speech and example sentences, improves the correlation with human judgments. our results also highlight the value of using contextual information from lexical dictionaries to augment contextual word embeddings. in particular, ernie dictionary sense max embeddings increase the correlation by about 5 percentage points for the full simverb dataset (excluding antonyms), and about 23 percentage points for the most polysemous word pairs. similar increases in correlation were observed from the simpler automated method of aggregating all dictionary senses together, as used in the ernie dictionary word embeddings. we hypothesise that these improvements arise because example dictionary sentences represent common uses of verb, which may reflect the way that humans judge word similarities when asked to judge two words without context. a different approach to control for the effects of polysemy used in several past studies is to ask participants to judge the similarity of words in the context of a specific sentence, thereby allowing for clearer sense disambiguation @xcite @xcite . however, it is difficult to ensure that participants do not simply judge the overall similarity of the sentences, or conversely ignore the context and consider the target words in isolation. furthermore, contextualised word embeddings are more difficult to interpret than static embeddings since they only apply to the word in a specific precise context. given that a concept is typically defined as a mental representation that is reasonably invariant across contexts @xcite , highly context-specific word embeddings are arguably of less value as cognitive models of concepts. as such, we believe there is also value in incorporating contextual information to improve static embeddings of polysemous words, as we have shown can be done by using example sentences from lexical dictionaries. in this paper we have focused on ernie embeddings, as they showed superior performance over competing models that are purely text-based. the performance of conceptnet embeddings provide an additional baseline that also incorporates expert linguistic knowledge. the results corroborates previous studies which found that adding expert knowledge can improve the performance of embeddings derived from word cooccurrence statistics @xcite @xcite . nevertheless, transformer models like ernie use much larger training corpuses and have more parameters than con-ceptnet @xcite , so the fact that con-ceptnet still outperforms all transformer embeddings is a notable finding. however, we do not seek to determine the effect of specific architectural choices or hyperparameters, as such analysis has been conducted in previous studies @xcite @xcite . ## conclusion in this study we have highlighted the problems of ignoring the implicit context in which humans make word similarity judgements. our results show that word meaning is judged in a contextdependent manner which decontextualised word embeddings struggle to adequately capture. future work focused on improving embeddings may require better datasets specifically focused on evaluating how humans rate the similarity of different forms of antonyms. also important is improving the representation of polysemy, which we have shown is possible by combining contextualised embeddings with carefully collated data from dictionaries and other knowledge banks. our analysis has primarily focused on verbs, and so further work focusing on nouns is also needed. overall, much work remains to be done to enhance the ability of vector-space semantic models to describe a wide range of semantic phenomena. ws353 sl999 men mt287 mt771 rw sv citation pmi model, skipgram, glove .71 .43 .78 .69 .51 levy, goldberg, et al. (2015) pmi model, cbow .79 .43 .79 .78 .71 de deyne et al. (2016) skip-gram .70 .34 .73 .66 .61 .40 chiu et al. (2016) count-based, cbow, glove, fasttext .70 .40 .78 wijnholds and sadrzadeh (2019) bert, gpt-2, roberta, xlnet, distilbert .72 .55 .45 bommasani et al. (2020) lsa, lda, cbow, skip-gram, glove, ri, fasttext, bert .71 .49 .79 .71 .48 .41 lenci et al. (2021)