Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy (2021)

Abstract: This paper presents a multilingual study of word meaning representations in context. We assess the ability of both static and contextualized models to adequately represent different lexical-semantic relations, such as homonymy and synonymy. To do so, we created a new multilingual dataset that allows us to perform a controlled evaluation of several factors such as the impact of the surrounding context or the overlap between words, conveying the same or different senses. A systematic assessment on four scenarios shows that the best monolingual models based on Transformers can adequately disambiguate homonyms in context. However, as they rely heavily on context, these models fail at representing words with different senses when occurring in similar sentences. Experiments are performed in Galician, Portuguese, English, and Spanish, and both the dataset (with more than 3,000 evaluation items) and new models are freely released with this study.

## introduction contrary to static vector models, which represent the different senses of a word in a single vector @xcite , contextualized models generate representations at token-level @xcite , thus being an interesting approach to model word meaning in context. in this regard, several studies have shown that clusters produced by some contextualized word embeddings (cwes) are related to different senses of the same word @xcite , or that similar senses can be aligned in cross-lingual experiments @xcite . however, more systematic evaluations of polysemy (i.e., word forms that have different related meanings depending on the context @xcite ), have shown that even though cwes present some correlations with human judgments @xcite , they fail to predict the similarity of the various senses of a polysemous word @xcite . as classical datasets to evaluate the capabilities of vector representations consist of single words without context @xcite or heavily constrained expressions @xcite , new resources with annotations of words in free contexts have been created, including both graded similarities @xcite or binary classification of word senses @xcite . however, as these datasets largely include instances of polysemy, they are difficult to solve even for humans (in fact, the highest reported human upper bound is about 80%) as the nuances between different senses depend on non-linguistic factors such as the annotator procedure or the target task @xcite @xcite . in this paper, we rely on a more objective and simple task to assess how contextualized approaches (both neural network models and contextualized methods of distributional semantics) represent word meanings in context. in particular, we observe whether vector models can identify unrelated meanings represented by the same word form (homonymy) and the same sense conveyed by different words (synonymy). in contrast to polysemy, there is a strong consensus concerning the representation of homonymous senses in the lexicon, and it has been shown that homonyms are cognitively processed differently than polysemous words @xcite . in this regard, exploratory experiments in english suggest that some cwes correctly model homonymy, approximating the contextualized vectors of a homonym to those of its paraphrases (lake and murphy, 2020), and showing stronger correlation with human judgments to those of polysemous words @xcite . however, as homonyms convey unrelated meanings depending on the context, it is not clear whether the good performance of cwes actually derives from the contextualization process or simply from the use of explicit lexical cues present in the sentences. taking the above into account, we have created a new multilingual dataset (in galician, portuguese, english, and spanish) with more than 3,000 evaluation items. it allows for carrying out more than 10 experiments and controlling factors such as the surrounding context, the word overlap, and the sense conveyed by different word forms. we use this resource to perform a systematic evaluation of contextualized word meaning representations. we compare different strategies using both static embeddings and current models based on deep artificial neural networks. the results suggest that the best monolingual models based on transformers @xcite can identify homonyms having different meanings adequately. however, as they strongly rely on the surrounding context, words with different meanings are represented very closely when they occur in similar sentences. apart from the empirical conclusions and the dataset, this paper also contributes with new bert and fasttext models for galician. 1 section 2 presents previous studies about word meaning representation. then, section 3 introduces the new dataset used in this paper. in section 4 we describe the models and methods to obtain the vector representations. finally, the experiments and results are discussed in section 5, while section 6 draws some conclusions of our study. ## related work a variety of approaches has been implemented to compute word meaning in context by means of standard methods of distributional semantics @xcite @xcite . as compositional distributional models construct sentence representations from their constituents vectors, they take into account contextualization effects on meaning @xcite @xcite . however, these approaches often have scalability problems as their representations grow exponentially with the size of the sentences. therefore, the datasets used to 1 dataset, models, and code are available at https:// github.com/marcospln/homonymy_acl21/ . the rise of artificial neural networks on natural language processing popularized the use of vector representations, and the remarkable performance of neural language models @xcite led to a productive line of research exploring to what extent these models represent linguistic knowledge @xcite . however, few of these works have focused on lexical semantics, and most of the relevant results in this field come from evaluations in downstream tasks. in this regard, @xcite found that clusters of bert embeddings @xcite seem to be related to word senses, while @xcite observed that clusters of polysemous words correspond to different senses in a cross-lingual alignment of vector representations. probing lstms on lexical substitution tasks, @xcite showed that these architectures rely on the lexical information from the input embeddings, and that the hidden states are biased towards contextual information. on an exploration of the geometric representations of bert, @xcite found that different senses of a word tend to appear separated in the vector space, while several clusters seem to correspond to similar senses. recently, @xcite evaluated the performance of bert models on several lexical-semantic tasks in various languages, including semantic similarity or word analogy. the results show that using special tokens @xcite or @xcite ) hurts the quality of the representations, and that these tend to improve across layers until saturation. as this study uses datasets of single words (without context), typelevel representations are obtained by averaging the contextualized vectors over various sentences. there are several resources to evaluate word meaning in free contexts, such as the stanford contextual word similarity @xcite and cosimlex (armendariz et al., 2020), both representing word similarity on a graded scale, or the word-in-context datasets (wic), focused on binary classifications (i.e., each evaluation item contains two sentences with the same word form, having the same or different senses) @xcite . these datasets include not only instances of homonymy but mostly of polysemous words. in this regard, studies on polysemy using transformers have obtained diverse results: @xcite found that bert embeddings correlate better with human ratings of co-predication than with similarity between word senses, thus suggesting that these representations encode more contextual information than word sense knowledge. nevertheless, the results of @xcite indicate that bert representations are correlated with human scores of polysemy. an exploratory experiment of the latter study also shows that bert discriminates between polysemy and homonymy, which is also suggested by other pilot evaluations reported by lake and murphy (2020) and @xcite . our study follows this research line pursuing objective and unambiguous lexical criteria such as the representation of homonyms and synonyms. in this context, there is a broad consensus in the psycholinguistics literature regarding the representation of homonyms as different entries in the lexicon (in contrast to polysemy, for which there is a long discussion on whether senses of polysemous words are stored as a single core representation or as independent entries @xcite ). in fact, several studies have shown that homonyms are cognitively processed differently from polysemous words @xcite . in contrast to the different senses of polysemous words, which are simultaneously activated, the meanings of homonyms are in conflict during processing, with the not relevant ones being deactivated by the context @xcite . to analyze how vector models represent homonymy and synonymy in context, we have built a new multilingual resource with a strong inter-annotator agreement, presented below. ## a new multilingual resource of homonymy and synonymy in context this section briefly describes some aspects of lexical semantics relevant to our study, and then presents the new dataset used in the paper. homonymy and homography: homonymy is a well-known type of lexical ambiguity that can be described as the relation between distinct and unrelated meanings represented by the same word form, such as match, meaning for instance 'sports game' or 'stick for lighting fire'. in contrast to polysemy (where one lexeme conveys different related senses depending on the context, e.g., newspaper as an organization or as a set of printed pages), it is often assumed that homonyms are different lexemes that have the same lexical form @xcite , and therefore they are stored as independent entries in the lexicon @xcite . there are two main criteria for homonymy identification: diachronically, homonyms are lexical items that have different etymologies but are accidentally represented by the same word form, while a synchronic perspective strengthens unrelatedness in meaning. even if both approaches tend to identify similar sets of homonyms, there may be ambiguous cases that are diachronically but not synchronically related (e.g., two meanings of banco -'bench' and 'financial institution'-in portuguese or spanish could be considered polysemous as they derive from the same origin, foot_0 but as this is a purely historical association, most speakers are not aware of the common origin of both senses). in this study, we follow the synchronic perspective, and consider homonymous meanings those that are clearly unrelated (e.g., they unambiguously refer to completely different concepts) regardless of their origin. it is worth mentioning that as we are dealing with written text we are actually analyzing homographs (different lexemes with the same spelling) instead of homonyms. thus, we discard instances of phonologically identical words which are written differently, such as the spanish hola 'hello' and ola 'wave', both representing the phonological form /ola/. similarly, we include words with the same spelling representing different phonological forms, e.g., the galician-portuguese sede, which corresponds to both /sede/ 'thirst', and /sede/ 'headquarters'. in this paper, homonymous senses are those unrelated meanings conveyed by the same (homonym) word form. for instance, coach may have two homonymous senses @xcite , which can be conveyed by other words (synonyms) in different contexts (e.g., by bus or trainer). structure of the dataset: we have created a new resource to investigate how vector models represent word meanings in context. in particular, we want to observe whether they capture (i) different senses conveyed by the same word form (homonymy), and (ii) equivalent senses expressed by different words (synonymy). the resource contains controlled sentences so that it allows us to observe how the context and word overlap affect word representations. ## models and methods this section introduces the models and procedures to obtain vector representations followed by the evaluation method. ## evaluation this section presents the experiments performed using the new dataset and discusses their results. ## conclusions and further work this paper has presented a systematic study of word meaning representation in context. besides static word embeddings, we have assessed the ability of state-of-the-art monolingual and multilingual models based on the transformers architecture to identify unambiguous cases of homonymy and synonymy. to do so, we have presented a new dataset in four linguistic varieties that allows for controlled evaluations of vector representations. the results of our study show that, in most cases, the best contextualized models adequately identify homonyms conveying different senses in various contexts. however, as they strongly rely on the surrounding contexts, they misrepresent words having different senses in similar sentences. in further work, we plan to extend our dataset with multiword expressions of different degrees of idiomaticity and to include less transparent -but still unambiguous-contexts of homonymy. finally, we also plan to systematically explore how multilingual models represent homonymy and synonymy in cross-lingual scenarios. first row shows examples of ex1. in figure 4a cálculos is correctly contextualized since layer 3. in figure 4b , the outlier sense of queixo is not correctly contextualized in any layer.