Let’s Play Mono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses (2021)

Abstract: Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analyzing this knowledge in LMs specifically trained for different languages (English, French, Spanish, and Greek) and in multilingual BERT. We perform our analysis on datasets carefully designed to reflect different sense distributions, and control for parameters that are highly correlated with polysemy such as frequency and grammatical category. We demonstrate that BERT-derived representations reflect words’ polysemy level and their partitionability into senses. Polysemy-related information is more clearly present in English BERT embeddings, but models in other languages also manage to establish relevant distinctions between words at different polysemy levels. Our results contribute to a better understanding of the knowledge encoded in contextualized representations and open up new avenues for multilingual lexical semantics research.

## introduction pre-trained contextual language models have advanced the state of the art in numerous natural language understanding tasks @xcite . their success has motivated a large number of studies exploring what these models actually learn about language @xcite @xcite . the bulk of this interpretation work relies on probing tasks that serve to predict linguistic properties from the representations generated by the models @xcite . the focus was initially put on linguistic aspects pertaining to grammar and syntax @xcite @xcite . the first probing tasks addressing semantic knowledge explored phenomena in the syntax-semantics interface, such as semantic role labeling and coreference @xcite , and the symbolic reasoning potential of lm representations @xcite . lexical meaning was largely overlooked in early interpretation work, but is now attracting increasing attention. pre-trained lms have been shown to successfully leverage sense annotated data for disambiguation @xcite . the interplay between word type and token-level information in the hidden representations of lstm lms has also been explored @xcite , as well as the similarity estimates that can be drawn from contextualized representations without directly addressing word meaning @xcite . in recent work, @xcite probe bert representations for lexical semantics, addressing out-of-context word similarity. whether these models encode knowledge about lexical polysemy and sense distinctions is, however, still an open question. our work aims to fill this gap. we propose methodology for exploring the knowledge about word senses in contextualized representations. our approach follows a rigorous experimental protocol proper to lexical semantic analysis, which involves the use of datasets carefully designed to reflect different sense distributions. this allows us to investigate the knowledge models acquire during training, and the influence of context variation on token representations. we account for the strong correlation between word frequency and number of senses @xcite , and for the relationship between grammatical category and polysemy, by balancing the frequency and part of speech representations for a poly word are obtained from sentences reflecting up to ten different senses (polybal), the same sense (poly-same), or natural occurrence in a corpus (poly-rand). importantly, our investigation encompasses monolingual models in different languages (english, french, spanish, and greek) and multilingual bert (mbert). we demonstrate that bert contextualized representations encode an impressive amount of knowledge about polysemy, and are able to distinguish monosemous (mono) from polysemous (poly) words in a variety of settings and configurations (cf. figure 1 ). importantly, we demonstrate that representations derived from contextual lms encode knowledge about words' polysemy acquired through pre-training which is combined with information from new contexts of use (sections 3-6). additionally, we show that bert representations can serve to determine how easy it is to partition a word's semantic space into senses (section 7). our methodology can serve for the analysis of words and datasets from different topics, domains and languages. knowledge about words' polysemy and sense partitionability has numerous practical implications: it can guide decisions towards a sense clustering or a per-instance approach in applications @xcite @xcite ; point to words with stable semantics which can be safe cues for disambiguation in running text @xcite @xcite ; determine the needs in terms of context size for disambiguation (e.g., in queries, chatbots); help lexicographers define the number of entries for a word to be present in a resource, and plan the time and effort needed in semantic annotation tasks @xcite . it could also guide cross-lingual transfer, serving to identify polysemous words for which transfer might be harder. finally, analyzing words' semantic space can be highly useful for the study of lexical semantic change @xcite @xcite . we make our code and datasets available to enable comparison across studies and to promote further research in these directions. foot_0 2 related work the knowledge pre-trained contextual lms encode about lexical semantics has only recently started being explored. works by @xcite and @xcite propose experiments using representations built from wikipedia and the semcor corpus @xcite , and show that bert can organize word usages in the semantic space in a way that reflects the meaning distinctions present in the data. it is also shown that bert can perform well in the word sense disambiguation (wsd) task by leveraging the sense-related information available in these resources. these works address the disambiguation capabilities of the model but do not show what bert actually knows about words' polysemy, which is the main axis of our work. in our experiments, sense annotations are not used to guide the models into establishing sense distinctions, but rather for creating controlled conditions that allow us to analyze bert's inherent knowledge of lexical polysemy. probing has also been proposed for lexical semantics analysis, but addressing different questions than the ones posed in our work. @xcite probe the hidden representations of a bidirectional (bi-lstm) lm for lexical (typelevel) and contextual (token-level) information. they specifically train diagnostic classifiers on the tasks of retrieving the input embedding of a word and a representation of its contextual meaning (as reflected in its lexical substitutes). the results show that the information about the input word that is present in lstm representations is not lost after contextualization; however, the quality of the information available for a word is assessed through the model's ability to identify the corresponding embedding, as in @xcite and @xcite . also, lexical ambiguity is only viewed through the lens of contextualization. in our work, on the contrary, it is given a central role: we explicitly address the knowledge bert encodes about words' degree of polysemy and partitionability into senses. vulić et al. (2020) also propose to probe contextualized models for lexical semantics, but they do so using ''static'' word embeddings obtained through pooling over several contexts, or extracting representations for words in isolation and from bert's embedding layer, before contextualization. these representations are evaluated on tasks traditionally used for assessing the quality of static embeddings, such as out-of-context similarity and word analogy, which are not tailored for addressing lexical polysemy. other contemporaneous work explores lexical polysemy in static embeddings @xcite , and the relation of ambiguity and context uncertainty as approximated in the space constructed by mbert using informationtheoretic measures @xcite . finally, work by @xcite provides useful observations regarding the impact of context on the representations, without explicitly addressing the semantic knowledge encoded by the models. through an exploration of bert, elmo, and gpt-2 @xcite , the author highlights the highly distorted similarity of the obtained contextualized representations which is due to the anisotropy of the vector space built by each model. 2 the question of meaning is not addressed in this work, making it hard to draw any conclusions about lexical polysemy. our proposed experimental setup is aimed at investigating the polysemy information encoded in the representations built at different layers of deep pre-trained lms. our approach basically relies on the similarity of contextualized representations, which amounts to word usage similar-ity (usim) estimation, a classical task in lexical semantics @xcite @xcite . the usim task precisely involves predicting the similarity of word instances in context without use of sense annotations. bert has been shown to be particularly good at this task @xcite . our experiments allow us to explore and understand what this ability is due to. ## analysis by frequency and pos given the strong correlation between word frequency and number of senses @xcite , we explore the impact of frequency on bert representations. our goal is to determine the extent to which it influences the good mono/poly detection results obtained in sections 3.4 and 4.1. ## classification by polysemy level our finding that word instance similarity differs across polysemy bands suggests that this feature can be useful for classification. in this section, we probe the representations for polysemy using a classification experiment where we test their ability to guess whether a word is polysemous, and which poly band it falls in. we use the poly-rand sentence pools and a standard train/dev/test split (70%/15%/15%) of the data. for the mono/poly distinction (i.e., the data used in section 3), this results in 584/126/126 words per set in each language. to guarantee a fair evaluation, we make sure there is no overlap between the lemmas in the three sets. we use two types of features: (i) the average self sim for a word; and (ii) all pairwise cosine similarities collected for its instances, which results in 45 features per word (paircos). we train a binary logistic regression classifier for each type of representation and feature. as explained in section 4, the three poly bands (low, mid, and high) and mono contain a different number of words. for classification into polysemy bands, we balance each class by randomly subsampling words from each band. in total, we use 1,168 words for training, 252 for development, and 252 for testing (70%/15%/15%) in english. in the other languages, we use a split of 840/180/180 words. we train multi-class logistic regression classifiers with the two types of features, self sim and paircos. we compare the results of the classifiers to a baseline that predicts always the same class, and to a frequencybased classifier which only uses the words' log frequency in google ngrams, or in the oscar corpus, as a feature. all classifiers outperform the same class baseline. these results are very encouraging, showing that bert embeddings can be used to determine whether a word has multiple meanings, and provide a rough indication of its polysemy level. results in the other three languages are not as high as those obtained in english, but most models give higher results than the frequency-based classifier. 26 7 word sense clusterability we have shown that representations from pretrained lms encode rich information about words' degree of polysemy. they can successfully distinguish mono from poly lemmas, and predict the polysemy level of words. our previous experiments involved a set of controlled settings representing different sense distributions and polysemy levels. in this section, we explore whether these representations can also point to the clusterability of poly words in an uncontrolled setting. ## conclusion we have shown that contextualized bert representations encode rich information about lexical polysemy. our experimental results suggest that this high quality knowledge about words, which allows bert to detect polysemy in different configurations and across all layers, is acquired during pre-training. our findings hold for the english bert as well as for bert models in other languages, as shown by our experiments on french, spanish, and greek, and to a lesser extent for multilingual bert. moreover, english bert representations can be used to obtain a good estimation of a word's partitionability into senses. these results open up new avenues for research in multilingual semantic analysis, and we can consider various theoretical and application-related extensions for this work. the polysemy and sense-related knowledge revealed by the models can serve to develop novel methodologies for improved cross-lingual alignment of embedding spaces and cross-lingual transfer, pointing to more polysemous (or less clusterable) words for which transfer might be harder. predicting the polysemy level of words can also be useful for determining the context needed for acquiring representations that properly reflect the meaning of word instances in running text. from a more theoretical standpoint, we expect this work to be useful for studies on the organization of the semantic space in different languages and on lexical semantic change.