P oly LM : Learning about Polysemy through Language Modeling (2021)

Abstract: To avoid the “meaning conflation deficiency” of word embeddings, a number of models have aimed to embed individual word senses. These methods at one time performed well on tasks such as word sense induction (WSI), but they have since been overtaken by task-specific techniques which exploit contextualized embeddings. However, sense embeddings and contextualization need not be mutually exclusive. We introduce PolyLM, a method which formulates the task of learning sense embeddings as a language modeling problem, allowing contextualization techniques to be applied. PolyLM is based on two underlying assumptions about word senses: firstly, that the probability of a word occurring in a given context is equal to the sum of the probabilities of its individual senses occurring; and secondly, that for a given occurrence of a word, one of its senses tends to be much more plausible in the context than the others. We evaluate PolyLM on WSI, showing that it performs considerably better than previous sense embedding techniques, and matches the current state-of-the-art specialized WSI method despite having six times fewer parameters. Code and pre-trained models are available at https://github.com/AlanAnsell/PolyLM .

## introduction much work in nlp has been dedicated to vector representations of words, but it has been recognized since as early as @xcite that such representations fail to capture the polysemous nature of many words, conflating their multiple senses into a single point in semantic space. there have been several attempts at embedding individual word senses to avoid this issue, termed the "meaning conflation deficiency" by @xcite in their survey on the area. we propose polylm, an unsupervised sense embedding model which is effective and easy to apply to downstream tasks. polylm can be thought of as both a (masked) language model and a sense model, as it calculates a probability distribution both over words and word senses at masked positions. the formulation is derived from two observations about word senses: firstly, that the probability of a word occurring in a given context is equal to the sum of the probabilities of its individual senses occurring; and secondly, that for a given occurrence of a word, one of its senses tends to be much more plausible in the context than the others. there are several reasons for the interest in sense representations. the first is the downsides associated with the meaning conflation deficiency. word embedding models can have difficulty distinguishing which sense of an ambiguous word applies in a given context @xcite . additionally, homonymy and polysemy cause distortion in word embeddings: for instance, we would find the unrelated words left and wrong unreasonably close in the vector space due to their similarity to two different senses of the word right, an effect noted by @xcite and illustrated in figure 1 . intuitively we would expect that sense embedding models could gain superior semantic understanding by avoiding these problems. in addition to well-established applications for sense representations such as word sense disambiguation (wsd) and induction (wsi), another interesting use case is the automatic construction of lexical resources @xcite . while there are existing human-curated word sense inventories for english such as such as wordnet @xcite , these are expensive to create and are unavailable for most languages. @xcite showed that sense embeddings learned using the model of @xcite could be linked with word senses contained in babelnet @xcite with a reasonable degree of precision, although the mapping struggled with recall. polylm represents a significant advance over @xcite s in terms of wsi performance, so @xcite and adjusttext @xcite . sense embeddings were learned by training polylm small with the standard 8 senses per word; word embeddings were learned by training polylm small , but with a single sense per word. note that both models were trained on unlemmatized data, unlike those used in the wsi experiments. the occurrence of closely related polysemous words nearby in the word embedding space (i.e. left and right) causes unrelated words to be closer together (e.g. left and wrong) and related words to be further apart (e.g. right and east) than they otherwise would be. the use of sense embeddings avoids such distortion. polylm is capable of detecting comparatively rare word senses, such as the political senses of left and right, and the use of smith and mason to refer to tradespeople. the emergence of contextualized models such as elmo @xcite and bert @xcite has had a tremendous impact on the area of semantic representation. rather than representing words using a single embedding, or even a set of sense embeddings, these models allow words to be represented using an infinite set of possible embeddings depending on the context. this approach has been very effective across nlp and many state-ofthe-art systems incorporate contextualized models, including systems for wsd and wsi. the success of contextualized models raises the question of whether there is still value in learning discrete sense representations. however, contextualized models still rely on word embeddings, and are therefore subject to the meaning conflation deficiency. furthermore it could be argued that it is inefficient to have the same representation size for all words regardless of how diverse their range of senses is. another drawback is that before they can be applied to word sense-related tasks, an adaptation step such as clus-tering to induce discrete senses or fine-tuning is generally required, which is often expensive in terms of both research and compute time. the contributions of this paper can be summarized as follows: we show that good performance on wsi can be achieved using the output of this layer alone, suggesting that it could be a useful component in many neural networks for language understanding. one of the first works in unsupervised learning of sense representations was by @xcite , who proposed a two-step process, where vector representations are first derived for each context containing an ambiguous word, and these are then clustered into a pre-defined number of groups. @xcite added a third step, where after senselabeling each word according to its context cluster, sense representations are learned through neural language modeling. a number of later approaches employed a joint training approach, where sense labeling and sense representation learning happen in parallel. neelakantan et al. (2014), li and jurafsky (2015) and @xcite each proposed multi-sense variants of the skip-gram model @xcite . various approaches were tried for determining the number of senses per word: for instance, @xcite chinese restaurant processes and dirichlet processes respectively to automatically learn an appropriate number of senses for each word. many joint training approaches have the disadvantage that they create ambiguity in the context representation by representing context words with word embeddings in order to avoid considering the exponential number of possible sense labelings for the context. @xcite and @xcite propose purely sense-based approaches which can sense-label the input efficiently. @xcite took a novel approach to the problem of learning word senses, demonstrating that the embedding learned by traditional techniques for an ambiguous word tends to be very close to a linear combination of the hypothetical vectors corresponding to its individual senses. they proposed a method for recovering the underlying sense vectors and coefficients, and evaluated their system on wsi. since the emergence of contextualized models, there have been a number of other systems which have exploited their powerful semantic representations for specific tasks such as word sense disambiguation @xcite and induction @xcite , however none of these methods creates explicit sense embeddings. ## experiments word sense induction (wsi) is the task of inferring the senses of a word in an unsupervised manner. this is precisely the aim of our method, and so is an ideal test task. we evaluate polylm on two wsi datasets, semeval-2010 task 14 @xcite and semeval-2013 task 13 @xcite . both datasets consist of passages containing one of a set of polysemous focus words. the occurrences of the focus words in the test set have been sense-labeled by human annotators according to a reference sense inventory. in the semeval-2010 dataset, each instance is labeled with a single sense, whereas in the semeval-2013 dataset an instance may be labeled with several relevant senses, each with a corresponding weight denoting its degree of applicability in the context. performance on semeval-2010 is measured using paired f-score (f-s) and v-measure (v-m), and on semeval-2013 using fuzzy b-cubed (fbc) and fuzzy normalized mutual information (fnmi). overall performance on each task (avg) is typically defined as the geometric mean of its two sub-metrics. currently, the best performing system on both datasets is that of @xcite . their system uses the idea of substitute vectors, first devised by @xcite . for each instance, a set of most likely words that could have occurred instead of the focus word is obtained from the output of a language model. these sets are then clustered, and each cluster is taken to correspond to a different sense of the focus word. amrami and goldberg use bert large as their language model. polylm can be used for wsi without any further training. for the semeval-2010 dataset, each instance c is labeled with the sense of the focus word w i which has the highest predicted probability, i.e. argmax @xmath0, c), where c is formed from c by replacing w i with @xcite . for semeval-2013, we consider a sense applicable if it has a predicted probability q p is (c , c) > p thresh , and the weight assigned to each applicable sense is its probability q p is (c , c). we arbitrarily set p thresh to 0.2. results are shown in polylm scales well for the sizes tested, with polylm base outperforming polylm small by 3.2 and 4.1 points in avg score on the two datasets with a 2.25x increase in the number of parameters. even if further increases in model dimensions yielded much smaller improvements in performance, it seems likely that a polylm model of bert large 's 340 million parameter size would achieve results significantly better than those of @xcite . ## conclusions polylm is a novel model of polysemy based on two assumptions about word senses: firstly, that the probability of a word occurring in a context is equal to the sum of its individual senses occurring, as expressed by the language modeling loss; and secondly, that generally only one sense of a word ought to have a high probability of occurring in a given context, as expressed by the distinctness loss. polylm does indeed learn word senses which correspond well to human notions, as demonstrated by its performance on word sense induction, which matches that of the previous state-of-the-art system despite having 6 times fewer parameters. it can be easily applied to many word-sense related tasks, as it generates a probability distribution over the senses of each word in the input text. it is not specific to any one contextualizer and so can be improved as contextualizers improve. yadollah yaghoobzadeh and hinrich schütze. 2016. intrinsic subspace evaluation of word embedding representations. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 236-246, berlin, germany. association for computational linguistics. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the 2015 ieee international conference on computer vision (iccv), iccv '15, pages 19-27, washington, dc, usa. ieee computer society. george k. zipf. 1950. human behavior and the principle of least effort. journal of clinical psychology, 6(3):306-306.