Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings (2021)

Abstract: The number of senses of a given word, or polysemy, is a very subjective notion, which varies widely across annotators and resources. We propose a novel method to estimate polysemy based on simple geometry in the contextual embedding space. Our approach is fully unsupervised and purely data-driven. Through rigorous experiments, we show that our rankings are well correlated, with strong statistical significance, with 6 different rankings derived from famous human-constructed resources such as WordNet, OntoNotes, Oxford, Wikipedia, etc., for 6 different standard metrics. We also visualize and analyze the correlation between the human rankings and make interesting observations. A valuable by-product of our method is the ability to sample, at no extra cost, sentences containing different senses of a given word. Finally, the fully unsupervised nature of our approach makes it applicable to any language. Code and data are publicly available https://github.com/ksipos/polysemy-assessment .

## introduction polysemy, the number of senses that a word has, is a very subjective notion, subject to individual biases. word sense annotation has always been one of the tasks with the lowest values of interannotator agreement @xcite . yet, creating high-quality, consistent word sense inventories is a critical pre-requisite to successful word sense disambiguation. towards creating word sense inventories, it can be helpful to have some reliable information about polysemy. that is, knowing which words have many senses and which words have only a few senses. such information can help in creating new inventories but also in validating and interpreting existing ones. it can also help select which words to include in a study (e.g., only highly polysemous words). we propose a novel, fully unsupervised, and data-driven approach to quantify polysemy, based on basic geometry in the contextual embedding space. contextual word embeddings have emerged in the last few years as part of the nlp transfer learning revolution. now, entire deep models are pre-trained on huge amounts of unannotated data and fine-tuned on much smaller annotated datasets. some of the most famous examples include ulm-fit @xcite and elmo @xcite , both based on recurrent neural networks; and gpt @xcite and bert @xcite , based on transformers @xcite . these models all are very deep language models. during pre-training on large-scale corpora, they learn to generate powerful internal representations, including fine-grained contextual word embeddings. for instance, in a well pre-trained model, the word python will have two very different embeddings depending on whether it occurs in a programming context (as in, e.g., "i love to write code in python") or in an ecological context ("while hiking in the rainforest, i saw a python"). our approach capitalizes on the contextual embeddings previously described. it does not involve any tool and does not rely on any human input or judgment. also, thanks to its unsupervised nature, it can be applied to any language (even those with limited resources), provided that contextual embeddings are available. the remainder of this paper is organized as follows. we detail our approach in section 2. then, we present our experimental setup (sec. 3), evaluation metrics (sec. 4), and report and interpret our results (sec. 6). in section 7, we briefly touch on two other interesting applications of our method. one that allows the user to sample sentences containing different senses of a given word and one that goes towards word sense induction. finally, related work is presented in section 8. ## experiments in this section, we describe the protocol we followed to test the extent to which our rankings match human rankings. ## evaluation metrics we used 6 similarity, correlation and information retrieval standard metrics to compare among methods: cosine similarity, spearman's rho @xcite 1904), kendall's tau @xcite , precision at k, normalized discounted cumulative gain @xcite , and rank biased overlap @xcite . to ensure a fair comparison, the scores in the rankings of all methods were normalized to be in the [0, 100] range before proceeding. also, each method played in turn the role of candidate and ground truth. this allowed us to compute not only the similarity between our rankings and the ground truth rankings, but also the similarity among the ground truth rankings themselves, which was interesting for exploration purposes. for each pair of evaluated and ground truth method, only the parts of the rankings corresponding to the words in common (intersection) were compared. thus, the rankings in each (candidate,ground truth) pair had equal length. ## implementation details to compute our scores, we built on the code of the pyramid match kernel from the grakel python library @xcite . we used the base r (r core team, 2018) cor() function foot_5 to compute the τ and ρ statistics. for rbo, we relied on a publicly available python implementation 10 . for all other metrics, we wrote our own implementations. full details about design choices, tokenizers, stopword lists etc., can also be found in our publicly available code repository: https: //github.com/ksipos/polysemy-assessment. ## results and observations our rankings correlate well with human rankings. results are shown in fig. 3 , as pairwise similarity matrices, for all six metrics. for readability, all scores are shown as percentages. for a given metric, our configuration that best matches, on average, all other methods (except random and frequency) is always shown as the first column. since all metrics except ndcg are symmetric, we only show the lower triangles of the other matrices. for ndcg, candidate methods are shown as columns and ground truths as rows. for each of the six evaluation metrics, it can be seen that the ranking generated by our unsupervised, data-driven method is well correlated with all human-derived ground truth rankings. this means that our method is robust to how one defines and measures correlation or similarity. in some cases, we even very closely reproduce the human rankings. for instance, our best configurations for cosine and ndcg get almost perfect scores of 86.5 and 99.72 when compared against wikipedia. in terms of kendall's tau, spearman's rho, p@k, and rbo, we are also very close to ontonotes (scores of 49.43, 35.23, 39.53, and 33.47, resp.). finally, the correlation between our rankings and the human rankings can also be observed to be, everywhere, much stronger than that between the baseline rankings (random and frequency) and the human rankings. statistical significance. we computed statistical significance for the spearman's rho and kendall's tau metrics. as shown in fig. 3 , the null hypothesis that there is no correlation between our rankings and the human-derived ground truth rankings was systematically rejected everywhere, with very high significance (@xmath0.0001). however, against the random baseline, the same null hypothesis (no correlation) was accepted everywhere. against frequency, the null was rejected, but very weakly (only at the @xmath1.01 level), and with very low correlation coefficients (6.53 for spearman and 4.44 for kendall). finally, the correlation between the random and frequency rankings and the ground truth rankings is never statistically significant, except for the pair frequency/ontonotes, but again, at a weak level (@xmath2.01). hyperparameters have a significant impact on performance, but optimal values are consistent across metrics. first, as can be observed from fig. 4 and fig. 5 , there is a large variability in performance when d (number of pca dimensions) and l (number of levels in the hierarchy) vary. however, for all six evaluation metrics, the best configurations are very similar: d2l10, d2l8, d2l8, d4l5, d3l9, and d4l10 11 . given the rather large grid we explored ([2, 20] × [2, 19] for d and l, resp.), with 342 combinations in total, we can say that all these optimal values belong to the same small neighborhood. this interpretation is confirmed by inspecting fig. 4 , where it can clearly be seen that the optimal area of the hyperparameter space is robust to metric selection and consistently corresponds to small values of d (around 3), and values of l at least above 3 or 4, ideally around 8. for larger values of l, performance plateaus (keeping d fixed). in other words, it is necessary to have some levels in the hierarchy, but having very deep hierarchies is not required for our method to work well. a benefit of having such small optimal values of d and l is their affordability, from a computational standpoint. all rankings derived from wordnet-based resources are highly correlated. it is interesting to note that the rankings generated from ontonotes, wordnet, wordnet reduced, and wordnet domains, all are highly similar. and this, despite the very different sense granularities they have. this means that despite the apparent differences in these resources, they all tend to produce similar polysemy rankings. the oxford rankings tend to be part of this high-similarity cluster as well, to a lesser extent. frequent words are not the most polysemous. finally, one last interesting observation is that while the frequency ranking is much better than the random ones, it still is far away from the human rankings. in other words, the frequency of appearance of a word (excluding stopwords, of course) is not as good an indicator of its polysemy as one 11 for rbo, d4l10 and d4l8 had the same score. a note on ties. to assess the impact of ties on the reported results, we repeated all of our experiments multiple times with different tie-breaking strategies (e.g., random, alphabetical...). results do not change: we find the same best parameter combinations, and the differences in the similarity matrices are minimal. ## other applications sampling diverse examples. an interesting byproduct of our discretization strategy is that it can be used to select sentences containing different senses of the same word, as illustrated in ## conclusion we proposed a novel unsupervised, fully datadriven geometrical approach to estimate word polysemy. our approach builds multiresolution grids in the contextual embedding space. through rigorous experiments, we showed that our rankings are well correlated (with strong statistical significance) to 6 different human rankings, for 6 different metrics. such fully data-driven rankings of words according to polysemy can help in creating new sense inventories, but also in validating and interpreting existing ones. increasing the quality and consistency of sense inventories is a key first step of the word sense disambiguation pipeline. we also showed that our discretization could be used, at no extra cost, to sample contexts containing different senses of a given word, which has useful applications in practice. finally, the unsupervised nature of our method makes it applicable to any language. while our scores are a good proxy for polysemy, they are not equal to word sense counts. moreover, we do not label each sense. future work should address these challenges by, e.g., automatically selecting bins of interest and generating labels for them (see section 7). future work should also perform some sort of extrinsic evaluation. for instance, the wordin-context task (pilehvar and camacho-collados, 2018) could be used, where two occurrences would be classified as having the same meaning if their two vectors fall in the same bin. another direction is investigating how different contextual embeddings (e.g., bert, bart) impact our rankings, including in languages other than english @xcite , and low-resource languages.