An Unsupervised, Geometric and Syntax-aware Quantification of Polysemy (2022)

Abstract: Polysemy is the phenomenon where a single word form possesses two or more related senses. It is an extremely ubiquitous part of natural language and analyzing it has sparked rich discussions in the linguistics, psychology and philosophy communities alike. With scarce attention paid to polysemy in computational linguistics, and even scarcer attention toward quantifying polysemy, in this paper, we propose a novel, unsupervised framework to compute and estimate polysemy scores for words in multiple languages. We infuse our proposed quantification with syntactic knowledge in the form of dependency structures. This informs the final polysemy scores of the lexicon motivated by recent linguistic findings that suggest there is an implicit relation between syntax and ambiguity/polysemy. We adopt a graph based approach by computing the discrete Ollivier Ricci curvature on a graph of the contextual nearest neighbors. We test our framework on curated datasets controlling for different sense distributions of words in 3 typologically diverse languages - English, French and Spanish. The effectiveness of our framework is demonstrated by significant correlations of our quantification with expert human annotated language resources like WordNet. We observe a 0.3 point increase in the correlation coefficient as compared to previous quantification studies in English. Our research leverages contextual language models and syntactic structures to empirically support the widely held theoretical linguistic notion that syntax is intricately linked to ambiguity/polysemy.

## introduction polysemy is a phenomenon prevalent in everyday language use where the same lexical unit (or word form) is associated with multiple distinct yet related meanings (or senses). determining which words are polysemous can help in filtering data for linguist studies, creation of sense corpora and the anthropological study of language. consider the following sentences: 1a his aunt is his legal guardian. polysemy is distinct from monosemy (a word form with only one meaning; 1a) and homonymy (multiple unrelated senses of the same word form; 2a-b). the polysemous senses of a word often have metonymic (3a-b) or metaphorical (3c) relations among them @xcite . polysemy is a central feature of natural languages and proliferates almost every word to varying degrees in the lexicon of a language. attempts @xcite at explaining the presence of ambiguity foot_0 in language suggest that polysemy is a desirable property for language systems since it allows efficient communication by allowing simpler units to be reused. ambiguity and polysemy have sparked debate among linguists and philosophers for decades but relatively little attention has been paid to analyze and measure polysemy in language by computational linguists. while a human listener is easily able to disambiguate the specific sense of the word being used in context, it is notoriously difficult for nlp systems to separate the distinct senses of a word being used @xcite . recently, there has been widespread attention on including syntactic knowledge in various computational linguistic systems and studies -ranging from syntax aware language models @xcite to syntax informed sentiment analysis @xcite . recent works have identified @xcite an intricate link between the syntactic properties of a lexical unit and its ambiguity (or lack thereof) since the meaning of a word is influenced by its syntactic as well as semantic context. the fact that most open class word forms are associated with multiple related senses hints at the possible role that syntax plays in influencing polysemy. syntactic structures can constrain the possible contexts a word form may be used in, thus there is an implicit relation between the semantics of a lexical unit and its associated polysemy. motivated by these recent linguistic findings, we operationalize the polysemy of a word form as being influenced by both -its semantic variability and its importance in the syntactic network. the level of polysemy a word possesses is highly subjective and varies widely across annotators @xcite . to aid annotators in creating, validating and qualitatively analysing sense inventories, having an estimate of the ambiguity a word possesses could be very helpful. this measure then acts as a proxy to how many (or how few) senses a word in a certain language possesses. a quantification of polysemy is also helpful in information retrieval systems as they can be used to rank more relevant results @xcite . polysemic knowledge can also help improve crosslingual alignment of embedding spaces and crosslingual transfer @xcite . while recent contextual embedding models like bert, xlm and roberta have been shown to possess the ability of distinguishing between different senses of a word @xcite , less attention has been paid towards quantifying the level of polysemy that a word represents -a measure which is continuous and can be compared across lexica. attempts at quantifying polysemy either rely on large amounts of data @xcite and/or on carefully tuned hyperparameters and embedding distortion due to dimensionality reduction of the contextual space of language models @xcite . we operationalize polysemy of a word form as a quantity influenced by its contextual semantic neighbors and its syntactic role in a syntactic network. in particular, we construct a contextual nearest-neighbor graph of lexical units using a pretrained language model like bert @xcite , the red edge (more negative) acts as a bridge connecting two distinct neighborhoods (distinct senses of the word bank) while the blue edge (more positive) is an edge within the same neighborhood (sense cluster of the monosemous word proton). the rest of the paper is organized as follows. section §2 discusses the background on ambiguity in linguistics and related topics. mathematical preliminaries and the proposed method are detailed in section §3 and section §4 respectively. section §5 discusses the implementation, data, experimental setup and the results. limitations and future work are discussed in section §6. ## background & related work lexical ambiguity ambiguity of language has been addressed as early as in the writings of aristotle but relatively recent linguistic research in the form of zipf's principle of least effort @xcite heralded a new understanding of human cognition and language systems positing the tradeoff between efficiency and brevity in communication systems @xcite . polysemy is a natural outcome of lexical semantic change @xcite by virtue of words gaining new meanings over time. recent works in computational linguistics for ambiguity mostly deal with word sense disambiguation @xcite , word-in-context tasks (pilehvar and camacho-collados, 2019) and analyzing polysemy in language models like bert @xcite . while some previous works @xcite acknowledge polysemy even in particular instances, relatively less attention has been paid towards quantifying polysemy using current nlp tools. @xcite measure ambiguity in language from an information-theoretic lens but their approach requires a large number of sentences to give a good upper bound on ambiguity estimates. @xcite leveraged contextual language models like bert to estimate polysemy but they rely on dimensionality reduction and sensitive hyperparameters. works like @xcite have explored the geometry of bert embeddings and their relation to polysemy levels thus highlighting the importance of neural embeddings in the quantification of polysemy levels of lexicons. lexical substitution lexical substitution is the task of finding relevant contextual replacements of a word given its context. to generate good quality contextual replacements, previous works have relied heavily on distributional semantic models like word2vec @xcite and specialized language models like context2vec @xcite . in all models, the generated substitutes are ranked based on some relation with the target word to be replaced. recent advances in language models like the transformer-based bert @xcite and xlnet @xcite rely on the bidirectional context and the special @xcite token based training to generate contextual substitutes. @xcite showed that bert performs poorly on lexical substitution and proposed a dropout based approach which is even more computationally expensive due to the large number of forward passes required. supervised approaches @xcite often rely on manually curated databases and sense inventories like wordnet, wikipedia or babelnet. @xcite is a recent neural lexical substitution method which injects information about the target word in the form of probability distribution of possible word subtitutes based on word frequencies. graphs and nlp traditional works in linguistics have used language networks and graphs for analyzing morphological complexity @xcite , ambiguity @xcite and phonetics @xcite . recent advances in graph neural networks (gnns) has opened new avenues to apply network based approaches to language problems. while language networks have been analysed before, gnns provide an alternative to traditional methods with more natural inductive biases for syntactic models to work with. the combination of graphs and language models has proved to be effective in incorporating semantics and syntax in language problems @xcite @xcite . ## proposed approach in this section, we introduce our proposed approach to quantify polysemy as illustrated in figure 2 . ## experiments in this section, we first describe the data used in the current study ( §5.1) followed by a description of the flow of the proposed approach ( §5.2). next we describe the evaluation metrics used ( §5.3) and the implementation details of the current study ( §5.4). finally, we discuss the results of the proposed approach ( §5.5) and perform an ablation study of investigating the individual contribution of semantics and syntax towards polysemy ( §5.6). ## discussion since our method aims to quantify the tendency of a word to have more meanings, words assigned higher values are assumed to be more polysemous. while this operationalisation does not explicitly allow for the discovery of new polysemy relations, we observe, for example, that "accord" (6 ground truth senses) is assigned a higher polysemy score relative to a word like "maximum" (4 ground truth sense). this case is interesting since wordnet provides very similar ratings for both while our method accentuates the difference between the two, intuitively, giving "accord" much higher score. conclusion in this study, based on previous linguistic evidence, we posit that including syntactic information in the form of dependency structural knowledge can help in the quantification of lexical ambiguity or polysemy of a wordform. to investigate this, we propose a simple operationalization of polysemy based on the ricci curvature of the contextual nearest neighbors graph of a word and the entropy of its combined syntactic network. we show that our proposed measure shows high correlations with number of word senses in word-net across multiple languages. our approach is fully unsupervised, simple and grounded in previously established linguistic theories. we hope that similar graph-based approaches can help in creation and validation of sense inventories across languages. limitations our work acts as a proxy for the ambiguity of a word form and the scores are continuous but it does not quantify the discrete counts of the senses of a word. we rely on the availability of good quality language-specific language models which can be used as the lexical substitution model in the semantic module. any errors in the language model may propagate into our score. we tested our framework on sentences sampled from the semcor 3.0 dataset which is a good resource for sense analysis in nlp but is naturally limited to sentences in formal english. a lack of diversely sourced corpora for a study in polysemy may limit the generalizability of a quantification measure to other domains. future work we leave the utility of polysemy quantification to improve extrinsic tasks word sense disambiguation or word in context for future work. we hope that works in polysemy quantification also lead to interesting linguistic analyses about the nature of ambiguity in natural languages and the relationship between syntactic information like part-of-speech tags and polysemy scores of word units.