S yn2 V ec: Synset Colexification Graphs for Lexical Semantic Similarity (2022)

Abstract: In this paper we focus on patterns of colexification (co-expressions of form-meaning mapping in the lexicon) as an aspect of lexical-semantic organization, and use them to build large scale synset graphs across BabelNet’s typologically diverse set of 499 world languages. We introduce and compare several approaches: monolingual and cross-lingual colexification graphs, popular distributional models, and fusion approaches. The models are evaluated against human judgments on a semantic similarity task for nine languages. Our strong empirical findings also point to the importance of universality of our graph synset embedding representations with no need for any language-specific adaptation when evaluated on the lexical similarity task. The insights of our exploratory investigation of large-scale colexification graphs could inspire significant advances in NLP across languages, especially for tasks involving languages which lack dedicated lexical resources, and can benefit from language transfer from large shared cross-lingual semantic spaces.

## introduction distributional models like word embeddings have been widely used in natural language processing (nlp) @xcite @xcite . they operate under the assumption that words appearing in similar contexts have similar meanings, and thus close representations. however, as do other unsupervised learning models, they suffer from classic limitations -i.e., there is no guarantee that all context words contribute to the meaning of the target word, while, in fact, it is possible that some low frequency words, with poorly-trained embeddings, are highly semantically connected. also, they don't distinguish between topically related words and near synonyms. dictionaries and thesauruses, on the other hand, have traditionally offered an alternative approach, through their discrete lists of fine-grained senses, textual definitions, and relationships with other senses. given a sufficiently large dictionary of many fine-grained sense representations in many of the world's languages, one could perform sophisticated semantic tasks on word senses @xcite . in fact, investigating universal and areal cross-linguistic variations in the lexicon has been the focus of lexical typology. one increasingly popular empirical method of investigating senses based on cross-linguistic comparison in typological studies has been that of colexification patterns. "a given language is said to colexify two functionally distinct senses if, and only if, it can associate them with the same lexical form" @xcite , reflecting natural semantic connections @xcite . for example, 'fire' and 'firewood' are colexified in kamoro (new guinea language) as 'uta' and in wayuu (arawakan language) as 'siki', but receive distinct lexemes in english and romanian. each polysemous lexeme as a whole is language-specific, yet a great number of lexical polysemies are each attested across multiple languages. in this paper, we investigate semantic structures in the lexicon as manifested by colexification patterns in a large number of languages and assess, at a large scale, their usefulness to the lexical semantic similarity (lsim) task @xcite . we used one of the largest digital lexical resources to date, babelnet 5.0 (953.4m (concept) lexicalizations in 499 languages foot_0 ) to build and process colexification graphs. specifically, we make the following contributions: 1) propose a simple, yet effective algorithm to automatically construct large-scale synset similarity graphs based on the principle of colexification, and use the graphs to generate high-quality synset and word representations. 2) demonstrate the importance of universality of our graph synset embedding representations with no need for any language-specific adaptation when evaluated on lsim. 3) show that our proposed approach significantly outperforms state-of-the-art synset and word embedding techniques on the lsim task. 4) when combined with knowledge-based approaches like our cross-lingual colexification patterns, purely unsupervised distributional models like fasttext and bert result in better alignment with human perception, as measured on the lsim task. our findings and models contribute to advances in computational modeling of natural language understanding across languages, and offer new insights into linguistic typology. ## related work the concept of colexification has been introduced by @xcite to distinguish senses in the grammatical domain, but has been formalized for the field of lexicon by @xcite who, in a cross-linguistic study of the world's lexicons, investigated colexification patterns captured in a semantic map. unlike haspelmath who showed that 12 diverse languages are sufficient to build a stable semantic map, francois posits that, in fact, the number of distinctions between senses increases with the number and variety of considered languages. following these studies, @xcite built a weighted colexification graph using data from 195 languages in 44 language families, with subsequent improved language coverage versions @xcite . here, closely-related or similar concepts tend to be often densely connected @xcite . @xcite constructed colexification graphs in the domain of natural objects (celestial and landscape) and investigated their polysemy distributions for the task of semantic similarity. we also take advantage of recurrent patterns in semantic structure across different language families. however, unlike them, we found some evidence that geographical and cultural differences matter in the human perception of our cross-linguistically connected concepts. @xcite distinguished between homonymy and polysemy by investigating the colexifications of 100 basic concepts. @xcite discovered cross-linguistic similarities based on colexification patterns of verbs of vision and hearing in the the do-main of perception-cognition. @xcite relied on colexification patterns to test the universality of emotion perception. like us, they show that, while there are shared structures of (affective) meanings across cultures, there are also some variations @xcite tested whether colexification patterns in multilingual resources are correlated with affective meaning similarity between words. @xcite showed that no two concepts are colexified in every language by analyzing colexification data from three resources: babelnet, open multilingual wordnet @xcite , and the database of cross-linguistic colexifications (clics3) @xcite . although the scope of research on colexification varies across projects, most studies have assumed that colexification captures some degree of semantic similarity. indeed, this is implied to some extent by the very definition of colexification, and supported by previous results in linguistics and nlp, suggesting that more commonly colexified meanings across languages require less cognitive effort to relate and recall @xcite . however, such a connection between cross-linguistic colexification patterns and semantic similarity has not been fully assessed at a large scale. given that features of the lexicon are not easily identifiable across many languages, one solution is to rely on large data repositories to unveil cross-linguistic generalizations. however, since most languages lack dedicated lexical resources for semantic similarity (henceforth, low-resource languages), one option is to transfer lexico-semantic knowledge from large shared cross-lingual semantic spaces. in this paper, as part of a large scale empirical study, we show that lexico-semantic associations captured by cross-lingual colexification patterns in babelnet contribute significantly in assessing if two words are semantically similar. ## synset cross-lingual colexification under the colexification framework, the primary unit of observation for lexical typology is no longer the word, but the sense -a functionally-based criterion of concept definition @xcite . in this project, however, we use the synset as the primary unit. for us, this is also a technical consideration, since only synsets can be compared across languages, especially in babelnet. lexical concepts are grouped into sets of cognitive (near) synonyms, called synsets, each encoding a distinct meaning. synsets are connected through lexical relations and conceptual/semantic relations (i.e., hyperonymy, hyponymy, meronymy, etc.). in this paper, we use only lexical relations to capture colexifications between meanings. for instance, the lexical relation between the senses of 'fire' and 'firewood' of the word uta (in kamaro) is a case of strict colexification. loose colexifications like derivationally-related forms can show interesting semantic associations, but are not considered here. two synset concepts that are colexified in at least one language are usually perceived as somehow semantically connected, either directly or indirectly. however, proving such connectedness is by no means an easy task. the accurate description of lexical data often requires taking into account the many functional properties of real-world referents as well as culture-specific aspects of a language or geographic area. such cases might capture underlying linguistic phenomena such as metaphor, metonymy, hyperonymy, analogical extension, and a rich set of cases of semantic shifts unique to each language or language family (juvonen and koptjevskaja-tamm, 2016) -whose analysis falls within the scope of semantic or etymological studies, and beyond our goal here. instead, our purpose is to organize cross-linguistic sense information in a way that captures various semantic connections between senses, allowing one to zoom in and out on aspects of the lexicon in cross-linguistic comparative studies. we rely here on the powerful structure of babelnet that maps concepts (i.e., synsets) across a large, typologically diverse set of languages. this allows us to empirically examine at a large scale the contribution of such a rich body of knowledge to the task of semantic similaritysuch empirical evidence is still lacking in the field. our model of semantic connection between synsets (we call syn2vec) is simple: given the set of concepts (synsets) of a lexeme, we assign a semantic link between every synset-synset pair. we want to investigate the idea that, as more and more languages are explored, and more and more senses are amassed, the resulting graph of cross-linguistic inter-connected synset concepts will capture aspects of semantic knowledge that might be missing in one language alone. given this intuition, next we briefly introduce the lexical resource used and our proposed algorithm to construct colexification graphs which model the synset semantic connections. we hypothesize algorithm 1 construction of colexification graph: given a set of languages l and corresponding vocabularies v , create graph edges between all colexified synset pairs (nodes). for a colexification graph g, nodes represent synsets, and edge weights the semantic connection between two synsets. we denote the weight for each edge {s 1 , s 2 } in the graph as g(s 1 , s 2 ) = g(s 2 , s 1 ) (undirected graph). algorithm 1 details the graph construction. in this study, we examine two types of colexification graphs: (1) monolingual, and (2) cross-lingual. for monolingual graphs, we choose one language l and provide @xmath0. for the cross-lingual graph, we use all languages, i.e. in babelnet, the same concept in each language will be mapped to a common synset and thus be represented as a node in the colexification graph. foot_2 c. creating synset and word embeddings. as we like to capture lexico-semantic associations, given a colexification graph g, we assume that vector representations of the nodes (synsets) that are close to one another are similar as computed by cosine distance in the embedding space. we use a recently-developed node embedding approach, prone @xcite , which, compared to other popular node embedding techniques like deepwalk and node2vec @xcite , is much faster and demonstrates superior node representations on several classification datasets using a lib-linear classifier. we use the python implementation from nodevectors foot_3 with all default hyperparameters. in predicting the lexical similarity of two words, we assume that their perceptual similarity is determined by summing the synset embeddings of each word, then comparing the results. thus, a word embedding w is computed as: ## baselines we evaluate the quality of our babelnet dictionary approach by comparing it to high-quality and pop-ular word and synset embedding approaches. we want to test whether the structural regularities observed in distributed text representations provide a route past some of the limitations of dictionaries, whether these two representations are comparable, and whether their combination might benefit the task of lexical similarity. specifically, we compare to the well-known static word embedding approach fasttext @xcite , and an approach that extracts contextualized representations of words from a pretrained bert language model @xcite , which we call "bert." we also compare to "ares", a recent synset/sense embedding model @xcite ) that builds representations for each synset by collecting relevant contexts and extracting contextual embeddings of lemmas belonging to each synset from a pretrained language model (bert). similar to @xcite , we rely on ares synset embeddings for our multilingual analysis. to compute ares word embeddings, we follow equation ( 1 ), but use the pretrained ares multilingual synset embeddings foot_5 . we re-implement the bert baseline foot_6 , but use pretrained word embeddings for fasttext foot_7 . ## experimental setup lexical semantic similarity (lsim) seeks to accurately measure the perceived similarity in meaning between two words and does so by the spearman's rank correlation foot_8 between similarity scores of human judgments and those computed automatically (cosine similarity of the words' vector representations). we rely here on multi-simlex @xcite , arguably the most comprehensive semantic similarity evaluation resource to date, which contains monolingual lists of 1,888 word pairs, with aligned concepts in 13 typologically diverse languages 10 . diverse criteria were used here to test whether two words are semantically similar, and not vaguely associated. thus, the multi-simlex datasets could be used as "an intrinsic evaluation benchmark to assess the quality of lexical representations based on monolingual, joint multilingual, and transfer learning paradigms" @xcite . of the original 13 languages, we limit our ## results and discussion we introduce here various experiments to compare multiple methods. since bert and fasttext can form representations from arbitrary string inputs, they have no out-of-vocabulary (oov) words, while all methods relying on external resources (ares and colex) require synset information for any word included in our experiments. for fair comparison, we limit our study to word pairs that include both words in the vocabulary of all approaches. we also use heatmaps (correlation plots) to give a clearer visual overview of the correlation between human judgments (i.e., gold standard) and our models' outputs, by word pair similarity rank (an ordinal measure) for all pairs in the test sets. figure 1 shows the overall correlation plots for all the test sets combined comparing our dictonary models (colex mono , colex cross ), the best baseline (fasttext), and our best fusion model (colex cross +fasttext). the color intensity of a square region corresponds to the number of word pairs in that region. we analyze the results of these different experiments next. ## suggestions for future improvement future work can expand our large-scale study of constructing synset and word representations from cross-lingual colexification principles in a number of directions. first, our cross-lingual embedding models seem specifically useful at ranking highly similar words just by amassing a large number of colexified synset pairs from many of the world's languages. however, while some colexification patterns might show more universal tendencies, others are very specific to a geographic area or language family, while others are more unique, identifying isolated cases of homonymy or other non-similarity phenomena. one possible solution is to represent as edge weight the number of languages that have a colexification pattern between two given synsets. this might result in a stronger model to identify either generalizations or more specific areal patterns (like language contact) by zooming out or in various areas of the graph, depending on one's research interests. our model of semantic similarity does not distinguish the degree of similarity captured by each colexified synset. figuring out a way to remove semantic links between colexified synsets that are only weakly or historically related may lead to higher quality synset and word representations capturing universal semantic tendencies, and thus run less the risk of an ethnocentric bias in favour of a specific language/area. since languages can be compared at various levels of linguistic organization, it would be interesting to empirically investigate how colexification patterns involving core vocabulary differ in their genealogical stability compared with patterns at the periphery of the lexicon @xcite . unlike fasttext and bert, which are fully unsupervised, our proposed approach relies on external resources (babelnet) for lexeme and synset information. moreover, babelnet is rather skewed in geographical coverage, typological diversity, and size of vocabulary across languages. from a sociolinguistic perspective, most of the babelnet coverage comes from socio-politically dominant modern languages, even heavily anglocentric (i.e., very rich, fine-grained distinctions of english lexicalizations). it would, thus, be interesting to test the efficacy of our model on a more balanced set of languages (as well as number of lexemes and synsets) from a more diverse (sub)set of language families. for our semantic similarity evaluation, we relied on multi-simlex whose perception ratings of wide coverage lexical words were determined in an out-of-context fashion via human subject questionnaires, and through translation from english. norm-generating studies involving large number of words have become increasingly popular across the cognitive sciences particularly due to their ability to provide greater statistical power, reduce experimenter bias in item selection, and increase study reliability @xcite . thus, correlation plots which intend to capture the relative strength of different colexification patterns are, in fact, an exploratory method and do not represent an attempt at rigorous hypothesis testing @xcite . a comparison of out-of-context vs. incontext judgments and of differences between universal vs. more culturally-specific types of knowledge would advance research in lexical semantics. ## conclusions this paper contributes to the investigation of lexico-semantic structures in the lexicon as manifested by colexification patterns captured in large synset graphs across babelnet's diverse set of 499 world languages. we introduced several approaches -monolingual and cross-lingual colexification graphs, popular distributional vector space models, as well as a fusion of such systems. we evaluated the extent to which these models correlate with human judgments on a semantic similarity task covering 9 typologically diverse languages. a deep analysis of the semantic similarity -relatedness/association continuum is not only important for research in lexical semantics and typology, but can also benefit a range of language understanding tasks in nlp. our large scale cross-lingual colexification graph investigations highlight an important contribution: our word representation approach relies on synset embeddings across languages as captured in colexification graphs, and thus, no adaptation of such word embeddings is necessary for cross-linguistic comparisons (i.e., there is no need for mapping monolingual embeddings to a shared bilingual vector space). we have also tested and validated our cross-lingual colexification models (tables 3 and 6 ) on the clsim task @xcite . the findings of our exploratory investigation of large-scale collexification graphs could inspire significant advances in nlp across languages, especially for tasks involving languages which lack dedicated lexical resources, and can benefit from language transfer from multilingual repositories.