C olex2 L ang: Language Embeddings from Semantic Typology (2023)

Abstract: In semantic typology, colexification refers to words with multiple meanings, either related (polysemy) or unrelated (homophony). Studies of cross-linguistic colexification have yielded insights into, e.g., psychology, historical linguistics and cognitive science (Xu et al., 2020; Brochhagen and Boleda, 2022; Schapper and Koptjevskaja-Tamm, 2022). While NLP research up until now has mainly focused on integrating syntactic typology (Naseem et al., 2012; Ponti et al., 2019; Chaudhary et al., 2019; Üstün et al., 2020; Ansell et al., 2021; Oncevay et al., 2022), we here investigate the potential of incorporating semantic typology, of which colexification is an example. We propose a framework for constructing a large-scale synset graph and learning language representations with node embedding algorithms. We demonstrate that cross-lingual colexification patterns provide a distinct signal for modelling language similarity and predicting typological features. Our representations achieve a 9.97% performance gain in predicting lexico-semantic typological features and expectantly contain a weaker syntactic signal. This study is the first attempt to learn language representations and model language similarities using semantic typology at a large scale, setting a new direction for multilingual NLP, especially for low-resource languages.

## introduction semantic typology studies cross-lingual semantic categorization @xcite . the term "colexification", which encompasses both polysemy and homophony, was introduced to the field of semantic typology by @xcite . this study focuses on cross-lingual colexification patterns, where the same lexical form is used in distinct languages to express multiple concepts. for instance, bla in monpa changprong and afu in rikou both express the concepts dust and ash @xcite . colexification was first used in linguistic typology to create semantic maps. @xcite created a semantic map with 12 languages, and franc ¸ois @xcite pointed out that the number of different senses increases with the number and variety of languages used. in recent years, big data, and improved data creation and curation techniques have led to the development of datasets like concepticon @xcite , and babelnet @xcite , which make large-scale cross-lingual semantic comparisons possible. the cross-linguistic colexifications (clics) database was created based on the concepticon collection and is being continuously maintained. the current version foot_1 clics 3 includes 4,228 colexification patterns across 3,156 languages. in this paper, we create a synset graph based on multilingual wordnet @xcite data from babelnet 5.0, compare it with the concept graph extracted from clics 3 , and explore the impact of data scope on language representation learning. we hypothesize that language representations learned using semantic typology encapsulate a distinct language signal, and the data size of colexifications has an impact on the learned language representations and the modelled language similari-ties. importantly, we expect that this type of signal can be used to improve semantically oriented downstream tasks in nlp. to test this hypothesis, we propose a framework colex2lang (cf. section 3) to learn language representations leveraging semantic typology, conduct typological feature prediction, and model language similarities. our experiments on typological feature prediction focus on the domain of semantic features, so as to investigate the extent to which a semantic signal is encapsulated by our language representations. specifically, we make the following contributions: (i) we generate and evaluate 24 sets of language embeddings based on large-scale colexification databases, using four advanced node embeddings algorithms, i.e., node2vec @xcite , prone @xcite , ggvec foot_2 , and glove foot_3 @xcite ; (ii) we conduct thorough experiments on typological feature prediction to compare colexificationinformed and more general language embeddings @xcite , which provides a strong benchmark for further research; (iii) we demonstrate the usability of modelling language similarities based on colexification patterns, and argue for the potential of utilising semantic typology in nlp applications. ## related work colexification cross-linguistic colexifications were first formalized by franc ¸ois (2008) for the creation of semantic maps. semantic maps represent the relation between recurring meaning expressions in a language graphically @xcite . the basic idea underpinning this method is that language-specific patterns of colexifications indicate semantic closeness or relatedness between the meanings that are colexified @xcite . when investigated cross-lingually, colexification patterns can provide insights in various fields, such as recognizing cognitive principles @xcite @xcite @xcite , diachronic semantic shifts in individual languages @xcite @xcite , and the evolution of language contact @xcite @xcite . @xcite investigated cross-lingual colexifications in the domain of emotions and found that languages have different associations between emotional concepts. for example, persian speakers associate the concept of grief with regret closely whereas dargwa speakers associate it with anxiety. the cultural variation and universal structure shown in the emotion semantics provide interesting insights into nlp. @xcite used colexification patterns to test whether the words linked by colexification patterns capture similar affective meanings, and subsequently expanded affective norms lexica to cover exhaustive word lists when additional data are available. inspired by @xcite , @xcite proposed emotion semantic distance, measuring how similarly emotions are lexicalized across languages, to improve cross-lingual transfer learning performance on sentiment analysis. @xcite show that there exists no universal colexification pattern by analyzing colexifications from babelnet, open multilingual word-net @xcite , and clics 3 . closely related to our work, harvill et al. (2022) constructed a synset graph from babelnet to improve performance on the task of lexical semantic similarity. instead of modelling only word similarity using colexification patterns, we strive to model language similarity in this study and show that the language embeddings learned on colexification patterns capture a unique semantic signal compared to language embeddings encapsulating syntactical signals. moreover, we experiment with different node embedding algorithms and compare three colexification datasets. the framework provides a strong benchmark for further investigating how semantic typological aspects of language embeddings can be leveraged in broader applications, especially for low-resource multilingual nlp. node embeddings node embeddings can be broadly classified into three different categories namely (i) matrix factorization-based models, (ii) random walk-based models, and (iii) deep neural network-based models, as discussed in @xcite . in matrix factorization-based models, an adjacency matrix is used to denote the topology of a network. matrix factorization techniques, such as singular value decomposition (svd) and non-negative matrix factorization (nmf), can be applied to address this problem. grarep @xcite considers k-hop neighbourhoods utilizing svd of the adjacency matrix. this model often only captures small-order proximity and has a significant computational complexity for large graphs. the asymmetric transitivity is preserved by the hope @xcite model as it converts the problem to a generalised svd problem reducing the complexity. prone @xcite introduces a sparse matrix factorization to achieve initial node representations efficiently. random walks are used to maintain local neighbourhoods of nodes and their attributes @xcite , by increasing the likelihood of a node's neighbourhood given its embedding using the skip-gram model @xcite . the objective behind these models is to optimize via stochastic gradient descent on a singlelayer neural network, resulting in decreased computing complexity. deepwalk @xcite randomly chooses a node and proceeds to walk to each neighbouring node until it reaches its maximum length (or some random length). line @xcite aims to embed nearby vertices that either have linkages between them (optimizing for first-order proximity) or have a shared 1hop neighbourhood (optimizing for second-order proximity). node2vec @xcite proposes a second-order random walk approach to sample the neighbourhood nodes with biasing parameters of breadth first search (bfs) and depth first search (dfs). a meta-strategy for graph embedding under recurrent construction of nodes and edges into condensed graphs with the same global structure is proposed by harp @xcite . these graphs serve as source initializations for the detailed graphs that are embedded, producing appropriate node and edge embeddings as a consequence. metapath2vec @xcite is an extension of deepwalk that formalizes meta-path-based random walks to build a node's neighbourhood, then uses a heterogeneous skip-gram model. ggvec algorithm directly minimizes distances between the related nodes and is designed for large networks. it uses negative sampling followed by minimization loss to learn the node embeddings based on the minimal dot product of edge weights. another node embedding model follows the word embedding model glove @xcite which is based on word cooccurrences and is beneficial for sparse matrices. the graph is represented by an adjacency matrix and the co-occurrence matrix is calculated using the frequency of node co-occurrences in the graph instead of word co-occurrences. typological feature prediction linguistic typologists analyse languages in terms of their structural properties @xcite . as documenting and categorising such cross-lingual variation across the languages in the world is one of the core activities in typology, one of the outcomes of research in linguistic typology is large typological databases (e.g. the world atlas of language structures (wals, @xcite ). while such variation can be found across the spectrum of languages, the earliest work in the field largely focused on morphosyntactic properties (e.g. greenberg (1957)), concretely looking at minimally meaning-bearing elements (morphemes), combinations thereof, and patterns of their use. for instance, well-documented features include word ordering (e.g. english is svo, japanese is sov) and affixation (german uses case suffixes, berber uses case prefixes). prediction of such features has gained interest in recent years @xcite @xcite , and it has been shown that embeddings trained solely from tasks such as machine translation @xcite or language modelling @xcite can encapsulate such features. further analysis has shown that the nature of the underlying data used to generate language embeddings can have a significant impact on what features are encapsulated @xcite , and even that such representations contain typological generalisations @xcite . previous work is limited in that it almost exclusively relates to morphosyntactic typological features. in this work, we aim to present initial evidence that a lexico-semantic signal can be better learned from a lexico-semantic data source. ## colex2lang to better understand and leverage semantic typological features in nlp, we propose a framework -colex2lang (fig. 1 ) -to model language representations based on a synset graph, created from large-scale databases, and evaluate and analyse the language representations. the framework colex2lang is composed of the following steps: building the synset/concept graph we use wordnet synsets, extracted from babel-net 5.0, to create a synset graph. the construction of a synset graph is formalized in @xcite (see details in appendix a). in babelnet, every synset is either a concept or a named entity or has no type. the dataset with only concepts and with all types of synsets from wordnet are created, denoted as "wordnet concept" and "word-net" respectively. in analogy, clics 3 provides a graph of concepts, from which we extracted the colexification patterns for all the languages having an iso 639-2 code 5 , denoted as clics. in this study, we use "concept" and "synset" interchangeably. the statistics of the curated datasets are shown in ## analyses and results comparing language embeddings as indicated in as shown in capturing lexicon typological features to better understand how the colexification-informed language embeddings better capture semantic typological information, we analyze the performance of lexicon feature prediction with several representative examples, as visualized in figure 2 . the left side of figure 2 presents the performance of clics (best) model and the corresponding random model in predicting each feature (e.g., number of basic colour categories), the colour of the circle represents the feature values (e.g., 6-6.5 and 11), and the size of the circles indicates its proportion of the data samples for the regarding values in the train data (e.g., there are more data samples for the feature value "11" than "6-6.5"). overall, clics outperforms random in almost each feature value across lexicon features. in comparison, clics excels at the uneven distribution of train data samples. for instance, for features "number of non-derived basic colour categories" and "number of basic colour categories", the feature values "4.5" and "11" have fewer samples compared to their counterparts, while random cannot detect them, clics obtained 50% and 80% performance. similar results are shown for the wordnet-based models and their corresponding random model, as shown on the right side of figure 2 . for the feature "number of basic colour categories", both wordnet and wordnet concept models achieve the perfect score compared to the random counterpart, which is not able to identify the minority class at all. whereas, wordnet concept outperforms wordnet for the feature "number of non-derived basic colour categories", wordnet concept has a 100% macro f1-score with wordnet and random failing to identify the minority class. these results demonstrate that the models trained with colexification-informed language embeddings have learned to better capture the semantic typology information compared to randomly initialized embeddings. the language embeddings could be further fine-tuned and applied to assist other nlp applications. language similarities having attested that the colexification-informed language embeddings capture the semantic typological aspects of languages, we investigate how well the language similarities represented by the semantic typology features and the language embeddings correlate. to represent languages by lexicon features, we generate a vector for each language by encoding a 13-dimensional vector with the feature values padded with -1, if the feature value is absent. the cosine similarities among the vectors are calculated. similarly, the cosine similarities are calculated for the language embeddings. the pearson correlation coefficient and p-value foot_5 for testing non-correlation are calculated between the lan- a set of nordic and baltic languages are selected to compare further the represented language similarities. both wordnet-based language embeddings present strong positive correlations, i.e., 0.7679 and 0.8515, respectively, and the correlations are statistically significant, as shown on the right side of we differentiated the wordnet concept from wordnet dataset, assuming that a dataset with only concepts would avoid data noises and render language embeddings able to better capture the semantic associations between languages. however, the analysed results do not corroborate the assumption. on the contrary, the language embeddings learned on all the wordnet synsets present a stronger correlation (+0.02) with lexicon typological features. to further investigate language similarities, we apply pca to the wordnet-based language embeddings (figure 4 ). we can observe that, e.g., scandinavian languages are clustered together, as expected. another observation is that finnish is relatively close to this cluster, owing to a relatively high amount of overlapping colexification patterns from language contact with swedish, as compared to estonian which is placed closer to one of its contact languages, lithuanian. ## conclusion and future work in this study, we have proposed a framework colex2lang to leverage colexifications to learn language representations and explored the potential of using semantic typology in nlp. a large-scale synset graph is constructed using wordnet source from babelnet, and three datasets of colexification are processed including clics. subsequently, within each dataset, 24 language embeddings variants are learned, and further evaluated and analysed by typology feature prediction and modelling language similarity. we have demonstrated, at a large scale, that colexificationinformed language embeddings capture a distinctive aspect of languages in terms of semantic typology, and the data scope of the curated synsets affects the performance of applying language embeddings. furthermore, the analysis of representing language similarities by using learned language embeddings illustrates a realistic approach. a large body of research has demonstrated the use of syntactic, genealogical and geographical information from linguistic typology to learn language representations, model language similarities, and further improve transfer learning performance in downstream tasks in nlp. our work is the first attempt to learn language representations and model language similarity by leveraging semantic typology. the framework provides a strong benchmark for further research in this direction. for future work, the benefits of applying colexification-informed language embeddings will be extensively explored. multilingual semantic parsing is a clear candidate, where a cross-lingual signal based on colexifications may prove useful. the language similarities represented by colexifications could further inspire multilingual transfer learning, as in leveraging high-resource languages with dedicated lexical data to improve performance in semantically similar low-resource languages.