Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphs (2023)

Abstract: In comparative linguistics, colexification refers to the phenomenon of a lexical form conveying two or more distinct meanings. Existing work on colexification patterns relies on annotated word lists, limiting scalability and usefulness in NLP. In contrast, we identify colexification patterns of more than 2,000 concepts across 1,335 languages directly from an unannotated parallel corpus. We then propose simple and effective methods to build multilingual graphs from the colexification patterns: ColexNet and ColexNet+ . ColexNet’s nodes are concepts and its edges are colexifications. In ColexNet+, concept nodes are additionally linked through intermediate nodes, each representing an ngram in one of 1,334 languages. We use ColexNet+ to train ColexNet+, high-quality multilingual embeddings that are well-suited for transfer learning. In our experiments, we first show that ColexNet achieves high recall on CLICS, a dataset of crosslingual colexifications. We then evaluate ColexNet+ on roundtrip translation, sentence retrieval and sentence classification and show that our embeddings surpass several transfer learning baselines. This demonstrates the benefits of using colexification as a source of information in multilingual NLP.

## introduction multilingual representations are beneficial in natural language processing (nlp) due to their ability to transfer knowledge across languages @xcite @xcite . typically, such representations are learned through pre-training large language models (llms) @xcite @xcite or multilingual word embeddings @xcite @xcite . however, llms require enormous amounts of data to train, limiting their use mostly to high-resource and medium-resource languages @xcite . alternatively, multilingual word embeddings are widely used in nlp because of their simplicity and good performance @xcite @xcite . however, most existing multilingual embeddings are learned through word-context information, without leveraging global cooccurrence information in individual languages or across languages, which can help distinguish distinct meanings conveyed by a lexical form. therefore, we see a pressing need in nlp for massively multilingual word embeddings that span a large number of languages (1,335 in our case) and that specifically account for global occurrence and are a good basis for crosslingual transfer learning. colexification has gained increasing attention in comparative linguistics and crosslingual nlp. according to @xcite , a language colexifies two distinct meanings if it expresses them with the same lexical form. different languages have different colexification patterns. for example, while english has separate words @xmath0, russian 'рукa' colexifies these two concepts. most prior work explores colexification @xcite @xcite using manually curated crosslingual datasets that consist of multilingual word lists such as clics @xcite @xcite . however, relying on these datasets has several limitations: extension to more languages and more concepts can be challenging; these datasets contain lists of lemmata and (in a corpus-based approach for low-resource languages without morphological resources) cannot easily be used for the processing of occurrences in context. to overcome these limitations and boost crosslingual transfer learning especially for low-resource languages, we use the parallel bible corpus (pbc) @xcite , which has verse-level aligned translations of the bible in 1,335 languages, to identify colexification patterns (a verse in pbc roughly corresponds to a sentence). with the identified patterns between a wide range of concepts, we propose novel algorithms that efficiently build large-scale multilingual graphs. to the best of our knowledge, this is the first work that constructs graphs of colexification and trains multilingual representations for crosslingual transfer learning directly from a parallel corpus on a large scale. we show that the graphs capture the links between concepts across languages and that the derived multilingual representations considerably improve crosslingual transfer on downstream tasks. previous work on building monolingual graphs @xcite or multilingual graphs @xcite @xcite is different as it (1) does not consider words in context and only uses lemmata, (2) is based on external sense inventories such as wordnet @xcite and @xcite , which are not available for many low-resource languages, and (3) does not investigate the crosslingual transferability of the multilingual representations on nlp downstream tasks such as sentence retrieval or classification in a crosslingual scenario. the contributions of this work are as follows: (i) we present colexnet, a graph of concepts based on colexification patterns that are directly extracted from a parallel corpus. (ii) by extending colexnet, we further present colexnet+, a large-scale multi-lingual graph that additionally contains ngrams in 1,334 languages that instantiate those patterns. (iii) we contribute to crosslingual transfer learning, by using colexnet+ to generate multilingual embeddings: one of the best-known and widely used multilingual resources is babelnet @xcite . babelnet has been used for learning or enhancing embeddings @xcite @xcite @xcite for lexical-level tasks such as semantic word similarity and word sense disambiguation @xcite @xcite . our focus is on the coverage of many more languages (i.e., larger scale in terms of languages) for crosslingual transfer learning. while hand-curated lexica often have better quality than automatically learned resources, they are not available for most of our languages. ultimately, the two approaches should be combined. colexification was introduced by @xcite in the context of grammatical semantics. @xcite then used colexification as the foundation for studying semantic change crosslinguistically. clics @xcite @xcite is a crosslingual database that facilitates research on colexification. languages can differ in their colexification patterns, which are influenced by many factors such as human cognition, language family, and geographic area @xcite @xcite ). an empirical study by @xcite indicates that no pair of concepts is colexified in every language. on the other hand, a recent investigation on conceptualization from the pbc shows some concepts are more likely to be involved in colexification than others @xcite . such universal colexification patterns across languages reflect crosslinguistic similarities @xcite . therefore, by integrating colexification patterns of as many languages as possible, we can generate multilingual representations that are suitable for massively crosslingual transfer learning. ## experiments to evaluate our proposed methods, we conduct the following experiments: (1) colexification identification; (2) roundtrip translation; (3) verse retrieval; and (4) verse classification. experiment (1) evaluates the colexification patterns we identify in colexnet. experiments (2), ( 3 ) and ( 4 ) evaluate the learned multilingual embeddings -------→ colexnet+. ## conclusion in this work, we present the multilingual graphs colexnet and colexnet+, based on colexifications extracted from a highly parallel corpus. comparing with clics, we show that we identify many gold-standard patterns in colexnet. in addition, we analyze the structure of colexnet and show it nearly forms a scale-free graph, with many communities of semantically related concepts. most importantly, we contribute to crosslingual transfer learning by inducing multilingual embeddings -------→ colexnet+ that are learned on colexnet+. our experiments indicate that -------→ colexnet+ largely represents concepts across languages in the same semantic space. we show that -------→ colexnet+ outperforms several approaches, including multilingual embeddings and pretrained models, on three downstream tasks. this indicates that embeddings learned from colexification graphs improve crosslingual transfer, especially for low-resource languages for which it is often infeasible to pretrain good models. finally, our embeddings exhibit robust transfer performance across many different source languages.