Aligning Alignments: Do Colexification and Distributional Similarity Align as Measures of cross-lingual Lexical Alignment? (2024)

Abstract: The data-driven investigation of the extent to which lexicons of different languages align has mostly fallen into one of two categories:colexification-based and distributional. The two approaches are grounded in distinct methodologies, operate on different assumptions, and are used in diverse ways.This raises two important questions: (a) are there settings in which the predictions of the two approaches can be directly compared? and if so, (b) what is the extent of the similarity and what are its determinants? We offer novel operationalizations for the two approaches in a manner that allows for their direct comparison, and conduct a comprehensive analysis on a diverse set of 16 languages.Our analysis is carried out at different levels of granularity. At the word-level, the two methods present different results across the board. However, intriguingly, at the level of semantic domains (e.g., kinship, quantity), the two methods show considerable convergence in their predictions.A detailed comparison of the metrics against a carefully validated dataset of kinship terms shows that the distributional methods likely capture a more fine-grained alignment than their counterpart colexification-based methods, and may thus be more suited for settings where fewer languages are evaluated.

## introduction to what degree do translation equivalents in different languages -for example, english red and french rouge -encode the same meaning? this question, in various forms, has long been a topic of interest in the cognitive sciences @xcite @xcite @xcite @xcite . indeed, lexicons are often viewed as reflecting the structure of human cognition; understanding how meaning is expressed across lan1 our code and data is available at https://github.com/ tai314159/aligning_alignments . figure 1 : colexification graph for the target concept "steigen::v" (which corresponds to the word rise in english, and nousta in finnish. each vertex corresponds to a concept that is colexified with the target concept either in english or finnish. the english lexicalizations of the target concept are in black and the finnish lexicalizations are in blue. the concepts themeselves are in green. each edge (marked by an arrow) denotes that a colexification exists in english/finnish (as labeled). a building block in answering such a question is the ability to evaluate the similarity between words that seemingly express a similar meaning (henceforth, translation pairs) across different languages. traditionally, in linguistic and cognitive research, comparing the meaning of words across languages involves methodologies and approaches that are less data-driven in nature, prioritizing in-depth, relatively small-scale exploration of meaning, such as descriptive comparisons @xcite , elicitation studies @xcite @xcite @xcite and semantic maps @xcite . the difficulty in defining lexical similarity between concepts, let alone translation equivalents, has motivated a transition from theoretical frameworks to data-driven approaches. indeed, a significant amount of recent works has focused on using data-driven methods to measure the equivalence of word pairs across different languages @xcite @xcite @xcite @xcite . all work on this question inherits an even more fundamental set of questions: how is meaning defined and how is the meaning of words captured? within this rich body of work, we can identify two main methodological approaches. the first approach is based on colexification patterns, which aims to compare the association between lexical form and senses across languages. colexification is the case where two or more concepts are lexicalized with a single form in a given language @xcite (see the second approach is based on distributional word embeddings (here, dista). this approach was recently proposed as a viable data-driven method for cross-lingual lexical semantic investigations @xcite @xcite @xcite , for improving cross-lingual transfer @xcite and for investigating multicultural knowledge in llms @xcite . while all distributional methods use the word embeddings of translation pairs for computing similarity, many different operationalizations of this general approach are possible. see §2.1. both approaches have had a substantial impact on the computational cognitive science literature @xcite @xcite . these approaches seek to reveal an abstract structure that underlies the relation between words and their meanings (e.g, languages from different language families might have the same structure of kinship terms). however, while both are data-driven and aim to capture similar phenomena, they rely on different data and methodologies, and in fact likely capture different aspects of linguistic meaning. colexification-based approaches set out to quantify similarity in lexicographical resources, while distributional embeddings use any signal that can be reliably extracted from the data. for example, dista may not represent rare senses, while colexification does not take frequency into account at all. they are also applied differently: colexification-based approaches often constructs intricate cross-lingual networks to explore meaning universality @xcite , while distributional alignment methods operate at the word level and can then be extended to larger word sets @xcite . in this work we seek to empirically compare the predictions of these two approaches. however, given the divergence in methodologies and underlying assumptions adopted by these various approaches, it is not clear if it is sound, or even possible, to compare them. moreover, obtaining a meaningful signal from colexification data typically requires aggregating information across thousands of languages @xcite and is rarely used for analysis at the word-pair level; instead, its strength lies in the analysis of intricate networks. therefore, working with a substantially smaller set of languages or even comparing a single language pair at a time, as is often the case in multilingual nlp research, requires adapting the approaches so they will yield compa-rable predictions. we ask whether these distinct approaches converge at interface settings -settings in which the two approaches offer coherent similarity measures that can be compared. we show that such cases of convergence exist ( §5) and, in these cases, ask whether -and when -the different approaches yield similar predictions. this is, to the best of our knowledge, the first time that these questions have been tackled within nlp. 3 analysis at various levels of granularity reveals that at the word-level, the two methodologies yield different results across the board. however, at the domain-level foot_2 , the trends presented by the two methods show substantially higher correlation. in general, there is an overall greater similarity across different distributional methods than between the two families of approaches, in terms of their predictions and the factors that influence them ( §5). moreover, while distributional methods are correlated in their alignment predictions with external similarity measures ( §5.4), the colexification approach is not. this suggests that the distributional approach captures more fine-grained aspects of meaning and is better suited for either delicate analysis of the results or when using a smaller set of languages. also, the domain-level might be a more robust level to report alignment than the word-level. additonally, we find that rate of lexical change is a significant predictor for cross-lingual alignment, across all methodologies. we discuss the implications of these results in §7. to recap, we (i) operationalize distributionbased and colexification-based approaches so as to enable a direct empirical comparison between them, (ii) perform in-depth comparison of different operationalizations of the two approaches, (iii) study the ramifications of different design choices that they incorporate. ## cross-lingual lexicon alignment much research on cross-lingual alignment between lexicons has sought to uncover whether certain concepts, notably in domains perceived as basic to the human experience, such as space, time, color, quantity, and family relations, are universal, on the one hand, or culturally-or historicallycontingent, on the other hand @xcite @xcite @xcite @xcite . alignment can either be defined with respect to individual words (i.e, word-level alignment) or with respect to domains (i.e, domain-level alignment). for example, we might expect the word sunday in english not to align well with the hebrew multiword expression denoting the same day of the week yom rishon, as the latter does not bear any of the religious connotations of sunday in english. the degree of their alignment is a word-level alignment. one can also compare the extent to which the concepts of time align more generally, in which case we might expect hebrew and english to be relatively similar, given that hebrew, spoken in israel, prima facie has a western conception of time, with, for example, a division of the year into twelve months, a division of the week into seven days, and so on. this is termed domain-level alignment. ## experimental setup we briefly describe our experimental setup, with full details in appendix §a. data & languages. we perform our analysis on a diverse set of 16 languages, spanning 7 different language families from many geographical areas across eurasia (see figure 3 ): english, french, italian, german, dutch, spanish, polish , finnish, estonian, turkish, chinese, korean, japanese , hebrew, hindi and arabic. the lexicon used in our analysis consists of 1,016 concepts sourced from northeuralex (nel) @xcite , a comprehensive linguistic resource containing these concepts with their word forms in 107 different languages. we map the concepts in nel to domains, using concepticon. 6 there are 20 domains (e.g, animals, kinship; full list is in appendix §a), each containing 22 -136 concepts. ## alignment metrics we now turn to presenting the metrics we use in the paper. each metric either follows the distributionbased alignment (dista) or the colexificationbased alignment (colexa) approach. for dista we follow the metrics and notations outlined in @xcite . notation. let c be the set of concepts in the nel dataset @xcite . we adopt the notion of a concept from the lexical typology literature (e.g., @xcite , and take it to mean a word sense defined independently of any specific language. let ω be a set of languages. a language @xmath0, and may lexicalize several concepts with one word (colexification). we denote the lexicon corresponding to c in a given language l with l, and note that |l|≤ |c| for every language. we assume that c is partitioned into domains, and denote the (non-overlapping) domains with d 1 , . . . , d m . given a concept @xmath1, we denote its lexicalization (the word expressing that concept) in language l with r l (c) ∈ l. a translation pair between languages l 1 and l 2 is a pair of words (w 1 , w 2 ) ∈ l 1 × l 2 , such that there exists @xmath2. for example, the concept song gives rise to the english-french translation pair (song,chanson). in principle, several translation pairs may correspond to a concept and language pair, but in the data we experiment with, this does not occur. for a given word w in a given language l, we denote its embedding with emb(w, l). we denote the embedding space corresponding to l with ℓ. ## qualitative analysis to further understand the nature of alignment and convergence of the various approaches, we manually examine data from four randomly-selected languages pairs (english-german, german-arabic, arabic-hebrew and spanish-hindi); specifically, for each method and language pair we take the top/bottom aligned 100 words, together with their 10 nearest neighbors in each language (for colexa we consider colexifications instead of neighbors). even within the most aligned domains, there is variability in the order of aligned words (e.g., in dista-cloud numbers such as seven and fifty are the most aligned, whereas in dista-static it is months, such as march). however, words in highly aligned domains tend to greatly overlap in their neighbors, and somewhat preserve their order of distances. it is difficult to draw conclusions at the wordlevel just by looking at the raw data (this is also reflected in our empirical analysis in disagreement between the methodologies, §5.1). this is especially true for colexa or for the least aligned words. we do find, however, that certain words exhibit highly consistent colexification patterns across languages. for instance, the word fingernail frequently colexifies with the word nail. based on this analysis, we hypothesize that words that colexify conceptualy similar senses (e.g., fingernail and nail/hand and claw/etc.) tend to have more universal colexification patterns and in turn more aligned (this echoes the finding that conceptual similarity shape colexification @xcite ), and that this is also reflected by high alignment in dista as this type of polysemy is less prone to affect the dissimilarity of neighbors across languages. conversely, when two distinct senses are colexified (e.g., bank in english colexifies a sense of financial institution and a sense of terrain), the neighbors are likely a mix of words relating to each sense, leading to lower distributional alignment. ## discussion distribution-based and colexification-based approaches both capture a data-driven notion of similarity between the lexicons of different languages. however, they rely on different methodologies and assumptions about the data that should be accounted for, and are commonly applied in distinct ways. this raises the question of whether they are comparable, and if so -whether their predictions converge. we find that despite the inherent differences between the methods, when viewed at the level of domains, the two appraoches show similar trends. we also find that the rate of lexical change is a strong predictor for alignment, words that change less have more stable meaning across languages. in contrast to colexa, dista is significantly correlated with extrinsic measures for meaning alignment across languages. a possible explanation is that colexa captures coarser aspects of meaning or that it is more suitable for scenarios which require aggregation across a more extensive range of languages. we still find this resource highly valuable, especially for investigations of high-level patterns of lexical similarity (e.g., variation in emotion concepts over the worlds languages @xcite ), since it is less prone to noise stemming from the training data than dista. however, for a more fine-grained analysis or when less languages are available, we encourage the use of dista. in this paper we lay the ground for a direct comparison of dista and colexa. our findings call for a more nuanced discussion of lexical alignment, and also underscore the importance of taking into account multiple approaches for similarity when drawing empirical conclusions about lexical similarity. different approaches and settings may well lead to different conclusions, which highlights the importance of justifying the technical approach taken in each paper.