Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective (2024)

Abstract: NLP research on aligning lexical representation spaces to one another has so far focused on aligning language spaces in their entirety. However, cognitive science has long focused on a local perspective, investigating whether translation equivalents truly share the same meaning or the extent that cultural and regional influences result in meaning variations. With recent technological advances and the increasing amounts of available data, the longstanding question of cross-lingual lexical alignment can now be approached in a more data-driven manner. However, developing metrics for the task requires some methodology for comparing metric efficacy. We address this gap and present a methodology for analyzing both synthetic validations and a novel naturalistic validation using lexical gaps in the kinship domain.We further propose new metrics, hitherto unexplored on this task, based on contextualized embeddings. Our analysis spans 16 diverse languages, demonstrating that there is substantial room for improvement with the use of newer language models. Our research paves the way for more accurate and nuanced cross-lingual lexical alignment methodologies and evaluation.

## introduction cross-lingual lexical semantic similarity can be approached from two complementary perspectives: local and global. the local perspective compares words or sets of words and characterizes how similarly meanings are lexicalized across languages @xcite @xcite @xcite @xcite . for example, whether translation equivalents, such as the english green and french vert, encode the same meaning. in contrast, the global perspective focuses on how similar languages are as a whole, examining broader figure 1 : distribution-based alignment. an illustration of the distribution-based alignment process for the word rise between english (l 1 , upper left) and finnish (l 2 , upper right). first, we find the k nearest neighbors of the word rise in l 1 (for ease of visualization @xmath0. we then translate the neighbors to l 2 . the neighbors do not necessarily coincide with the nearest neighbors of the finnish translation of rise. here two of the nearest neighbors are not translations (marked in grey) and are ignored in the calculation). we then measure the correlation between the distance vectors. we repeat this in the opposite direction, l 2 → l 1 , and compute the average. patterns, relationships, and structures within languages, rather than focusing on individual words. for instance, english and bulgarian are both svo languages, although the latter has a more flexible word-order structure. such differences can influence their global alignment and the ability to transfer knowledge between them @xcite . in the field of natural language processing (nlp), most research on cross-lingual similarity focuses on global similarity measures @xcite @xcite , as they facilitate cross-lingual transfer in a multitude of tasks, such as machine translation and bilingual lexicon induction @xcite @xcite . these approaches typically involve aligning entire vector spaces of different languages through methods such as linear transformations, aiming to create a unified semantic space where languages can be directly compared. the local perspective, although often overlooked in nlp, has long been a topic of interest in cognitive sciences and linguistics @xcite @xcite @xcite @xcite . to the extent that lexicons reflect the structure of human cognition, understanding how meaning is expressed across languages offers insight as to how humans categorize and represent the world. examples of such inquiries span various semantic domains (a way of grouping words together based on common aspects of meaning or function), including colors @xcite and emotions @xcite . traditionally, in linguistic and cognitive research, comparing the meaning of words across languages involves methodologies and approaches that are less data-driven in nature, but rather prioritize in-depth, relatively small-scale exploration of meaning, such as descriptive comparisons @xcite , elicitation studies @xcite @xcite @xcite and semantic maps @xcite . however, defining and operationalizing the notion of cross-lingual lexical similarity is notoriously difficult and controversial. the difficulty in defining lexical similarity has motivated a turn from theory-driven to data-driven approaches. indeed, considerable recent work pursued data-driven approaches to the quantification of equivalency between word pairs in different languages @xcite @xcite . while some studies employed nlp methods @xcite @xcite , their application has been limited to static word representations and their validation have focused on converging evidence, i.e, getting the same (or similar) result in different means and looking for confounds. e.g, comparing to other tests from the cognitive science literature, such as picture naming @xcite or translation norms @xcite . however, these test are only ap-plicable to a very small set of languages and word lists. moreover, in nlp, it is common to compare to some external reference point that we perceive is one of high quality. for brevity, we simply refer to it as ground truth (section §5.2). in this work, we focus on the challenging task of word-level semantic similarity across languages, addressing the question of how word meanings vary across languages. we use a range of metrics designed to assess and compare the nuanced meanings of translation equivalents in different languages, and extend them in a novel way to include contextualized word representations (section 4.2). one of the main challenges in developing metrics to quantify differences in lexical semantics across languages is the difficulty to validate them, given that there are no available resources tailored to define ground truth in this area. to address this, we not only conduct extensive synthetic analyses of the metrics (section §5.1) but also establish a novel validation method, adapting a newly compiled linguistic resource of lexical gaps in the kinship domain (section §5.2; @xcite . we perform a detailed comparison between the various metrics at two levels of granularity: wordlevel and domain level. we also analyze what features affect the alignment, using a combination of lexical and environmental features. we show that at the domain-level there is substantial similarity between the methods and that it offers a more stable level for such analysis. moreover, while the contextualised embeddings (ce's) based metrics are substantially correlated with our naturalistic validation, the other methods are not, suggesting that there is definitely room for improvement in this area, using newer models. to summarize our contributions we (1) formulate the question of cross-lingual lexical similarity as an nlp task; (2) compare and analyze various metrics for this task; (3) introduce novel metrics based on contextualised word embedding; (4) offer a comprehensive validation suite to support our findings, including a novel validation method against a highquality linguistic resource specifically tailored to the kinship domain. ## related work a multitude of different approaches for computing distributional similarity have been explored in nlp, of which we select a number of representative examples. distributional metrics can be clas-sified based on whether they employ a joint space for the embeddings for the languages in question, or whether the spaces are trained monolingually and then aligned @xcite . 1 the latter approaches have been a key facilitator of cross-lingual transfer in nlp and are especially important in low-resource settings. however, for identifying patterns of divergence and convergence in the usage of specific words and domains, this approach is suboptimal. 2 globally optimal alignment (one that minimizes the distance between the image of one language in the space of another language) may distort the alignment of some words subsets, in the interest of improving the alignment of other, larger word sets. 3 local alignment, or the extent to which translation pairs like english home and spanish casa, hold a similar meaning across languages, is a well studied open question in cognitive science @xcite @xcite @xcite , that had only recently been approached with nlp tools @xcite . however, existing metrics are limited to static word embeddings and do not accommodate newer models that support contextualization. additionally, understanding the variability in meaning across languages can provide valuable insight into cultural differences, revealing how various societies conceptualize their unique experiences and worldviews @xcite @xcite . this line of research aligns with the recent surge in studies concerning multicultural knowledge in llms, which assess whether models like gpt variants or multimodal lms possess diverse cultural knowledge or exhibit biases favoring western cultures @xcite @xcite . 1 we are aware of one study of cross-lingual lexical comparison that used global alignment to project languages to a shared space, and defined the degree of alignment between a translation pair to be the distance of the image of one word to the embedding of the other @xcite . however, due to the reasons stated above, we do not consider their approach in this paper. ## experimental setup we provide a brief description of our experimental setup, with full details available in appendix §a. languages. we perform our analysis on a diverse set of 16 languages, spanning 7 different toplevel language families from many geographical areas across eurasia: english (eng), french (fra), italian (ita), german (deu), dutch (nld), spanish (spa), polish (pol), finnish (fin), estonian (est), turkish (tur), chinese (chn), korean (kor), japanese (jap), hebrew (heb), hindi (hin) and arabic (arb). ## cross-lingual lexicon alignment in this section, we lay the groundwork for crosslingual lexicon alignment, focusing on word-level semantic similarity between languages. we use local metrics @xcite @xcite , extend them to novel ones and provide a framework for comparing word meanings across languages. foot_4 ## analysis having established a set of metrics and an extensive validation suite, we now turn to analyzing the alignment results. this analysis is conducted at two levels of granularity: the word level and the domain level. we compare various metrics, identifying which words and domains are most and least aligned. furthermore, we examine factors affecting alignment, such as lexical properties and environmental features, to uncover the underlying causes of semantic divergence across languages. ## qualitative analysis we conduct a small scale qualitative analysis to further understand the results. 21 drawing conclusions at the word level solely from the raw data is challenging, as evidenced by the discrepancies between methodologies in our empirical analysis. we give a few examples of factors that impact the alignment. 22 the first is cultural differences. to illustrate, the word "pig" is less aligned between english and arabic than between english and german. manual examination of the data reveals that the nearest neighbors of "pig" in english and german include animals and food items, such as "butter" and "salt," whereas in arabic, they are exclusively animals. this difference might arise from the cultural context in islam, where pork is prohibited. another reason for divergence is illustrated by the word "soft," which is one of the least aligned words across languages. the neighbors of "soft" are highly varied and come from different semantic domains, even within the same language, making their connection to the word itself less immediate. for example, in english, some neighbors for the word "soft" are "red", "sea" and "hand," while in german, they include "meat," "beautiful" and "rich". this diversity in semantic association contributes to the lower alignment for words like "soft". 21 the qualitative analysis was conducted by one of the authors. the full details of the setup are in appendix §e. 22 analysis done for snc-cloud. differences in word senses also contribute to misalignment. for example, the word "brother" is less aligned between english and hebrew, which might be attributed to the homonymy in hebrew between the senses of "brother" and "hearth". this difference is reflected in their nearest neighbors. this issue is directly related to the influence of polysemy on alignment, which we address in appendix §f and leave to future work. ## discussion how and why languages vary in their use of words to carve up the semantic space has long been a question of central interest in cognitive sciences and linguistics, but has often been overlooked in nlp. recent advances in nlp allow addressing this gap, and performing such analyses well and at scale. in this paper, we formulate this question as an nlp task, and provide a methodology for computing the efficacy of different metrics in addressing the task. we use existing metrics and extend them to contextualized word representations. we evaluate the metrics across multiple scenarios, using both synthetic and naturalistic validation approaches. we observe consistent trends across all metrics and architectures: the rate of change is a strong predictor of alignability. additionally, internally structured domains such as time, quantity, and kinship show the highest degree of alignment across languages, consistent with @xcite . one of the major challenges in analyzing crosslingual alignment for individual words or domains is the lack of ground-truth data for validation. this impedes the comparison between different metrics, and thereby hinder progress on this task. to address this, we provide both synthetic and naturalistic validations using a newly created linguistic resource, based on the kinship domain. our validation shows that all the snc metrics we propose effectively capture cross-lingual semantic variability, with a slight preference for the metrics that are based on contextualized representations. in future work, we plan to leverage cross-lingual alignment to investigate its impact on cross-lingual transfer on various nlp tasks. additionally, we aim to use this approach to examine the extent to which llms encode cultural knowledge, a topic that has recently garnered significant attention @xcite @xcite .