Exploring Alignment in Shared Cross-lingual Spaces (2024)

Abstract: Despite their remarkable ability to capture linguistic nuances across diverse languages, questions persist regarding the degree of alignment between languages in multilingual embeddings. Drawing inspiration from research on high-dimensional representations in neural language models, we employ clustering to uncover latent concepts within multilingual models. Our analysis focuses on quantifying the alignment and overlap of these concepts across various languages within the latent space. To this end, we introduce two metrics CALIGN and COLAP aimed at quantifying these aspects, enabling a deeper exploration of multilingual embeddings. Our study encompasses three multilingual models (mT5, mBERT, and XLM-R) and three downstream tasks (Machine Translation, Named Entity Recognition, and Sentiment Analysis). Key findings from our analysis include: i) deeper layers in the network demonstrate increased cross-lingual alignment due to the presence of language-agnostic concepts, ii) fine-tuning of the models enhances alignment within the latent space, and iii) such task-specific calibration helps in explaining the emergence of zero-shot capabilities in the models.

## introduction the emergence of multilingual contextualized embeddings has been a ground-breaking advancement, in the ever-evolving landscape of natural language processing. adept at capturing the linguistic nuances across different languages, these embeddings have spurred a multitude of studies @xcite @xcite seeking to understand the underlying mechanisms. how these models achieve multilinguality without explicit cross-lingual supervision during training is a particularly interesting question to answer. cross-lingual embeddings are designed to encode linguistic concepts that bridge equivalent semantic meaning across diverse languages. the question is: how well is this achieved in practice? when considering two arbitrary languages, how well aligned are the embeddings of those languages? and how language agnostic are these multilingual embeddings in reality? addressing these questions necessitates a comprehensive approach. in high-dimensional spaces, neural language models exhibit a capability to group words with shared linguistic associations, as highlighted by @xcite . expanding upon this foundational insight, recent research endeavors @xcite @xcite delved into conducting representation analysis within pre-trained models. our objective, in this work, is to uncover encoded concepts within multilingual models and analyze their alignment and overlap across various languages within the latent space. we discover latent concepts by applying clustering to the underlying contextualized representations. the premise is that these clusters potentially signify latent concepts, encapsulating the language knowledge assimilated by the model. we build our work on top of this foundation to quantify concept alignment and overlap within multilingual latent space. we propose two metrics calign and colap to quantify these two aspects and carry out analysis to study the following questions: we conducted a study employing three multilingual transformer models: mt5 @xcite , mbert @xcite , and xlm-roberta ( @xcite . these models were finetuned for three downstream nlp tasks: machine translation, named-entity recognition and sentiment analysis, spanning sequence generation, labeling and classification respectively. our analysis revealed intriguing insights, including: ## methodology the high-dimensional latent spaces learned within neural language models have been shown to encapsulate concepts formed by common linguistic attributes @xcite . our approach is rooted in this foundational insight where we discover latent concepts for interpreting representational spaces in multilingual neural language models. more precisely, our study endeavors to gauge the degree of alignment and overlap of concepts across the latent spaces acquired through training models on a diverse array of languages. to this end, we introduce two metrics to quantify these phenomena. the first metric calign (concept alignment) involves measuring alignment by identifying concepts that are semantically equivalent. this provides a nuanced understanding of how concepts in one language align with their counterparts in another, capturing the semantic coherence within the multilingual framework. our second metric colap (concept overlap) delves into investigating the existence of overlapping crosslingual latent spaces within the model's representation. this metric aims to highlight multilingual concepts that maintain multiple languages in a close latent space. by probing the shared latent spaces, we gain insights into the intricate relationships be-tween concepts across languages, contributing to a more comprehensive understanding of multilingual model representations, and how they evolve when the model is trained for specific tasks. figure 1 gives an overview of our approach. in the following sections, we detail each stage of our methodology. ## results and analysis cross-lingual representations are deemed to capture unified linguistic concepts across languages which enables them to generalize and to carry out the tasks for low resource languages and zero-shot scenarios. we use latent concept analysis of multilingual models to address the following questions: i) how latent space aligns and overlaps across languages in multilingual model? ii) how is the representation space calibrated as the model is tuned towards different downstream tasks? and iii) what impact does this re-calibration have on the alignment and overlap of concepts representing zero-shot languages? (which were not used for fine-tuning). ## related work numerous studies have explored the domain of multilingual embedding, investigating how deep neural language models encode knowledge across various languages without explicit supervision. pires et al. (2019) demonstrated mbert's ability to learn multilingual representations, enabling cross-lingual transfer even for languages with different scripts, (a) mbert base (b) mbert (ner) figure 8: quantifying concept overlap in mbert provided they share topological similarities. cao et al. (2020) employ a contextual word retrieval task where the model is tasked with finding corresponding words and sentences across parallel corpora. dufter and schütze (2020) identified critical architectural and linguistic properties for multilinguality, emphasizing the necessity of common positional embeddings, shared special tokens, and a restricted parameter space. papadimitriou et al. (2021) investigated higher-order grammatical feature representation across languages using probing classifiers trained on mbert embeddings. their successful zero-shot cross-lingual transfer demonstrated parallel representation of grammatical features. wen-yi and mimno ( 2023 ) conducted analysis on the embedding layer of mt5 and xlm-r, uncovering the diverse language encoding patterns within these models and highlighting the semantic encoding across languages. @xcite investigated the conceptual correspondence between structural concepts in linguistically diverse languages, emphasizing the correlation between conceptual alignment and cross-lingual transfer. they proposed a metalearning approach to align these linguistic spaces, enabling zero-shot and few-shot generalization. our approach diverges from prior research methodologies by using an unsupervised approach to unveil multilingual concepts learned within the latent space of these models. we identify latent concepts across different languages and assess alignment across these concepts using our proposed metrics calign and colap. unlike previous approaches that focus on individual words and local alignment, our multilingual concept analysis provides insight into how different linguistic concepts align and overlap across multilingual spaces. we illustrate the alignment and overlap within these spaces and track their recalibration as the models undergo fine-tuning for downstream tasks. while prior research often examines if individual words have aligned counterparts in target languages, our work extends this by enforcing whether the latent spaces themselves are similarly constructed. this means that the neighbors of a word in one language correspond to neighbors of the target word in another language, introducing a stronger evidence of multilinguality at a fundamental level. our findings suggest that this calibration of latent space enhances the model's performance in zero-shot scenarios, presenting a distinct analysis and revealing results that significantly differ from previous research. ## conclusion the emergence of multilingual contextualized embeddings has sparked interest in understanding their mechanisms. we introduce two metrics, concept alignment (calign) and concept overlap (colap), to quantify alignment and overlap within multilingual models. our analysis reveals: i) deeper layers exhibit increased alignment due to presence of semantic concepts, ii) fine-tuning enhances alignment across cross-lingual concepts, facilitating zero-shot capabilities, iii) divergent patterns in encoder and decoder spaces and higher overlaps between closely related languages are observed. our insights shed light on the dynamics of multilingual embeddings and lay the groundwork for a more comprehensive understanding of multilingual nlp models.