High-Dimensional Interlingual Representations of Large Language Models (2025)

Abstract: Large language models (LLMs) trained on massive multilingual datasets hint at the formation of interlingual constructs–a shared region in the representation space. However, evidence regarding this phenomenon is mixed, leaving it unclear whether these models truly develop unified interlingual representations, or present a partially aligned constructs. We explore 31 diverse languages varying on their resource-levels, typologies, and geographical regions; and find that multilingual LLMs exhibit inconsistent cross-lingual alignments. To address this, we propose an interlingual representation framework identifying both the shared interlingual semantic region and fragmented components, existed due to representational limitations. We introduce Interlingual Local Overlap (ILO) score to quantify interlingual alignment by comparing the local neighborhood structures of high-dimensional representations. We utilize ILO to investigate the impact of single-language fine-tuning on the interlingual alignment in multilingual LLMs. Our results indicate that training exclusively on a single language disrupts the alignment in early layers, while freezing these layers preserves the alignment of interlingual representations, leading to improved cross-lingual generalization. These results validate our framework and metric for evaluating interlingual representation, and further underscore that interlingual alignment is crucial for scalable multilingual learning.

## introduction interlingua, a universal language-neutral representation, is pivotal for cross-lingual generalization. grounded in both linguistic theories and computational practice, this concept aims to treat languages equitably and capture universal semantic structures independent of any specific language @xcite @xcite ; rayner et @xcite . the advent of llms trained on extensive multilingual corpora suggests the potential of interlingual constructs naturally emerging without any explicit objectives @xcite @xcite . this is attributed to their ability to map representations from different languages into a shared multilingual representation space @xcite @xcite @xcite . however, evidence remains mixed on whether they converge all language-specific representations into a unified single interlingual representation space, and raising questions about whether llms can retain the interlingual representations in diverse linguistic typology, geographical distribution, and resource-level settings. it is unclear whether llms form a unified interlingual construct or if fragmentation occurs across different language groups. a critical question persists: do llms develop a uni-versal interlingua representation, or present a partially aligned construct toward certain languages? our preliminary experiments reveal that llms represent parallel semantic input differently across languages. notably, their neuron activations align better within high-resource pairs and the same familial or regional roots, suggesting that llms exhibit varying alignment consistencies across differing language groups. building upon these insights, we introduce a novel interlingual representation framework aimed at enhancing the understanding of how llms encapsulate interlingual semantics. our framework identifies both the core region that captures shared semantics across languages, and addresses fragmented components due to representational limitations underscoring the importance of interlingual alignment across diverse linguistic contexts. with the framework, we introduce a novel metric, interlingual local overlap (ilo), which quantifies intrinsic interlingual alignment consistencies by comparing the local neighborhood structures of high-dimensional representations. inspired by graph theory @xcite @xcite , the ilo score is derived from the harmonic mean of two measurements, on the extent to which representations of a given language within the multilingual space: individually neighboring diverse other languages (bridge) and collectively connect diversely with other languages (reachability). we demonstrate the effectiveness our framework and metric through an in-depth analysis of llms' internal states on a multilingual mathematical reasoning task, chosen for its language-agnostic properties. we first observe that training multilingual llms on a single-language causes catastrophic forgetting @xcite @xcite degrading their crosslingual generalization @xcite . these degradations are correlated with the disruption of interlingual alignment that originate in the early layers of llms. to ensure the preservation of interlingual alignments, we adopt a strategy of selectively-freezing parameters during the single-language fine-tuning. evaluations using ilo highlight that this approach effectively safeguards the interlingual alignments across all layers and maintains the levels observed prior, which results in significant improvements in cross-lingual generalization. ultimately, our findings underscore the pivotal role of interlingual semantic alignment in the pursuit of scalable multilingual learning. 2 related works syntactical interlingua representations interlingua has played a huge role throughout the development of nlp. various representations of interlingua have been developed along with the advancement of nlp. in the early years, a logically formalized interlingua representation for mechanical translation has been proposed @xcite . in the early days, interlingua is presented as delexicalized grammar extracted from the original text that can be mapped to other language interlingua delexicalized grammar. in this case, each language has its own interlingua form which can then be mapped into other language with a dictionary lookup @xcite . a more sophisticated method involves interlingua representation as a common abstract syntax that are shared across all languages @xcite . this method has been applied in various systems such as spoken languge translator @xcite , parc's xle @xcite , and verbmobil @xcite . despite its advancement, this method tends to be incomplete and difficult to scale to new languages @xcite . ## interlingual representations in multilingual llms to explore the emergence of interlingual representation in llms, we assess the semantic alignment of their hidden states to understand whether the latent structures capture universal semantics across languages. we presume that multilingual llms adhere to a "first align, then predict" pattern @xcite and that their aligned states represent semantically similar features across languages. ideally, these features map parallel semantic inputs from many languages to similar vector representations that overlaps in the high-dimensional space. consider the high-dimensional representation space h ⊆ r d learned by llms, where d is the model's hidden-states dimension. for an input x in language ℓ, the model uses language-specific encoding functions f ℓ (x) ∈ h. here, h serves as a shared multilingual space where different encod-ing functions f ℓ (x) align semantic and syntactic patterns across languages. building on this, we define semantic alignment α of representations from parallel inputs x and x ′ in languages ℓ and ℓ ′ as: here, ϕ denotes a similarity function and d ℓ,ℓ ′ is the distribution of semantically equivalent input pairs. a higher α(ℓ, ℓ ′ ) indicates better alignment. ## semantic alignment of multilingual llms representations we explore the presence and characteristics of the components m c and m f within multilingual llms through assessing the semantic alignment between its hidden-states, derived from parallel inputs on various languages. initially, we project llms' internal hidden-state embeddings into a 2d space to broadly assess proximities of parallel language representations and observe whether parallel input pairs in different languages clusters or overlaps. we then measure the cross-lingual alignment across the parallel hidden-state embeddings through neuron activation consistency w.r.t their resource-level, linguistic features, and geographical region. we sample 31 diverse language subsets of flores-200 @xcite varied on its resource-level, region, and family (eberhard et al., 2024) (see tables 1 and a1) as proxies to typological and morphological features @xcite . over experiments, we assess several multilingual llms: aya expanse (8b) @xcite , llama-3.1 (8b) @xcite , gemma-2 (9b) @xcite , qwen-2.5 (7b) @xcite . we observe a universal phenomenon from these models, as described in the following sections. we put the further comparison details in appendix d. ## intrinsic interlinguality of llms in section 4, we empirically demonstrated that multilingual llms' behavior aligns closely with the theoretical framework introduced in section 3. linguistically-diverse semantically-parallel inputs, to indicate and quantify their intrinsic interlingual alignment in the high-dimensional space. ## conclusion the emergence of multilingual llms demonstrates that interlingual constructs naturally arise, even in the absence of explicit objectives. we introduce a conceptual framework to understand interlingual representations, identifying both the core interlingual region that captures shared semantics, and fragmented components that reveal representational limitations in aligning with this core region. to advance the understanding of interlingual semantic alignment, we propose the interlingual local overlap (ilo) score which quantifies alignment in the local neighborhood structures of interlingual high-dimensional representations. our proposed framework and metric illuminates the critical role of semantic alignment, offering a quantitative view into the high-dimensional alignment of multilingual representations. this study emphasizes interlingual semantic alignment and provides critical insights to optimize multilingual llms in the context of diverse linguistic tasks.