Language Surgery in Multilingual Large Language Models (2025)

Abstract: Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across tasks and languages, revolutionizing natural language processing. This paper investigates the naturally emerging representation alignment in LLMs, particularly in the middle layers, and its implications for disentangling language-specific and language-agnostic information. We empirically confirm the existence of this alignment, analyze its behavior in comparison to explicitly designed alignment models, and demonstrate its potential for language-specific manipulation without semantic degradation. Building on these findings, we propose Inference-Time Language Control (ITLC), a novel method that leverages latent injection to enable precise cross-lingual language control and mitigate language confusion in LLMs. Our experiments highlight ITLC’s strong cross-lingual control capabilities while preserving semantic integrity in target languages. Furthermore, we demonstrate its effectiveness in alleviating the cross-lingual language confusion problem, which persists even in current large-scale LLMs, leading to inconsistent language generation. This work advances our understanding of representation alignment in LLMs and introduces a practical solution for enhancing their monolingual and cross-lingual performance.

## introduction large language models (llms) have revolutionized natural language processing, demonstrating remarkable generalization capabilities across diverse tasks and languages @xcite @xcite @xcite . their ability to adapt to new tasks in few-shot and even zeroshot settings highlights their efficiency and versatility @xcite . * equal contributions. see appendix k for further details. by exploiting this behavior, we are able to achieve inference-time language control (itlc), alleviating the language confusion problem in llms. prior works have identified a naturally emerging representation alignment across layers in llms, particularly in the middle layers of llms @xcite . this emerging alignment in llms is the key factor in their ability to handle multiple languages @xcite @xcite , which is pivotal for their cross-lingual capabilities. however, several questions remain open, such as whether this emerging alignment behaves similarly to alignment in models trained with enforced alignment objectives @xcite @xcite @xcite , how this alignment can be utilized to further enhance llms, etc. in this work, we investigate the phenomenon of representation alignment in llms, focusing on its occurrence, distinction, and potential applications. we aim to confirm the presence of representation alignment and contrast it with alignment in llms with strictly designed alignment, such as multilingual @xcite or @xcite . our findings highlight that, unlike llms with strictly designed alignment, the naturally emerging alignment in recent llms demonstrates a much stronger retention of language-specific information with much smaller performance drop in the aligned representation compared to the unaligned layers which we conjecture to be the minimum required languagespecific information required to perform do decoding in the correct language. to this end, we exploit the bottleneck of language-specific information in the aligned representation and develop a simple test-time intervention method to control the decoding language, namely inference-time language control (itlc). specifically, we extracted a low-rank language vector from the aligned representations using linear discriminant analysis @xcite , aggregated them per language to create language vectors, and perform a simple vector translation to control the decoding language as shown in figure 1 foot_0 . we show the effectiveness of itlc in mitigating the language confusion problem @xcite . furthermore, we conduct an extensive evaluation to test that, unlike other approaches, itlc can control the language with minimal loss of semantic. our contribution in this work is fourfold: ## understanding representation alignment in llms prior works @xcite @xcite @xcite demonstrate the existence of emerging representation alignment in llms. we take a step further to provide a deeper understanding to this behavior by contrasting it with alignment in strictly-aligned llms. specifically, we observe the correlation between the degree of alignment with the cross-lingual generalization and language identification (lid) capability, which are the proxies to their language-agnostic and language-specific capabilities, respectively. ## inference-time language control building on the insights presented in §3, we explore a method to control the language of the generated sequence with minimal semantic loss. specifically, we develop a method to extract languagespecific information at the layer where representation alignment occurs in llms. using this information, we gather language-specific vectors from each language and use them to manipulate the language-specific information during the inference time. with this language-specific intervention, we aim to steer the model toward utilizing languagespecific features, allowing us to perform inference-time language control (itlc). itlc offers two key advantages over existing intervention methods: unlike existing intervention methods that are limited to either crosslingual @xcite or monolingual @xcite scenarios, and unlike approaches that method qwen2.5-0.5b qwen2.5-0.5b-instruct qwen2.5-7b qwen2.5-7b-instruct llama-3.1-8b llama-3.1-8b-instruct ## impact of itlc we demonstrate the effectiveness of itlc on mitigating the language confusion problem @xcite . we also compare our method with another test-time intervention methods specifically designed for language confusion (sterz et al., 2025) foot_2 . furthermore, we showcase that itlc can also perform language control while being highly efficient with minimal semantic loss compared to other existing test-time intervention methods @xcite . ## conclusion our work explores the phenomenon of representation alignment in llms, confirming its occurrence and elucidating its behavior compared to strictly designed alignment models. we have demonstrated the potential for disentangling language-specific and language-agnostic information, enabling effective language-specific manipulation without semantic loss. furthermore, we have shown the practical applications of language control manipulation in enhancing language control and mitigating confusion problems. our itlc method demonstrates significant gains on the language confusion benchmark, achieving an average improvement of 9% in monolingual and 26.7% in cross-lingual settings. it also achieves comparable performance to existing test-time intervention approaches, while being much more efficient (requiring only a single middle layer intervention). ultimately, our work not only advances the theoretical understanding of representation alignment in llms but also introduces a practical and effective solution for enhancing crosslingual capabilities, paving the way for more robust and versatile lllms in multilingual contexts.