Mechanistic Understanding and Mitigation of Language Confusion in E nglish-Centric Large Language Models (2025)

Abstract: Language confusion—where large language models (LLMs) generate unintended languages against the user’s need—remains a critical challenge, especially for English-centric models. We present the first mechanistic interpretability (MI) study of language confusion, combining behavioral benchmarking with neuron-level analysis. Using the Language Confusion Benchmark (LCB), we show that confusion points (CPs)—specific positions where language switches occur—are central to this phenomenon. Through layer-wise analysis with TunedLens and targeted neuron attribution, we reveal that transition failures in the final layers drive confusion. We further demonstrate that editing a small set of critical neurons, identified via comparative analysis with a multilingual-tuned counterpart, substantially mitigates confusion while largely preserving general competence and fluency. Our approach matches multilingual alignment in confusion reduction for many languages and yields cleaner, higher-quality outputs. These findings provide new insights into the internal dynamics of LLMs and highlight neuron-level interventions as a promising direction for robust, interpretable multilingual language modeling.

## introduction current large language models (llms), such as gpt-4 @xcite , palm 2 @xcite , and llama 3 @xcite , have demonstrated exceptional linguistic competence across a wide range of complex tasks that require abstract knowledge and reasoning @xcite @xcite . early llms were predominantly trained on massive amounts of english text data, with some limited exposure to other languages, resulting in initially constrained multilingual capabilities @xcite such as multilingual continued pretraining and instruction tuning, have substantially extended these models' ability to support multiple languages @xcite @xcite @xcite . as a result, contemporary english-centric llms have become foundational tools for multilingual communication, multilingual content generation, and cross-lingual applications @xcite @xcite @xcite . however, despite their impressive capabilities, a persistent and underexplored limitation remains: llms can fail to generate text in the user's intended language, even when explicitly instructed-a phenomenon termed language confusion @xcite . language confusion manifests as full-response, line-level, or word-level switches into unintended languages, severely undermining user experience and model reliability, especially for non-english speakers (figure 1a ). recent work by @xcite provides the first systematic characterization of language confusion, introducing the language confusion benchmark (lcb) and associated metrics to quantify this phenomenon across a diverse set of languages and models. their evaluation revealed that even state-of-the-art llms are susceptible to language confusion, with english-centric llms such as llama2, llama3, and mistral exhibiting particularly high rates of unintended language switching, especially in the absence of targeted multilingual alignment (figure 1c ). while @xcite propose several mitigation strategies, including decoding adjustments, prompting techniques, and multilingual fine-tuning, these approaches remain largely surface-level, offering limited insight into the internal mechanisms that give rise to language confusion. a key observation from prior work is the identification of confusion points-specific positions in the generation process where the model abruptly switches to an unintended language. however, the model's internal dynamics leading to these confusion points and their causal role in language confusion remain largely unexplored. this gap is particularly salient given the parallels to human bilingual code-switching, where switch points between languages are cognitively significant as extensively studied in psycholinguistics @xcite . further discussions on the relationship between code-switching in human communication and language confusion in llm text generation are provided in appendix a. in this work, we move beyond behavioral evaluation to open the black box of llms, leveraging mechanistic interpretability (mi) methods @xcite @xcite to investigate the internal representations and neuron-level processes underlying language confusion. we first empirically demonstrate that confusion points are critical drivers of language confusion: targeted replace-ment at these points can substantially reduce confusion across languages. building on the cp token, we employ mi tools such as tunedlens @xcite to trace the evolution of language representations through the model's layers, revealing that confusion typically arises from transition failures in the final layers, where latent conceptual representations are mapped to surface forms in the target language (figure 1b ). to further elucidate the mechanism, we conduct a neuron-level analysis, identifying specific neurons in the last layers whose activity is predictive of successful or failed language transitions at confusion points. inspired by recent advances in neuron attribution and editing, we show that targeted manipulation of only 100 neurons can effectively mitigate language confusion, offering a novel, model-internal approach to improving multilingual reliability. our findings provide the first mechanistic account of language confusion in llms, bridging the gap between behavioral benchmarks and internal model dynamics. by highlighting the central role of confusion points and their neural mechanisms, we lay the groundwork for more robust, interpretable, and cognitively informed multilingual language models. our work makes the following contributions: (1) we provide the first mechanistic interpretability study of language confusion in english-centric llms, revealing the central role of confusion points in unintended language switching; (2) we employ layer-wise and neuron-level analyses to trace the internal dynamics leading to language confusion and identify critical late-layer neurons responsible for transition failures; (3) we propose and validate a principled neuron selection and editing strategy that effectively mitigates language confusion and preserves the model's general competence and output quality. ## mitigating language confusion via neuron editing a central challenge in mitigating language confusion via neuron editing is to identify a set of neurons whose intervention effectively reduces confusion without degrading the model's general competence or fluency. insights from our previous mechanistic analysis indicate that language confusion is primarily driven by a subset of late-layer ffn neurons. however, indiscriminate deactivation of important neurons risks harming the model's overall performance. thus, a principled neuron selection strategy is essential. ## conclusions this work provides the first mechanistic interpretability account of language confusion in english-centric llms. by tracing confusion points to failures in late-layer transitions and localizing the critical neurons responsible, we demonstrate that targeted neuron editing can robustly mitigate language confusion without sacrificing general competence or fluency. our approach achieves results comparable with multilingual-tuned models for many languages, while preserving cleaner output quality, though multilingual sft remains stronger for some low-resource cases. these findings highlight the promise of neuron-level interventions for more reliable and interpretable multilingual language modeling.