What Language Do Non- E nglish-Centric Large Language Models Think in? (2025)

Abstract: In this study, we investigate whether non-English-centric large language models, ‘think’ in their specialized language. Specifically, we analyze how intermediate layer representations, when projected into the vocabulary space, favor certain languages during generation—termed as latent languages. We categorize non-English-centric models into two groups: CPMs, which are English-centric models with continued pre-training on its specialized language, and BLMs, which are pre-trained on a balanced mix of multiple languages from scratch. Our findings reveal that while English-centric models rely exclusively on English as their latent language, non-English-centric models activate multiple latent languages, dynamically selecting the most similar one based on both the source and target languages. This also influences responses to culture difference questions, reducing English-centric biases in non-English models. This study deepens our understanding of language representation in non-English-centric LLMs, shedding light on the intricate dynamics of multilingual processing at the representational level.

## introduction large language models (llms) need multilingual capability to effectively serve a global audience by facilitating communication and task execution across diverse languages. nevertheless, stateof-the-art llms remain predominantly englishcentric @xcite . despite their robust performance in english, these models often exhibit reduced proficiency in non-english languages, and their outputs may reflect an inherent bias toward english-centric perspectives. recent studies on the llama-2 family sug-figure 1 : logit-lens for intermediate layers of three models when doing translation. the input is "français: "préparation" -中文: "." the figure shows the highest probability token from the intermediate layers starting from layer 20. the chinese answer "准备" is expected, where 0xe5 is the first utf-8 byte of "准." to mitigate these challenges, researchers have developed non-english-centric models designed to enhance performance in a specialized language and reduce biases. two primary strategies have emerged. one approach adapts english-centric models by continuing pre-training with languagespecific corpora (cpms) @xcite , while the other constructs a model from scratch using a balanced corpus that contains english and the specialized language (blms) @xcite ) @xcite . while these models demonstrate improved performance, it remains unclear how their internal processing differs from english-centric models. to explore this, we investigate the open question: when processing their specialized language, in what latent language do these models 'think'? specifically, we seek to determine whether these models employ the dominant language of their training data as latent language when processing monolingual cloze tasks. to address this question, we conduct experiments on four languages-japanese, chinese, french, and arabic. an example finding of japanese-specialized models indicate that while the english-centric model predominantly processes information in english, the blm model primarily utilizes japanese as a single latent language in its intermediate layer; the cpm model exhibits a mixed pattern of both english and specialized language utilization as latent languages. while non-english-centric models 'think' in their specialized language when processing tasks in that language, an intuitive question arises: what latent language do these models employ when handling cross-lingual tasks? to address this, we systematically vary both the source and target languages across various non-english-centric models in a translation task. our experiments reveal that the latent language in intermediate layers of these models follows a dynamic pattern: earlier layers tend to reflect latent language similar to the source language, while later layers increasingly utilize latent language similar to the target language-eventually yielding outputs in the target language. notably, blms exhibit a noticeable tendency to adopt a single latent language (i.e., '準 備' in figure 1(c )), whereas cpms tend to intermix activations across languages (i.e. '準備' and '_prepar'). we refer to this phenomenon-where the model's probability distribution shifts stepwise from a language akin to the source to one more similar to the target, culminating in the final output-as the 'probabilistic cascade'. given that non-english-centric models have been shown to reduce biases @xcite , it is crucial to understand how their internal latent language patterns contribute to shaping cultural biases. in particular, we investigate: how do the latent language patterns influence semantic representations when processing culturally specific questions? to address this, we analyze the models' internal responses when handling culture difference questions. when asked about the longest river in japan, english-centric model initially produces latent representations biased toward english-centric cultural narratives (i.e., referencing the mississippi river). although later layers gradually adjust the output toward the target language context, the final answer remains culturally inappropriate. in contrast, non-english-centric models realign their latent language more effectively toward the target culture, resulting in more accurate and culturally relevant outputs. this investigation thus elucidates how latent language patterns in intermediate layers can shape cultural bias. in summary, we demonstrate the aforementioned experiments, the model subjects, and covered non-english languages in figure 2 . our contributions are threefold: current frontier large language models, such as gpt-4 @xcite , gemini @xcite , and llama-2 @xcite , are primarily trained with english-centric corpora, with other languages constituting only a small portion of the training data. significant research efforts have been contributed to enhance these models' multilingual capabilities through various methods. one approach involves continued pre-training on specialized language data @xcite @xcite , as demonstrated by models like japanese swallow @xcite , chi-nesellama @xcite , claire @xcite and sambalingo @xcite , all of which are based on llama-2. another approach is training from scratch with bilingual data @xcite @xcite , exemplified by models such as llm-jp @xcite , baichuan @xcite , lucie @xcite and jais @xcite . while these two approaches have proven effective, the community still knows little about the underlying mechanism. our research substantially fills this gap. ## methodology in this section, we first introduce logit lens, which is used to detect the latent language of certain llms. we define two categories of non-englishcentric llms and collect models across four non-english lanuguages-chinese, japanese, french, and arabic-as our research subjects. we design three tasks including monolingual cloze, crosslingual translation, and culture difference qa tasks to examine the three research questions described in the introduction. ## experimental settings to derive general conclusions considering linguistic diversity, we selected one cpm and one blm of comparable size for japanese, chinese, french, and arabic, respectively, and conducted our experiments alongside the english-centric llama 2 family to investigate how training data influences latent language probabilities. details of the selected models are presented in appendix 1. to ensure that the model can complete the task successfully, we use few-shot prompting (in the same language setting) to teach the llm the task format in all experiments, with each shot structured as described in section 3.4. we then monitored the probabilities of different language versions of the answers being generated at each layer and visualized the results in graphical form. ## conclusion in this study, we leverage logit lens to analyze the latent languages of non-english-centric llms. our findings in the monolingual cloze task indicate that cpms exhibit a mixture of latent languages, blending their specialized language with english, while blms activate the latent language most similar to the input dynamically. while conducting cross-lingual translation, both source and target languages influence latent language activation, with higher linguistic similarity leading to stronger activation. a typical pattern termed 'probabilistic cascade' is observed: the probability of latent languages peaks and then declines alternately, and ultimately shifts the peak to the target language. finally, we observe that english-centric models introduce cultural biases, whereas non-english-centric models better capture their respective cultural contexts. these insights contribute to understanding multilingual bias and guiding future improvements.