C o C o- C o L a: Evaluating and Improving Language Adherence in Multilingual LLM s (2025)

Abstract: Multilingual Large Language Models (LLMs) develop cross-lingual abilities despite being trained on limited parallel data. However, they often struggle to generate responses in the intended language, favoring high-resource languages such as English. In this work, we introduce CoCo-CoLa (Correct Concept - Correct Language), a novel metric to evaluate language adherence in multilingual LLMs. Using fine-tuning experiments on a closed-book QA task across seven languages, we analyze how training in one language affects others’ performance. Our findings reveal that multilingual models share task knowledge across languages but exhibit biases in the selection of output language. We identify language-specific layers, showing that final layers play a crucial role in determining output language. Accordingly, we propose a partial training strategy that selectively fine-tunes key layers, improving language adherence while reducing computational cost. Our method achieves comparable or superior performance to full fine-tuning, particularly for low-resource languages, offering a more efficient multilingual adaptation.

## introduction multilingual llms are pre-trained on raw text from multiple languages, typically consisting of separate corpora for each language. remarkably, despite this lack of explicit parallel data to facilitate cross-lingual associations, these models develop an implicit understanding of inter-language relations and cross-lingual word associations @xcite . instruction tuning further enhances their ability to follow prompts, and models trained on multilingual data often exhibit zero-shot cross-lingual transfer of instruction-following capabilities @xcite . how-figure 1 : evaluation of correctness and language adherence on french input. the soda level visualizes the coco-cola ratio, with higher levels indicating stronger adherence to the input language. our results show that partially fine-tuning the final layers of an english-tuned model on french achieves language adherence and accuracy comparable to a model fully fine-tuned on french. a crucial step toward addressing the limitations of multilingual llms is understanding how they internally process and encode multilingual knowledge. interpretability research has traditionally focused on monolingual models, leveraging techniques such as representation probing @xcite and model patching @xcite . these methods have been widely used to examine llms' performance across tasks such as mathematics @xcite , and general knowledge @xcite @xcite @xcite . studies on model internals suggest that multi-layer perceptrons (mlps) retrieve task-relevant information, while attention layers refine and promote the correct response @xcite . furthermore, knowledge is often identified in earlier layers and reinforced in later layers @xcite . however, these interpretability techniques have primarily been applied to monolingual models, which were initially dominant due to the early focus on english-language pertaining @xcite @xcite . the rise of multilingual llms trained on diverse languages @xcite @xcite , necessitates extending interpretability research beyond english. multilingual llms present additional challenges: representations of different languages are intertwined within a shared space; cross-lingual alignment varies across languages; and shared tokens between languages impact their process. these complexities make it difficult to isolate languagespecific knowledge, benchmark cross-lingual generalization, and interpret how multilingual llms acquire and apply linguistic information. given the prevalence of mid-and low-resource languages, understanding these mechanisms is crucial not only for improving cross-lingual transfer but also for mitigating the "curse of multilinguality" -the performance degradation observed as the number of supported languages increases. recent efforts have begun tackling these challenges by probing internal representations @xcite , analyzing the emergence of cross-lingual transfer @xcite , and studying token representation alignment on cross-lingual transfer @xcite . furthermore, researchers attempt to separate the linguistic abilities from task abilities by developing language-and taskspecific adapters @xcite , subnetworks @xcite , or layers @xcite . however, despite this progress, most prior works treat multilinguality as a monolithic phenomenon, focusing on general cross-lingual transfer or aggregating all languages into a single block of linguistic knowledge. less attention has been given to understanding how llms process individual languages at a more granular level, particularly within the context of task learning. in this work, we focus on language adherence by first identifying both shared and distinct patterns in cross-lingual task acquisition, revealing how multilingual models internalize and apply linguistic knowledge (section 3). we find that training on a task in one language improves performance in other languages. however, this benefit is not always directly observable due to an inherent model bias towards generating output in a preferred language, rather than strictly adhering to the input language (section 4.1). to quantify this bias, we introduce coco-cola (correct concept, correct language), a novel metric designed to assess a model's ability to generate responses in the intended input language, particularly for languages not included in supervised finetuning (sft). furthermore, we propose a partial training method that selectively fine-tunes specific model layers which reveals the relation between language adherence and model layers (section 4.2). this approach enables more efficient language adaptation, achieving comparable or even superior performance compared to full model retraining, especially for low-resource languages. finally, we show that the issue of language adherence can be addressed by finetuning only the final layers of llms on a small balanced multilingual data (section 4.3). ## related work this work builds on several active research areas that inform our study of multilingual task learning in llms. specifically, we draw from ( 1 (3) adapters, which separate language knowledge from task-specific knowledge, offering a structured framework for understanding their interactions; and (4) subnetworks, which identify task-and language-specific parameters within existing models, offering an alternative to external adapters and directly informing our approach to efficient partial training. interpretability. @xcite use probing techniques to analyze accuracy changes across layers in llms, showing that high-resource languages exhibit patterns similar to english, with accuracy increasing from lower to upper layers. however, this pattern is inconsistent for low-resource languages. @xcite examine cross-lingual transfer by analyzing neuron overlap in different languages using checkpoints from bloom's pre-training @xcite . they find a strong correlation between neuron overlap and cross-lingual transfer, though neuron overlap does not increase monotonically during training, and patterns vary across model sizes. similarly, @xcite investigate language-specific neurons and assess how these neurons affect both english and non-english language performance. representation alignment. beyond studying multilingualism in llms, some research focuses on improving model performance across languages through representation alignment. @xcite align english and arabic model representations using a bilingual dictionary before fine-tuning on a target task. @xcite align english representations with other languages using question-translation data before instruction-tuning. additionally, @xcite introduce a pixel representation method to enhance alignment and improve translation quality. adapters. another approach for cross-lingual transfer involves integrating adapters into the model. this technique is based on the assumption that task-solving knowledge can be separated from language knowledge. pfeiffer et al. (2020) introduce mad-x, a framework where language and task adapters are trained separately, with each block's representations passing through a language adapter before a task adapter. building on this, later works aim to refine adapter creation and composition methods. for instance, @xcite propose bad-x, which replaces monolingual adapters with bilingual adapters, improving performance for low-resource languages. @xcite introduce adamergex, where adapters for language-task pairs are trained independently and later combined through linear operations (addition and subtraction) to generate adapters for new language-task pairs. subnetworks. to enhance cross-lingual transfer without adding new parameters, some methods focus on identifying existing task-and languagespecific parameters within the model. @xcite fine-tune models for specific languages or tasks, extract the most affected neurons, and use the resulting subnetworks to enable multilingual task adaptation. @xcite take a layerwise approach in multiple steps: they train separate language-and task-expert models, analyze parameter changes to identify key layers for language and task learning, and use layer-swapping techniques to create a math expert in a new language. consistent with @xcite , their findings suggest that initial and final layers primarily encode linguistic information, while middle layers are task-specific. ## preliminary analysis in the preliminary section of this paper, we first isolate language effects from task learning by choosing multi-lingual parallel qa data (section 3.1), examine fine-tuning performance across multiple languages (section 3.2), explore how well llms generalize knowledge across languages (section 3.3), and which model components are most affected during training (section 3.4). then, in section 4.1, we introduce coco-cola metric to measure language adherence in multilingual llms followed by an efficient partial training method to increase the model adherence (section 4.2). ## approach our previous analysis suggests that while task knowledge is largely shared across languages, the way this knowledge is processed and accessed differs. although a jaccard index analysis revealed substantial overlap in correct answers, our investigation of parameter updates showed that models trained on non-english languages required more substantial modifications in their mlp modules compared to english, even when achieving comparable accuracy. this raises an important question: do these modifications reflect deviations in knowledge acquisition, or are they more related to language generation? in this section, we first introduce a metric to analyze linguistic bias in multilingual llm outputs. then, we propose a partial training strategy aimed at reducing this bias by selectively fine-tuning specific model components. ## conclusion in this work, we first analyzed shared knowledge across seven languages and identified key differences in the parameters most affected when training models for each language. building on these insights, we proposed the coco-cola ratio, a metric for evaluating language adherence in multilingual llms, and used it to evaluate both pre-trained and fine-tuned llms. our findings show that pretrained models tend to generate english outputs regardless of the input language and that fine-tuning on english further amplifies this bias. to address this problem, we leveraged insights from parameter updates and coco-cola results to develop a partial training method that improves language adherence in english-trained models. our analysis demonstrated a more efficient alternative to full fine-tuning, achieving comparable or even superior performance while significantly reducing the number of updated parameters. additionally, we showed that partial training on balanced multilingual data achieves similar language adherence to monolingual training. given the widespread availability of instruction-tuned and task-specific english models, partial training of final layers presents a fast and efficient approach for improving language adherence and adapting llms to new languages.