Cross-lingual Transfer Dynamics in BLOOMZ : Insights into Multilingual Generalization (2025)

Abstract: Multilingual large language models have emerged as a promising solution for resource-constrained settings, with significant efforts aimed towards improving multilingual capabilities of English-centric pretrained models. However, the broader cross-lingual implications of fine-tuning interventions remain understudied. This work examines instruction tuning (IT) over the BLOOMZ model for Question Answering (QA) in low-resource settings, with special emphasis on transfer dynamics across several languages. Our findings reveal two critical insights: first, IT on the target language can negatively impact its own performance in constrained short-span generation tasks due to overgeneration tendencies; second, in QA tasks, IT appears to suppress performance in some interfering languages, thereby enhancing capabilities in some target Indic languages by more than doubling QA performance. These results highlight important trade-offs in multilingual LLM adaptation and enhance our understanding of cross-lingual transfer mechanisms.

## introduction large language models (llms) excel in tasks like classification, text generation, and information extraction. recently, cross-lingual alignment has been widely studied to enhance the multilingual capabilities of llms @xcite @xcite . since most of the world's languages can be deemed low-resource owing to the limited amounts of high-quality data @xcite , cross-lingual alignment is an important problem to tackle. prior work on multilinguality has largely focused on cross-lingual dynamics within english-centric models and pretrained decoder-only models @xcite @xcite . in this work, we study the cross-lingual abilities of bloomz, a multilingual, multi-task instructiontuned model @xcite . we focus on a constrained generation task, closed question answering (qa), that can be objectively evaluated (unlike open-ended generation tasks like machine translation) while still being vulnerable to generation-related artefacts (unlike classification tasks). we examine the impact of instruction tuning (it) on qa on several indic and non-indic languages. surprisingly, we find significant performance improvements using languages that transcend language family relatedness and surface-level script similarities, indicating that bloomz exhibits cross-lingual generalization beyond typological proximity @xcite . we also present a new multilingual logit lens-based analysis to provide more insights into cross-lingual dynamics that result in performance improvements or degradations. our analysis reveals two key phenomena: 1) suppression of the target language and 2) a tendency to over-generate in the target language, both of which significantly affect the model's output as illustrated in figure 1 . alignment with prior work. a growing consensus from recent work @xcite is that english-centric models like llama @xcite and @xcite process multilingual prompts by internally mapping to an english "thinking space" in intermediate layers, while the initial and final layers are multilingual in nature. concurrent work explores language-specific neurons in the multilingual llms and reveals their significant roles in the outer layers @xcite . @xcite argues that stronger alignment between english and non-english middle layer embeddings correlate with better crosslingual transfer. while current llms and adaptation methods demonstrate promising surface-level cross-lingual abilities (on tasks like style transfer), they struggle with deeper cross-lingual reasoning and knowledge transfer. this limitation suggests the presence of a cross-lingual knowledge barrier, as noted by chua et al. (2024). towards addressing this gap, it has been observed that fine-tuning on certain languages can improve the performance of others, indicating the presence of cross-lingual bridging mechanisms (singh et al., 2024b; bai et al., 2024; ifergan et al., 2024; wang et al., 2024; bai et al., 2023). our experiments also support this possibility of cross-lingual bridging mechanisms. our findings align with ifergan et al. (2024) who documented bloom's unique ability to facilitate factual recall across languages with different scripts. ## results and discussion suppression for performance improvements. we highlight token-f1 scores for 5 indic languages: gujarati (gu), kannada (kn), malayalam (ml), odia (or), and punjabi (pa) as these showed significant improvements in performance. metrics are presented in through analysis of the multilingual logit lens plots (figure 2 ), we identify distinct "hill-like" patterns in the middle-layer latents of languages such as german (de), estonian (et), swedish (sv), xhosa (xh), finnish (fi), indonesian (id), and malay (ms). we verify that, to some extent, these languages can facilitate cross-lingual transfer, in accordance with the findings of @xcite . we compare the logit lens plots for or-xquad and pa-xquad test sets between the base model and the de-it model (i.e., the base model finetuned on german train set). we focus on samples where the base model answers wrongly, but the de-it model provided correct predictions, shown in figure 2 . these plots highlight high-resource fine-tuning is not universally beneficial. contrary to prior work, our findings challenge the notion that fine-tuning on highresource languages universally improves perfor-mance across the multilingual spectrum. the performance metrics of en-it model on gu-xquad test set serves as a clear example of this. surprisingly, certain medium to low-resource languages, such as kannada, malayalam, thai and turkish contribute significantly to overall model improvement. this suggests that the effectiveness of fine-tuning languages in enhancing alignment and generalization is not solely dependent on data availability. self-performance trade-off. self-it (i.e., it using language x evaluated on test samples of language x) appears to negatively impact performance on qa-style tasks that require concise, spanbased answers. this is likely due to the model's tendency to generate verbose (and sometimes hallucinatory) responses, that negatively affects task accuracy. logit lens plots in figure 2 illustrate this tendency for de-it on de-xquad, with rising latent probabilities in the final layers indicative of over-generation. ## conclusion this work highlights the intricate nature of multilingual task specific fine-tuning and its diverse effects across languages. we demonstrate that instruction tuning with a very small set of samples is unlikely to acquire substantial new knowledge, but can induce shifts in linguistic structures, particularly in the later layers, leading to suppression of interfering language latents, contributing to improved performance. however, these improvements are neither uniform nor guaranteed, as high-resource fine-tuning does not always yield positive effects, and self-it performance trade-offs often emerge. moreover, the quality of training data significantly influences outcomes, with better translation quality directly correlating with improved multilingual alignment.