The Effect of Language Diversity When Fine-Tuning Large Language Models for Translation (2025)

Abstract: Prior research diverges on language diversity in LLM fine-tuning: Some studies report benefits while others find no advantages. Through controlled fine-tuning experiments across 132 translation directions, we systematically resolve these disparities. We find that expanding language diversity during fine-tuning improves translation quality for both unsupervised and—surprisingly—supervised pairs, despite less diverse models being fine-tuned exclusively on these supervised pairs. However, benefits plateau or decrease beyond a certain diversity threshold. We show that increased language diversity creates more language-agnostic representations. These representational adaptations help explain the improved performance in models fine-tuned with greater diversity.

## introduction general-purpose llms like llama 3 @xcite show promise for machine translation but require targeted fine-tuning beyond their incidental bilingualism @xcite to match the performance of specialized translation systems. through fine-tuning approaches ranging from twostage methods @xcite to more sophisticated optimization techniques @xcite , llms such as tower @xcite now outperform traditional nmt systems @xcite . current research presents conflicting evidence on multilingual fine-tuning strategies. some studies show that scaling the number of tasks or languages during instruction tuning improves (crosslingual) generalization @xcite @xcite , while others report that just 1-3 fine-tuning languages effectively trigger cross-lingual transfer @xcite . recent inference-only experiments by @xcite across 132 translation directions highlight this uncertainty, showing variance in translation quality with offtarget generations for non-english sources and inconsistent performance across languages. while non-english over-tokenization and typological distance provide partial explanations, controlled finetuning experiments on the effects of language diversity during fine-tuning remain unexplored. we address these conflicting findings through systematic experimentation with varying translation directions, measuring effects on both seen and unseen language pairs. through controlled fine-tuning across 132 translation directions, we demonstrate that increasing language diversity consistently improves translation quality in all categories. counterintuitively, models fine-tuned on the most diverse language sets outperform others even on fully supervised language pairs that less diverse models are specifically optimized to handle. however, experiments with even larger language sets (272 directions) reveal that benefits plateau or decrease beyond a certain diversity threshold. analysis of model activations shows that fine-tuning on diverse language directions creates more target language-agnostic representations in middle layers, explaining the performance improvements in our most diverse models. ## fine-tuning and evaluation design following @xcite , we categorize our language pairs into three groups based on their presence in the fine-tuning data of the tower model we build upon: fully supervised (pairs between de, en, ko, nl, ru and zh), zeroshot (pairs involving cs, is, ja, pl, sv and uk), and partially supervised (pairs combining supervised and zero-shot languages). this yields 132 translation directions across 12 typologically diverse languages with varying pre-training representation, enabling comprehensive assessment across different data conditions (see appendix a). fs (seen directions), fs+ps+un (all directions), evaluated on fully supervised (de/en/ko/nl/ru/zh pairs), unsupervised (cs/is/ja/pl/sv/uk pairs), and partially supervised (combining supervised and unsupervised) language pairs. numbers above bars show mean scores. training on more diverse sets improves all categories, with fs+ps+un achieving best comet-strict scores even for fully supervised pairs. fs substantially reduces off-target rates for unsupervised directions compared to base and fsec, despite these pairs being absent from its fine-tuning data. ## results increased diversity leads to better performance figure 1 (left) demonstrates that expanding language diversity during fine-tuning yields consistent performance improvements across all language pair categories. the comet-strict scores show a clear progression from base to fsec to fs to fs+ps+un models, with the most diverse model achieving the highest scores in every category. surprisingly, the fs+ps+un model (fine-tuned on all 132 directions) outperforms specialized models even on fully supervised language pairs (0.880 vs. 0.876 for fsec), despite the latter being specifically optimized for these directions. the benefits become more pronounced for partially supervised (0.812 vs. 0.448 for base) and unsupervised (0.739 vs. 0.253 for base), although this improvement is ## conclusion our systematic investigation across 132 translation directions resolves conflicting findings on language diversity in llm fine-tuning. we show that fine-tuning on broader language sets consistently improves translation across all categories: fully supervised, partially supervised, and zero-shot pairs. consequently, we recommend fine-tuning with diverse language directions even when optimizing for a limited subset of target translation pairs, as our most diverse model outperformed models specialized exclusively for those target pairs. however, we advise identifying an optimal diversity threshold, as too many languages diminishes performance for well-supported pairs while still benefiting lessrepresented languages. our representational analysis attributes the diversity improvements to specific adaptations in middle layers, revealing increased language-agnostic representations, which explains the enhanced cross-lingual transfer.