RBG - AI : Benefits of Multilingual Language Models for Low-Resource Languages (2025)

Abstract: This paper investigates how multilingual language models benefit low-resource languages through our submission to the WMT 2025 Low-Resource Indic Language Translation shared task. We explore whether languages from related families can effectively support translation for low-resource languages that were absent or underrepresented during model training. Using a quantized multilingual pretrained foundation model, we examine zero-shot translation capabilities and cross-lingual transfer effects across three language families: Tibeto-Burman, Indo-Aryan, and Austroasiatic. Our findings demonstrate that multilingual models failed to leverage linguistic similarities, particularly evidenced within the Tibeto-Burman family. The study provides insights into the practical feasibility of zero-shot translation for low-resource language settings and the role of language family relationships in multilingual model performance.

## introduction the development of multilingual machine translation (mmt) systems presents a critical opportunity to bridge communication gaps for the world's estimated 7,000+ languages. among them many remain severely under-resourced in digital contexts. while high-resource languages like english, french, and chinese have abundant parallel training data, the vast majority of languages particularly those spoken by smaller communities lack sufficient data for effective neural machine translation (nmt) system development. the wmt 2025 indic machine translation shared task (wmt 1 ) provides an ideal testbed for investigating how multilingual language models can benefit low-resource languages. the indic 1 https://www2.statmt.org/wmt25/indic-mt-task.html , accessed on august 2025 language family presents a diverse landscape of linguistic resources, ranging from relatively highresource languages like hindi and bengali to extremely low-resource languages with minimal digital presence. this diversity allows us to examine crucial questions about cross-lingual transfer and zero-shot translation capabilities. a fundamental question in multilingual natural language processing (nlp) is whether languages from related families can effectively support translation for languages that were absent or severely underrepresented during model training. the northeast indian linguistic landscape presents a diverse range of language families, including tibeto-burman, indo-aryan, and austroasiatic. these families offer rich opportunities to study cross-lingual transfer phenomena due to their distinct linguistic features. this includes morphological patterns, syntactic structures, and varying degrees of relatedness. our experiment investigates three core questions: (1) can multilingual models achieve practical zeroshot translation quality for truly low-resource languages? (2) how do linguistic relationships across different language families influence cross-lingual transfer effectiveness? (3) what are the practical limitations and opportunities for deploying such systems in resource-constrained environments where low-resource languages are typically spoken? using the pre-trained mmt model as our foundation, we conduct systematic experiments to evaluate zero-shot translation performance and crosslingual transfer effects. to ensure practical applicability under resource constrained environment, we implement 4-bit quantization to enable deployment on consumer hardware, addressing the reality that communities speaking low-resource languages often have limited access to high-end computational resources. ## related work cross-lingual transfer learning has emerged as a promising approach for supporting low-resource languages by leveraging knowledge from related high-resource languages. early work demonstrated that multilingual nmt models could achieve reasonable performance on unseen language pairs through parameter sharing @xcite . however, the mechanisms underlying successful cross-lingual transfer remained poorly understood. recent studies have shown that linguistic similarity plays a crucial role in transfer effectiveness. @xcite demonstrated that typologically similar languages benefit more from multilingual training, while @xcite found that shared script and language family membership are strong predictors of transfer success. @xcite further showed that multilingual models develop language-agnostic representations that facilitate zero-shot transfer, particularly within language families. the concept of "cursing of multilinguality" suggests that adding more languages to multilingual models can hurt performance on existing languages @xcite . however, @xcite argued that this effect is primarily observed when languages are linguistically distant, and that related languages can actually benefit from shared training. @xcite first demonstrated zeroshot translation in multilingual nmt, showing that models could leverage shared representations to translate between unseen language pairs via pivoting through shared languages. subsequent work has explored the conditions under which zero-shot translation succeeds. @xcite found that zero-shot performance is highly dependent on the linguistic similarity between source and target languages and the pivot languages seen during training. pires et al. (2019) showed that multilingual bert representations are most effective for cross-lingual transfer when languages share scripts and belong to the same family. for indic languages specifically, @xcite demonstrated that sanskrit-related languages show strong cross-lingual transfer effects. inline with it, @xcite found that multilingual indic models benefit from careful language grouping based on linguistic relationships. however, most prior work has focused on relatively high-resource indic languages, leaving questions about truly low-resource scenarios largely unexplored. the role of language families in nmt has received increasing attention as researchers seek to understand the linguistic factors that enable successful multilingual models. tan et al. (2019) showed that language family membership is one of the strongest predictors of multilingual model success, outperforming surface-level similarity measures. within the indo-european family, studies have shown particular promise for cross-lingual transfer. @xcite demonstrated that indo-aryan languages share sufficient structural similarity to enable effective multilingual training, while tamil and other dravidian languages require different modeling approaches due to their distinct linguistic properties. recent work on massively multilingual models like mt5 and madlad-400 has shown that scaling to hundreds of languages can improve zero-shot performance, but the specific benefits for extremely low-resource languages remain unclear @xcite . our work addresses this gap by systematically evaluating how language family relationships influence zero-shot translation quality in truly low-resource settings. on other hand, recent advances in model compression, particularly quantization techniques, have made large multilingual models more accessible @xcite . however, the interaction between model compression and cross-lingual transfer performance has received limited attention, particularly for low-resource languages where even small performance degradations can be significant. ## data and methodology our evaluation uses the wmt 2025 shared task datasets, supplemented with low-resource language pairs to enable comprehensive analysis @xcite @xcite . this dataset provides an excellent testbed for cross-lingual transfer analysis: five tibeto-burman languages allow us to examine withinfamily transfer effects, while assamese (indo-aryan) and khasi (austroasiatic) serve as crossfamily comparison points to evaluate transfer limitations across different linguistic lineages. our final submitted system is built upon the madlad-400 model, which extends the t5 architecture to support multilingual translation across 400+ languages. the model was selected after qualitative benchmarking against several multilingual alternatives, including mbart-50 and nllb-200 @xcite . as given in we utilize the t5-compatible tokenizer associated with madlad-400, which handles the diverse scripts and writing systems of indic languages. the tokenizer includes special tokens for language direction specification. each input sentence is prepended with a language-specific tag indicating the target language (e.g., <2en> for english, <2hi> for hindi). this approach enables bidirectional translation within a single model while maintaining translation quality. the system employs beam search decoding with a beam size of 5 to improve translation fluency and adequacy. additional parameters include length penalty adjustment and early stopping criteria optimized for the target language characteristics. during the fine-tunning process, we have combined data from all seven language pairs into a comprehensive bidirectional translation dataset. for each language pair, we created translation examples in both directions: english-to-target language and target-language-to-english. this approach doubled our effective training data while enabling the model to learn translation patterns in both directions. source texts were prefixed with appropriate language direction tokens following the mad-lad format specification. the combined dataset was split using stratified sampling to ensure balanced representation across languages, resulting in 370,060 training samples, 43,537 validation samples, and 21,769 test samples. ## experiments: model adaptation we first established baseline performance using zero-shot inference with the pre-trained madlad-400 3b model. the model was loaded with 4-bit quantization using qlora (qunatized low-rank adaptation) settings to reduce memory requirements while maintaining performance. zero-shot translation was performed by prepending languagespecific direction tokens to source sentences, following the format used during pre-training where tokens such @xmath0. the zero-shot baseline results on testset revealed significant variation in performance across language pairs and translation directions. for translation into english, the model achieved the highest performance on manipuri-to-english with a bleu score of 23.2, followed by khasi-to-english at 19.4 and bodo-to-english at 19.0. assamese-to-english showed moderate performance with 11.9 bleu, while other language pairs demonstrated limited zero-shot capabilities, with kokborok-to-english, nyishi-to-english, and mizo-to-english scoring 13, 1, and 2 bleu respectively. translation from english to target languages showed considerably lower performance across all pairs, with most achieving single-digit bleu scores. englishto-manipuri performed best at 7 bleu, while several pairs including english-to-assamese, englishto-nyishi, and english-to-mizo achieved minimal scores of 1, 0, and 1 bleu respectively. these baseline results highlighted the model's stronger capability for translating into english compared to generating text in low-resource languages, estab-lishing the need for targeted fine-tuning to improve bidirectional translation performance. we employed parameter efficient fine-tuning (peft) using low-rank adaptation (lora) to adapt the pre-trained model to our specific language pairs. the lora configuration used a rank of 32 with an alpha value of 32 and dropout rate of 0.1. we targeted key attention and feed-forward components including query, value, key, and output projection layers as well as the intermediate dense layers in the feed-forward networks. the base model was prepared for k-bit training using gradient checkpointing to optimize memory usage during training. this configuration resulted in 94.4 million trainable parameters, representing only 3.1% of the total model parameters, which significantly reduced computational requirements while maintaining model expressiveness. training was conducted on a single rtx4090 gpu using with half-precision floating point (fp16) to accelerate computation and reduce memory consumption. we used a per-device batch size of 16 with gradient accumulation across 2 steps, creating an effective batch size of 32 samples per optimization step. the learning rate was set to 5e-5 with a weight decay of 0.01 to prevent overfitting. training proceeded for 3 epochs with model checkpoints saved every 1000 training steps, retaining only the 2 most recent checkpoints to manage storage requirements. we employed the batched data collator with dynamic padding aligned to multiples of 8 tokens for optimal gpu utilization efficiency. the training process utilized the sequence-tosequence model trainer from the transformers library, which handled the complete training loop including automatic loss computation and backpropagation through the lora adapters. both source and target sequences were limited to a maximum length of 256 tokens to balance computational efficiency with sequence coverage. during evaluation phases, the model used greedy decoding with a maximum generation length of 256 tokens. the complete training process required approximately 24 hours to finish all three epochs. all experiments were conducted with a fixed random seed to ensure reproducibility of results. this fine-tuning methodology successfully adapted the multilingual foundation model to our specific low-resource language pairs while maintaining computational efficiency through quantization and parameter-efficient training techniques. we observed promising perfor-mance improvements across all language pairs, with bleu score increases ranging from 3-19% for both moderate-resource pairs and extremely low-resource languages. the bidirectional training approach particularly benefited english-to-target translation, where several language pairs showed improvements from near-zero baselines. ## results and analysis our analysis of the actual wmt 2025 results reveals significant insights into zero-shot and finetuned translation performance across language families and resource levels. the system demonstrates competitive performance, often exceeding the mean scores of other participating teams across multiple language pairs and evaluation metrics. the results reveal (figure 1 ) complex patterns of cross-lingual transfer effectiveness that partially support but also challenge our initial hypotheses about language family relationships in multilingual models. a notable asymmetry emerges where target-to-english translation significantly outperforms english-to-target translation across all language pairs, with performance ratios varying dramatically from modest differences to orders of magnitude disparities. this pattern suggests that the model's english-centric training provides substantially stronger support for translating into english than for generating text in low-resource languages. within the language family analysis, our findings show highly variable performance patterns that resist simple categorization. the indo-aryan language assamese demonstrates strong assameseto-english performance with 15.26 bleu, substantially exceeding the competition mean of 11.05, but shows weaker english-to-assamese performance at 9.09 bleu compared to the competition mean of 13.51. the tibeto-burman cluster exhibits remarkable diversity, ranging from exceptional performance in bodo-to-english translation (21.67 bleu) to very poor results in nyishi-to-english (9.61 bleu) and catastrophic failure in englishto-bodo translation (0.21 bleu). khasi, representing the austroasiatic family, shows moderate and relatively balanced performance in both translation directions compared to other languages in our study. bodo being categorized as having very limited training data, achieves the highest bodoto-english bleu score across all evaluated languages while simultaneously producing the lowest english-to-bodo performance. this strong asym- interms of overall performance, our quantized madlad-400 system demonstrates competitive performance relative to other participating teams, outperforming the competition mean in eight out of fourteen language-direction pairs. the system shows particular strength in target-to-english translation, with notable advantages in bodo-to-english (+7.42 bleu), nyishi-to-english (+3.62 bleu), and assamese-to-english (+4.21 bleu) directions. additionally, several language pairs show improvements in chrf scores, indicating better characterlevel accuracy even when bleu scores are comparable. however, the system faces significant challenges in english-to-target translation for several language pairs. performance gaps appear most visible in english-to-assamese (-4.42 bleu), englishto-khasi (-5.21 bleu), and english-to-mizo (-3.68 bleu) directions. the most severe limitation appears in english-to-bodo translation. here, our system achieves only 0.21 bleu compared to the competition mean of 11.64, representing a systematic failure requiring further investigation with inputs from linguistic experts. ## conclusion and future work this study provides empirical evidence about how multilingual language models benefit low-resource languages through cross-lingual transfer, based on our competitive performance in the wmt 2025 shared task. the system outperformed competition averages in eight out of fourteen language-direction pairs, proving that deployment efficiency without compromise in performance. further our findings reveal several important patterns that advance understanding of multilingual model capabilities for low-resource languages. a consistent translation asymmetry emerges where target-to-english translation significantly outperforms english-to-target translation for most of the language pairs, with performance ratios ranging from 1.5x to 100x. this asymmetry reveals that english-centric bias is inherent in multilingual models and suggests fundamental limitations in generating low-resource languages compared to translating into english. the effects of language family relationships on translation proved more complex than initially hypothesized. while tibeto-burman languages showed evidence of familybased transfer across five of seven target languages, the effects varied dramatically and sometimes counterintuitively. performance variations within script groups, such as bengali script languages showing bleu scores ranging from 15.26 (assamese) to 1.58 (kokborok) for source-to-english translation. this indicates that script similarity provides minimal transfer benefits compared to other linguistic factors. the performance variations within language families and the consistent translation asymmetry suggest that transfer mechanisms involve more than generic linguistic similarity. wide performance variations within the tibeto-burman family sug-gest that family membership alone is insufficient to predict transfer success. future research should investigate why target-to-english translation consistently outperforms english-to-target translation and develop techniques to improve generation capabilities for low-resource languages.