Can Big Models Help Diverse Languages? Investigating Large Pretrained Multilingual Models for Machine Translation of I ndian Languages (2023)

Abstract: Machine translation of Indian languages is challenging due to several factors, including linguistic diversity, limited parallel data, language divergence, and complex morphology. Recently, large pre-trained multilingual models have shown promise in improving translation quality. In this paper, we conduct a large-scale study on applying large pre-trained models for English-Indic machine translation through transfer learning across languages and domains. This study systematically evaluates the practical gains these models can provide and analyzes their capabilities for the translation of the Indian language by transfer learning. Specifically, we experiment with several models, including Meta’s mBART, mBART-manyto-many, NLLB-200, M2M-100, and Google’s MT5. These models are fine-tuned on small, high-quality English-Indic parallel data across languages and domains. Our findings show that adapting large pre-trained models to particular languages by fine-tuning improves translation quality across the Indic languages, even for languages unseen during pretraining. Domain adaptation through continued fine-tuning improves results. Our study provides insights into utilizing large pretrained models to address the distinct challenges of MT of Indian languages.

## introduction india is a linguistically diverse nation with 22 official scheduled languages and numerous dialects representing various language families like indo-aryan, dravidian, and tibeto-burman. this linguistic diversity presents unique challenges for developing machine translation systems between indian languages. complex morphology, linguistic diversity, language divergence, and limited parallel data make translation of indian languages difficult. recent advances in neural machine trans-lation (nmt) have improved translation quality for many language pairs. however, performance for low-resource indian languages still lags behind european and east asian languages. recently, pretrained multilingual language models like mbart @xcite and translation models like m2m-100 @xcite have shown promising results by leveraging knowledge transferred from high-resource languages through large-scale pretraining. these models can translate between multiple languages with higher quality than previous approaches by leveraging their broad linguistic knowledge gained through pretraining. however, their capabilities have not been thoroughly explored for indian languages on a large scale specifically. this paper conducts a large-scale study on applying pretrained models to english-indic machine translation across diverse languages and domains. specifically, we investigate several models, including meta's mbart @xcite , mbart50-many-to-many @xcite , @xcite , m2m-100 @xcite , and google's mt5 @xcite for translation of indian languages. the models are fine-tuned on high-quality human-translated parallel data between english and 16 indic languages. the languages encompass those included during model pretraining along with unseen languages to cover high to low-resource scenarios. through extensive experiments, we evaluate the improvements enabled by transfer learning and examine model capabilities for handling indian language diversity. our analysis provides practical insights into utilizing pretrained models for english-indic translation. our contributions are as follows: ## focus languages we focus on 16 official languages of india with varying quantities of available data, including highresource languages such as hindi and bengali and very low-resource languages such as sanskrit, with ancient texts being its available corpus. ## experimental setup in this section, we discuss the data and settings used in our experiments. we conducted experiments to fine-tune pretrained models for 16 different languages. for 14 of these languages, we used the bpcc-wiki dataset @xcite to focus on language-specific features. for the remaining two languages (manipuri and kashmiri), we used the nllb-seed @xcite and ilci (choudhary and jha, 2014) corpus. we used the flo-res @xcite dev set and devtest set for 13 out of 16 languages for development and testing. for the remaining three languages (bodo, konkani, and dogri), we used a randomly selected 1k sentences as the dev set and the flores dev set translated by bpcc corpus as the test set. to adapt the pretrained models to a specific domain, we used the pmi dataset @xcite . it contains speeches and news from the indian prime minister. for development and testing, we used the wat 2021 development and test sets @xcite . for languages not included in the wat 2021 set, we randomly sampled sentences from the corpus. it is worth noting that the fine-tuning parallel data for all languages were less than 50k sentences, as shown in the ## result and discussion we discuss the results of transfer across languages and domain in this section. ## conclusion and future work in conclusion, this paper shows that large pretrained models, such as mbart50-m2m, nllb-200, and m2m-100, can significantly improve the quality of english-indic translations through transfer learning, across diverse indian languages. our analysis shows adapting these models by finetuning on small amounts of high-quality parallel data for a particular language or domain substantially boosts translation quality. improvements are seen even for low-resource languages unseen during model pretraining. this indicates pretrained models acquire linguistic knowledge that transfers across languages to handle complex morphology and diversity. future work includes expanding language coverage and exploring semi-supervised methods for continued improvements. as pretrained models grow in scale and linguistic breadth, their capabilities for indic language translation will also improve. we hope our study provides insights to leverage these large pretrained models to address the unique challenges of indian language diversity.