Machine Translation Advancements for Low-Resource I ndian Languages in WMT 23: CFILT - IITB ’s Effort for Bridging the Gap (2023)

Abstract: This paper is related to the submission of the CFILT-IITB team for the task called IndicMT in WMT23. The paper describes our MT systems submitted to the WMT23 IndicMT shared task. The task focused on MT system development from/to English and four low-resource North-East Indian languages, viz., Assamese, Khasi, Manipuri, and Mizo. We trained them on a small parallel corpus resulting in poor-quality systems. Therefore, we utilize transfer learning with the help of a large pre-trained multilingual NMT system. Since this approach produced the best results, we submitted our NMT models for the shared task using this approach.

## introduction in the realm of machine translation, the wmt23 indicmt shared task emerges as an arena where the boundaries of translation technology are stretched to their limits. our efforts revolve around the translation of the 'en-x' pair in both directions, where 'en' signifies english and 'x' encompasses assamese, a member of the indo-aryan language family, and manipuri, a representative of the tibeto-burman family. as the task focused on english to and from low-resource indian languages, we were provided with a small parallel corpus for each 'en-x' pair. furthermore, participants had access to a substantial amount of monolingual data for assamese and manipuri, creating an ideal setting for trying out new and creative approaches. in the realm of machine translation, the neural machine translation paradigm has emerged as a dominant force, as evidenced by seminal works such as @xcite and the comprehensive survey by @xcite . however, neural machine translation models are notoriously data-hungry, leading to performance degradation when confronted with low-resource languages, as highlighted by @xcite . to tackle this challenge, we turn to the promising technique of transfer learning, a well-established approach in machine learning where knowledge gained from one task is leveraged to enhance performance in a related task. in our pursuit of improving translation capabilities for low-resource languages, we harness the multilingual indictrans2 model, as introduced by @xcite . our methodology involves fine-tuning this model using the 'en-x' parallel data provided for the task. by adopting this approach, we aim to capitalize on the acquired knowledge during training to significantly bolster the performance of the model in the specific translation task at hand. indictrans2 is rooted in the transformerbased encoder-decoder architecture pioneered by @xcite . it was trained on the extensive bharat parallel corpus collection (bpcc), a publicly accessible repository encompassing both pre-existing and freshly curated data for all 22 scheduled indian languages, this model boasts a comprehensive understanding of the linguistic diversity within the indian subcontinent. to enhance its linguistic prowess, indictrans2 has undergone auxiliary training utilizing the rich resource of backtranslated monolingual data. the model was then trained on human-annotated data to achieve further improvements. we used this model and fine-tuned it on the training data provided by wmt23. the fine-tuned indictrans2 achieves good scores; hence we are using it for our final submission. we hypothesize that its stellar performance can be attributed to the amalgamation of language knowledge acquired during its initial training, coupled with the domain-specific expertise gleaned from the fine-tuning process, facilitated by the training data made available through wmt23. ## data we use the indictrans2 model and fine-tune it on the wmt23. the original indictrans2 was trained on the bharat parallel corpus collection (bpcc) corpus. they have used flores-200 as their validation set for assamese and extended @xcite for manipuri. for auxiliary training which includes back-translated monolingual sentences, they have used indiccorp v2 @xcite and one side of nllb data as monolingual corpus. they have used standard test sets like flores-200, but they have also created a new benchmark called the in22 test set which is an n-way parallel corpus for all 22 indian scheduled languages. we have fine-tuned the model using the wmt23 parallel corpus. the 'english-assamese' pair has 50k parallel sentences, and the 'english-manipuri' pair has around 21.6k sentences. the validation set consisted of the wmt23 validation set. the size of the validation set for the 'english-assamese' is 2k sentences; for the 'english-manipuri' pair, it was 1k sentences. the test set for both pairs was the wmt23 test set. ## system overview in the pursuit of enhancing machine translation for low-resource languages, various approaches have emerged, such as phrase-pair injection and back-translation, aimed at enhancing performance. our system, on the other hand, takes a distinct path and relies on the knowledge gained from the multilingual training of indictrans2 and applies it to different low-resource languages. phrase-pair injection (pti) @xcite , @xcite and @xcite utilized a technique to combine both statistical machine translation (smt) and neural machine translation (nmt). the utilization of the phrase table during training is pivotal in statistical machine translation (smt) as it probabilistically maps phrases from the source to the target language. by incorporating these phrase mappings from the table into the existing parallel corpora, the training dataset for the neural machine translation (nmt) model is significantly enriched. consequently, this enrichment empowers the nmt model to excel in its translation performance. back-translation back-translation @xcite is a technique that is used to improve the performance of low-resource translation systems using monolingual data. in this technique, a reverse model is employed to generate a parallel corpus from a monolingual corpus. this is a clever way to use the monolingual corpus to improve the translation performance of the nmt models. we do not include back-translated sentences for training since we could not see any significant performance improvement. transfer learning transfer learning is a machine learning technique where a model trained on one task is adapted for a second related task. instead of starting the training of a new model from scratch, transfer learning leverages the knowledge learned from the first task to improve learning on the second task. we have used indictrans2 (ai4bharat et al., 2023), a powerful model that performs well for english-to-indic and indic-to-english translation for 22 scheduled indian languages. this knowledge can be used to translate other indian languages to and from english. our approach entailed the fine-tuning of this model, leveraging the parallel corpus provided by the wmt23 for the indicmt task. this fine-tuning process equipped the model with the expertise required to proficiently translate assamese and manipuri to and from english, ultimately yielding the most outstanding results. we do not inject phrase pairs since for such a low resource setting, it is difficult to see performance improvements even with phrase pair injections due to the inability of nmt models to capture the low resource language. ## conclusion in this paper, we present how cfilt-iitb utilized the power of multilingual models for the wmt23 indicmt low-resource machine translation of indian languages shared task. since, the data for low-resource languages is scarce, utilizing pretrained multilingual translation models is very crucial. but to have reasonable to good performance over these models, it is helpful to have a model that is trained on similar languages. for this task, indian languages like assamese and manipuri share similar structure and vocabulary with many indian languages like bengali which can be considered a high resource language for india. training models over similar language does boost performance although to cover a wide variety of low-resource languages, one must face the curse of multilingualism. our most proficient system attains an average bleu score of 41.39 for indic-english translation and 22.25 for english-indic language pairs, specifically assamese and manipuri.