Attentive fine-tuning of Transformers for Translation of low-resourced languages @ L o R es MT 2021 (2021)

Abstract: This paper reports the Machine Translation (MT) systems submitted by the IIITT team for the English→Marathi and English⇔Irish language pairs LoResMT 2021 shared task. The task focuses on getting exceptional translations for rather low-resourced languages like Irish and Marathi. We fine-tune IndicTrans, a pretrained multilingual NMT model for English→Marathi, using external parallel corpus as input for additional training. We have used a pretrained Helsinki-NLP Opus MT English⇔Irish model for the latter language pair. Our approaches yield relatively promising results on the BLEU metrics. Under the team name IIITT, our systems ranked 1, 1, and 2 in English→Marathi, Irish→English, and English→Irish respectively. The codes for our systems are published1 .

## introduction today, a large number of text and written materials are present in english. however, with roughly around 6,500 languages in the world foot_1 @xcite @xcite , every native monoglot should not be deprived of this knowledge and information. the manual translation is a tedious job involving much time and human resources, giving rise to machine translation (mt). machine translation involves the automated translation of text from one language to another by using various algorithms and resources to produce quality translation predictions @xcite @xcite . neural machine translation (nmt) brought about a great improvement in the field of mt by overcoming flaws of rule-based and statistical machine translation (smt) @xcite @xcite @xcite . nmt incorporates the training of neural networks on parallel corpora to predict the likeliness of a sequence of words. sequence-to-sequence neural models (seq2seq) @xcite are the widely adopted as the standard approach by both industrial and research communities @xcite @xcite . even though nmt performs exceptionally well for all the languages, it requires a tremendous amount of parallel corpus to produce meaningful and successful translations @xcite . with little research on low resourced languages, finding a quality parallel corpus to train the models can be arduous. the two low-resourced languages worked on in this paper are marathi (mr) and irish (ga). with about 120 million marathi speakers in maharashtra and other states of india, marathi is recognized as one of the 22 scheduled languages of india foot_2 . the structural dissimilarity which occurs while translating from english (subject-verb-object) to marathi (subject-object-verb) or vice versa adds up to issues faced while translation @xcite . the irish language was recognized as the first official language of ireland and also by the eu @xcite . belonging to the goidelic language family and the celtic family @xcite , irish is also claimed as one of the low resourced languages due to its limited resources by the meta-net report @xcite . our paper represents the work conducted for the loresmt @ mt summit 2021 foot_3 shared task to build mt systems for the low-resourced marathi and irish languages on covid-19 related parallel corpus. we implement transformer-based @xcite nmt models to procure bleu scores @xcite of 24.2, 25.8, and 34.6 in @xmath0, @xmath1, and @xmath2. ## related works neural machine translation has been exhaustively studied over the years @xcite , with several intuitive approaches involving collective learning to align and translate @xcite , and a language-independent attention bridge for multilingual translational systems @xcite . there have been several approaches to nmt, with zero-shot translational systems, between language pairs that have not seen the parallel training data during training @xcite . the introduction of artificial tokens has reduced the architectural changes in the decoder @xcite . there have been some explorations towards neural machine translation in low resource languages, with the development of a multisource translational system that targets the english string for any source language @xcite . there has been subsequent research undertaken by researchers for machine translation in low-resource indian languages. chakravarthi et al. surveyed orthographic information in machine translation, examining the orthography's influence on machine translation and extended it to under-resourced dravidian languages @xcite . another approach of leveraging the information contained in rule-based machine translation systems to improve machine translation of low-resourced languages was employed @xcite . several approaches involving the improvement of wordnet for low-resourced languages have been explored @xcite . @xcite mmdravi, a multilingual multimodal machine translation dataset for low-resourced dravidian languages, extending it from the flickr30k dataset, and generating translations for the captions using phonetic transcriptions (u @xcite . there have been relatively fewer approaches experimented with and benchmarked when it comes to translating from marathi to english and vice versa. @xcite tried to build multilingual nmt systems, comprising 103 distinct languages and 204 translational directions simultaneously. @xcite developed a machine translation system for marathi to english using transformers on a parallel corpus. other works on improving machine translation include @xcite proposing an english-marathi nmt using local attention. the same can be stated to irish, as it is a poorly resourced language, as the quality of the mt outputs have struggled to achieve the same level as well-supported languages @xcite . in recent years, several researchers tried to overcome the resource barrier by creating artificial parallel data through back-translation @xcite , exploiting out-of-domain data @xcite , and leveraging other better-resourced languages as a pivot @xcite . ## dataset we use the dataset provided by the organizers of loresmt @ mt summit 2021. the datasets can be found here foot_4 . it is a parallel corpus for english and the low resourced language, i.e., irish and marathi, mostly containing text related to covid-19 @xcite . we have used bible-uedin foot_5 @xcite an external dataset for marathi. it is a multilingual parallel corpus dataset containing translations of the bible in 102 languages @xcite ) and shows the possibility of using the bible for research and machine translation. english-marathi corpus contains 60,495 sentences. cvit pib foot_6 @xcite has also been used for the purpose of this research. it contains 1,14,220 parallel corpora for english-marathi. ## results and analysis for marathi, it is distinctly visible that our system model, i.e., indictrans fine-tuned on the training data provided by the organizers or tra and the bible-uedin dataset, gave the best bleu scores. it was surprising how the model fine-tuned on a parallel corpus of 60,495 sentences of bible-uedin surpassed the model fine-tuned on 1,14,220 sentences from the cvit pib dataset. the possible explanation is the higher correlation between the sentences of the bible-uedin dataset with the test dataset than the cvit pib dataset. another reason could be the presence of excessive noise in the cvit pib dataset. the other reason for this could be noise and a lower quality of translations in the cvit pib dataset compared to bible-uedin. to infer the same, 1000 random pairs of sentences were picked from the datasets, and the average labse or language-agnostic bert sentence embedding @xcite ) scores were found out. labse gives a score between 0 and 1, depending on the quality of the translation. it was seen that the average was 0.768 for the bible-uedin dataset, while it was 0.58 for the cvit pib dataset. this might have been one of the reasons for the better bleu scores. the model also showed constant overfitting after the second and third epoch, as the bleu scores reduced considerably as they reached the 6th epoch. the bleu scores decreased by a difference of 6.the validation loss starts increasing after the third epoch, thus, showing the overfitting occurring in training. so, the model was fine-tuned for three epochs while maintaining a low learning rate of around 3e-5 to get a bleu score of 24.2. @xcite training a model to predict for a low-resourced language was highly challenging due to the absence of prominent pretrained models @xcite @xcite . however, as an experiment, two models from huggingface transformers 11 , m2m100 @xcite and opus-mt from helsinki nlp (tiedemann, 2020) were compared. for the dev data, opus mt produced a bleu score of 30.4 while m2m100 gave 25.62 for translations from english to irish and 37.2 and 30.37 respectively for irish to english translations. probably, the individual models pretrained on numerous datasets gave opus mt an edge over m2m100. this led us to submit the opus mt model for the loresmt shared task 2021. the model gave exceptional bleu scores of 25.8 for english to irish, which ranked second in the shared task, while 34.6 for irish to english stood first. ## conclusion it is arduous and unyielding to get accurate translations for low-resourced languages due to limited datasets and pretrained models. however, our paper puts forward a few methods to better the already existing accuracies. ranked 1, 1, and 2 in @xmath3, @xmath4, and @xmath5, indictrans fine-tuned on the bible-uedin, and the dataset provided by the organizers manages to surpass the other models due to its high correlation with the test set and minimal noise for the marathi language. the irish language task was dominated by the opus mt model by helsinki-nlp, outperforming other transformer models, m2m100.