A comparative study of transformer and transfer learning MT models for E nglish- M anipuri (2023)

Abstract: In this work, we focus on the development of machine translation (MT) models of a lowresource language pair viz. English-Manipuri. Manipuri is one of the eight scheduled languages of the Indian constitution. Manipuri is currently written in two different scripts: one is its original script called Meitei Mayek and the other is the Bengali script. We evaluate the performance of English-Manipuri MT models based on transformer and transfer learning technique. Our MT models are trained using a dataset of 69,065 parallel sentences and validated on 500 sentences. Using 500 test sentences, the English to Manipuri MT models achieved a BLEU score of 19.13 and 29.05 with mT5 and OpenNMT respectively. The results demonstrate that the OpenNMT model significantly outperforms the mT5 model. Additionally, Manipuri to English MT system trained with OpenNMT model reported a BLEU score of 30.90. We also carried out a comparative analysis between the Bengali script and the transliterated Meitei Mayek script for English-Manipuri MT models. This analysis reveals that the transliterated version enhances the MT model performance resulting in a notable +2.35 improvement in the BLEU score.

## introduction in an increasingly interconnected world, the role of machine translation cannot be overstated. it serves as a critical bridge for breaking down linguistic barriers and enabling effective communication across diverse cultures and languages. however, the efficacy of machine translation (mt) systems largely depends on the availability of adequate linguistic resources, particularly parallel corpora which are essential for training and finetuning neural machine translation models. while widely spoken languages benefit from abundant parallel data, many minority and indigenous languages with scant linguistic resources available for their inclusion in the machine translation landscape are left in the shadows of the digital age. manipuri language called meiteilon is mainly spoken in the state of manipur which lies in the northeastern part of india. some speakers exist in the states of assam, tripura and mizoram few speakers are also located in the country like bangladesh and myanmar. manipuri language is also facing the challenge of linguistic resource scarcity. this situation impedes access to information and hampers communication for manipuri speakers in an increasingly globalized world. bridging this linguistic gap is not only a matter of preserving cultural heritage but also essential for promoting effective communication, education and access to vital information. in this paper, we embark on a journey to analyse the challenge of english to manipuri mt in a low-resource setting. our approach hinges on the powerful techniques of transfer learning and pretrained models which have shown remarkable success in natural language processing tasks including machine translation. transfer learning allows us to harness the knowledge learned from high-resource languages and adapt it to the low-resource english-manipuri translation task. our goal is to investigate the potential of these techniques in enhancing the translation quality and fluency for manipuri, despite the constraints of limited parallel data. by leveraging the wealth of linguistic information embedded in pre-trained models, we aim to bridge the language gap and contribute to the development of language technology for minority languages like manipuri. this paper is organized as follows: in section 2, we provide an overview of related work in the fields of machine translation, transfer learning, and low-resource languages. section 3 outlines the methodology employed, data preparation, and use of opennmt @xcite and mt5 @xcite pre-trained models for english to ma-nipuri mt. section 4 provides the results and evaluation of the mt models using automatic metrics and qualitative analysis. in section 5, we present insightful conclusions derived from the key findings. ## related work over the past decade, numerous studies have been conducted on mt models for low-resource languages @xcite including unsupervised (singh and singh, 2020), transfer learning @xcite and multimodal @xcite ) approaches among others. singh and bandyopadhyay (2010) carried out a study on supervised statistical methods in which the authors conducted a persuasive examination of the impact of morphosyntactic information and dependencies in the context of statistical machine translation employing bengali script. the field of mt has experienced substantial progress primarily propelled by the introduction of neural machine translation (nmt) models as evidenced by @xcite . these innovative deep learning techniques have gradually supplanted traditional phrase-based and statistical methods resulting in remarkable enhancements in translation quality. nonetheless, the effectiveness of these systems critically hinges on the availability of parallel corpora which consist of matching sentences in both the source and target languages @xcite . high-resource languages such as english, spanish and chinese benefit from extensive parallel dataset, leading to precise and fluent translations. conversely, low-resource languages @xcite often spoken by marginalized communities grapple with significant challenges in procuring adequate training data for machine translation. transfer learning has the potential to help lowresource languages overcome challenges by leveraging insights from rich languages. this method @xcite has gained prominence in natural language processing domains, including machine translation. techniques like cross-lingual embedding and multilingual pre-trained models have been explored, enabling models to adapt and excel in resource-scarce environments. crosslingual embedding involves translating words or phrases from diverse languages into a shared vector space while multilingual pre-trained models capture cross-lingual representations during ini-tial training allowing for fine-tuning on languagespecific tasks. these strategies have shown promising results in low-resource machine translation scenarios, providing hope for underrepresented languages, such as minority or indigenous languages which face a critical threat of extinction due to a lack of support for documentation, educational materials and communication tools. this paper explores the potential of transfer learning and pre-trained models to improve english to manipuri translation in low-resource contexts. ## methodology in this section, we describe the methodology used to develop english-manipuri mt systems in a lowresource setting (figure 1 ). our approach centers around supervised transformer based mt model and transfer learning based mt model to adapt the specific characteristics of the english-manipuri language pair. pre-processing normalisation tokenizer opennmt mt5 dataset collection english to manipuri (bengali script) bleu score comparision language pair transliteration bengali to meitei mayek ## conclusion in this paper, we report a comprehensive study by implementing mt models using transformer based model (opennmt) and transfer learning technique using pretrained model (mt5) for english-manipuri in a low resource setting. the experimental results demonstrate that opennmt model outperform the mt5 model by a factor of 9 in terms of bleu score. the performance of manipuri to english using opennmt model report a bleu score of 30.90 using bengali script. opennmt model shows a promising results of bleu score of 33.25 for english to manipuri languages pair using transliterated meitei mayek script. despite the fact that transfer learning technique emerge as an approach to build machine translation models for resource-constrained languages by leveraging pre-trained models, the study's findings indicate that this technique performs less effectively than the transformer model trained on the same parallel dataset. future research may be needed to explore the nuances of transfer learning in this specific domain and to identify potential refinements or enhancements to improve its efficacy for machine translation in resource-constrained languages. based on subjective evaluation, the translation quality is generally deemed satisfactory taking into account of the small dataset and the use of a single test reference dataset. this research underscores the potential of supervised transformer based model and transfer learning technique using particulars of input/output and models sample input and output sample 1 1eng (english) long term measures 1mnib(manipuri in bengali script) অশাংবা মতমগী ওইবা েথৗরাংশীং 1mnim(manipuri in meetei mayek) ꯑꯁꯥ ꯡꯕꯥ ꯃꯇꯃꯒꯤ ꯑꯣꯢꯕꯥ ꯊꯩ ꯔꯥ ꯡꯁꯤꯡ eng to mnib 1mt5 output মতম শাংনা পাংেথাকপা পাৈ িশং 1opennmt output অশাংবা মতমগী ওইবা পাৈ শীং eng to mnim 1mni outputm(meetei mayek) ꯑꯁꯥ ꯡꯕꯥ ꯃꯇꯃꯒꯤ ꯑꯣꯢꯕꯥ ꯄꯥ ꯝ ꯭ ꯕ ꯩ ꯁꯤꯡ mnib to eng 1opennmt bt output: long term measures sample 2 2eng lack of financial resources 2mnib ফাইনাি এল িরেসাসর্ শীং হ বা 2mnim ꯐꯥ ꯢꯅꯥ ꯟ ꯭ ꯁ ꯤꯑꯦꯂ ꯔꯤꯁꯣꯔ ꯭ ꯁ ꯁꯤꯡ ꯍꯟ ꯭ ꯊ ꯕꯥ eng to mnib 2mt5 output ফাইনাি এল িরেসাসর্ শীং ৱাৎপা 2opennmt output ফাইনাি এল িরেসাসর্ শীং য়ামদবা eng to mnim 2mni outputm ꯐꯥ ꯢꯅꯥ ꯟ ꯭ ꯁ ꯤꯑꯦꯂ ꯔꯤꯁꯣꯔ ꯭ ꯁ ꯁꯤꯡ mnib to eng 2opennmt bt output decreased financial resources, sample 3 3eng every pregnancy is special and every pregnant woman must receive special care . 3mnib মীেরানবা খু িদংমক মরুওই অমসু ং মীেরানবী নু পী খু িদংম া অখ বা েকয়র েশায়দনা ফংগদবিন । 3mnim ꯃꯤꯔꯣꯅꯕꯥ ꯈꯨ ꯗꯤꯡꯃꯀ ꯃꯔꯨ ꯑꯣꯢ ꯑꯃꯁꯨ ꯡ ꯃꯤꯔꯣꯅꯕꯤ ꯅꯨ ꯄꯤ ꯈꯨ ꯗꯤꯡꯃꯛ ꯭ ꯅ ꯥ ꯑꯈꯟ ꯭ ꯅ ꯕꯥ ꯀꯦꯌꯔ ꯁꯣꯌꯗꯅꯥ ꯐꯡꯒꯗꯕꯅꯤ ꯫ eng to mnib 3mt5 output মীেরানবা খু িদংমক অখ বা ওই অমসু ং মীেরানবী নু পী পু ম া অখ বা েকয়র েলৗগদবিন । 3opennmt output মীেরানবা খু িদংমক মরুওই অমসু ং মীেরানবী নু পী খু িদংম া অখ বা েকয়র ফংহনবা তাই । eng to mnim 3mni outputm ꯃꯤꯔꯣꯅꯕꯥ ꯈꯨ ꯗꯤꯡꯃꯀ ꯃꯔꯨ ꯑꯣꯢ ꯑꯃꯁꯨ ꯡ ꯃꯤꯔꯣꯅꯕꯤ ꯅꯨ ꯄꯤ ꯈꯨ ꯗꯤꯡꯃꯛ ꯭ ꯅ ꯥ ꯑꯈꯟ ꯭ ꯅ ꯕꯥ ꯀꯦꯌꯡꯔ ꯂꯩ ꯒꯗꯕꯅꯤ ꯫ mnib to eng 3opennmt bt output every pregnancy is special and every pregnant woman should have access to specialized facilities .