H indi- M arathi Cross Lingual Model (2020)

Abstract: Machine Translation (MT) is a vital tool for aiding communication between linguistically separate groups of people. The neural machine translation (NMT) based approaches have gained widespread acceptance because of its outstanding performance. We have participated in WMT20 shared task of similar language translation on Hindi-Marathi pair. The main challenge of this task is by utilization of monolingual data and similarity features of similar language pair to overcome the limitation of available parallel data. In this work, we have implemented NMT based model that simultaneously learns bilingual embedding from both the source and target language pairs. Our model has achieved Hindi to Marathi bilingual evaluation understudy (BLEU) score of 11.59, rank-based intuitive bilingual evaluation score (RIBES) score of 57.76 and translation edit rate (TER) score of 79.07 and Marathi to Hindi BLEU score of 15.44, RIBES score of 61.13 and TER score of 75.96.

## introduction mt is a well-known task of natural language processing (nlp) wherein automatic translation is performed between different languages. broadly, mt is categorized into rule-based and corpus-based, where rule-based is based on a pre-defined rules on the concerned languages and corpus-based finds a generalized approach after being trained on a large corpus. mt switches from rule-based approach to the corpus-based which blots out the need for linguistic expertise. in the corpus-based approach, example-based machine translation (ebmt), statistical machine translation (smt) and nmt techniques are available. the disadvantage of ebmt is that even though the corpus is large, all examples are not covered. to mitigate the issues of the contemporary approach smt is introduced @xcite ; @xcite . the smt based system makes an assumption based on probability scores of the translated text. and hence, the ranking is done. smt also faces many issues like system complexity, long term dependency problem, context-analyzing inability, word-alignment and the rare word problem. the inefficiency of smt leads to the development of the nmt @xcite . but like smt, the nmt based model also suffers the requirement of sufficient training parallel corpus, which is a challenge in the case of low resource languages. for this reason, there is a demand for direct translation among similar language pairs by utilizing similarity features and monolingual data, so that less availability of the parallel data does not pose a challenge. however, the nmt technique achieves state-of-the-art approach in mt because of its transformer model @xcite . for low resource language pair translation, nmt models have been improved with monolingual corpus @xcite ; @xcite ; @xcite . in this work, we have adopted cross-lingual language model (xlm) @xcite to implement an nmt model for hindi-marathi similar language translation task because xlm shows significant improvements for low-resource languages by utilizing the monolingual corpora. ## related work hindi-marathi translation lacks background work. however, similar work is found on hindi-nepali pair at wmt19 shared task of similar language translation @xcite . the literature survey mainly focuses on nmt for low resource language pairs since nmt outperforms conventional smt on low resource pairs like english to mizo, english to hindi, english to punjabi, and english to tamil @xcite ; @xcite ; @xcite . it is noticed that train- ## system description our approach consists of the two principal approaches viz. the pre-training step and the finetuning step which are discussed in the following sub-sections 4.1 and 4.2. . ## experimental setup we have trained the transformer based cross language model (xlm) @xcite also known as mlm + tlm task. we have used 6 layers with 8 attention heads. an embedding layer is also used with size 256. given the comparatively smaller marathi dataset as discussed in section 3.1, and limited availability of computational resources foot_0 we trained the smaller model instead of the usual 12 layers and 16 attention heads as proposed by @xcite . batch size of 32 was used. following settings of @xcite , attention dropout was set to 0.1, gelu activation was used. also, adam was used as an optimizer with an initial learning rate of 0.0001. rest of the parameters are same as used by conneau and @xcite in their experiments and as given in their github repository foot_1 . ## result and analysis the wmt20 organizer declared result for the shared task of similar language translation on hindi to marathi foot_2 and marathi to hindi foot_3 and the results of our system's is reported in our system's rank is 10 with bleu score 11.59 for hindi to marathi translation type source: hindi target: marathi short source test sentence अवसर क समानता है । predicted test sentence सं धीची समानता आहे . google translation सं ध ची समानता आहे . bing translation सं ध ची समानता आहे . medium source test sentence यह मे रे लए एक ब त ही सु खद अनु भू त रही है । predicted test sentence ही मा यासाठ अ तशय सु खद अनु भू ती आहे . google translation ही मा यासाठ खू प आनं ददायी भावना आहे . bing translation मा यासाठ ही खू प सु खद भावना आहे . long source test sentence बि क यह एक सकारा मक शां त है जहां हम सब क णा और ान के आधार पर सं वाद, स ाव और याय को बढ़ावा दे ने के लए काम करते ह । predicted test sentence ही एक सकारा मक शां तता आहे जथे आपण क णा आ ण ाना या आधारे सं वाद स ावने ला ो साहन दे यासाठ काम करतो. google translation उलट ही एक सकारा मक शां ती आहे जथे आपण सव क णा आ ण ानावर आधा रत सं वाद, सु सं वाद आ ण यायाला चालना दे यासाठ काय कर त आहोत. bing translation उलट ही एक सकारा मक शां ती आहे जथे आपण सव जण क णा आ ण ानावर आधा रत सं वाद, सामं ज य आ ण याय ◌ाला ो साहन दे याचे काम करतो. ## conclusion and future work our nmt system adopts cross lingual model for a similar language translation task of hindi-marathi pair in both forward and backward directions. the evaluated result and in-depth analysis of the predicted sentences shows that our nmt system performs well for the short and medium types of sentences and shows poor performance in long sentences. however, our nmt system needs more marathi monolingual corpus and in the future works, multilingual nmt system will be developed to overcome the limitation of corpus for such low resource language pair translation.