Investigation of E nglish to H indi Multimodal Neural Machine Translation using Transliteration-based Phrase Pairs Augmentation (2022)

Abstract: Machine translation translates one natural language to another, a well-defined natural language processing task. Neural machine translation (NMT) is a widely accepted machine translation approach, but it requires a sufficient amount of training data, which is a challenging issue for low-resource pair translation. Moreover, the multimodal concept utilizes text and visual features to improve low-resource pair translation. WAT2022 (Workshop on Asian Translation 2022) organizes (hosted by the COLING 2022) English to Hindi multimodal translation task where we have participated as a team named CNLP-NITS-PP in two tracks: 1) text-only and 2) multimodal translation. Herein, we have proposed a transliteration-based phrase pairs augmentation approach, which shows improvement in the multimodal translation task. We have attained the second best results on the challenge test set for English to Hindi multimodal translation with BLEU score of 39.30, and a RIBES score of 0.791468.

## introduction the multimodal nmt (mnmt) concept aims to include different input modalities, such as images in addition to text and attempts to improve lowresource pair translation by merging visual features in addition to textual features @xcite . the attention-based encoder-decoder architecture for nmt handles various issues of long-term dependency and variable-length phrases via sequenceto-sequence learning and attains a state-of-the-art technique of machine translation (mt) @xcite . also, nmt shows remarkable performance for low-resource indian languages @xcite @xcite @xcite . further, to handle the data scarcity problem, the authors @xcite augmenting phrase pairs and the source language transliteration-based @xcite approach to enhance text-only based for low-resource pair translation. this paper aims to investigate the english to hindi multimodal translation task in wat2022 with a proposed transliteration-based phrase pairs augmentation approach (as discussed in 3.2). the rest of the paper is organized as follows: section 2 presents the review of related works. the system description is briefly discussed in section 3. section 4 reports the results and section 5 concludes the paper with future scope. ## related works the literature survey explores existing works on mnmt for english-hindi language pair @xcite @xcite . we participated in wat2020 on multimodal translation task for english to hindi translation and attained the best results with a bleu score of 33.57 on the challenge test set(laskar et al., 2020b) using rnn-based mnmt model @xcite and taking advantage of pretrained word embeddings of the monolingual corpus. later, we improved the results in wat2021 @xcite using phrase pairs augmentation. in this work, we have investigated a proposed transliteration-based phrase pairs augmentation approach to enhance the multimodal translational performance of english to hindi. ## system description the experiments are carried out in four operations, namely, transliteration-based phrase pairs augmentation, data preprocessing, model training, and testing. the opennmt-py @xcite tool is utilized to build multimodal and text-only models independently. the difference between our previous work @xcite and this work is that the current work uses transliteration-based phrase pairs augmentation. ## result and analysis the wat2022 shared task organizer @xcite published the evaluation result foot_3 of the multimodal translation task for english to hindi. we participated with the team name cnlp-nits-pp in the multimodal and text-only submission tracks of the same task where four teams participated. the automatic evaluation metrics, bleu ## conclusion and future work in this work, we have proposed the use of transliteration-based phrase pairs augmentation in the wat2022 multimodal translation task for english to hindi translation. our multimodal nmt attained a higher score than that of the text-only nmt model and existing work of @xcite . a multilingual-based approach will be investigated to improve the translational performance of low-resource multimodal nmt.