Multimodal Neural Machine Translation for E nglish to H indi (2020)

Abstract: Machine translation (MT) focuses on the automatic translation of text from one natural language to another natural language. Neural machine translation (NMT) achieves state-of-the-art results in the task of machine translation because of utilizing advanced deep learning techniques and handles issues like long-term dependency, and context-analysis. Nevertheless, NMT still suffers low translation quality for low resource languages. To encounter this challenge, the multi-modal concept comes in. The multi-modal concept combines textual and visual features to improve the translation quality of low resource languages. Moreover, the utilization of monolingual data in the pre-training step can improve the performance of the system for low resource language translations. Workshop on Asian Translation 2020 (WAT2020) organized a translation task for multimodal translation in English to Hindi. We have participated in the same in two-track submission, namely text-only and multi-modal translation with team name CNLP-NITS. The evaluated results are declared at the WAT2020 translation task, which reports that our multi-modal NMT system attained higher scores than our text-only NMT on both challenge and evaluation test set. For the challenge test data, our multi-modal neural machine translation system achieves Bilingual Evaluation Understudy (BLEU) score of 33.57, Rank-based Intuitive Bilingual Evaluation Score (RIBES) 0.754141, Adequacy-Fluency Metrics (AMFM) score 0.787320 and for evaluation test data, BLEU, RIBES, and, AMFM score of 40.51, 0.803208, and 0.820980 for English to Hindi translation respectively.

## introduction multi-modal nmt aims to draw information from the input data from different modalities like text, image, and audio. by combining information from more than one modality, it attempts to amend the quality of low resource language translation. the work undertaken by @xcite merges the visual features of images from the corresponding input data with textual features of the input bitext to translate sentences, which outperforms text-only translation. for text-only based nmt, encoder-decoder architecture is a widely used technique in the mt community. because it handles various issues like variable-length phrases using sequence to sequence learning, the problem of long term dependency using long short term memory (lstm) @xcite . however, in the case of very long sentences, the basic encoderdecoder architecture is unable to encode all the information. to resolve this issue, the attention mechanism is proposed which pays attention to all source words locally as well as globally @xcite . for indian language translation, attention-based nmt yields remarkable performance @xcite @xcite . besides, without modifying the system architecture, nmt performance can be improved using monolingual data @xcite , which is very effective in the case of low resource language translation. this paper investigates english to hindi translation using the multimodal concept with monolingual data to improve the translation quality at the wat2020 translation task. ## related works the literature survey finds out very limited existing works on english-hindi language pair translation using multi-modal nmt @xcite @xcite . the work by @xcite @xcite . nmt settings of @xcite , achieves bleu score 24.2 for hindi to english translation. moreover, in the wat2019 multi-modal translation task of english to hindi, @xcite based on recurrent neural network (rnn) @xcite achieves bleu score of 12.58, 28.45 for the challenge and evaluation test respectively. and, on the same task of wat2019, we have achieved the highest bleu score of 20.37, 40.55 for the challenge and evaluation test respectively @xcite . we have used rnn encoder and doubly-attentive rnn decoder based model @xcite . the loophole in the existing works of english to hindi translation using multi-modal nmt is that they have not used monolingual data to improve the performance of multi-modal nmt @xcite . in this paper, we have used monolingual corpus in the pre-training step to enhance the performance of the multi-modal nmt for english to hindi translation respectively. ## dataset description hindi visual genome 1.1 consists of parallel text and image data, which is provided by the wat2020 organizers @xcite . the original train parallel text data consists of 28,930 sentences and 28,928 images. we have removed three duplicate sentences (id:2328549, 2385507, 2391240) from the parallel data. also, we have removed one image (id:2326837) from the image dataset since it is not available in train parallel text data. therefore, the total number of parallel sentences and image become 28,927 in hindi visual genome 1.1 dataset as summarized in iitb foot_0 @xcite and english monolingual data from wmt16 foot_1 as shown in ## system description we have used opennmt-py @xcite to setup our multi-modal nmt and text-only nmt systems. the key process of the operations include data preprocessing, system training to generate an optimum trained model, and then obtained trained model is used in the testing/translation process to predict translation on the given unseen data. ## result and analysis the wat2020 translation task organizer declared the evaluation result 3 of multi-modal translation task for english to hindi and our system's results are presented in ## conclusion and future work this work participates in two different translation tracks at wat2020 multi-modal translation task of english to hindi namely: multi-modal and textonly. in this competition our multi-modal nmt achieves higher bleu, ribes and amfm scores than text-only nmt. from the best of our knowledge, our multi-modal nmt achieves best score on english to hindi multi-modal translation. in future work, more experiments, analysis will be carried out to enhance the performance of multi-modal nmt.