Improved E nglish to H indi Multimodal Neural Machine Translation (2021)

Abstract: Machine translation performs automatic translation from one natural language to another. Neural machine translation attains a state-of-the-art approach in machine translation, but it requires adequate training data, which is a severe problem for low-resource language pairs translation. The concept of multimodal is introduced in neural machine translation (NMT) by merging textual features with visual features to improve low-resource pair translation. WAT2021 (Workshop on Asian Translation 2021) organizes a shared task of multimodal translation for English to Hindi. We have participated the same with team name CNLP-NITS-PP in two submissions: multimodal and text-only NMT. This work investigates phrase pairs injection via data augmentation approach and attains improvement over our previous work at WAT2020 on the same task in both text-only and multimodal NMT. We have achieved second rank on the challenge test set for English to Hindi multimodal translation where Bilingual Evaluation Understudy (BLEU) score of 39.28, Rank-based Intuitive Bilingual Evaluation Score (RIBES) 0.792097, and Adequacy-Fluency Metrics (AMFM) score 0.830230 respectively.

## introduction multimodal nmt (mnmt) intends to draw insights from the input data through different modalities like text, image, and audio. combining information from more than one modality attempts to amend the quality of low resource language translation. @xcite show, combining the visual features of images with corresponding textual features of the input bitext to translate sentences outperform text-only translation. encoder-decoder architecture is a widely used technique in the mt community for text-only-based nmt as it handles various issues like variable-length phrases using sequence to sequence learning, the problem of longterm dependency using long short term memory (lstm) @xcite . nevertheless, the basic encoder-decoder architecture cannot encode all the information when it comes to very long sentences. the attention mechanism is proposed to handle such issues, which pays attention to all source words locally and globally @xcite . the attention-based nmt yields substantial performance for indian language translation @xcite @xcite @xcite . moreover, nmt performance can be enhanced by utilizing monolingual data @xcite @xcite and phrase pair injection @xcite , effective in low resource language pair translation. this paper aims english to hindi translation using the multimodal concept by taking advantage of monolingual data and phrase pair injections to improve the translation quality at the wat2021 translation task. ## related works for the english-hindi language pair, the literature survey revealed minor existing works on translation using multimodal nmt @xcite @xcite . @xcite uses synthetic data, following multimodal nmt settings @xcite , and attains a bleu score of 24.2 for hindi to english translation. however, in the wat 2019 multimodal translation task of english to hindi, we achieved the highest bleu score of 20.37 for the challenge test set @xcite . this score was improved later in the task of wat2020 @xcite to obtain the bleu score of 33.57 on the challenge test set. in (laskar et al., 2020c), we have used bidirectional rnn (brnn) at encoder type, and doubly-attentive rnn at decoder type following default settings of @xcite and utilizes pre-train word embeddings of the monolingual corpus and additional parallel data of iitb. this work attempts to utilize phrase pairs @xcite to enhance the translational performance of the wat2021: english to hindi multimodal translation task. ## system description to build multimodal and text-only nmt models, opennmt-py @xcite ) tool is used. there are four operations which include data augmentation, preprocessing, training and testing. our multi-model nmt gets advantages from both image and textual features with phrase pairs and word embeddings. ## result and analysis the wat2021 shared task organizer published the evaluation result foot_4 of multimodal translation task for english to hindi and our team stood second position in multimodal submission for challenge test set. our team name is cnlp-nits-pp, and we have participated in the multimodal and text-only submission tracks of the same task. in both multimodal and text-only translation submission tracks, a total of three teams participated in both evaluation and challenges test data. the results are evaluated using automatic metrics: bleu @xcite , ribes @xcite and amfm @xcite . the results of our system is reported in it is because the combination of textual and visual features outperforms text-only nmt. furthermore, our system's results are improved as compared to our previous work on the same task @xcite . it shows the bleu, ribes, amfm scores of present work show (+5.71, +9.41), (+0.037956, +0.055641), (+0.04291, +0.04835) increments on the challenge test set for multimodal and text-only nmt, where it is realised that phrase pairs augmentation improves translational performance. the sample examples of best and worst outputs, along with google translation and transliteration of hindi words, are presented in figure 2 and 3. in figure 2 and 3, highlighted the region in the image for the given caption by a red colour rectangular box. ## conclusion and future work in this work, we have participated in a shared task at wat2021 multimodal translation task of english to hindi, where translation submitted at tracks: multimodal and text-only. this work investigates phrase pairs through data augmentation approach in both multimodal and text-only nmt, which shows better performance than our previous work on the same task @xcite . in future work, we will investigate a multilingual approach to improve the performance of multimodal nmt.