IITP at WAT 2021: System description for E nglish- H indi Multimodal Translation Task (2021)

Abstract: Neural Machine Translation (NMT) is a predominant machine translation technology nowadays because of its end-to-end trainable flexibility. However, NMT still struggles to translate properly in low-resource settings specifically on distant language pairs. One way to overcome this is to use the information from other modalities if available. The idea is that despite differences in languages, both the source and target language speakers see the same thing and the visual representation of both the source and target is the same, which can positively assist the system. Multimodal information can help the NMT system to improve the translation by removing ambiguity on some phrases or words. We participate in the 8th Workshop on Asian Translation (WAT - 2021) for English-Hindi multimodal translation task and achieve 42.47 and 37.50 BLEU points for Evaluation and Challenge subset, respectively.

## introduction recent progress in neural machine translation (nmt) focuses on translating a source language into a particular target language. various methods have been proposed for this task and most of them deal with the textual data. there are certain drawbacks while performing machine translation using only textual datasets. human performs translation which is based upon language grounding: our sense of meaning emerges from interacting with the world. nmt methods do not have any mechanism to perform language grounding; thus they are devoid of capturing the true meaning of sentences or phrases while translating them into the other languages. for example, it * equal contribution needs to translate the word "cricket", it can get confused if it is the game cricket or the insect cricket. but the visual information can clear the ambiguity. multi-modal translation aims to alleviate this issue by training an nmt model on textual data along with associated images to perform language grounding. this shared task deals with developing multi-modal nmt models for english-hindi translation. the choice of languages depends on the following issues: i). hindi is the most spoken language in india and the fourth most spoken language in the world with 600 million speakers 1 . despite the huge amount of speakers, suitable resources in hindi is limited due to the various factors. ii) automatic translation of texts from one language to the another is a difficult task. specifically, when one or both of them are resource-poor and distant from each other. in multimodal nmt (mnmt), information from the other modalities like audio, image, video, etc. are used along with text to generate the translation. in low-resource languages, this is particularly used to improve the low-quality translations as even though vocabularies, grammar of two languages are different but their visual representation is the same. there are several proposed multi-modal methods for translations that exploit the features of the associated image for better translation. stateof-the-art methods might achieve better accuracy than the models we used. our main motivation for using simplistic models is to demonstrate a proof-of-concept to be used for multi-modal translation among the resource-poor language pairs. we achieved good results on both challenge and evaluation set in different evaluation metrics including bleu, ribes, amfm. in subsequent modifications, we aim to develop our models incorporating several state-of-the-art features. the following sections describe our processes in greater details. ## related works there have been many attempts to use information other than the source for better translation. uni-modal systems include document-level nmt @xcite , sentence-level nmt with contextual information @xcite , etc. among multimodal systems, @xcite used an object detection system and extracted local and global image features. thereafter, they used those image features as additional inputs to encoder and decoder. @xcite used attention mechanism on visual inputs for the source hidden states. ( lin et al., 2020) ## experimental results we obtain impressive results on our submissions. there are two sets designed for evaluating our model, i) evaluation set, ii) challenge set. we evaluate our model on both of these test set and tabulate our results in ## conclusion we participate in wat-2021 multimodal translation task for english to hindi. we achieve good results on both the challenge and evaluation sets achieving 42.47 and 37.50 bleu points, respectively. we rank second place on evaluation set and third place on challenge set on wat-2021 multimodal translation task for english to hindi. in future, we would like to extend our work by training with additional monolingual data and better ways to incorporate multimodal features.