NLPH ut’s Participation at WAT 2021 (2021)

Abstract: This paper provides the description of shared tasks to the WAT 2021 by our team “NLPHut”. We have participated in the English→Hindi Multimodal translation task, English→Malayalam Multimodal translation task, and Indic Multi-lingual translation task. We have used the state-of-the-art Transformer model with language tags in different settings for the translation task and proposed a novel “region-specific” caption generation approach using a combination of image CNN and LSTM for the Hindi and Malayalam image captioning. Our submission tops in English→Malayalam Multimodal translation task (text-only translation, and Malayalam caption), and ranks second-best in English→Hindi Multimodal translation task (text-only translation, and Hindi caption). Our submissions have also performed well in the Indic Multilingual translation tasks.

## introduction machine translation (mt) is considered to be one of the most successful applications of natural language processing (nlp) 1 . it has significantly evolved especially in terms of the accuracy of its output. though mt performance reached near to human level for several language pairs (see e.g. @xcite , it remains challenging for low resource languages or translation effectively utilizing other modalities (e.g. image, @xcite ). the workshop on asian translation (wat) is an open evaluation campaign focusing on asian languages since 2013 @xcite . in wat2021 @xcite multimodal track, a new indian language malayalam was introduced for @xmath0, multimodal translation, and malayalam image captioning task. 2 this year, the multiindic foot_1 task covers 10 indic languages and english. in this system description paper, we explain our approach for the tasks (including the subtasks) we participated in: section 2 describes the datasets used in our experiment. section 3 presents the model and experimental setups used in our approach. section 4 provides the official evaluation results of wat2021 foot_2 followed by the conclusion in section 5. ## dataset we have used the official datasets provided by the wat2021 organizers for the tasks. ## experimental details this section describes the experimental details of the tasks we participated in. ## conclusions in this system description paper, we presented our systems for three tasks in wat 2021 in which we participated: (a) @xmath1, (b) @xmath2, and (c) indic multilingual translation task. as the next steps, we plan to explore further on the indic multilingual translation task by utilizing all given data and using additional resources for training. we are also working on improving the region-specific image captioning by fine-tuning the object detection model. gold: एक लड़की टे िनस खे ल रही है gold: आदमी समु द्र में सर्िं फ ग gloss: a girl is playing tennis gloss: man surfing in ocean output:एक टे िनस रै के ट पकड़े हुए आदमी output: पानी में एक व्यिक्त gloss: a man holding a tennis racket gloss: a man in the water gold: एक कु त्ता कू दता है gold: हे लमे ट पहनना gloss: a dog is jumping gloss: wearing helmet output: कु त्ता भाग रहा है output: एक आदमी के िसर पर एक काला हे लमे ट gloss: a dog is running gloss: a black helmet on the head of a person gold: തിളക്കമുള്ള പച്ച ൈകറ്റ് gold: ഒരു വത്തിെല ാഫിക് ൈലറ്റ് gloss: bright green kite gloss: traffic light at a pole output:ആകാശത്ത് പറ ന്ന ൈക-റ്റ് output: ാഫിക് ൈലറ്റ് ചുവപ്പ് തി-ള gloss: kite flying in the sky gloss: the traffic light glows red gold: തൂങ്ങി കിട ന്ന ഒരു കൂട്ടം വാ-ഴപ്പഴം gold: ചുമരിൽ ഒരു ഘടികാരം വാഴ-പ്പഴം gloss: a bunch of hanging bananas gloss: a clock on the wall output: ഒരു കൂട്ടം വാഴപ്പഴം output: ചുമരിൽ ഒരു ചി ം gloss: a bunch of bananas gloss: a picture on the wall