V i TA : Visual-Linguistic Translation by Aligning Object Tags (2021)

Abstract: Multimodal Machine Translation (MMT) enriches the source text with visual information for translation. It has gained popularity in recent years, and several pipelines have been proposed in the same direction. Yet, the task lacks quality datasets to illustrate the contribution of visual modality in the translation systems. In this paper, we propose our system under the team name Volta for the Multimodal Translation Task of WAT 2021 from English to Hindi. We also participate in the textual-only subtask of the same language pair for which we use mBART, a pretrained multilingual sequence-to-sequence model. For multimodal translation, we propose to enhance the textual input by bringing the visual information to a textual domain by extracting object tags from the image. We also explore the robustness of our system by systematically degrading the source text. Finally, we achieve a BLEU score of 44.6 and 51.6 on the test set and challenge set of the multimodal task.

## introduction machine translation deals with the task of translation between language pairs and has been an active area of research in the current stage of globalization. in the task of multimodal machine translation, the problem is further extended to incorporate visual modality in the translations. the visual cues help build a better context for the source text and are expected to help in cases of ambiguity. with the help of visual grounding, the machine translation system has scope for becoming more robust by mitigating noise from the source text and relying on the visual modality as well. in the current landscape of multimodal translation, one of the issues is the limited datasets available for the task. another contributing factor is that often the images add irrelevant information to the sentences, which may act as noise instead of an added feature. the available datasets, like multi30k @xcite , are relatively smaller when compared to large-scale text-only datasets @xcite . the scarcity of such datasets hinders building robust systems for multimodal translation. to address these issues, we propose to bring the visual information to a textual domain and fine-tune a high resource unimodal translation system to incorporate the added information in the input. we add the visual information by extracting the object classes by using an object detector and add them as tags to the source text. further, we use mbart, a pretrained multilingual sequenceto-sequence model, as the base architecture for our translation system. we fine-tune the model on a textual-only dataset released by @xcite consisting of 1,609,682 parallel sentences in english and hindi. further, we finetune it on the training set enriched with the object tags extracted from the images. we achieve state-of-the-art performance on the given dataset. the code for our proposed system is available at https://github.com/kshitij98/vita . the main contributions of our work are as follows: the rest of the paper is organized as follows. we discuss prior work related to multimodal translation. we describe our systems for the textual-only and multimodal translation tasks. further, we report and compare the performance of our models with other systems from the leaderboard. lastly, we conduct a thorough error analysis of our systems and conclude with a direction for future work. ## related work earlier works in the field of machine translation largely used statistical or rule-based approaches, while neural machine translation has gained popularity in the recent past. @xcite released the first deep learning model in this direction, and later works utilize transformer-based approaches @xcite @xcite @xcite for the problem. multimodal translation aims to use the visual modality with the source text to help create a better context of the source text. @xcite first conducted a shared task on the problem and released the dataset, multi30k @xcite . it is an extended german version of flickr30k @xcite , which was further extended to french and czech @xcite . for multimodal translation between english and hindi, @xcite propose a subset of visual genome dataset @xcite and provide parallel sentences for each of the captions. although both english and hindi are spoken by a large number of people around the world, there has been limited research in this direction. dutta chowdhury et al. (2018) created a synthetic dataset for multimodal translation of the language pair and further used the system proposed by calixto and liu (2017 @xcite . laskar et al. (2020) also proposed a similar architecture and pretrained on a large textual parallel dataset @xcite in their system. ## system overview in this section, we describe the systems we use for the task. ## results and discussion the bleu score @xcite is the official metric for evaluating the performance of the models in the leaderboard. the leaderboard further uses ribes @xcite and amfm @xcite metrics for the evaluations. we report the performance of our models after tokenizing the hindi outputs using indic-tokenizer foot_2 in it can be seen that our model is able to generalize well on the challenge set as well and performs better than other systems by a large margin. to ## conclusion we propose a multimodal translation system and utilize the textual-only pre-training of a neural machine translation system, mbart, by extracting object tags from the image. further, we explore the robustness of our proposed multimodal system by systematically degrading the source texts and observe improvements from the textual-only counterpart. we also explore the shortcomings of the currently available object detectors and use groundtruth annotations in our experiments to show the scope of our methodology. the addition of colors and adjectives further adds to the robustness of the system and can be explored further in the future.