Silo NLP ’s Participation at WAT 2022 (2022)

Abstract: This paper provides the system description of “Silo NLP’s” submission to the Workshop on Asian Translation (WAT2022). We have participated in the Indic Multimodal tasks (English->Hindi, English->Malayalam, and English->Bengali, Multimodal Translation). For text-only translation, we used the Transformer and fine-tuned the mBART. For multimodal translation, we used the same architecture and extracted object tags from the images to use as visual features concatenated with the text sequence for input. Our submission tops many tasks including English->Hindi multimodal translation (evaluation test), English->Malayalam text-only and multimodal translation (evaluation test), English->Bengali multimodal translation (challenge test), and English->Bengali text-only translation (evaluation test).

## introduction machine translation (mt) is a classic sub-field in nlp which investigates the usage of computer software to translate text or speech from one language to another without human involvement @xcite . although mt performance has reached near the level of human translators for many highresource languages, it remains challenging for many low-resource languages @xcite . also, effective usage of other modalities (e.g. image) in mt is an important research area in the past few years @xcite . the wat is an open evaluation campaign focusing on asian languages since 2013 @xcite . in the wat2022 multimodal track, a new indian language bengali was introduced for @xmath0, multimodal translation, and bengali image captioning task. 1 1 https://ufal.mff.cuni. cz/bengali-visual-genome/ wat-2022-english-bengali-multim the multimodal translation tasks in wat2022 consist of image caption translation, in which the input is a descriptive source language caption together with the image it describes, while the output is a target language caption. the multimodal input enables the use of image context to disambiguate source words with multiple senses. in this system description paper, we explain our approach for the tasks (including the sub-tasks) we participated in: task 1: @xmath1: @xmath2: @xmath3 ## data sets we used the data sets specified by the organizer for the related tasks along with additional synthetic data for performance improvement. the use of additional data places some foot_0 of our submissions in the unconstrained track. task 1: @xmath4, the organizers provided hindivisu-algenome 1.1 @xcite 3 dataset (hvg for short). the training part consists of 29k english and hindi short captions of rectangular areas in photos of various scenes and it is complemented by three test sets: development (d-test), evaluation (e-test) and challenge test set (c-test). our wat submissions were for e-test (denoted "ev" in wat official tables) and c-test (denoted "ch" in wat tables). for the synthetic image features, we use the flickr8k data set @xcite . even though it is an image captioning data set, we discard the images, treating the data set as in-domain monolingual data. we use a machine translation into hindi @xcite as the target side, and generate image features using the procedure described in section 3.4. the statistics of the datasets are shown in task 2: @xmath5, the organizers provided malayalamvisualgenome 1.0 dataset foot_2 (mvg for short). mvg is an extension of the hvg dataset for supporting malayalam, which belongs to the dravidian language family @xcite . the dataset size and images are the same as hvg. while hvg contains bilingual english-hindi segments, mvg contains bilingual english-malayalam segments, with the english, shared across hvg and mvg, see task 3: @xmath6, the organizers provided ben-galivisualgenome 1.0 dataset foot_3 (bvg for short). bvg is an extension of the hvg dataset for supporting bengali. the dataset size and images are the same as hvg, and mvg, see ## experimental details this section describes the experimental details of the tasks we participated in. ## results we report the official automatic evaluation results of our models for all the participating tasks in on the e-test sets, our multimodal systems receive the highest bleu scores for @xmath7, and our text-only systems outperform other text-only systems for @xmath8. our multimodal systems consistently outperform our text-only systems, with increases between +1.1 and +10.2 bleu. on the c-test challenge sets, we have the highest bleu score for @xmath9. again, our multimodal systems outpertype of data data set images source text target text sentences mmt hindi visual genome ✓ ✓ ✓ 29k monolingual flickr8k + ✓ + 40k captioning translation mt feat. synth. figure 3: process for generating synthetic image features. green arrows indicate processes to synthesize data, and green plus signs indicate the resulting synthetic data. mt is short for machine translation, feat. synth. for image feature synthesis. wat bleu human note system and wat task label silo nlp best comp silo nlp best comp @xmath10.2 42.9 mmevmm22en-hi 42.0 39.4 mmchtext22en-hi 29.6 41.8 2.715 3.078 mmchmm22en-hi 39.1 39.3 @xmath11.8 30.6 mmevmm22en-ml 41.0 -mmchtext22en-ml 14.6 19.5 2.013 not available mmchmm22en-ml 20.4 -@xmath12.0 41.0 mmevmm22en-bn 42.1 43.9 mmchtext22en-bn 22.6 32.9 2.658 3.525 competitor used external data mmchmm22en-bn 28.7 28.7 it should be noted that our @xmath16, making use of substantial additions of in-domain data. for these language pairs, the increase in translation quality can not be attributed entirely to multimodality. however, for @xmath17, making these systems comparable. the @xmath18.2 bleu for e-test and +5.7 bleu for c-test. as the c-test challenge set is constructed to contain translational ambiguity, the improvement is an indication that the image features are useful for disambiguation. the human evaluation scores from the wat organizers for the available sub-tasks are updated in we demonstrate examples of translations obtained for the challenge test set in figure 2 . the extracted image features in the form of object tags are also shown for each image in the figure. we ob-serve that the multimodal translations are notably better than the text-only translations. this is consistent with the pattern of the bleu scores in ## conclusions in this system description paper, we presented our system for three tasks in wat2022: (a) @xmath19, (b) @xmath20, and (c) @xmath21. we released the code through github for research foot_5 . in the next step, we will explore the usage of synthetic features in multimodal translation for the malayalam and bengali languages, to make use of available text-only corpora for these language pairs.