BITS - P at WAT 2023: Improving I ndic Language Multimodal Translation by Image Augmentation using Diffusion Models (2023)

Abstract: This paper describes the proposed system for mutlimodal machine translation. We have participated in multimodal translation tasks for English into three Indic languages: Hindi, Bengali, and Malayalam. We leverage the inherent richness of multimodal data to bridge the gap of ambiguity in translation. We fine-tuned the ‘No Language Left Behind’ (NLLB) machine translation model for multimodal translation, further enhancing the model accuracy by image data augmentation using latent diffusion. Our submission achieves the best BLEU score for English-Hindi, English-Bengali, and English-Malayalam language pairs for both Evaluation and Challenge test sets.

## introduction machine translation (mt) is the nlp task of translation between language pairs. multimodal machine translation (mmt) is the translation process that utilizes information from multiple modalities, not just text. the most popular approach is to use extra visual context in addition to the source text input. the visual context is presented in the form of images that are relevant to the text to be translated and may help in cases of ambiguity. in 10th workshop on asian translation (wat 2023), we investigate multimodal machine translation for english to hindi, bengali, and malayalam languages by fine-tuning the 'no language left behind' (nllb) pre-trained machine translation model by @xcite and using image data augmentation techniques. figure 1 shows an example where the visual context of an image helps generate accurate machine-translated text. the rest of the paper is organized as follows: section 2 presents the review of related works. the data and system are briefly described in section 3. section 4 reports the results, followed by the future scope and conclusion in section 5. ## related work in the literature survey, there are some multimodal machine translation works that take both text and image as input, and learn joint multimodal representations from images and text @xcite . @xcite incorporated an object detection system, extracting local and global image features as additional inputs to the encoder and decoder. @xcite utilized dynamic context-guided capsule network (dccn) for iterative extraction of related visual features. most of them investigated multimodal mt for high-resource european languages. there are only a few works for indic languages. dutta @xcite who employed synthetic data for training and used multimodal, attention-based mt incorporating visual features into the encoder and decoder @xcite . @xcite further demonstrated the advantage of jointly learning textimage interaction rather than modeling them separately using attentional networks. @xcite proposed a subset of the visual genome dataset @xcite ) for multimodal translation between english and hindi, a less explored language pair in this context. @xcite used the bengali visual genome @xcite ) and adopted the vita @xcite approach, with mbart @xcite for encoding english sentences with object tags and decoding bengali translations. our current work builds upon these foundations, introducing a novel approach using nllb and stable diffusion @xcite to multimodal translation, with a specific focus on the english to hindi, bengali, and malayalam language pairs. ## system overview in this section, we describe the dataset, data augmentation technique, and model approach for the proposed system we use for the multimodal translation task. ## experimental results our model achieves promising results during the automated evaluation foot_3 . to evaluate our model, we used two test sets: i) evaluation set (e-test) with 1595 sentences, and ii) challenge set (c-test) with 1400 sentences. we test our model on both of these test sets and present the results in the english-hindi multimodal machine translation model trained with additional synthetic image data achieves 52.10 and 45 bleu score on c-test and e-test, while the baseline model trained only on hindi visual genome data achieves 51.20 and 44.90 bleu score on c-test and e-test respectively. the data augmentation technique improved the bleu score by +0.9 on challenge set (c-test). ## conclusion in this paper, we described our multimodal machine translation system. our system scored 50.60 and 48.70 bleu points for bengali; 45.00 and 52.10 bleu points for hindi, and 51.90 and 42.20 bleu points for malayalam for evaluation and challenge test sets, respectively. the data augmentation strategy of using synthetic images generated by diffusion models improved the system by +0.9 bleu score. in the future, we would further explore data augmentation approaches for text data using image captioning frameworks.