Multimodal Neural Machine Translation System for E nglish to B engali (2021)

Abstract: Multimodal Machine Translation (MMT) systems utilize additional information from other modalities beyond text to improve the quality of machine translation (MT). The additional modality is typically in the form of images. Despite proven advantages, it is indeed difficult to develop an MMT system for various languages primarily due to the lack of a suitable multimodal dataset. In this work, we develop an MMT for English-> Bengali using a recently published Bengali Visual Genome (BVG) dataset that contains images with associated bilingual textual descriptions. Through a comparative study of the developed MMT system vis-a-vis a Text-to-text translation, we demonstrate that the use of multimodal data not only improves the translation performance improvement in BLEU score of +1.3 on the development set, +3.9 on the evaluation test, and +0.9 on the challenge test set but also helps to resolve ambiguities in the pure text description. As per best of our knowledge, our English-Bengali MMT system is the first attempt in this direction, and thus, can act as a baseline for the subsequent research in MMT for low resource languages.

## introduction over the last decade, deep neural networks (dnn) achieved state-of-the-art results for many tasks including computer vision, natural language processing, and speech processingwhich encouraged researchers to design a system that will get benefit from the fusion of multiple modalities @xcite . english text: a girl playing tennis. bengali text: একিট েমেয় েটিনস েখলেছ figure 1 : a sample from the bvg dataset: an image with a specific region marked and its description in english and bengali. multimodal translation refers to the extraction of information from more than one modality where it is assumed that alternative views would be used for input data @xcite @xcite . the tasks and applications in multimodal translation involve translation of image captions, translation of video content, translation of spoken language, and others. these applications exploit more than one modality such as translation from video content includes audio and visual modality, and translation of image captions includes visual modality and caption text. although there are different opinions on the performance of machine translation using visual modality but under limited resources, visual input generates better translation @xcite . it is observed that multimodal translation systems do not leverage the visual signal to produce the correct translation in case of mistakes in the source language sentence (chowdhury and elliott, 2019). when images are considered as an additional modality, the recent research can be divided into two major approaches based on their utilization of image features for the mmt: processing the global image features @xcite , and processing the object tags derived from the images @xcite . cross-lingual visual pre-training which learns multimodal cross-lingual representations also found effective in mmt @xcite . bengali (also known as bangla) is an indo-aryan language widely spoken in india and bangladesh and considered as the 6-th most spoken language of the world with approximately 230 million speakers. although english-to-bengali text-only parallel corpora @xcite are available for building mt systems @xcite @xcite ; the multimodal dataset for bengali did not exist. thus, english-bengali mmt systems have not been developed until now. recently, the first english-bengali multimodal dataset: bengali visual genome (bvg) has been published @xcite ) -which has facilitated research and development of corresponding multimodal as well as image captioning tasks. the primary objective of this paper is to develop an mmt system for bengali where the multimodal input is provided as an image and its description in english. we have used the bvg dataset to demonstrate our mmt system. the bvg consists of image descriptions (or captions) in the bilingual corpus for a specific rectangular region in the image as shown in figure 1 . the bounded box region information (x, y, width, height) for each of the images is provided in the dataset. the mmt system uses both text and the associated image to build the model to translate into the target bengali text. we extracted the object tags as image features. then the object tags are appended to the original english sentence which is then translated using mbart @xcite , a multilingual sequence to sequence model trained on millions of unsupervised multilingual sentences. we also perform a comparative study between the english-bengali text-to-text translation system and the built mmt system. ## related work there is limited research conducted in the domain of multimodal machine translation for indian languages. other than hindi, no mmt system is available in other indian languages due to the unavailability of the multimodal dataset for translation. the flickr30k dataset with hindi description is used for multimodal nmt task by dutta @xcite . they attempted to conduct multimodal translation from hindi to english and examined whether visual image features can improve translation performance. they used synthetic hindi descriptions for the flickr30k dataset and provided validation and test corpus of english translations of the flickr30k dataset. similarly, @xcite considered the flickr30k dataset and asked five different crowd workers to provide hindi translation of an image from the dataset and generated english captions with evaluating the quality of the translation. laskar et al. (2020) used hindi visual genome 1.1 dataset @xcite and used opennmt-py to build text-only nmt and multimodal nmt. they had used pretrained cnn with vgg19 for extracting local and global features from the images for the multimodal translation. the multimodal nmt performs better as compared to textonly nmt. ## description of the mmt system in this section, we describe the multimodal translation system developed for @xmath0. our model is adapted from vita @xcite foot_0 which uses mbart @xcite , a multilingual sequence-to-sequence denoising auto-encoder that has been pre-trained using the bart objective @xcite . @xcite built a @xmath1.1). similar to the vita approach, we first derive the list of object tags for a given image using the pretrained faster r-cnn with resnet-101-c4 backbone. based on their confidence scores, we pick the top 10 object tags. in cases where less than 10 object tags are detected, we consider all the tags. the object tags are then concatenated to the english sentence which needs to be translated to bengali. the concatenation is done using the special token '##' as the separator. the separator is followed by comma-separated object tags. adding objects enables the model to utilize visual concepts which may not be readily available in the original sentence. the english sentences along with the object tags are fed to the encoder of the mbart model. the mbart's decoder generates the bengali translations autoregressively. the block diagram of the multimodal translation using object tags is shown in figure 2 . @xcite applied vita for english to hindi translation by using the mbart-25 model which has been pre-trained using the bart objective @xcite . for this pre-training, only multilingual unsupervised data spanning 25 languages was used @xcite . then they finetune the model for the machine translation task using 1.6 million english-hindi parallel sentences. finally, they finetune the model on the english-hindi multimodal data with the addition of object tags by (a) first masking out 15% of the english tokens in the input and then (b) with no masking. for translating from english to bengali, however, we do not perform a large-scale machine translation pre-training using a million training examples. we instead use the pre-trained mbart-50 foot_1 model finetuned on the machine translation task in a one-to-many setup using multilingual data which contains merely 4487 english-bengali parallel sentences @xcite . we take this pre-trained model and train it further on the machine translation task using the bengali visual genome multimodal data by adding object tags to the english source sentences. because of the scarcity of the pretraining machine translation english-bengali parallel data (nearly 320 times smaller) as compared english-hindi parallel data, our system represents a low-resource scenario. although the main approach we use is similar to the vita method by @xcite , we state the one modification in our implementation and the reason behind it. the original vita method stochastically masks out 15% of the tokens in the input english sentence. this is done to incentivize the model to utilize the object tags while generating the bengali translation and not rely only on the english sentence. however, in our experiments, we do not mask out 15% of the english tokens. this is done because we already see gains above the text-only results without masking. using masking can potentially improve the multimodal translation scores further. ## experiment details in this section, we first describe the dataset used followed by details of model training configurations. ## result and discussion we have used the popular machine translation metric bleu @xcite for the automatic evaluation, computed using sacre-bleu toolkit @xcite . the development bleu scores during training are shown in figure 3 . the development bleu score increases as the training progress. the mbart based scores (both text-to-text and multimodal) reach a notably high bleu score even after one epoch of training. this is because of the prior knowledge acquired from pre-training, which is missing in the case of the text-to-text transformer. the mmt results on the d-test, e-test, and c-test are shown in text mbart performs better as compared to the text-to-text transformer model. the multimodal mbart performs the best overall. the object tags added to the original english sentences provide more context about the image and enable the generation of a better translation, which is indicated by the higher overall bleu scores. the improvement is seen in translating c-test as well. we performed the comparison of mmt system with the best text-to-text translation system without using any image features. there is a performance improvement (bleu score) of +1.3 on d-test, +3.9 on e-test, and +0.9 on c-test. apart from performance, mmt systems help to resolve ambiguities as shown in the the mmt system can correctly translate the ambiguous word court which the text-totext mt system fails. we compared the translation output between both text-to-text and mmt systems and observed the mmt system produces a better translation, correct word order, no ambiguity as compared with the textto-text mt system. ## manual evaluation to validate the automatic scoring, we manually annotated 100 randomly selected sen-tences from the c-test set as translated by the text-to-text machine translation system and mmt system. the annotation was performed by the native bengali speaker. bengali captions in the mmt translation outcomes fall under five different sets where some of them are translated perfectly without any issue, some of them are very close to perfect, some of them have parts of speech or grammar issues, some of them have ambiguity in meanings, and some of them have lack of words than original annotation. in this annotation, each annotated segment gets exactly one label from the following set @xcite : flawless for translations without any error (typesetting issues with diacritic marks due to different tokenization are ignored), good for translations which are generally ok and complete but need a small correction, partly correct for cases where a part of the segment is correct but some words are mistranslated, ambiguity for segments where the mt system "misunderstood" a word's meaning, and incomplete for segments that run well but stop too early, missing some content words. this category also includes the relatively rare cases where the text-to-text or mmt system produced just a single word, unrelated to the source. the manual evaluation results are summarized in following graph figure 4 . the mmt system generates more flawless and good translation output as compared to the text-to-text system (see figure 4 ). the text-to-text system obtained more partial correct and incomplete translation output. it observed that still ## conclusion in this paper, we build an english-bengali mmt system utilizing bi-lingual text and the associated images which improves the translation quality (based on an automatic evaluation) and resolve ambiguities. our work helps to build a better english-bengali mt system and encourages researchers to explore the mmt system for bengali. the future work includes exploring other state-of-the-art mmt systems on the bvg dataset and performs a comparison analysis @xcite @xcite . flawless/good: window on second level ি তীয় েমেঝেত উইে ািট gloss: window on second level blue tennis court area নীল েটিনস েকাটর্ অঞ্চল gloss: blue tennis court area three baseball players on baseball field. িতনিট েবসবল েখেলায়াড় মােঠ রেয়েছ gloss: three baseball players on baseball field. partly correct: railing on second floor of building ি তীয় তলায় েরিলংেয় েজিলং gloss: gelling on second floor railing (output does not convey the intended meaning in the target language) the hand is going to press a button হাত জন একিট েবাতােম যাে gloss: the hand is going to a button ('press' is completely omitted from the translated outcome) ambiguity: brown tennis court floor আদালেত ধূ সর রং gloss: brown color on court floor (confusing tennis court with judicial) street sign on a pole in english and chinese ইংিলশ এবং চীনা ভাষায় একিট েমরুেত রা ার িচহ্ন gloss: street sign on a pole in english and chinese (confusing street-side pole with 'south pole and north pole' in bengali) translation. arxiv preprint arxiv:1609.03976. ozan caglayan, menekse kuyu, mustafa sercan amac, pranava swaroop madhyastha, erkut erdem, aykut erdem, and lucia specia. 2021. cross-lingual visual pre-training for multimodal machine translation. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 1317-1324. ozan caglayan, pranava swaroop madhyastha, lucia specia, and loïc barrault. 2019. probing the need for visual context in multimodal