Multiple Captions Embellished Multilingual Multi-Modal Neural Machine Translation (2021)

Abstract: Neural machine translation based on bilingual text with limited training data suffers from lexical diversity, which lowers the rare word translation accuracy and reduces the generalizability of the translation system. In this work, we utilise the multiple captions from the Multi-30K dataset to increase the lexical diversity aided with the cross-lingual transfer of information among the languages in a multilingual setup. In this multilingual and multimodal setting, the inclusion of the visual features boosts the translation quality by a significant margin. Empirical study affirms that our proposed multimodal approach achieves substantial gain in terms of the automatic score and shows robustness in handling the rare word translation in the pretext of English to/from Hindi and Telugu translation tasks.

## introduction the machine translation (mt) systems by @xcite has been the defacto standard which are based on parallel dataset. but, in recent times, use of monolingual data @xcite or incorporating multiple lan guages in a jointly trained single multilingual model @xcite has improved the translation quality of low re source languages. compared to training separate bilingual models with the same parameters, the ability to handle translation between multiple lan guage pairs provides an inherent advantage of hav ing relatively compact model parameters. typi cally, in such models, the encoder and decoders are shared among all the languages and attention. the sharing of the encoder is crucial to learn the ini tial multilingual crosslingual information @xcite however, a single shared de coder is often insufficient in handling the transla tion of multiple languages. this decoder degener acy is addressed by partial sharing of decoder and attention parameters @xcite or through a languageagnostic universal models @xcite . image features along with the text data has been used in sequence generation tasks such as the im age caption generation @xcite and multimodal machine translation (mmt) which in corporates visual features into ordinary nmt sys tems for low resource languages. with the intro duction of mmt datasets such as multi-30k @xcite and hindi visual genome @xcite , mt researchers @xcite have highlighted improvement in translation qual ity by incorporating image features in the mt sys tems. in this work, we adopt a single shared multilin gual machine translation system between english and under resourced languages viz., hindi and tel ugu, aided by linguistic information in the form of multiple captions. the inclusion of multiple cap tions during training makes the system implicitly robust to lexical and syntactic diversity. in addi tion to the multiple captions, we infuse our multi caption multilingual model with the visual infor mation in a multimodal @xcite el liott and kádár, 2017; yao and wan, 2020) setting. english and hindi belong to the indoeuropean language family, while telugu is a dravidian lan guage. all three languages use different scripts; roman for english, devanagari for hindi and tel ugu is written in telugu script, an abugida writing system from the brahmic family of scripts. and found that their method improves over the sin gle parallel corpora pbsmt baseline in terms of the overall word coverage and the translation qual ity. paraphrase has also been leveraged as a data augmentation technique to improve dialog gener ation @xcite and question generation @xcite tasks. more similar to our pro posed work @xcite decomposed the paraphrase as a foreign language in a multilingual scenario. similar to the findings of @xcite , @xcite found that their method improves the word coverage with diverse lexical choices. however, these works are purely unimodal, and on the other hand, a visually informed multimodal system involves the extraction of the global se mantic features from the image and initialize ei ther the encoder or decoder to fuse the visual con text along with the textual input @xcite . in cases where the textual context is re stricted, @xcite showed that vi sual features could help to generate better transla tions. similar to the proposed work, @xcite trained a multimodal machine transla tion system in the pretext of tamil, kannada and malayalam by generating a synthetic dataset from flickr30k @xcite . they showed that transliteration of the dravidian languages into latin script and the multilingual setup improves the multimodal system over the bilingual multi modal baseline. ## methodology the proposed multicaption enabled multimodal multilingual machine translation system employs two major steps: first, we create the training cor pus and then train the multicaption multilingual system fused with the visual features. ## results and analysis this section presents the quantitative results ob tained by all the models and their analysis. ## conclusion this work presents the multimodal machine trans lation embellished with multiple captions in a mul tilingual setup. we find that the multiple captions benefit both the textonly and the multimodal mod els by introducing lexical diversity, making the sys tem more robust to handle the rare word translation and thus increasing the translation quality. addi tionally, the visual features which provide the con text information further alleviate the translation quality. however, this work is experimented on synthetic data, hence the trained systems possesses the accumulated errors present in the training data. in our future work, we intend to address this issue with some counter measures.