Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation (2023)

Abstract: Multimodal machine translation (MMT) aims to improve translation quality by incorporating information from other modalities, such as vision. Previous MMT systems focus on better access and use of visual information and tend to validate their methods on image-related datasets. However, these studies face two challenges. First, they can only utilize a limited amount of data that is composed of bilingual texts and images (referred to as “triple data”), which is scarce. Second, current benchmarks for MMT are restricted and do not correspond to realistic scenarios. Therefore, this paper correspondingly establishes new methods and a new dataset for MMT. We propose a novel framework for MMT that addresses these challenges by utilizing large-scale non-triple data, such as monolingual image-text and parallel text-only data. Additionally, we construct a new e-commercial multimodal translation dataset, named EMMT, of which the test set is specifically designed to include ambiguous words that require visual context for accurate translation. Experiments show that our method is well-suited for real-world scenarios and can significantly improve translation performance with more non-triple data. In addition, our model also rivals or surpasses various SOTA models in conventional multimodal translation benchmarks.

## introduction multimodal machine translation (mmt) is a machine translation task that utilizes data from other modalities, such as images. previous studies propose various methods to improve translation quality by incorporating visual information and showing promising results @xcite @xcite . however, manual image annotation is relatively expensive; at this stage, most mmt work is applied on a small and specific dataset, @xcite . the current performance of the mmt system still lags behind the large-scale text-only neural machine translation (nmt) system, which hinders the real-world applicability of mmt. we summarize the limitations of the current mmt in two aspects. the first limitation is the size of the training data. usually, the performance of mmt heavily relies on the triple training data: parallel text data with corresponding images. the triplets are much rarer for collection and much more costly for annotation than monolingual imagetext and parallel text data, as in figure 1 . considering that current mt systems are driven by a massive amount of data @xcite , the sparsity of multimodal data hinders the large-scale application of these systems. some researchers have proposed retrieve-based approaches @xcite , aiming to construct pseudo-multimodal data through text retrieval. however, their constructed pseudo-data face problems like visual-textual mismatches and sparse retrieval. besides, the models still cannot take advantage of monolingual image-text pairs. the second limitation is the shortage of proper benchmarks. although several researchers have examined the benefit of visual context upon the translation when textural information is degradated @xcite , the improvements remain questionable. @xcite and @xcite argue that vision contributes minor in previous mmt systems, and the images in the previous benchmark dataset provide limited additional information. in many cases, the translation of sentences relies on textual other than image information. the texts contain com-plete contexts and are unambiguous, leaving the usage of images doubtful. therefore, a benchmark in that the sentences can not be easily translated without visual information is much needed. to address these limitations, we propose models to make the most of training data and build a challenge and real-world benchmark to push the realworld application of mmt research. at first, we propose a new framework, named 2 /3-triplet, which can use both parallel text and image-text data. it provides two different ways of exploiting these data based on the continuous vision feature and discrete prompt tokens, respectively. the two approaches are not mutually exclusive and can be used jointly to improve performance within the same framework. it is also worth mentioning that the prompt approach is easy to deploy without modifying the model architecture. in addition, we present a new real-world dataset named emmt. we collect parallel text-image data from several publicly available e-commerce websites and label the translation by 20 language experts. to build a challenge test set, we carefully select ambiguous sentences that can not be easily translated without images. this high-quality dataset contains 22k triplets for training and 1000 test examples, along with extra image-text and parallel text data. comprehensive experiments show that 2 /3-triplet rivals or surpasses text-only and other mmt competitors on emmt, as well as previous benchmarks. especially, 2 /3-triplet consistently improves the strong text-only baseline by more than 3 bleu scores in various settings, showing the importance of visual information. ## related work researchers applied multimodal information to enhance machine translation systems since the statistical machine translation era @xcite . with the rise of neural networks in machine translation, researchers have focused on utilizing image information more effectively. early work used image features as initialization for neural mt systems @xcite . more recent studies proposed multimodal attention mechanisms @xcite , enhanced text-image representations using graph neural networks @xcite , latent variable models or capsule networks @xcite , and used object-level visual grounding information to align text and image @xcite . @xcite found that a stronger vision model is more important than a complex architecture for multimodal translation. as we discussed earlier, these methods are limited to bilingual captions with image data, which is scarce. therefore, some researchers @xcite also design retrievalbased mmt methods that retrieve images with similar topics for image-free sentences. alternatively, elliott and kádár (2017) proposed visual "imagination" by sharing visual and textual encoders. recently, @xcite and @xcite have questioned whether the most common benchmark multi30k @xcite is suited for multimodal translation since they found images contribute little to translation. @xcite have contributed a new dataset of the e-commercial product domain. however, we find their datasets still have similar drawbacks. several relevant studies about translation and multimodality are noteworthy. @xcite used visual content as a pivot to improve unsupervised mt. @xcite proposed a pretraining model by using modality embedding as prefix for weak supervision tasks. @xcite introduced the valhalla, which translates under guidance of hallucinated visual representation. ## approach for the fully supervised condition in mmt, we have triplet {(x, y, i)}, where x is the source text, y is the target text, and i is the associated image. since the triplet is rare, we attempt to utilize partially parallel data like {(y, i)} and {(x, y)}, which are referred as monolingual image-text data and parallel text data in this paper. in this section, we propose a new training framework 2 /3-triplet with two approaches to utilize triple and non-triple data at the same time. we name these two approaches as fusion-based and prompt-based, as shown in figure 2 . for each approach, the model can conduct a mix training with three kinds of data: ((x, i) → y), ((x) → y), and ((y * , i) → y), where y * indicates the masked target text. fusion-based approach resembles the conventional models where the encoded vision information is taken as model input and the model is trained in end2end manners, and our design makes it possible to utilize bilingual corpus and image-text pairs image encoder black bass t-shirt ⿊⾊ 鲈⻥ [mask] ② parallel text ⿊⾊ 鲈⻥ t恤 ③ monolingual caption ① triplet + text encoder text input image feature random text mask ⿊⾊ 鲈⻥ t恤 text decoder textual modality visual modality fusion gate black bass t-shirt black bass t-shirt image encoder 吉他 image caption image embedding (a) fusion-based black bass t-shirt [sep] 吉他 t恤 caption model ① triplet ② parallel text black bass t-shirt [sep] ⻉斯 random text mask sample from target ⿊⾊ ⻉斯 t恤 text encoder-decoder text input translations: ⿊⾊ black 吉他 guitar t恤 t-shirt ⻉斯 bass (guitar) 鲈⻥ bass (fish) black bass t-shirt ⿊⾊ [mask] t恤 [sep] 吉他 t恤 ⿊⾊ ⻉斯 t恤 black bass t-shirt ⿊⾊ ⻉斯 t恤 ③ monolingual caption other than multilingual triplets. prompt-based approach is inspired by the recent nlp research based on prompts @xcite @xcite , where we directly use the image caption as a prompt to enhance the translation model without any modification to the model. ## dataset as mentioned before, in previous test sets, many sentences can be easily translated without the image context, for all information is conveyed in the text and has no ambiguity. to deeply evaluate visual information usage, we propose a multimodalspecific dataset. we collect the data based on real-world ecommercial data crawled from tiktok shop and shoppee. we crawled the product image and title on two websites, where the title may be in english or chinese. we filter out redundant, duplicate sam-ples and those with serious syntax errors. based on this, we conduct manual annotations. we hired a team of 20 professional translators. all translators are native chinese, majoring in english. in addition, another translator independently samples the annotated corpus for quality control. we let the annotators select some samples specifically for the test set, which they found difficult to translate or had some confusion without images. the total number of triples annotated is 22, 500 of which are carefully selected samples as testset. we also randomly selected 500 samples as devsets among the full-set while the remaining as training set. besides the annotated triplets, we clean the rest of the crawled data and open sourced it as the monolingual caption part of the data. since our approach features in utilizing bilingual data to enhance multimodal translation, we sample 750k ccalign (el-kishky et al., 2020) english-chinese as a bilingual parallel text. the selection is motivated by the corpus's properties of its diversity in sources and domains, and it is more relevance to real-world compared to other corpus. the sampled data scale is decided based on both the model architecture and the principles of the neural scaling law @xcite . we also encourage future researchers to explore the use of additional non-triple data to further enhance performance, as detailed in the appendix. we summarize the dataset statistics in ## conclusion this paper devises a new framework 2 /3-triplet for multimodal machine translation and introduces two approaches to utilize image information. the new methods are effective and highly interpretable. considering the fact that current multimodal benchmarks are limited and biased, we introduce a new dataset emmt of the e-commercial domain. to better validate the multimodal translation systems, the testset is carefully selected as the image are crucial for translation accuracy. experimental results and comprehensive analysis show that 2 /3-triplet makes a strong baseline and emmt can be a promising benchmark for further research.