Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation (2023)

Abstract: One of the major challenges of machine translation (MT) is ambiguity, which can in some cases be resolved by accompanying context such as images. However, recent work in multimodal MT (MMT) has shown that obtaining improvements from images is challenging, limited not only by the difficulty of building effective cross-modal representations, but also by the lack of specific evaluation and training data. We present a new MMT approach based on a strong text-only MT model, which uses neural adapters, a novel guided self-attention mechanism and which is jointly trained on both visually-conditioned masking and MMT. We also introduce CoMMuTE, a Contrastive Multilingual Multimodal Translation Evaluation set of ambiguous sentences and their possible translations, accompanied by disambiguating images corresponding to each translation. Our approach obtains competitive results compared to strong text-only models on standard English→French, English→German and English→Czech benchmarks and outperforms baselines and state-of-the-art MMT systems by a large margin on our contrastive test set. Our code and CoMMuTE are freely available.

## introduction multimodal machine translation (mmt) typically refers to the use of additional non-textual data in text-based machine translation (mt). here, we focus on the case where source texts are accompanied by images, the idea being to exploit visual data to improve the translation of ambiguous sentences. for example, in figure 1 , the english word glasses can either be translated as french verres 'drinking vessels' or lunettes 'spectacles', an ambiguity which is resolved using the image. a main research direction of mmt has been how to best exploit image representations and combine the image and text modalities @xcite @xcite . it has typically been difficult to surpass strong text-only baselines, the image modality often being ignored @xcite . a major issue holding back progress is that most current stateof-the-art mmt models @xcite @xcite are trained solely on the ∼30k examples of the multi30k dataset @xcite , comprising image captions and their translations. this causes two issues: (i) the models do not exploit the large amount of text-only data available and therefore perform poorly in comparison to state-of-the-art text-only mt systems, and (ii) we show that very few examples require images to be correctly translated, which means that the datasets are ill-adapted to evaluating the use of the image modality. in this article, we aim to overcome these problems by proposing (i) a new mmt approach that is able to exploit (text-only) monolingual and parallel data as well as (multimodal) captioning data, and that reaches a good balance between maintaining high mt quality and effectively exploiting images, and (ii) a test set, commute, containing contrastive evaluation pairs, where images provide the necessary context to disambiguate between multiple meanings of the same source sentence. our suggested model is inspired by work on adapting frozen language models (lms) to multimodal inputs @xcite @xcite ; we propose to adapt a strong mt model to multimodal inputs with lightweight modules @xcite to exploit the large amount of textual data it was trained on. we also propose to better exploit the image by introducing guided self-attention and by combining the standard mmt objective with a visually-conditioned masked language modelling (vmlm) objective @xcite @xcite . our model obtains competitive results compared to strong text-only baselines on standard @xmath0,de,cs} mmt benchmarks @xcite @xcite and outperforms them and state-of-the-art mmt models on our lexically ambiguous contrastive test set. 3 2 related work multimodal mt data. the reference dataset to train and evaluate mmt models is multi30k @xcite . however, recent work has shown that most mmt systems trained and evaluated on it do not effectively exploit the image information; @xcite showed that replacing the ground truth image with a random one does not lead to the drop in performance that would be expected, while @xcite argued that the observed gain in performance was due to a regularisation effect. it is also notoriously difficult to beat text-only baselines on this benchmark @xcite . this may be due to (i) some subsets of multi30k having been translated independently from the images @xcite and (ii) most of the time, the source text being sufficient in theory to produce a perfect translation (i.e. the image is not necessary; see section 5.2 for our own analysis). based on this, alternative test sets and evaluation methods have been proposed. @xcite proposed to probe the use of images in mmt models, while @xcite proposed another training corpus and evaluation benchmark to evaluate mmt systems, but their work is only based on gender ambiguity and requires specific training data to train mmt models. lala and specia (2018) released a lexically ambiguous mmt evaluation dataset to evaluate models ability to disambiguate source sentences, but we found that text context is generally sufficient to translate the evaluation dataset correctly. ## our approach: vgamt the two main aims of our approach are to (i) exploit a maximum available data (not just multimodal parallel text data) and to (ii) provide an effective way to combine image and text modalities. our approach, shown in figure 2 , consists in taking a strong text-only mt model foot_2 and adapting it to multimodal mt. to adapt this strong text-only model to multimodal inputs, we add several lightweight modules-bottleneck adapters @xcite and linear visual projection layers-to the otherwise frozen initial model. the bottleneck adapters are lightweight linear layers introduced after each attention block and each feedforward layer to project embeddings down before projecting them up. in terms of representing visual information, we choose to use two types of representation. we concatenate local (mdetr) features and global (clip) features to the text inputs. we choose to use global features too, since the source sentence can describe more general aspects of the image than mere objects (such as scenes). we jointly train the non-frozen parts of our model on two distinct objectives: multimodal mt (mmt) and visuallyconditioned masked language modelling (vmlm), as described in section 3.1. we also introduce a guided self-attention to exploit image information in a straightforward manner (see section 3.2) in the encoder (while the decoder uses regular self-and cross-attentions and can only attend to embeddings related to text positions). we call our approach visually guided and adapted machine translation (vgamt). ## contrastive multilingual multimodal translation evaluation (commute) to overcome the flaws of existing benchmarks (see section 5.2), we introduce commute, a contrastive multilingual multimodal translation evaluation dataset foot_5 . it is composed of 155 lexically ambiguous sentences in english, each associated with two translations corresponding to two of the possible meanings of each sentence and two images that determine which of the translations is correct. it covers @xmath1, @xmath2. an example is given in figure 4 . 2018 ), and we created the remaining ones. 7 we collected two images for each sentence under creative commons license (either google images or our own photos), so that the image illustrates without ambiguity one of the two meanings of the sentence. we do not restrict the image-text relation to be strictly descriptive (as for image captions) in order to have a more general evaluation dataset. each sentence was translated into two possible translations (each corresponding to one of the images) by a native speaker of the target language. appendix a provides some basic statistics. the idea of commute is to use mmt models to rank each of the two translations based on image information. the perplexity of a sentence for a given model is defined as: p p l q (y) = n @xmath3, where q is the probability distribution output by the model, n is the sequence length and y 1 , . . . , y n is the sequence of tokens. now, let y 1 , . . . , y n 1 be the sequence of tokens of the correct translation and y ′ 1 , . . . , y ′ n 2 the sequence of tokens of the incorrect translation, a model makes a correct prediction if p p l q (y) ≤ p p l q (y ′ ). i.e. the model considers the correct translation more likely than the incorrect one. for each example, we rank each of the translations based on each of the images (2 comparisons per example), and report the accuracy over all the examples. as commute is perfectly balanced, a text-only model will get exactly 50% accuracy on this task. 5 experiments ## 2 il va falloir se débarrasser de cette taupe. figure 4 : example from commute. the english word 'mole' can refer either to 'a small dark, raised lump on a the skin' (1) or 'a small burrowing mammal' (2). subtitles2018 foot_8 @xcite , wikipedia @xcite , ted talks (reimers and gurevych, 2020) and the books datasets (tiedemann, 2012). we preprocess the data using moses scripts @xcite ). foot_9 ## ablation study to better understand the role of vgamt's components, we carry out several ablations for @xmath4. scores on all test sets except test2017. notably, the unfrozen model's accuracy score of 60.5 on com-mute is 6.6 points lower than our final vgamt model. as well as providing a more lightweight solution that does not involve fine-tuning all parameters, using neural adapters and freezing other weights is useful in terms of performance. ## conclusion we propose a new mmt approach (vgamt) based on (i) adapting a strong text-only mt model with lightweight adapters and (ii) introducing better use of the text and image modalities through a novel guided self-attention mechanism and joint mmt and vmlm training. we also introduce