Noise-robust Cross-modal Interactive Learning with T ext2 I mage Mask for Multi-modal Neural Machine Translation (2022)

Abstract: Multi-modal neural machine translation (MNMT) aims to improve textual level machine translation performance in the presence of text-related images. Most of the previous works on MNMT focus on multi-modal fusion methods with full visual features. However, text and its corresponding image may not match exactly, visual noise is generally inevitable. The irrelevant image regions may mislead or distract the textual attention and cause model performance degradation. This paper proposes a noise-robust multi-modal interactive fusion approach with cross-modal relation-aware mask mechanism for MNMT. A text-image relation-aware attention module is constructed through the cross-modal interaction mask mechanism, and visual features are extracted based on the text-image interaction mask knowledge. Then a noise-robust multi-modal adaptive fusion approach is presented by fusion the relevant visual and textual features for machine translation. We validate our method on the Multi30K dataset. The experimental results show the superiority of our proposed model, and achieve the state-of-the-art scores in all En-De, En-Fr and En-Cs translation tasks.

## introduction multi-modal neural machine translation (mnmt) aims to optimize the conventional text-only machine translation systems by using multi-modal information (eg., image, video, sound), which has received growing research attentions in the fields of cv and nlp, recently. a reasonable assumption is that visual information is helpful to improve textual-level machine translation @xcite @xcite , and many studies have been carried out to conduct the benefits of image for nmt @xcite @xcite . as expected, the fusion of visual information actually improves the performance of machine translation @xcite . most existing mnmt methods mainly focus on how to design a excellent multi-modal fusion framework to bridge the semantic gap between image and text, while visual noise is often ignored. unfortunately, it is often difficult to obtain the image that exactly match the textual information. what is worse, image information and textual information may even be weakly correlated with each other. visual noise is generally unavoidable @xcite . as shown in figure 1 (left), objects such as old man, brown hat and bench included in the image, these visual objects correspond to the 'old man', 'brown hat' and 'bench' in the source sentence, which is the useful visual information for machine translation. however, the image also contains some irrelevant visual information ( e.g., tree, flower, grasse) for the source sentence, the mismatched visual-textual information may distract the multi-modal fusion and then lead to machine translation performance decay. therefore, it is necessary to consider noiserobust text-image fusion problem for mnmt. how to effectively and efficiently extract useful visual information is one of the core issues of mnmt, there are three main multi-modal fusion methods: 1) multi-modal attention mechanism, such as cross-modal interactive attention mechanism @xcite @xcite and adaptive feature selection mechanism @xcite @xcite between visual features and textual features. 2) multi-modal transformer fusion methods, which utilizes transformer to encode textual features and visual features separately @xcite , and then a multi-head cross-modal attention mechanism @xcite @xcite is figure 1 : an example of an @xmath0. this paper endeavors to address visual noiserobust multi-modal fusion problem for mnmt, we attempt to explore robust multi-modal interactive fusion strategy with cross-modal relation-aware mask mechanism for mnmt in transformer framework. concretely, a text-image relation-aware attention module is constructed in the visual transformer encoder by cross-modal interactive mask mechanism, and the visual features are extracted based on text-to-image interactive mask knowledge. then a noise-robust multi-modal fusion approach is adopted to integrate visual features into seq2seq framework more efficiently and effectively. comparing with previous works, the major contributions of our paper are three-fold. zhao et al. (2021) utilized object detection features with an additional regiondependent attention mechanism to fusion visual regional features and textual features; nishihara et al. (2020) presented a supervised cross-modal attention module to align textual features and visual features; song et al. (2021) employed a coattention graph updating module at each transformer encoder layer to align multi-modal features. yao and wan (2020) used multi-modal tranformer to align both visual features and textual features; yin et al. (2020) proposed a graph-based mnmt approach to extract multi-model features through text-image gating attention mechanism; lin et al. (2020) adopted a gating mechanism to fuse visual features extracted by a dynamic context-guided capsule network; all the above methods focus on multimodal feature fusion methods, and they assume that visual information is closely related to textual information, which heavily restricts their robustness. however, text and its corresponding image may not match exactly, visual noise is generally inevitable. in this work, we systematically investigate whether masking visual noise helps machine translation. mask strategy mask strategy is one of the most effective ways of representation learning, which has been widely used in vision and textual pretraining models. we summarize the existing mask strategies in the three aspects as follows: 1) vision mask-based pre-trained models @xcite @xcite , the main purpose is to mask image patch-level for better visual robust-representation learning. 2) textual mask-based pre-trained models @xcite @xcite , the tokens of the input sentences are randomly masked, and then are predicted in decoder, which aims to generate more fine-grained textual representations. 3) crossmodal mask-based pre-trained models @xcite @xcite , both text tokens and vision tokens are randomly masked, which aims to learn multimodal representations between vision and language in a pre-training manner. the mask strategy has been shown effective in many pre-training representation learning tasks. inspired by @xcite , in this work, we try to exploit the mask strategy to address the noiserobust multi-modal fusion problem for mnmt. ## methodology in this section, we introduce our proposed noiserobust multi-modal neural machine translation approach, as illustrated in figure 2 . our proposed model is based on the structure of transformer, which contains four subnetworks, 1) source sentence encoder, 2) image encoder with robust masking matrix, 3) cross-modal gated fusion module and 4) target sentence decoder. without loss of generality, input words are embedded via traditional embedding layer with position embedding. as an example, denote by x @xmath1, • • •, x j n } and v j as the j-th data-pair of source sentence input and its corresponding image, respectively, where n is the source length of x j . formally, the source sentence representation e x j and visual representation e v j are calculated as e x @xmath2, where, emb x is the textual embedding layer with both word embedding and position embedding, emb v is the visual feature extraction layer with resnet-101, e x @xmath3. ## conclusion in this paper, we propose a noise-robust multimodal interactive fusion approach with crossmodal relation-aware mask mechanism to address image noise in mnmt. experiment results and analysis on three benchmark translation tasks demonstrate the effectiveness and superiority of our proposed method. further ablation experiments demonstrate that masking irrelevant visual information helps machine translation. in future work, we will continue to explore how to more effectively remove noisy information in vision.