Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective (2022)

Abstract: Multimodal machine translation (MMT) aims to improve translation quality by equipping the source sentence with its corresponding image. Despite the promising performance, MMT models still suffer the problem of input degradation: models focus more on textual information while visual information is generally overlooked. In this paper, we endeavor to improve MMT performance by increasing visual awareness from an information theoretic perspective. In detail, we decompose the informative visual signals into two parts: source-specific information and target-specific information. We use mutual information to quantify them and propose two methods for objective optimization to better leverage visual signals. Experiments on two datasets demonstrate that our approach can effectively enhance the visual awareness of MMT model and achieve superior results against strong baselines.

## introduction multimodal machine translation (mmt) typically improves text translation quality by introducing an extra visual modality input. this task hypothesizes that the corresponding visual context is helpful for disambiguation and omission completion when the source sentence is ambiguous or even incorrect. compared to the purely text-based machine translation, the critical point of mmt is to find out an effective approach to integrating images into a mt model that makes the most of visual information. most of the existing studies have dedicated their efforts to the way of extracting multi-granularity visual features for integration @xcite @xcite @xcite or designing model architectures for better message passing across various modalities @xcite @xcite @xcite * corresponding author @xcite , which achieve promising performances in most of the multimodal scenarios. despite the success of these methods, some researchers reveal that visual modality is underutilized by the modern mmt models @xcite . @xcite leverage an adversarial evaluation method to appraise the awareness of visual contexts of mmt models. an interesting observation is that the disturbed image input generally shows no significant effects on the performance of public mmt models, indicating the insensitivity to the visual contexts. recently, @xcite revisit the need for visual context in mmt and find that their strong mmt model tends to ignore the multimodal information. they suggest that the gains from the multimodal signals over text-only counterparts are, in fact, due to the regularization effect. in practice, the existing approaches mainly focus on a better interaction method between textualand visual-information. in training, they generally employ maximum likelihood estimation (mle) to optimize the entire model. in most cases, the source sentence carries sufficient context for translation, leading to the neglect of visual modality under such training paradigm. however, the visual information is still crucial when the source sentence faces the problem of incorrection, ambiguity and genderneutrality . accordingly, we argue that the visual modality remains an excellent potential to facilitate translation, which inspire us to explore taking full advantage of the visual modality in mmt. to this end, we propose increasing visual awareness from an information theoretic perspective. specifically, we quantify the amount of visual information with an information-theoretic metric, namely mutual information (mi). the quantification is then divided into two parts: 1) the sourcespecific part, which means the information between source texts and images; 2) the target-specific part, which means the mutual information between target texts and images given source texts, namely conditional mutual information (cmi). for the sourcespecific part, we maximize the mutual information by resorting to a lower bound of mi. for the target-specific part, we propose a novel optimization method for maximizing the conditional mutual information (see details in 3.2). we evaluate our approach on two public datasets. the experimental results and in-depth analysis indicate that our method can effectively enhance the mmt model's sensitivity to visual context, which manifests that mmt has great potential when the utilization of visual context is adequate. to conclude, the main contributions of our paper are three-fold: ## methodology in this section, we mathematically describe the basic notion of our proposed method from an information theoretic perspective. let x denotes a random variable over source sentences, y denotes a random variable over target sentences, and z is a random variable over images. they are jointly distributed according to some true p.m.f. p(x, y, z). we formulate the visual awareness of a mmt model as a joint mutual information i(x, y; z), which indicates how much the visual modality contributes to the overall system. a high joint mutual information suggests a strong sensitivity of the mmt model to visual information. however, it is intractable to directly maximize this joint mutual information in training. thus, in this work we decompose i(x, y; z) into two parts by the chain rule of mutual information and optimize these two text encoder image encoder text decoder x x - z f(x) r(z) f(x -) r(z ) p(y | x, z) p(y | x, z ) max i(y; z | x) max i(x; z) l tmi l smi l mmt gate fusion drop in for m ation parts separately: as shown in figure 1 , the green part i(x; z) is the source-specific mutual information (smi), which indicates the mutual information between source texts and images; the yellow part i(y; z|x) is the target-specific conditional mutual information (tmi), which indicates the mutual information between target texts and images given source texts. figure 2 illustrates the overview of our proposed method. in the following, we describe how to optimize the source-specific and target-specific mutual information in detail. ## limitations since we conduct all experiments on the machine translation task, it is unclear whether our approach can benefit other multimodal tasks, e.g. visual qa or multimodal dialog. another thing worth noting is that the image-source-target triplet data is not easily available in reality. thus how to effectively use the unpaired image in the unsupervised way is a more promising research.