Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs (2023)

Abstract: This paper presents an in-depth study of multimodal machine translation (MMT), examining the prevailing understanding that MMT systems exhibit decreased sensitivity to visual information when text inputs are complete. Instead, we attribute this phenomenon to insufficient cross-modal interaction, rather than image information redundancy. A novel approach is proposed to generate parallel Visual Question-Answering (VQA) style pairs from the source text, fostering more robust cross-modal interaction. Using Large Language Models (LLMs), we explicitly model the probing signal in MMT to convert it into VQA-style data to create the Multi30K-VQA dataset. An MMT-VQA multitask learning framework is introduced to incorporate explicit probing signals from the dataset into the MMT training process. Experimental results on two widely-used benchmarks demonstrate the effectiveness of this novel approach. Our code and data would be available at: https://github.com/libeineu/MMT-VQA .

## introduction broadening the scope of traditional machine translation, multimodal machine translation (mmt) enhances the quality of text translation by factoring in an auxiliary visual modality @xcite . the key challenge of mmt is finding an effective way to integrate images into the translation model, thereby maximizing the utilization of visual information @xcite @xcite . meanwhile, it operates on the premise that the relevant visual context can help clarify or fill in gaps when the source sentence is unclear or incomplete. while, researchers soon discovered that mmt systems do not always perform as expected: the visual component often contributes little to the trans-lation process when the text is complete. surprisingly, mmt systems can still behave well even the image input is unrelated to the text @xcite @xcite . this raises questions about the actual role of visual input in the mmt framework. aiming this, @xcite pointed out that vision features are helpful when the source sentence miss important patterns. along this line, @xcite designed more specific probing tasks to evaluate how mmt behave in a limited textual context. but a severe problem still remains, that mmt systems exhibit less sensitivity to the information conveyed by the vision modality when the text is complete. our hypothesis posits that this may not result from the redundancy of visual data but rather from the lack of effective interaction between the text and visual modalities. it is a reasonable progression to consider the probing task proposed by @xcite , which explicitly measures the crossmodality ability. this task involves masking critical entities and evaluating whether mmt can produce accurate translation by attending to the image. however, this process falls short in providing any actionable feedback to mmt as it serves as a evaluation metric. consequently, it is worth investigating whether we can incorporate the probing signals into the training process, instead of using it solely for evaluation. in this study, we focus on augmenting mmt systems by incorporating probing signals during training to fortify the interplay between text and visual data. this can actively stimulating the textual representation to engage with visual data. to achieve this goal, we leverage large language models (llms) to identify intricate contextual words and generate questions, thereby transforming original text into visual question-answering (vqa) style pairs. this results in a new multi30k-vqa dataset. on this basis, we introduce a mmt-vqa multi-task learning framework, prompting the mmt model to actively "probe" visual data guided by the textual context. our main contributions can be summarized as follows: ## an empirical study on advanced vision features the use of stronger vision models as image encoders in mmt has gained widespread attention. @xcite demonstrated that the application of more robust pre-trained vision models, such as the vision transformer (vit) @xcite , considerably enhances mmt systems, as evidenced by the substantial improvements in the accuracy metrics of the proposed probing tasks. continuing along this research trajectory, we question whether more recent pre-trained vision models can offer additional advantages to mmt systems, a topic yet to be fully explored. in this context, we chose mae @xcite due to its notable impact in the field of computer vision, as well as two vision-language pre-trained models, namely clip @xcite and blip @xcite , for their potential as vision encoders in a mmt system. ## analysis here, we aim to take more in-depth analysis on the importance of mmt-vqa. without specific notion, all results were conducted on multi30k en-de task. ## related work the task of mmt forms an intersection between nlp and cv, with the goal of enhancing translation outcomes through the integration of image-based context. the earlier approaches focused primarily on the effective amalgamation of these dual-modal representations @xcite @xcite . subsequent research, however, began to question the significance of visual modality, finding that irrelevant images did not substantially influence translation quality. indeed, the absence of images did not lead to a substantial drop in bleu scores @xcite . @xcite provided further insight into this phenomenon by demonstrating that visual modality could offer utility in contexts with limited linguistic data, but its influence was less pronounced when full sentences were available. in recent studies, @xcite attributed improvements to the effects of regularization during training. they further showed that even a tiny configuration could outperform earlier models in terms of bleu score. concurrently, additional efforts include the use of hallucination representations augmentation @xcite , noise-input to mask the additional noise thereby guiding the model to focus on meaningful interactions @xcite , and dual-level interactive mixup for feature alignment @xcite have emerged. however, existing mmt systems still face difficulties when dealing with complete textual contexts, a concern that also forms the crux of the present study. distinctively, we focus on fortifying text-visual interactions through the deployment of probing signals during training. our approach shares thematic similarity with @xcite , who explored this issue from a mutual information standpoint. ## conclusions in this work, we offer a novel perspective on multimodal machine translation (mmt), attributing reduced system sensitivity to visual information in complete text inputs to insufficient cross-modal interaction rather than image information redundancy. we propose a unique approach that generates visual question-answering (vqa) style pairs from source text via llms to model the probing signal in mmt explicitly. as a bonus, we propose a multi30k-vqa dataset and design a mmt-vqa multitask learning framework. our methodology is validated by experimental results on two representative benchmarks, evidencing its effectiveness and driving forward the mmt research.