Multimodal Machine Translation with Text-Image In-depth Questioning (2025)

Abstract: Multimodal machine translation (MMT) integrates visual information to address ambiguity and contextual limitations in neural machine translation (NMT). Some empirical studies have revealed that many MMT models underutilize visual data during translation. They attempt to enhance cross-modal interactions to enable better exploitation of visual data. However, they only focus on simple interactions between nouns in text and corresponding entities in image, overlooking global semantic alignment, particularly for prepositional phrases and verbs in text which are more likely to be translated incorrectly. To address this, we design a Text-Image In-depth Questioning method to deepen interactions and optimize translations. Furthermore, to mitigate errors arising from contextually irrelevant image noise, we propose a Consistency Constraint strategy to improve our approach’s robustness. Our approach achieves state-of-the-art results on five translation directions of Multi30K and AmbigCaps, with +2.35 BLEU on the challenging MSCOCO benchmark, validating our method’s effectiveness in utilizing visual data and capturing comprehensive textual semantics.

## introduction multimodal machine translation (mmt) utilizes modalities beyond text, especially visual data, to clarify ambiguous words and supplement incomplete contexts, thereby improving translation quality @xcite @xcite . some mmt studies focus on the extraction of visual information, which are devoted to extract text-related visual features for subsequent translation @xcite @xcite . other studies expect to enhance crossmodal fusion and alignment to improve translation quality @xcite @xcite . source: a woman rides her bike while on her smartphone . target: eine frau fährt fahrrad , während sie mit ihrem smartphone beschäftigt ist . ❌ ❌ ❌ ✅ eine frau fährt ihr fahrrad , während sie auf ihrem handy fährt . (a woman rides her bike on her phone.) nmt: eine frau fährt ihr fahrrad , während sie auf ihrem handy fährt . (a woman rides her bike le on her phone.) mmt-vqa: eine frau fährt mit ihrem fahrrad, während sie auf ihrem smartphone ist. (a woman rides her bike on her smartphone.) gpt4: eine frau fährt fahrrad , während sie mit ihrem smartphone beschäftigt ist . (a woman rides her bike while using her smartphone.) consqa-mmt: figure 1 : "on her smartphone" in english conveys "busy with her phone" in this scenario. the first three approaches directly translate the preposition "on" to "auf" in german, inaccurately shifting the meaning to "located above her phone." our method effectively leverages image information to capture correct semantics and produce a precise translation. later, people found no difference in translation performance of the above models with relevant or irrelevant images and they questioned that many mmt models' utilization of images is inadequate @xcite @xcite . @xcite and @xcite attributed this to insufficient interaction between two modalities, opening up a new research direction. for instance, @xcite introduced a progressive transformer model with modality difference awareness to enhance the visual-textual interaction. @xcite designed a dual-objective framework, jointly training mmt and visually conditioned masked language modeling, allowing better utilization of images in translation. @xcite converted the masked source text into questionanswering pairs and jointly trained mmt and visual question-answering (vqa) to strengthen cross-modal interaction, enforcing their model to be sensitive to visual information. however, these works focus on simple interactions between specific types of words (nouns, characters, colors) and corresponding entities in the image, while accurate translation often requires attention to overall semantics and contexts. as shown in figure 1 , the phrase "on her phone" in this context means "using her phone" but nmt translates it literally into "auf ihrem handy" which can only mean "located above her phone" in german. nmt translates incorrectly due to the incomplete context of source text. when paired with an image showing "the woman is using her phone," previous mmt methods still fail to get correct translation. according to their methods, the model can only focus on the interaction and alignment of "woman" "bike" "smartphone" in this example, lacking a deeper understanding of the relationships between entities. to address this, we design a text-image in-depth questioning method, leveraging the extensive knowledge and understanding capabilities of large language model (llm). for example in figure 1 , the question is "what is the woman doing in addition to riding her bike?" and the answer is "using her phone". our approach enhances the depth of text-image interaction, providing the correct translation result. furthermore, due to the limitations of visual information or the misalignment between text and images, the answers may not be derived from images. forcing a strict question-answering loss will interfere with model training and negatively impact translation. to address this, we propose a consistency constraint strategy that relaxes the strict requirement for giving correct answers, focusing on aligning image-to-source and image-to-target semantic distances. the semantic distances are measured by the accuracy of the question-answering task. our main contributions can be summarized as follows: ## related work mmt has emerged as a rapidly growing research domain @xcite . early investigations in this field focused on three key areas: text-aware visual feature extraction techniques @xcite @xcite @xcite , effective crossmodal representation learning @xcite @xcite , and enhancing model robustness through mitigating noise propagation from irrelevant visual inputs @xcite , error accumulation in cross-modal fusion @xcite , and semantic drift during bilingual decoding @xcite @xcite . however, elliott (2018) conducted adversarial experiments and revealed that replacing irrelevant images had no significant impact on translation results. @xcite further pointed out that the performance improvement of multimodal models might be due to regularization effects rather than the utilization of visual information. @xcite designed an entity mask probing task, demonstrating that image has little effect on the translation when text is complete. @xcite suggested that visual information serves a supplementary role in mmt and can be substituted. to effectively leverage the auxiliary role of visual modalities in mmt, @xcite proposed using deliberation networks and structured visual information to generate draft translations, which are then refined based on the visual context. @xcite and @xcite enhanced cross-modal interaction and visual attention through mechanisms like bidirectional and coattention networks. @xcite introduced a modality difference-aware module that progressively fuses visual features layer by layer to bridge modality gaps. @xcite focused on ambiguous sentences that benefit from visual cues, enhancing both multimodal and text-only translation approaches. @xcite proposed a new framework with lightweight adapters and guided self-attention, jointly training mmt and visual masked language models. @xcite converted the text, previously masked in the detection task, into question-answer pairs and trained mmt with vqa. @xcite proposed a mmt architecture named gram to train models on multimodal datasets with masked sentences, improving visual context utilization. recently, an emerging paradigm @xcite @xcite ) employed text-conditioned generative or retrieval models to synthesize and integrate visual features into translation process. these studies laid the foundation for image-text alignment in mmt, while our work advances crossmodal interaction mechanisms to enhance translation accuracy and robustness. ## method in this section, we first introduce the framework and the basic loss function implemented in our mmt task ( §3.1). next, we introduce our proposed consqa-mmt, shown in figure 2 , which features the text-image in-depth questioning method ( §3.2) and the consistency constraint strategy ( §3.3). finally, we outline our method's overall training objective ( §3.4). ## conclusion we proposed an effective and robust multimodal machine translation model, consqa-mmt, with novel cross-modality interaction mechanisms. we designed a text-image in-depth questioning method that enhances the interaction and alignment of the two modalities through refined question-answer pairs and joint training. the proposed consistency constraint strategy reduces the influence of irrelevant images on translations by relaxing loss constraints on vqa branches. moreover, incorporating text-only translation constraints improves translation accuracy and fluency. collectively, consqa-mmt significantly enhances the overall quality of multimodal machine translation and achieves stateof-the-art performance.