Detect, Disambiguate, and Translate: On-Demand Visual Reasoning for Multimodal Machine Translation with Large Vision-Language Models (2025)

Abstract: Multimodal machine translation (MMT) aims to leverage additional modalities to assist in language translation. With limited parallel data, current MMT systems rely heavily on monolingual English captioning data. These systems face three key issues: they often overlook that visual signals are unnecessary in many cases, they lack transparency in how visual information is used for disambiguation when needed, and they have yet to fully explore the potential of large-scale vision-language models (LVLMs) for MMT tasks. To address these issues, we propose the Detect, Disambiguate, and Translate (DeDiT) framework, the first reasoning-based framework for MMT leveraging LVLMs. DeDiT detects ambiguity in the input sentence, performs visual reasoning only when ambiguity is found, and generates the final translation.We implemented two versions of DeDiT: a prompting method for large proprietary LVLMs and a fine-tuning method for smaller LVLMs using synthetic data. Experiments on the Multi30K and CoMMuTE benchmarks show that DeDiT outperforms state-of-the-art models in disambiguation accuracy and translation quality. We also introduce an improved evaluation metric for disambiguation accuracy that enhances performance assessment and can be applied to proprietary models accessed via APIs.

## introduction multimodal machine translation (mmt) has emerged as an active area of research that aims to improve machine translation quality by leveraging additional modalities beyond text. in real-world scenarios, the text to be translated is often accompanied by images, videos, or other contextual signals. these additional modalities can help disambiguate meaning and resolve ambiguities that are inherent in natural language. due to the scarcity of multimodal multilingual translation datasets, recent work in mmt focuses on creating new mmt datasets @xcite @xcite and adapting pre-trained text-only machine translation systems for mmt using monolingual captioning data @xcite @xcite . however, these approaches have three major issues. first, in real-world situations, not all input sentences contain ambiguity. for non-ambiguous sentences, incorporating visual embeddings can introduce noise, decreasing translation accuracy when the visual context is unnecessarily included in the model's input @xcite . current models are unable to distinguish between ambiguous and non-ambiguous data, treating all inputs uniformly, which can lead to performance degradation. a detailed discussion is provided in the appendix a. second, existing mmt models function as black boxes, producing translations without providing insight into their reasoning processes. this lack of transparency leaves users uncertain about how visual information is utilized, resulting in poor explainability and diminished trust in the models. ideally, an mmt model should clearly indicate whether ambiguity exists in the input and explain how visual information is used to resolve it. finally, large-scale vision-language models (lvlms) have shown strong zero-shot and fewshot performance across various downstream multimodal tasks, achieving excellent results with little or no task-specific data. however, these models have yet to be thoroughly explored for mmt tasks. given that mmt fundamentally involves disambiguation, the extensive world knowledge acquired during large-scale pre-training could be particularly beneficial for detecting and resolving ambiguities. to address these challenges, we propose the detect, disambiguate, and translate (dedit) framework, the first reasoning framework for mmt based on lvlms that generates an explicit, ondemand reasoning process. specifically, dedit models first detect whether ambiguity exists in the input sentence. if ambiguity is detected, the model analyzes the accompanying visual information and determines how to utilize it to resolve the ambiguity before producing the final translation. if no ambiguity is detected, the model bypasses the visual reasoning step and directly generates the final translation based on the input sentence alone. we implemented two different versions of the dedit framework: a prompting based method for large proprietary lvlms and a fine-tuning method using synthetic data from larger models to adapt smaller open-source lvlms. we conducted experiments on two benchmarks: multi30k, which contains both ambiguous and nonambiguous sentences, making it more representative of real-world multimodal translation scenarios; and commute, a newly developed dataset with contrastive examples designed to assess a model's disambiguation capabilities. we refined the disambiguation accuracy evaluation on the commute benchmark by removing its dependence on model logits or probability distributions, which allows the evaluation to better reflect the model's actual disambiguation performance based on its outputs. additionally, it enables the evaluation to be applied to proprietary llms accessed via apis, making it more versatile and applicable to a broader range of models. our experimental results on both benchmarks demonstrate that the two dedit implementations outperform previous state-of-the-art models, validating the effectiveness of the dedit reasoning framework for mmt tasks. the contributions of our work can be summarized as follows: ## methodology the dedit reasoning framework consists of the following stages: (1) detect whether the input sentence contains any ambiguity and identify the specific ambiguous terms; (2) perform visual disambiguation reasoning if ambiguity is detected, analyzing the content of the accompanying image to determine the accurate translation of the ambiguous term identified in the first step. if no ambiguity is detected, this step is skipped; and (3) generate the final translation sentence. in this paper, we implemented two versions of the dedit: (1) for large proprietary lvlms with strong zero-shot capabilities (e.g., claude-3, gpt-4o), we employed dedit prompting by designing specific instructions to guide the models through the reasoning process. (2) for smaller, finetuneable lvlms, we fine-tuned them on synthetic dedit reasoning data generated by larger models, enabling these smaller models to also acquire dedit reasoning capabilities. ## conclusion in this paper, we introduce the detect, disambiguate, and translate (dedit) framework, a novel approach to mmt task that effectively addresses the challenges of on-demand ambiguity resolution in real-world scenarios. our work demonstrates that while visual information can enhance translation quality in ambiguous cases, it can also introduce noise in non-ambiguous instances dedit tackles this issue through on-demand reasoning, selectively utilizing visual data only when ambiguity is detected. we implemented two versions of dedit: a prompting method for large proprietary lvlms and a fine-tuning approach for smaller lvlms using synthetic dedit reasoning data. our experiments on the multi30k and commute benchmarks show that both implementations consistently outperform previous sota models. this demonstrates the effectiveness of lvlms for mmt tasks, and also shows that our proposed dedit reasoning framework can effectively enhance disambiguation and improve multimodal machine translation performance, while providing an on-demand, interpretable, and transparent reasoning process.