I n I mage T rans: Multimodal LLM -based Text Image Machine Translation (2025)

Abstract: Multimodal large language models (MLLMs) have shown remarkable capabilities across various downstream tasks. However, when MLLMs are transferred to the text image machine translation (TiMT) task, preliminary experiments reveal that MLLMs suffer from serious repetition and omission hallucinations. To alleviate these issues, this paper first designs an efficient MLLM named InImageTrans for TiMT and then proposes a simple and effective method named multi-conditional direct preference optimization (mcDPO) for advancing the TiMT. Particularly, the proposed mcDPO not only guides the MLLM in rejecting repetition output by creating text output preference pairs automatically, but also guides the MLLM in paying more attention to text information in images by creating image input preference pairs. Furthermore, we build a high-quality benchmark called MCiT for comprehensively evaluating the TiMT capabilities of InImageTrans. Experimental results show that the proposed method significantly outperforms existing open-source MLLMs on MCiT.

## introduction currently, multimodal large language models (mllms) have shown remarkable capabilities in various downstream tasks @xcite @xcite @xcite . take multimodal machine translation (mmt) as an example. typically, visual information, which describes the full or partial related content of one source text information, is simultaneously encoded with this source text by mllms as a fusion representation. mllms are conditioned on this fusion representation to generate the target output, which has gained ## method in this section, we first introduce an efficient mllm called inimagetrans especially for timt, and propose the mcdpo method to mitigate the repetition and omission hallucinations. ## mcit benchmark the current timt benchmarks fail to simultaneously challenge the ocr capabilities of mllms as well as their reasoning and knowledge-based translation skills. therefore, we manually annotate an english-chinese benchmark for timt called mcit. as shown in appendix c, the datasets encompass various real-world scenarios, including documents, scenes, and posters, which require mllms to possess strong ocr recognition capabilities. moreover, the complex textual content demands advanced knowledge and reasoning abilities for accurate translation. during annotation, ten professional english and chinese speakers manually translate the text in paragraphs to ensure semantic consistency and completeness of translation. furthermore, ten annotators are asked to verify each other's translation results. ## discussion according to equ (5), theoretically, there are two optimization directions for rpo: increasing the probability of the golden labels or decreasing the probability of the repetition tokens. to verify it, we select 20 examples whose repetition hallucinations are solved by rpo and measure the output probabilities of the repetition token and the information entropy of the model output at seven generation steps. as shown in figure 7 , removing rpo causes repetition tokens to be generated with high probabilities and low information entropy, indicating the model's confidence in generating repetition tokens. using rpo, we observe that the generation probability of repetition tokens drops sharply. besides, the initial increase in information entropy indicates that the model reduces the confidence of repetition tokens while increasing the confidence of other tokens. subsequently, the decrease in information entropy suggests that the model has started to confidently generate the correct tokens. finally, the above experiments confirm that the proposed rpo method effectively avoids repetition hallucination by dynamically adjusting the confidence of output tokens. ## conclusion in this paper, we investigate the severe repetition and omission hallucinations for existing mllms 20263 on timt task. then we design an efficient mllm named inimagetrans specially for timt and propose a multi-conditional direct preference optimization (mcdpo) approach for advancing the timt to mitigate hallucinations and improve translation quality. furthermore, we build a highquality benchmark named mcit for effectively evaluating the timt capabilities of mllms. experimental results show that the proposed method significantly outperforms existing opensource mllms in terms of both translation quality and hallucination mitigation and approaches the performance of proprietary mllms.