Improving MLLM ’s Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency (2025)

Abstract: Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model’s existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept “Bilingual Cognitive Advantage”. Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks. The code will be released upon acceptance.

## introduction multimodal large language models (mllms) have achieved significant advancements in various document image understanding tasks, particularly in optical character recognition (ocr), which plays a crucial role in extracting text from scanned documents or images. these improvements have led to notable progress in tasks, such as visual question answering (vqa), and information extraction (ie) @xcite @xcite . however, mllms still face challenges towards document image machine translation (dimt)-the task of ',07,' ',07&' 2&5 94$ 4zhq9/%dvh 4zhq9/6)7 translating text in document images from one language to another. @xcite ). an intuitive approach to enhancing mllm's dimt ability is to apply supervised fine-tuning (sft) @xcite on annotated dimt datasets. however, a major challenge with sft is catastrophic forgetting, where fine-tuning mllm on translation tasks often causes a loss of the model's original ocr capability. as shown in figure 1 , while the fine-tuned mllm performs well on translation tasks, achieving a bleu score of 53.92 on the in-domain dimt task, it struggles to accurately extract text from images, with an accuracy of only 5.96 on the ocr task. this significant drop in ocr performance indicates a nearcomplete loss of the mllm's ocr proficiency. to address the challenges associated with sft, existing continual learning methods have been proposed @xcite @xcite @xcite . these methods aim to mitigate catastrophic forgetting and enhance domain generalization through various strategies, such as replay-based methods and regularization-based methods. however, challenges persist in effectively balancing the retention of prior knowledge with the acquisition of new skills, especially in complex tasks like dimt. inspired by the concept of "bilingual cognitive advantage" @xcite @xcite @xcite @xcite , as shown in figure 2 , a learning paradigm that focuses on retaining and leveraging human's existing monolingual strengths while learning new languages, we introduce a simple yet effective fine-tuning paradigm called synchronized self-reviewing (ssr), where the mllm generates the ocr text in the source language first, followed by the translation text in the target language. by synchronous learning, ssr enables the mllm to leverage its strong monolingual ocr proficiency while extending its capabilities to new languages, thereby improving its cross-lingual performance on the dimt task. additionally, ssr enhances the mllm's generalization ability, making it more robust across various domains and tasks. furthermore, the method benefits from the use of large amounts of unsupervised data, reducing the need for extensive parallel datasets, which are often scarce in the dimt task. in summary, this paper presents a novel method to improve dimt performance by using synchronously self-reviewing to preserve monolingual ocr proficiency while enabling cross-lingual dimt. we demonstrate, through extensive experiments, that ssr significantly enhances the mllm's generalization across both ocr and dimt tasks, addressing challenges such as catastrophic forgetting and poor domain generalization. our contributions are summarized as follows: ## related work different from text machine translation @xcite @xcite , document image machine translation aims to translate text within document images from one language to another while preserving the logical layout @xcite . recent advancements in dimt can be categorized into two primary approaches: (1) cascade systems @xcite @xcite , which employ multiple models sequentially and encounter issues such as structural redundancy, error propagation, and high latency. recent advancements in mllms have significantly improved the processing and understanding of text-rich document images @xcite @xcite @xcite @xcite . @xcite explores adding fine-grained vision perception for document images to the mllm without affecting its existing natural image understanding capabilities. @xcite proposes shifted window attention to achieve cross-window connectivity at higher input resolutions and token ## method in this section, we will introduce ssr, a novel finetuning paradigm that leverage the mllm's monolingual (ocr) proficiency to enhance its crosslingual (dimt) ability. the overview of our approach is shown in figure 3 . the key idea is to train the model to first generate the source text (ocr result) before producing the target text (translation text). this approach enables the model to incorporate both image and source text information when generating the target text. although the self-generated source text may contain misrecognized or repeated text, since it is sampled from the model's original distribution, it contributes to a smoother convergence of the model's loss curve during training, which will be discussed in section 5.1. furthermore, this self-review process helps in the retention of the model's original monolingual capabilities. ## conclusion in this paper, we propose a novel fine-tuning paradigm, ssr, to enhance mllms' dimt capabilities by leveraging their ocr proficiency, offering three key advantages. first, monolingual proficiency preserves the mllm's original monolingual competence by maintaining the source text format. second, cross-lingual enhancement enables the mllm to establish relationships between different modalities, enriching target text generation with additional information. finally, our ap-proach can be extended to utilize large-scale unsupervised data to further enhance performance. extensive experiments validate the effectiveness of ssr, demonstrating its superiority in strengthening cross-lingual capabilities while preserving monolingual proficiency.