PATIMT -Bench: A Multi-Scenario Benchmark for Position-Aware Text Image Machine Translation in Large Vision-Language Models (2025)

Abstract: Text Image Machine Translation (TIMT) aims to translate texts embedded within an image into another language. Current TIMT studies primarily focus on providing translations for all the text within an image, while neglecting to provide bounding boxes and covering limited scenarios. In this work, we extend traditional TIMT into position-aware TIMT (PATIMT), aiming to support fine-grained and layout-preserving translation, which holds great practical value but remains largely unexplored. This task comprises two key sub-tasks: region-specific translation and full-image translation with grounding. To support existing models on PATIMT and conduct fair evaluation, we construct the PATIMT benchmark (PATIMT-Bench), which consists of 10 diverse real-world scenarios. Specifically, we introduce an Adaptive Image OCR Refinement Pipeline, which adaptively selects appropriate OCR tools based on scenario and refines the results of text-rich images. To ensure evaluation reliability, we further construct a test set, which contains 1,200 high-quality instances manually annotated and reviewed by human experts. After fine-tuning on our data, compact Large Vision-Language Models (LVLMs) achieve state-of-the-art performance on both sub-tasks. Experimental results also highlight the scalability and generalizability of our training data.

## introduction text image machine translation (timt) is a challenging branch of neural machine translation (nmt), offering broad application prospects in both academic research and commercial applications. conventional timt methods @xcite @xcite typically focus on generating plain text or markdownformatted translations of all text within an image, failing to precisely preserve the original layout of source text in the image. this limitation gives rise to a critical position alignment problem in realworld applications, where users cannot reliably match translations to the corresponding source text. additionally, all these methods overlook the localized translation requirements. this significantly limits their practical usability. in this paper, we explore position-aware timt (patimt) which contains two core sub-tasks: region-specific translation and full-image translation with grounding as shown in figure 1 . regionspecific translation enables users to manually select one region of an image for translation, which allows fine-grained, user-controllable timt. fullimage translation with grounding ensures precise positional alignment between the translation and source text in the image, enabling seamless rendering of translated image version. recently, large vision-language models (lvlms) @xcite @xcite show remarkable performance across diverse multimodal benchmarks such as ocr @xcite , image understanding @xcite @xcite @xcite and visual grounding @xcite @xcite . they appear to have great potential to perform regionspecific translation and full-image translation with grounding. however, existing lvlms usually fail to follow the above two types of translation instructions, as illustrated in figure 2 . this limitation primarily derives from data scarcity. available timt datasets @xcite @xcite typically lack bounding box annotations or suffer from limited scenarios and scale, making them difficult to support accurate position-aware timt. moreover, there is a lack of comprehensive benchmark. existing timt benchmarks mainly focus on evaluating plain text or markdown translations and only specialize in a single scenario @xcite @xcite . although mit-10m @xcite covers diverse image categories, it lacks bounding box annotations and is confined to simple scenarios, excluding document, infographic images and so on. nevertheless, constructing multi-scenario pa-timt datasets remains challenging for three main reasons. 1) general ocr tools typically provide line-by-line recognition results, leading to semanti-cally incoherent annotations; 2) document-specific ocr tools sometimes ignore text-containing areas and are not always optimal for other scenarios; 3) manual annotation is labor-intensive and expensive. in this work, we address these issues by introducing an automated data processing pipeline to construct a high-quality, multi-scenario patimt dataset and a comprehensive benchmark: first, we introduce an adaptive image ocr refinement pipeline that combines a general easyocr foot_0 with a pdf-optimized mineru @xcite to adaptively process images from different scenarios and refine the results of text-rich samples. second, we propose patimt-bench, which is explicitly designed to evaluate region-specific translation and full-image translation with grounding for images from diverse domains. specifically, we use our pipeline to construct training data, which provides fine-grained bounding boxes in proper layout for text within the images. as for the test set, we select 1200 images with high-quality manual annotations that are carefully reviewed by human experts. experimental results demonstrate that all compact lvlms achieve state-of-the-art performance on patimt-bench after fine-tuning on our training data, outperforming larger models such as qwen2.5-vl-72b and closed-source models like gpt-4o. a series of systematic analyses are conducted which demonstrate the scalability and generalizability of our dataset. ## adaptive image ocr refinement pipeline to develop a high-quality patimt dataset for diverse real-world scenarios, we first extensively collect existing open-source image-text datasets and classify these images into corresponding scenarios. secondly, we introduce an adaptive processing with refinement strategy to adaptively process images from different scenarios. finally, we prompt gpt-4o @xcite to generate the instruction tuning data. figure 3 illustrates the overall pipeline. ## patimt benchmark in this section, we present a detailed description of our patimt-bench. firstly, we formally define the patimt images ocr boxes boxes src words tgt words train 48,884 1,307,516 417,066 24,827,252 30,437,907 test 1,200 -11,102 564,656 685,375 ## conclusion in this paper, we extend the conventional timt task into patimt task, which encompasses two sub-tasks: region-specific translation and fullimage translation with grounding. confronted with data scarcity, we construct the patimt-bench, a benchmark featuring 10 distinct image scenarios. we introduce an adaptive image ocr refinement pipeline to construct training data, which adaptively selects suitable ocr tools according to different image scenarios and refines the results for text-rich images to ensure high-quality annotations. notably, to ensure the accuracy of evaluation, we manually annotate bounding boxes and review the translation results of 1,200 instances to construct the test set. lvlms fine-tuned on our data achieve state-of-the-art performance on patimt-bench, and demonstrate the scaling ability of our training data. in the future, we will explore the application of our dataset to domains including in-image machine translation @xcite and visual text generation @xcite @xcite . additionally, we aim to expand our benchmark into a large-scale multilingual version.