A Query-Response Framework for Whole-Page Complex-Layout Document Image Translation with Relevant Regional Concentration (2025)

Abstract: Document Image Translation (DIT), which aims at translating documents in images from source language to the target, plays an important role in Document Intelligence. It requires a comprehensive understanding of document multi-modalities and a focused concentration on relevant textual regions during translation. However, most existing methods usually rely on the vanilla encoder-decoder paradigm, severely losing concentration on key regions that are especially crucial for complex-layout document translation. To tackle this issue, in this paper, we propose a new Query-Response DIT framework (QRDIT). QRDIT reformulates the DIT task into a parallel response/translation process of the multiple queries (i.e., relevant source texts), explicitly centralizing its focus toward the most relevant textual regions to ensure translation accuracy. A novel dynamic aggregation mechanism is also designed to enhance the text semantics in query features toward translation. Extensive experiments in four translation directions on three benchmarks demonstrate its state-of-the-art performance, showing significant translation quality improvements toward whole-page complex-layout document images.

## introduction document image translation is a fundamental task in which model translates documents in images from source language to the target @xcite . it plays a critical role in various applications such as cross-lingual document retrieval, summarization, and information extraction @xcite . however, performing dit in real-world scenarios faces many difficulties including intricate layouts, complex multi-modality semantics, and cross-lingualities, etc. basically, dit is formulated as an image/textto-text transformation task. early works (afli and way, 2016; @xcite directly use text contents extracted by optical character recognition (ocr) engine as the inputs of text translation model, and have been proven adequate for simple-layout text images such as single-line movie subtitle or multi-line, single-paragraph text. to better exploit multi-modal information of documents, recent methods propose to incorporate visual layout by encoding image pixels @xcite @xcite or layout positions @xcite , and combine them with text features, achieving good results in dit. however, as shown in fig. 1 , when translating whole-page documents with complex layouts, contents of each sentence change greatly in their textual regions, often taking up a relevant but local area of the whole document space. therefore, a favorable dit framework should first be aware and capable of concentrating on the most relevant region for each sentence and then generating the translation conditioning on these key regional texts. nevertheless, these vital steps are severely overlooked in prior methods since they lack ex-plicit objectives or modules targeting the relevant regional concentration and thus suffer degraded performance when translating complex-layout documents. therefore, how to incorporate relevant regional identification and concentration capabilities into dit models to centralize their focus toward the most relevant regional texts, has become a vital step to improve whole-page complex-layout dit. to address this problem, this paper presents the query-response document image translation (qrdit) network. it reformulates dit as a novel query-response procedure where specific modules & objectives are introduced for relevant regional concentration. specifically, qrdit converts dit into a response process of queries (i.e., relevant regional texts), which includes a query stage and a response stage. first, the query stage extracts document multi-modal features, and then carries out sequence labeling to identify each query's prefix word. all prefixes interact with the multi-modal feature through a detr-like @xcite cross-attention to form query embeddings. then, the text area relevant to each source sentence is identified by adaptively gathering the most relevant words, of which word-level relevance scores for each query are computed and employed as the gathering measure. then, at the response stage, model first utilizes a dynamic gate-based aggregation to enhance the text semantics in query features. this mechanism alleviates the feature deviation issue (caused by prior query tasks), keeping features oriented at the translation goal. finally, these enhanced query features are fed into a translation decoder, which employs a per-query strategy to simultaneously generate translation result as response for each query/source sentence. the query and response stages are integrated as an end-toend framework with feature flowing across them consecutively. such a query-response framework successfully incorporates regional concentration capability into dit and therefore leads to significant improvements. extensive experiments in four translation directions on three public benchmarks demonstrate qrdit's sota performance. our contributions are: ## task formulation the inputs of dit include document image i and its ocr results -text words @xmath0. each box b @xmath1, y tl , x br , y br ) i reports the topleft layout position (x tl , y tl ) i and bottom-right layout position (x br , y br ) i of i-th word t i on the image. given these multi-modal inputs (i, t , and b) for a document image, dit aims to generate its targetlanguage document-level translation ŷ : ## methodology as shown in fig. 2 , by reformulating the dit task as a novel query-response procedure, our proposed qrdit consists of 1) an ocr stage, 2) a query stage for query extraction, and 3) a response stage for response/translation generation. firstly, the multi-modal information including text, layout, and image are obtained from preconditioned ocr results. the words and bounding boxes are sorted with a "top-left to bottom-right" rule to keep a fixed input format. afterward, in query stage, inputs are deeply aggregated as multi-modal document representation, based on which the prefix word of each query (i.e., source sentence) is identified. then, with each prefix acting as the "anchor", the most relevant regions are allocated predominant concentrations and words within that region are picked up (with an adaptive extraction strategy) to complement the query. finally, all translation queries are semantically enhanced and then prompt response modules to generate their translations in parallel, which are concatenated as the final document translation results while also preserving each translation's layout position. document transformer transformer encoder n query prefix feat. transformer decoder mlp ⊗ n query mask emb. translation decoder 传真 封面 页 公司 ：philip morris 有限 公司 layout emb. image emb. text emb. ocr q 1 q 2 q n ... multi-modal feat. text words, bounding boxes & image formalized document translation results 传真 封面 页 日期 ：1999 年 1 月 14 日 文件 编号 ： 1129.01 收件人 ： alice 和 bob 公司 ：philip morris 股份 有限 公司 ... ocr processing query extraction response/translation generation query prefix identification document feat. extraction relevant regional concentration adaptive query extraction adaptive τ dynamic gate query feat. text emb. text-enhanced feat. text semantic enhancement pre-query response ## related work deep neural models have demonstrated significant effectiveness and driven the expansion of machine translation from plain text to multimodal domains @xcite @xcite @xcite . as a multimodal machine translation task, document image translation (dit) involves the collaborative processing of document text and visual layout. there have been many works oriented at the simple-layout single-line/paragraph text image (e.g., movie subtitle or street view text) translation. most of them attempt to alleviate the image-text modality gap in feature space, with techniques such as multi-modal codebook @xcite , crossmodal knowledge transfer @xcite or distillation @xcite , and multi-modal contrastive learning @xcite . some recent studies @xcite begin exploring the image-format output for both text translation and visual style preserving. however, these methods may suffer deteriorated performance for whole-page complex-layout documents due to the lack of visual layout modeling toward complex layouts. for complex-layout documents, early cascade solutions rely on an ocr-translation pipeline @xcite or add an extra layout parser @xcite for layout analysis. many recent works have verified incorporating visual layout into an end-to-end dit model by encoding image pixels @xcite or word positions @xcite . nevertheless, these methods lack special awareness of key text areas during translation. as a resolution, our proposed framework deploys specific modules and objectives toward relevant regional concentration to ensure more accurate source texts & translations. ## conclusion this paper proposes the query-response framework. it integrates relevant regional concentration capabilities, achieving accurate relevant text extraction and improving translation. we also propose a dynamic aggregation mechanism to enhance query feature text semantics toward translation. experiments in four directions on three datasets demonstrate qrdit significantly outperforms prior methods and achieves new sota performance.