Document Image Machine Translation with Dynamic Multi-pre-trained Models Assembling (2024)

Abstract: Text image machine translation (TIMT) is a task that translates source texts embedded in the image to target translations. The existing TIMT task mainly focuses on text-line-level images. In this paper, we extend the current TIMT task and propose a novel task, **D**ocument **I**mage **M**achine **T**ranslation to **Markdown** (**DIMT2Markdown**), which aims to translate a source document image with long context and complex layout structure to markdown-formatted target translation.We also introduce a novel framework, **D**ocument **I**mage **M**achine **T**ranslation with **D**ynamic multi-pre-trained models **A**ssembling (**DIMTDA**).A dynamic model assembler is used to integrate multiple pre-trained models to enhance the model’s understanding of layout and translation capabilities.Moreover, we build a novel large-scale **Do**cument image machine **T**ranslation dataset of **A**rXiv articles in markdown format (**DoTA**), containing 126K image-translation pairs.Extensive experiments demonstrate the feasibility of end-to-end translation of rich-text document images and the effectiveness of DIMTDA.

## introduction text image machine translation (timt) is an emerging field focused on translating text from one language to another within images, as explored by @xcite . recent studies in timt fall into two primary categories: (1) cascade systems @xcite @xcite , which involve sequential multi-model operations. these systems often grapple with structural redundancy, error propagation, and high latency. existing end-to-end methods @xcite @xcite have shown promising results on the timt task. however, as shown in figure 1 , these methods are primarily tailored for text-line-level applications with short texts, such as road signs, shop billboards, and subtitles. but document images are also prevalent in real-world scenarios, such as academic papers, magazines, and scanned documents. unfortunately, direct application of these end-to-end timt methods to document images with long context and complex layout encounters significant challenges. as shown in we think there are three reasons for the poor performance of the current timt models on document images: (1) long context: the number of tokens contained in text-line-level images is typically less than 50, whereas the number of tokens in document images ranges from several hundred to over a thousand; (2) complex layout: document images contain complex layouts, like title, paragraph, table, figure, and formula, which severely hinder the performance of the model; (3) data scarcity: there is no large-scale public dataset available for the end-to-end document image machine translation (dimt) task. these three reasons lead to the slow convergence and poor performance of the end-to-end model. 3 to address the above issues, we extend the current end-to-end timt task and introduce a novel document image machine translation to markdown (dimt2markdown) task. this novel task aims to translate document images from one language to another while meticulously preserving the logical layout in markdown format, as shown in figure 1 . moreover, to alleviate the convergence problem of the end-to-end model, we propose a novel framework for document image machine translation with dynamic multi-pre-trained models assembling (dimtda), which leverages the multiple pre-trained models to initialize the endto-end dimt model. dimtda is a pure end-toend framework that contains a model assembler to dynamically connect multi-pre-trained models. specifically, it contains four components: (1) a pretrained optical character recognition (ocr) model to capture the textual information; (2) a pre-trained layout model to encode the layout structure information; (3) a pre-trained translation decoder to generate translated texts; (4) a dynamic model assembler to fuse the pre-trained models. besides, to alleviate data scarcity, we construct a large-scale dataset for the dimt2markdown task, dota, containing 126k scientific document images collected from arxiv paired with chinese 3 this can be confirmed by our main contributions are concluded as follows: ## related work text image machine translation is to translate the source text image to target translations. in recent years, various end-to-end methods @xcite @xcite @xcite have been proposed. @xcite uses a convolutional encoder and an autoregressive transformer decoder to build the model. @xcite trains the end-to-end timt model with text translation as an auxiliary task. @xcite proposes an end-to-end timt framework that bridges the modality gap with pretrained models. while these end-to-end methods have demonstrated satisfactory performance, their effectiveness is limited to images with short context and simple layout structure. multi-model machine translation aims to improve text machine translation performance by incorporating visual information @xcite . there have been many studies @xcite @xcite focused on this area of research. the biggest distinction between multi-model machine translation and our dimt2markdwon task lies in the input difference. multi-model machine translation takes source text as input with images as auxiliary, while the only input for our task is a document image. with the advancement of document image research, numerous pre-trained models @xcite @xcite @xcite , especially for document images, have emerged. these models have shown excellent performance in their respective downstream tasks, demonstrating robust capabilities in understanding document images. this dynamic gate (b) dynamic model assembler feed forward cross-attention self-attention ... layout queries self-attention cross-attention feed forward ocr queries ... ... ocr qformer layout qformer source texts (a) pre-trained models (c) assembled models target texts in markdown format pre-trained translation decoder pre-trained ocr encoder pre-trained layout encoder source image dynamic model assembler layout encoder layout decoder source image layout stucture ocr encoder ocr decoder source image translation encoder translation decoder source texts target texts figure 2: the diagram of the proposed dimtda with an ocr encoder, a layout encoder, a dynamic model assembler, and a translation decoder. the ocr encoder, layout encoder, and translation decoder are initialized from pre-trained models. ## dota dataset to facilitate the research community, we build a novel large-scale document image machine translation dataset of arxiv articles in markdown format (dota). specifically, we randomly select 18,496 papers published on arxiv from 2020 to 2023 and download the corresponding pdf files and l a t e x source codes. the category distribution of these articles is shown in following @xcite , we employ latexml to process source codes, transforming them into html files, and then to markdown files. throughout this process, formulas and tables are retained in their original l a t e x source code, while figures are omitted. then, we split the markdown files according to the page breaks in the pdf file and rasterize each page as an image to create the final paired dataset. several techniques are used to ensure the quality of the dataset. see the appendix a for details. for translation, we translate english texts in the markdown files of the train set into chinese with google translate api. the valid set and test set are translated by professional translators. for formulas and tables, we utilize special tokens to substitute them before the translation process, reinstating them after the translation is completed. to guarantee translation quality, we use comet @xcite to score each translation pair and remove the lowest scoring 10% of pairs. foot_3 besides, we randomly sample 1000 translation pairs and employ three professional translators to evaluate these translation texts on a scale of 1 to 5, with 1 indicating the lowest quality and 5 indicating the highest, in terms of fluency and fidelity. the average score of fluency and fidelity are 3.70 and 3.53, finally, 126,345 paired images, english markdown files, and their corresponding chinese translations are obtained. more details are illustrated in ## conclusion in this paper, we propose a novel text image machine translation task, dimt2markdown, taking a source document image as input and outputting translations in markdown format, which broadens the research of text image machine translation. we also construct a large-scale dataset named dota for this task. besides, a novel end-to-end framework, dimtda, is introduced, which dynamically assembles multi-pre-trained models to fulfill this task. comprehensive experiments demonstrate promising prospects for the direct translation of document images into markdown-formatted texts and the effectiveness of dimtda.