L ayout DIT : Layout-Aware End-to-End Document Image Translation with Multi-Step Conductive Decoder (2023)

Abstract: Document image translation (DIT) aims to translate text embedded in images from one language to another. It is a challenging task that needs to understand visual layout with text semantics simultaneously. However, existing methods struggle to capture the crucial visual layout in real-world complex document images. In this work, we make the first attempt to incorporate layout knowledge into DIT in an end-to-end way. Specifically, we propose a novel Layout-aware end-to-end Document Image Translation (LayoutDIT) with multi-step conductive decoder. A layout-aware encoder is first introduced to model visual layout relations with raw OCR results. Then a novel multi-step conductive decoder is unified with hidden states conduction across three step-decoders to achieve the document translation step by step. Benefiting from the layout-aware end-to-end joint training, our LayoutDIT outperforms state-of-the-art methods with better parameter efficiency. Besides, we create a new multi-domain document image translation dataset to validate the model’s generalization. Extensive experiments show that LayoutDIT has a good generalization in diverse and complex layout scenes.

## introduction document image translation (dit) aims to translate text embedded in scanned documents from one language to another. automated dit is a crucial research area for business and academic values. different from plain text translation, texts on document images are arranged with certain layouts. it requires the model to understand the visual layouts and text semantics simultaneously. figure 1 showcases visual layout plays a vital role in dit. missing or error layout information could result in subsequent translation failure. however, existing works struggle to capture the visual layout with a logical reading order on document images @xcite @xcite @xcite . as shown in figure 2 , given a sequence of word pieces extracted by optical character recognition (ocr), existing cascade methods @xcite get the rational translation through four separate processes:(1) layout parsing, (2) logical order detection, (3) sentence segmentation, and (4) translation. they are handled as isolated individuals and optimized independently, which leads to severe error propagation. another problem is that it ignores the semantic correlation shared in different modules. to address the above problems, in this work, we propose a unified end-to-end trainable layoutaware document image translation (layoutdit) network to map a document image processed by ocr to its document translation. layoutdit could simultaneously achieve layout understanding, reading order detection, sentence segmentation, and translation, and share computation and semantic information among these complementary tasks. specifically, layoutdit comprises a layout-aware encoding phase and a multi-step conductive decoding phase. the layout-aware encoding phase aims to model document image's texts with layouts by jointly embedding text tokens and their 2d layouts. the multi-step conductive decoding phase is to capture the reading logic and to get rational translation results. the decoding phase identifies a chain of three decoding sub-processes: reading order decoding, sentence boundary decoding, and translation decoding. it decodes intermediate inferences for each sub-process to approach the final document translation step by step. each intermediate decoding step is fulfilled dedicatedly with a step-decoder. all step-decoders are unified into a multi-step decoder with the conduction of hidden states across them, thus enabling layoutdit's sub-module interaction and end-to-end training. we evaluate layoutdit's strong ability on the public readingbank @xcite for general-domain document image translation. in addition, we create a new multi-domain document image translation dataset (ditrans), which covers three different document domains to test layoutdit's domain-specific capacity. experimental results show that the proposed layoutdit can achieve state-of-the-art translation performance with better parameter efficiency. further analysis demonstrates layoutdit's great power in crossdomain adaptation and few-shot learning, indicating its superiority in these practical dit scenarios. the contributions of this work are summarized as follows: ## layout-aware document image translation in this section, we describe in detail our model architecture, as seen in figure 3 . layoutdit consists of a layout-aware encoding phase and a multi-step conductive decoding phase. the former employs a layout-aware encoder for document image understanding by producing its layout-aware representations; the latter generates the document translation following a "divide-and-conquer" philosophy. concretely, a multi-step conductive decoder is proposed to divide the decoding phase into a chain of three sub-processes and conquers them with dedicated step-decoders, one by one: the final document translation can be obtained by concatenating all sentences' translations. text tokens "the" "strategy" "years" "vision" 1d position @xmath0, y 0 , x 1 , y 1 , w, h) 1 (x 0 , y 0 , x 1 , y 1 , w, h) 5 (x 0 , y 0 , x 1 , y 1 , w, h) 6 (x 0 , y 0 , x 1 , y 1 , w, h) l layout-aware encoder document image reordering step-decoder segmentation step-decoder bos ios bos ios … … … … … … x 1 * x 5 * x 6 * x l * … … … … h 1 ssd h 5 ssd h 6 ssd h l ssd … … … … idx 1 idx 5 idx 6 idx l … … decoder hidden states h 1 rsd h 5 rsd h 6 rsd h l rsd translation step-decoder translation step-decoder translation step-decoder … h 1 sent h 2 sent h m sent … sentences context memory target sent 1 target sent 2 target sent m sent. boundary classes bos -begin of sentence ios -inside of sentence decoding steps step 1 -reading order decoding step 2 -sent. boundary decoding step 3 -translation decoding … -sentence segmentation rule layout-aware encoding phase multi-step conductive decoding phase ® ® ® ® with the conduction of hidden states, all three step-decoders are unified into one multi-step conductive decoder, enabling layoutdit's sub-module interaction and end-to-end training. ## multi-domain dit dataset to the best of our knowledge, there is only one publicly available dataset named readingbank @xcite that consists of web-crawled general-domain document images with pseudo labels. therefore, we first manually annotate a multidomain dit dataset named ditrans to test layout-dit's domain-specific capability. ditrans contains sophisticatedly selected images of three domains: political report, newspaper, and advertisement. we hire 35 professional annotators to label the reading order and sentence boundaries and 30 translators for document translation. translators are shown a document image with reading order and sentence boundaries and are required to produce correct and fluent translations for them in chinese. in addition, we hire 8 professional annotators to sample and check the annotated instances for quality control. we annotate 1,675 instances in total with 34,456 parallel sentence pairs. figure 4 presents document image examples from our ditrans and the public readingbank.