Exploring Better Text Image Translation with Multimodal Codebook (2023)

Abstract: Text image translation (TIT) aims to translate the source texts embedded in the image to target translations, which has a wide range of applications and thus has important research value. However, current studies on TIT are confronted with two main bottlenecks: 1) this task lacks a publicly available TIT dataset, 2) dominant models are constructed in a cascaded manner, which tends to suffer from the error propagation of optical character recognition (OCR). In this work, we first annotate a Chinese-English TIT dataset named OCRMT30K, providing convenience for subsequent studies. Then, we propose a TIT model with a multimodal codebook, which is able to associate the image with relevant texts, providing useful supplementary information for translation. Moreover, we present a multi-stage training framework involving text machine translation, image-text alignment, and TIT tasks, which fully exploits additional bilingual texts, OCR dataset and our OCRMT30K dataset to train our model. Extensive experiments and in-depth analyses strongly demonstrate the effectiveness of our proposed model and training framework.

## introduction in recent years, multimodal machine translation (mmt) has achieved great progress and thus received increasing attention. current studies on mmt mainly focus on the text machine translation with scene images @xcite @xcite @xcite @xcite . however, a more common requirement for mmt in real-world applications is text image translation (tit) @xcite , which aims to translate the source texts embedded in the image to target translations. due to its wide figure 1 : an example of text image translation. the bounding box in red represents the text to be recognized. we can observe that the incorrect ocr result will negatively affect the subsequent translation. current studies on tit face two main bottlenecks. first, this task lacks a publicly available tit dataset. second, the common practice is to adopt a cascaded translation system, where the texts embedded in the input image are firstly recognized by an optical character recognition (ocr) model, and then the recognition results are fed into a textonly neural machine translation (nmt) model for translation. however, such a method tends to suffer from the problem of ocr error propagation, and thus often generates unsatisfactory translations. as shown in figure 1 , "富锦消防" ("fu jin xiao fang") in the image is incorrectly recognized as "富锦消阳" ("fu jin xiao yang"). consequently, the text-only nmt model incorrectly translates it into "fujin xiaoyang". furthermore, we use the commonly-used paddleocr foot_0 to handle several ocr benchmark datasets. as reported in in this paper, we first manually annotate a chinese-english tit dataset named ocrmt30k, providing convenience for subsequent studies. this dataset is developed based on five chinese ocr datasets, including about 30,000 image-text pairs. besides, we propose a tit model with a multimodal codebook to alleviate the ocr error propagation problem. the basic intuition behind our model is that when humans observe the incorrectly recognized text in an image, they can still associate the image with relevant or correct texts, which can provide useful supplementary information for translation. figure 3 shows the basic architecture of our model, which mainly consists of four modules: 1) a text encoder that converts the input text into a hidden state sequence; 2) an image encoder encoding the input image as a visual vector sequence; 3) a multimodal codebook. this module can be described as a vocabulary comprising latent codes, each of which represents a cluster. it is trained to map the input images and ground-truth texts into the shared semantic space of latent codes. during inference, this module is fed with the input image and then outputs latent codes containing the text information related to ground-truth texts. 4) a text decoder that is fed with the combined representation of the recognized text and the outputted latent codes, and then generates the final translation. moreover, we propose a multi-stage training framework for our tit model, which can fully exploit additional bilingual texts and ocr data for model training. specifically, our framework consists of four stages. first, we use a large-scale bilingual corpus to pretrain the text encoder and text decoder. second, we pretrain the newly added multimodal codebook on a large-scale monolingual corpus. third, we further introduce an image encoder that includes a pretrained vision transformer with fixed parameters to extract visual features, and continue to train the multimodal codebook. additionally, we introduce an image-text alignment task to enhance the ability of the multimodal codebook in associating images with related texts. finally, we finetune the entire model on the ocrmt30k dataset. particularly, we maintain the image-text alignment task at this stage to reduce the gap between the third and fourth training stages. our main contributions are as follows: ## related work in mmt, most early attempts exploit visual context via attention mechanisms @xcite @xcite @xcite . afterwards, ive et al. (2019) employ a translate-and-refine approach to improve translation drafts with visual context. meanwhile, calixto et al. (2019) incorporate visual context into mmt model through latent variables. different from these studies focusing on coarse-grained visual-text alignment information, yin et al. (2020) propose a unified multimodal graph based encoder to capture various semantic relationships between tokens and visual objects. lin et al. (2020) present a dynamic context-guided capsule network to effectively capture visual features at different granularities for mmt. obviously, the effectiveness of conventional mmt heavily relies on the availability of bilingual texts with images, which restricts its wide applicability. to address this issue, @xcite first build a token-image lookup table from an image-text dataset, and then retrieve images matching the source keywords to benefit the predictions of target translation. recently, @xcite present a phrase-level retrieval-based method that learns visual information from the pairs of source phrases and grounded regions. besides, researchers investigate whether visual information is really useful for machine translation. @xcite finds that irrelevant images have little impact on translation quality. @xcite attribute the gain of mmt to the regularization effect. unlike these conclusions, @xcite and @xcite observe that mmt models rely more on images when textual ambiguity is high or textual information is insufficient. to break the limitation that mmt requires sentence-image pairs during inference, researchers introduce different modules, such as image prediction decoder @xcite , generative imagination network @xcite , autoregressive hallucination transformer @xcite , to produce a visual vector sequence that is associated with the input sentence. significantly different from the above studies on mmt with scene images, several works also explore different directions in mmt. for instance, @xcite and @xcite investigate product-oriented machine translation, and other researchers focus on multimodal simultaneous machine translation @xcite . moreover, there is a growing body of studies on video-guided machine translation @xcite @xcite . these studies demonstrate the diverse applications and potential of mmt beyond scene images. in this work, we mainly focus on tit, which suffers from incorrectly recognized text information and is more practicable in real scenarios. the most related work to ours mainly includes @xcite @xcite . @xcite first explore in-image translation task, which transforms an image containing the source text into an image with the target translation. they not only build a synthetic in-image translation dataset but also put forward an end-toend model combining a self-attention encoder with two convolutional encoders and a convolutional decoder. @xcite focus on the tit task, and propose to combine ocr and nmt into an endto-end model with a convolutional encoder and an autoregressive transformer decoder. along this line, @xcite apply multi-task learning to this task, where mt, tit, and ocr are jointly trained. however, these studies only center around synthetic tit datasets, which are far from the real scenario. ## dataset and annotation to the best of our knowledge, there is no publicly available dataset for the task of tit. thus we first manually annotate a chinese-english tit dataset named ocrmt30k, which is based on five commonly-used chinese ocr datasets: rctw-17 @xcite , casia-10k @xcite , icdar19-mlt @xcite , icdar19-lsvt @xcite and icdar19-art @xcite . we hire eight professional translators for annotation over five months and each translator is responsible for annotating 25 images per day to prevent fatigue. translators are shown an image with several chinese texts and are required to produce correct and fluent translations for them in english. in addition, we hire a professional translator to sample and check the annotated instances for quality control. we totally annotate 30,186 instances and the number of parallel sentence pairs is 164,674. figure 2 presents an example of our dataset. ## conclusion in this paper, we release a chinese-english tit dataset named ocrmt30k, which is the first publicly available tit dataset. then, we propose a novel tit model with a multimodal codebook. typically, our model can leverage the input image to predict latent codes associated with the input sentence via the multimodal codebook, providing supplementary information for the subsequent translation. moreover, we present a multi-stage training framework that effectively utilizes additional bilingual texts and ocr data to refine the training of our model. in the future, we intend to construct a larger dataset and explore the potential applications of our method in other multimodal tasks, such as videoguided machine translation.