In-Image Neural Machine Translation with Segmented Pixel Sequence-to-Sequence Model (2023)

Abstract: In-Image Machine Translation (IIMT) aims to convert images containing texts from one language to another. Traditional approaches for this task are cascade methods, which utilize optical character recognition (OCR) followed by neural machine translation (NMT) and text rendering. However, the cascade methods suffer from compounding errors of OCR and NMT, leading to a decrease in translation quality. In this paper, we propose an end-to-end model instead of the OCR, NMT and text rendering pipeline. Our neural architecture adopts encoder-decoder paradigm with segmented pixel sequences as inputs and outputs. Through end-to-end training, our model yields improvements across various dimensions, (i) it achieves higher translation quality by avoiding error propagation, (ii) it demonstrates robustness for out domain data, and (iii) it displays insensitivity to incomplete words. To validate the effectiveness of our method and support for future research, we construct our dataset containing 4M pairs of De-En images and train our end-to-end model. The experimental results show that our approach outperforms both cascade method and current end-to-end model.

## introduction machine translation (mt) is an important technique for achieving communication between different languages @xcite . recently researches on mt mainly focus on text and speech modality @xcite , and there is less work based on image modality mt. in-image machine translation (iimt) is an imageto-image mt task which aims to transform images containing texts from one language to another. the technique of iimt enables visual text translation across different languages, which helps users to understand street signs, scanned documents without the input of text manually @xcite . compared with previous researches on image modality mt @xcite @xcite , the most significant distinction lies in the outputs of iimt which are images containing translated texts, making users understand the meaning of images more easily. iimt is a novel machine translation task, with inherent research value and a wide range of application scenarios. an intuitive method of iimt is decomposing the task into several subtasks: firstly, recognizing the text in the image, then translating the text, and rendering the translated text into the image, which is commonly referred to as the cascade method. specifically, the cascade method of iimt usually contains three parts: (1) recognizing: recognize the text in the origin image with an optical character recognition (ocr) model. (2) translating: translate the text recognized by ocr model to the target language with a neural machine translation (nmt) model. (3) rendering: eliminate text content in the original image and render the translated text. however, a disadvantage of most cascade methods is the issue of error propagation. if errors occur in the output of the previous stage model, it can directly impact the performance of the following models. as shown in figure 1 , "stadt" in the image is incorrectly recognized as "staot" by the ocr model, and the nmt model incorrectly translates it. consequently, the incorrect translated text is rendered into the image, resulting in the incorrect output image. besides, some of the texts in the images may be incomplete because of the blocked by obstacles, resulting in the ocr model recognizing incomplete texts, thereby leading to the decreasing quality of nmt translations and the output images. one of the methods to reduce the influence of error propagation and incomplete texts is designing end-to-end models instead of cascade methods. end-to-end models take into account the entire task directly and learn the transformation from input images to output images without explicit intermediate outputs, eliminating the need for intermediate steps or stages. building end-to-end iimt models is full of challenge because it is hard to generate images with specific texts @xcite , and few iimt dataset is publicly available. @xcite propose end-to-end models for iimt firstly. but the translation quality of their end-toend models is much lower than that of cascade methods. in this paper, we aim to develop an end-to-end iimt model which achieves better translation quality than the cascade method. we regard images as pixel sequences, transforming the image-to-image generating task to a pixel sequence-to-sequence translation task and applying sequence-to-sequence architecture to build our end-to-end model. we also build iimt datasets for training and validating our iimt models. the main contributions of this paper are as follows: ## data construction to our knowledge, there is no publicly available dataset for iimt. since collecting real image pairs with aligned parallel texts for iimt is costly, it is more practical to generate image pairs by rendering the texts from a parallel corpus. therefore, we build datasets containing images with black font and white background of one-line text. specifically, we render texts in the parallel corpora into images with white background foot_0 to build the image pairs, replacing the text pair ⟨x, y ⟩ in parallel corpus to the image pair ⟨x, y⟩. for an image @xmath0, we set height @xmath1, and width w is proportional to the length of the text x. an image pair that contains parallel german-english (de-en) text in the iimt dataset is shown in figure 2 . our method can be used to construct iimt datasets for existing parallel corpora, making it easier to expand data. besides, the font of texts in images can be changed by changing the font file, and the size or color of texts in images can be changed by adjusting parameters in the program, which means our method has strong scalability. we perform inspections on the constructed datasets to ensure the quality, specifically verifying the integrity of text within the images and ensuring the absence of garbled characters. ## analysis in this section, we focus on the following research questions (rqs): (1) does the segmentation of different iterations influence the translation quality? (2) compared to the cascade method, does our model has better translation quality for images with incomplete texts? (3) does the position of texts in images influence the performance of our model? (4) is our method effective on datasets with multiple font sizes or font types? (5) how does error propagation influence the translation quality of the cascade method? ## conclusion in this paper, we propose an end-to-end model for iimt based on segmented pixel sequences to reduce the influence of error propagation. experimental results on iimt datasets show that our model has better translation quality compared with both the current end-to-end model and the cascade method. furthermore, we conduct analysis to demonstrate our end-to-end model has better performance on different types of iimt datasets. in future work, we will try to build more realistic iimt datasets, and design new model architecture to achieve a better translation quality.