PEIT : Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation (2023)

Abstract: Image translation is a task that translates an image containing text in the source language to the target language. One major challenge with image translation is the modality gap between visual text inputs and textual inputs/outputs of machine translation (MT). In this paper, we propose PEIT, an end-to-end image translation framework that bridges the modality gap with pre-trained models. It is composed of four essential components: a visual encoder, a shared encoder-decoder backbone network, a vision-text representation aligner equipped with the shared encoder and a cross-modal regularizer stacked over the shared decoder. Both the aligner and regularizer aim at reducing the modality gap. To train PEIT, we employ a two-stage pre-training strategy with an auxiliary MT task: (1) pre-training the MT model on the MT training data to initialize the shared encoder-decoder backbone network; and (2) pre-training PEIT with the aligner and regularizer on a synthesized dataset with rendered images containing text from the MT training data. In order to facilitate the evaluation of PEIT and promote research on image translation, we create a large-scale image translation corpus ECOIT containing 480K image-translation pairs via crowd-sourcing and manual post-editing from real-world images in the e-commerce domain. Experiments on the curated ECOIT benchmark dataset demonstrate that PEIT substantially outperforms both cascaded image translation systems (OCR+MT) and previous strong end-to-end image translation model, with fewer parameters and faster decoding speed.

## introduction image translation (it), transforming an image containing text in the source language to an image * equal contribution. † corresponding author. as a cross-modal task, a major challenge of it is the representation discrepancy across the textual and visual modality. the text contained in an image is in its visual modality, unlike the text input for text-only machine translation. its meaning also correlates with the visual context in the image. such visual modality and text-vision correlation make it difficult for it models to capture the meaning of the text in the context of the image and hence deteriorate translation quality. previous efforts to e2e it, e.g., the method presented in @xcite , use resnet as the visual encoder to encode the latent semantic representations of images, and a pre-trained text-only decoder to generate target translations. such framework may not be able to sufficiently leverage crossmodal knowledge as it only uses convolutional neural networks (cnn) to model both image and visual text contained in the image and do not explicitly deal with the modality gap issue. to mitigate this problem, we propose peit that bridges the modality gap with pre-trained models for end-to-end it. the peit is composed of four essential components: a visual encoder, a shared encoder-decoder backbone network, a vision-text representation aligner and a cross-modal regularizer. we use a two-stage pre-training strategy to pre-train peit. in the first pre-training stage, we pre-train an nmt model on a huge amount of mt training data, which is used to initialize the shared encoder-decoder network, transfer knowledge to e2e it and unify cross-modal representations. following previous e2e it practice @xcite , we also pretrain the shared encoder-decoder backbone network on a synthesized dataset with rendered images containing sentences from the mt training data after the network has been initialized by the pre-trained mt model. during the second pre-training stage, the aligner equipped with the shared encoder is jointly trained to align vision-text input representations in the same semantic space via contrastive learning. the regularizer stacked over the shared decoder is also optimized to force the decoder to generate the same translation for the same input in different modalities. to the best of our knowledge, there is no public dataset available for it task. we hence curate a large-scale image translation dataset in ecommerce domain, ecoit, containing product images automatically crawled from a chinese ecommerce website foot_2 paired with post-edited target translations (480k sentences with 3.64m source tokens). we fine-tune peit on the constructed ecoit to perform the it task. the main contributions of this work are summarized as follows: ## related work recent years have witnessed increasing attention on multimodal machine translation (mmt) that translates a source sentence into the target language accompanied with an additional modality @xcite . given the additional modality and its relation to the source sentence, mmt can be roughly divided into image-guided translation @xcite , video-guided translation @xcite , speech translation @xcite , it @xcite . image-guided mmt aims to leverage visual context to aid textual machine translation @xcite . the significant difference between image-guided translation and image translation is that the latter embeds the source sentence in its visual modality in the image while the former has the image and the source sentence separated and the image is used to provide additional information for translating the source sentence. in contrast to image-guided translation, it has not yet been fully explored in the literature probably due the lack of publicly available datasets for it. both @xcite and @xcite propose end-to-end approaches to it. @xcite uses a convolutional encoder to encode the image and transformer decoder to generate target translation. the end-to-end it model is able to locate characters in image, performs implicit tokenization on the source text, and then extracts latent semantic representations from them. this model can extract the latent token representations of image and text, and map into a shared space to implement the e2e it. while they provide an initial definition of the it task, they neither consider the modality gap nor verify the effect of the proposed models on real-world images. for speech translation (st), recent efforts have shifted towards end-to-end speech-to-text translation that directly translates a speech in the source language into a text in the target language @xcite . this is because end-toend st is of less error propagation and low latency compared with traditional cascaded st (inaguma @xcite . however, e2e st suffers from the high cost of speech-to-text parallel data creation. pre-training and multitask learning strategies have been explored to mitigate this data scarcity issue @xcite . in addition, similar to e2e it, e2e st is also confronted with the cross-modality issue, which can be mitigated by sharing the same semantic space for audio and text representations @xcite . partially motivated by e2e st, we propose an endto-end framework for it from the perspectives of pre-training with data of the mt task, sharing parameters across modalities, knowledge transfer via multitask learning, attempting to address the data scarcity and modality gap issues in it. ## large-scale parallel image translation dataset: ecoit in order to facilitate the training and evaluation of e2e it and hence promote its research, we build a large-scale e-commerce parallel it dataset, ecoit, based on the taobao foot_3 e-commerce platform. the reason for building the dataset in the e-commerce domain is that product descriptions and advertising slogans are often contained in the images of products to attract shoppers and promote sales. in other words, e-commerce provides a huge amount of images containing text from a wide range of domains, which much fits into the motivation of it. to build this dataset, we first crawl ∼ 600,000 images that contain chinese texts. we then use an ocr detector foot_4 ,with a high accuracy of 90%, to automatically recognize the chinese texts in images. recognized texts are manually scrutinized: those with over 3 incorrectly recognized chinese characters are removed while those with less than 3 wrong characters are manually corrected. after this manual scrutinization, we have 479,490 image-sentence pairs. we automatically translate these chinese texts into english with google translate api. to guarantee translation quality, we hire crowd-sourced workers who are chinese-english bilingual speakers to manually post-edit english translations to ensure both flu-ency and adequacy. more than 80% of automatic english translations have been post-edited. 2,000 image-translation pairs are selected as the validation set while 1,020 pairs are selected as the test set. ## methodology this section starts with the task formulation of it, followed by an overview of the model architecture of peit and the two-stage pre-training strategy that leverages mt knowledge from both the encoder and decoder to reduce the modality gap. ## fine-tuning after the two-stage pre-training, we continue to fine-tune our peit on the curated image translation dataset ecoit so as to endow peit with the ability to translate real-world images containing text into the target language. for fine-tuning and inference, we only keep the visual encoder and the shared encoder-decoder backbone network, removing the pre-trained mt module together with the visiontext aligner and cross-modal regularizer. the kept components are fine-tuned with the cross-entropy loss l v on ecoit. ## experiments we conducted extensive experiments with a large mt training corpus, a synthesized dataset with rendered images based on the mt training corpus and the curated image translation data to examine the effectiveness of the proposed peit against previous end-to-end and cascaded baselines. ## conclusion in this paper, we have presented peit, an end-toend image translation framework that attempts to bridge the modality gap with pre-trained models, as well as ecoit, a large-scale high-quality chinese-english image translation benchmark dataset with real-world e-commerce images containing text, which facilitates future research on this emerging direction. peit, containing the vision-text representation aligner and cross-modal regularizer for modality bridging, is pre-trained in two stages and fine-tuned on the curated dataset. experiments and in-depth analyses demonstrate that peit is significantly better than both cascaded image translation systems and previous end-to-end image translation models.