Towards End-to-End In-Image Neural Machine Translation (2020)

Abstract: In this paper, we offer a preliminary investigation into the task of in-image machine translation: transforming an image containing text in one language into an image containing the same text in another language. We propose an end-to-end neural model for this task inspired by recent approaches to neural machine translation, and demonstrate promising initial results based purely on pixel-level supervision. We then offer a quantitative and qualitative evaluation of our system outputs and discuss some common failure modes. Finally, we conclude with directions for future work.

## introduction end-to-end neural models have emerged in recent years as the dominant approach to a wide variety of sequence generation tasks in natural language processing, including speech recognition, machine translation, and dialog generation, among many others. while highly accurate, these models typically operate by outputting tokens from a predetermined symbolic vocabulary, and require integration into larger pipelines for use in user-facing applications such as voice assistants where neither the input nor output modality is text. in the speech domain, neural methods have recently been successfully applied to end-to-end speech translation @xcite @xcite , in which the goal is to translate directly from speech in one language to speech in another language. we propose to study the analogous problem of in-image machine translation. specifically, an image containing text in one language is to be transformed into an image containing the same text in another language, removing the dependency of any predetermined symbolic vocabulary or processing. ## data generation to our knowledge, there are no publicly available datasets for the task of in-image machine translation task. since collecting aligned natural data for in-image translation would be a difficult and costly process, a more practical approach is to bootstrap by generating pairs of rendered images containing sentences from the wmt 2014 german-english parallel corpus. the dataset consists of 4.5m german-english parallel sentence pairs. we use newstest-2013 as a development set. for each sentence pair, we create a minimal web page for the source and target, then render each using headless chrome foot_1 to obtain a pair of images. the text is displayed in a black 16-pixel sans-serif font on a white background inside of a fixed-size 1024x32-pixel frame. for simplicity, all sentences are vertically centered and left-aligned without any line-wrapping. the consistent position and styling of the text in our synthetic dataset represents an ideal scenario for in-image translation, serving as a good test-bed for initial attempts. later, one could generalize to more realistic settings by varying the location, size, typeface, and perspective of the text and by using non-uniform backgrounds. ## model our goal is to build a neural model for the inimage translation task that can be trained end-toend on example image pairs (x * , y * ) of height and width h and w using only pixel-level supervision. we evaluate two approaches for this task: convolutional encoder-decoder model and full model that combines soft versions of the traditional pipeline in order to arrive at a modular yet fully differentiable solution. ## results in figure 3 we share various source images and corresponding predicted target images generated by the full model. despite the very high dimensionality of the output, the model occasionally succeeds at predicting the full output translation as shown in figure 3a . additionally, figure 3b shows examples where model makes a minor typo in the earlier proposals and is able to correct itself further during generation process. figure 3c and figure 3d show two major failure modes of our model where it is either able to generate the first part of the sentence and fails to generate the rest or completely fails at generating the image. to better understand the source of errors made by the full model we visualize the probabilities of pixels in figure 4 . we can see that model is uncertain between producing word however and nevertheless which leads to artifacts when taking the arg max value of each pixel during decoding similar to ones displayed in figure 3d . on model train set dev set nll conv baseline 0.120 0.117 full model 0.028 0.025 bleu conv baseline -0.5 full model -7.7 ## conclusion in this paper, we introduce the task of in-image neural machine translation and develop an end-toend model that shows promising results on learning to translate text through purely pixel-level supervision. by doing so, we demonstrate a viable first step towards applying such models in more natural settings, such as translating texts, menus, or street signs within real-world images. future work should explore models that do not rely on off-the-shelf text tokenizers to decompose the very hard image generation problem into sequence of simpler image predictions. we hypothesize that discrete latent variables @xcite are best suited to implicitly segment the image and capture the sequential nature of this task.