Using Visual Feature Space as a Pivot Across Languages (2020)

Abstract: Our work aims to leverage visual feature space to pass information across languages. We show that models trained to generate textual captions in more than one language conditioned on an input image can leverage their jointly trained feature space during inference to pivot across languages. We particularly demonstrate improved quality on a caption generated from an input image, by leveraging a caption in a second language. More importantly, we demonstrate that even without conditioning on any visual input, the model demonstrates to have learned implicitly to perform to some extent machine translation from one language to another in their shared visual feature space. We show results in German-English, and Japanese-English language pairs that pave the way for using the visual world to learn a common representation for language.

## introduction there has been great interest in learning visual representations from images paired with natural language annotations. while tasks such as image caption generation e.g. @xcite have focused mostly on english text, there is a growing body of work extending to a larger set of languages @xcite @xcite . images annotated in multiple languages offer the possibility of studying grounded models of languages along with their commonalities and intrinsics in direct connection with the visual world. we focus in the multilingual image description generation setting where we train an image encoder with soft-attention @xcite and multiple text decoders for each target language. then, we demonstrate that information from one language can be transferred to another language using energy based inference @xcite in an iterative fashion by leveraging the backpropagation algorithm at test time. effectively, we demonstrate that the common visual feature space used to generate text in the target languages also learns implicitly alignments between them and thus acts as its own form of "visual language". figure 1 shows some example images and textual descriptions in german and english, as well as a general outline of our approach. we demonstrate our findings by ( 1 ) showing that a textual description in a second language helps improving generated image description quality in a target language, and (2) showing how to use the visual feature space in an image encoder to translate sentences among target languages even in the absence of visual input. stated otherwise, our claim is that multi-lingual image captioning models can act as incidental machine translators. more broadly, our work explores the possibility of using visually grounded representation learning as a unifying medium across languages, where a single model is used for learning mappings across an exhaustive number of language pairs among target languages. we demonstrate our approach on two datasets of images annotated with german, english, and japanese, english respectively. ## background our work is different from work in both general neural machine translation (nmt) @xcite @xcite , and multimodal machine translation (mmt) @xcite @xcite in that we do not use parallel corpora across languages. this distinction is important and perhaps confusing as we rely on the multi30k dataset for which several versions and tasks exist @xcite . the first task, task 1, is perhaps the most popular, containing parallel text among languages (german, english, french and czech) describing 30,000 images from the flickr30k dataset @xcite with a single caption in each language. this task has often been used also as a pure machine translation benchmark by discarding the image information. the second task, task 2, is the one that concerns our work and is one of the tasks we leverage for training, which is the mutilingual image description generation task, where each of the 30,000 images is annotated with 5 independent (unpaired) captions in german and english. using visual features as "pivoting" variables is related to using conditional latent variables to iteratively perform inference using backpropagation. a version of this idea was perhaps first mentioned in @xcite as noted by @xcite . besides work on generative adversarial networks @xcite , there are only a few works since then that have independently proposed to use iterative inference with backpropagation including @xcite ; domke (2013); @xcite . we particularly adopt the single layer version of the most recenlty proposed feedback propagation approach of @xcite as it was more directly applied to convolutional neural networks for visual recognition. unlike this previous work, we are the first to show that feedback propagation can leverage its latent space to use interactions among target variables even in the absence of any visual input at test time. ## method as mentioned earlier, our base model consists of the image captioning model with "soft" attention proposed by @xcite but trained with independent textual decoders for each target language. in this model, the image encoder consists of a convolutional neural network and the textual decoders consist of recurrent neural networks with long short term memory (lstm) units. the output soft spatial attention vector computed from the input image is used as input for the decoders to generate captions in each target language. let the input image be i, and let us consider the bilingual case of language a and language b where the targets are text sequences t a and t b respectively. the model can then be expressed as: visual input + second language in order to use the latent feature space to predict t a conditioned on t b and i, we estimate a pivoting variable ẑ by iteratively minimizing using backpropagation the following: ## related work our work is closely related to the problem of lexicon induction from images which has been used to address the issue when paired texts are not available for machine translation. works that have leveraged visual features to build such lexicon include @xcite ; @xcite ; @xcite . other works with similar goals include @xcite where visual features are used to assemble a weakly supervised set of text pairs, @xcite where the objective is to leverage both image-caption pairs and multilingual parallel corpora, and @xcite where the images are used as pivot between languages to learn multimodal multilingual common representations. our work leverages only unpaired data and does not aim to train a machine translation model or obtain multimodal representations explicitly. related to our goals is also work image: a man in a white shirt is jumping in the air. ## conclusions we show that visual feature space can be used as a pivot for transferring information across languages. we demonstrated this by showing how having access to captions in a second language can improve the generated caption quality in a target language. moreover, we present the key result that we can perform arbitrary mappings among target languages in an image conditioned model, even when removing the requirement of visual input, essentially demonstrating the model learns mappings across languages similar to machine translation models.