X-LXMERT : P aint, C aption and A nswer Q uestions with M ulti- M odal T ransformers (2020)

Abstract: Mirroring the success of masked language models, vision-and-language counterparts like VILBERT, LXMERT and UNITER have achieved state of the art performance on a variety of multimodal discriminative tasks like visual question answering and visual grounding. Recent work has also successfully adapted such models towards the generative task of image captioning. This begs the question: Can these models go the other way and generate images from pieces of text? Our analysis of a popular representative from this model family – LXMERT – finds that it is unable to generate rich and semantically meaningful imagery with its current training setup. We introduce X-LXMERT, an extension to LXMERT with training refinements including: discretizing visual representations, using uniform masking with a large range of masking ratios and aligning the right pre-training datasets to the right objectives which enables it to paint. X-LXMERT’s image generation capabilities rival state of the art generative models while its question answering and captioning abilities remains comparable to LXMERT. Finally, we demonstrate the generality of these training refinements by adding image generation capabilities into UNITER to produce X-UNITER.

## introduction the past year has seen a spate of bert-style @xcite transformer-based architectures @xcite @xcite proposed for vision-and-language tasks. these models are typically pre-trained on large image captioning corpora, extending ideas from masked language modeling to mask both the image and text modalities and produce state of the art results on a variety of vision and language tasks including visual question answering, visual grounding and image retrieval. these impressive results as well as recent probing mechanisms @xcite suggest that these models are able to capture a variety of semantics in images including objects, attributes and their relationships and ground these in natural language. while these models have been extensively evaluated over several discriminative tasks, relatively little attention has been paid to their generative capabilities. bidirectional transformer models like bert which exploit context preceding and following the current token are not explicitly designed for generation. recent work for language-only transformers @xcite @xcite adapt these models towards this capability using sampling procedures. such techniques have also been adapted successfully for image captioning -inputting an image and sampling the textual side of the model to generate a relevant caption @xcite . this begs the question: can we go the other way and sample images from input pieces of text? i.e. do vision-and-language bert models know how to paint? in this work, we probe the ability of a powerful and popular representative from this family of models -lxmert @xcite , to produce high fidelity and semantically meaningful images conditioned on captions. interestingly, our analysis leads us to the conclusion that lxmert in its current form does not possess the ability to paintit produces images that have little resemblance to natural images. this is a somewhat surprising finding given lxmert's masked training objectives for both modalities and its impressive performance on tasks that seemingly require a similar skill set. we find that this is largely due to the regression training objective used by this family of models to predict masked features on the visual side. this is in contrast with the textual side, where they predict masked tokens within a large discrete vocabulary using a classification objective. regressing features in high dimensional spaces is challenging to optimize and introduces noise at inference. this gets compounded when using iterative sampling procedures to predict the entire set of visual features. a downstream image generator consuming these predictions isn't able to recover from this noise even when fine-tuned on lxmert's predictions. we introduce x-lxmert that builds upon lxmert and enables it to effectively perform discriminative as well as generative tasks. our key refinements include: (a) simplifying the visual inputs to use grid features instead of object detection bounding boxes, (b) discretizing visual representations, (c) using uniform masking with a large range of masking ratios to enable the model to predict the entire set of visual clusters at inference time and (d) aligning the right pre-training datasets to the right objectives. when coupled with our proposed image generator, x-lxmert is able to generate rich imagery that is semantically consistent with the input captions. importantly, x-lxmert's image generation capabilities rival state-of-the-art image generation models (designed only for generation), while its question answering capabilities show little degradation compared to lxmert. these refinements are not lxmert-specific. they are designed to be easily applicable to a wide variety of multimodal bert models. we find that uniter, a single stream model for vision-andlanguage tasks, produces very poor images when coupled with a generator, but with our extensions, the resulting x-uniter produces images of a similar quality to x-lxmert. in summary, we present x-lxmert, a unified multimodal transformer model that can answer questions, and also generate captions and images. our extensions to enable these capabilities are not tied to lxmert's underlying architecture. we expect that the entire family of multimodal bert models can be enhanced with image generative capabilities using our introduced strategy. ## related works visual-language transformer models recent multi-modal pre-training models show significant improvements on a wide range of downstream tasks, including discriminiative (eg., visual question answering) and generation task (eg. image captioning @xcite ). some methods use a single transformer architecture to jointly encode text and image @xcite @xcite @xcite @xcite , while others use two-stream architectures @xcite @xcite . these models typically consume object detection features. we probe this family of models at the task of image generation and present extensions that enable them to reliably generate images. sequence generation with undirectional transformer when generating sequences with conventional transformer language models, it is natural to sample tokens from left to right. however, since undirectional transformers (eg. bert) are not trained with a specific generation order, a line of works has investigated different strategies for sequence generation with undirected models. wang and cho (2019 text-to-image synthesis synthesizing images from text descriptions continues to be challenging. since the pioneering work of @xcite , many methods have adopted gans @xcite to generate high-fidelity images. @xcite generate images that maximizes activation of a pretrained captioning model. recent works @xcite @xcite use multi-stage generation, where low-resolution images are initially sampled, then gradually upsampled and improved in later stages. these models are specialized toward image generation, whereas our model can not just generate images, but also answer questions and generate captions. also, our design is modular in nature. while we use a compact image generator with x-lxmert, one can also replace it with either of the aforementioned model architectures. there is another line of works predicting object layouts from text and generating image based on the layouts @xcite . these models use bounding box annotations to train layout predictors, while x-lxmert implicitly learns the layouts only from text and image alignments. ## probing lxmert's ability to paint in order to probe lxmert's ability to paint, we first modify its input image representation to a grid based feature set (sec. 4.1) and then pass these to an image generator (sec. 4.2). ## x-lxmert in this section, we present x-lxmert foot_0 that extends lxmert, enabling it to paint, while still maintaining a high performance on discriminative tasks. x-lxmert has three key refinements that enable it to paint (sec. 5.1): discretizing visual representations, using uniform masking with a large range of masking ratios, and aligning the right pretraining datasets to the right objectives. we then leverage gibbs sampling to generate visual features given textual input (sec. 5.2). ## experimental setup in this section we present experimental setups to evaluate image generation, visual question answering and visual reasoning. ## experimental results we now present a comparison of x-lxmertwith several baselines on the generative and discriminative tasks, along with ablation studies and qualitative results. we also show the generality of our techniques via extending uniter to create x-uniter. ## conclusion we develop a probing mechanism and find that lxmert, a powerful vision-and-language transformer model, is not able to generate meaningful images conditioned on text. we present x-lxmert, a unified model for image generation, captioning, qa and visual reasoning, and show that our extensions can easily be applied to other vision-and-language transformer models.