What BERT Sees: Cross-Modal Transfer for Visual Question Generation (2020)

Abstract: Pre-trained language models have recently contributed to significant advances in NLP tasks. Recently, multi-modal versions of BERT have been developed, using heavy pre-training relying on vast corpora of aligned textual and image data, primarily applied to classification tasks such as VQA. In this paper, we are interested in evaluating the visual capabilities of BERT out-of-the-box, by avoiding pre-training made on supplementary data. We choose to study Visual Question Generation, a task of great interest for grounded dialog, that enables to study the impact of each modality (as input can be visual and/or textual). Moreover, the generation aspect of the task requires an adaptation since BERT is primarily designed as an encoder. We introduce BERT-gen, a BERT-based architecture for text generation, able to leverage on either mono- or multi- modal representations. The results reported under different configurations indicate an innate capacity for BERT-gen to adapt to multi-modal data and text generation, even with few data available, avoiding expensive pre-training. The proposed model obtains substantial improvements over the state-of-the-art on two established VQG datasets.

## introduction in artificial intelligence, several works have investigated the longstanding research question of whether textual representations encode some sort of visual information. this has been done primarily for word embeddings, e.g, by applying them to zero-shot learning @xcite , or sentence embeddings, e.g, by applying them to image captioning @xcite . in this paper, we are interested in evaluating the visual capacities of pre-trained language models; in our case, bert @xcite . to do so, we choose the visual question generation (vqg) @xcite task, for the following reasons. first, from a practical standpoint, the vqg task has several applications: robots or ai assistants could ask questions rooted in multi-modal data (e.g. fusing conversational data with visual information from captors and cameras), in order to refine their interpretation of the situation they are presented with. second, it could also allow systems relying on knowledge-bases to gain visual common sense and deal with the human reporting bias @xcite , which states that the content of images and text are intrinsically different, since visual common sense is rarely explicitly stated in text. moreover, unlike image captioning (where the input is only visual) or vqa (where the input is visual and textual), vqg is a multi-modal task where input can be textual and/or visual: this is of particular interest to analyze the impact of each modality. finally, vqg relies on textual generation, which is challenging since bert is not primarily designed for generation. bert-based multi-modal language models have been proposed @xcite to tackle multi-modal tasks, relying on heavy pretraining and large corpora of aligned textual and visual data. from these works, it is left to explore whether the cross-modal capacities come from the pre-traning, or are to some extent already encoded in bert's representations. it has recently been shown that bert can generalize to another language, with great results, in a zero-shot manner @xcite , i.e. without supervision between languages. in preliminary experiments, we extended this work to another modality: we found out that, in vqg, without any supervision between the images and the questions, the cross-modal alignment was not successfully learnt. this discrepancy between multi-lingual and multi-modal results might find its root cause in the intrinsic semantic difference between textual and visual modalities @xcite . nonetheless, we hypothesize that bert contains some abstractions that generalize across modalities. if so, it may transfer knowledge to the visual modality even with few training data rather than expensive pre-training and complex architectures. thus, in contrast with latter multi-modal bert approaches, we explicitly avoid using the following complex mechanisms: (1) multi-modal supervision: we do not exploit explicit supervision between images and captions through a pre-training step; (2) image-specific losses: specific pre-training losses such as masked roi classification with linguistic clues @xcite or sentence-image prediction @xcite ; (3) non-linearities: we explore a scenario in which the only learnable parameters, for aligning image representations to bert, are those of a simple linear projection layer. furthermore, to the best of our knowledge, this paper is the first attempt to investigate multi-modal text generation using pre-trained language models. we introduce bert-gen, a text generator based on bert, that can be applied both in mono and multi-modal settings. we treat images similarly to text: while a sentence is seen as a sequence of (sub)word tokens, an image is seen as a sequence of objects associated to their corresponding positions (bounding boxes). we show how a simple linear mapping, projecting visual embeddings into the first layer, is enough to ground bert in the visual realm: text and image object representations are found to be effectively aligned, and the attention over words transfers to attention over the relevant objects in the image. our contributions can be summarized as follows: (1) we introduce bert-gen, a novel method for generating text using bert, that can be applied in both mono and multi-modal settings; (2) we report state-of-the art results on the vqg task; (3) we show that the semantic abstractions encoded in pretrained bert can generalize to another modality without pre-training of any sort; (4) we provide extensive ablations and qualitative analyses to interpret the behavior of bert-gen under different configurations (mono-or multi-modal). concerning models jointly treating information from images and text, visual features extracted from the image are used as "visual words", and a [sep] special token is employed to separate textual and visual tokens. in the literature, visual features are object features extracted with a faster r-cnn @xcite -with the notable exception of @xcite who used pooling layers from a cnn. a first body of work exploit single-stream transformers in which visual features are incorporated in a bert-like transformer: this is the case for visualbert @xcite and vl-bert @xcite . other works, such as vilbert @xcite and lxmert @xcite have investigated two-stream approaches: these models employ modality-specific encoders built on standard transformer blocks, which are then fused into a cross-modal encoder. interestingly, none of the aforementioned models have been used for generation tasks such as vqg, tackled in this work. ## model in vqg, the objective is to generate a relevant question from an image and/or its caption. the caption x txt is composed of m tokens txt 1 , ..., txt m ; these tokens can be words or subwords (smaller than word) units depending on the tokenization strategy used. as bert uses subword tokenization, throughout this paper we will refer to subwords as our tokenization units. the proposed model is illustrated in figure 1 . in 3.1, we detail how images are incorporated in the transformer framework. in 3.2, we present bert-gen, a novel approach to use bert for text generation. ## experimental protocol our main objective is to measure whether the textual knowledge encoded in pre-trained bert can be beneficial in a cross-modal task. thus, we define the three following experimental setups, which we refer to as step 1, 2, and 3: ## results finally, in our third step (image + caption), we obtain the highest scores, for all metrics. this indicates that the model is able to effectively leverage the combination of textual and visual inputs. indeed, complementary information from both modalities can be exploited by the self-attention mechanism, making visual and textual tokens interact to generate the output sequences. again, we additionally report the results obtained bypassing the intermediate steps 1 and 2: for the model denoted as step 3 (from scratch) (last row of in ## model discussion what does the model look at? to interpret the behavior of attention-based models, it is useful to