VD-BERT : A U nified V ision and D ialog T ransformer with BERT (2020)

Abstract: Visual dialog is a challenging vision-language task, where a dialog agent needs to answer a series of questions through reasoning on the image content and dialog history. Prior work has mostly focused on various attention mechanisms to model such intricate interactions. By contrast, in this work, we propose VD-BERT, a simple yet effective framework of unified vision-dialog Transformer that leverages the pretrained BERT language models for Visual Dialog tasks. The model is unified in that (1) it captures all the interactions between the image and the multi-turn dialog using a single-stream Transformer encoder, and (2) it supports both answer ranking and answer generation seamlessly through the same architecture. More crucially, we adapt BERT for the effective fusion of vision and dialog contents via visually grounded training. Without the need of pretraining on external vision-language data, our model yields new state of the art, achieving the top position in both single-model and ensemble settings (74.54 and 75.35 NDCG scores) on the visual dialog leaderboard. Our code and pretrained models are released at https://github.com/salesforce/VD-BERT .

## introduction visual dialog (or visdial) aims to build an ai agent that can answer a human's questions about visual content in a natural conversational setting @xcite . unlike the traditional single-turn visual question answering (vqa) @xcite , the agent in visdial requires to answer questions through multiple rounds of interactions together with visual content understanding. the primary research direction in visdial has been mostly focusing on developing various attention mechanisms @xcite for a better fusion of vision and dialog contents. compared to vqa that predicts an answer based only on the question about the image (figure 1 (a)), visdial needs to additionally consider the dialog history. typically, most of previous work @xcite @xcite uses the question as a query to attend to relevant image regions and dialog history, where their interactions are usually exploited to obtain better visual-historical cues for predicting the answer. in other words, the attention flow in these methods is unidirectional -from question to the other components (figure 1(b) ). by contrast, in this work, we allow for bidirectional attention flow between all the entities using a unified transformer @xcite encoder, as shown in figure 1(c ). in this way, all the entities simultaneously play the role of an "information seeker" (query) and an "information provider" (key-value), thereby fully unleashing the potential of attention similar to @xcite . we employ the transformer as the encoding backbone due to its powerful representation learning capability exhibited in pretrained language models like bert @xcite . inspired by its recent success in vision-language pretraining, we further extend bert to achieve simple yet effective fusion of vision and dialog contents in visdial tasks. recently several emerging works have attempted to adapt bert for multimodal tasks @xcite @xcite . they often use self-supervised objectives to pretrain bert-like models on large-scale external vision-language data and then fine-tune on downstream tasks. this has led to compelling results in tasks such as vqa, image captioning, image retrieval @xcite , and visual reasoning @xcite . however, it is still unclear how visual dialog may benefit from such vision-language pretraining due to its unique multiturn conversational structure. specifically, each image in the visdial dataset is associated with up to 10 dialog turns, which contain much longer contexts than either vqa or image captioning. in this paper, we present vd-bert, a novel unified vision-dialog transformer framework for vis-dial tasks. specifically, we first encode the image into a series of detected objects and feed them into a transformer encoder together with the image caption and multi-turn dialog. we initialize the encoder with bert for better leveraging the pretrained language representations. to effectively fuse features from the two modalities, we make use of two visually grounded training objectives -masked language modeling (mlm) and next sentence prediction (nsp). different from the original mlm and nsp in bert, we additionally take the visual information into account when predicting the masked tokens or the next answer. visdial models have been trained in one of two settings: discriminative or generative. in the discriminative setting, the model ranks a pool of answer candidates, whereas the generative setting additionally allows the model to generate the answers. instead of employing two types of decoders like prior work, we rely on a unified transformer architecture with two different self-attention masks @xcite to seamlessly support both settings. during inference, our vd-bert either ranks the answer candidates according to their nsp scores or generates the answer sequence by recursively applying the mlm operations. we further fine-tune our model on dense annotations that specify the relevance score for each answer candidate with a ranking optimization module. in summary, we make the following contributions: ## related work visual dialog. the visual dialog task has been recently proposed by @xcite , where a dialog agent needs to answer a series of questions grounded by an image. it is one of the most challenging vision-language tasks that require not only to understand the image content according to texts, but also to reason through the dialog history. previous work @xcite @xcite @xcite @xcite focuses on developing a variety of attention mechanisms to model the interactions among entities including image, question, and dialog history. for example, @xcite proposed dan, a dual attention module to first refer to relevant contexts in the dialog history, and then find indicative image regions. redan, proposed by @xcite , further explores the interactions between image and dialog history via multi-step reasoning. different from them, we rely on the selfattention mechanism within a single-stream transformer encoder to capture such interactions in a unified manner and derive a "holistic" contextualized representation for all the entities. similar to this, @xcite proposed fga, a general factor graph attention that can model interactions between any two entities but in a pairwise manner. there are recent works @xcite also applying the transformer to model the interactions among many entities. however, their models neglect the important early interaction of the answer entity and cannot naturally leverage the pretrained language representations from bert like ours. [cls] 𝑝# 𝑜@xmath0 … segment image vd-bert (disc/gen) ## the vd-bert model we first formally describe the visual dialog task. given a question q t grounded on an image i at t-th turn, as well as its dialog history formulated as h @xmath1, (q 1 , a 1 ), ..., (q t-1 , a t-1 )} (where c denotes the image caption), the agent is asked to predict its answer a t by ranking a list of 100 answer candidates { â1 t , â2 t , ..., â100 t }. in general, there are two types of decoder to predict the answer: a discriminative decoder that ranks the answer candidates and is trained with a cross entropy loss, or a generative decoder that synthesizes an answer and is trained with a maximum log-likelihood loss. figure 2 shows the overview of our approach. first, we employ a unified vision-dialog transformer to encode both the image and dialog history, where we append an answer candidate ât in the input to model their interactions in an early fusion manner ( §3.1). next, we adopt visually grounded mlm and nsp objectives to train the model for effective vision and dialog fusion using two types of self-attention masks -bidirectional and seq2seq. this allows our unified model to work in both discriminative and generative settings ( §3.2). lastly, we devise a ranking optimization module to further fine-tune on the dense annotations ( §3.3). ## experimental setup datasets. we evaluate our model on the visdial v0.9 and v1.0 datasets @xcite . specifically, v0.9 contains a training set of 82,783 images and a validation set of 40, 504 images. the v1.0 dataset combines the training and validation sets of v0.9 into one training set and adds another 2,064 images for validation and 8, 000 images for testing (hosted blindly in the task organizers' server). each image is associated with one caption and 10 question-answer pairs. for each question, it is paired with a list of 100 answer candidates, one of which is regarded as the correct answer. for the v1.0 validation split and a part of v1.0 train split (2,000 images), extra dense annotations for the answer candidates are provided to make the evaluation more reasonable. the dense annotation specifies a relevance score for each answer candidate based on the fact that some candidates with similar semantics to the ground truth answer can also be considered as correct or partially correct, e.g., "brown and tan" and "brown" in figure 2 . evaluation metric. following @xcite , we evaluate our model using the ranking metrics like recall@k (@xmath2, 5, 10}), mean reciprocal rank (mrr), and mean rank, where only one answer is considered as correct. since the 2018 visdial challenge (after the acquisition of dense annotations), ndcg metric that considers the relevance degree of each answer candidate, has been adopted as the main metric to determine the winner. configurations. we use bert base as the backbone, which consists of 12 transformer blocks, each with 12 attention heads and a hidden state dimensions of 768. we keep the max input sequence length (including 36 visual objects) to 250. we use adam (kingma and ba, 2015) with an initial learning rate of 3e -5 and a batch size of 32 to train our model. a linear learning rate decay schedule with a warmup of 0.1 is employed. we first train vd-bert for 30 epochs on a cluster of 4 v100 gpus with 16g memory using mlm and nsp losses (with equal coefficients). here we only utilize one previous dialog turn for training efficiency. for instances where the appended answer candidate is incorrect, we do not conduct mlm on the answer sequence to reduce the noise introduced by the negative samples. after that, we train for another 10 epochs with full dialog history using either nsp in the discriminative setting or mlm on the answer sequence in the generative setting. for dense annotation fine-tuning in the discriminative setting, we train with the listnet loss for 5 epochs. ## results and analysis we first compare vd-bert with state-of-the-art models on visdial datasets ( §5.1). then we conduct ablation studies to examine various aspects of our model ( §5.2), followed by an in-depth analysis of fine-tuning on dense annotations ( §5.3). lastly, we interpret how it attains the effective fusion of vision and dialog via attention visualization ( §5.4). ## conclusion we have presented vd-bert, a unified visiondialog transformer model that exploits the pretrained bert language models for visual dialog. vd-bert is capable of modeling all the interactions between an image and a multi-turn dialog within a single-stream transformer encoder and enables the effective fusion of features from both modalities via simple visually grounded training. besides, it can either rank or generate answers seamlessly. without pretraining on external visionlanguage datasets, our model establishes new stateof-the-art performance in the discriminative setting and shows promising results in the generative setting on the visual dialog benchmarks. (a) it learns three apparent attention patterns for the example in figure 5 : attentions that a token puts to its previous token, to itself, and to the next token a brown building stands at the corner of a street on a sunny day q1: is there any people in the photo? a1: yes far away q2: are there any streets visible in the photo? a2: yes q3: are there cars parked along the streets? a3: no q4: what kind of buildings appear in the photo? a4: older architectural building like a library q5: is there any signs visible in the photo? a5: there is a no entry sign q6: are there traffic lights visible? a6: no q7: can you see the sidewalk? a7: yes q8: are the sidewalks clean? a8: immaculate q9: are there any trees planted near the building? a9: there is a tree next to the building q10: is the tree short or tall? a10: tall q1: is there any people in the photo? a1: yes q2: are there any streets visible in the photo? a2: no on the street q3: are there cars parked along the streets? a3: no q4: what kind of buildings appear in the photo? a4: yes shops q5: is there any signs visible in the photo? a5: yes q6: are there traffic lights visible? a6: 1 light pole q7: can you see the sidewalk? a7: yes q8: are the sidewalks clean? a8: yes very clean q9: are there any trees planted near the building? a9: yes q10: is the tree short or tall? a10: tall q1: is there any people in the photo? a1: yes q2: are there any streets visible in the photo? a2: yes q3: are there cars parked along the streets? a3: no q4: what kind of buildings appear in the photo? a4: older architectural building like a library q5: is there any signs visible in the photo? a5: yes q6: are there traffic lights visible? a6: no q7: can you see the sidewalk? a7: yes q8: are the sidewalks clean? a8: yes q9: are there any trees planted near the building? a9: there is a tree next to the building q10: is the tree short or tall? a10: tall q1: is there any people in the photo? a1: yes q2: are there any streets visible in the photo? a2: yes q3: are there cars parked along the streets? a3: no q4: what kind of buildings appear in the photo? a4: not sure q5: is there any signs visible in the photo? a5: yes q6: are there traffic lights visible? a6: no q7: can you see the sidewalk? a7: yes q8: are the sidewalks clean? a8: yes, they are clean q9: are there any trees planted near the building? a9: no there isn't q10: is the tree short or tall? a10: tall man pointing at tv display in a store q1: is the picture in color? a1: yes q2: how many people are in the picture? a2: i see 2 people q3: how many tv's are in the picture? a3: 18 screens on the display but there are some in boxes as well q4: is 1 of the people a worker in the store? a4: no it doesn't look like it q5: is there an image on the tv that he is pointing at? a5: yes it is an cartoon q6: is the man standing or sitting? a6: standing q7: what is the brand of tv? a7: i am unable to tell q8: can you tell what store it is? a8: no q9: are there any other products in view? a9: i just see tv's and boxes q10: what color is the floor? a10: blue q1: is the picture in color? a1: yes q2: how many people are in the picture? a2: 2 q3: how many tv's are in the picture? a3: 2 q4: is 1 of the people a worker in the store? a4: yes, but they are taking a break q5: is there an image on the tv that he is pointing at? a5: yes it is an cartoon q6: is the man standing or sitting? a6: standing q7: what is the brand of tv? a7: it's in japanese q8: can you tell what store it is? a8: yes it is a list of rules q9: are there any other products in view? a9: there is papers all over the desk so i would say yes q10: what color is the floor? a10: blue q1: is the picture in color? a1: yes q2: how many people are in the picture? a2: 2 q3: how many tv's are in the picture? a3: 18 screens on the display but there are some in boxes as well q4: is 1 of the people a worker in the store? a4: there are no visible workers q5: is there an image on the tv that he is pointing at? a5: yes it is an cartoon q6: is the man standing or sitting