Learning to Ground Visual Objects for Visual Dialog (2021)

Abstract: Visual dialog is challenging since it needs to answer a series of coherent questions based on understanding the visual environment. How to ground related visual objects is one of the key problems. Previous studies utilize the question and history to attend to the image and achieve satisfactory performance, while these methods are not sufficient to locate related visual objects without any guidance. The inappropriate grounding of visual objects prohibits the performance of visual dialog models. In this paper, we propose a novel approach to Learn to Ground visual objects for visual dialog, which employs a novel visual objects grounding mechanism where both prior and posterior distributions over visual objects are used to facilitate visual objects grounding. Specifically, a posterior distribution over visual objects is inferred from both context (history and questions) and answers, and it ensures the appropriate grounding of visual objects during the training process. Meanwhile, a prior distribution, which is inferred from context only, is used to approximate the posterior distribution so that appropriate visual objects can be grounding even without answers during the inference process. Experimental results on the VisDial v0.9 and v1.0 datasets demonstrate that our approach improves the previous strong models in both generative and discriminative settings by a significant margin.

## introduction with the development of deep learning, various vision-language tasks have been introduced and attracted widespread attention, such as image captioning @xcite @xcite @xcite , visual question answering @xcite @xcite @xcite and visual dialog @xcite @xcite h0: a group of women walk down a road h1: what age are the women ? they are teenagers h2: are they all the same race ? most of them look to be q3: any other people ? ) @xcite @xcite . specifically, visual dialog, which aims to hold a meaningful conversation @xcite with a human about a given image, is a challenging task that requires models to locate related visual objects in an image and answer the current question based on the history and the located visual objects. in order to answer the question correctly, we need to accurately locate the question-related visual objects. most existing methods utilize kinds of attention mechanism @xcite @xcite @xcite to capture the target visual objects. redan @xcite and dmam @xcite use multi-step reasoning based on dual attention to iteratively update related visual objects. dan @xcite , mcan @xcite and @xcite utilize multi-head attention mechanisms to manage multimodal intersection and obtain weight distributions. moreover, there are some approaches @xcite @xcite @xcite using graph-based structures to capture related visual objects. fga @xcite realizes a factor graph attention mechanism, which constructs the graph over all the multi-modal features and estimates their interactions to ground visual objects. cag @xcite focuses on an iterative question-conditioned context-aware graph to locate related visual objects. however, the methods mentioned above obtain the prior distribution of visual objects through various interactions of questions, history and images, and finally use the prior distribution to obtain the final representation of the image. the prior distribution of visual objects is not enough to ground accurate visual objects, thus obtaining the wrong representation of the image. in this paper, we propose a method to learn to ground visual objects in visual dialog. specifically, we obtain the posterior distribution over visual objects by utilizing contexts and answers, while the prior distribution works without knowing answers in advance. then we minimize the distance between the two distributions. during the training process, our model is trained to minimize the kl divergence between the prior distribution and the posterior distribution so that our model can approximate the posterior distribution accurately using the prior distribution. then, during the inference process, the model grounds visual objects merely based on the prior distribution (i.e., without any posterior information). we show that through this process, the model can effectively learn to ground visual objects accurately and give informative and accurate responses by utilizing appropriate visual objects. we test the effectiveness of our proposed model on two large-scale datasets: visdial v0.9 and v1.0 @xcite . the contributions of this work are summarized as follows: context encoder visual encoder history question answer answer encoder prior module posterior module visual object manager decoder 𝑥 " 𝑥 " 𝑣 $ 𝑣 % 𝑣 & 𝑣 ' … 𝑦 " 𝑣 * figure 2: architecture overview 2 methodology following das et al. (2017), a visual dialog agent is given three inputs, i.e., an image i, history (the caption and question-answer pairs) till round t -1: ## application of our approach we take the strong baseline ltmi (nguyen et al., 2020) as a base model to introduce our approach, which mainly consists of the following components: context encoder and answer encoder: we use two bi-directional lstm encoders to extract token-level representations @xmath0. we use another bi-directional lstm encoder to extract sentence-level representations @xmath1. λ is the length of questions and answers with paddings, t is the turn of dialog and d q is the dimension. q and h are fused into a context representation x with multi-head attention @xcite . visual encoder: similar to @xcite , we extract the image features by using a pretrained faster rcnn @xcite . we select µ object proposals for each image, where each object proposal is represented by a 2048-dimension feature vector. we transform the obtained visual region features by a multi-layer perceptron and obtain the image features @xmath2. prior module: we use multi-head attention @xcite as f cv (•, •) to manage the multi-modal interaction. a cross-attention layer is firstly applied to outputs of the texutal and visual encoders: posterior module: we simply utilize the add operation as f cy (•, •) to manage the interaction of x and y: we replace x in eq.( 6 ) -eq.( 8 ) with x y and thus obtain the posterior distribution g and v post generative and discriminative decoder: we utilize another lstm as our discriminative and generative decoders following the previous studies @xcite . receiving the representation of context, images and the candidate answers, the two decoders compute the score of each candidate answer in different ways. the objective function of the base model is to minimize the negative log-likelihood l g of answer generated for the generative decoder or the crossentropy loss l d for the discriminative decoder. we utilize the kullback-leibler (kl) divergence loss to narrow the gap. the objective functions of the student are as follows: @xcite 52.59 42.29 62.85 68.88 17.06 51.86 47.99 38.18 57.54 64.32 18.60 hciae @xcite 53.86 44.06 63.55 69.24 16.01 59.70 49.07 39.72 58.23 64.73 18.43 corefnmn @xcite datasets and implementation details. we conduct experiments on the visdial v0.9 and v1.0 datasets @xcite to verify our approach. visdial v0.9 contains 83k dialogs on cocotrain @xcite and 40k dialogs on cocoval images as the test set, for a total of 1.23m dialog question-answer pairs. visdial v1.0 dataset is an extension of visdial v0.9 dataset with additional 10k coco-like images. visdial v1.0 dataset contains 123k, 2k, and 8k images as train, validation, and test splits, respectively. to represent image regions, we use faster r-cnn @xcite with resnet-101 @xcite finetuned on the visual genome dataset @xcite , thus obtaining a 2048-dimension feature vector for each region. following (nguyen et al., 2020), we detect µ = 100 objects from each image. our model is implemented based on pytorch @xcite . in experiments, we use adam (kingma and ba, 2014) optimizer for training, with the mini-batch size as 32. for the choice of the learning rate, we employ the warm-up strategy @xcite . specifically, we begin with a learning rate of 0.001, the learning rate is decreased by 1/4 for every 2 epochs up to 20 epochs. we use 4 titan-xp gpu for training. we spend about 4 hours / 1 epoch for the discriminative setting and 1 hour / 1 epoch for the generative setting. our student model is the same as ltmi, with the total parameters 42.20m. the λ sets to 1. automatic evaluation. we use a retrieval setting to evaluate individual responses at each round of a dialog, following @xcite . specifically, at test time, apart from the image, ground truth dialog history and the question, a list of 100candidate answers is also given. the model is evaluated on retrieval metrics: (1) rank of human response, (2) existence of the human response in top -k ranked responses, i.e., r@k (3) mean reciprocal rank (mrr) of the human response and (4) normalized discounted cumulative gain (ndcg) for visdial v1.0. human evaluation. we randomly extract 100 samples for human evaluation @xcite and then ask 3 human subjects to guess whether the last response in the dialog is human-generated or machine-generated. if at least 2 of them agree it is generated by a human, we think it is humangenerated (m1). we record the percentage of responses that are evaluated better than or equal to human responses (m2), according to the human subjects' evaluation. ## related work recent several works @xcite @xcite els. while visual dialog models focus on the intersection of questions, history and images. how to locate the related visual objects is quite important. mn @xcite , hciae @xcite , corefnmn @xcite , coatt @xcite , rva @xcite , dvan @xcite utilize kinds of attention mechanisms as the backbone to locate the related visual objects. visualbert @xcite and vdbert @xcite exploit large extra datasets to explore in visual dialog via pretraining language models. gnn-em @xcite , fga @xcite , dualvd @xcite , cag @xcite and kbgn @xcite utilize graph neural networks to obtain the representation of visual objects. however, most existing visual dialog models condition visual objects simply on history and questions, which we regard as a prior distribution over visual objects. in this paper, we propose an approach to learn to ground visual objects via bridge the gap between the prior distribution and the posterior distribution. ## conclusion in this paper, we propose a novel approach to learn to ground visual objects for visual dialog, which employs a novel visual objects grounding mechanism where both prior and posterior distributions over visual objects are used to facilitate visual objects grounding. experimental results on two largescale datasets show that our approach improves the previous models by a significant margin.