History for Visual Dialog: Do we really need it? (2020)

Abstract: Visual Dialogue involves “understanding” the dialogue history (what has been discussed previously) and the current question (what is asked), in addition to grounding information in the image, to accurately generate the correct response. In this paper, we show that co-attention models which explicitly encode dialoh history outperform models that don’t, achieving state-of-the-art performance (72 % NDCG on val set). However, we also expose shortcomings of the crowdsourcing dataset collection procedure, by showing that dialogue history is indeed only required for a small amount of the data, and that the current evaluation metric encourages generic replies. To that end, we propose a challenging subset (VisdialConv) of the VisdialVal set and the benchmark NDCG of 63%.

## introduction recently, there has been an increased interest in visual dialog, i.e. dialog-based interaction grounded in visual information @xcite @xcite @xcite @xcite . one of the most popular test beds is the visual dialog challenge (visdial) @xcite , which involves an agent answering questions related to an image, by selecting the answer from a list of possible candidate options. according to the authors, nearly all interactions (98%) contain dialog phenomena, such as co-reference, that can only be resolved using dialog history, which makes this a distinct task from previous visual question answering (vqa) challenges, e.g. @xcite . for example, in order to answer the question "about how many?" in figure 1 , we have to infer from what was previously said, that the conversation is about the skiers. @xcite as a ranking problem, where for the current question (blue), the agent ranks list of 100 candidate answers (yellow). relevance weights for each candidate were collected via crowd-sourcing. previous dialog history (red) together with the caption (green) forms the contextual information for the current turn. in the original paper, @xcite find that models which structurally encode dialog history, such as memory networks @xcite or hierarchical recurrent encoders @xcite improve performance. however, "naive" history modelling (in this case an encoder with late fusion/concatenation of current question, image and history encodings) might actually hurt performance. @xcite take this even further, claiming that visdial can be modeled without taking history or even visual information into account. @xcite rebutted by showing that both features are still needed to achieve state-of-the-art (sota) results and an appropriate evaluation procedure has to be used. in this paper, we show that competitive results on visdial can indeed be achieved by replicating the top performing model for vqa @xcite -and effectively treating visual dialog as multiple rounds of question-answering, without taking history into account. however, we also show that these results can be significantly improved by encoding dialog history, as well as by fine-tuning on a more meaningful retrieval metric. finally, we show that more sophisticated dialog encodings outperform naive fusion on a subset of the data which contains "true" dialog phenomena according to crowd-workers. in contrast to previous work on the visdial dataset, e.g. @xcite @xcite @xcite , we are the first to conduct a principled study of dialog history encodings. our contributions can thus be summarized as follows: ## visual dialog models in this section, we extend modular co-attention networks, which won the vqa challenge 2019 @xcite and adapt it to visual dialog. different from previous co-attention networks @xcite , mca networks use guided attention to model dense relations between the question and image regions for better visual grounding. in the following, we explore mca networks with different input encodings following a '[model]-[input]' convention to refer to our mca model variants; see figure 3 for an overview. whenever unspecified, images are represented as a bag of bottom-up features, i.e. object level representations (see section 3). ## implementation we use pytorch foot_0 @xcite for our experiments foot_1 . following @xcite , we use bottom-up features of 36 proposals from images using a faster-rcnn @xcite pre-trained on visual genome @xcite to get a bag of object-level 2048-d image representations. input question and candidate options are tokenized to a maximum length of 20 while the conversational history to 200. token embeddings in text are initialized with 300-d glove vectors @xcite and shared among all text-based encoders. the rnn encodings are implemented using lstms @xcite . we use the adam optimizer (kingma and ba, 2015) both for training and fine-tuning. more training details can be found in appendix a. ## results in the following, we report results on the visdial v1.0 val set, ( mca-i-h is our best performing model. it achieves state-of-the-art performance: it outperforms the rva baseline by almost 5 ndcg points on the val set and by over 7 points on the test set. on the official challenge test set, mca-i-h ranks 2 nd : it improves over 7 ndcg over the best single model but loses by 2 points against a 6-strong rva ensemble model (2019 winning entry). ## error analysis in the following, we perform a detailed error analysis, investigating the benefits of dialog history en-coding and the observed discrepancy between the ndcg results and the other retrieval based metrics. ## discussion and related work our results suggest that the visdial dataset only contains very limited examples which require dialog history. other visual dialog tasks, such as guesswhich? @xcite and guesswhat?! @xcite take place in a goal-oriented setting, which according to @xcite , will lead to data containing more natural dialog phenomena. however, there is very limited evidence that dialog history indeed matters for these tasks @xcite . as such, we see data collection to capture visual dialog phenomena as an open problem. nevertheless, our results also show that encoding dialog history still leads to improved results. this is in contrast with early findings that a) "naive" encoding will harm performance @xcite ; at least 13.3% of answers are non-commital (i cannot tell, not sure, i can't tell). see mca-i-hconcq in furthermore, we find that our model learns to provide generic answers by taking advantage of the ndcg evaluation metric. learning generic answers is a well-known problem for open-domain dialog systems, e.g. @xcite . while the dialog community approaches these phenomena by e.g. learning better models of coherence @xcite , we believe that evaluation metrics also need to be improved for this task, as widely discussed for other generation tasks, e.g. @xcite @xcite . as a first step, bert score @xcite could be explored to measure ground-truth similarity replacing the noisy ndcg annotations of semantic equivalence. ## conclusion and future work in sum, this paper shows that we can get sota performance on the visdial task by using transformerbased models with guided-attention @xcite , and by encoding dialog history and finetuning we can improve results even more. of course, we expect pre-trained visual bert models to show even more improvements on this task, e.g. vilbert @xcite , lxmert @xcite , uniter @xcite etc. however, we also show the limitations of this shared task in terms of dialog phenomena and evaluation metrics. we, thus, argue that progress needs to be carefully measured by posing the right task in terms of dataset and evaluation procedure.