Reasoning Over History: Context Aware Visual Dialog (2020)

Abstract: While neural models have been shown to exhibit strong performance on single-turn visual question answering (VQA) tasks, extending VQA to a multi-turn, conversational setting remains a challenge. One way to address this challenge is to augment existing strong neural VQA models with the mechanisms that allow them to retain information from previous dialog turns. One strong VQA model is the MAC network, which decomposes a task into a series of attention-based reasoning steps. However, since the MAC network is designed for single-turn question answering, it is not capable of referring to past dialog turns. More specifically, it struggles with tasks that require reasoning over the dialog history, particularly coreference resolution. We extend the MAC network architecture with Context-aware Attention and Memory (CAM), which attends over control states in past dialog turns to determine the necessary reasoning operations for the current question. MAC nets with CAM achieve up to 98.25% accuracy on the CLEVR-Dialog dataset, beating the existing state-of-the-art by 30% (absolute). Our error analysis indicates that with CAM, the model’s performance particularly improved on questions that required coreference resolution.

## introduction visual dialog is the task of answering a sequence of questions about a given image such that responding to any one question in the dialog requires context from the previous dialog history. the task of visual dialog @xcite brings together several fundamental building blocks of intelligent systems: visual understanding, natural language understanding and complex reasoning. the multimodal nature of visual dialog requires approaches that jointly model and reason over both * equal contribution modalities. furthermore, visual dialog necessitates the ability to resolve visual coreferences, which arise when two phrases in the dialog refer to the same object in the image. visual coreference resolution requires both an ability to reason over coreferences in the dialog, as well as ground the entities from the language modality in the visual one. in contrast to large-scale realistic datasets for visual dialog, such as visdial @xcite , @xcite introduce clevr-dialog as a diagnostic dataset for visual dialog. in contrast to other visual dialog datasets, clevr-dialog is synthetically generated -this allows it to be both large-scale and structured in nature. this diagnostic dataset allows for improved fine-grained analysis, using the structured nature of the images and language. this fine-grained analysis allows researchers to study the different components in isolation and identify bottlenecks in end-to-end systems for visual dialog. highly structured models have performed well on visual question answering and visual dialog @xcite by leveraging explicit program modules to perform composition reasoning. @xcite , which leverages explicit program modules for coreference resolution, was the previous stateof-the-art model on the clevr-dialog dataset. however, the explicit definition of program modules requires handcrafting and limits generalizability. as such, we explore mechanisms of relaxing the structural constrains by using mac networks @xcite and adapting it to the task of dialog. specifically, we introduce the context-aware attention and memory (cam) to serve as an inductive bias that allows mac networks to explicitly capture the necessary context from the dialog history. cam consists of a context-aware attention mechanism and a multi-turn memory state. the contextaware attention mechanism attends over the control states of past dialog turns, to determine the control states for the current dialog turn. since control states in a mac networks are analogous to program modules, the attention effectively leverages past reasoning operations to inform current reasoning operations. for example, if the mac network had to locate the "the red ball", a future turn which refers to "the object to the left of the previous red object" can attend to the control state responsible for locating the red ball. meanwhile the multi-turn memory remembers information extracted to answer previous questions in the dialog. similar to the explicit programs of corefnmn, cam serves to model properties of dialog (e.g., coreference resolution, history dependent reasoning). however, unlike corefnmn, cam does not require explicit handcrafting, can be trained end-to-end and is capable of generalization. our methods attain state-of-the-art performance on clevr-dialog, with a 30% improvement over the prior work. further, cam provides strong performance gains over mac networks across several different experimental setups. analysis shows that cam's attention weights are meaningful, and particularly useful for questions that require coreference resolution across dialog turns. ## results in when none of these three extensions are present, we obtain the vanilla mac network which does not have the ability to reason over the dialog context. we see that vanilla mac achieves 10% higher accuracy than nmn, which is also history-agnostic, and is surprisingly close to corefnmn, which explicitly reasons over the dialog history. the fact that a single turn model can correctly answer twothirds of the questions in a very large dataset raises some concerns regarding how representative is the dataset of an actual dialog task. adding contextaware attention to the mac improves the accuracy of the model considerably to 89.43%. introducing multi-turn memory to this model yields an accuracy of 97.98% accuracy -an improvement of 30% (absolute) on the performance of vanilla mac network and benchmark corefnmn. these results emphatically establish the efficacy of cam and establish a new state-of-the-art for clevr-dialog. perhaps most notably, concatenating the dialog history to the current query works remarkably well for mac networks, achieving 98.08% accuracy with no other augmentations to the mac network. introducing context-aware attention further improves accuracy to 98.25%, which yet again evidences the efficacy of the attention mechanism we propose. however, introducing the multi-turn memory results in a slight decrease in performance, indicating that the memory mechanism is not useful when the entire dialog context is present. we think it is important to mention here that, concatenating the dialog history is a naive method, and this method becomes computationally inefficient when the dialog history is longer and the questions themselves are longer. while it is true that the context aware attention mechanism also stores additional data -the past control states. however, since the control states are fixed sized, the additional memory and computation required is in o(t p), where t is the maximum number of dialog turns and p is the number of reasoning steps to be performed. on the other hand, the increase memory and computation requirements for concatenation is in o(|q max |t p) where |q max | is the length of the longest question. without concatenating the dialog history, incorporating the multi-turn memory greatly improves accuracy (89.43% → 97.98%), while being more computationally efficient. these results indicate that relaxing the structure of the model by eliminating hand-crafted modules gives the model much more flexibility in how it processes the input query, allowing it to perform more complex reasoning than the programs assembled by neural module networks. 6 analysis 6.1 performance breakdown in order to better understand the results presented above, we breakdown the accuracy of the models along dialog turns and question types. ## conclusion we present context-aware attention and memory (cam), a set of dialog-specific augmentations to mac networks @xcite . cam consists of a context-aware attention mechanism which attends over the mac control states of past dialog turns and a persistent, multi-turn memory which is accumulated over multiple turns of the dialog. these augmentations serve as an inductive bias that allow the architecture to capture various important properties of dialog, such as coreference and history dependency. our methods attain state-of-the-art performance on clevr-dialog, with our best model attaining an accuracy of 98.25%, a 30% improvement over all prior results. further, cam attains strong performance gains over vanilla mac networks, especially for question types that require coreference resolution and later dialog turns. ablation experiments indicate that both of the components in cam provide significant improvement in performance. we also verified that the context aware attention mechanism in indeed captures coreferences between the dialog turns. our results are indicative of the flexibility of weakly structured models, like mac networks and their ability to adapt to different problem settings. to adapt mac networks for visual dialog we had to devise a mechanism to provide it with contextual information from past turns. thereafter, the other components of the model were able to adapt and use this information to improve performance on the task, whereas to adapt nmn to visual dialog @xcite had to devise specialized modules to handle specific types of questions.