DVD : A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue (2021)

Abstract: A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial and temporal scene variations. Building such dialogue systems is a challenging problem, involving various reasoning types on both visual and language inputs. Existing benchmarks do not have enough annotations to thoroughly analyze dialogue systems and understand their capabilities and limitations in isolation. These benchmarks are also not explicitly designed to minimise biases that models can exploit without actual reasoning. To address these limitations, in this paper, we present DVD, a Diagnostic Dataset for Video-grounded Dialogue. The dataset is designed to contain minimal biases and has detailed annotations for the different types of reasoning over the spatio-temporal space of video. Dialogues are synthesized over multiple question turns, each of which is injected with a set of cross-turn semantic relationships. We use DVD to analyze existing approaches, providing interesting insights into their abilities and limitations. In total, DVD is built from 11k CATER synthetic videos and contains 10 instances of 10-round dialogues for each video, resulting in more than 100k dialogues and 1M question-answer pairs. Our code and dataset are publicly available.

## introduction research in visual question answering (vqa) aims to develop intelligent systems that can reason and answer questions about visual information. earlier datasets have been introduced to study this problem, focusing on images as the visual input @xcite @xcite q1: until the end of the cube 's rotation , what types of actions does the big thing undertake the most ? a1: flying q2: during the same time period , how many sliding objects are there ? a2: 2 q3: among them , there is a ball . during the whole video , what type of action does it undertake second ? a3: no action q4: how about up until now ? a4: sliding q5: during the red thing 's last slide , how many things are behind the earlier mentioned large object ? a5: 2 q6: how about left of it ? a6: 0 ... and @xcite recently, many qa benchmarks have been proposed to extend the visual information from the image to video domain @xcite @xcite . while image qa problems require a system to learn cross-modality interaction, video qa problems go beyond and capture visual information with temporal variance. as an orthogonal extension from vqa problems, another line of research investigates image/video qa in a dialogue setting @xcite @xcite @xcite . in this problem, questions about a given video or image are positioned in a multi-turn dialogue. in each dialogue turn, a question usually exhibits different types of cross-turn relations to other questions in prior dialogue turns, such as object co-reference and topic alignment. in this work, we investigate the problem of multi-turn video question answering (qa), also known as video-grounded dialogue. numerous approaches to video-grounded dialogue have shown remarkable performance in build-ing intelligent multimodal systems @xcite @xcite @xcite . however, most of these methods exhibit marginal performance gain, and our ability to understand their limitations is impeded by the complexity of the task. existing benchmarks are not designed with enough information to determine whether current approaches are capable of sophisticated reasoning and not just exploiting biases, which has been a common concern in vision-language systems @xcite @xcite . to address the limitations of existing benchmarks and analyze dialogue systems more efficiently, we propose dvd, a diagnostic dataset for video-grounded dialogues. we demonstrate an example dialogue in dvd in figure 1 . from scene graphs and object action annotation of a cater video @xcite , we simulate questions based on reasoning structures, also known as functional programs in clevr @xcite . compared to clevr, we introduced 17 novel functional modules, designed for video and dialogue input components. as illustrated in figure 1 , at each dialogue turn, a dvd question tests dialogue systems to perform different types of reasoning on videos, such as action recognition and spatio-temporal reasoning. across turns, we generate questions to be related to each other by incorporating different types of semantic relationships, including: (1) temporal relation, which requires a system to learn to localize different temporal segments of the video from turn to turn; (2) object reference, which requires a system to resolve visual objects mentioned throughout the dialogue history in either short-term references (pronouns) or longterm references (e.g. "the earlier mentioned large object"); and (3) topic transfer, which requires a system to maintain a memory of the last question turn to solve the question in the current turn. on dvd, we trained a set of baseline methods and analyzed the results by several aspects of visual and linguistic complexity (section 4). we found that these methods struggle on questions requiring both video temporal and spatial localization. they are also vulnerable to long-term reasoning in both videos and dialogues as they are not designed to track active visual objects or relevant video segments throughout dialogue context. we hope the dvd dataset will lead to new research avenues to develop intelligent systems capable of complex reasoning on video and dialogue medium (further discussion in the supplementary material). the dvd dataset and code will be made public. ## related work we compared dvd to existing datasets from the following four angles: 1) vision-linguistic. vision-linguistic understanding benchmarks have been proposed, including captioning @xcite @xcite , phrase grounding or object reference @xcite , scene graph learning @xcite , and text-to-clip @xcite . our benchmark, dvd, is more related to vqa in which a visual input is given and a system is required to answer a question about this input @xcite @xcite . another related line of research is the research of navigation systems in a physical environment @xcite . compared to the prior benchmarks, one major difference of dvd is the extension of single-turn interaction to a multi-turn human-machine dialogue. ## the dvd dataset our benchmark provides a dataset that can be used to conduct rich diagnostics to better understand the reasoning capabilities of dialogue systems. ## dialogue systems on dvd the video-grounded dialogue task in dvd is defined as a turn-based retrieval task from multiplechoice candidate answers. at each dialogue turn i (@xmath0, 2, ..., 10), video input v i , the groundtruth dialogue context, including question and answer pairs up to the last dialogue turn, ## discussion and conclusion we have introduced dvd, a diagnostic dataset designed to analyze video-grounded dialogue systems. dvd dataset is generated with tight control of data bias through balancing the question and answer distribution and questions are built based on a principled approach to reflect the complexity in videos and dialogues. our results have shown that dvd can provide interesting insights into system abilities and limitations. specifically, our analysis has revealed some key shortcomings of current models, including: (1) limited ability to efficiently integrate visual information from both spatial and temporal space; (2) limited ability to recognize and compile multiple actions in long-ranged video intervals; (3) inconsistent performance across dialogue turns, especially in cases when systems are required to switch attention temporally; and (4) unstable performance to resolve object co-reference in the dialogue context, especially when the turn distance of the object references increases. these insights provide potential avenues where we hope dvd will be a useful benchmark to explore new ideas. specifically, we discuss two research directions: dialogue object tracking. to further diagnose a dialogue system, we aim to study their long-term memory reasoning ability to track objects and their attributes mentioned in the dialogue context. we are inspired by research work of dialogue state tracking in task-oriented dialogues @xcite and propose to use tracking accuracy metric in video-grounded dialogue systems. at each turn t, a video-grounded dialogue system should be able to track and update a dialogue state s t , defined as a set of all mentioned objects o t i and their attributes, including sizes z t i , colors c t i , materials m t i , and shapes s t i : s @xmath1, o t 2 , ...) = ((z t 1 , c t 1 , m t 1 , s t 1 ), (z t 2 , c t 2 , m t 2 , s t 2 ), ...). we define two tracking metrics, including joint accuracy, measuring the accuracy of prediction of all objects and attributes as a set, and slot accuracy, measuring the accuracy of predicted attributes individually. the introduction of these evaluation metrics necessitates a new learning task, dialogue object tracking (dot) in video-grounded dialogue systems, to better understand current systems' long-term reasoning ability. video interval tracking. another aspect of dialogue systems that we want to diagnose is their ability to localize video segments in a multi-turn setting. each question turn often focuses on different parts of the video as the dialogue extends over time. it is important to learn how a system can localize the right segments of the video from turn to turn. similar to dot, we define a new learning task for video interval tracking (vit) in a similar nature as text-to-clip tasks @xcite . the task can be defined as a ranking task of segment candidates to choose the relevant segments in each question turn. this task is evaluated by ranking metrics such as rank@1 or rank@2, and mean intersection over union (miou). alternatively, we can adapt grounding, a simple metric used by @xcite to assess spatial attention of image regions. in dvd, grounding can be used in temporal attention-based approaches to determine model ability to localize the right position of video intervals in question. finally, we want to emphasize that dvd is designed as a synthetic dataset for diagnosis purposes to systematically evaluate model capabilities. the benchmark should not be used to replace data of human dialogues but be used to supplement realworld dialogue datasets. amir zadeh, michael chan, paul pu liang, edmund tong, and louis-philippe morency. 2019. social-iq: a question answering benchmark for artificial social intelligence. in proceedings of the ieee conference on computer vision and pattern recognition, pages 8807-8817. yuke zhu, oliver groth, michael bernstein, and li fei-fei. 2016. visual7w: grounded question answering in images. in proceedings of the ieee conference on computer vision and pattern recognition, pages 4995-5004.