TVQA +: Spatio-Temporal Grounding for Video Question Answering (2020)

Abstract: We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos. We first augment the TVQA dataset with 310.8K bounding boxes, linking depicted objects to visual concepts in questions and answers. We name this augmented version as TVQA+. We then propose Spatio-Temporal Answerer with Grounded Evidence (STAGE), a unified framework that grounds evidence in both spatial and temporal domains to answer questions about videos. Comprehensive experiments and analyses demonstrate the effectiveness of our framework and how the rich annotations in our TVQA+ dataset can contribute to the question answering task. Moreover, by performing this joint task, our model is able to produce insightful and interpretable spatio-temporal attention visualizations.

## introduction we have witnessed great progress in recent years on image-based visual question answering (qa) tasks @xcite @xcite . one key to this success has been spatial attention @xcite @xcite , where neural models learn to attend to relevant regions for predicting the correct answer. compared to image-based qa, there has been less progress on the performance of video-based qa tasks. one possible reason is that attention techniques are hard to generalize to the temporal nature of videos. moreover, due to the high cost of annotation, most existing video qa datasets only contain qa pairs, without providing labels for the key clips or regions needed to answer the question. inspired by previous work on grounded image and video captioning @xcite , we propose methods that explicitly localize video clips as well as spatial regions for answering videobased questions. such methods are useful in many scenarios, such as natural language guided spatiotemporal localization, and adding explainability to video question answering, which is potentially useful for decision making and model debugging. to enable this line of research, we also collect new joint spatio-temporal annotations for an existing video qa dataset. in the past few years, several video qa datasets have been proposed, e.g., moviefib @xcite , movieqa @xcite , tgif-qa @xcite , pororoqa @xcite , marioqa @xcite , and tvqa @xcite . tvqa is one of the largest video qa datasets, providing a large video qa dataset built on top of 6 famous tv series. be-cause tvqa was collected on television shows, it is built on natural video content with rich dynamics and complex social interactions, where questionanswer pairs are written by people observing both videos and their accompanying dialogues, encouraging the questions to require both vision and language understanding to answer. movie @xcite and television show @xcite videos come with the limitation of being scripted and edited, but they are still more realistic than cartoon/animation @xcite and game @xcite videos, and they also come with richer, real-world-inspired inter-human interactions and span across diverse domains (e.g., medical, crime, sitcom, etc.), making them a useful testbed to study complex video understanding by machine learning models. one key property of tvqa is that it provides temporal annotations denoting which parts of a video clip are necessary for answering a proposed question. however, none of the existing video qa datasets (including tvqa) provide spatial annotation for the answers. actually, grounding spatial regions correctly could be as important as grounding temporal moments for answering a given question. for example, in fig. 1 , to answer the question of "what is sheldon holding when he is talking to howard about the sword?", we need to localize the moment when "he is talking to howard about the sword?", as well as look at the region of "what is sheldon holding". hence, in this paper, we first augment a subset of the tvqa dataset with grounded bounding boxes, resulting in a spatio-temporally grounded video qa dataset, tvqa+. it consists of 29.4k multiplechoice questions grounded in both the temporal and the spatial domains. to collect spatial groundings, we start by identifying a set of visual concept words, i.e., objects and people, mentioned in the question or correct answer. next, we associate the referenced concepts with object regions in individual frames, if there are any, by annotating bounding boxes for each referred concept (see examples in fig. 1 ). our tvqa+ dataset has a total of 310.8k bounding boxes linked with referred objects and people, spanning across 2.5k categories (more details in sec. 3). with such richly annotated data, we then propose the task of spatio-temporal video question answering, which requires intelligent systems to localize relevant moments, detect referred objects and people, and answer questions. we further design several metrics to evaluate the performance of the proposed task, including qa accuracy, object grounding precision, temporal localization accuracy, and a joint temporal localization and qa accuracy. to address spatio-temporal video question answering, we propose a novel end-to-end trainable model, spatio-temporal answerer with grounded evidence (stage), which effectively combines moment localization, object grounding, and question answering in a unified framework. we find that the qa performance benefits from both temporal moment and spatial region supervision. additionally, we provide visualization of temporal and spatial localization, which is helpful for understanding what our model has learned. comprehensive ablation studies demonstrate how each of our annotations and model components helps to improve the performance of the tasks. to summarize, our contributions are: language-guided retrieval grounding language in images/videos is an interesting problem that requires jointly understanding both text and visual modalities. earlier works @xcite @xcite focused on identifying the referred object in an image. recently, there has been a growing interest in moment retrieval tasks @xcite @xcite , where the goal is to localize a short clip from a long video via a natural language query. our work integrates the goals of both tasks, requiring a system to ground the referred moments and objects simultaneously. ## dataset in this section, we describe the tvqa+ dataset, the first video question answering dataset with both spatial and temporal annotations. tvqa+ is built on the tvqa dataset introduced by lei et al.. tvqa is a large-scale video qa dataset based on 6 popular tv shows, containing 152.5k multiple choice questions from 21.8k, 60-90 second long video clips. the questions in the tvqa dataset are compositional, where each question is comprised of two parts, a question part ("where was sheldon sitting"), joined via a link word, ("before", "when", "after"), to a localization part that temporally locates when the question occurs ("he spilled the milk"). models should answer questions using both visual information from the video, as well as language information from the naturally associated dialog (subtitles). since the video clips on which the questions were collected are usually much longer than the context needed for answering the questions, the tvqa dataset also provides a temporal timestamp annotation indicating the minimum span (context) needed to answer each question. while the tvqa dataset provides a novel question format and temporal annotations, it lacks spatial grounding information, i.e., bounding boxes of the concepts (objects and people) mentioned in the qa pair. we hypothesize that object annotations could provide an additional useful training signal for models to learn a deeper understanding of visual information. therefore, to complement the original tvqa dataset, we collect frame-wise bounding boxes for visual concepts mentioned in the questions and correct answers. since the full tvqa dataset is very large, we start by collecting bounding box annotations for qa pairs associated with the big bang theory. this subset contains 29,383 qa pairs from 4,198 clips. ## methods our proposed method, spatio-temporal answerer with grounded evidence (stage), is a unified framework for moment localization, object grounding and video qa. first, stage encodes the video and text (subtitle, qa) via frame-wise regional visual representations and neural language representations, respectively. the encoded video and text representations are then contextualized using a convolutional encoder. second, stage computes attention scores from each qa word to object regions and subtitle words. leveraging the attention scores, stage is able to generate qa-aware representations, as well as automatically detecting the referred objects/people. the attended qa-aware video and subtitle representation are then fused together to obtain a joint frame-wise representation. third, taking the frame-wise representation as input, stage learns to predict qa relevant temporal spans, then combines the global and local (span localized) video information to answer the questions. in the following, we describe stage in detail. ## experiments as introduced, our task is spatio-temporal video question answering, requiring systems to temporally localize relevant moments, spatially detect referred objects and people, and answer questions. in this section, we first define the evaluation metrics, then compare stage against several baselines, and finally provide a comprehensive analysis of our model. additionally, we also evaluate stage on the full tvqa dataset. ## conclusion we collected the tvqa+ dataset and proposed the spatio-temporal video qa task. this task requires systems to jointly localize relevant moments, detect referred objects/people, and answer questions. we further introduced stage, an end-to-end trainable framework to jointly perform all three tasks. comprehensive experiments show that temporal and spatial predictions help improve qa performance, as well as providing explainable results. though our stage achieves state-of-the-art performance, there is still a large gap compared with human performance, leaving space for further improvement.