In-the-Wild Video Question Answering (2022)

Abstract: Existing video understanding datasets mostly focus on human interactions, with little attention being paid to the “in the wild” settings, where the videos are recorded outdoors. We propose WILDQA, a video understanding dataset of videos recorded in outside settings. In addition to video question answering (Video QA), we also introduce the new task of identifying visual support for a given question and answer (Video Evidence Selection). Through evaluations using a wide range of baseline models, we show that WILDQA poses new challenges to the vision and language research communities. The dataset is available at https: //lit.eecs.umich.edu/wildqa/.

## introduction video understanding plays an important role in the development of intelligent ai systems, as it enables the effective processing of different modalities of information @xcite . various tasks have been proposed to examine the ability of models' to understand videos, including video question answering (video qa), video captioning, and fill-inthe-blank tasks @xcite @xcite *: equal contribution @xcite . recent years have witnessed significant progress in video understanding, including new benchmarks @xcite as well as advanced sophisticated benchmarksmodels @xcite . there are however several drawbacks associated with existing video understanding research. first, existing video understanding benchmarks focus on common human activities as typically appearing in cooking videos @xcite or in movies @xcite , leading to a limited set of video domains. second, most video understanding benchmarks adopt a multiple-choice format, where models select an answer from a set of candidates @xcite . models trained under such a setting cannot be used in real-life applications because candidate answers are not provided @xcite . third, videos included in existing benchmarks are typically short @xcite , and the performance of models on longer videos is not well studied. we address these challenges in our dataset construction process. first, we propose the wildqa dataset in which we collect "in the wild" videos that are recorded in the outside world, going beyond daily human activities. figure 2 shows the differ-figure 2 : examples from movieqa @xcite , tvqa @xcite , and our wildqa dataset. the previous datasets mostly focus on human interactions in a multiple-choice setting, while ours focus on scenes recorded in the outside world in an open-ended setting. we only list a single answer here for illustration purposes. using the wildqa dataset, we address two main tasks. first, we address the task of video question answering (video qa), aiming to generate open-ended answers. second, we introduce the task of retrieving visual support for a given question and answer (video evidence selection). finding the relevant frames in a video for a given question-answer pair can help a system in its reasoning process, and is in line with ongoing efforts to build interpretable models @xcite . for each of these two tasks, we evaluate several baseline models, including multi-task models that combine the two tasks together. figure 1 shows an example from our dataset, including an example of a question, answer, and supporting video evidence. to summarize, the main contributions of this paper are: researchers have also developed various methods to handle the video qa task, including joint reasoning of the spatial and temporal structure of a video @xcite @xcite , integrating memory to keep track of past and future frames @xcite @xcite @xcite , various attention mechanisms @xcite @xcite @xcite , and others. recently, pre-trained models have proved to be useful in various visual and language tasks @xcite @xcite . however, the pretrained visual and language models are typically encoder-only and cannot generate an answer in natural language on their own. thus, such pre-trained encoder-only models do not fit into the open-end video question answering setting in our task. additionally, previous work has also investigated various reasoning tasks in a multimodal setting @xcite @xcite . although it is not our focus, some questions in our dataset require a certain level of reasoning ability. moreover, since our dataset is created by domain experts, there is domain knowledge involved in the questions as well. moment retrieval. moment retrieval is the task of retrieving a short moment from a large video corpus given a natural language query @xcite . researchers have proposed or adapted various datasets for this task @xcite @xcite . the task of retrieving relevant parts in the video given the question (video evidence selection) in our proposed dataset is akin to moment retrieval. however, moment retrieval focuses on retrieving the part of videos that the question describes, while video evidence selection is to find parts of videos that can support the answer to the questions as shown in figure 1 . prior work such as tutorial-vqa @xcite also adopt the setting of providing parts of the videos as answers to the question, but they did not include any text answers in their dataset. ## wildqa dataset video selection and processing. following @xcite ; @xcite , we start by collecting videos from youtube. first, we identify five domains that primarily consist of outdoor scenes and are representative for the outside world, namely, agriculture, geography, human survival, natural disasters, and military. we then manually collected videos from relevant youtube channels for each domain. because the raw videos can be as long as an hour, we split the raw videos into short clips using pyscenedetect, foot_0 and concatenate these short clips so that the output video is approximately one minute. we use the output videos for the annotation process described below. more details for the video selection and video processing steps are discussed in appendix a.1. ## video question answering following @xcite , we adopt free-form open-ended video question answering for our video question answering (video qa) task. given a question q and a video v, the task is to generate an answer a in natural language. we adopt a few-shot learning setting on our dataset, where models are fine-tuned on questionanswer pairs corresponding to 30% of the videos for each domain. the tuned models are tested on data for the remaining 70% videos. the reason is that the time to annotate 30% of the data is around 23 hours, during which there are around 50 data points annotated for each domain, which is acceptable. we hypothesize that it is realistic to have such a setting because the potential endusers could spend around a day or two collecting data, and we can then quickly tune a model using it. moreover, no repeated videos appear in different splits, following @xcite . we end up having 264 question-answer pairs for 108 videos in our dev set and 652 pairs for 261 videos in the test set. we adopt bleu @xcite and rouge @xcite as the metrics to measure the quality of the generated answer. we run each model 3 times and report the scores of @xmath0: comparison between our wildqa and other existing datasets. ve?: whether the dataset provides "video evidences"?; mc: "multiple choice" question answering; oe: "open ended" question answering; es: "evidence selection". we adapt the comparison table from zhong et al. (2022). ## video evidence selection similar to @xcite , given a video v and a question q, the video evidence selection task consists of predicting {(s 1 , e 1 ), (s 2 , e 2 ), . . .}, where (s i , e i ) represents the time for start s and end e of a singles span within the video v. we also adopt the few-shot learning setting as described in section 4 for the task of video evidence selection. similar to deyoung et al. (2020), we design an intersection-over-union (iou) metric borrowed from @xcite . we define iou as follows: given two time spans in the video, iou is defined as the length of their intersection divided by the length of their union. the prediction is counted as a match if it overlaps with any of the ground truth spans by more than the threshold (0.5, following deyoung et al., 2020). we use these partial matches to calculate an f1 score (iou-f1 scores). as described in section 4, we run each model three times and report the scores of @xmath1. ## conclusion in this paper, we introduced a new and challenging benchmark, wildqa, to promote domain diversity for video understanding. specifically, we focused on five domains that involve long videos recorded in the outside world, which can be useful for applications in these domains. instead of the traditional multiple-choice setting for video question answering, we proposed to generate open-ended answers. we believe open-end answer generation can help construct systems that can answer end users' questions in a more natural way. to help the model attend to the relevant parts in the videos, we also proposed the task of video evidence selection. through experiments, we showed the feasibility of these tasks, and also showed that jointly training for both video question answering and video evidence selection can improve the models' performance. in addition, we found it is easier to understand models' behavior by knowing which part of the video the model attends to when answering a question. we believe that this is an important step towards a trustworthy, explainable multimodal system. the dataset is available at https: //lit.eecs.umich.edu/wildqa/ .