Video Question Answering: Datasets, Algorithms and Challenges (2022)

Abstract: This survey aims to sort out the recent advances in video question answering (VideoQA) and point towards future directions. We firstly categorize the datasets into 1) normal VideoQA, multi-modal VideoQA and knowledge-based VideoQA, according to the modalities invoked in the question-answer pairs, or 2) factoid VideoQA and inference VideoQA, according to the technical challenges in comprehending the questions and deriving the correct answers. We then summarize the VideoQA techniques, including those mainly designed for Factoid QA (e.g., the early spatio-temporal attention-based methods and the recently Transformer-based ones) and those targeted at explicit relation and logic inference (e.g., neural modular networks, neural symbolic methods, and graph-structured methods). Aside from the backbone techniques, we delve into the specific models and find out some common and useful insights either for video modeling, question answering, or for cross-modal correspondence learning. Finally, we point out the research trend of studying beyond factoid VideoQA to inference VideoQA, as well as towards the robustness and interpretability. Additionally, we maintain a repository, https://github.com/VRU-NExT/VideoQA, to keep trace of the latest VideoQA papers, datasets, and their open-source implementations if available. With these efforts, we strongly hope this survey could shed light on the follow-up VideoQA research.

## introduction recent years have witnessed a flourish of research in vision-language understanding @xcite @xcite @xcite , of which, video question answering (videoqa) is one of the most prominent, given its promise to develop interactive ai to communicate with the dynamic visual world via natural languages. despite the popularity, videoqa remains one of the greatest challenges, because it demands the models to comprehensively understand the videos to correctly answer questions. the questions involve not only the recognition of visual objects, actions, activities and events, but also the inference of their semantic, spatial, temporal, and causal relationships @xcite @xcite @xcite @xcite . to tackle the challenges, techniques such as spatio-temporal attention @xcite , motion-appearance memory @xcite , and spatio-temporal or hierarchical graph models @xcite have been proposed and demonstrated their effectiveness on different videoqa datasets. however, we find that the datasets, the defined challenges, and the corresponding algorithms are varied and a bit messy. there is a lack of a meaningful survey to categorize the datasets and to organize the technique developed, which seriously impedes the research. although a handful of recent works @xcite @xcite have tried to review videoqa, they mostly follow an old-to-new fashion to summarize the literature and lack an effective taxonomy to classify them. in terms of the contents, these works focus merely on factoid questions and neglect the inference questions (see fig. 1 for the difference). furthermore, lots of recent new techniques (e.g., pre-training and transformer) are missing. this paper thus gives a more comprehensive and meaningful survey to videoqa, in the hope of learning from the past and shaping the future. our contributions are as follows. (1) we provide a clear taxonomy to videoqa. we can either classify existing videoqa tasks into factoid videoqa and inference videoqa according to the fundamental challenges embodied in qas, or classify them into normal videoqa, multi-modal videoqa, and knowledge-based videoqa according to the multimodal information invoked in the qas. (2) we categorize existing videoqa techniques as memory, transformer, graph, neural modular network, and neural-symbolic method. along with the techniques, some meaningful insights are also included: attention modeling, cross-modal pre-training, hierarchical learning, multi-granular ensemble, and progressive reasoning. (3) we analyze existing methods from the perspective of the challenges encountered in the various videoqa tasks and provide our prospects for future research. ## conclusion this paper gives a quick overview to the broad aspect of video question answering. we mainly categorized the related datasets and techniques. also, we discussed some meaningful insights and analyzed the performances of different techniques on different type of datasets. we finally concluded several promising future directions. with these efforts, we hope this survey can shed light and attract more research to videoqa, and eventually, foster more efforts towards strong ai systems that can demonstrate their understanding of the dynamic visual world by making meaningful responses to our natural language instructions or queries.