Dynamic Multistep Reasoning based on Video Scene Graph for Video Question Answering (2022)

Abstract: Existing video question answering (video QA) models lack the capacity for deep video understanding and flexible multistep reasoning. We propose for video QA a novel model which performs dynamic multistep reasoning between questions and videos. It creates video semantic representation based on the video scene graph composed of semantic elements of the video and semantic relations among these elements. Then, it performs multistep reasoning for better answer decision between the representations of the question and the video, and dynamically integrate the reasoning results. Experiments show the significant advantage of the proposed model against previous methods in accuracy and interpretability. Against the existing state-of-the-art model, the proposed model dramatically improves more than 4\%/3.1\%/2\% on the three widely used video QA datasets, MSRVTT-QA, MSRVTT multi-choice, and TGIF-QA, and displays better interpretability by backtracing along with the attention mechanisms to the video scene graphs.

## introduction video question answering (video qa) aims to answer questions according to the given videos. it is usually defined as a classification task, where the most appropriate answer is chosen from a candidate list for the given question and video. existing methods for video qa conduct direct answering selection based on the multimodal encoding of questions and videos @xcite @xcite . in recent years, researchers have proposed many optimization strategies for better performance in video question answering, e.g., designing delicate encoding mechanisms @xcite @xcite @xcite @xcite @xcite , introducing video scene graphs @xcite , adopting video pre-trained language models @xcite @xcite @xcite , and leveraging external knowledge or resources @xcite @xcite @xcite . compared with conventional monomodal question answering tasks such as text qa @xcite @xcite and table qa @xcite . video qa is more difficult due to the need for crossmodal understanding and reasoning of the video and the question. existing methods are mainly concerned with how to encode the crossmodal features better. when faced with a complex question, they usually lack the abilities of deep understanding and complex reasoning. similar to the situations in text qa and table qa, it is also necessary for video qa to deeply understand the semantics of the context, namely, the video, and the reasoning on the context and the question. statistical analysis on several datasets embedding text encoder vision encoder video representation memory sparse sampling video-level decision gnn answer question video embedding text encoder vision encoder sparse sampling aggregating (mean/max/logsumexp) answer question text encoder text encoder clip-level decision clip-level decision clip-level decision video embedding text encoder vision encoder dense sampling answer question cross-modal encoding video-level decision (a). existing methods (b). clipbert (2021) (c). ours stop gradient video scene graph graph consturction dynamic multistep reasoning read figure 2: comparison of our model architecture with (a). popular video-and-language learning paradigm and (b).clipbert. in contrast to most previous methods that adopt simple and implicit reasoning mechanisms, our method uses video scene graphs to represent the video semantic information and adopt a dynamic multistep reasoning mechanism. in this work, we propose a novel dynamic reasoning model for video qa to overcome the weakness of previous models in deep understanding and reasoning. it first creates the video semantic representation from the video scene graph, which is composed of the semantic elements of the video and the semantic relations between these elements. then it conducts multistep reasoning of the question based on the video semantic representation to generate a series of video-aware question representations. finally, it generates the most appropriate question representation for the final answering decision by dynamically integrating these video-aware question representations according to the reasoning complicity prediction. figure 2 shows the overall architecture of the proposed model and the comparison with previous methods. it simulates the reasoning procedure of human beings, while previous methods follow the pipeline of multimodal encoding and answering selection. in addition, the proposed model enables the decomposition of question understanding and video understanding, thus leading to more opportunities for future optimization. on the one hand, more external knowledge resources and better reasoning architectures can be introduced for better video qa performance. on the other hand, it can act as a unified framework for different qa tasks such as video qa, table qa , and text qa. we verify the proposed model on three wellknown datasets, msrvtt-qa @xcite , msrvtt multi-choice @xcite , and tgif-qa @xcite , widely used in recent video qa works @xcite @xcite @xcite @xcite . experiments show that our model achieves dramatic improvement over the powerful state-of-the-art model clipbert @xcite , with an average accuracy increment of more than 3 percentage points. ablation studies show that the dynamic reasoning strategy significantly outperforms previous implicit simple reasoning strategies. the video semantic representation based on the video scene graph makes the dynamic reasoning strategy work better. backtracing along with the at- ... 1 f 2 f n train f 1 n 2 n m n video scene graph r 0 reader r 1 updater r s-1 r s-1 ... r 0 r 1 r s-1 ... r s reasoner query v s-1 r s .. . 1 cilp 2 cilp n train cilp tie man suit photo o f we ar w ea r beach o n p la y frisbee . . . o f group people video representation memory graph construction evaluator integrator r f [cls] what does the video show old footage of a man and also of ? . ## background given the question q, video question answering requires choose the correct answer â from the candidates set ω a according to the video content v . as figure 2 (a) shows, most existing works utilize offline (stop gradient) extracted dense video features and text features. as figure 2 (b) shows, clipbert @xcite achieves the state-ofthe-art by using sparsely sampled clips and raw text for end-to-end training, yet suffer from two main drawbacks: (i) lack of a deep understanding of the video content and reasoning on the question based on the video. (ii) strong coupling between video and question modeling process, which needs additional image-text pair data for pre-training to enhance the ability of the text encoder to model multimodal features and leads to poor scalability and repeated computation. as shown in figure 2 (c), we propose a simple but effective architecture to solve the weakness of previous works. it first de-couples the question understanding module and the video understanding module. then, we introduce the video scene graph to get a better representations of the video semantic information, which can enhance the understanding of the video content and its graph structure is also better for reasoning. at last, we design a dynamic multistep reasoning mechanism to iteratively deepen the understanding of the question according to the video content. ## experiments in this section, we validate our method on three mainstream video qa datasets. we conduct comparison experiments with previous works and perform ablation experiments to analyze the critical improvement in our proposed method. we use standard train/val/test splits for all datasets and use accuracy to measure the performance. all experimental results are the mean and standard deviation of ten replicate experiments. ## conclusion we propose a dynamic reasoning mechanism based on video scene graph for video qa to alleviate the drawback of existing methods, that is, lack of deep understanding and multistep reasoning. experiments show that our method significantly surpasses previous methods on multiple video qa datasets due to better understanding and reasoning mechanisms and achieves much better interpretability by backtracing along with the attention mechanism to the video scene graph. in addition, different from the conventional manners that perform classification after crossmodal feature encoding, the model realizes the decoupling of question understanding and video understanding and the decoupling of understanding and decision-making, thus providing more possibilities for improvement. on the one hand, we can optimize video qa by introducing external knowledge, designing more effective reasoning mechanisms, defining and constructing better video scene graphs. on the other hand, we can also jointly model multimodal qa, such as qa based on videos, tables, texts, and graphs.