Semantic-Aware Dynamic Retrospective-Prospective Reasoning for Event-Level Video Question Answering (2023)

Abstract: Event-Level Video Question Answering (EVQA) requires complex reasoning across video events to obtain the visual information needed to provide optimal answers. However, despite significant progress in model performance, few studies have focused on using the explicit semantic connections between the question and visual information especially at the event level. There is need for using such semantic connections to facilitate complex reasoning across video frames. Therefore, we propose a semantic-aware dynamic retrospective-prospective reasoning approach for video-based question answering. Specifically, we explicitly use the Semantic Role Labeling (SRL) structure of the question in the dynamic reasoning process where we decide to move to the next frame based on which part of the SRL structure (agent, verb, patient, etc.) of the question is being focused on. We conduct experiments on a benchmark EVQA dataset - TrafficQA. Results show that our proposed approach achieves superior performance compared to previous state-of-the-art models. Our code is publicly available at https://github.com/lyuchenyang/Semantic-aware-VideoQA .

## introduction this paper focuses on one specific variant of video question answering (vqa) @xcite @xcite , namely eventlevel vqa (evqa) @xcite . in general, the objective of the vqa task is to provide an answer to a visual-related question according to the content of an accompanying video. despite significant recent progress in vqa, evqa still remains one of the most challenging vqabased tasks since it requires complex reasoning over the events across video frames @xcite @xcite . to * *corresponding author tackle the challenges in evqa, a number of approaches have been proposed @xcite . @xcite propose a temporal-aware bidirectional attention mechanism for improving event reasoning in videos, while @xcite propose a novel model named energy-based refinedattention mechanism (erm), which obtains better performance compared to previous approaches with a smaller model size. @xcite , on the other hand, incorporate visual-linguistic causal dependencies based on graph convolutional networks @xcite for enhancing cross-modal event reasoning for evqa. despite recent advances, conventional evqa approaches generally fail to take into account the explicit semantic connection between questions and the corresponding visual information at the event level. therefore, we propose a new approach that takes advantage of such semantic connections, using the semantic role labeling (srl) @xcite @xcite structure of questions. the model uses srl information to learn an explicit semantic connection between the text-based questions and visual information in videos. additionally, we carry out a multi-step reasoning mechanism over video frames to avoid adapting to spurious correlation and shortcuts by explicitly learning the reasoning process itself @xcite @xcite @xcite . specifically, in each reasoning step, the model should explicitly decide which frame should be focused on by predicting the reasoning direction (retrospective or prospective). in terms of the question, in each reasoning step, we focus on one or more specific srl arguments with high attention weights, and model its connection with the visual information (i.e., video frames) contained within the corresponding video. for example, for a question such as [arg1: how many cars] were [verb: involved] [arg2: in the accident?], the model con- update reasoner coverage retrospec@xmath0ve reasoning h t h t-1 answer predictor figure 1: overview of our approach for multi-step visual reasoning. in each reasoning step, the model predicts the reasoning direction (either retrospective or prospective) and focuses on a specific srl argument with high attention weights. a coverage mechanism is employed to improve the coverage of srl arguments in the question. to investigate the effectiveness of the proposed approach, we conduct experiments on a benchmark evqa dataset: trafficqa. results reveal the model to achieve performance superior to that of existing baselines for a range of reasoning types (e.g., counterfactual, prospective). ## methodology an overview of our approach is shown in figure 1 . suppose the input of our model consists of a video v composed of n image frames sampled from it: meanwhile, we use a text encoder to obtain the representations of the question with its corresponding srl arguments: q ′ ∈ r 1×d and s ′ ∈ r n ×d . we then perform multi-step reasoning in which we iteratively update the hidden state vector h with the visual information from frame representations based on the attention weights between them and the srl arguments of the question. h is updated from the initial step h 0 to the final step h t -1 where t is the total number of reasoning steps. finally, we predict the most probable answer a based on h t -1 . ## conclusion and future work in this paper, we propose a multi-step dynamic retrospective-prospective approach for evqa. our approach employs a multi-step reasoning model that explicitly learns reasoning based on the semantic connection of the srl structure of a question and corresponding video frames. we additionally proposed a coverage mechanism to improve the coverage of srl arguments in the reasoning process. experimental results show that the proposed