Generating Questions, Answers, and Distractors for Videos: Exploring Semantic Uncertainty of Object Motions (2025)

Abstract: Video Question-Answer-Distractors (QADs) show promising values for assessing the performance of systems in perceiving and comprehending multimedia content. Given the significant cost and labor demands of manual annotation, existing large-scale Video QADs benchmarks are typically generated automatically using video captions. Since video captions are incomplete representations of visual content and susceptible to error propagation, direct generation of QADs from video is crucial. This work first leverages a large vision-language model for video QADs generation. To enhance the consistency and diversity of the generated QADs, we propose utilizing temporal motion to describe the video objects. In addition, We design a selection mechanism that chooses diverse temporal object motions to generate diverse QADs focusing on different objects and interactions, maximizing overall semantic uncertainty for a given video. Evaluation on the NExT-QA and Perception Test benchmarks demonstrates that the proposed approach significantly improves both the consistency and diversity of QADs generated by a range of large vision-language models, thus highlighting its effectiveness and generalizability.

## introduction video question-answer-distractors (qads) @xcite @xcite facilitates the evaluation of video-language understanding across modalities, offering a benchmark for both human and machine performance. for instance, online education videos mooc requires students to answer questions to assess their understanding of the video materials. besides, video qads serve as established benchmarks for evaluating multimodal models @xcite @xcite . the manual generation of video qads necessitates the analysis * corresponding author. two primary approaches currently dominate the automatic generation of qads for videos. the first approach relies on textual information extracted from video captions @xcite or transcriptions @xcite , which are susceptible to incomplete representations @xcite and error propagation @xcite . the second approach generates qads directly from the video content. existing research in this domain has primarily focused on leveraging question types @xcite or solely object information @xcite for question-answer pair generation. however, this crucial aspect of video understanding remains unaddressed by current methodologies. as illustrated in figure 1 , the qads generated by llava-next cannot effectively assess the understanding of motions and interactions of the videos. in contrast, the qads generated by our model, maxsem, are specifically focused on the temporal object motions and interactions present within the video. in this work, we propose maxsem, a framework to maximize the semantic uncertainty of generated question-answer-distractors for video, ensuring both consistency and diversity. we introduce temporal object motions, which provide a description of object movements over time within the video. the generated qads leverage object motions and interactions, which are widely present in the ground truth datasets, thereby enhancing consistency. furthermore, a selection mechanism is proposed to sample diverse combinations of temporal object motions when generating multiple qads from a single video, thereby maximizing semantic uncertainty and ensuring qads diversity. we conduct extensive experiments on the next-qa and perception test benchmarks, empirical results reveal that the proposed method significantly improves both the consistency and diversity of qads generated by lvlms. the contribution of this paper lies in the automatic generation of qads to evaluate video understanding capabilities. moreover, a prospective direction for future research involves utilizing the generated qads to improve the performance of existing models. the main contributions of this work are listed as follows: the majority of previous research has concentrated on the generation of a part of or parts of qads, namely questions, answers, or distractors. video question generation involves generating meaningful questions from video content or captions. wang et al. (2020) leveraged video frames, detected objects, and subtitles as input to generate semantically meaningful questions. su et al. (2021) introduced a generator-pretester network that generates question-answer pairs and subsequently validates the generated question by trying to answer it. guo et al. (2020) extended this work to the multiturn setting, generating multiple questions based on both dialogue history and video content. video question answering (le et al., 2020; xiao et al., 2022b,a) requires the models to understand both the complex video and language data to correctly generate the answers. wang et al. (2022) proposed vidil, a framework that instructs a large language model on video-language tasks based on the temporal-aware template. li et al. (2022) proposed an invariant grounding framework (igv) to distinguish the causal scene and emphasize its causal effect on the answer. li et al. (2023) developed a differentiable selection module that adaptively collects question-critical moments and objects using cross-modal interaction. visual distractor generation targets to generate challenging distractors according to the visual content. lu et al. (2022) introduced a reinforcement learning approach to generate distractors from visual images. luo et al. (2024) proposed leveraging multimodal large language models with chain-of-thought reasoning to generate both questions and distractors. ding et al. (2024a) first proposed to generate questions, answers, and distractors jointly. ding et al. (2024b) introduced a framework to generate qads that focus on different regions in the image. however, their work is confined to the analysis of individual images. in this paper, we explore the simultaneous generation of multiple qads from a video, incorporating considerations of both consistency and diversity. ## conclusion this paper presents maxsem, a novel method that leverages temporal object motions and maximum semantic uncertainty to generate multiple qads from a given video. specifically, we use temporal object motions to describe the object movement over time and the intersections between different objects. meanwhile, we introduce a selection module to select different temporal object motions to guide the generation. we conduct experiments on the next-qa and perception test benchmarks to demonstrate a significant improvement of our proposed methods for different models.