Causal and Temporal Inference in Visual Question Generation by Utilizing Pre-trained Models (2024)

Abstract: Visual Question Generation is a task at the crossroads of visual and language learning, impacting broad domains like education, medicine, and social media. While existing pre-trained models excel in fact-based queries with image pairs, they fall short of capturing human-like inference, particularly in understanding causal and temporal relationships within videos. Additionally, the computational demands of prevalent pre-training methods pose challenges. In response, our study introduces a framework that leverages vision-text matching pre-trained models to guide language models in recognizing event-entity relationships within videos and generating inferential questions. Demonstrating efficacy on the NExT-QA dataset, which is designed for causal and temporal inference in visual question answering, our method successfully guides pre-trained language models in recognizing video content. We present methodologies for abstracting causal and temporal relationships between events and entities, pointing out the importance of consistent relationships among input frames during training and inference phases and suggesting an avenue for future exploration.

## introduction visual question generation (vqg) is an emerging task of multi-modal learning, integrating vision and language. since its inception @xcite , vqg has influenced diverse domains like education @xcite , social media @xcite , and human-computer interaction @xcite . existing datasets primarily cater to factoid question answering, extracting direct answers from visual content @xcite . however, factoid question answering lacks inherent depth in human thinking, exemplified by the disparity between a fact-based query like "was anyone injured in the crash?" and a more insightful, causal 1 the code is available at this address. this research addresses a critical gap in the vqg landscape: the absence of studies exploring inference aligned with human thinking. moreover, unlike singular images, videos offer richer details of relationships between events and entities, prompting our focus on two fundamental types of reasoning-causal inference and temporal inference. through this approach, we aim to introduce a new challenge of inferential question generation originating from videos and auxiliary text and advance the field of vqg. meanwhile, despite advancements in vqg, the computational demands @xcite of pre-training models, particularly visual transformers @xcite , pose challenges. our work distinguishes itself by harnessing pretrained vision-to-text matching models instead of embarking on resource-intensive model training from scratch. inspired by prior successes @xcite , our approach expedites question generation by leveraging the knowledge embedded in existing models, thereby enhancing the quality and efficiency of the process. the contributions of this paper are as follows: ## methods the overall framework for vqg is displayed in figure 1. this section introduces our training strategies and inferential relationship abstraction methods. ## conclusion this paper bridges the gap in aligning machinegenerated visual questions, focusing on inferential questions in video vqg. our framework utilizes pre-trained models to enhance event-entity inferential relationships and question generation. we additionally introduce a grounding metric and pro-pose techniques for causal and temporal abstraction. through extensive experiments, we achieve significant improvement across all metrics, highlighting our framework's efficacy in promoting visual content recognition. we underscore the importance of consistent relationships between input frames during training and inference for event-entity relationship inference. this research opens a promising avenue for future work, focusing on methods to enhance consistent frame-based relationships in causal and temporal video inference.