Exploring Question Guidance and Answer Calibration for Visually Grounded Video Question Answering (2024)

Abstract: Video Question Answering (VideoQA) tasks require not only correct answers but also visual evidence. The “localize-then-answer” strategy, while enhancing accuracy and interpretability, faces challenges due to the lack of temporal localization labels in VideoQA datasets. Existing methods often train the models’ localization capabilities indirectly using QA labels, leading to inaccurate localization. Moreover, our experiments show that despite high accuracy, current models depend too heavily on language shortcuts or spurious correlations with irrelevant visual context. To address these issues, we propose a Question-Guided and Answer-Calibrated TRansformer (QGAC-TR), which guides and calibrates localization using question and option texts without localization labels. Furthermore, we design two self-supervised learning tasks to further enhance the model’s refined localization capabilities. Extensive experiments on three public datasets focused on temporal and causal reasoning show that our model not only achieves accuracy comparable to large-scale pretrained models but also leads in localization aspects. Code will be available on GitHub.

## introduction videos, with their rich information, engaging content, and user connectivity, have become highly sought-after multimedia data @xcite @xcite . however, sifting through extensive video content is time-consuming, making it hard for users to fully understand and find specific content. therefore, there is a growing demand for technology that can quickly locate and answer questions to aid user understanding of video content. recent studies have devoted significant effort to videoqa tasks, including transformer-based mod- (2) for the second question, without the calibration of options for localization, current models @xcite easily focus on the orange area and predict 'feed girl' as the answer. to address the aforementioned challenges, we propose the qgac-tr, which aims to fully utilize the guidance and calibration of questions and options to achieve precise localization of video segments without localization labels, focusing on providing correct answers based on visual content. specifically, after employing a cross-attentionbased transformer encoder to enforce the involvement of question text in extracting video representations, we introduce a question relevance token representing the entire video to serve as a standard for adaptive localization, designing an input-adaptive localizer. subsequently, we designed two selfsupervised learning tasks. the first involves constructing negative pairs by mixing original videoquestion pairs, training the model to suppress the question relevance scores of these negative pairs. through this process, we expect the model to develop the ability to discern relevant video segments. the second task uses the relevance scores generated from video-question pairs as an anchor, treating scores related to correct and incorrect options as positive and negative samples, respectively, for self-supervised contrastive learning. this process aims to more comprehensively and accurately locate video segments, compensating for the shortcomings of the visual clues provided solely by question text. in summary, our contributions are as follows: video temporal grounding. this task closely relates to the localization phase in videoqa. traditional methods are categorized into proposal-based and proposal-free approaches (see appendix a.1 for details). recent advancements include detrbased and regression-based approaches, with methods like umt @xcite incorporating additional audio modalities, and @xcite @xcite developing detr architectures. however, their direct application to videoqa is challenging without localization labels and when questions contain only partial visual cues. language shortcuts in vision-language tasks. in terms of reducing language shortcuts and spurious visual correlations, our research aligns with other domains. for instance, @xcite addressed the issue of language shortcuts in vqa by creating vqav2, which pairs questions with additional images that have similar content but different answers. @xcite and @xcite mitigate this issue by adjusting prediction scores. @xcite developed x2-vlm through multi-level vision-language pretraining to enhance spatial localization. these studies attempt to avoid shortcut learning in image processing by collecting new datasets, employing specialized learning strategies, or focusing on spatial grounding. methods suitable for videoqa remain largely unexplored. model architecture overview. our overall model architecture is shown in figure 3 . (i) the model uses pretrained visual and text encoders to extract features. (ii) tokens along with a learnable relevance token are fed into input-adaptive localizer. two self-supervised learning tasks are also designed, namely mixing different samples of videos and questions for negative pair learning (l neg ), and question-option pairs with correct and incorrect options serving as positive and negative samples for contrastive learning (l cl ). (iii) the questions and localized frames are processed through another cross-attention transformer encoder for qa reasoning and prediction. ## conclusion qgac-tr introduces an innovative approach to videoqa by using question and option texts for guided localization in the absence of labels. it employs a transformer-based architecture with cross-attention and an input-adaptive localizer to enhance localization precision. additionally, two self-supervised learning tasks further improve the model's understanding and localization of video content. the experimental results show that qgac-tr not only achieves competitive qa accuracy but also excels in localization precision. this demonstrates that it deeply exploits the helpfulness of question and option texts for localization, and effectively mitigates the model's reliance on language shortcuts and spurious correlations.