Large Language Models are Temporal and Causal Reasoners for Video Question Answering (2023)

Abstract: Large Language Models (LLMs) have shown remarkable performances on a wide range of natural language understanding and generation tasks. We observe that the LLMs provide effective priors in exploiting \textit{linguistic shortcuts} for temporal and causal reasoning in Video Question Answering (VideoQA). However, such priors often cause suboptimal results on VideoQA by leading the model to over-rely on questions, \textit{i.e.} , \textit{linguistic bias} , while ignoring visual content. This is also known as ‘ungrounded guesses’ or ‘hallucinations’. To address this problem while leveraging LLMs’ prior on VideoQA, we propose a novel framework, Flipped-VQA, encouraging the model to predict all the combinations of \langle V, Q, A \rangle triplet by flipping the source pair and the target label to understand their complex relationships, \textit{i.e.} , predict A, Q, and V given a VQ, VA, and QA pairs, respectively. In this paper, we develop LLaMA-VQA by applying Flipped-VQA to LLaMA, and it outperforms both LLMs-based and non-LLMs-based models on five challenging VideoQA benchmarks. Furthermore, our Flipped-VQA is a general framework that is applicable to various LLMs (OPT and GPT-J) and consistently improves their performances. We empirically demonstrate that Flipped-VQA not only enhances the exploitation of linguistic shortcuts but also mitigates the linguistic bias, which causes incorrect answers over-relying on the question. Code is available at https://github.com/mlvlab/Flipped-VQA.

## introduction large language models (llms) have exhibited an impressive ability to free-form generation tasks and multi-choice question-answering tasks in natural language processing @xcite @xcite @xcite . these llms have achieved human-level performance on a wide range of challenging tasks like professional & academic qa @xcite , science qa @xcite , mathematics qa @xcite , code generation @xcite , and commonsense reasoning @xcite since they are pretrained with largescale corpora (e.g., commoncrawl, bookcorpus, and wikipedia) which entail massive human knowledge. with such pretraining data, usually comprising a series of contexts, llms are trained to predict the next token given the preceding tokens. therefore, llms learn to predict the next context given a series of contexts during pretraining, so they implicitly learn temporal and causal reasoning ability. to assess llms' temporal and causal reasoning ability, we explore a popular multi-modal understanding task, video question answering (videoqa), which requires the model to predict the correct answer (a) given a video (v) and question (q) pair. recent challenging videoqa benchmarks demand the model to answer the question which asks temporal and causal relationships, e.g., the next event of a video or the reason why a scene happens. we observe that llms effectively handle such challenging videoqa benchmarks by leveraging their strong prior knowledge of temporal and causal reasoning learned from the pretraining phase. for example, in fig. 1a , llms correctly answer causal questions solely based on the text question and options without referring to the visual content by exploiting linguistic shortcut. also, fig. 1b shows that a language-only qa model equipped with a larger language model, e.g., llama 33b, denoted by qa 33b outperforms a videoqa model with opt 125m trained with full ⟨v, q, a⟩ on causal and temporal questions by a large margin of 13%. although llms' prior knowledge is effective for addressing complex temporal and causal questions, this sometimes leads to suboptimal answers when the model overly depends on inaccurate linguistic prior, i.e., linguistic bias, while ignoring the visual content. this is known as the 'hallucination problem' of visual question-answering models equipped with llms @xcite . although in the literature linguistic shortcut and linguistic bias are interchangeably used, in this paper, we use the former specifically when the linguistic prior is correct and the latter otherwise. here, we propose a novel learning framework, flipped-vqa, predicting all the combinations of ⟨v, q, a⟩ triplet by flipping the source pair and the target label, i.e., @xmath0, @xmath1, and @xmath2. in other words, to understand complex relationships between the video, question, and answer, llms are asked to additionally predict the question given a video-answer pair and the video given a questionanswer pair by leveraging their knowledge of temporal and causal reasoning. in our experiments, we develop llama-vqa by applying flipped-vqa to @xcite and it outperforms other baselines on five challenging videoqa benchmark datasets: next-qa @xcite , star @xcite , dramaqa @xcite , vlep @xcite , and tvqa @xcite with a small number of learnable parameters (only 0.06% of total model parameters). furthermore, flipped-vqa improves the performance of gpt-j @xcite and opt @xcite , implying that our framework is generally applicable to other decoder-only llms. we empirically demonstrate that flipped-vqa encourages llms to exploit linguistic shortcuts by leveraging their prior knowledge and also mitigates linguistic bias which causes incorrect answer over-relying on the question. to sum up, our contributions are as follows: ## related works llms for temporal and causal reasoning. exposed to a wide range of corpora during pretraining, llms perform diverse reasoning tasks @xcite @xcite @xcite @xcite . particularly, a line of works @xcite @xcite @xcite focuses on the temporal and causal reasoning skills of llms. for temporal reasoning, @xcite assesses llms by asking opendomain time-sensitive questions in both closed and open settings. similarly, hobbhahn et al. (2022) investigates the ability through querying subject and relation pairs of different time periods. furthermore, some works have also explored the causal capabilities by evaluating whether llms can understand the causal implications given in the sentence. long et al. (2023) uses simple causal graphs and determines if llms can understand the relationship between nodes. in this work, we further examine the temporal and causal reasoning skills of llms expanded to the multi-modal setting of challenging videoqa. llms for multi-modal understanding. various lines of work attempt to incorporate different modalities into llms to leverage the models' generation power and knowledge in performing multimodal downstream tasks. there exist various techniques @xcite @xcite in the literature to bridge the gap between different modalities. for instance, flamingo @xcite ingests visual content into the frozen chinchilla @xcite through a perceiver resampler. llama-adapter @xcite fine-tunes llama by applying linear projection along with the adaption of prompts to incorporate the visual information. recently, there have been several approaches to develop a llmsbased video chat model trained on massive multimodal instruction tuning dataset that enables comprehensive understanding across different modalities @xcite . specifically, videochat @xcite proposes video-centric instruction dataset that primarily emphasizes spatio-temporal reasoning and causal rela-tionships present in the video. ## method we present flipped-vqa, a simple yet effective framework for video question answering (videoqa) that leverages the llms' prior knowl-edge of temporal and causal reasoning. in addition to the target task of predicting an answer a given a video v and a question q (i.e., @xmath3, our framework flips the role of inputs and outputs, requiring the model to predict v given qa and q given va. we apply our framework to @xcite and develop llama-vqa but it is applicable to any decoderonly llms. in this section, we first describe the overall architecture of llama-vqa and then introduce our objective. the overall architecture is illustrated in fig. 2 . ## experiments we verify the effectiveness of our framework to leverage the powerful prior knowledge induced by an llm. for a thorough analysis, our framework is applied to various llms: llama (7b, 13b, and 33b), opt (125m ∼ 6.7b), gpt-j (6b). we conduct experiments and analyses to answer the following research questions: q1. do llms possess the knowledge of temporal and causal reasoning? q2. is flipped-vqa effective for dealing with challenging videoqa? q3. how does flipped-vqa alleviate linguistic bias? datasets. we experiment on five multiple-choice videoqa benchmark datasets (next-qa, star, dramaqa, vlep, and tvqa) which require challenging temporal and causal reasoning abilities. further experimental settings and implementation details are provided in sec. a and sec. b. ## conclusion in this paper, we investigate the large language models' (llms') temporal and causal reasoning abilities on the challenging multi-modal video question answering (videoqa) task. we observe that the larger llms possess more powerful prior knowledge of temporal and causal reasoning in addressing complex videoqa. moreover, we propose a novel framework, flipped-vqa, that effectively leverages the llms' knowledge of temporal and causal reasoning on understanding the complex ⟨v, q, a⟩ triplet by introducing three generative objectives: l vqa , l vaq , and l qav . our in-depth analyses show that flipped-vqa not only enhances the exploitation of linguistic shortcuts but also mitigates linguistic bias that causes hallucination and ungrounded guess problems. acknowledgments. this work was partly supported by ict creative consilience program (iitp-2023-2020-0-01819) supervised by the iitp, the national research foundation of korea (nrf) grant funded by the korea government (msit) (nrf-2023r1a2c2005373), and kakaobrain corporation.