A Simple LLM Framework for Long-Range Video Question-Answering (2024)

Abstract: We present LLoVi, a simple yet effective **L**anguage-based **Lo**ng-range **Vi**deo question-answering (LVQA) framework. Our method decomposes the short- and long-range modeling aspects of LVQA into two stages. First, we use a short-term visual captioner to generate textual descriptions of short video clips (0.5-8 seconds in length) densely sampled from a long input video. Afterward, an LLM aggregates the densely extracted short-term captions to answer a given question. Furthermore, we propose a novel multi-round summarization prompt that asks the LLM first to summarize the noisy short-term visual captions and then answer a given input question. To analyze what makes our simple framework so effective, we thoroughly evaluate various components of our framework. Our empirical analysis reveals that the choice of the visual captioner and LLM is critical for good LVQA performance. The proposed multi-round summarization prompt also leads to a significant LVQA performance boost. Our method achieves the best-reported results on the EgoSchema dataset, best known for very long-form video question-answering. LLoVi also outperforms the previous state-of-the-art by **10.2%** and **6.2%** on NExT-QA and IntentQA for LVQA. Finally, we extend LLoVi to grounded VideoQA, which requires both QA and temporal localization, and show that it outperforms all prior methods on NExT-GQA. Code is available at https://github.com/CeeZh/LLoVi.

## introduction recent years have witnessed remarkable progress in short video understanding (5-15s in length) @xcite @xcite @xcite . however, extending these models to long videos (e.g., several minutes or hours in length) is not trivial due to the need for sophisticated long-range temporal reasoning capabilities. most figure 1 : comparison between llovi (ours) and the recent frozenbilm @xcite ) video qa method. like most prior methods, frozenbilm is best suited for short-range video understanding. thus, as illustrated in the figure , it fails to answer a question that requires reasoning about complex human activities in a long video. in comparison, our method effectively reasons over long temporal extents and produces a correct answer. recently, large language models (llms) have shown impressive capability for long-range reasoning on a wide range of tasks such as document understanding @xcite @xcite and long-horizon planning @xcite @xcite . motivated by these results in the natural language and decision-making domain, we explore using llms for long-range video question answering (lvqa). specifically, we propose llovi, a sim-ple yet effective language-based framework for long-range video understanding. unlike prior longrange video models, our approach does not require specialized long-range video modules (e.g., memory queues, state-space layers, etc.) but instead uses a short-term visual captioner coupled with an llm, thus exploiting the long-range temporal reasoning ability of llms. our simple two-stage framework tackles the lvqa task by decomposing it into short and long-range modeling subproblems: to further enhance the effectiveness of our framework, we also introduce a novel multi-round summarization prompt that asks the llm first to summarize the short-term visual captions and then answer a given question based on the llmgenerated video summary. since the generated captions may be noisy or redundant, such a summarization scheme enables filtering out potentially distracting/irrelevant information and eliminating redundant sentences, which significantly improves the reasoning ability of the llm for lvqa. additionally, we conduct an empirical study on egoschema to investigate the factors behind our framework's success. specifically, we study (i) the selection of a visual captioner, (ii) the choice of an llm, (iii) the llm prompt design, and (iv) optimal video processing configurations. our key empirical findings include: ## method our method, llovi, decomposes lvqa into two subtasks: 1) short-term video clip captioning and 2) long-range text-based video understanding. our decomposed lvqa framework brings several important advantages. first, our approach is simple as it does not rely on complex/specialized longrange video modeling operators (e.g., memory queues, state-space layers, space-time graphs, etc.). second, our framework is training-free, which makes it easy to apply it to lvqa in zero-shot settings. third, our framework enables us to leverage the strong existing short-term visual captioners (e.g., lavila, llava) and powerful zero-shot llms (e.g., gpt-3.5, gpt-4, llama). fourth, our method is highly flexible, i.e., it can incorporate various visual captioners and llms, and also benefit from future improvements in visual captioning/llm model design. figure 2 presents a detailed illustration of our high-level approach. be- ## conclusion in this work, we present a simple, yet highly effective llm-based framework for long-range video question-answering (lvqa). our framework outperforms all prior models on the newly introduced egoschema benchmark. furthermore, we demonstrate that our approach generalizes to other lvqa benchmarks such as next-qa, intentqa, and it can also be extended to grounded lvqa tasks. lastly, we thoroughly evaluate various design choices of our approach and analyze the key factors behind the success of our method. we hope that our simple lvqa framework will help inspire new ideas and simplify model design in long-range video understanding.