V ideo RAG : Scaling the context size and relevance for video question-answering (2024)

Abstract: Recent advancements have led to the adaptation of several multimodal large language models (LLMs) for critical video-related use cases, particularly in Video Question-Answering (QA). However, most of the previous models sample only a limited number of frames from video due to the context size limit of backbone LLM. Another approach of applying temporal pooling to compress multiple frames, is also shown to saturate and does not scale well. These limitations cause videoQA on long videos to perform very poorly. To address this, we present VideoRAG, a system to utilize recently popularized Retrieval Augmented Generation (RAG) pipeline to select the top-k frames from video, relevant to the user query. We have observed a qualitative improvement in our experiments, indicating a promising direction to pursue. Additionally, our findings indicate that videoRAG demonstrates superior performance when addressing needle-in-the-haystack questions in long videos. Our extensible system allows for trying multiple strategies for indexing, ranking, and adding QA models.

## introduction in the realm of video-based language models (video-llms), the ability to accurately answer questions about long-form video content remains a significant challenge. videorag, a novel system introduced by researchers, aims to address this issue by retrieving more relevant and informative video frames to enhance the context for video-llms. several recent works adapt the pre-trained imagetext multimodal llms to video data, by encoding multiple video frames into a sequence of features. however, this approach is limited by the trade-off between the number of frames and the computation our demo screencast is available at video link cost. pllava @xcite , a sota videoqa model, uniformly samples only 16 frames, which is bound to miss the key details in long videos. straightforward approach of averaging the spatial and temporal dimensions, as implemented in the videochatgpt system @xcite , leads to the loss of substantial spatial information and fails to achieve optimal performance as the training dataset is scaled @xcite . in contrast, videorag employs a retrievalaugmented approach, where the system dynamically retrieves the most relevant video frames to supplement the input to the video-llm. vide-orag allows indexing of multiple video frames using multiple encoders, and searching the top-k most relevant frames given a query. we also release a human-annotated benchmark dataset along with our system, specially curated for long videos. the evaluation of videorag on questionanswering tasks shows that it outperforms baseline video-llms that rely on direct frame averaging or fixed-frame sampling, particularly on long videos. ## videorag system our videorag system leverages standard components of the retrieval-augmented generation (rag) pipeline and image encoding. it is platformagnostic, supporting the addition of various strategies at each component level. the system architecture comprises six main components: video processing we extract the candidate frames from the video, focusing on efficient and scalable processing to manage the memory overhead associated with storing frame information. other aspects of the video are extracted as well, for example the audio signal, and the objects appearing the video. we also process the videos to extract chunks to track the activities happening during these chunks. video encoding entities from video, such as frames, chunks, and other extracted information is encoded into fixed-size vector semantic representations for further indexing. we explore multiple strategies to encode the videos intelligently, including the features like image frames, audio signal, and the objects appearing in the video. we rely on multiple strategies like sentencetransformers @xcite , @xcite , @xcite for encoding the text query. indexing we employ a vector database, to store the frame embeddings effectively. this is a standard component in the rag pipeline. these vector databases allow for efficient lookup algorithms for retrieval of top-k samples. ## usage and evaluation videorag is planned to be used for easy experimentation, and evaluation of long form video question-answering systems. our system allows for easy extension to additional indexing mechanisms, ranking strategies, and videoqa models. we compare our videorag outputs with that of non-rag baseline systems. we curated a dataset of a few long videos and corresponding question-answer pairs.