V ideo RAG : Retrieval-Augmented Generation over Video Corpus (2025)

Abstract: Retrieval-Augmented Generation (RAG) is a powerful strategy for improving the factual accuracy of models by retrieving external knowledge relevant to queries and incorporating it into the generation process. However, existing approaches primarily focus on text, with some recent advancements considering images, and they largely overlook videos, a rich source of multimodal knowledge capable of representing contextual details more effectively than any other modality. Also, while very recent studies explore the use of videos in response generation, they either predefine query-associated videos without retrieval or convert videos into textual descriptions, losing multimodal richness. To tackle these, we introduce VideoRAG, a novel framework that not only dynamically retrieves videos based on their relevance with queries but also utilizes both visual and textual information. The operation of VideoRAG is powered by recent Large Video Language Models (LVLMs), which enable the direct processing of video content to represent it for retrieval and the seamless integration of retrieved videos jointly with queries for response generation. Also, inspired by that the context size of LVLMs may not be sufficient to process all frames in extremely long videos and not all frames are equally important, we introduce a video frame selection mechanism to extract the most informative subset of frames, along with a strategy to extract textual information from videos (as it can aid the understanding of video content) when their subtitles are not available. We experimentally validate the effectiveness of VideoRAG, showcasing that it is superior to relevant baselines. Our code is available at https://github.com/starsuzi/VideoRAG.

## introduction recently, large foundation models, such as large language models and their extension to the vision * equal contribution modality called large vision-language models, have become the standard for addressing diverse tasks due to their remarkable capabilities @xcite @xcite . in particular, these models, trained on extensive textual and multimodal corpora, encode vast amounts of knowledge within their large-scale parameters. however, they are still prone to generating factually incorrect outputs, as their parametric knowledge can be inaccurate or outdated @xcite . this limitation highlights the need for incorporating knowledge from external knowledge sources, with retrieval-augmented generation (rag) emerging as an essential mitigator for it. specifically, rag typically operates by retrieving query-relevant information and then generating answers grounded in the retrieved content @xcite . however, while existing rag approaches have been widely adopted for various real-world appli-cations, they have primarily focused on retrieving and incorporating textual content @xcite , with only recent attempts beginning to explore images (or text-image pairs) as the additional source of external knowledge @xcite . on the other hand, we argue that there remains a rapidly expanding yet underutilized medium, called videos, which provides unparalleled multimodal richness and might be a compelling resource for augmenting the knowledge landscape of current rag systems. specifically, videos combine temporal dynamics, spatial details, and multimodal cues, which collectively enable them to capture complex processes, context-dependent interactions, and non-verbal signals that static modalities (e.g., text and images) often fail to convey. moreover, given the increasing popularity of video-sharing platforms (such as youtube), the availability of diverse, high-quality video data has grown, ranging from educational tutorials and scientific demonstrations to personal experiences and real-time events, all of which may be useful when formulating responses to user queries. a few recent studies have started considering video content to handle user queries; however, they have limitations. for instance, some assume that videos relevant to queries are already known and instead focus on identifying query-relevant frames within that specified video @xcite . while effective in scenarios where the relevant video is explicitly provided, it is suboptimal for more general-use cases, where users expect systems to dynamically identify and retrieve videos to provide answers. on the other hand, other studies handle videos by converting them into textual formats, such as subtitles, and utilizing these textual representations under off-the-shelf text-based rag pipelines @xcite . however, while this text-only strategy may offer a convenient workaround, it inherently sacrifices the multimodal richness of video data by discarding critical information, such as temporal dynamics captured in the visual context, during the conversion process. for example, consider a query: "how does the expression of the dog change when it is angry?". while textual transcriptions might describe the dog's barking or growling, they fail to capture visual cues (baring teeth, raised hackles, or narrowed eyes), which are needed for accurately interpreting the emotional state of the dog and subsequently formulating the answer to the query. to address the aforementioned limitations, we introduce a novel framework, called videorag, which aims to offer another fruitful angle to existing rag frameworks by enabling a more comprehensive utilization of video content for its holistic retrieval and incorporation (see figure 1 ). specifically, in response to queries, the proposed vide-orag retrieves relevant videos from a large video corpus but also integrates both visual and textual elements into the answer-generation process. also, we operationalize this by harnessing the advanced capabilities of recent large video language models (lvlms), which are capable of directly processing video content, consisting of visual and textual information, within the unified framework, thereby more effectively capturing its multimodal richness. however, there exist a couple of remaining challenges in integrating videos into rag frameworks. first, videos are inherently long and redundant, oftentimes making it infeasible for lvlms to process all frames due to their limited context capacity as well as unnecessary since not all frames contribute meaningfully for retrieval and generation. to address this, we introduce a frame selection model that is trained to extract the most informative subset of frames to maximize retrieval and generation performance. also, we observe that, while the joint utilization of visual and textual features is needed for the effective representation of videos and subsequently their retrieval, the textual descriptions of videos (e.g., subtitles) are oftentimes not available. to tackle this, we further present a simple yet effective mitigation strategy that utilizes automatic speech recognition techniques to generate textual transcripts from videos, allowing us to leverage both visual and textual modalities for every video. to validate the effectiveness of videorag, we conduct experiments by using overlapping queries from the wikihowqa dataset (bolotova-baranova et al., 2023) (consisting of query-answer pairs) and the howto100m dataset @xcite (including query-video pairs without answers). also, based on this, we automatically collect the dataset for rag over videos and then evaluate models on it. then, the experimental results show the significant performance improvement of the proposed vide-orag framework over relevant baselines, demonstrating the efficacy of leveraging videos for rag. ## method we present videorag that retrieves query-relevant videos and generates responses grounded in them. ## experiment we now describe experimental setup and results. ## related work retrieval-augmented generation rag is a strategy that combines retrieval and generation processes to produce accurate answers by grounding them in external knowledge @xcite . to be specific, during the retrieval step, documents (relevant to queries) are selected from a large corpus by calculating their similarity to the query, which can be done with retrievers @xcite @xcite . in the generation step, these retrieved documents serve as input for generating answers that are rooted in the provided information @xcite @xcite , with some advancements using iterative retrieval-generation cycles @xcite or adapting different rag strategies based on query complexity @xcite . however, despite the fact that much of the real-world knowledge is inherently multimodal in nature @xcite @xcite , the majority of rag studies have focused on the textual modality, with little effort on incorporating images, leaving a significant gap in leveraging the full spectrum of available knowledge for the holistic operation of rag. multimodal rag there has been growing interest in expanding rag to incorporate multimodal information (beyond text), such as images @xcite @xcite , code @xcite , tables @xcite , and audio @xcite . however, unlike them, videos offer a unique and orthogonal advantage for rag, as they encapsulate temporal dynamics, spatial details, and multimodal cues in ways unmatched by other modalities. inspired by this fact, very recent studies have started exploring the usage of video content within rag pipelines; however, existing approaches leverage it in a suboptimal way. to be specific, some focus on extracting queryrelevant frames from the preselected video and generating answers based on them, which, while useful in controlled scenarios, limits their real-world applicability in open-domain settings @xcite . also, some other studies attempt to sidestep the complexity of handling video data by converting it into textual representations (such as subtitles or captions); however, while directly applicable to existing text-based rag frameworks, they sacrifice the multimodal richness embedded within videos (such as temporal dynamics and spatial patterns) @xcite @xcite . to address these, we propose videorag which is capable of dynamically retrieving and holistically utilizing video content in rag, powered by lvlms discussed next. large video language models building on the remarkable success of llms @xcite @xcite @xcite , there has been a growing interest in extending them to encompass diverse modalities, such as images @xcite @xcite and code @xcite . also, this expansion has recently extended to another modality called video, leading to the emergence of lvlms that are capable of directly processing video content. they excel in solving traditionally challenging (yet straightforward) tasks, such as object or action detection, and their capabilities have rapidly advanced to tackle more challenging tasks, such as analyzing spatio-temporal dynamics to predict event sequences, inferring causal relationships, and generating context-aware descriptions of intricate scenarios @xcite @xcite @xcite @xcite , even in zero-shot settings @xcite . however, their potential has yet to be explored in the context of rag; thus, in this work, we aim to bridge this gap with videorag. ## conclusion we presented videorag, a framework that expands the current landscape of rag by leveraging a video corpus as the external knowledge source. specifically, unlike existing works that use the textual representations of videos or assume the existence of query-relevant videos without retrieval, the proposed videorag retrieves videos based on their relevance to queries but also integrates their multimodal richness (including visual and textual elements) into the rag pipeline, with adaptive frame selection to leverage only the most informative subset of full frames for effectiveness and efficiency. also, through comprehensive analyses, we demonstrated how the inclusion of visual or textual features, or a combination of both, improves retrieval and generation performance, and, inspired by the critical role of textual features (for retrieval quality) but their absence in some videos, we presented a simple yet effective mitigator that uses automatic speech recognition to generate textual transcripts. overall, experimental results validated the superiority of our videorag over existing rag methods, and we believe it makes a significant step toward holistic rag systems that can utilize videos.