O m A gent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer (2024)

Abstract: Recent advancements in Large Language Models (LLMs) have expanded their capabilities to multimodal contexts, including comprehensive video understanding. However, processing extensive videos such as 24-hour CCTV footage or full-length films presents significant challenges due to the vast data and processing demands. Traditional methods, like extracting key frames or converting frames to text, often result in substantial information loss. To address these shortcomings, we develop OmAgent, efficiently stores and retrieves relevant video frames for specific queries, preserving the detailed content of videos. Additionally, it features an Divide-and-Conquer Loop capable of autonomous reasoning, dynamically invoking APIs and tools to enhance query processing and accuracy. This approach ensures robust video understanding, significantly reducing information loss. Experimental results affirm OmAgent’s efficacy in handling various types of videos and complex tasks. Moreover, we have endowed it with greater autonomy and a robust tool-calling system, enabling it to accomplish even more intricate tasks.

## introduction large language models (llms) have advanced remarkably in recent years, greatly expanding their capabilities across various applications @xcite @xcite . as these models have evolved, they have been increasingly applied to multimodal contexts, allowing them to process and interpret not just text but also images and other media types. initially, the focus was on single images, utilizing the models' ability to generate and understand detailed descriptions or responses based on static visuals @xcite @xcite however, as llms have become more complex and powerful, there has been growing interest in applying them to more dynamic media, such as video content @xcite @xcite @xcite . this interest arises from the potential to provide deeper and more nuanced interpretations of video data, similar to how humans understand and interact with moving images and sound. currently, most video understanding models are limited to processing short videos, typically only a few minutes or even several seconds long. despite these advancements, significant challenges remain, especially in handling long video inputs like 24-hour cctv footage or full-length movies, which involve massive amounts of data and require substantial processing power. traditionally, one solution has been to extract key frames from these long videos or convert all frames into textual descriptions before processing @xcite . while this approach makes the task more manageable for llms, it often results in information loss. key frame extraction might miss subtle but important details in the omitted frames, and converting visual data to text can oversimplify or misrepresent visual nuances, leading to a less accurate understanding of the content. to address these limitations, the application of retrieval-augmented generation (rag) technology in video understanding has emerged as a promising solution @xcite @xcite . rag enables the storage and efficient retrieval of video frames based on their relevance to a query. this method allows for more precise and contextual responses by referencing specific content directly from the video, rather than relying on potentially incomplete or inaccurate textual summaries. however, given that video is a continuous stream of information, when it is stored, this continuous flow is segmented into discrete blocks of data. the information lost during this segmentation is irretrievable. to address the aforementioned issues, we attempt to analyze the human approach to handling complex, long video question-answering tasks, seeking breakthroughs from this perspective. when a person watches a content-rich, lengthy video, such as a movie, they retain a general impression of the movie in their mind. this impression includes a rough outline of the video's content at various time points. when asked about specific details, the person may not recall the details immediately but can quickly locate the relevant time point in the video and rewatch the segment to retrieve the missing information. the key insight of omagent is to replicate this process by integrating multimodal rag and generalist ai agent. omagent consists of two main components: (1) a video2rag video preprocessor to extract and store the generalized information from the video, akin to the foundational impression a video imprints upon the viewer's memory. (2) a divide-and-conquer loop (dnc loop) for task planning and execution which equipped with tool invocation capabilities. we abstract the human ability to reposition and review video details as a tool named "rewinder," which can be autonomously selected and utilized by an ai agent, similar to how a person might use a video player's progress bar to navigate to points of interest. omagent not only can retrieve detailed information from videos but also can actively seek external information, enabling more advanced video understanding and question-answering. existing benchmarks are insufficient to accurately quantify these capabilities, so we propose a new complex video understanding benchmark to fulfill this task. the contributions are: (1) omagent, the first complex video understanding framework integrating multimodal rag and generalist ai agent. (2) a benchmark dataset that contains 2000+ q&a pairs for evaluating video understanding systems. (3) experiments that shows the proposed agentic method is able to outperform strong baselines for solving complex video understanding problems. ## related work video llms analyzing and understanding video content using large-scale language models (llms) typically involves fine-tuning or pretraining methods. pre-training strategies, such as supervised or contrastive learning, develop video llms, while instruction fine-tuning updates adapter parameters to enable video comprehen-sion @xcite . for example, lavila @xcite enhances video subtitle generation through a cross-attention module and rewriting mechanism, improving coverage and diversity. video-llama @xcite addresses spatio-temporal visual variations using separate video and audio encoders with an advanced audiovisual q-former, significantly boosting video comprehension. video-llava @xcite connects multimodal representations into a unified semantic space with llms, improving video understanding tasks. however, these methods often consume a lot of computational resources and time during the training process, and the models can usually only target specific tasks related to the training data. in addition, video llms trained from scratch may not be able to achieve the expected performance when dealing with longer or previously unseen videos, showing shortcomings in understanding long videos and dealing with complex video question-answer tasks. ## method the process of omagent's video understanding can be bifurcated into two primary parts: video2rag and dnc loop. as illustrated in figure 1 , all video data must undergo preliminary processing before being stored in the knowledge database in preparation for subsequent tasks. the preprocessing phase of video2rag encompasses a series of model identification and vectorization procedures, culminating in the extraction of the core content of the video files for storage. when undertaking video understanding tasks, the initial step is to extract temporal information from the query. this information will then be used to filter the retrieved results. subsequently, the query is encoded by text encoder, and the embedding is employed to retrieve pertinent video segment information from the knowledge database. the retrieved video clip information and the original task will be transmitted to dnc loop, the intelligent agent capable of autonomously planning and executing tasks, for processing. complex tasks will be recursively subdivided into executable subtasks. if at any point the agent deems that specific video details need to be reviewed, it will utilize the rewind tool to examine the relevant content. once all subtasks are successfully completed, the execution results will be conveyed to a node dedicated to synthesizing the final answer. ## experimental settings to validate the efficacy of the omagent system in addressing complex problems within real-world scenarios, we have designed a two-phase experimental approach: ## conclusion omagent is a powerful video comprehension agent that integrates multimodal rag with a generalist ai agent, enabling several advanced capabilities. it offers a theoretical near-infinite length video understanding capacity and incorporates a secondary recall mechanism for detailed video information, which significantly mitigates information loss. additionally, omagent autonomously invokes tools based on video comprehension tasks, allowing it to execute more intricate operations. these capabilities have resulted in remarkable performance on the mbpp, freshqa, and our proposed long video complex task test datasets.