T rave LER : A Modular Multi- LMM Agent Framework for Video Question-Answering (2024)

Abstract: Recently, image-based Large Multimodal Models (LMMs) have made significant progress in video question-answering (VideoQA) using a frame-wise approach by leveraging large-scale pretraining in a zero-shot manner. Nevertheless, these models need to be capable of finding relevant information, extracting it, and answering the question simultaneously. Currently, existing methods perform all of these steps in a single pass without being able to adapt if insufficient or incorrect information is collected. To overcome this, we introduce a modular multi-LMM agent framework based on several agents with different roles, instructed by a Planner agent that updates its instructions using shared feedback from the other agents. Specifically, we propose TraveLER, a method that can create a plan to "**Trave**rse” through the video, ask questions about individual frames to "**L**ocate” and store key information, and then "**E**valuate” if there is enough information to answer the question. Finally, if there is not enough information, our method is able to "**R**eplan” based on its collected knowledge. Through extensive experiments, we find that the proposed TraveLER approach improves performance on several VideoQA benchmarks without the need to fine-tune on specific datasets. Our code is available at https://github.com/traveler-framework/TraveLER.

## introduction over the last few years, large multimodal models (lmms) have demonstrated tremendous progress in the area of video understanding, particularly for the video question-answering (videoqa) domain @xcite . more recently, lmms have been able to achieve impressive results through video-based models @xcite @xcite . however, video models require a high level * equal contribution. of computational complexity to fine-tune, and annotations are difficult and expensive to collect. as a result, many recent approaches @xcite @xcite operate on the frame level, leveraging large-scale image-based pretrained models in a zero-shot manner. moreover, these models may need to do several tasks simultaneously in a single step. in particular, they should identify the correct events in videos by understanding what information is relevant and ignoring irrelevant information. next, they would need to extract specific and question-relevant visual details and use them to answer the question. as such, one iteration might not be enough to collect all the necessary information. for example, many current approaches use simple captioning, which is often too general to extract specific details, or they might miss important events. in these cases, current approaches cannot revisit the video to find additional information. decomposing this process into different components allows each component to adapt to newly collected information. to address this, we introduce a modular multi-lmm agent framework for videoqa. a planner agent instructs agents in charge of different tasks, such as navigating through the video, extracting visual information through a question-answering process, and reviewing this information to select an answer. feedback from the agents is then passed on to each other and back to the planner, who uses the feedback to update its instructions. in this way, we can revisit the video to narrow our focus or expand our search to more relevant information and extract specific details to answer the question. consider the example in figure 1 . suppose we are asked why the boy turned over in the middle of the video. in the first iteration, our method uses temporal cues from the question to skip to the middle of the video and asks questions to find the relevant frames. in the next iteration, we gather more information. asking about what the boy is doing, traveler creates a plan (in blue) to "traverse" (in orange) through the video, asks questions regarding individual frames (in yellow) to "locate" and store key information and, "evaluates" whether there is sufficient information to answer the question (in green), and "replans" using past collected knowledge if there is not enough information. ## traveler framework we begin by describing some background on the llm and lmm architectures (section 3.1), then introduce each component of our framework (section 3.2) and implementation details (section 3.3). our method is shown in figure 2 . ## evaluation we evaluated our traveler framework on several benchmarks described in section 4.1, and compared it to multiple baselines in section 4.2. the results and ablations are in section 4.3 and sec-tion 4.4. additional results and ablations are in the supplementary in section a. ## conclusion we design a modular, multi-lmm agent framework for video-question answering based on several agents with different roles, instructed by a planner agent that updates its instructions using shared feedback between the other agents. our method creates a plan to "traverse" through the video, asking questions about individual frames to "locate" and store key information, and then "evaluate" if there is enough information to answer the question, "replanning" using new feedback if necessary. through extensive experiments and ablations, we find that the proposed traveler approach is not only easy to employ with different models but also improves performance on several video questionanswering benchmarks without the need to finetune on specific datasets.