V ideo LLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format (2025)

Abstract: Recent researches on video large language models (VideoLLM) predominantly focus on model architectures and training datasets, leaving the interaction format between the user and the model under-explored. In existing works, users often interact with VideoLLMs by using the entire video and a query as input, after which the model generates a response. This interaction format constrains the application of VideoLLMs in scenarios such as live-streaming comprehension where videos do not end and responses are required in a real-time manner, and also results in unsatisfactory performance on time-sensitive tasks that requires localizing video segments. In this paper, we focus on a video-text duet interaction format. This interaction format is characterized by the continuous playback of the video, and both the user and the model can insert their text messages at any position during the video playback. When a text message ends, the video continues to play, akin to the alternative of two performers in a duet. We construct MMDuetIT, a video-text training dataset designed to adapt VideoLLMs to video-text duet interaction format. We also introduce the Multi-Answer Grounded Video Question Answering (MAGQA) task to benchmark the real-time response ability of VideoLLMs. Trained on MMDuetIT, MMDuet demonstrates that adopting the video-text duet interaction format enables the model to achieve significant improvements in various time-sensitive tasks (76% CIDEr on YouCook2 dense video captioning, 90% mAP on QVHighlights highlight detection and 25% R@0.5 on Charades-STA temporal video grounding) with minimal training efforts, and also enable VideoLLMs to reply in a real-time manner as the video plays.

## introduction videos are becoming an increasingly important medium to acquire information on a daily basis. powered by recent advancements in large language models (llms) @xcite @xcite @xcite and vision encoders @xcite @xcite @xcite , several video large language models (videollm) @xcite @xcite @xcite have already demonstrated strong abilities for holding conversations and answering questions about videos. a common feature of these models is using visual encoders to encode all frames sampled from the entire video at first, and integrate them into text input by concatenating them to input embeddings or using cross attention. recent research on videollms has primarily concentrated on model architectures and training datasets, with limited exploration of the interaction format between the user and the model. in this paper, the "interaction format" of videollms comprises the following two aspects: (1) a chat template used to convert input sources, e.g., video, user text query, and model response, into a sequence of tokens; (2) a turn-taking rule organizing inputs of different sources to finalize an interaction format. for example, for most existing videollms, the interaction format is: (1) for the chat template, the model uses (frames sampled from) the full video and a text query as input, and then outputs a response; (2) for the turn-taking rule, usually the model is permitted to take its turn to generate a response when both the whole video content and user query have ended, e.g., when @xmath0. we refer to this traditional interaction method as "whole video" in the rest of this paper. however, this all-along used whole video interaction has the following two defects, which hinder the performance and real-world usage scenarios of videollms: firstly, it does not admit timely interactions. as the video is often input as a whole, this … … what is happening in the nasa space center? video keeps playing assistant decides whether to start a response after each frame ? a woman is working in the lab. video keeps playing ? a man is standing on a stage. user assistant user assistant whole video interaction format video-text duet interaction format figure 1: an example of the common whole video interaction format and our video-text duet interaction format. in this work, we formalize the video-text duet interaction format, an interaction method that aims to enhance videollms by addressing the aforementioned issues. an illustration of the whole video interaction format and the video-text duet interaction format is shown in fig. 1 . with our video-text duet interaction format, the video is continuously played and input to the model frame-byframe. both the user and model can insert their text messages right after any frame during the video play. when a dialogue turn from either the user or the model ends, the video stream can have the floor and input video frames to the model until another turn is started by either the user or the model, akin to the show of two performers in a duet. this improves the timeliness of interaction and better suits real-world applications such as live-streaming or surveillance video comprehension. moreover, by inserting responses to the video where is most relevant, the model can learn to generate responses by referencing a smaller but fine-grained fraction of the video before this position. in this manner, it facilitates information retrieval to describe lengthy videos, as well as enables a response to be "grounded" at the targeted position of the video. we believe this design contributes to addressing the above discussed issues of existing whole video videollms. to prove the effectiveness of the video-text duet interaction format, we construct mmduetit, a dataset to facilitate the training of a versatile vide-ollm following the video-text duet interaction format. we propose multi-answer video grounded qa (magqa), a novel task that requires the model to generate answers at appropriate timespans in a real-time manner to align with potential applications of live-streaming video comprehension. we also train mmduet, a videollm that implements our proposed video-text duet interaction format. initialized with llava-onevision @xcite and trained with mmduetit at a low cost, mmduet achieves significant performance improvement in various time-sensitive tasks, and is able to generate responses in real-time as the video plays. ## related works the advancement of large language models (llms) and visual encoders has led to numerous efforts on their integration, aiming to utilize the powerful understanding and generation abilities of existing llms for video-related tasks @xcite @xcite @xcite . these models exhibit a decent ability of video understanding such as captioning or summarizing @xcite . however, their performance on time-sensitive tasks is still unsatisfactory. recent works attempt to empower videollms with the ability to localize and represent segments in videos, and thus achieve better performance on tasks like temporal video grounding or dense video captioning. these works explore new ways on how to easily represent video clips with texts, such as second numbers of timestamp (timechat @xcite ), timeline percentage (vtimellm @xcite ) or using special textual tokens @xcite , grounded-videollm @xcite ). however, their performance has not been satisfactory yet, possibly due to llms' limited ability to accurately count and generate numbers @xcite to localize each video frame. to alleviate this issue, hawkeye @xcite uses a coarsegrained method by referring to a larger fraction of the video, but it requires multiple rounds of recursive grounding to precisely locate a segment and may not express multiple segments at a time. the work most similar to our motivation is videollm-online (chen et al., 2024), which proposes a framework named live for training vide-ollms to interrupt video streams and insert responses. however, they only finetune a model on ego4d @xcite and coin @xcite to demonstrate the live training and inference, and do not explore on how the model capabilities vary with this new type of interaction, especially the zero-shot performance on timesensitive tasks. our work differs from videollm-online at: firstly, providing a more general description of the video-text dual interaction format, including a wider variety of criteria for determining whether a response should be generated, and its application on new tasks such as temporal video grounding and grounded question answering; secondly, introducing a new dataset mmduetit and the method on building such datasets; thirdly, proposing a new task magqa; lastly, proposing a more powerful model mmduet that has state-of-the-art performance on various time-sensitive tasks and zeroshot generalization ability. ## the video-text duet interaction format in section 1, we have defined the concept of "interaction format" with two aspects (i.e., chat template & turn-taking rule), as well as the drawbacks of the commonly-used whole video interaction format. now we re-emphasize and formalize our videotext duet interaction format, which is completely different from previous to implement videollms. ## mmduetit: dataset for training mmduet we build mmduetit, a dataset for training the mm-duet model to learn to calculate the informative and relevance scores, and autonomously output replies at any necessary time in the play of the video. mmduetit is composed of three different types of tasks that benefit our model training: dense captioning, multi-answer grounded video question answering, and temporal video grounding. an example of the input format for each task is listed in appendix d. ## experiments implementations mmduet is initialized with llava-onevision @xcite . we train the model on mmduetit for one epoch. the training takes about one day on a node with 8 tesla v100 gpus, and the inference runs on 1 tesla v100 gpu. more implementation details are listed in appendix b.2. baselines as mmduet mainly focuses on timesensitive video tasks, we use the following baselines that are able to represent time spans in videos by different representation formats: timechat @xcite , vtimellm (7b) @xcite , hawkeye @xcite , vtg-llm @xcite , and videollm-online (chen et al., 2024). for videollm-online, we experimented with θ ∈ {0.5, 0.6, 0.7, 0.8} as suggested in their paper and report the best results (0.8 for both dense video captioning and magqa). since the initialization of mmduet is stronger than that of the baselines, for a fair comparison we also conduct a controlled experiment in which the only difference is the interaction format. specifically, we use the same initialization model (llava-onevision), training data (mmduetit) and training schedule, but reformat the data to the respective interaction formats and video segment representation formats used by timechat and vtimellm to train two baseline models. we name these models as llava-ov-tc and llava-ov-vt. ## conclusion in this paper, we first formalize the video-text duet interaction format. we collect mmduetit for training models to follow the video-text duet interaction format. based on mmduetit we train mmduet, a model with significant improvements on various time-sensitive tasks and is able to automatically decide when to response in a real-time manner. we believe such improvements can be a substantial step towards building powerful and useful video comprehension applications.