Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Support for Streaming Video Benchmarks with Multi-Round Generation
Root cause: Unimplemented Feature Gap

Hello maintainers and contributors,
First of all, thank you for creating and maintaining lmms-eval, we really appreciate your work!

I’m currently working on integrating streaming video benchmarks such as OVOBench and StreamingBench into the framework. These benchmarks often require multi-round generation, where a model interacts with a sequence of video chunks over time, producing responses iteratively based on evolving context.
However, I’ve noticed two key limitations:
1. For simple-type models, almost none implement the generate_until_multi_round method, which is needed to support iterative generation.
2. For chat-type models, while multi-round interaction is conceptually supported, the ConfigurableMessagesTask interface (specifically the construct_requests method) does not currently expose or facilitate multi-round generation at the task level.

Are there any plans to officially support streaming video benchmarks and multi-round generation patterns in lmms-eval?
If so, what would be the recommended approach for implementing this? Are there any design guidelines or ongoing efforts I could align with?
Thank you again for your amazing work!

Hi, may I ask how would the multi turn benchmark looks like in QA side? I think some how needs to format the messages into a Chat Messages used in chat model and iteratively let the model to generate in the generate multi-round generation

If we’re talking about streaming video benchmarks, the typical request construction logic is as follows:
For each sample that requires multi-round generation, a long video is split into chunks according to the required context window.
The model receives each chunk along with a question — for example, one asking about an event that should occur in the video.
Responses from the model for each chunk are then aggregated across the entire sample.
The final score for that sample depends on how timely the model’s answer was relative to the target question.

I feel this some how can be reformatted as a single round QA task type and aggregate the score later under current lmms-eval framework. Because you can split the video into multiple QAs, the rest part is just how to aggregate the score from different QAs according to a key together. 

Although I do agree this is not a general approach for many many multi turn benchmarks, I have not yet find a perfect solution to come up with a general abstraction for most of the multi-turn benchmarks to prepare the task and QA. Feel free to discuss and suggest if you have any idea on this abstraction for most of the multi-turn benchmarks.

Thank you for your suggestion! I also considered this approach to reorganize samples in the benchmark. However, it’s not truly universal — for example, in StreamingBench, users can specify the context window size and split the video in-place during processing of a single dataset sample.

It seems the most universal way to handle multi-turn dialogues would be to allow multiple calls to the model per sample — which is exactly what generate_until_multi_round implies. I believe it’s correct that the logic and number of model calls should be controlled at the level of each benchmark’s processing.

Therefore, my original question remains relevant: is there any plan to support generate_until_multi_round in the framework?

Hi, I think the generate multi round until currently works only for the mmsearch task and I am not sure how generalizable it is to extend to other tasks. In current dev plan, I think generate_until_multi_round won't be a main feature to be add in shortly. We may try to design some more generalizable solution in the future.