QEVA : A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering (2025)

Abstract: Video-to-text summarization remains underexplored in terms of comprehensive evaluation methods. Traditional n-gram overlap-based metrics and recent large language model (LLM)-based approaches depend heavily on human-written reference summaries, limiting their practicality and sensitivity to nuanced semantic aspects. In this paper, we propose QEVA, a reference-free metric evaluating candidate summaries directly against source videos through multimodal question answering. QEVA assesses summaries along three clear dimensions: Coverage, Factuality, and Temporal Coherence. We also introduce MLVU(VS)-Eval, a new annotated benchmark derived from the MLVU dataset, comprising 800 summaries generated from 200 videos using state-of-the-art video-language multimodal models. This dataset establishes a transparent and consistent framework for evaluation. Experimental results demonstrate that QEVA shows higher correlation with human judgments compared to existing approaches, as measured by Kendall’s \tau_b , \tau_c , and Spearman’s \rho . We hope that our benchmark and metric will facilitate meaningful progress in video-to-text summarization research and provide valuable insights for the development of future evaluation methods.

## introduction text-based video summarization has become increasingly crucial due to the explosive growth of video content and significant advances in video-large multimodal models (video-lmms). despite substantial progress in generating comprehensive textual summaries from videos, insufficient attention has been paid to reliably evaluating the quality of these summaries. currently, evaluation primarily relies on reference-based metrics such as rouge ## the qeva method we introduce qeva, a novel evaluation framework designed to assess how comprehensively and faithfully a textual video summary captures the original video's content. qeva is grounded in the principle that "a good summary should serve as an effective substitute for the source video". to operationalize this intuition, qeva employs multimodal question answering (qa) as its core mechanism: a high-quality summary should allow accurate answering of critical questions derived from the source video content. unlike prior work relying solely on direct llm-based judgments, qeva integrates both video-lmms and llms within a collaborative pipeline, thus mitigating the inher-you are an instructor in "deep video comprehension through summaries". given an entire video, generate 10 diverse quiz questions measuring high-level comprehension (themes, motivation, causal relations)… (continues) avoid timestamps, subtitles, or trivial factual questions; require inference and synthesis… (continues) output example: {"question_1": "what is the video's central conflict?", ...} input: <entire @xmath0 we conduct comprehensive experiments to validate the effectiveness of our proposed qeva metric. we first introduce mlvu(vs)-eval, a novel benchmark dataset for evaluating video-to-text summarization metrics ( §4.1). we then compare qeva against existing metrics in terms of correlation with human judgments ( §4.2). additionally, we analyze model-wise performance of qeva ( §4.3), and demonstrate the robustness of qeva via ablation studies ( §4.4). ## conclusion we propose qeva, a novel reference-free metric for evaluating narrative video summarization leveraging multimodal question answering. qeva demonstrates significantly higher correlation with human judgments compared to existing metrics, while eliminating reliance on costly reference summaries. our approach facilitates scalable, accurate, and practical evaluation of video summarization systems, accelerating the development and deploy-ment of video-lmms in real-world multimodal applications.