MDSE val: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization (2025)

Abstract: Multimodal Dialogue Summarization (MDS) is a critical task with wide-ranging applications. To support the development of effective MDS models, robust automatic evaluation methods are essential for reducing both cost and human effort. However, such methods require a strong meta-evaluation benchmark grounded in human annotations. In this work, we introduce MDSEval, the first meta-evaluation benchmark for MDS, consisting image-sharing dialogues, corresponding summaries, and human judgments across eight well-defined quality aspects. To ensure data quality and richfulness, we propose a novel filtering framework leveraging Mutually Exclusive Key Information (MEKI) across modalities. Our work is the first to identify and formalize key evaluation dimensions specific to MDS. Finally, we benchmark state-of-the-art modal evaluation methods, revealing their limitations in distinguishing summaries from advanced MLLMs and their susceptibility to various bias.

## introduction human communication is inherently multimodal, encompassing text, images, videos, and audio. this has led to the emergence of multimodal large language models (mllms), which integrate information across modalities to facilitate more natural and effective human-machine interactions. a key application in this space is multimodal dialogue summarization (mds) @xcite @xcite , which aims to distill salient information from multimodal conversations (e.g., slack chats). to accelerate the development of mds models, reliable automatic evaluation methods are essential, as they enable rapid iteration and significantly reduce the cost of manual assessments. however, * work done during an internship at aws ai labs. the development of effective automatic evaluation methods requires a meta-evaluation benchmark-a dataset with detailed human-annotated quality assessments-to serve as ground truth for evaluating the evaluators. to address this gap, we introduce mdseval, the first comprehensive meta-evaluation benchmark, specifically for the mds task. mdseval comprises image-sharing dialogues, multiple candidate summaries, and human evaluations across eight quality dimensions. this benchmark enables systematic comparison of evaluation methods, exposes their limitations, and provides actionable insights for developing more accurate and human-aligned assessment techniques for multimodal summarization. the overall data curation pipeline is illustrated in figure 1 . mdseval contains 198 high-quality image-sharing dialogues, carefully curated from the photochat @xcite and dialogcc @xcite datasets. to ensure the suitability and challenge level of these dialogues for the summarization task-specifically, that effective summaries must draw on information uniquely provided by both text and images-we introduce a novel data filtering framework based on the mutually exclusive key information (meki) criterion. meki is designed to identify information that is uniquely conveyed by one modality and not inferable from the other, thereby emphasizing the need for true multimodal comprehension. empirical analysis shows that meki scores correlate strongly with human judgments (spearman ρ ≈ 0.80). each image-sharing dialogue is paired with five summaries generated by state-of-the-art (sota) mllms. in additional to typical evaluation aspects such as coherence, conciseness and fine-grained faithfulness, we identify and define new aspects focusing on cross-modal information coverage, information balance and topic progression. each summary is annotated by three experienced human experts. based on the human annotations, we benchmark latest multimodal evaluation techniques on mdseval, revealing two critical findings: 1) current multimodal evaluation methods struggle to differentiate summaries generated by recent llms. 2) existing evaluation methods suffer from significant biases. mdseval aims to advance the development of robust, human-aligned multimodal evaluation methods. ultimately, our work paves the way for more sophisticated multimodal conversational agents capable of seamlessly integrating and interpreting information across modalities. we summarize our main contributions as follows: ## mdseval benchmark this section provides a detailed overview of the mdseval dataset. we describe the multi-stage construction pipeline, illustrated in figure 1 , and present the resulting dataset statistics. ## conclusion in this work, we introduce mdseval, the first metaevaluation benchmark for multimodal dialogue summarization. mdseval includes human annotations across eight carefully defined evaluation aspects. we detail our data curation pipeline and propose novel metrics and considerations-such as the mutually exclusive key information (meki) scores-which will inspire future research. furthermore, we assess the performance of several sota multimodal evaluation methods on mdseval and find that current mllms still struggle to deliver human-aligned judgments when evaluating summaries generated by the latest mllms.