Automatic, Meta and Human Evaluation for Multimodal Summarization with Multimodal Output (2024)

Abstract: Multimodal summarization with multimodal output (MSMO) has attracted increasing research interests recently as multimodal summary could provide more comprehensive information compared to text-only summary, effectively improving the user experience and satisfaction. As one of the most fundamental components for the development of MSMO, evaluation is an emerging yet underexplored research topic. In this paper, we fill this gap and propose a research framework that studies three research questions of MSMO evaluation: (1) Automatic Evaluation: We propose a novel metric mLLM-EVAL, which utilizes multimodal Large Language Model for MSMO EVALuation. (2) Meta-Evaluation: We create a meta-evaluation benchmark dataset by collecting human-annotated scores for multimodal summaries. With our benchmark, we conduct meta-evaluation analysis to assess the quality of different evaluation metrics and show the effectiveness of our proposed mLLM-EVAL. (3) Human Evaluation: To provide more objective and unbiased human annotations for meta-evaluation, we hypothesize and verify three types of cognitive biases in human evaluation. We also incorporate our findings into the human annotation process in the meta-evaluation benchmark. Overall, our research framework provides an evaluation metric, a meta-evaluation benchmark dataset annotated by humans and an analysis of cognitive biases in human evaluation, which we believe would serve as a valuable and comprehensive resource for the MSMO research community.

## introduction with the exponentially growing amount of multimedia data online, multimodal summarization with multimodal output (msmo) has garnered more and more attention from researchers. unlike unimodal output (e.g., text-only summary), msmo aims at extracting the most salient information from different modalities and generating multimodal summaries, with a text summary and the most relevant images @xcite @xcite @xcite @xcite . compared to a text-only summary, a multimodal summary is more informative and could provide more comprehensive and userfriendly content to the readers, as well as effectively improve their reading experience and satisfaction @xcite @xcite , which could thus be suitable for many applications (e.g., multimedia news summarization). evaluation metrics for msmo are essential to objectively measure the quality of the multimodal summary outputs. however, only a few works have focused on the evaluation for multimodal summaries @xcite @xcite . meanwhile, it is also important to establish a meta-evaluation system to assess the quality of the msmo metrics to promote more exploration of effective and unbiased evaluation metrics. despite its significance, there is no such benchmark or study in meta-evaluation. furthermore, metaevaluation requires human-annotated ratings for the correlation test since human annotations are considered as the gold standard, but the analysis of cognitive biases in the human annotations remains unexplored. to fill these gaps and call for more research in this area, we propose a research framework (as in fig. 1 ) and investigate the following research questions: rq1. automatic evaluation: how to properly evaluate the quality of a multimodal summary? the existing evaluation methods proposed by @xcite require the human-written references for evaluation (e.g., rouge score @xcite ) for text quality, selection precision for image quality, supervised regression for overall quality). however, it is costly to collect human-written multimodal summaries as references for evaluation. inspired by the success of large language models (llms) in reference-free text summary evaluation @xcite @xcite , as well as the development of multimodal llms (e.g., gpt-4 (openai, 2023)), we propose mllm-eval, an msmo metric utilize the emergent abilities of multimodal llms that shows a high correlation with human judgments. rq2. meta-evaluation: how to evaluate the evaluation metrics for multimodal summarization? furthermore, to evaluate our proposed metric and encourage further exploration of msmo metrics, we create the first msmo meta-evaluation benchmark dataset by asking three experts to assess the quality of multimodal summaries. using our metaevaluation benchmark, we evaluate and analyze the quality of the existing metrics and mllm-eavl by testing how well they correlate with human annotations, where our proposed method outperforms the current evaluation metrics. rq3. human evaluation: is human evaluation truly the gold standard? in our meta-evaluation (rq2), we require human evaluation to examine how well the msmo metrics correlate with human judgments, as human evaluation is widely considered as the gold standard. however, humans could potentially bring biases into the evaluation due to cognitive biases, which would make the comparison unfair and biased. to this end, we hypothesize and verify three types of cognitive biases: anchoring bias; lead bias in image selection; text bias in overall evaluation. we also incorporate our findings to avoid such biases, thus providing more objective and unbiased human annotations for meta-evaluation (rq2), which makes our metaevaluation more reliable. the main contribution of this paper is: (1) this is a pioneering work to study automatic evaluation, meta-evaluation and cognitive biases analysis in human evaluation for msmo. (2) we propose mllm-eval using multimodal llms for reference-free msmo evaluation. (3) we create the first msmo meta-evaluation benchmark by collecting human annotations and then conduct meta-evaluation to assess the quality of various msmo metrics. (4) we study the cognitive biases in human evaluation for msmo. overall, we aim to establish a valuable research foundation to significantly benefit the multimodal summarization community. ## calls for future research we hope this paper will be a valuable resource for future research on multimodal summarization evaluation and models. the study in this work demonstrates the need for future research on (1) better automatic evaluation metrics that could properly and objectively evaluate the quality of multimodal summaries; (2) more meta-evaluation in multimodal summarization to update with the current advanced systems and datasets (3) more study on cognitive biases in human evaluation for multimodal summarization, to have a more fair and accurate comparison of different systems or evaluation metrics. we hope this work could demonstrate the importance of these issues and call for future research (evaluation, meta-evaluation, human evaluation), which would significantly benefit the multimodal summarization research community. ## conclusion in this paper, we propose a research framework, where we investigate the automatic evaluation, meta-evaluation and cognitive bias analysis in human evaluation for multimodal summarization. for automatic evaluation, we propose a reference-free metric based on multimodal llms that correlates well with human judgments. furthermore, we collect human annotations for multimodal summaries and release a meta-evaluation benchmark to evaluate various evaluation metrics. for the human evaluation, we study and verify three types of cognitive biases in human evaluation. we believe our work would be a valuable resource for multimodal summarization research, and hope this work could demonstrate the importance of these topics as well as encourage further research in this area.