What’s under the hood: Investigating Automatic Metrics on Meeting Summarization (2024)

Abstract: Meeting summarization has become a critical task considering the increase in online interactions. Despite new techniques being proposed regularly, the evaluation of meeting summarization techniques relies on metrics not tailored to capture meeting-specific errors, leading to ineffective assessment. This paper explores what established automatic metrics capture and the errors they mask by correlating metric scores with human evaluations across a comprehensive error taxonomy. We start by reviewing the literature on English meeting summarization to identify key challenges, such as speaker dynamics and contextual turn-taking, and error types, including missing information and linguistic inaccuracy, concepts previously loosely defined in the field. We then examine the relationship between these challenges and errors using human annotated transcripts and summaries from encoder-decoder-based and autoregressive Transformer models on the QMSum dataset. Experiments reveal that different model architectures respond variably to the challenges, resulting in distinct links between challenges and errors. Current established metrics struggle to capture the observable errors, showing weak to moderate correlations, with a third of the correlations indicating error masking. Only a subset of metrics accurately reacts to specific errors, while most correlations show either unresponsiveness or failure to reflect the error’s impact on summary quality.

## introduction the rise in remote collaboration has increased the need for effective meeting summarization @xcite , beneficial for participants and non-attendees. however, established evaluation metrics do not fully capture the challenges of meeting transcripts such as contextual turn-taking and discourse structure @xcite . the commonly used rouge metric @xcite has limitations in reflecting summary quality @xcite @xcite , and newer metrics like bertscore @xcite have not been thoroughly tested for meeting summarization @xcite . yet, this assessment would be crucial because meeting summarization comes with unique challenges, such as high abstraction, low extraction rate, and complex reasoning @xcite . this study examines how automatic metrics relate to human annotations for meeting summarization and what they actually measure in their scores. we aim to create a unified understanding of the challenges in meeting summarization and the errors that occur when these challenges are unmet. using the qmsum dataset @xcite , we have experts annotate the challenges in the transcripts and errors in automatically generated summaries using various models, including domain-standard encoder-decoder architectures and notable decoderonly models. this setup allows us to find connections between challenges, errors, and eight automatic metrics. the results show problems with current automatic metrics for evaluating meeting summarization. structural disorganization errors are often penalized, matching human judgments, but hallucination errors are sometimes rewarded. rouge @xcite struggles to distinguish the impact of different errors on quality, even though it is good at penalizing omissions. surprisingly, about a third of the metrics-error combinations either ignore or reward errors. for example, perplexity favors incorrect references, and lens favors structural disorganization. these observations highlight the need for better evaluation methods in meeting summarization. our contributions are threefold: the codebase, annotations, and guidelines are available on github: ## methodology we conducted a focused literature review to identify challenges and observable errors when failing these in english abstractive meeting summarization. we target papers published between the introduction of the bart model @xcite in 2019, which serves as the typical backbone model for the field, and 2024. this scope allows us to focus on challenges pertaining to currently used transformer-based models. we set additional inclusion criteria to prioritize relevance. we consider works that introduce novel methodologies or contribute to defining challenges, with a preference for transformer architectures, and exclude only loosely related studies, multi-lingual, multi-modal approaches, non-abstractive, and non-transformerbased approaches. we considered publications from 2024, irrespective of their citation index, acknowledging their emergent influence. the survey follows the prisma checklist @xcite to prevent bias. google scholar is chosen as the primary database for its comprehensive coverage and advanced query capabilities, yielding more works conforming to our review scope than web of science and dblp. our search comprises two stages: the initial phase involves broader queries combining "meeting summarization" with adjunctive terms such as challenge, literature review, or survey, while the subsequent refinement uses challenge-related keywords and synonyms. papers are ranked based on google scholar metrics, considering the top 100 from each query, and serve as primary source for our analysis. we apply our criteria to the initial pool of 300 papers to select 18 core articles directly addressing challenges and 40 additional papers discussing challenges and errors implicitly through methodological contributions. each shortlisted paper undergoes detailed data extraction to identify relevant challenges, methodologies, foundational models, and metrics collated in a structured format for crossvalidation. this approach allows us to construct a comprehensive view of current challenges and observable errors in english meeting summarization. ## related work research on meeting summarization typically addresses challenges and error types in isolation, leaving their interrelation unexplored. for dialogue summarization, challenges are collected (chen and yang, 2020) but lack the detail required for meeting contexts. previous works by @xcite ; @xcite lay the groundwork for our error typologies, enhanced with insights from diverse studies and annotator feedback. chen and yang (2020) correlate challenges and errors in dialogue summarization, but their findings do not entirely transfer to english abstractive meeting summarization, though their annotation approach informs ours. automatic metrics for meeting summaries are underexplored, but similar studies exist for text summarization using the cnn/dailymail dataset @xcite and dialogue summarization via the samsum dataset @xcite . we build upon these works by selectively adopting their metrics and augmenting them with measures commonly reported in recent meetings and dialogue summarization studies. ## conclusion we developed a resource suite for meeting summarization evaluation, covering domain-specific challenges, linked error types, predictive correlations for model architectures with known challenges, and expert annotations of the qmsum subset. our analysis highlighted the limitations and misalignments of current metrics with human judgment in discerning error nuances. metrics that work well for other summarization tasks either did not react to errors or cannot reflect the impact on quality in their scores. a composite metric may be more effective, but its formulation requires further research. recent advancements highlighted the potential of llmbased metrics. utilizing an llm prompted with our detailed annotator guidelines and supplemented by examples presents a promising approach for detecting complex errors like structural disorganization and incorrect reasoning. given the current evolution of techniques, we plan to extend the work as a (dialogue) summarization benchmarking pipeline to capture better how upcoming techniques handle challenges and how well new metrics capture their performance. we encourage the research community to contribute model outputs and introduce new metrics to this initiative.