Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization? (2024)

Abstract: Text summarization, a key natural language generation (NLG) task, is vital in various domains. However, the high cost of inaccurate summaries in risk-critical applications, particularly those involving human-in-the-loop decision-making, raises concerns about the reliability of uncertainty estimation on text summarization (UE-TS) evaluation methods. This concern stems from the dependency of uncertainty model metrics on diverse and potentially conflicting NLG metrics. To address this issue, we introduce a comprehensive UE-TS benchmark incorporating 31 NLG metrics across four dimensions. The benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable. We also assess the performance of 14 common uncertainty estimation methods within this benchmark. Our findings emphasize the importance of considering multiple uncorrelated NLG metrics and diverse uncertainty estimation methods to ensure reliable and efficient evaluation of UE-TS techniques. Our code and data are available: https://github.com/he159ok/Benchmark-of-Uncertainty-Estimation-Methods-in-Text-Summarization.

## introduction text summarization @xcite is a representative nlg task that generates summaries for given texts. this study researches abstractive summarization @xcite , which is more flexible than extractive summarization @xcite . in many scenarios (e.g., finance and health), there are serious consequences if relying on false predicted summaries. for instance, an inaccurate financial report summary could lead to incorrect financial decisions, resulting in financial losses @xcite . consequently, the task of ue-ts, which measures the likelihood that a generated summary is low-quality, has garnered significant interest @xcite @xcite . however, an overlooked question persists regarding the validity (the ability to accurately measure what it intends) and robustness (consistency across various scenarios and datasets) of the ue-ts evaluation framework. this concern arises due to two primary factors. first, common ue-ts evaluation metrics, such as forced-choice evaluation @xcite and prediction rejection ratio (prr) @xcite , rely on the alignment between two types of sample rankings: those based on uncertainty scores from an estimation method (e.g., entropy of generation semantics @xcite ) and those derived from a specific nlg metric (e.g., @xcite ). while higher alignment suggests better uncertainty estimation, the reliance on a single nlg metric may not fully capture the nuances of summary quality. second, the inherent label diversity in nlg tasks, particularly text summarization, necessitates the use of various nlg metrics. for instance, in text summarization, summaries like "the cat chased the mouse" and "the mouse was pursued by the kitty" convey identical messages using different expressions. due to the multiple valid responses possible in such tasks @xcite , it is unreasonable to expect nlg predictions to perfectly match the labels, unlike in nlu tasks @xcite . consequently, each nlg metric emphasizes different aspects of the diverse labels. for instance, rouge @xcite and bleu @xcite assess the similarity between the prediction and label based on n-gram overlap, while @xcite examines the level of hallucination between the generation and reference text. collectively, previous evaluations of ue-ts models @xcite @xcite typically relied on one or two uncertainty estimation metrics, each dependent on a single type of nlg metric. this limited evaluation raises a critical question: how reliable and valid are current methods for evaluating uncertainty estimation in text summarization? answering this question is essential for developing reliable ue-ts methods that can mitigate the risks associated with inaccurate summaries in critical applications, such as financial decision-making or healthcare. moreover, our research aims to explore the broader relationship between uncertainty estimation and nlg metric scores, the impact of nlg metric diversity on experimental design and analysis, and the effective selection of diverse uncertainty estimation methods. this comprehensive exploration will inform and enhance future ue-ts research. to address these questions, we introduce a ue-ts benchmark incorporating a diverse set of text generation metrics. our contributions are summarized below. ## related work text summarization. there are two types of text summarization: extractive text summarization @xcite , which extracts the original sentences from the text for summarization, and abstractive text summarization @xcite , which directly generates summaries from the text. due to the flexibility of abstractive text summarization, we focus on abstractive text summarization. recent models for abstractive text summarization are typically divided into two categories: plms (e.g., @xcite ) and llms (e.g., llama 2 @xcite ). to conduct comprehensive research on our question, we test abstractive summarization models from both plm and llms. ## conclusion recognizing the dependency of uncertainty model metrics on diverse nlg metrics, we introduce a comprehensive ue-ts benchmark to investigate the impact of nlg metric choice on the evaluation of uncertainty estimation methods in text sum-marization. this benchmark encompasses three datasets, fourteen uncertainty estimation methods (both white-box and black-box), and thirty-one nlg metrics spanning four evaluation dimensions. additionally, we assess one black-box llm, one white-box llm, and one plm for text summarization. our findings not only highlight the current limitations in ue-ts model evaluation but also contribute to a deeper understanding of the complex relationship between nlg metrics, uncertainty estimation methods, and human annotation. this knowledge can guide future research, inform the development of more effective uncertainty estimation methods, and ultimately enhance evaluation protocols for ue-ts and other nlg tasks.