STORYSUMM : Evaluating Faithfulness in Story Summarization (2024)

Abstract: Human evaluation has been the gold standard for checking faithfulness in abstractive summarization. However, with a challenging source domain like narrative, multiple annotators can agree a summary is faithful, while missing details that are obvious errors only once pointed out. We therefore introduce a new dataset, StorySumm, comprising LLM summaries of short stories with localized faithfulness labels and error explanations. This benchmark is for evaluation methods, testing whether a given method can detect challenging inconsistencies. Using this dataset, we first show that any one human annotation protocol is likely to miss inconsistencies, and we advocate for pursuing a range of methods when establishing ground truth for a summarization dataset. We finally test recent automatic metrics and find that none of them achieve more than 70% balanced accuracy on this task, demonstrating that it is a challenging benchmark for future work in faithfulness evaluation.

## introduction as large language models (llms) are able to perform more open generation tasks, challenges in evaluation have arisen @xcite . summarization is one such task. some aspects of summary quality like readability or coherence @xcite can be judged by looking at the summary alone. however, judging faithfulness (whether all details in the summary are faithful to the source) requires carefully checking a multi-sentence summary against a multi-paragraph source document @xcite . summaries that misrepresent source documents can easily spread disinformation, so it is critical we evaluate summary faithfulness, despite how labor-intensive it is. methods for detecting inconsistencies have generally used one of two tools: 1) trained models, or 2) human crowdworkers. model-based approaches typically build on qa or entailment strategies. qa strategies generate questions about the summary and compare answers retrieved from the summary vs. the source document @xcite . entailment-based approaches align facts in the summary with evidence from the source and determine for each pair if the evidence entails the fact @xcite @xcite . more recent work explores prompting strategies for llms to identify faithfulness errors @xcite @xcite @xcite . with human annotators, prior work has shown that human judgments have increased variability when evaluating long summaries @xcite . reducing the problem to evaluating individual sentences or claims helps to produce more reliable results @xcite @xcite . however, these works have focused on factuality in news summaries or real-world articles where ground truth is based in reality and facts are stated explicitly. as llms continue to grow in capabilities, there is a pressing need for evaluation of their accuracy to grow with them. we therefore produce a new benchmark, storysumm, which can be used to improve evaluation methods for faithfulness. sto-rysumm consists of 96 short stories and llmgenerated summaries with over 500 sentencelevel faithfulness labels and explanations. each unfaithful summary is labeled as easy or hard to detect. llm summaries often contain subtle errors, particularly for narrative text which requires nuanced interpretation. this benchmark therefore introduces new challenges when compared to factchecking or summarization datasets in the news domain. the example in figure 1 demonstrates that assessing the summary requires correct interpretation of sentences like, her heart was mine., which have multiple meanings and are misleading without carefully reading the entire story. by focusing on faithfulness in narrative summarization and using real-world data from llms and reddit, storysumm poses a realistic but hard benchmark to push our methods forward. we first explore how to establish ground-truth on this dataset by comparing different human annotation protocols and manually inspecting the results. we try different protocols and pools of annotators to see if there is an approach that helps average annotators pay attention and understand this challenging task more consistently. we find that different protocols catch unique but legitimate inconsistencies and have only fair agreement with each other. we therefore manually review and merge label sets across three annotation protocols. we analyze the errors found by each protocol, and formulate a set of recommendations for human evaluation of faithfulness in narrative summarization. most importantly, we show that it is important to use a variety of annotators and protocols when establishing ground truth for faithfulness. we then explore how well recent automatic metrics perform on this dataset. we find that no metric achieves more than 70% balanced accuracy on this task and even the best metric misses almost 50% of the hard inconsistencies. ## storysumm dataset we design our benchmark with a focus on three principles which distinguish it from existing datasets. first, the stories need to be short enough that humans can easily read them, so that we can affordably test human protocols. second, the stories should not be so famous that llms have likely trained on summaries of them, potentially biasing llm summary or evaluation quality. third, the summaries should be representative of powerful llms so that we can assess how difficult it is to find errors in fluent and convincing summaries. motivated by these principles, we opt for short narratives from reddit and use gpt-series and claude-series models to generate summaries. we do not include any human-written summaries as the purpose of this dataset is to improve detection of errors in llm-generated summaries. we show summary statistics for the dataset in ## benchmarking automatic metrics having established a source of ground truth, we benchmark recent automatic methods against our labels. we try the following metrics: binary. we prompt gpt-4 @xcite , claude-3 foot_2 , and mixtral-8x7b foot_3 to assign a binary faithfulness label to each summary using the same definition of faithfulness as used for the human annotators. cot. we prompt gpt-4, claude-3, and mixtral-8x7b to assign a binary faithfulness label to each summary, but to first provide some reasoning in a chain-of-thought style @xcite . models are prompted to: consider whether there are any details in the summary that are inconsistent with the story and provide a couple sentences of reasoning for why the summary is or is not consistent with the story. ## related work datasets. there are many datasets for factchecking or inconsistency detection in news @xcite @xcite @xcite @xcite and dialogue @xcite summarization. however, the summarization datasets specifically for narrative either use books and stories that most llms have trained on @xcite or use books that have to be purchased @xcite . automatic metrics. many inconsistency detectioin methods have been developed on the above datasets, which we cite in section 1 and section 4. we test the current best metrics on our benchmark. calibration against humans. @xcite , @xcite also propose recommendations for human evaluation of faithfulness. other works @xcite @xcite have demonstrated that standard evaluation metrics are not well correlated with human judgments. subbiah et al. (2024 ## conclusion we introduce a new benchmark for testing methods for faithfulness evaluation. in producing the benchmark, we demonstrate that faithfulness in narrative summarization is still a significant concern for llms, and we formulate recommendations for better evaluation of faithfulness in summaries. finally, we demonstrate that recent automatic evaluation metrics have room for improvement on this task. in the future, we hope to use this dataset to improve methods for reliable evaluation of narrative summarization. in particular, we would like to develop automatic methods to merge error sets across evaluation protocols and check for correctness in error reason, not just localization.