SN a C : Coherence Error Detection for Narrative Summarization (2022)

Abstract: Progress in summarizing long texts is inhibited by the lack of appropriate evaluation frameworks. A long summary that appropriately covers the facets of that text must also present a coherent narrative, but current automatic and human evaluation methods fail to identify gaps in coherence. In this work, we introduce SNaC, a narrative coherence evaluation framework for fine-grained annotations of long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowdworkers. Furthermore, we show that the collected annotations allow us to benchmark past work in coherence modeling and train a strong classifier for automatically localizing coherence errors in generated summaries. Finally, our SNaC framework can support future work in long document summarization and coherence evaluation, including improved summarization modeling and post-hoc summary correction.

## introduction as pre-trained models for news summarization @xcite @xcite have improved drastically, researchers have begun tackling increasingly challenging settings, particularly long document summarization and generation of longer summaries @xcite @xcite . summaries in these settings differ considerably from the newswire summaries of past research efforts @xcite : models now need to extract salient information from different parts of a significantly in court, mr. darnay is accused of treason. however, he is acquitted after his patriot friend, roger cly, testifies against him. why restore to life? is her father sick? who is mr. darnay? who is mr. lorry? sudden shift in scene (what happened to miss mannette?) event unclear character unclear scene transition character unclear miss manette receives a letter from the bank informing her that information about her father's small property has been discovered. she wants to travel to france to identify him and restore him to life. mr. lorry explains that her father has been found under another name and is being held in a house in paris. five years later, in 1780, a young frenchman, named charles darnay, is accused of being a traitor and a spy. lucie @xcite and her father are reluctant witnesses for the prosecution @xcite corresponding excerpt from the human-written summary properly contextualizes the new character. figure 1 : excerpt from a generated book summary by openai's 175b model @xcite . individual segments do not follow a coherent structure and extra information is often needed to understand the narrative. this shift in the scope of the summarization task calls for a reexamination of the summarization evaluation framework. even for short newswire summaries, @xcite showed that automated metrics are inadequate, and consequently, reporting results from a human evaluation study has become the standard practice. however, human evaluation is rarely done for longer summaries possibly due to the associated labor costs of reading and evaluating long text. it is also unclear whether a/b testing or likert-scale based annotation frameworks transfer to long summary settings. establishing human evaluation protocols is critical for comparing different modeling approaches and measuring progress. recently, @xcite proposed a strong book summarization model but showed that although generated summaries covered important information from the books, they read like a list of events stapled together without any coherent narrative structure (see figure 1 ). we found similar context: john fenwick, an aspiring artist, accepts a loan from mr. morrison to move to london to pursue his art career. in london, he becomes infatuated with madame de pastourelles, a beautiful and intelligent artist. ## long narrative summarization we study coherence errors in two domains, books and movie screenplays, although our taxonomy and annotation methodology are broadly applicable. ## snac annotation methodology we design our methodology to: 1) simplify the summary-level annotation task into smaller subtasks, and 2) provide a structured framework that allows annotators to specify the type of coherence error, instead of evaluating coherence holistically. ## data collection we collect annotations from two types of annotators: experts and crowdworkers. ## discussion our analysis of current narrative summarization models reveals that these do not generate coherent narratives; in fact, each generated summary contains ~30 coherence errors of varying degrees of severity. moreover, both automatic and human approaches for coherence evaluation fail to reliably measure coherence. snac addresses this gap. however, we stop short of providing a prepackaged metric: which errors are more severe is application-dependent and subjective, and overall error counts cannot be compared. we encourage future work to focus on fine-grained error annotations, like those we present here, instead of sentence-or document-level annotations that do not provide actionable insights. we also recommend fine-grained error modeling for future coherence systems as well. while previous modeling has targeted document-or sentence-level coherence, our models trained on snac data can detect span-level coherence errors, particularly chare errors with high accuracy. this automatic error localization opens up future avenues of post-hoc error correction systems built on top of coherence models. ## related work coherence frameworks inspired by centering theory @xcite , @xcite proposed entity-grid models to measure coherence through transitions of entity roles. this was further extended to incorporate non-head entities @xcite , discourse roles @xcite , and other improvements @xcite , including neural variations @xcite @xcite to better model text coherence. however, these models have been evaluated primarily on document-level essay scoring tasks @xcite or artificial sentence-ordering tasks @xcite , and not on model-generated coherence errors. summarization evaluation automatic metrics such as bleu @xcite , meteor @xcite , rouge @xcite , bertscore @xcite , and others have been used to evaluate summarization, but @xcite showed that these correlate poorly with summary quality. human evaluation is widely considered the gold standard for generation tasks, however, recent work @xcite demonstrated that humans are not reliable for evaluating strong models like gpt-3. ## conclusion we introduce snac, a narrative coherence evaluation framework for long summaries. we develop an error taxonomy grounded in coherence errors made by current models and annotate data to provide the first characterization of such errors in narrative summaries. we also make our annotation tool publicly available to support future research efforts.