On the Role of Summary Content Units in Text Summarization Evaluation (2024)

Abstract: At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs areconcise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages?ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategiesto approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when rankingshort summaries, but may not help as much when ranking systems or longer summaries.

## introduction judging the quality of a summary is a challenging task. besides being short and faithful to its source document, a summary should particularly excel in relevance, that is, a summary should select only the most relevant or salient facts from a source document. an attractive method for assessing such notion of relevance is the pyramid-method @xcite ) that is based on so-called summary content units (scus) which decompose a reference summary into concise human-written english sentences. with scus available from one or different reference summaries, we can then more objectively assess the degree to which a generated summary contains the relevant information. with the aim to fully automate the pyramid method, @xcite suggest that the required human effort can be partially and even fully alleviated, by i) automatically generating scus and ii) validating the relevance of a generated summary with a natural language inference (nli) system that checks how many scus are entailed by the generated summary. since strong nli systems are available off-theshelf and are known to be useful in natural language generation (nlg) evaluation @xcite , the generation of scus appears as the most challenging and leastunderstood part of an automated pyramid method. indeed, while @xcite show that scus can be approximated by phrasing semantic role triplets using a semantic role labeler and coreference resolver, we lack availability and understanding of possible alternatives as well as their potential impact on downstream-task summary evaluation in different scenarios. in this work, we proposed two novel approaches to approximate scus: semantic meaning units (smus) that are based on abstract meaning representation (amr) and semantic gpt units (sgus) that leverage sota large language models (llms). we carry out experiments to systematically evaluate the intrinsic quality of scus and their approximations. on the downstream task evaluation, we find that although scus remain the most effective metric to rank different systems or generated summaries across three meta-evaluation datasets, surprisingly, an efficient sentence-splitting baseline already yields competitive results when compared to scus. in fact, the sentence-splitting baseline outperforms the best scu approximation method on a few datasets when ranking systems or long generated summaries. in summary, our work provides important insights into the application of automation of the pyramid method in different scenarios for evaluating generated summaries. we make the code publicly available at https://github.com/ tristanratz/scu-text-evaluation/ . ## related work over the past two decades, researchers have proposed a wide range of human-in-the-loop or automatic metrics to assess the quality of generated summaries in different dimensions, including linguistic quality, coherence, faithfulness, and content quality. for more in-depth surveys on this topic, please refer to @xcite , @xcite . in this work, we focus on evaluating the content quality of a generated summary that assesses whether the summary effectively captures the salient information of interest from the input document(s). in the reference-based metrics, content quality is assessed by comparing system-generated summaries to human-written reference summaries. the pyramid method @xcite is regarded as a reliable and objective approach for assessing the content quality of a generated system summary. below we provide a brief overview of the pyramid method and highlight several previous efforts to automate this process. pyramid method. the original pyramid method @xcite comprises two steps: scus creation and system evaluation. in the first step, human annotators exhaustively identify summary content units (scus) from the reference summaries. each scu is a concise sentence or phrase that describes a single fact. the weight of an scu is determined by the number of references in which it occurs. in the second step, the presence of each scu in a system summary is manually checked. the system summary's pyramid score is calculated as the normalized sum of the weights of the scus that are present. later, @xcite introduce a revised version of the original pyramid method where they eliminate the merging and weighting of scus, thereby enabling scus with the same meaning to coexist. ## scu approximation i: smu from amr abstract meaning representation (amr) @xcite is a widely-used semantic formalism employed to encode the meaning of natural language text in the form of rooted, directed, edgelabeled, and leaf-labeled graphs. the amr graph structure facilitates machine-readable explicit representations of textual meaning. motivated by @xcite 's observation that stus based on semantic roles cannot well present single facts in long reference summary sentences that contain a lot of modifiers, adverbial phrases, or complements, we hypothesize that amr has the potential to capture such factual information more effectively. this is because, in addition to capturing semantic roles, amr models finer nuances of semantics, including negations, inverse semantic relations, and coreference. to generate semantic meaning units (smus) from a reference summary, we employ a parser that projects each sentence of the reference summary onto an amr graph, then split the amr graph into meaningful, event-oriented subgraphs. finally, we use a generator to generate a text (an scu approximation) from each subgraph. 1 while there may be various conceivable ways to extract subgraphs, for our experiment we use simple and intuitive splitting heuristics. given an amr graph, we first extract all predicates to figure 2: two amr sub-graphs for the sentence "godfrey elfwick recruited via twitter to appear on world have your say" ## scu approximation ii: sgu from llm recently, it became widely known that pre-trained large language models (llms) are able to generate high-quality output according to prompts given by humans, optionally exploiting a few shown examples through in-context learning @xcite . therefore, we try to approximate scus using gpt models from openai, calling the resulting units semantic gpt units (sgus). specifically, we use gpt-3.5-turbo which is built on instruct-gpt @xcite and gpt-4 (openai, 2023) to generate sgus (sgus_3.5 and sgus_4) for each reference summary using the same prompt and a one-shot example. please refer to appendix a.1 for more details. ## discussion and conclusions this work focuses on automating the pyramid method by proposing and evaluating two new methods to approximate scus. we found out that there are more effective ways of approximating scus than with stus only, and our extrinsic evaluation suggests that costly scus and approximations may even be unnecessary for system comparisons. there are several aspects worth discussing. firstly, as shown by comparing, e.g., the results of stus and smus in