APPLS : Evaluating Evaluation Metrics for Plain Language Summarization (2024)

Abstract: While there has been significant development of models for Plain Language Summarization (PLS), evaluation remains a challenge. PLS lacks a dedicated assessment metric, and the suitability of text generation evaluation metrics is unclear due to the unique transformations involved (e.g., adding background explanations, removing jargon). To address these questions, our study introduces a granular meta-evaluation testbed, APPLS, designed to evaluate metrics for PLS. We identify four PLS criteria from previous work—informativeness, simplification, coherence, and faithfulness—and define a set of perturbations corresponding to these criteria that sensitive metrics should be able to detect. We apply these perturbations to extractive hypotheses for two PLS datasets to form our testbed. Using APPLS, we assess performance of 14 metrics, including automated scores, lexical features, and LLM prompt-based evaluations. Our analysis reveals that while some current metrics show sensitivity to specific criteria, no single method captures all four criteria simultaneously. We therefore recommend a suite of automated metrics be used to capture PLS quality along all relevant criteria. This work contributes the first meta-evaluation testbed for PLS and a comprehensive evaluation of existing metrics.

## introduction plain language summaries of scientific information are important to make science more accessible @xcite and inform public decision-making @xcite . recently, generative models have made gains in translating scientific information into plain language approachable to lay audiences @xcite @xcite . despite figure 1 : we present appls, the first granular testbed for analyzing evaluation metric performance for plain language summarization (pls). we assess performance of 14 existing metrics, including automated scores, lexical features, and llm prompt-based evaluations. our goal is to assess how well existing metrics capture the multiple criteria of pls. we identify four criteria, informed by prior work @xcite @xcite , that a pls metric should be sensitive to: informativeness, simplification, coherence, and faithfulness. we introduce a set of perturbations to probe metric sensitivity to these criteria, where each perturbation is designed to affect a single criterion with ideally minimal impact to others. 2 by incrementally introducing perturbations to the texts of two scientific pls datasets, cells @xcite and plaba @xcite , we create our meta-evaluation testbed appls. we analyze 14 metrics using appls, including the most widely used metrics in text simplification and summmarization literature, and recentlyproposed prompt-based methods @xcite . we find that established metrics like rouge @xcite , bertscore @xcite , and qaeval @xcite do not capture simplification and are inconsistent at capturing perturbations in informativeness, coherence, and faithfulness; sari score @xcite , explicitly crafted for text simplification, is the only automated score that displays sensitivity towards simplification perturbations but not to other perturbations. llm prompt-based evaluations are effective for assessing informativeness, faithfulness, and simplification, but not for coherence. our analysis suggests that a single overall score cannot simultaneously respond to all four criteria. our main contributions are as follows: • we present appls, the first granular testbed for analyzing evaluation metric performance for plain language summarization; the testbed is created by applying 11 perturbations along four dimensions to two scientific pls datasets ( §3, 4); robust analysis with synthetic data synthetic data has been widely used in nlp to evaluate metrics, for tasks such as text generation @xcite , natural language inference @xcite , question answering @xcite , and reading comprehension @xcite ). yet, no prior work has focused on the pls task or incorporated simplification into their benchmarks. additionally, previous studies lack granular analyses to capture nuanced relationships between text changes and score changes. our research endeavors to bridge these gaps by crafting perturbations that mirror real-world errors found in pls. ## pls evaluation criteria we identify four criteria that an effective pls evaluation metric should be sensitive to based on both abstractive summarization @xcite and plain language summarization paradigms @xcite @xcite . as in @xcite , we define sensitivity as being correlated in the expected direction with the amount of change in that criteria. ## constructing the appls testbed to assess metric sensitivity, we develop perturbations along each evaluation criteria dimension. we implement our perturbations in two large-scale pls datasets, described in §4.1. we follow with a discussion of how perturbations are incorporated into these datasets and our approach for managing perturbation magnitude ( §4.2) and validating perturbation quality ( §4.3). ## evaluation metrics our analysis spans 8 established evaluation metrics, including the 5 most commonly reported in acl'22 summarization/generation papers (empirical results in app. d). we also assess 5 lexical features associated with text simplification ( §5.2) and llm-based evaluations ( §5.3). ## analysis results automated metric responses to perturbations are in figure 2 , responses of lexical features are in figure 3 , and prompt-based evaluation results are shown in figure 4 . all trends are consistent across two random seeds. to contextualize metric performance in appls, we survey metric changes reported in acl'22 papers on text generation and summarization (full results in app. d). median reported improvements for the most commonly reported metrics and sari are: rouge (+0.89), bleu (+0.69), ppl (-2.06), meteor (+0.50), bertscore (+0.55), and sari (+1.71), as shown in figure 11 . aside from sari, current metrics exhibit shortcomings in evaluating simplicity. metrics that are sensitive to simplification should consistently distinguish between more and less simplified text. sari is the only automated metric among those we tested that is consistently sensitive to simplified text. as shown in figure 2 , metrics that exhibit sensitivity to simplification perturbations are gpt-ppl (decreasing as more perturbations are introduced; lower ppl is better) and sari score. however, in follow-up evaluations with plaba (shown in app. figure 16 ), we see that gpt-ppl has undesirable sensitivity to text length, as found in prior work @xcite . rouge, bleu, meteor, bertscore, and qaeval decrease in response to the simplification perturbation. while they show consistent response relative to the degree of perturbation, they are nonetheless not useful for assessing text simplicity. when we report metric changes swapping sources and targets (perturbing simplified texts to increase complexity), these metrics also decrease (app. figure 13 ), suggesting that they are sensitive to n-gram changes and not text simplicity. lens behaves erraticly with increasing simplification perturbation percentage, indicating that it is not a good metric for text simplicity. in addition to using gpt-4 @xcite ) to produced simplified text for the simplification perturbation, we also test three other llms: gpt-3 (brown et al., 2020), llama 2 @xcite , and claude. 7 in figure 5 , we show metric changes to the simplification perturbation generated by all four models. similar score changes are observed for all models (except gpt-3 for sari score, which is an outlier), demonstrating that the simplicity perturbation in our testbed is a reasonable and consistent measure of metric response to text simplification, and that sari is generally able to distinguish between more and less simplified text. metrics effectively capture informativeness, coherence, and faithfulness, with room for improvement. for informativeness, rouge, bleu, bertscore, gpt-ppl, and qaeval are sensitive to information deletion and irrelevant additions, but decrease with the addition of background explanations through keyword definitions. for coherence, bertscore and lens excel in detecting perturbations, largely due to their ability to assess between-sentence relationships. bertscore, gpt-ppl, and qaeval generally perform well for faithfulness-related perturbations, although gpt-ppl and bertscore are somewhat sensitive to synonym verb swaps instead of antonym verb swaps, which is an undesirable trait. qaeval is best at being unresponsive to synonym verb swaps. number swaps, however, remain undetected by all metrics. results in figure 2 . lexical features are useful measures of text simplicity. figure 3 illustrates the response of lexical features to degrees of text simplification in cells, confirming trends observed in previous studies @xcite @xcite . as simplification increases, sentence length decreases; common words and verbs increase; and nouns, adjectives, and term specificity decrease. although prior work emphasizes the importance of conjunctions for comprehension @xcite , our study reveals a reduction rather than increase in conjunctions as texts become simpler. overall, these trends demonstrate that lexical features are valuable indicators for text simplification. results on plaba are similar, with an inverse trend for paragraph length (app. figure 12 ). llm prompt-based evaluations show promise in distinguishing between pls criteria. promptbased scores demonstrate sensitivity to perturbations in informativeness, faithfulness, and simplification, while showing less sensitivity to changes in coherence (figure 4 ). while providing a single criterion, all criteria, and all criteria with an explanation mostly yield similar results, trends for simplification and some types of faithfulness perturbations are more clear and consistent when all criteria are provided. this suggests that providing all criteria and requesting all scores simultaneously is most efficient and accurate. our results also indicate that additional explanations are not essential for pls evaluation (results for settings b and c are similar). however, further studies are required to better understand the decision-making process of the llm, assess the validity of its explanations, and explore how the quality of explanations impacts the score. prompts and detailed results are provided in app. g. ## discussion & conclusion recent advances point to the possibility of automated pls; however, the multifaceted nature of pls makes evaluation challenging. we introduce the first-to our knowledge-meta-evaluation testbed, appls, for evaluating pls metrics. in appls, we apply controlled text perturba-tions to existing pls datasets based on several criteria (informativeness, simplification, coherence, and faithfulness). using appls, we find that while some metrics reasonably capture informativeness, faithfulness, and coherence, sari is uniquely sensitive to simplification perturbations, but exhibits insensitivity to other perturbations. similar challenges are observed for qaeval, as no single metric was consistently sensitive to all perturbations across criteria. therefore, an evaluation metric suite should be considered based on all desired criteria. from our results on appls, we identify the following metrics for each criterion as most promising from among those we tested: sari for simplicity, gpt-ppl for informativeness, lens for coherence, and qaeval for faithfulness. however, we warn that all of these automated metrics have limitations as identified in our results. further research is necessary to identify more robust metrics for a comprehensive evaluation of pls. the quickly improving performance of language models on a variety of tasks has placed greater emphasis on extrinsic human evaluations @xcite evaluating models in real-world use cases, often with end-users. however, extrinsic evaluations are time-intensive, difficult to implement correctly, and costly, making them only viable for the most promising models. automated metrics offer a fast and low-cost method for identifying improvement trends even if they do not perfectly measure absolute improvement, or improvement at the instance level. our selection of automated metrics grounded in criteria from the health communication literature offer a viable first step in evaluating systems. initial automated evaluations can then be followed by extrinsic evaluations to ensure comprehensive analysis for real-world use. our appls testbed allows for extensible evaluation of pls evaluation metrics. although this study focuses on the health domain, appls can be adapted to other domains by changing the diagnostic dataset. depending on the domain, evaluation criteria may need to be adjusted. for example, in legal contexts, faithfulness might be prioritized over informativeness, and additional criteria such as language specificity (e.g., avoidance of vague terminology) may be necessary. using our perturbation pipeline, appls can transform any pls dataset into a granular meta-evaluation testbed. new perturbations can be introduced, and new evaluation metrics can also be incorporated easily into analysis. our testbed lays the groundwork for further advancements in automated pls and pls evaluation, aiming to foster more impactful, accessible, and equitable scientific communication.