Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA (2023)

Abstract: Large language models (e.g., GPT-4) are uniquely capable of producing highly rated text simplification, yet current human evaluation methods fail to provide a clear understanding of systems’ specific strengths and weaknesses. To address this limitation, we introduce SALSA, an edit-based human annotation framework that enables holistic and fine-grained text simplification evaluation. We develop twenty one linguistically grounded edit types, covering the full spectrum of success and failure across dimensions of conceptual, syntactic and lexical simplicity. Using SALSA, we collect 19K edit annotations on 840 simplifications, revealing discrepancies in the distribution of simplification strategies performed by fine-tuned models, prompted LLMs and humans, and find GPT-3.5 performs more quality edits than humans, but still exhibits frequent errors. Using our fine-grained annotations, we develop LENS-SALSA, a reference-free automatic simplification metric, trained to predict sentence- and word-level quality simultaneously. Additionally, we introduce word-level quality estimation for simplification and report promising baseline results. Our data, new metric, and annotation toolkit are available at https://salsa-eval.com.

## introduction text simplification aims to improve a text's readability or content accessibility while preserving its fundamental meaning @xcite . traditional human evaluation for text simplification often relies on individual, shallow sentence-level ratings @xcite , easily affected by the annotator's preference or bias. @xcite recently proposes a more reliable and consistent human evaluation method by ranking and rating multiple simplifications altogether. however, as text simplification involves performing a series of transformations, or edits, such as paraphrasing, removing irrelevant details, or splitting a long sen- this is sentence split splits content into two sentences bad substitution over generalize with inaccurate meaning grocery inflation the cost of groceries structure change changes tense reaches increased has paraphrase substitutes with simpler wording reaches increased to good deletion removes non-critical info high of info. rewrite different meaning amid due to complex sentence: simplification by gpt-4: ## salsa framework we introduce salsa, an edit-based human evaluation framework for text simplification. salsa is defined by a typology of 21 linguistically-grounded edit types with the aim of capturing both successes and failures (i.e., quality changes and errors, see figure 1 ). the annotation methodology of salsa is structured as a decision tree and implemented via an easy-to-use interface, illustrated in figure 2 . our interface is designed with thresh @xcite , and we release our configuration to encourage adaptation to other text rewriting tasks 1 ) selecting edits, (2) identifying information change, (3) classifying edit type and (4) rating efficacy/severity. @xcite or collecting fine-grained human feedback @xcite foot_0 . in the following, we describe each step of the annotation process. ## data collection we describe our use of salsa to collect 19k edit annotations covering 11.6k spans on 840 modelgenerated and human-written simplifications. ## key analysis we use salsa to evaluate state-of-the-art simplification by collecting annotations on our extended version of the simpeval corpus @xcite , which includes fine-tuned, llm-and human-written simplifications. our resulting data collection includes 19k edit annotations across 840 simplifications. we present our primary results in figures 4 , 5 , and 6. figures 4 and 5 illustrate the frequency of quality and error edit types. as edits vary in length, we calculate edit coverage: the length of each edit in proportion to the total length of the simplification and report the average edit coverage for different efficacy and severity ratings in 6, showing a view of edit ratings adjusted for length. additionally, we include figure 7 , which compares simplifications generated by recent instruction fine-tuned language models. the following are our key findings: models primarily write good edits, but still trail humans (fig. 4 , 5 ). we observe that 16% of modelgenerated edits are errors, with the best-performing model, few-shot gpt-3.5, producing errors in only 9% of edits. we find this still trails human simplifications, which have an error rate of 6%. muss and gpt-3.5 have a median count of 1 error per simplification and 63% of their simplifications contain at least one error, showing these errors are not concentrated in a few 'bad' simplifications but instead often occur among many good edits. language models elaborate, while humans generalize (fig. 4 ). when simplifying content, all models (excluding t5) tend to elaborate at a higher ratio than humans, for example, gpt-3.5 attempts to insert content 17% more often. as llms have shown to encode world knowledge in their parameters @xcite , gpt-3.5 elaboration is far more effective than muss, for example: ## evaluating metric edit sensitivity while automatic metrics are traditionally evaluated using correlation with sentence-level, likert scale human ratings on dimensions of adequacy, fluency and simplicity, this fails to understand the ability of automatic metrics to capture the subtleties of lexical, syntactic, and conceptual simplification. with our salsa annotations, we study how well current automatic metrics capture these distinct simplification approaches. additionally, we introduce lens-salsa, a reference-free metric fine-tuned on salsa annotations. ## word-level quality estimation word-level quality estimation (qe) is the task of predicting the quality of each token in a generation, and has substantial downstream application to evaluating and refining text simplification. despite word-level qe being a well understood task in machine translation @xcite , it has not yet been studied for text simplification due to a lack of appropriately annotated data. in this section, we use salsa annotations to demonstrate baseline approaches and highlight potential for future work. task. we define word-level simplification qe as classifying each token in the complex and simplified sentences as quality, error, or ok. to adapt salsa for the qe task, we label each token by the average efficacy/severity rating of its associated edit: < 0 as error, = 0 as ok, @xmath0. words that are not part of any edits default to the ok label. we deconstruct split and structure edits into their constituent edits, only label the simplified spans for substitution edits, and exclude reorder edits due to their low frequency. the final label counts for our train, validation, test splits are: 6.8k/1.8k/27k, 2.7k/627/11k, and 1.7k/484/6.9k for quality/error/ok respectively. ## related work model evaluation. simplification work broadly agrees some typology of simplification operations exists @xcite , starting with early rulebased systems which explicitly defined specific syntax operations @xcite . past work has experimented with designing models to control the extent of each operation by using a pipeline to perform simplification operations independently @xcite , predicting edit operations @xcite or augmenting finetuned models with learned control tokens @xcite . however, evaluation only considers a sentence in its entirety rather than rating individual operations, either by automatic metrics @xcite , shown to be an inadequate representation of quality @xcite , or by surface-level likert ratings, typically asking crowd-sourced annotators to rate on scales of fluency, adequacy, and simplicity. these scores are difficult to interpret and capture no detail into the type of simplification being written @xcite . additionally, despite current systems' often producing simplification errors @xcite , annotating error has primarily been performed through inspection, and has not been incorporated into human or automatic evaluation @xcite . linguistic inspection. manual inspection attempts to understand the behavior of simplification models or datasets, characterized by detailed typologies and often conducted by authors or domain experts. @xcite performs detailed inspection of the asset simplification test corpus (alva-manchego et al., 2020a) to study the behavior of automatic metrics and cumbicus-pineda et al. (2021a) propose a framework for evaluating success and failure by answering a series of checklist items, with sentences given a capability score based on the number of requirements fulfilled. yamaguchi et al. (2023) annotates simplifications of earlier models such as dress (zhang and lapata, @xcite and suc @xcite using a taxonomy of 62 error categories, but do not analyze the sota, muss, or llms. @xcite proposes an interactive linguistic inspection interface, but this interface is not designed for human evaluation of model outputs and does not provide ratings for measuring performance. fine-grained human evaluation. human evaluation performed on a span-level has been previously proposed for a variety of nlp tasks. in translation, the multidimensional quality metrics (mqm) @xcite , categorizes error into accuracy and fluency sub-types and is later extended by @xcite to weight errors by severity and combine into a single quality score. @xcite proposes scarecrow to capture errors appearing in open-ended text generation. however, as these span-based evaluation schemes exclusively annotate error, they encourage generic outputs and punish interesting or diverse generations. for summarization, the frank typology @xcite aggregates errors into broader categories to benchmark metrics that measure factuality. inspired by frank, @xcite introduces a framework to evaluate factuality for text simplification. ## conclusion in this work, we introduce salsa, a novel editbased evaluation framework incorporating error and quality evaluation, and dimensions of lexical, syntax and conceptual simplification and demonstrate salsa benefits in granularity, accuracy, and consistency. we employ salsa to collect a 19k edit annotation dataset and analyze the strengths and limitations of fine-tuned models, prompted llms, and human simplifications. finally, we use salsa annotations to develop a reference-free automatic metric for text simplification and demonstrate strong baselines for word-level quality estimation, showing promising avenues for the development of fine-grained human evaluation.