Adapting Sentence-level Automatic Metrics for Document-level Simplification Evaluation (2025)

Abstract: Text simplification aims to enhance the clarity and comprehensibility of a complex text while preserving its original meaning. Previous research on the automatic evaluation of text simplification has primarily focused on sentence simplification, with commonly used metrics such as SARI and advanced metrics such as LENS being trained and evaluated at the sentence level. However, these metrics often underperform on longer texts. In our study, we propose a novel approach to adapt existing sentence-level metrics for paragraph- or document-level simplification. We benchmark our approach against a wide variety of existing reference-based and reference-less metrics across multiple domains. Empirical results demonstrate that our approach outperforms traditional sentence-level metrics in terms of correlation with human judgment. Furthermore, we evaluate the sensitivity and robustness of various metrics to different types of errors produced by existing text simplification systems.

## introduction text simplification involves rewriting a text to improve its ease of understanding, while maintaining the original meaning @xcite . this refinement greatly improves the readability of documents, making them more accessible to diverse audiences, including children @xcite , non-native speakers @xcite , and individuals with learning disabilities @xcite . text simplification also makes specialized documents, such as medical articles @xcite and legal texts @xcite , easier to understand for non-expert readers. one major obstacle for text simplification is reliable automatic evaluation of simplified texts. while the simplification of long texts, such as documents and paragraphs, holds practical utility, existing research has primarily focused on the automatic evaluation of sentence simplification @xcite @xcite . commonly used metrics for text simplification, such as sari @xcite , which measures n-gram overlap between the simplified text and human references, and bertscore @xcite , which assesses semantic similarity using bert embeddings @xcite , are primarily designed for sentence-level evaluation. although @xcite propose a variant of sari for longer texts called dsari, lexical overlap metrics like sari and dsari struggle to effectively capture paraphrases @xcite . in contrast, semantic similarity metrics such as bertscore focus on meaning preservation, often lacking correlation with simplicity @xcite . on the other hand, the rise of pretrained language models has led to the development of supervised metrics such as lens @xcite and referee @xcite , which are fine-tuned on human judgments and effectively capture diverse styles of simplification. however, these metrics are primarily trained and evaluated for sentence simplification, resulting in suboptimal performance on longer texts. for instance, figure 1 demonstrates that lens, the state-of-the-art metric for sentence simplification, exhibits the highest correlation with human judgments on shorter texts (under 200 tokens) but shows diminished performance on longer texts. here, source texts and references are from the cochrane dataset @xcite , and human judgments for the simplified texts are collected by @xcite . section 3 and to address these limitations, we propose a simple yet effective method to adapt sentence-level metrics for paragraph-or document-level simplification. our approach first decomposes long texts into shorter segments using a specialized semantic similarity model and a graph-based alignment strategy. it then employs a sentence-level metric to compute evaluation scores for these shorter texts and aggregates the results. this method can be applied to any reference-based or reference-less metric, with or without the source document. we compare our proposed approach to a wide variety of existing simplification metrics in terms of correlation with human judgments, robustness, and sensitivity to minor errors. empirical results demonstrate that our approach enhances the correlation of sentence-level metrics across three domains: wikipedia, news, and medical texts. our approach also boosts the robustness of existing metrics on longer texts as illustrated in fig 1 (agg-lens vs lens). our main contributions include: (a) a novel approach for adapting sentence-level metrics to long text simplification; (b) benchmarking this approach alongside a comprehensive set of reference-based and reference-less simplification metrics across multiple domains; and (c) evaluating the sensitivity and robustness of automatic metrics to various types of errors produced by existing text simplification systems. ## aggregating sentence-level metrics for long texts in this section, we present a novel method for adjusting sentence-level metrics to evaluate paragraph-or document-level simplification. we employ a specialized sentence alignment model @xcite alongside a graph alignment strategy to identify smaller units of related text across the input, simplified, and reference texts. these related texts can encompass multiple sentences, allowing our graph alignment strategy to effectively handle multi-sentence simplification edits, such as sentence reordering, fusing sentences, and selecting relevant content across sentences @xcite . we then calculate metric values for these smaller units and average them to derive the final metric value. foot_0 step 1: construct similarity matrices. given a complex text @xmath0, . . . , c i , . . . , c m ), its simplification @xmath1, . . . , s j , . . . , s n ), a reference @xmath2, . . . , r k , . . . , r p ), and a sentence-level metric m (.), the goal is to compute a score z that captures the overall quality of s. here, c i , s j , and r k correspond to sentences in c, s, and r respectively. first, we compute two sentence similarity matrices: a @xmath3, s j ) and a @xmath4, r k ). we utilize jiang et al. (2020)'s sentence pair similarity model to construct a cs and a cr . this model was trained to measure similarity of sentences in parallel complex articles and their simplified versions. 2 however, our approach is agnostic to the type of sentence aligner and can be replaced with any aligner that better suits the target dataset or task, offering flexibility for different contexts. ## evaluating correlation with human judgements in this section, we benchmark existing simplification metrics and their aggregated versions constructed using our proposed approach. ## evaluating robustness using adversarial attacks while correlation with human ratings gauges the capability of automatic metrics to evaluate the overall quality of simplification, it may not effectively capture how sensitive metrics are to minor errors in simplification system outputs. considering that state-of-the-art generation models such as gpt4 @xcite are capable of producing high-quality text, it is crucial for evaluation metrics to detect even minor errors. in this section, we outline various types of errors typically generated by paragraph-level simplification systems and suggest methods for perturbing a well-written simplification to introduce each error with minimal alterations. subsequently, we present results obtained by different automatic metrics in detecting these minor errors. ## ablation analysis we analyze the design decisions that are essential for the effective performance of our approach: (a) a threshold of 0.5 for sentence pair similarity, (b) graph-based alignment of sentences, and (c) sentence pair similarity model trained on parallel simplification corpora. ## conclusion in this work, we propose a novel approach for adapting sentence-level automatic metrics for long text simplification. results show that our approach enhances the correlation with human judgments of sentence-level metrics across multiple domains. we also conduct the first systematic study of automatic evaluation metrics for document-level simplification by benchmarking a comprehensive range of metrics, spanning traditional lexical and semantic measures to recent learned approaches. finally, we evaluate the robustness of simplification metrics using adversarial attacks that simulate different errors made by long text simplification systems.