LENS : A Learnable Evaluation Metric for Text Simplification (2023)

Abstract: Training learnable metrics using modern language models has recently emerged as a promising method for the automatic evaluation of machine translation. However, existing human evaluation datasets for text simplification have limited annotations that are based on unitary or outdated models, making them unsuitable for this approach. To address these issues, we introduce the SimpEval corpus that contains: SimpEval_past, comprising 12K human ratings on 2.4K simplifications of 24 past systems, and SimpEval_2022, a challenging simplification benchmark consisting of over 1K human ratings of 360 simplifications including GPT-3.5 generated text. Training on SimpEval, we present LENS, a Learnable Evaluation Metric for Text Simplification. Extensive empirical results show that LENS correlates much better with human judgment than existing metrics, paving the way for future progress in the evaluation of text simplification. We also introduce Rank & Rate, a human evaluation framework that rates simplifications from several models in a list-wise manner using an interactive interface, which ensures both consistency and accuracy in the evaluation process and is used to create the SimpEval datasets.

## introduction text simplification is a text-to-text generation task that aims to make a text easier to read while preserving its original meaning @xcite . automatic evaluation of text simplification is challenging because a sentence can be simplified in many ways, such as paraphrasing complex words, deleting insignificant information, and splitting long sentences into shorter ones. an ideal automatic metric should accommodate these diverse choices while capturing semantic similarity and fluency. however, existing metrics such as sari @xcite and bertscore @xcite struggle original sentence from our simpeval 2022 : the book is illustrated with more than 200 newly commissioned color photographs by the british architectural photographer morley von sternberg, as well as older images where needed to illustrate buildings that have been significantly altered or demolished. ne w be nch ma r ne w be nch ma rk! reference (written by human): the book contains more than 200 newly commissioned color photographs by morley von sternbeg. it also contains older images which show buildings that were changed significantly or destroyed. more than 200 brand-new, colored photographs have been taken by british architectural photographer morley von sternberg for this book. additionally, older images have been used to show buildings that have changed or been destroyed. the book has more than 200 newly commissioned colour photographs by the british architectural photographer morley von sternberg. there are also older images where needed to show buildings that have been significantly altered or demolished. the book is illustrated with more than 200 newly commissioned color photographs by the british architectural photographer morley von sternberg, as well as older images that have not been used a lot since the book was first published. simplifications: figure 1 : automatic metric and human evaluation scores on simplifications of a complex sentence from our simpeval 2022 . lens achieves the best correlation with humans. sari penalizes simplifications that deviate from the reference (first two examples), and bertscore fails to penalize hallucinations with high lexical overlap (third example). to capture all the aspects and achieve a high correlation with human evaluation (alva-manchego et al., 2021) (see figure 1 ). these metrics fail even more when evaluating high-quality systems that have close performance, calling for a more robust and accurate metric for text simplification. prior work on machine translation @xcite has seen the success of using language models as an automatic metric by training them on human judgments, such as direct assessment (da) that rates generations on a 0-100 scale. however, existing human evaluation datasets @xcite ) for text simplification are not suitable for training metrics because they include a limited number of annotations or systems. besides, these datasets do not include state-of-the-art generation models such as gpt-3.5 @xcite that have been shown to generate human-like quality text @xcite . in this work, we introduce lens, a learnable evaluation metric for text simplification. lens is the first supervised metric for the task and uses an adaptive ranking loss to promote fair comparison of simplifications that have undergone different edits (i.e., splitting, paraphrasing, and deletion). to train lens, we collect 12k human judgments on 2.4k simplifications by 24 simplification systems from the literature, which we name as the simpeval past dataset. we also create simpe-val 2022 to evaluate lens and other metrics in a realistic and challenging setting of assessing simplifications by state-of-the-art language models on the more recent, complex, and longer sentences published on wikipedia after oct 22nd, 2022. sim-peval 2022 contains over 1k human ratings on 360 simplifications generated by 4 sota models, including gpt-3.5, and 2 humans. empirical experiments show that lens achieves a higher correlation of 0.331 with human ratings on simpeval 2022 , which is more than twice as high as the correlation scores of 0.112 and 0.149 by bertscore and sari, respectively. we further demonstrate that incorporating lens into decoding process, using minimum bayes risk framework @xcite , can directly improve the automatic text simplification system's performance. we expect that our data collection method, including rank & rate, a list-wise human evaluation interface, can be easily adapted to other text generation tasks. ## automatic evaluation metric to tackle the challenge of limited human evaluation data, we curate simpeval, a corpus containing over 13k human judgements on 2.8k simplification from 26 systems. this facilitates the training and evaluation of lens (a learnable evaluation metric for text simplification), the first supervised automatic metric for text simplification evaluation. in this section, we first describe the creation of simpeval datasets in §3.1 and then lens in §3.2. ## experiments we benchmark lens metric against the existing automatic metrics (i) for evaluating text simplification system outputs and (ii) for training better represent the kendall tau-like correlation for paraphrase-focused, split-focused, and all simplifications, respectively. we report the pearson correlation coefficients along three dimensions for wiki-da and newsela-likert. the best values are marked in bold and the second best values are underlined. figure 3 : the 95% confidence intervals for kendall tau-like correlation (τ all ) on simpeval 2022 and for pearson correlation with simplicity ratings on wiki-da and newsela-likert, calculated by bootstrapping @xcite . lens is more reliable with smaller intervals and has higher correlation with human judgments. ## lens analysis in this section, we delve into the impact of the underlying model architecture and the number of references on the performance of lens. model architecture. ## rank & rate framework to facilitate consistent evaluation and enable the training of lens, we develop rank & rate , a human evaluation framework to assist annotators in efficiently comparing and rating many (>20) system outputs at once in a list-wise ranking manner. ## conclusion we introduce lens, the first supervised automatic metric for text simplification. we show that lens exhibits higher human correlation than other automatic metrics. we also introduce rank & rate framework, which allows annotators to evaluate multiple systems' outputs at once in a list-wise manner. using it, we create simpeval past to train lens, and simpeval 2022 as a new metric evaluation benchmark for text simplification. we hope our metric, data, and framework will facilitate future research in text simplification evaluation.