Simplicity Level Estimate ( SLE ): A Learned Reference-Less Metric for Sentence Simplification (2023)

Abstract: Automatic evaluation for sentence simplification remains a challenging problem. Most popular evaluation metrics require multiple high-quality references – something not readily available for simplification – which makes it difficult to test performance on unseen domains. Furthermore, most existing metrics conflate simplicity with correlated attributes such as fluency or meaning preservation. We propose a new learned evaluation metric — SLE — which focuses on simplicity, outperforming almost all existing metrics in terms of correlation with human judgements.

## introduction text simplification involves the rewriting of a text to make it easier to read and understand by a wider audience, while still expressing the same core meaning. this has potential benefits for disadvantaged end-users @xcite , while also showing promise as a preprocessing step for downstream nlp tasks @xcite @xcite . although some recent work considers simplification of entire documents @xcite the majority of work focuses on individual sentences, given the lack of high-quality resources @xcite @xcite . a major limitation in evaluating sentence simplification is that most popular metrics require highquality references, which are rare and expensive to produce. this also makes it difficult to assess models on new domains where labeled data is unavailable. another limitation is that many metrics evaluate simplification quality by combining multiple criteria (fluency, adequacy, simplicity) which makes it difficult to determine where exactly systems succeed and fail, as these criteria are often highly correlated -meaning that high scores could be spurious indications of simplicity @xcite 2021b). ## a metric for simplicity the sle metric. we propose sle, a learned metric which predicts a real-valued simplicity level for a given sentence without the need for references. given some sentence t, the system predicts a score sle(t) ∈ r, with high values indicating higher simplicity. this can not only be used as an absolute measure of simplicity for system output ŷ, but also to measure the simplicity gain relative to input x: in this paper we primarily focus on ∆sle, as it is the most applicable variant under common sentence simplification standards. model. as the basis for the metric, we fine-tune a pretrained roberta model foot_1 to perform regression over simplicity levels given sentence inputs, using a batch size of 32 and @xmath0. we ran training experiments on a computing grid with a nvidia a40 gpu. data. we use newsela @xcite , which consists of 1,130 news articles manually rewritten at five discrete reading levels (0-4), each increasing in simplicity. existing works often assume sentences have the same reading level as the document they are from @xcite ; however, we expect there to be a lot of variation in the simplicity of sentences within documents and overlap across levels. as such, merely training to minimize error with respect to these labels would likely result in mode collapse within levels (peaky, low-entropy distribution) and strong overfitting to the newsela corpus. to address this mismatch between document-and sentence-level simplicity, we take the following two mitigating steps to allow the model to better differentiate between sentences from the same reading level. label softening. we attempt to mitigate peakiness in the output distribution by softening the quantized reading levels assigned to each sentence in the training data. specifically, we interpolate regression labels throughout overlapping class regions (±1) according to their flesch-kincaid grade level (fkgl) @xcite . fkgl is a readability metric often used in education as a means to judge the suitability of books for students (high @xmath1. if l is the set of sentences belonging to some reading level, we define an intra-level ranking according to re-scaled, negative fkgls: from this, we derive the final revised labels: we report results for a model using softened labels (sle) as well as a variant using the original quantized labels (sle z ). ## results results on the regression task can be seen in correlations with human ratings of simplicity are shown in some studies from the automatic readability assessment (ara) literature use quantized newsela reading levels as labels to train regression models. @xcite do so in order to predict the readability of full documents, which does not extend to sentence simplification. @xcite predict a reading level accuracy within an rl reward for sentence simplification, but do so using the reading levels that were assigned to each document. this too does not transfer well to sentence-level evaluation, given the imprecision and noise introduced by the use of quantized ratings that were assigned at the document level. these approaches have not been applied to the actual evaluation of sentence simplification systems. ## future directions in this paper we explore the efficacy of sle as a measure of raw simplicity or relative simplicity gain (∆sle). however, given the flexibility of not relying on references, sle can potentially be used in other ways. for example, one could measure an error with respect to a target simplicity level, l * : this could be useful in the evaluation of controllable simplification systems, which should be able to satisfy simplification requirements of specific user groups or reading levels @xcite @xcite . as it is trained with aggregate document-level accuracy in mind, sle could also possibly be used to evaluate document simplification -either via averaging sentence scores or using some other aggregation method. ## conclusion in this paper we presented sle -a reference-less evaluation metric for sentence simplification that is competitive with or better than the best performing reference-based metrics in terms of correlation to human judgements of simplicity. we reconsider the ability of popular metrics to accurately gauge simplicity when controlling for other factors such as fluency and semantic adequacy, confirming suspicions that many do not measure simplicity directly. we hope this work motivates further investigation into the efficacy of standard simplification evaluation techniques and the proposal of new methodologies.