Gauging the Gap Between Human and Machine Text Simplification Through Analytical Evaluation of Simplification Strategies and Errors (2023)

Abstract: This study presents an analytical evaluation of neural text simplification (TS) systems. Because recent TS models are trained in an end-to-end fashion, it is difficult to grasp their abilities to perform particular simplification operations. For the advancement of TS research and development, we should understand in detail what current TS systems can and cannot perform in comparison with human performance. To that end, we first developed an analytical evaluation framework consisting of fine-grained taxonomies of simplification strategies (at both the surface and content levels) and errors. Using this framework, we annotated TS instances produced by professional human editors and multiple neural TS systems and compared the results. Our analyses concretely and quantitatively revealed a wide gap between humans and systems, specifically indicating that systems tend to perform deletions and local substitutions while excessively omitting important information, and that the systems can hardly perform information addition operations. Based on our analyses, we also provide detailed directions to address these limitations.

## introduction text simplification (ts) is the task of reducing the content and structural complexity of text while retaining the core part of the original meaning (alva-manchego et al., 2020). ts can not only facilitate the text reading by children or language learners, but also improve the performance of downstream nlp applications, including machine translation and summarization (siddharthan et al., 2004; štajner and popovic, 2016). early studies on ts have separately dealt with lexical simplification (glavaš and štajner, 2015) and syntactic simplification (scarton et al., 2017), and developed simplification techniques specialized for particular linguistic phenomena. in contrast, recent studies have tackled ts as a task of aarti has a growth mindset. ## related work to evaluate ts systems, automatic evaluation metrics such as sari (xu et al., 2016), bleu (papineni et al., 2002), and flesch-kincaid grade level (fkgl) (kincaid et al., 1975) are widely used. sari and bleu use n-gram overlap between target sentences and human-created references, whereas fkgl uses the number of syllables and words in the output sentences. these metrics can be easily calculated if references are available and are indispensable for the rapid cycle of system development and evaluation. however, their limitations and pitfalls have been acknowledged. sulem et al. (2018a), for example, reported that bleu is negatively correlated with human evaluation scores, such as simplicity and grammaticality. tanprasert and kauchak (2021) also showed that the fkgl score can easily be manipulated by minor modifications, such as adding periods randomly. subjective human evaluation has also been implemented (štajner and nisioi, 2018; sulem et al., 2018b; al-thanyyan and azmi, 2021). in many cases, certain aspects of ts quality, namely grammaticality/fluency, meaning preservation/adequacy, and simplicity, are rated on a three-to five-point likert scale based on the evaluation criteria. importantly, all the abovementioned evaluation methods only provide summative numerical scores. these scores are useful for comparing the general performances of different systems, but do not necessarily provide a guidepost for achieving higher system performance. to gain a detailed understanding of what ts systems can/cannot do vis-à-vis editing operations by humans, analytical evaluation methods are required. the analytical evaluation of ts can be broadly divided into strategy and error analyses. the former concerns the type of editing operation (strategy) performed to produce the simplified text. previous studies have acknowledged general strategies, such as paraphrasing, deletion, and splitting (shard-low, 2014), and document-level strategies, such as sentence reordering and sentence-joining operations (alva-manchego et al., 2019b). however, these roughly typify superficial textual changes rather than detailed content-level changes that capture editing operations peculiar to ts. the latter concerns the type of error in the resulting simplified text. in contrast to automatic and human evaluations, fewer attempts have been made to conduct an error analysis (maddela et al., 2021). @xcite proper implementation of analytical evaluation requires well-formulated frameworks to classify textual phenomena observed in the outputs. the general editing strategies mentioned above and some guidelines for human writers (mitkov and štajner, 2014) are not sufficiently concrete for finegrained analysis. although several typologies of simplification operations (e.g., amancio and specia, 2014; brunato et al., 2014; koptient et al., 2019) and editing guidelines for human writers (mitkov and štajner, 2014) have been proposed, the following limitations can be generally acknowledged: (1) content-level operations are not fully covered; (2) their applicability to outputs of automatic ts systems has not been verified. in the field of translation studies, a wide variety of translation strategies have been proposed to describe the differences between source and target texts (e.g., vinay and darbelnet, 1958; molina and hurtado albir, 2002). chesterman (2016), for example, developed a comprehensive taxonomy of translation strategies that consists of syntactic, semantic, and pragmatic categories, and each includes ten strategies. taxonomies of translation errors have also been developed and are widely used in practice, such as multidimensional quality metrics (mqm) (lommel et al., 2014). although these existing frameworks may be useful as points of departure, detailed ones dedicated to ts tasks are still lacking. ## framework of analytical evaluation we developed taxonomies of simplification strategies and errors as the analytical evaluation framework for ts. simplification strategies consist of two independent components: surface strategies, which capture superficial operations for grammatical or textual elements, and content strategies, which capture semantic or content changes from the viewpoint of simplification. in this framework, each ts instance (i.e., a minimally decomposed editing operation) is first judged as an error category listed in the error taxonomy. if it is not, the instance is then independently labeled a surface and content strategy (see also figure 1 ). our framework includes guidelines for annotating surface and content strategies in the form of a decision tree. 2 ## experimental setup to clarify the potential and limitations of current ts systems in comparison with human performance, we designed an experiment to annotate ts instances produced by human editors and recent neural ts systems using our taxonomies of simplification strategies and errors described in §3. we also conducted a human evaluation to better understand the general tendencies of how strategies and errors affect the ts quality. ## conclusions and outlook to better advance ts research and practice, in this study, we conducted an analytical evaluation of current neural ts systems and showed their potential and limitations in comparison with human performance. using our proposed evaluation framework consisting of taxonomies of surface strategies, content strategies, and errors, we annotated both the human references and outputs of six systems (three models trained on in-domain and out-of-domain datasets). the results demonstrated that, while current ts systems can perform deletions and local substitutions, their performance is far behind human parity, owing to the following limitations: • the systems have difficulties in substituting a linguistic unit larger than a phrase, including sentence splitting. our analytical evaluation also suggests detailed paths to overcome these issues. for example, in addition to improving the capacity of end-to-end neural models, utilizing technologies tailored to particular operations such as sentence splitting and explanation generation can be helpful. to mitigate the excessive deletion, it would be effective to refine the alignment methods. exploiting documentlevel contexts on both input and output sides and/or document-external knowledge is a necessary task for successful content addition.