Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension (2024)

Abstract: Automatic text simplification (TS) aims to automate the process of rewriting text to make it easier for people to read. A pre-requisite for TS to be useful is that it should convey information that is consistent with the meaning of the original text. However, current TS evaluation protocols assess system outputs for simplicity and meaning preservation without regard for the document context in which output sentences occur and for how people understand them. In this work, we introduce a human evaluation framework to assess whether simplified texts preserve meaning using reading comprehension questions. With this framework, we conduct a thorough human evaluation of texts by humans and by nine automatic systems. Supervised systems that leverage pre-training knowledge achieve the highest scores on the reading comprehension tasks among the automatic controllable TS systems. However, even the best-performing supervised system struggles with at least 14% of the questions, marking them as “unanswerable” based on simplified content. We further investigate how existing TS evaluation metrics and automatic question-answering systems approximate the human judgments we obtained.

## introduction rewriting text so that it is easier to understand has the potential to help a wide range of audiences including non-native speakers @xcite @xcite , children @xcite , or people with reading or cognitive disabilities @xcite access information more easily @xcite . online resources such as newsela inc (2023) and onestopenglish @xcite or the cochrane systematic reviews @xcite , provide text articles simplified by human editors so that they are easier * work done while at the university of maryland. to understand by k-12 students, english speakers with limited proficiency, and lay people seeking to understand medical literature, respectively. this has motivated a wealth of natural language processing research on text simplification, framed as the task of rewriting an input text into a simpler version while preserving the core meaning of the original @xcite , which has been addressed with approaches ranging from dedicated supervised systems @xcite @xcite @xcite @xcite @xcite to prompting black-box pre-trained models @xcite . however, texts that are easier to read are not helpful if they mislead readers by providing information that is not consistent with the original document. this can happen with automatic text simplification (ts) outputs where deletions or inaccurate rewrites can change how a text is understood @xcite . assessing to what extent the meaning of the original text is preserved should therefore be a critical dimension of ts evaluation @xcite , and a pre-requisite to determining whether and how ts can be used in practice. additionally, evaluating individual sentences out of context may not be sufficient to establish whether model-generated texts preserve meaning, as human simplifications often occur at the document or the passage level @xcite . yet, ts outputs are primarily evaluated intrinsically, with automatic metrics that compare system outputs with human-written reference simplifications and/or the original source @xcite @xcite , or with generic human assessments of simplicity and meaning preservation of individual sentences outside of a context of use @xcite . while these evaluations can guide model development, they do not address whether readers get information from the simplified text that is consistent with the original content. in this work, we conduct a human evaluation of the ability of state-of-the-art ts systems to preserve the meaning of the original text by measuring how well people can answer questions about key facts from the original text after reading a simplified version. we design reading comprehension (rc) tasks to directly assess meaning preservation in ts, different from prior uses of reading comprehension to assess people's reading efficiency @xcite . this framework lets us conduct a controlled comparison of simplified texts, whether written by humans or by ts systems: we compare people's ability to answer questions about the original text, a simplified version written by humans, and nine ts-generated versions that represent a diverse set of supervised and unsupervised approaches from the recent ts literature. 1 we first discuss relevant literature for ts evaluation and the use of rc exercises to assess simplified or other model-generated texts in section 2. next, section 3 elaborates on our rc-centered human evaluation framework, and section 4 delves into the various design choices we made. section 5 demonstrates the robustness of our evaluation and presents the main results. as we will see, supervised systems that utilize pre-training knowledge achieve the highest level of accuracy in rc tasks compared to other automatic controllable ts systems ( §5.2). however, at least 14% of the questions remain unanswerable even for the best-performing system due to the errors introduced by these systems ( §5.3). in section 6, we shift our focus towards a meta-evaluation of existing automatic ts evaluation metrics which indicates that the 3-way comparison used in sari makes it a reliable metric for system-level evaluation at the paragraph level. finally, we include a preliminary discussion and analysis of the potential for automating the rc-based evaluation through the application of model-based question-answering techniques in section 7. 1 collected annotations and code are released at https:// github.com/sweta20/ats-eval.git . ## background how to design human and automatic evaluation protocols for ts is a research question unto itself. while automatic metrics are key to system development, commonly used metrics like bleu @xcite , sari @xcite , or the flesch-kincaid grade level @xcite have low correlation with human judgments of simplicity @xcite @xcite . this suggests that these metrics can fail to capture meaningful differences between simplified texts. furthermore, there is no standardized framework for measuring the adequacy of simplified outputs @xcite , where adequacy refers to the degree to which the generated text accurately conveys the meaning from the original text @xcite ). 2 prior work highlights the importance of manually evaluating ts systems. for instance, @xcite introduce rank & rate, a human evaluation framework that rates simplifications from several models at the sentence level by leveraging automatically annotated edit operations that are verified by annotators. these edit operations are then used in rating the output texts on a scale of 0 to 100. however, this rating is meant to jointly account for meaning preservation, simplicity, and fluency. @xcite show that factual errors often appear in both human and automatically generated simplified texts (at the sentence level), and define an error taxonomy to account for both the nature and severity of errors. yet, these intrinsic evaluations do not directly tell us whether people correctly understand key facts conveyed in the original after reading a simplified text. moreover, the evaluation is only performed at the sentence level without accounting for the context in which they appear which can impact the overall assessments as noted by @xcite . reading comprehension tests are standard tools used by educators to assess readers' understanding of text materials, and thus provide an assessment of ts that is more in line with its intended use. they have been used to show that human-simplified texts are easier to comprehend by l2 learners @xcite @xcite @xcite , as well as secondary and post-secondary students @xcite . for instance, long and ross (1993) conduct a reading comprehension study with 483 japanese students with varying english language proficiency levels and found that participants who had access to linguistically simplified or elaborated texts scored higher on the rc tasks than those who read the original text. similarly, @xcite showed a linear effect of text complexity on comprehension even when accounting for individual language and reading proficiency differences as well as their background knowledge. rets and rogaten's (2021) study with 37 adult english l2 users showed better comprehension and faster recall for participants with low english proficiency levels with simplified texts. finally, in a within-subject study with four original and four simplified texts involving 103 participants with varying levels of english proficiency (beginner to native), @xcite show that utilizing controlled language @xcite for ts improves reading comprehension. all these studies are conducted at the paragraph or the document level based on human-written simplifications which are implicitly assumed to be correct. the use of reading comprehension for the evaluation of automatically simplified texts has been more limited. @xcite first used reading comprehension to evaluate automatically simplified texts from multiple ts models, with non-native readers of english. they conduct a multiple-choice test using five news summaries chosen from the breaking news english website, originally at reading level 6 (hard) and simplified manually or via automatic ts systems. their study found no significant differences between the comprehension accuracy of different user groups when reading automatically generated simplifications. however, they note that the drop in comprehension scores for some of these systems could be accounted to the content removal which can make some questions unanswerable. hence, it is not clear whether the differences are non-significant due to user understanding, errors introduced by ts systems, or the effectiveness of simplifications. @xcite also conduct a reading comprehension study to evaluate the usefulness of automatic ts outputs with automatically generated questions that can be answered by original text and human-written references. they found that shorter passages generated by automatic ts systems lead to a speed-up in the rc task completion time regardless of simplicity. however, the automatic generation of questions mostly limited them to factoid, thus limiting the scope of understanding tests, and it is unclear whether the ts errors could render the rc questions unanswerable. evaluating text generation via automatic question answering (qa) has also received much attention, including for machine translation @xcite , and for text summarization, where it has been used to assess the factuality @xcite or faithfulness @xcite of model-generated summaries. for summarization evaluation, questions have been automatically created based on key information from the model-generated summary, such as important nouns or entities. an automatic qa system is then employed to generate answers to these questions using the original document as a reference. the quality of the generated summary is determined by comparing these answers using metrics that measure semantic similarity or exact matches. however, unlike summarization, where the primary goal is to condense a text (either in an extractive or abstractive fashion), text simplification also involves making structural and linguistic changes so that the text is easier to comprehend which the existing automatic qa-based evaluations are not equipped to assess. in this work, we design a reading comprehension task to assess the ability of ts systems to preserve meaning via carefully constructed multiplechoice questions targeting language comprehension and use it to conduct a thorough controlled evaluation of a diverse set of state-of-the-art ts systems. we conduct our human evaluation at the paragraph level as humans naturally tend to simplify complex text at this granularity, and utilizing complete texts for measuring rc would yield more accurate results compared to relying on individual sentences @xcite . ## a reading comprehension-based human evaluation framework overview our human evaluation is based on the following task: participants are presented with text and then are asked questions to test their understanding of some of the information conveyed in the text, as illustrated in figure 1 . we seek to measure whether participants who read simplified versions of the original paragraph can answer questions as well as those who read the original. however, our goal is not to assess the participants but the ts systems: when working with participants who are proficient in the language tested, we assume that differences in reading comprehension accuracy indicate differences between the quality of ts systems that produce the different simplifications. onestopqa within this simple framework, the design of the rc questions and answers is critical to directly evaluate the correctness of automatic ts systems. we build on the onestopqa reading comprehension exercises created using the starc (structured annotations for reading comprehension) annotation framework @xcite , which is well suited to our task since it targets the real-world need of supporting readers with low english proficiency, and there is already evidence that it is a sound instrument to capture differences in reading comprehension from human-written text. specifically, onestopqa is based on texts from the onestopenglish.com english language learning portal @xcite , which are drawn from the guardian newspaper. questions are designed to assess language comprehension rather than numerical reasoning or extensive external knowledge. more importantly, these questions cannot be answered with simple string-matching and guessing strategies. furthermore, the answer options under the starc annotation framework follow a structured format that reflects four fundamental types of responses, ordered by miscomprehension severity: a indicates correct comprehension, b shows the ability to identify essential information but not fully comprehend it, c reflects some attention to the passage's content, and d shows no evidence of text comprehension @xcite . participants are presented with the answer options in a randomized order to minimize any potential bias or pattern recognition. the correct answer typically is not present verbatim in the critical span, a text span from the passage upon which the question is formulated. 3 we note that the questions only target a subset of the information conveyed in a passage, and hence, our evaluation framework does not provide a measure of completeness. in other words, correctly answering the rc questions does not require understanding every piece of salient information from the original. 4 further, prior work suggests that on-estopenglish text and onestopqa questions provide a sound basis for evaluating automatic ts, as they can capture differences in reading comprehension from manually simplified text: @xcite found a statistically significant difference between users scrolling interactions and the text difficulty level in a 518-participant study and @xcite showed that the nature of the reading comprehension questions can impact text understanding. ## experimental setup first, we describe experiment details including data, participant selection, and study design. then, we outline the selected ts systems for evaluation. ## results we first analyze the results to show the validity of the evaluation set-up, before comparing ts systems on the accuracy and answerability metrics. ## evaluating automatic ts evaluation metrics we now turn towards investigating to what extent automatic ts evaluation metrics frequently used in the literature capture the system rankings obtained via the rc task (al-thanyyan and azmi, 2021; @xcite . we compute the spearman-rank correlation of the system-level scores using selected automatic metrics and the rc accuracy scores in furthermore, qafacteval exhibits only a weak correlation (0.395) at best with human judgments. this is consistent with the current findings by @xcite , who discuss and show how automatically extracting facts from summaries could lead to a fundamental problem in the evaluation where current qa-based frameworks not only struggle to accurately identify errors in the generated summaries but also perform worse than straightforward exact match comparisons. ## model-based question answering our evaluation so far has relied on human-written questions answered by crowd workers, using 13 https://github.com/salesforce /qafacteval . either model-generated or human-written texts. automating one or both components would help scale the evaluation and port it to new settings more flexibly. recent work suggests that this might be plausible: @xcite show that automatically generated questions and answers can be used to evaluate machine translation systems at the sentence level, and automatic qa techniques @xcite have been used to assess the factuality and faithfulness of summarization systems. here, we assess the performance of a state-of-the-art qa system in recovering the gold-standard ranking induced by human judgments, leaving the more complex study of multiple-choice rc question generation to future work. we use unifiedqa v2 foot_8 a qa model, trained to answer questions in 4 different formats using 20 different datasets. this model has been shown to support better generalization to unseen datasets compared to models specialized for individual datasets. we use the format recommended in the original paper: {question} \n (a) {choice 1} (b) {choice 2} ... \ n {paragraph} to generate answers for all the conditions. the spearman-rank correlation between exact match (em) and ground truth accuracies (c) for all systems is 0.838 and 0.744, excluding kis. however, we note that the system's ability to distinguish closely competing systems (highlighted by the same color) is limited, as shown in figure 6 . interestingly, qa using human-simplified text achieves higher accuracy than using original unmodified text. this finding is in line with prior work where ts has been shown to improve the performance of multiple downstream nlp tasks such as information extraction @xcite , parsing @xcite , semantic role labeling @xcite , machine translation @xcite @xcite @xcite , among others @xcite . this suggests that automating part of the evaluation framework is a direction worth investigating in more depth in future work. ## conclusion we introduced an evaluation framework based on reading comprehension to directly assess whether ts systems correctly convey salient information from the original texts to readers. this framework lets us conduct a thorough human evaluation of the adequacy of 10 simplified texts: a human-written version and outputs from nine ts systems. supervised systems that leverage pre-trained knowledge (muss, t5) produce texts that lead to the highest reading comprehension accuracy, approaching the scores obtained on human-written texts. prompted llms (chatgpt) perform well but are not as accurate as supervised systems. however, we find that even those systems do not preserve the meaning of the original text, with at least 14% of questions marked as ''unanswerable'' on the basis of the text they generate. when human evaluation is not practical, our analysis suggests that sari is a better metric than meaning-preservation metrics such as bertscore and bleu to rank systems by adequacy, and that model-based qa can approximate system rankings but at the cost of reduced discriminative power across systems and can introduce other confounding factors. overall, these results confirm the importance of directly evaluating the accuracy of the information conveyed by ts systems, and suggest that while some systems are overall correct enough to warrant usability studies, all systems still make critical errors. this motivates future work on machine-in-the-loop workflows to let editors and readers rely on ts appropriately @xcite , and on improving the over-deletion of content by current ts systems. our human evaluation framework provides a blueprint for evaluating whether correct ts outputs improve reading comprehension for people who have difficulty understanding complex texts, which we intend to investigate in future work.