Evaluation of Automatic Text Simplification: Where are we now, where should we go from here (2022)

Abstract: Évaluation de la simplification automatique de textes : où nous en sommes et vers où devonsnous aller. L’objectif de la simplification automatique de textes consiste à adapter le contenu de documents afin de les rendre plus faciles à comprendre par une population donnée ou bien pour améliorer les performances d’autres tâches TAL, comme le résumé automatique ou extraction d’information. Les étapes principales de la simplification automatique de textes sont plutôt bien définies et étudiées dans les travaux existants, alors que l’évaluation de la simplification reste sous-étudiée. En effet, contrairement à d’autres tâches de TAL, comme la recherche et extraction d’information, la structuration de terminologie ou les questions-réponses, qui s’attendent à avoir des résultats factuels et consensuels, il est difficile de définir un résultat standard de la simplification. Le processus de simplification est très subjectif et souvent non consensuel parce qu’il est lourdement basé sur les connaissances propres des personnes. Ainsi, plusieurs facteurs sont impliqués dans le processus de simplification et son évaluation. Dans ce papier, nous présentons et discutons quelques uns de ces facteurs : le rôle de l’utilisateur final, les données de référence, le domaine des documents source et les mesures d’évaluation.

## introduction the purpose of automatic text simplification is to adapt the content of documents in order to make them easier to understand @xcite by a given population including children @xcite @xcite , foreigners or people with low literacy @xcite , people with neurodegenerative disorders @xcite , or people without specialized knowledge @xcite @xcite ). text simplification can also help other nlp applications and has been used for example to prepare documents further for syntactic analysis @xcite , semantic annotation @xcite , summarization @xcite , automatic translation @xcite , indexing @xcite , or information retrieval and extraction @xcite . simplification can be performed at different linguistic levels : lexical, syntactic, semantic, or pragmatic. the main steps and requirements for the automatic text simplification systems are now quite well defined and researched. we can mention for instance text difficulty assessment, complex words identification, disambiguation, word difficulty ranking, building of models and resources, or sentence alignment. yet, the evaluation of simplification remains understudied. indeed, contrary to other nlp tasks, like information retrieval and extraction, terminology structuring, or question-answering, which expect factual and consensual outputs of the systems, it is difficult to define standard output of simplification : (1) it is not factual because it relies on series of transformations more or less managed by human simplificators and automatic systems, and (2) it is not consensual because it is heavily based on own knowledge of people and because everyone has an opinion on the simplification output. as it has been noticed, native simplified-language speaker does not exist @xcite . hence, several factors are involved in the simplification process and its assessment. in this paper, we propose to discuss the role of four important factors : the role of end users (section 2), the reference data (section 3), the domain of source documents (section 4), and the evaluation measures (section 5). ## end user different users may need different simplification strategies. for instance, children do not have the same needs as people with neurodegenerative disorders or foreigners : types of documents and their content differ, as well as required adaptation of documents. besides, the level of literacy of people varies a lot, including within the a given population (children, foreigners...). six standard literacy levels are usually distinguished @xcite , and rely on the following skills : 0. read brief texts on familiar topics, locate a single piece of information, know basic vocabulary, 1. read short texts, locate synonymous information, recognize basic vocabulary, determine the meaning of sentences, 2. match between the text and information, paraphrase, make low-level inferences, 3. read and navigate in dense, lengthy or complex texts, 4. integrate, interpret information from complex texts, identify and understand non-central ideas, interpret or evaluate subtle evidence-claim or persuasive discourse relationships, 5. search for, and integrate, information across multiple texts, construct syntheses of similar and contrasting ideas, evaluate evidence based arguments, understand subtle cues, make high-level inferences, use specialized background knowledge. the levels 2 or 3 correspond to the high school level : persons can read and understand quite complex non-specialized texts, make some inferences and paraphrase content. to ensure that a given person belongs to a given literacy level, specific tests are used. such tests are specific to a given population, needs, and language, like those dedicated to health literacy in different languages @xcite @xcite . usually, the real user is not involved in the evaluation leaving unexplored the question of how useful the simplification is. to perform a correct evaluation of the simplification, the evaluation must be user-dependent and consider the needs of final users. this accounts for instance for the type of target population, its literacy level, the type of documents, the readability level of these documents, the expected simplification rules (lexical, syntactic...), and the expected simplification level. ## reference data the reference data play a very important role in the development and evaluation of simplification systems. several approaches exist for the creation of such data, which we present and discuss : hence, the reference data must fit a given literacy level for a given population. this is an important yet complicated issue because there is no clear indications on the simplification requirements for each literacy level. existing simplification guidelines @xcite @xcite are very vague and indicate only general principles without specification of the literacy levels : use short words, use frequent and non-ambiguous words, avoid abbreviations, limit the variability of the vocabulary, make syntactically simple sentences, avoid sentences in passive or negative voice, use personal style, explain difficult concepts, use pictures, etc. this means that each work on simplification should reimplement such principles in order to create clear and exploitable simplification rules. the reference data created for simplification are available for several languages : basque (gonzalez-dios et al., 2018), danish @xcite , english @xcite @xcite , french @xcite , german @xcite , italian @xcite , japanese @xcite , portuguese @xcite , russian @xcite , spanish @xcite . these corpora mainly contain parallel and aligned sentences from existing sources or are crafted manually. some corpora also indicate transformations due to the simplification types @xcite . the content and quality of these corpora are not assessed. ## domain of source documents the domain of source documents has impact on the simplification result and its evaluation. for instance, documents from general and specialized languages do not require the same simplification transformations or resources. the particularity of specialized languages is double : (1) in some domains (medicine, biology, physics...), they convey specific terminology, which requires intense simplification at the lexical and semantic levels. (2) in other domains (legal and administrative), they have specific syntactic structures, which requires their syntactic simplification. this situation may increase distance between the source and simplified texts, which causes lower scores for some evaluation measures. in the following example from the biomedical domain, the two sentences are semantically identical yet they are very distant lexically and syntactically : medication inhibiting the peristalsis are counter-indicated in this situation. in this case, do not take medication for stopping or decreasing the intestinal transit. when simplification with synonyms or equivalent expressions is impossible, it becomes necessary to use more general terms or add explanations, which also leads to an increasing lexical and syntactic distance between source and simplified sentences and documents. ## evaluation measures the purpose of evaluation measures is to assess the quality in simplification research. this is a sensitive issue but still under-studied. we present and discuss several evaluation measures, which can be divided in two types : human (section 5.1) and automatic (section 5.2) evaluation. ## conclusion evaluation of text simplification systems is still an understudied area. it is heavily based upon evaluation approaches exploited in other nlp tasks. yet, the simplification task is different from these other nlp tasks, mainly because its output is subjective. as noticed, this is due to the facts that native simplified-language speaker does not exist, simplification guidelines are vague, there is little consensus on simplification output, and simplification needs depend on the population aimed. this introduces inherent difficulty when creating the reference data and evaluating the simplification, especially for languages other than english, as several of the evaluation measures are language-dependent. in future work, it is necessary to research further the evaluation of simplification : to propose clearer principles for manual evaluation, to define a stronger association between the evaluation criteria and measures, and to propose new and more flexible evaluation metrics. besides, if the simplification systems are designed for specific target population, then this population should be involved in the development of the solution and the evaluation of the results.