Linguistic Corpus Annotation for Automatic Text Simplification Evaluation (2022)

Abstract: Evaluating automatic text simplification (ATS) systems is a difficult task that is either performed by automatic metrics or user-based evaluations. However, from a linguistic point-of-view, it is not always clear on what bases these evaluations operate. In this paper, we propose annotations of the ASSET corpus that can be used to shed more light on ATS evaluation. In addition to contributing with this resource, we show how it can be used to analyze SARI’s behavior and to re-evaluate existing ATS systems. We present our insights as a step to improve ATS evaluation protocols in the future.

## introduction automatic text simplification (ats) is a task that consists in automatically adapting a text to make it more accessible for readers. it is the focus of more and more attention from researchers, with a growing number of publications every year and some surveys published recently @xcite @xcite . as it is the case in many domains, ats tasks are currently mostly explored using deep learning @xcite @xcite , although rule-based systems are still being explored as well @xcite @xcite . one serious hurdle that the field is currently facing is how to evaluate those systems @xcite . there are two common approaches: human judgment and automatic evaluation metrics. human judgment is collected by asking people to rate the output of a system on a likert scale using three criteria: grammaticality, meaning preservation and simplicity. this method is certainly encompassing, but requires lots of time and effort. in contrast, automatic evaluation metrics represent a reproducible and quick way to measure the performance of ats systems. the most common metrics currently used are: bleu @xcite , a metric created to evaluate the performance of machine translation systems; sari @xcite , a metric specifically designed for ats; and the flesch-kincaid readability formula @xcite . although these metrics have serious shortcomings @xcite @xcite , their use is widespread in the field due to their ease of use. bleu and sari require reference data (human made simplifications) and are known to be more reliable when more references are available. in other words, their behavior is a function of the dataset used as the reference. since different audiences have different needs when it comes to simplification @xcite , making sure that a dataset will reflect those specific needs is important. two other automatic metrics have been used in text simplification research but are not broadly used: bertscore (zhang et al., 2020) -a metric designed for language generation evaluation -and samsa @xcite ) -specifically designed for ats but not easy to use as it requires semantic annotation. in english, mostly three corpora are used for evaluation: turkcorpus @xcite , asset @xcite and newsela @xcite . the first two share the same source sentences, with different crowdsourced simplifications. for other languages, with less attention from the community, systems are evaluated against ad-hoc corpora @xcite @xcite . little is known about the way automatic metrics are related to the different types of simplification operations. recent work @xcite investigated the relation of those metrics to basic computational operations such as add, delete or insert, but no work focuses on the linguistic simplifications listed in existing typologies @xcite @xcite . in this paper, we aim to investigate the following hypothesis: evaluation metrics react differently depending on the type of linguistic operations applied to produce the reference(s) and the output of a system. to verify this hypothesis, we have annotated the asset corpus with a broad set of linguistic operations. the annotated resource is the first contribution of the paper and we hope it can spur research on new automated evaluation metrics. a second contribution is the annotation process itself (see section 3), as we provide an annotation guide that has been validated with 9 annotators and could be reused to annotate other corpora in a similar way. in section 4, we study sari's behavior according to the operations we annotated and we calculate the sari and bleu scores of published systems on various subsets of asset (section 4.2). this new angle of observations on automatic ats evaluation, in relation with our hypothesis, is the third contribution of this paper. ## literature review historically, before statistical and then neural methods became the main focus of ats research, typologies were a requirement for building ats systems, as they were the conceptual basis of rule-based methods. as documented by @xcite , authors of ats systems were mostly concerned with syntax @xcite @xcite and produced typologies of the syntactic operations they aimed at performing on sentence structures. in more recent works, we could identify two main axes that were used to build typologies of simplification operations: one based on linguistic descriptions and the other one based on string edits. ## annotation this section presents the dataset (section 3.1) and the typology of the operations we use for annotation (section 3.2). we then make a statement on what we chose not to annotate but could have been expected given the data and the current practices in ats evaluation (section 3.3). we describe the annotation process (section 3.5) and we analyze the annotated set that served for the inter-annotator agreement (section 3.5), then we describe the resulting resource (section 3.6. we finally compare our result with other similar works (section 3.7. ## experiments we propose two experiments that show how asset ann can help studying evaluation protocols. first, we analyze the behavior of sari, with respect to asset ann . second, we re-evaluate existing ats systems from the literature, to have a view on how they handle linguistic operations. ## discussion we performed the annotation of asset with the goal of shedding light on automatic ats evaluation using linguistic information. this enabled us to make several insightful contributions. regarding the analysis of current automatic evaluation, we have shown that sari's subcomponents can inform on the linguistic operations present in references that appear in a system output, whereas this is lost when averaging the three subcomponents into one single score. this insight is an argument in favor of reporting the sub-components' scores while performing evaluation, as some works started to do @xcite . the analyses we performed also make the case for further exploration in bridging linguistic operations and computational operations. we believe that asset ann and our experiments can serve to pave the way for future ats evaluation practices that would go further in that direction. we selected subsets of references to use for the evaluation according to predetermined sets of operations. creating such subsets allows to focus on relevant operations for a given target audience's specific needs in the evaluation framework. asset ann is a first step in that direction. as discussed in section 3.3, we did not evaluate simplicity. though it can be time consuming, we recommend to assess simplicity based on the precise definition of the task. during the resource 2% of the sentences identified as erroneous simplification. creation, annotators told us they thought that some simplifications were, in fact, obviously not simpler. working with members of the target audience of a given task in order to identify references that should not be taken into account would allow for a more accurate evaluation protocol. finally, as a future work we would like to leverage the annotated dataset to automatically reproduce the annotation. ## conclusion in this work, we produced an annotated version of the asset test set. the resource was annotated by 9 annotators, following a typology that we proposed based on existing similar typologies. based on the annotation, we could clean the test set by removing 227 sentence pairs with manifest errors and produce a curated version of asset. we reevaluated systems for which the outputs were available and provided the community with updated performance results using sari and bleu. the different resources described in the paper and the code used for our analyses are available online. in addition, we performed an extensive analysis of sari's sub-components and found links between those and linguistic operations. we see these results as a promising direction to improve automatic ats evaluation by exploring the relationship between computational and linguistic operations.