Reproducing a Manual Evaluation of the Simplicity of Text Simplification System Outputs (2022)

Abstract: In this paper we describe our reproduction study of the human evaluation of text simplic- ity reported by Nisioi et al. (2017). The work was carried out as part of the ReproGen Shared Task 2022 on Reproducibility of Evaluations in NLG. Our aim was to repeat the evaluation of simplicity for nine automatic text simplification systems with a different set of evaluators. We describe our experimental design together with the known aspects of the original experimental design and present the results from both studies. Pearson correlation between the original and reproduction scores is moderate to high (0.776). Inter-annotator agreement in the reproduction study is lower (0.40) than in the original study (0.66). We discuss challenges arising from the unavailability of certain aspects of the origi- nal set-up, and make several suggestions as to how reproduction of similar evaluations can be made easier in future.

## introduction against a background of growing interest in approaches to reproducibility assessment in general, and specific reproduction studies in particular, this paper reports a reproduction study of a human evaluation of text simplicity carried out as part of the reprogen shared task 2022 on reproducibility of evaluations in nlg. we participated with a contribution in track a, carrying out a reproduction study of the human evaluation of sentence simplicity reported by @xcite , one of the five papers offered in the track. in the original paper, nine automatic text simplification systems were evaluated by human annotators for four different criteria: correctness and number of changes, meaning preservation, grammaticality, and simplicity. in this paper, we concentrate only on simplicity. we first summarise the original study and describe the details of our reproduction study (section 2. we then present the results from both studies (section 3) in terms of the system-level simplicity scores of the nine systems, and the inter-annotator agreement estimated as quadratic cohen's kappa. we also report pearson's correlation coefficient between the original and the reproduction system scores. we finish (section 4) with a discussion of the differences between the two studies and the impact of missing information about the original set-up, and suggest how to make future human evaluations easier to repeat. ## experimental design in original and reproduction study a commonly cited motivation for automatic text simplification (ats) systems is that texts containing uncommon words or long and complicated sentences can be difficult to read and understand by people as well as difficult to analyze by machines. ats is the process of transforming one text into another text which ideally has the same meaning, but is easier to read and understand by a wider audience and also easier to process with nlp tools. ats systems can be rule-based or corpus-based, namely trained on parallel corpora consisting of original texts and their simplified versions. for human evaluation of ats systems, the usual quality criteria are meaning preservation (the degree to which the meaning of the original text is retained in the simplified output; analogous to adequacy in mt), grammaticality (whether the grammar of the generated output is good), and simplicity (how difficult/simple the generated output is). this paper focuses on simplicity evaluation in the form of comparing the automatically simplified output with the original text that was the input: the original sentence is presented together with its automatically simplified version, and the evaluators are asked whether the simplified version is simpler, equally simple/difficult, or more difficult than the original. ## conclusions this paper reported the results of a reproduction study of a human evaluation of text simplicity. the obtained scores confirm some of the findings of the original paper, however findings relating to whether or not re-ranking with bleu or sari helped were not aligned in the two studies, in some cases showing opposite effects. pearson correlation between the studies was moderate to high at 0.766. the inter-annotator agreement was lower in the reproduction study, 0.40 vs. 0.66, but we do not know whether it was computed in a comparable way. a deeper analysis of these differences is unfortunately not possible because we lack too many details for the original set-up. also, sentence-level human annotations which would be helpful are not published (while the models and the automatic evaluation scripts are). it appears to be the case that there is a tendency for comprehensive details about the human evaluation process to be reported only in papers dealing with human evaluation itself, although even in these, the provided information is not often fully complete. in papers where human evaluation is not the focus but only a method to assess the system(s), usually only very shallow information is provided, if any. moreover, it is often the case that the authors themselves perform evaluations, sometimes with no overlap, which makes it impossible to report iaa. fully reporting such details is disincentivised as doing so may lead to more negative reviews. human evaluation is time and resource-expensive and it is usually not possible to (i) evaluate large amounts of text, (ii) involve a large number of evaluators, or (iii) evaluate large portions of text by several evaluators for iaa, because all these factors increase cost further. as in previous work @xcite )), we conclude that reporting more details about human evaluation experiments would be of benefit scientifically. details of human evaluations should be provided in each paper, even if the conditions were not perfect (and they often are not). it is more scientifically rigorous as well as more useful to provide full details than not providing information for fear of negative review.