Evaluating Factuality in Text Simplification (2022)

Abstract: Automated simplification models aim to make input texts more readable. Such methods have the potential to make complex information accessible to a wider audience, e.g., providing access to recent medical literature which might otherwise be impenetrable for a lay reader. However, such models risk introducing errors into automatically simplified texts, for instance by inserting statements unsupported by the corresponding original text, or by omitting key information. Providing more readable but inaccurate versions of texts may in many cases be worse than providing no such access at all. The problem of factual accuracy (and the lack thereof) has received heightened attention in the context of summarization models, but the factuality of automatically simplified texts has not been investigated. We introduce a taxonomy of errors that we use to analyze both references drawn from standard simplification datasets and state-of-the-art model outputs. We find that errors often appear in both that are not captured by existing evaluation metrics, motivating a need for research into ensuring the factual accuracy of automated simplification models.

## introduction simplification methods aim to make texts more readable without altering their meaning. this may permit information accessibility to a wide range of audiences, e.g., non-native speakers @xcite , children @xcite , as well as individuals with aphasia @xcite and dyslexia @xcite . simplification may also help laypeople digest technical information that would otherwise be impenetrable @xcite . recent work has made substantial progress by designing sequence-to-sequence neural models that "translate" complex sentences into simplified versions @xcite . an important but mostly overlooked aspect of automated simplification-especially in the conditional text generation regime-is whether outputs are faithful to the inputs that they are simplifying. consider, for example, automatically simplifying medical texts @xcite : presenting individuals with readable medical information that contains factual errors is probably worse than providing no such access at all. recent work has acknowledged factuality and faithfulness as key issues to be addressed in other conditional generation tasks like summarization @xcite @xcite , yet so far little research has thoroughly studied the kinds of errors that simplification datasets and system outputs exhibit. this work seeks to close this research gap. we collect human ratings of factuality for these aspects on two widely used simplification corpora: wikilarge @xcite and newsela @xcite . automatically aligned sentences from these two datasets are typically used to train and evaluate supervised simplification systems. we find that errors occur frequently in the validation and test sets of both datasets, although they are more common in newsela (section 6). we then evaluate outputs from several modern simplification models @xcite @xcite , as well as a fine-tuned t5 @xcite model. compared to rnn-based models, transformer-based ones tend to have less severe deletion and substitution errors; however, the pre-trained t5 produced more hallucinations on the more abstractive newsela dataset. we find that existing quality metrics for simplification such as sari @xcite correlate poorly with factuality. although deletion errors correlate with existing semantic similarity measures, they fail to capture insertion and substitution. as an initial step towards automatic factuality assessment in simplification, we train roberta @xcite -based classification models using our annotated data, and use synthetically generated data to supplement training. we demonstrate that this is a challenging task. our code and data can be found at https://github.com/ashologn/evaluatingfactuality-in-text-simplification . ## related work factuality (and the lack thereof) has been identified as critical in recent work in unsupservised simplification @xcite and medical simplification @xcite . @xcite incorporated textual entailment into their simplification task via an auxillary loss. they showed that this improved simplifications with respect to standard metrics and human assessments of output fluency, adequacy, and simplicity, but they did not explicitly evaluate the resultant factuality of outputs, which is our focus. given the paucity of prior work investigating factuality in the context of automated simplification, the most relevant thread of research to the present effort is work on measuring (and sometimes improving) the factuality in outputs from neural summarization systems. @xcite proposed using textual entailment predictions as a means to identify errors in generated summaries. elsewhere, @xcite used weak supervisionheuristic transformations used to intentionally introduce factual errors-to train a model to identify inaccuracies in outputs. @xcite enlisted humans to evaluate hallucinations (content found in a summary but not in its corresponding input) in automatically generated outputs. they report that for models trained on the xsum dataset @xcite , over 70% of summaries contain hallucinations. this corroborates other recent work @xcite , which has also found that rouge is a weak gauge of factuality. @xcite proposed qags, which uses automated question-answering to measure the consistency between reference and generated summaries. elsewhere, @xcite proposed evaluating textual factuality independent of surface realization via semantic role labeling (srl). finally, @xcite introduced the frank (meta-)benchmark for evaluating factuality metrics for summarization. while frank is tailored towards summarizationspecific error categories including discourse, our ontology broadly reflects the goal of simplification (retaining content with simpler language) from the perspective of information precision, recall, and accuracy. ## information errors in simplification above we reviewed various recently proposed frameworks and methods for assessing the factual 0: there is no new information included in the simplified text, or the new information is trivial (e.g., a single insignificant word). ## data and models we annotate data from the simplification datasets themselves (we will call these reference examples), as well as from model-generated text. thus we assess how the distribution of errors in the references compares to that of errors in system outputs and glean insights that might relate model architecture and training choices to the kinds of errors produced. ## factuality of reference examples quantitative analysis simplifications in newsela are more abstractive @xcite , i.e., simplified sentences copy fewer phrases verbatim from inputs. this can be quantified via normalized edit distance @xcite , which yielded a median of 0.46 for newsela examples compared to the 0.38 for wikilarge (after noise filtering described in appendix b). these results suggest that while reducing sentence length and rewording can be beneficial @xcite , too much can negatively impact factuality. ## factuality of system outputs a clear relationship between dataset and system output distributions does not exist for insertion and substitution errors. for dress and editnts, this is due to the fact that the minor differences in insertion errors are dwarfed by the larger number of -1 (gibberish) labels assigned to newsela outputs. interestingly, outputs from the t5 model were rarely labeled as -1 errors, so the difference in insertion errors is more apparent. in the case of substitution, the newsela outputs for dress and t5 models show much higher rates of substitution errors than the wikilarge outputs, despite the opposite being true for the datasets themselves. editnts does not show the same pattern, but again, the high rate of -1 errors subsumes every other trend. one possible reason for this phenomenon could be that the higher abstractiveness of newsela encourages models to rewrite the input sentence to a greater extent and destroy the original meaning in the process. in general the models produce substitution errors more frequently than are found in the dataset, meaning that they are introduced by the models themselves and not merely learned from the data. ## automatic factuality assessment since manual annotation is costly and timeconsuming, as a first step towards large-scale evaluation, we present an initial attempt at automating factuality assessment by training a model on human annotations. to supplement training, we explore methods of generating synthetic data to improve model performance. we framed automatic factuality assessment as a classification task in which a separate classifier is trained for each category (insertion, deletion, and substitution), for each of the levels 0, 1, and 2. we treat the annotations used in our previous analyses as the test set and have additional data annotated to function as the training set for this task. we therefore collected a total of 1004 additional examples annotated across wikilarge, newsela, access outputs on wikilarge, and t5 outputs on newsela and wikilarge. we fine-tuned roberta @xcite with a classification head. synthetic data generation as training and evaluation the model is evaluated using the f1-scores with respect to each class (0,1,2), and when selecting checkpoints during training, the average of the label 1 and 2 f1 scores is used. the deletion model was trained directly on its training data, whereas the insertion and substitution models were initially pretrained on the synthetic datasets. training details are provided in appendix e.3. ## conclusion we have presented an evaluation of the factuality of automated simplification corpora and model outputs, using an error typology with varied degrees of severity. we found that errors appear frequently in both references and generated outputs. in the datasets, deletion errors are quite frequent, with newsela containing more than wikilarge. the system outputs indicate that the models also tend to delete information, which is likely a behavior learned from the training data. model outputs contain more substitution errors than the datasets, so that behavior is probably a model bias rather than something picked up from the data. although we examined the two commonly used sentence-level datasets, factuality errors do extend to other domains and larger units of text. our initial analysis of factuality in medical text simplification @xcite found errors of all three types, an indication that factual simplification is an open problem in such high-stake areas. the details of our analysis are in appendix f. we also found that factuality errors are not well captured by existing metrics used in simplification such as sari @xcite . while semantic similarity metrics correlate with deletion errors, they poorly correlate with insertion or substitution. we further present an initial model for automatic factuality assessment, which we demonstrate is a challenging task.