Improving Text Simplification with Factuality Error Detection (2022)

Abstract: In the past few years, the field of text simplification has been dominated by supervised learning approaches thanks to the appearance of large parallel datasets such as Wikilarge and Newsela. However, these datasets suffer from sentence pairs with factuality errors which compromise the models’ performance. So, we proposed a model-independent factuality error detection mechanism, considering bad simplification and bad alignment, to refine the Wikilarge dataset through reducing the weight of these samples during training. We demonstrated that this approach improved the performance of the state-of-the-art text simplification model TST5 by an FKGL reduction of 0.33 and 0.29 on the TurkCorpus and ASSET testing datasets respectively. Our study illustrates the impact of erroneous samples in TS datasets and highlights the need for automatic methods to improve their quality.

## introduction text simplification (ts) is a natural language processing (nlp) task that considers the reduction of text's complexity towards increasing its readability and understandability while retaining its original meaning. ts can increase the accessibility of information to a wider audience, including youngsters, those with little literacy, people who are not native speakers, the elderly, and people with disabilities @xcite @xcite . additionally, numerous studies have also demonstrated that ts can support other nlp tasks as a preprocessing step @xcite . the current ts domain @xcite @xcite is dominated by fine tuning large sequenceto-sequence language models on existing parallel datasets, the main ones being wikilarge @xcite and newsela @xcite . however, several studies have revealed that these training datasets suffer from factuality errors. @xcite factuality errors occur when the samples provided do not accurately or properly represent the task. in the ts context, two main sources of factuality errors are bad alignment, i.e., loss of content preservation, and bad simplification, i.e., the target sentence is not simpler than the source @xcite . the existence of parallel training samples with factuality errors can impact significantly the performance of the ts models. in this study, we investigated methods to detect parallel samples with factuality errors in the wikilarge dataset. we explored the impact of decreasing the loss weight of the detected samples during training in the ts task performance. we re-trained the state-of-the-art (sota) ts model tst5 (sheang and saggion, 2021) using the modified wikilarge dataset and observed a significant performance improvement when tested on the turkcorpus and as-set datasets. ## factuality error detection in this study, we implemented a rule-based algorithm to detect factuality errors in the wikilarge dataset. for the detected samples, the loss of the ts model was subsequently scaled down during training to reduce their impact on the model's learning performance. to detect bad simplification, we utilized the flesch-kincaid grade level (fkgl) metric @xcite , which was designed for evaluating text readability and has also been used as an evaluation metric in multiple previous works @xcite @xcite . fkgl was originally calculated at the paragraph level based on the average length of the sentence ( n words nsentences ) and the number of syllables ( n syllables n words ). to apply fkgl to the sentence level, instead of calculating the average length, the length of the sentence itself was used (eq. 1). with the assumption that readability reflected simplicity, any sentence pairs for which the source sentence y had higher fkgl score than its target counterpart x were marked as bad simpli-f @xmath0.39n words + 11.8 n syllables n words -15.59 (1) bad alignment was recognized based on named entity recognition. named entity refers to a phrase that clearly identifies one item from a set of other items that have similar attribute. we identified locations, name, time, and organization in both target and source sentences. this was performed through a pretrained classifier provided by the nltk library 1 @xcite . here, we assumed that simplification might reduce but should not add entities. according to this, we calculated the cosine similarity between all the entities in source sentences and target sentences. because each named entity may contain different number of words, we used a contextual embedding model based on transformers to create embeddings for each named entity rather than a word level encoder such as word2vec @xcite . bad alignment was recognized if there existed an entity e t in the target sentence that did not have a corresponding entity e s in the source sentence with cosine similarity higher than a predefined threshold t . for the pairs marked with factuality error, their corresponding weights were scaled down during training as shown in eq. 2 and 3 for the bad simplification and bad alignment respectively. the effect of the factuality error samples suppression was explored by experimenting with different scaling parameters α 1 and α 2 . the resulting weights of bad simplification and bad alignment were multiplied together, and the outcome was then normalized by the total weight. thus, sentence pairs that were found to be both unaligned and unsimplified were further suppressed. ## discussion our factuality detection rate was aligned with the work of @xcite 's experiment on the bad simplification case (23% and 33% respectively), however, it identified a higher number of bad alignment samples (31% in comparison to 17%). this could be due to sensitivity differences between the two approaches. our ts results ( from our results illustrate that the existence of factuality errors in the training datasets used for ts, can induce a significant impact in the performance of the ts models. this indicates a general need for new reliable datasets exploration. better error detection methods, including more thorough tuning, and further validation is needed with other ts models and other parallel datasets such as newsela, which is part of our future work. the trade-off between error detection sensitivity and dataset size reduction is crucial and needs further investigation. ## conclusion in this paper, we designed a model-independent factuality error detection mechanism to support ts model training. we demonstrated that our mechanism could significantly improve the performance of the sota ts model (tst5) based on recognized ts metrics. our study raises the need for high quality parallel datasets, as well as automated factuality error detecton methods to improve the performance of ts models.