How May I Help You? Using Neural Text Simplification to Improve Downstream NLP Tasks (2021)

Abstract: The general goal of text simplification (TS) is to reduce text complexity for human consumption. In this paper, we investigate another potential use of neural TS: assisting machines performing natural language processing (NLP) tasks. We evaluate the use of neural TS in two ways: simplifying input texts at prediction time and augmenting data to provide machines with additional information during training. We demonstrate that the latter scenario provides positive effects on machine performance on two separate datasets. In particular, the latter use of TS improves the performances of LSTM (1.82–1.98%) and SpanBERT (0.7–1.3%) extractors on TACRED, a complex, large-scale, real-world relation extraction task. Further, the same setting yields improvements of up to 0.65% matched and 0.62% mismatched accuracies for a BERT text classifier on MNLI, a practical natural language inference dataset.

## introduction the goal of text simplification (ts) is to reduce text complexity (while preserving meaning) such that the corresponding text becomes more accessible to human readers. previous works explored how ts can assist children @xcite , nonnative speakers @xcite , and people with disabilities @xcite . while this can be achieved in a variety of approaches @xcite , most ts research has focused on two major approaches: rule-based and neural sequence-to-sequence (seq2seq). since 2017, there is a significant increase of neural seq2seq ts methods @xcite @xcite @xcite . in this paper, we analyze another potential use of the latter ts direction: assisting machines performing natural language processing (nlp) tasks. to this end, we investigate two possible directions: (a) using ts to simplify input texts at prediction time, and (b) using ts to augment training data for the respective nlp tasks. we empirically analyze these two directions using two neural ts systems @xcite , and two nlp tasks: relation extraction using the tacred dataset @xcite , and multi-genre natural language inference (mnli) @xcite . further, within these two tasks, we explore three methods: two relation extraction approaches, one based on lstms @xcite and another based on transformer networks, spanbert @xcite , and one method for mnli also based on transformer networks, bert @xcite . our analysis shows that simplifying texts at prediction times does not improve results, but using ts to augment training data consistently helps in all configurations. in particular, after augmented data is added, all approaches outperform their respective configurations without augmented data on both tacred (0.7-1.98% in f1) and mnli (0.50-0.65% in accuracies) tasks. the reproducibility checklist and the software are available at this link: https://github.com/vanh17/textsim . ## related work recent work have effectively proven the practical application of neural networks and neural deep learning approaches to solving machine learning problems @xcite @xcite . with respect to input simplification, several works have utilized ts as a pre-processing step for downstream nlp tasks such as information extraction @xcite @xcite , parsing @xcite , semantic role labeling @xcite , and machine translation @xcite . however, most of them focus on the use of rule-based ts methods. in contrast, we investigate the potential use of domain-agnostic neural ts systems in simplifying inputs for downstream tasks. we show that, despite the complexity of the tasks investigated and the domain agnosticity of the ts approaches, ts improves both tasks when used for training data augmentation, but not when used to simplify evaluation texts. on data augmentation for natural language processing downstream tasks, previous work show significant benefits of introducing noisy data on the machine performance @xcite . previous efforts used ts approaches, e.g. lexical substitution, to augment training data for downstream tasks such as text classification @xcite . however, these methods focused on replacing words with thesaurus-based synonyms, and did not emphasize other important lexical and syntactic simplification. here, we use two out-of-the-box neural ts systems that apply both lexical and syntactic sentence simplification for data augmentation, and show that our data augmentation consistently leads to better performances. note that we do not use rulebased ts systems because they have been proven to perform worse than their neural counterparts @xcite . further, rule-based ts systems are harder to build in a domain-independent way due to the many linguistic/syntactic variations across domains. ## approach we investigate the impact of text simplification on downstream nlp tasks in two ways: (a) simplifying input texts at prediction time, and (b) augmenting training data for the respective nlp tasks. we discuss the settings of these experiments next. ## results and discussion all in all, our experiments suggest that our data augmentation approach using ts is fairly general. it does not depend on the actual ts method used, and it improves three different methods from two different nlp tasks. further, our results indicate that our augmentation approach is more beneficial for tasks with lower resources (e.g., tacred), but its impact decreases as more training data is available (e.g., mnli). ## conclusion we investigated the effects of neural ts systems on downstream nlp tasks using two strategies: (a) simplifying input texts at prediction time, and (b) augmenting data to provide machines with additional information during training. our experiments indicate that the latter strategy consistently helps multiple nlp tasks, regardless of the underlying method used to address the task, or the neural approach used for ts.