On the Helpfulness of Document Context to Sentence Simplification (2020)

Abstract: Most of the research on text simplification is limited to sentence level nowadays. In this paper, we are the first to investigate the helpfulness of document context on sentence simplification and apply it to the sequence-to-sequence model. We firstly construct a sentence simplification dataset in which the contexts for the original sentence are provided by Wikipedia corpus. The new dataset contains approximately 116K sentence pairs with context. We then propose a new model that makes full use of the context information. Our model uses neural networks to learn the different effects of the preceding sentences and the following sentences on the current sentence and applies them to the improved transformer model. Evaluated on the newly constructed dataset, our model achieves 36.52 on SARI value, which outperforms the best performing model in the baselines by 2.46 (7.22%), indicating that context indeed helps improve sentence simplification. In the ablation experiment, we show that using either the preceding sentences or the following sentences as context can significantly improve simplification.

## introduction text simplification is a hot issue in the field of natural language generation (nlg). it is also one of the critical needs of society @xcite . text simplification aims to adapt a complex text into a more readable version with the same meaning @xcite , which will benefit young children @xcite and non-native english speakers @xcite . it includes many ways to deal with the input text, such as deletion, reordering, paraphrase, and sentence separation @xcite . besides, text simplification is closely related to many natural language processing (nlp) tasks, such as machine translation @xcite , paraphrase generation @xcite @xcite ) and text summarization @xcite @xcite . in recent years, many researchers have proposed various models to improve the performance of text simplification. however, few people have explored the impact of document context (i.e., the preceding and following sentences of the original sentence to be simplified in a document) on text simplification, let alone establish a large-scale dataset containing context. many examples like the one in in this paper, we are committed to investigating the influence of document context on text simplification and proposing a neural model to improve simplification using context. using the wikipedia datasets @xcite , we first construct a dataset in which the document context ## related works text simplification has developed rapidly in the past decade. @xcite proposed the pbmt-r model while @xcite put forward the hybrid model. both models were based on statistic machine learning. using a small number of manual simplifications and a large number of paraphrases, @xcite adapted the method of the statistical machine translation. @xcite were the first to apply the sequence-to-sequence model to automatic text simplification, using the framework of neural machine translation and making specific improvements. @xcite proposed the dress model which rewards the simple, fluent output sentences with proper meaning. @xcite used memory-augmented neural networks to adapt the existing architecture and @xcite used multi-task learning to improve the entailment and paraphrase capabilities. recently, @xcite used two techniques to solve the problem that the model tends to copy words directly, resulting in a long and complicated output sentence. @xcite proposed a method to simplify the original sentences to different level sentences. different from most sequenceto-sequence models, @xcite proposed a neural programmer-interpreter approach to predict explicit edit operations directly. @xcite proposed the neural crf model to get better sentence alignment. however, up to now, most of the research on text simplification is limited to sentence level, ignoring the influence of document context on sentence simplification. @xcite are the first to empirically demonstrate that discourse relations are closely related to the perceived quality of the text. so far, @xcite are the first and the only ones to focus on discourse level factors of text simplification. their results have shown that using discourse level factors is useful for predicting sentence deletion. nevertheless, different from our research focusing on sentence simplification, this work mainly analyzes and predicts the sentence deletion in document simplification. in addition to deletion operation, sentence simplification also includes reservation, separation, synonym replacement, and other operations @xcite . whether the preceding sentences and the following sentences have different effects on the original sentence has not been taken into account. even in a similar field of machine translation, most of the research focuses on the preceding sentences. based on the transformer model, @xcite used a new encoder to represent the context and proposed a new model. @xcite proposed a hierarchical attention model that captures the context in a structured and dynamic way. both of the models that have received widespread attention only focuses on the preceding sentences. in addition to the preceding sentences, the following sentences also contain the information of keeping, paraphrasing, and deleting the words in the original sentence in text simplification. to the best of our knowledge, we are the first to study the effects of the preceding sentences and the following sentences and apply them to a sequence-to-sequence model in the field of text simplification. ## our model the suc model consists of four parts: the transformer model, context information module, pointergenerator network, and coverage mechanism. among them, the context information module is an original module proposed by us to get the context representation and apply it to the transformer model. ## conclusion in this paper, we propose a new model named suc which makes full use of the context information for text simplification. from the results of the automatic evaluation and human evaluation, it can be seen that our model outperforms the baselines, which proves that context information is helpful to sentence simplification. in the ablation experiment, we show that sentence pairs with the preceding sentences or following sentences can both significantly improve simplification.