Document-Level Text Simplification: Dataset, Criteria and Baseline (2021)

Abstract: Text simplification is a valuable technique. However, current research is limited to sentence simplification. In this paper, we define and investigate a new task of document-level text simplification, which aims to simplify a document consisting of multiple sentences. Based on Wikipedia dumps, we first construct a large-scale dataset named D-Wikipedia and perform analysis and human evaluation on it to show that the dataset is reliable. Then, we propose a new automatic evaluation metric called D-SARI that is more suitable for the document-level simplification task. Finally, we select several representative models as baseline models for this task and perform automatic evaluation and human evaluation. We analyze the results and point out the shortcomings of the baseline models.

## introduction text simplification is a valuable technique that deserves to be studied in depth @xcite . one definition of text simplification is to simplify the original text to a more understandable text, while keeping the main meaning of the original text unchanged @xcite . it can provide convenience for non-native speakers @xcite @xcite , non-expert readers @xcite and children @xcite . ## related works sentence simplification aims to rewrite an original sentence into a more straightforward sentence @xcite . the input and output of the model are just sentences instead of articles. based on the english wikipedia and the simple english wikipedia, many researchers have built high-quality datasets such as wikilarge @xcite , wikismall @xcite , and so on @xcite . based on newsela, @xcite established the newsela dataset. the above datasets are widely used in the field of sentence simplification. most of the early simplification models were based on statistical machine translation @xcite . @xcite improved the machine translation model to obtain a new simplification model. @xcite developed a reinforcement learning model and achieved excellent results. @xcite introduced a new memory-augmented neural network to enhance the results. two new approaches were proposed by @xcite to solve the problem of long and complicated simplified outputs. scarton and specia (2018) and nishihara et al. (2019) investigated how to simplify sentences to different difficulty levels. dong et al. (2019) studied the three explicit edit operations in sentence simplification and proposed a new model. štajner et al. (2017), paetzold et al. (2017), and jiang et al. (2020) proposed sentence alignment methods to improve sentence simplification. sun et al. (2020) used the preceding and following sentences to help simplify a specifically given sentence. there are very few works related to documentlevel text simplification. alva-manchego et al. (2019) focused on cross-sentence transformations in text simplification and analyzed them, concluding that document-level simplification cannot be achieved by merely selecting parts of the content then simplifying individual sentences. subsequently, zhong et al. (2020) used discourse-level factors to predict whether a sentence should be deleted and achieved good results. although some previous works focus more or less on document-level information, the task of document-level text simplification is still not operation sentence joining sentence splitting sentence deletion sentence reordering sentence addition anaphora resolution percentage(%) 96 84 91 92 92 92 ## problem formulation the document-level text simplification task can be defined as follows. given an original complex article c, the article consists of n sentences, denoted as @xmath0, s 2 , ...s n }. document-level simplification aims to simplify c into m sentences, which form the simplified article f , denoted as @xmath1, t 2 , ...t m }, and m may not be equal to n. f retains the primary meaning of c and is more straightforward than c, making it easier for people to understand. the operations for sentence-level simplification include word reservation and deletion, synonym replacement, etc. @xcite based on the work of @xcite , we define six types of document-level simplification operations, namely, sentence joining, sentence splitting, sentence deletion, sentence reordering, sentence addition, and anaphora resolution. see appendix a for the specific definition and example of each operation. in our definition, document-level simplification should allow the loss of information but should not allow the loss of important information. @xcite pointed out that sentence deletion is a prevalent phenomenon in document simplification. we believe that information that has little relevance to the primary meaning should be removed to improve readability. ## the d-sari metric currently, the most commonly used automatic evaluation metric for sentence-level simplification is the sari metric @xcite . the correlation of sari with human judgments of the simplicity indicator proved to be high in sentence-level simplification @xcite . however, this metric has shortcomings when used directly to evaluate document-level simplification. for better understanding, in this section, we conduct a qualitative analysis and give the following example: original article: marengo is a town in and the county seat of iowa county , iowa , united states . simplified article 2: marengo is a city in iowa , the us . it has served as the county seat since august 1845 , even though it was not incorporated . the population was 2,528 in the 2010 census , a decline from 2,535 in 2010 . simplified article 3: marengo is a town in iowa . marengo is a town in the us . in the us . the population was 2,528 . the population in the 2010 census . simplified article 4: marengo is a town in iowa , united states . in 2010 , the population was 2,528 . reference article: marengo is a city in iowa in the us . the population was 2,528 in 2010 . the sari values of the four simplified articles are shown in the left half of the simplified article 1 generates "us", a word that does not appear in the original article, which raises the overall sari value. intuitively, however, it makes no sense to generate several simplified words if the simplified article is too short to convey the main meaning of the original article. therefore, we believe that a penalty factor lp 1 should be added to the f add score. if the generated simplified article is shorter than the reference article, the f add score will be penalized. the simplified article 2 does a good job of simplifying the first sentence of the original article but retains much useless information and difficult words. paradoxically, its p del score is the highest among the four simplified essays. a more common scenario is that the original article is much longer than the simplified article, then according to the formula of p del , removing fewer words will have a limited effect on p del . therefore, we believe that a penalty factor lp 2 should be added to the p del score, penalizing the p del score if the generated simplified article is longer than the referenced article. the simplified article 3 finds the important information in the original article and does a good job of simplifying it. nevertheless, it performs duplicate generation, which leads to a severe decrease in readability and should not get such a high f keep value. according to the formula of f keep , if the duplicate n-grams also appear in the reference, then the f keep value will not decrease. therefore, we believe that the lp 2 should also be added to the f keep score. besides, we add a sentence-level penalty factor slp to penalize the f keep score if the number of generated sentences is far from the number of sentences in the reference article. in summary, based on the sari metric @xcite , we propose the d-sari metric for the document-level simplification task. we retain the idea of calculating the scores of add, keep and delete separately in sari, which proved to be effective in sentence simplification. the d-sari metric is shown as below: i, o, and r represent the number of words (including punctuation) in the input article, the output article, and the reference article, respectively. o s and r s represent the number of sentences in the output article and the reference article, respectively. due to the limitation of space, please refer to @xcite for the calculation of f add , f keep and p del . we also calculate the d-sari values for each of the simplified articles in the given example, as shown in the right half of as we analyzed from the given example, it is reasonable to penalize the three components in sari. in the d-sari metric, the penalty is based on length. the motivation comes from bleu @xcite . a candidate should be neither too long nor too short, and an evaluation metric should enforce this. the difference between the length of a simplified sentence and the original sentence in sentence-level text simplification is not very large, while the opposite is true for documentlevel text simplification. an original article may be long, while a simplified article may contain only one sentence. it is simple enough, but not a good simplification of the original article. it is a reasonable proposition that the length of the simplified article should be close to the length of the reference article. we also conduct an empirical analysis of the d-sari metric. in section 7.3, we use spearman's rank correlation coefficient @xcite to show that the d-sari metric has the strongest correlation among several metrics with human ratings. ## baseline models we selected four representative models as the baselines for the document-level simplification task, which are: all the models were tested on our delineated test sets. we used the fairseq toolkit and performed replicate experiments. see appendix b for detailed parameters. ## conclusion in this paper, we are committed to promoting research on document-level text simplification. we established a large-scale high-quality dataset named d-wikipedia and proposed a new automatic evaluation metric called d-sari. we also selected several representative models as baselines for this task. the results demonstrate that the dataset is of high quality and the metric is reliable.