SW i PE : A Dataset for Document-Level Simplification of W ikipedia Pages (2023)

Abstract: Text simplification research has mostly focused on sentence-level simplification, even though many desirable edits - such as adding relevant background information or reordering content - may require document-level context. Prior work has also predominantly framed simplification as a single-step, input-to-output task, only implicitly modeling the fine-grained, span-level edits that elucidate the simplification process. To address both gaps, we introduce the SWiPE dataset, which reconstructs the document-level editing process from English Wikipedia (EW) articles to paired Simple Wikipedia (SEW) articles. In contrast to prior work, SWiPE leverages the entire revision history when pairing pages in order to better identify simplification edits. We work with Wikipedia editors to annotate 5,000 EW-SEW document pairs, labeling more than 40,000 edits with proposed 19 categories. To scale our efforts, we propose several models to automatically label edits, achieving an F-1 score of up to 70.9, indicating that this is a tractable but challenging NLU task. Finally, we categorize the edits produced by several simplification models and find that SWiPE-trained models generate more complex edits while reducing unwanted edits.

## introduction text simplification (ts) aims to make complex documents accessible to larger audiences by lowering the barrier of reading for children, non-native speakers, and novice readers in technical domains. ts has primarily been approached in a sentencelevel sequence-to-sequence (seq2seq) manner, following the methodology of mature nlg tasks such as machine translation. prior work framed at the sentence level has focused on simplification edits that occur within sentence units, such as lexical replacements @xcite and sentence splitting @xcite . yet, many simplification operations, such as background elaborations @xcite 2020) or content reordering @xcite require document-level context. a major roadblock to advances in documentlevel simplification has been the lack of large-scale and high-quality datasets. the two most popular sources of data for the english language are either the news-based newsela which is not available publicly or the combination of english wikipedia (ew) and simple english wikipedia (sew) foot_0 , which is large-scale but requires non-trivial processing to align wikipedia articles with their simplified versions @xcite . the alignment task has predominantly been framed as finding pairs of semantically similar sentences within the latest revisions of ew and sew pages. our first contribution is to adapt the wikipedia content alignment task to document-level granularity. we explore the entire revision history of wikipedia pages and match individual revisions of sew pages with best-aligned ew revisions, rather than rely on the most recent revisions which might yield factually misaligned pairs due to outdated information. by applying our alignment method to the entire revision history of sew -and processing two orders of magnitude more content -we create the swipe dataset, a high-quality and large-scale document-level simplification dataset. swipe consists of 145,161 document pairs, which we processed into an alignment sequence composed of three operations: unchanged text, insertion, and deletion. figure 1 provides an illustrative alignment sequence of a swipe sample. our second contribution is a comprehensive analysis of edits that occur in swipe. we propose a 19-category edit taxonomy based on prior work and expanded for document-level edits. the categories are organized into four coarse-grained classes representing simplification objectives: lexical, syntactic, semantic, and discourse-level edits. we collaborate with active sew editors to annotate 5,000+ alignment sequences of swipe. the collected annotations of around 40,000 edits reveal that all four edit classes are prevalent in swipe (each occurs in at least 40% of annotated documents). document-level context is required for at least 43% of edits, and diverse edits often co-occur within documents, as sew editors combine editing strategies when producing sew pages. our third contribution is to propose models that can automatically identify edit categories and models that generate document-level simplified text. for the task of edit identification, our best model achieves a categorization f-1 score of 70.6, leaving room for future improvement. when analyzing simplification models based on the edits they produce, we find that swipe-trained models can produce more complex edits than prior work while generating fewer undesirable edits that potentially introduce factually incorrect content. we release the swipe data, the models, and experimental code publicly foot_1 . ## edit-level annotation in upcoming sections, we use the term document to refer to a particular page version. given two matched documents, they can be represented as a single alignment sequence using a string-alignment algorithm such as levenshtein @xcite ). an alignment sequence consists of a series of three operations: unchanged text, inserted text, and removed text, as illustrated in figure 1 . to understand the types of edits that occur in swipe, we collaborated with simple wikipedia editors to annotate a subset of the dataset. ## automatic edit identification we investigate whether edits can be identified automatically, which could automate annotation of the entire swipe dataset -estimated to require 2,900 hours of manual work -or facilitate analysis of generative simplification models. ## text simplification baselines we leverage swipe and its cleaned alternative to fine-tune two bart-large models: bart-swipe and bart-swipe-c and compare them to recent simplification systems. we experiment with two existing simplification systems: access @xcite , a state-of-the-art controllable sentence-level simplification model trained on wikilarge (zhang and lapata, 2017), and keep it simple (kis) @xcite , an unsupervised paragraph-level model optimized to produce lexical and syntactic edits. we also train bart-wikilarge a bart-large model trained on wikilarge to understand the effect of the dataset under a fixed pretrained model. finally, we include a prompt-based gpt3-davinci-003 using a task prompt that did not specify edit categories to apply. model details and example outputs are in appendix e. we run experiments on the validation set of swipe. for each model, we report the n-grambased sari score @xcite , the flesch-kincaid grade @xcite , and the distribution of edit categories identified by bic (merged into 10 groups). results are in figure 5 . swipe-trained models achieve the highest performance in terms of sari, confirming a similarity to reference simplifications, and the lowest estimated grade-level scores, validating the model's ability to improve readability. the access sentence-level model performs moderately well on the sari metric, but worst on the grade-level estimation, and makes fewer complex edits such as reorderings or elaborations, confirming that sentence-level models focus on simpler edits, such as lexical and syntactic edits. all other models attempt a large proportion of all edits, including a large number of edits tagged as extraneous information (i.e., information not in the original document). when simplified by human editors, extraneous information often comes from other documents or background knowledge and is not likely harmful. on the contrary, recent nlg work has shown that model-generated extraneous information is often hallucinated, can be factually incorrect, and is undesirable. example model outputs in appendix e.2 show example problematic outputs from the kis and bart-wikilarge models which include factual errors, for example confusing centimeters and micrometers, or the length and width of a hair. the kis, bart-wikilarge, bart-swipe, and gpt-3 models all produce a larger proportion of extraneous information edits than elaborations, confirming prior work showing that problematic hallucinations can occur for the simplification task as well @xcite . bart-swipe-c is able to produce elaborations while having a reduced rate of extraneous information, giving preliminary evidence that the edit-based dataset cleaning process we adopt can mitigate -but not solvethe generation of extraneous information. similar to recent work in summarization showing that zero-shot gpt3 can tie or surpass supervised models @xcite , we observe that gpt3 can generate a wide range of simplification edits and does not mirror priors of the dataset -such as producing more sentence splits than fusions -indicating it has potential for use as a general-purpose simplification model. similar to prior work, gpt3-based candidates score poorly on reference-based metrics. we note that the analysis is preliminary, and future work should assess the efficacy, factual consistency, and simplicity of generated edits with target readers as done in prior work @xcite to gain a thorough understanding of model performance. ## conclusion we introduce swipe, a large-scale document-level simplification dataset based on wikipedia. swipe is created by collecting pairs of pages from the english and simple english wikipedia and matching their revision histories to build document pairs that align in terms of content presented. we collaborated with simple wikipedia editors to annotate 5,000 document pairs in swipe, finding that many complex edits that require document-level context such as elaborations frequently occur in the dataset. we experimented with the automatic identification of edits, finding that even though the task is challenging, some models are able to achieve performance above 0.7 f-1 at edit categorization, making them viable to analyze model-generated simplifications. an analysis of generative simplification models reveals that sentence-level models are limited in the types of edits they propose and that document-scoped models are likely to produce hallucinated content. finally, a model fine-tuned on a cleaned version of swipe produces less extraneous content while continuing to generate complex edits, pointing towards simplification models that can generate complex yet factually consistent edits. difficulty in retracing original editing. by matching revisions of wikipedia pages that are factually aligned, and working with sew editors to annotate the edits, we attempted to match the process used to create the resource. it is however not possible to recruit all 5,000+ sew editors and for some page pairs the annotations are another editor's best attempt to reconstruct the intended edits by the original editor.