ASSET : A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations (2020)

Abstract: In order to simplify a sentence, human editors perform multiple rewriting transformations: they split it into several shorter sentences, paraphrase words (i.e. replacing complex words or phrases by simpler synonyms), reorder components, and/or delete information deemed unnecessary. Despite these varied range of possible text alterations, current models for automatic sentence simplification are evaluated using datasets that are focused on a single transformation, such as lexical paraphrasing or splitting. This makes it impossible to understand the ability of simplification models in more realistic settings. To alleviate this limitation, this paper introduces ASSET, a new dataset for assessing sentence simplification in English. ASSET is a crowdsourced multi-reference corpus where each simplification was produced by executing several rewriting transformations. Through quantitative and qualitative experiments, we show that simplifications in ASSET are better at capturing characteristics of simplicity when compared to other standard evaluation datasets for the task. Furthermore, we motivate the need for developing better methods for automatic evaluation using ASSET, since we show that current popular metrics may not be suitable when multiple simplification transformations are performed.

## introduction sentence simplification (ss) consists in modifying the content and structure of a sentence to make it easier to understand, while retaining its main idea and most of its original meaning @xcite . simplified texts can benefit non-native speakers @xcite , people suffering from aphasia @xcite , dyslexia @xcite or autism @xcite . they also help language processing tasks, such as parsing @xcite , summarisation (silveira and * equal contribution @xcite , and machine translation @xcite . in order simplify a sentence, several rewriting transformations can be performed: replacing complex words/phrases with simpler synonyms (i.e. lexical paraphrasing), changing the syntactic structure of the sentence (e.g. splitting), or removing superfluous information that make the sentence more complicated @xcite @xcite . however, models for automatic ss are evaluated on datasets whose simplifications are not representative of this variety of transformations. for instance, turkcorpus @xcite , a standard dataset for assessment in ss, contains simplifications produced mostly by lexical paraphrasing, while reference simplifications in hsplit @xcite focus on splitting sentences. the newsela corpus @xcite contains simplifications produced by professionals applying multiple rewriting transformations, but sentence alignments are automatically computed and thus imperfect, and its data can only be accessed after signing a restrictive publicsharing licence and cannot be redistributed, hampering reproducibility. these limitations in evaluation data prevent studying models' capabilities to perform a broad range of simplification transformations. even though most ss models are trained on simplification instances displaying several text transformations (e.g. wikilarge @xcite ), we currently do not measure their performance in more abstractive scenarios, i.e. cases with substantial modifications to the original sentences. in this paper we introduce asset (abstractive sentence simplification evaluation and tuning), a new dataset for tuning and evaluation of automatic ss models. asset consists of 23,590 human simplifications associated with the 2,359 original sentences from turkcorpus (10 simplifications per original sentence). simplifications in asset were collected via crowdsourcing ( § 3), and encompass a variety of rewriting transformations ( § 4), which make them simpler than those in turkcorpus and hsplit ( § 5), thus providing an additional suitable benchmark for comparing and evaluating automatic ss models. in addition, we study the applicability of standard metrics for evaluating ss using simplifications in asset as references ( § 6). we analyse whether bleu @xcite or sari @xcite scores correlate with human judgements of fluency, adequacy and simplicity, and find that neither of the metrics shows a strong correlation with simplicity ratings. this motivates the need for developing better metrics for assessing ss when multiple rewriting transformations are performed. we make the following contributions: ## creating asset we extended turkcorpus @xcite by using the same original sentences, but crowdsourced manual simplifications that encompass a richer set of rewriting transformations. since turkcorpus was adopted as the standard dataset for evaluating ss models, several system outputs on this data are already publicly available @xcite @xcite . therefore, we can now assess the capabilities of these and other systems in scenarios with varying simplification expectations: lexical paraphrasing with turkcorpus, sentence splitting with hsplit, and multiple transformations with asset. ## rewriting transformations in asset we study the simplifications collected for asset through a series of text features to measure the abstractiveness of the rewriting transformations performed by the annotators. from here on, the analysis and statistics reported refer to the test set only (i.e. 359 original sentences), so that we can fairly compare asset, turkcorpus and hsplit. ## rating simplifications in asset here we measure the quality of the collected simplifications using human judges. in particular, we study if the abstractive simplifications in asset (test set) are preferred over lexical-paraphrase-only or splitting-only simplifications in turkcorpus (test set) and hsplit, respectively. ## evaluating evaluation metrics in this section we study the behaviour of evaluation metrics for ss when using asset's simplifications (test set) as references. in particular, we measure the correlation of standard metrics with human judgements of fluency, adequacy and simplicity, on simplifications produced by automatic systems. ## conclusion we have introduced asset, a new dataset for tuning and evaluation of ss models. simplifications in asset were crowdsourced, and annotators were instructed to apply multiple rewriting transformations. this improves current publicly-available evaluation datasets, which are focused on only one type of transformation. through several experiments, we have shown that asset contains simplifications that are more abstractive, and that are consider simpler than those in other evaluation corpora. furthermore, we have motivated the need to develop new metrics for automatic evaluation of ss models, especially when evaluating simplifications with multiple rewriting operations. finally, we hope that asset's multi-transformation features will motivate the development of ss models that benefit a variety of target audiences according to their specific needs such as people with low literacy or cognitive disabilities.