Small but Mighty: New Benchmarks for Split and Rephrase (2020)

Abstract: Split and Rephrase is a text simplification task of rewriting a complex sentence into simpler ones. As a relatively new task, it is paramount to ensure the soundness of its evaluation benchmark and metric. We find that the widely used benchmark dataset universally contains easily exploitable syntactic cues caused by its automatic generation process. Taking advantage of such cues, we show that even a simple rule-based model can perform on par with the state-of-the-art model. To remedy such limitations, we collect and release two crowdsourced benchmark datasets. We not only make sure that they contain significantly more diverse syntax, but also carefully control for their quality according to a well-defined set of criteria. While no satisfactory automatic metric exists, we apply fine-grained manual evaluation based on these criteria using crowdsourcing, showing that our datasets better represent the task and are significantly more challenging for the models.

## introduction split and rephrase is the task of rewriting a presumably long and complex sentence into shorter and simpler sentences, while maintaining the same meaning. for example, one possible way to split the sentence "voiced by aoi koga, kaguya is the series' titular character, popular among a wide audience." would result in "kaguya is voiced by aoi koga. kaguya is the series' titular character. kaguya is popular among a wide audience." while the split sentences have to be coherent, paraphrasing is not enforced. for example, the word "titular" does not have to be replaced. this type of text simplification is challenging as its natural language generation process potentially involves multiple sub-processes such as co-reference resolution, named-entity recognition, semantic role labelling, etc. split and rephrase has two main real-world uses: first, to benefit systems whose performance improves with decreasing length of sentences e.g. entity extraction @xcite and machine translation @xcite by acting as a pre-processing step; second, to benefit human readers, especially those less proficient with the language when reading complex documents such as terms and agreements, in understanding the meaning more easily and accurately @xcite . datasets of the split and rephrase task contain pairs of a complex sentence and a presumably meaning-preserving simplified rewrite containing multiple simpler sentences. the task was introduced by @xcite , with the release of the websplit corpus. afterwards, @xcite proposed the state-of-the-art model to date, a sequence-to-sequence model @xcite with a copy mechanism @xcite with the observation that most texts are unchanged during a split and rephrase operation. later, @xcite introduced the wikisplit corpus to be used as large but noisy training data, which the authors reported to be unsuitable as the evaluation data. also, @xcite studied the problems of using bleu as the evaluation metric for this task, while proposing a manually constructed test set called hsplit. we argue that the widely used benchmark dataset of split and rephrase, the websplit test set (known as simply websplit below), is not suitable for evaluation. apart from its series of limitations already reported, such as a small vocabulary, unnatural expressions, etc. @xcite , we further show that its complex sentences systematically fol-low only 3 syntactical patterns marked by lexical cues (section 2). to demonstrate the implication of such limitations of websplit, we show that a simple, unsupervised rule-based model with only 3 corresponding operations can perform even slightly better than the state-of-the-art neural model (section 3). to remedy the limitations of websplit, we crowdsource two new benchmarks with significantly more diverse syntax in the wikipedia and legal contract domain with hundreds of humanwritten complex-simple sentence pairs (section 4). we carefully control for their quality based on 6 well-defined criteria of what constitutes a good split and rephrase rewrite. while most related work reports model performance using the widely criticized bleu score and manual evaluation with no clear rubric, we perform fine-grained model evaluation using these 6 criteria, rated by crowd workers, showing that our benchmarks present models with greater challenges (section 5). ## issues with websplit websplit and wiki-split are two widely used datasets for the split and rephrase task. because wikisplit is derived from the edit history of wikipedia, versions of passages are not necessarily written by split and rephrases operations, as the meaning may not be preserved during edits. hence, wikisplit is reported by its authors to be noisy and ill-suited for evaluation for this task @xcite . websplit is used in multiple previous works as the evaluation benchmark. it was created by automatically matching sentences in the webnlg corpus @xcite according to partitions of their meaning representations. the dataset has been shown to have various limitations, such as unnatural expressions, repetition of phrases @xcite , etc. furthermore, our preliminary study shows that websplit contains several recurring syntactic patterns marked with lexical cues. to demonstrate this, we randomly sample 100 complex sentence from the test set, and are able to categorize them with only 3 syntactical patterns marked by lexical cues (underlined), at which some almost trivial split and rephrase operations can take place: relative clause (rc) (48 out of 100): scott adsit voiced baymax which was created by duncan rouleau. it can be further noticed that most complex sentences in websplit are short and require only one split and rephrase operation. we next show that a rule-based model which only exploits these patterns can perform on par with the state-of-the-art neural model on websplit. ## rule-based model we design a simple rule-based model to exploit the syntactic cues widely present in websplit. ## new benchmark datasets considering the limitations of websplit, an ideal benchmark must not only be challenging with diverse patterns, but also ensure that the rewrites are strictly meaning-preserving split and rephrase. with these two goals, we collect two benchmark datasets, wiki benchmark (wiki-bm) from wikipedia and contract benchmark (cont-bm) from the legal documents. these two datasets are to be used as gold standard for the evaluation of split and rephrase. to systematically control for the quality, we define 6 criteria of what constitutes a good split and rephrase, and validate the collected rewrites based on these criteria. ## model performance previous work reports the model performance on this task using two metrics: bleu on the entire benchmark and manual ratings on a small subset. however, bleu has long been shown to have little correlation with human judgements in text simplification foot_2 @xcite . while other alternatives exist, the focus of our work is not the metrics, but rather the quality and difficulty of benchmarks, which can be illustrated no better than by human evaluation. previously, manual evaluation has been done without a well-established rubric on what makes a split and rephrase rewrite correct. to address these problems, we use crowdsourcing following the process of phase 2, by asking 3 crowd workers to rate model outputs based on the 6 finegrained criteria described above. both the rule-based and seq2seq model have large rooms for improvement, as they significantly underperform crowd workers in almost all criteria, with significantly lower performance in our proposed benchmarks than in websplit. even for crowd workers, the percentage of overall correct is less than 70% in our new benchmarks, whose complex sentences are much more challenging to split and rephrase. ## reliability of the crowd can we use crowdsourcing to evaluate models no less reliably as experts or authors, as done in previous work? as experts of this task, we manually rate a subset of model-output rewrites as the ground truth for rating, and compare it against the crowd's rating. since there are 3 benchmarks and 3 models (including human, whose outputs are crowd rewrites we have collected in wiki-bm and cont-bm, but not websplit), there are 8 combinations in total. from the crowd ratings of these combinations, we assign each complex-output pair into one of 4 buckets, determined by the number of correct ratings out of 3 crowd ratings. for each bucket, we sample 2 complex-output pairs. in total, 8 × 4 × 2 = 64 complex-output pairs are sampled. the expert rates them independently following the same 6 criteria as the crowd workers. this gives the proportion of expert's correct ratings among each bucket. these statistics allow us to fit a beta distribution for expert rating conditional on each crowd rating bucket, using laplace prior smoothing. the results are shown in figure 1 . each distribution corresponds to a bucket with 0, 1, 2, or 3 out of 3 correct crowd ratings. for example, the right-most curve represents the probability density function where both the expert and the 3 crowd raters agree on a correct rating. according to the figure with a 90% one-sided confidence, when all 3 crowd raters rate a rewrite as correct, the expert also rates correct in more than around 80% of the samples; when none of the 3 crowd raters rate a rewrite as correct, the expert rates correct for less than around 10% of the samples. this shows that crowdsourcing can be a reliable way to evaluate models for this task, with variable reliability depending on the number of raters per sample and their agreement. ## conclusion and future work after showing the flaws of the current benchmarks in split and rephrase, we release two crowdsourced benchmarks, wiki benchmark and contract benchmark, created from wikipedia articles and legal documents respectively. our benchmarks contain significantly more diverse syntax and provide additional challenges to models. using fine-grained crowdsourcing evaluation on 6 welldefined criteria, we show that they provide a greater challenge to models. we hope our benchmark datasets and human judgements facilitate model development and metric design, respectively. moreover, future work should inspect the effect of split and rephrase on downstream tasks such as machine translation or information retrieval, and examine if models' performance on these tasks correlate with that on our benchmarks. at the wrong place or unnecessarily? 6. does the rewritten text have one or more sentences that should be further split? each question is accompanied by a positive and negative example, the same as in the previous section. the crowd workers answer the first two questions by dragging a draw bar between "strongly disagree" and "strongly agree", and the last four questions by choosing "yes/no" radio boxes.