B i SECT : Learning to Split and Rephrase Sentences with Bitexts (2021)

Abstract: An important task in NLP applications such as sentence simplification is the ability to take a long, complex sentence and split it into shorter sentences, rephrasing as necessary. We introduce a novel dataset and a new model for this ‘split and rephrase’ task. Our BiSECT training data consists of 1 million long English sentences paired with shorter, meaning-equivalent English sentences. We obtain these by extracting 1-2 sentence alignments in bilingual parallel corpora and then using machine translation to convert both sides of the corpus into the same language. BiSECT contains higher quality training examples than the previous Split and Rephrase corpora, with sentence splits that require more significant modifications. We categorize examples in our corpus and use these categories in a novel model that allows us to target specific regions of the input sentence to be split and edited. Moreover, we show that models trained on BiSECT can perform a wider variety of split operations and improve upon previous state-of-the-art approaches in automatic and human evaluations.

## introduction understanding long and complex sentences is challenging for both humans and nlp models. nlp tasks like machine translation @xcite and dependency parsing @xcite tend to perform poorly on long sentences. text simplification @xcite is often formulated with a specific step to break longer sentences into shorter sentences. this task is referred to as split and rephrase @xcite . several past efforts have created split and rephrase training sets, which consist of long, complex input sentences paired with multiple shorter sentences that preserve the meaning of the input sentence. @xcite introduced the websplit corpus based on decomposing a long sentence into rdf triples (a form of semantic representation), and generating shorter sentences from subsets of these triples. however, the reliance on rdf triples and a limited vocabulary results in unnatural expressions @xcite and repeated syntactic patterns @xcite . more recently, the wikisplit corpus @xcite was introduced. it contains one million training examples of sentence splitting that were mined from the revision history of english wikipedia. while this yields an impressive number of training examples, the data are often quite noisy, with around 25% of wikisplit pairs containing significant errors (detailed in §3.2). this is because wikipedia editors are not only trying to split a sentence, but also often simultaneously modifying the sentence for other purposes, which results in changes of the initial meaning. in this paper, we introduce a novel methodology for creating split and rephrase corpora via bilingual pivoting @xcite . figure 1 demonstrates the process. first, we extract all 1-2 and 2-1 sentence-level alignments @xcite ) from bilingual parallel corpora, where a single sentence in one language aligns to two sentences in the other language. we then machine translate the foreign sentences into english. the result is our bisect corpus. split and rephrase corpora, including bisect, contain pairs with variable amounts of rephrasing. some pairs only edit around the split location, while others require more involved changes to maintain fluency. in this work, we leverage this knowledge by introducing a classification task to predict the amount of rephrasing required, and a novel model that targets that amount of rephrasing. the main contributions of this paper are: ## related work the idea of splitting a sentence into multiple shorter sentences was initially considered a sub-task of text simplification @xcite . however, the structural paraphrasing required to split a sentence makes for an interesting problem in itself, with many downstream nlp applications. thus, @xcite proposed the split and rephrase task, and introduced the websplit corpus, created by aligning sentences in webnlg @xcite . websplit contains duplicate instances and phrasal repetitions @xcite , and most splitting operations can be trivially classified @xcite , so subsequent split and rephrase corpora have been created to improve training @xcite and evaluation @xcite . the main work we compare against is wikisplit, a corpus created by extracting split sentences from wikipedia edit histories @xcite . concurrent work used a subset of wikisplit to focus on sentence decomposition @xcite . while this approach is able to both extract many potential sentence splits and transfer across languages, edited sentences do not necessarily have to retain the same meaning. in contrast, our corpus bisect is created from aligned parallel documents. bilingual corpora is generally leveraged for monolingual tasks with bilingual pivoting @xcite , which assumes that two english phrases that translate to the same foreign phrase have similar meaning. this technique was used to create the paraphrase database @xcite , a collection of over 100 million paraphrase pairs, and to improve neural approaches for sentential paraphrasing @xcite @xcite and sentence compression @xcite . in introducing the split and rephrase task, @xcite also reports the performance of several baseline models, where the strongest is an lstm-based model. subsequent works have improved performance using a copy-attention mechanism @xcite . we instead start with a bert-initialized transformer model @xcite , and train it with an adaptive loss function to emphasize split-based edits. concurrent work also introduced an additional neural graph-approach for split and rephrase @xcite . ## bisect corpus to address the need of split and rephrase data that is both meaning preserving and sufficient in size for training, we present the bisect corpus. ## our model the bisect corpus contains a significant amount of paraphrasing along with sentence splitting, and models trained on bisect tend to alter the lexical choices made in the input sentence. although this is desirable in some situations, like for the task of sentence simplification, sometimes it can alter the meaning of the input sentence. we propose a novel model that allows finer-grained control over what parts of the sentence are changed. our approach leverages the sentence split categories described in §3.3 to identify the split-based edits and incorporates them into a customized loss function as distantly supervised labels. this section describes the base model and its variant that adapts a high paraphrasing bisect corpus to a sentence splitting task with minimal rephrasing. ## experiments and results in this section, we compare different split and rephrase models trained on our new bisect corpus. we also conduct a carefully designed human evaluation as automatic metrics are not totally reliable. our model trained on bisect establishes a new start-of-the-art for the task. ## conclusion in this work, we introduce bisect, a new corpus for the split and rephrase task in several languages. we create this by making use of bilingual parallel corpora, and translating instances of aligned split sentences. we show that the sentence splitting models trained on our new corpus generate fewer errors than their counterparts trained on the existing datasets. to further improve meaning preservation and diversity, we propose a novel approach that identifies split-related edits in a training pair using linguistic rules and trains the model solely on splitbased edits. our proposed approach trained on bisect outperforms existing systems in terms of both automatic and human evaluations. we plan to investigate and create better automatic evaluation metrics for future work.