Discourse-Based Sentence Splitting (2021)

Abstract: Sentence splitting involves the segmentation of a sentence into two or more shorter sentences. It is a key component of sentence simplification, has been shown to help human comprehension and is a useful preprocessing step for NLP tasks such as summarisation and relation extraction. While several methods and datasets have been proposed for developing sentence splitting models, little attention has been paid to how sentence splitting interacts with discourse structure. In this work, we focus on cases where the input text contains a discourse connective, which we refer to as discourse-based sentence splitting. We create synthetic and organic datasets for discourse-based splitting and explore different ways of combining these datasets using different model architectures. We show that pipeline models which use discourse structure to mediate sentence splitting outperform end-to-end models in learning the various ways of expressing a discourse relation but generate text that is less grammatical; that large scale synthetic data provides a better basis for learning than smaller scale organic data; and that training on discourse-focused, rather than on general sentence splitting data provides a better basis for discourse splitting.

## introduction sentence splitting segments a sentence into two or more shorter sentences. it is a key component of sentence simplification. it has also been shown to help human comprehension @xcite and to be a useful preprocessing step for several nlp tasks, such as relation extraction @xcite and machine translation @xcite @xcite . there is a large body of work on sentence splitting. it has been studied in the context of many text simplification systems @xcite @xcite @xcite @xcite and is the focus of so-called, split-and-rephrase models @xcite @xcite . so far however, little attention has been paid to how discourse splitting interacts with discourse structure. as illustrated in importantly, in the case of discourse-based splitting, the discourse relation which holds in the input must be preserved in the split output. this is illustrated in in this paper, we focus on discourse-based sentence splitting and make the following contributions: ## related work together with deletion, reordering and substitution, sentence splitting is one of the main operations used in text simplification. early work on simplification used a rule based approach to splitting @xcite . for instance, @xcite defines 26 handcrafted rules for simplifying apposition and/or relative clauses in dependency structures and 85 rules to handle subordination and coordination. further work focused on learning statistical simplification models from parallel datasets of complex-simplified sentences derived from english wikipedia and simple english wikipedia. @xcite introduces a syntax-based machine translation model where splitting probabilities are learned from syntactic structure. @xcite induced a grammar from the parallel wikipedia corpus annotated with syntactic trees and use an integer linear programming model for selecting the most appropriate simplification from the space of possible rewrites generated by the grammar. they report learning 438 rules for sentence splitting. probabilistic models have also been proposed. @xcite determine splitting points using a dedicated probabilistic module trained on the parallel wikipedia corpus annotated with semantic structures while @xcite extends this approach to an unsupervised setting where splitting points are determined based on the maximum likelihood of sequences of thematic role sets present in the simplified version of english wikipedia. more recent work has directly adressed the sentence splitting task. @xcite introduce a dataset for training sentence splitting models called websplit and report results for various neural models trained on this data, comparing a vanilla sequence-to-sequence model with a multisource and a semantically informed model. @xcite present an alternative train/dev/test partition for websplit which better supports generalisation and show that adding a copy mechanism helps improve results. one limitation of the websplit corpus is that it uses a small vocabulary. to remedy this shortcoming, @xcite create a new dataset called wikisplit by mining wikipedia's edit history. wikisplit contains one million naturally occurring sentence splits. the authors show that incorporating wikisplit as training data produces a model which outperforms prior results on the websplit test data by 32 bleu points. while these efforts are focused on syntax-or semantic-based sentence splitting, our work targets discourse-based sentence splitting. closest to our work, @xcite ,a,c) defines a set of 35 hand-crafted transformation rules to recursively decompose sentences into a hi-erarchical structure relating core sentences linked via rhetorical relations. they do not generate a wellformed text and the proposed rule-based approach will fail to easily generalise to other languages. furthermore, because they focus on producing sentences representing minimal semantic units, their system outputs contain a large number of very short sentences which poses some readability issues. in contrast, we present a dataset for training discourse splitting models and transformer-based, encoder-decoder models for generating discourse splits. the included examples exhibit a single split per sentence and do not rely on a deep hierarchical representation of the discourse structure, thereby preserving readibility. ## models given a complex sentence c with discourse tree t and split output s, we consider and compare two approaches: an end-to-end approach c2s where the split output s is directly generated from c; and a pipeline approach p l which uses c's discourse tree to mediate the split i.e., first mapping c to its discourse tree t and second, mapping this tree to the split output s. we try both of these approaches in order to investigate how difficult it is for an endto-end model to incorporate the discourse structure on its own and to what extent, if any, explicit mediation of this information aids the performance of discourse-based splitting. for each of these two approaches, we explore different ways of combining the training data: using only the synthetic data (synth), only the organic data (organic) or both (synth+organic). we also investigate a pre-training and fine-tuning approach where we pre-train on the synthetic data and finetune on the organic data; and a multi-task learning approach where we multi-task on the intermediate mapping tasks (mapping c to t and mapping t to s) and on the end-to-end task (mapping c to s). d-reln d-con d-adv temporal:asynchronous "and afterwards", "but afterwards", "after which", "then", "after that", "after this", "but, after that", "and after this", "after which", "eventually", "and eventually", "and in turn", "in turn", "which, in turn", "and then", "and so", "later", "and later", "but later", "next", "before", "followed by", "when", "thereafter", "and thereafter", "after which", "before that", "but before that", "although before that", "prior to this", "earlier", "and earlier", "formerly", "previously", "after", "and previously", "recently" "afterward(s)", "after that", "eventually", "in turn", "later", "next", "thereafter", "before that", "earlier", "previously" temporal:synchrony "in the meantime", "but in the meantime", "whilst", "meanwhile", "while in the meantime", "while", "simultaneously", "and simultaneously" "in the meantime", "meanwhile", "simultaneously" contingency:cause "accordingly", "so", "as such", "and as such", "as a result", "and as a result", "however", "so that", "resulting in", "consequently", "and therefore", "and so", "with", "therefore", "which means", "which means that", "thus", "and thus", "thusly" "accordingly", "as a result", "consequently", "therefore", "thus" comparison:contrast "by comparison", "in comparison", "while", "compared to", "whilst", "by contrast", "in contrast", "and in contrast", "while", "although", "conversely", "and conversely", "nevertheless", "but", "none the less", "yet", "however", "on the other hand", "and on the other hand", "but on the other hand", "but", "whereas" "by/in comparison", "by/in contrast", "conversely", "nevertheless", "on the other hand" texpansion:conjunction "additionally", "and additionally", "and also", "and is also", "besides", "besides this", "aside from", "further", "furthermore", "and furthermore", "and further", "in addition to", "likewise", "and likewise", "moreover", "indeed", "similarly", "and similarly", "while" "additionally", "also", "besides", "furthermore", "in addition", "likewise", "moreover", "similarly" expansion:instantiation "for example", "for instance", "such as", "in particular" "for example", "for instance", "in particular" expansion:alternative "instead", "but instead", "though", "but rather", "rather" "instead", "rather" ## experimental setup all of our generative models use the bart architecture @xcite and were trained on a computing grid using 4 nvidia rtx 2080 ti gpus. each experiment starts by fine-tuning the facebook/bart-base model hosted by hugging-face foot_3 , which has 6 layers in each of the encoder and decoder, a hidden size of 768, and was pretrained to perform reconstruction of corrupted documents on a combination of books and wikipedia data. during training, we used a learning rate of 3e - foot_4 , a batch size of 16, and performed dropout with a rate of 0.1 and early stopping as regularisation measures. for each experiment we set aside 5% of the training set for validation. during generation, we perform beam search with a beam size of 4. we compare the following models: split baseline (bl split ) pre-trained bart finetuned on a 1m example dataset of both syntax-and discourse-based splittings (wikisplit). this baseline allows us to compare training with very large heterogeneous training data (bl split ) vs. learning from smaller, discourse-split data (bl dsplit ). discourse-split baseline (bl dsplit ) pretrained bart fine-tuned on a discourse-focused subset of wikisplit (d-wikisplit). this baseline is to be directly compared with bl split . parser pipeline baseline (p l p arse ) a pipeline of two models. the first uses the discourse parser process used to generate t s from cs in section 3 (c2t) and the second is a pre-trained bart finetuned on (t, s) data (t2s). we experimented training the t2s component on various datasets and found the best to be that trained purely on synthetic data. thus, any pipeline mentioned in the remainder of this paper refers to a specific c2t component connected to this same t2s component. this baseline allows us to compare pipeline models whose c2t component is learned on the split data vs. one where the c2t component uses an existing discourse parser. end-to-end model (e2e) pre-trained bart fine-tuned on discourse-split data. we report results for variants trained on d-cc-news-s (e2e synth ), d-wikisplit and d-muss (e2e organic ), and all three combined (e2e both ). pipeline model (pl) a pipeline of two models. the first model is pre-trained bart finetuned on (c, t ) data and the second is a pretrained bart fine-tuned on (t, s) data from d-ccnews-s. we report results for pipelines with a c2t component trained on all d-ccnews data (p l synth ), d-muss data (p l organic ), and d-ccnews combined with d-wikisplit and d-muss data (p l both ). pre-training and fine-tuning (pt+ft) pretrained bart fine-tuned on one data set before being further fine-tuned on another. we try training first on either synthetic or standard wikisplit data and then fine-tuning on d-wikisplit and d-muss data. using wikisplit for the first step was found to be the best performing configuration for the endto-end system (e2e ptf t ), while using d-ccnews proved better for the pipeline (p l ptf t ). ## results and discussion pipeline vs. end-to-end. while no single configuration outperforms all others, p l synth ranks high for meaning preservation (samsa and bleu) and for discourse structure (d-acc and d-rel). more generally, we see that p l models universally outperform their e2e variant in terms of discourse structure (rel-and d-acc). conversely, the e2e models tend to show better results in terms of meaning preservation (samsa and bleu). this suggests that while the p l models are good at producing valid connectives and the correct sentence order (high d-acc), their generative capacity needs improvement. synthetic vs organic data. another clear trend is that models trained with synthetic data have significantly higher d-acc than those trained with organic data. this confirms our hypothesis that, because it includes multiple variants of the same discourse split using different connectives and orderings, the synthetic data helps to train discourse splitting models that are better able to generalise i.e., are able to generate with different connectives for the same relation. ## conclusion in this paper we introduced the task of discoursebased sentence splitting together with a largescale dataset of both organic and synthetic discourse splits. experimental evaluation revealed that discourse-based, pipeline models have better discourse relation preservation capabilities than endto-end models, and that synthetic data is critical for learning models that can generalise i.e. that can generate mutliple variants of the same discourse relation. in future work, we would like to create more document-aware models incorporating both syntax-and discourse-based sentence splitting at the document level.