A Simple yet Effective Method for Sentence Ordering (2021)

Abstract: Sentence ordering is the task of arranging a given bag of sentences so as to maximise the coherence of the overall text. In this work, we propose a simple yet effective training method that improves the capacity of models to capture overall text coherence based on training over pairs of sentences/segments. Experimental results show the superiority of our proposed method in in- and cross-domain settings. The utility of our method is also verified over a multi-document summarisation task.

## introduction and background document coherence understanding plays an important role in natural language understanding, where a coherent document is connected by rhetorical relations, such as contrast, elaboration, narration, and justification, allowing us to communicate cooperatively in understanding one another. in this work, we measure the ability of models to capture document coherence in the strictest setting: sentence ordering @xcite @xcite , a task of ordering an unordered bag of sentences from a document, aiming to maximise document coherence. the task of sentence ordering is to restore the original order for a given bag of sentences, based on the coherence of the resulting document. the ability of a model to reconstruct the original sentence order is a demonstration of its capacity to capture document coherence. figure 1 presents such an example, where the (shuffled) sentences are from a paper abstract discussing the relationship between word informativeness and pitch prominence, and the gold-standard sentence ordering is (4, 5, 1, 7, 3, 2, 6). furthermore, the task of sentence ordering is potentially beneficial for downstream tasks such as multi-document summarisation @xcite but there are others who express doubts about such a correlation. traditional approaches to sentence ordering used hand-engineered features to capture document coherence @xcite @xcite @xcite , e.g. using an entity matrix @xcite or graph @xcite to represent entity transitions across sentences, and maximising transition probabilities between adjacent sentences. neural work has modelled the task either generatively @xcite @xcite @xcite @xcite @xcite or discriminatively @xcite . as example genera-tive approaches, @xcite obtain sentence and paragraph representations from bert @xcite and then use a pointer network to decode the sentence ordering for a given paragraph, whereas @xcite use a graph-based neural network over sentences and entities. the shortcoming of generative methods is the difficulty in obtaining good paragraph representations, especially for longer paragraphs. to mitigate this, various attention mechanisms have been explored @xcite @xcite . discriminative approaches, on the other hand, can readily capture the relative order between sentence pairs, and paragraph decoding can then be achieved through methods such as beam-search @xcite or topological sort @xcite . however, even with exact decoding methods such as topological sort, issues remain, including: (1) coherence scores for sentence pairs that are distant in the document tend to be noisy; and (2) it can be difficult to determine the relative order of adjacent sentences without broader context. to mitigate these two drawbacks, we propose a simple yet effective training method. instance pairs are only constructed from adjacent segments to provide stronger coherence signals, but to capture broader context, up to 3 continuous sentences are combined to form a single segment in an instance pair. the effectiveness of our method is demonstrated across multiple datasets, in in-and cross-domain settings, and the setting of multi-document summarisation. ## methodology the method proposed by @xcite exploits the relative order between any two sentences in a given paragraph. as in figure 2a , the pairs connected by blue and red lines (pointing right and left, resp.) are the resulting positive and negative coherence instances for sentence s 2 , respectively. these instances are used to train a text coherence model, which we denote as "allpairs". in contrast, our method utilises the relative order between adjacent segments only, resulting in an order of magnitude less training data than allpairs (o(n) vs. o(n 2 )) but stronger supervision signal; we denote this as "adjonly". as in figure 2b , the blue/red lines connect adjacent sentences for sentence s 2 , resulting in positive/negative coherence instances. to capture broader context, we also construct pairs based on segments made up of multiple continuous sentences (not shown in the figure), such as (s 1:2 , s 2:3 ) and (s 1:3 , s 2:4 ) as positive instances, and (s 2:3 , s 1:2 ) and (s 2:4 , s 1:3 ) as negative instances, where s i:i+j denotes the concatenation of sentences s i to s i+j inclusive (@xmath0. in this work, we experiment with @xmath1, 1, 2} (i.e. sentence unigrams, bigrams, and trigrams), resulting in (at most) 6(n -2) instances for a paragraph with n sentences (noting that the segment cannot extend beyond the extremities of the document). at test time, following prabhumoye et al. (2020), we predict the relative order of each sentence pair (only sentence unigram), then order the sentences with topological sort. we also trialled other training methods -including regressing over the distance between two sentences, and training with constraints over sentence triplets inspired from @xcite in computer vision -but observed no improvement. ## conclusion and future work we propose a simple yet effective training method to predict the relative ordering of sentences in a document, based on sentence adjacency and topological sort. experiments on six datasets from different domains demonstrate the superiority of our