Is Everything in Order? A Simple Way to Order Sentences (2021)

Abstract: The task of organizing a shuffled set of sentences into a coherent text has been used to evaluate a machine’s understanding of causal and temporal relations. We formulate the sentence ordering task as a conditional text-to-marker generation problem. We present Reorder-BART (Re-BART) that leverages a pre-trained Transformer-based model to identify a coherent order for a given set of shuffled sentences. The model takes a set of shuffled sentences with sentence-specific markers as input and generates a sequence of position markers of the sentences in the ordered text. Re-BART achieves the state-of-the-art performance across 7 datasets in Perfect Match Ratio (PMR) and Kendall’s tau. We perform evaluations in a zero-shot setting, showcasing that our model is able to generalize well across other datasets. We additionally perform several experiments to understand the functioning and limitations of our framework.

## introduction constructing coherent text requires an understanding of entities, events, and their relationships. automatically understanding such relationships among nearby sentences in a multi-sentence text has been a longstanding challenge in nlp. sentence ordering task was proposed to test the ability of automatic models to reconstruct a coherent text given a set of shuffled sentences @xcite . coherence modeling has wide applications in natural language generation like extraction-based multi-document summarization @xcite @xcite , retrieval dependent qa @xcite and concept-to-text generation @xcite . earlier studies on coherence modeling and sentence ordering focused on exploiting different categories of features like coreference clues (elsner and * authors contributed equally. i packed my raincoat. the forecast called for rainy. it never rained. the weather is never predictable. instead it started to snow. 1 2 3 4 5 the forecast called for rainy. i packed my raincoat. it never rained. instead it started to snow. the weather is never predictable. ## related work the problem of sentence ordering can be formulated as finding an order with maximum coherence. earlier works focused on modeling local coherence using linguistic features @xcite @xcite . a line of work have leveraged neural networks to encode sentences and retrieve the final order using pointer network @xcite by comparing them in a pairwise manner @xcite @xcite @xcite . han @xcite and tgcm @xcite used an attention based pointer network for decoding. b-tsort (prabhumoye et al., 2020) uses topological sorting to retrieve the final order from sentence pairs. @xcite encode sentence-level relationships as constraint graphs to enrich sentence representations. the state-of-the-art approach @xcite introduced a novel pointer decoder with a deep relational module. other works considered reframing the task as a ranking problem. @xcite proposed a model which relies on a ranking framework to retrieve the order of sentence pairs. kumar et al. (2020) utilized a bert @xcite encoder to generate scores for each sentence which were used to sort them into the correct order. different from these approaches, we formulate sentence ordering as a conditional text generation task. we use a sequence-to-sequence model in our framework where the decoder encapsulates the functioning of a pointer network while generating output sentence positions. our code is available at: https://github.com/fabrahman/rebart . ## re-bart given a sequence of shuffled sentences @xmath0, s 2 , . . . , s n s }, where s i denotes the i th sentence and n s denotes the number of input sentences, the task is to generate the ordered output sequence s * = {s 1 , s 2 , . . . , s n s }. we solve the sentence ordering task using a textto-marker framework shown in figure 2 . specifically, taking a shuffled sequence of sentences (s ) as input, we generate a sequence of position markers @xmath1, y 2 , . . . , y n s } as output, where y i denotes the position of i th sentence of the corresponding ordered sequence (s i ) in the shuffled input. the ordered output sequence can then be reconstructed as ŝ = {s y 1 , s y 2 , . . . , s y n s }. our goal is to train a probabilistic model p θ (y|s ) by optimizing: the functioning of re-bart model is shown in figure 2 . re-bart consists of a sequenceto-sequence model with an encoder to receive a shuffled set of sentences, and a decoder to generate position markers (2, 1, 3 etc.), which is then used to retrieve the final ordered sequence. we use bart @xcite as the underlying sequence-to-sequence model, since our task can benefit from the sentence permutation pre-training objective. additionally, to provide the model with a supervision signal to generate position markers, we append each sentence in the shuffled input with sentence markers (<s1>, <s2>, etc.). 1 sentence markers were added as special tokens to the tokenizer. re-bart learns to attend to the markers while generating the final order y . the proposed text-to-marker framework has two advantages over an alternate text-to-text framework, where the model directly generates the entire text sequence instead of marker outputs. first, the model performs better as the output space is much smaller. this also makes it less susceptible to neural text degeneration @xcite , as significantly fewer output tokens are generated. second, when generating the entire text sequence in the text-to-text framework, we observe that the model often generates text which is not part of the input, rendering the output invalid for the task. ## results in this section, we evaluate the performance of re-bart on several benchmark sentence ordering datasets. we also conduct a series of experiments to better understand the working of our model and investigate its generalization capability. we observe that re-bart's performance on wikipedia movie plots is relatively poor compared to other datasets. this could be because this dataset has relatively longer input sequences ( comparison with text-to-text framework: from results in ## analysis in this section, we perform experiments to explore re-bart's functioning with variation in different aspects of inputs. ## conclusion in this work, we address the task of sentence ordering by formulating it as a conditional text generation problem. we observe that simply generating output text from shuffled input sequences is difficult due to neural text degeneration. we solve this problem by proposing re-bart, a text-tomarker generation framework. re-bart achieves the state-of-the-art performance on 7 benchmark datasets and is able to generalize well across different domains in a zero-shot setup. we investigated the limitations of our model, and found that re-bart is sensitive to various factors like number of input sentences and degree of shuffling. future works can focus on developing models which are robust to such factors.