ST a CK : Sentence Ordering with Temporal Commonsense Knowledge (2021)

Abstract: Sentence order prediction is the task of finding the correct order of sentences in a randomly ordered document. Correctly ordering the sentences requires an understanding of coherence with respect to the chronological sequence of events described in the text. Document-level contextual understanding and commonsense knowledge centered around these events are often essential in uncovering this coherence and predicting the exact chronological order. In this paper, we introduce STaCK — a framework based on graph neural networks and temporal commonsense knowledge to model global information and predict the relative order of sentences. Our graph network accumulates temporal evidence using knowledge of ‘past’ and ‘future’ and formulates sentence ordering as a constrained edge classification problem. We report results on five different datasets, and empirically show that the proposed method is naturally suitable for order prediction. The implementation of this work is available at: https://github.com/declare-lab/sentence-ordering .

## introduction coherence is an essential quality for any natural language text @xcite . correct ordering of sentences is a necessary attribute of coherence. as such, there has been much research in correct sentence order detection due to its application to various down-stream tasks, such as, question answering @xcite , multidocument summarization @xcite , automated content addition to text @xcite ), text generation @xcite , and others. it also has potential applications in the evaluation of the quality of machinegenerated documents. existing approaches to sentence order prediction can be broadly classified into two categories: (1) sequence generation methods, and (2) pair-wise methods @xcite . while the former considers tagging the entire sequence, the latter takes one sentence pair at a time and predicts their relative order. pair-wise methods ignore the importance of document level global information, i.e., while predicting the relative order of two sentences (s i , s j ), other sentences s k from the same document do not play any role. ## background coherence and the problem of sentence order prediction have been extensively studied in literature due to their applicability in various downstream problems. early work in this direction mainly used domain knowledge and handcrafted linguistic features to model the relation between sentences in a document @xcite @xcite . sentence ordering methods in recent literature are primarily based on neural network architectures, and can be broadly categorized into two main families -i) sequence generation methods and ii) pair-wise methods. sequence generation methods use the entire sequence of the randomly ordered document to model local and global information. this information is then used to predict the correct order. the sentences and documents are typically encoded using a recurrent or transformer-based network @xcite @xcite . hierarchical encoding schemes are also common @xcite . prediction is then generally performed with a pointer network decoder @xcite based on beam search. alter-natively, ranking losses @xcite have also been explored to circumvent the expensive beam search algorithm @xcite . such models predict a score for each sentence, which are then sorted to obtain the final order. the pair-wise methods are motivated by a different principle of sentence ordering. these models aim to predict the relative order of each pair of sentences in the document. the final order is then constrained on all of the predicted relative orders. the constraint solving problem is generally tackled with topological sorting @xcite , or more sophisticated neural network models @xcite . our proposed stack framework falls under this family of pair-wise models. in stack, temporal commonsense is modelled using the commonsense transformers (comet) model @xcite . the comet model uses a bart @xcite sequence-to-sequence encoder decoder framework and is pretrained on the atomic 20 20 commonsense knowledge graph @xcite . the pretraining objective is to take a triplet {s, r, o} from the knowledge graph, and generate the object phrase o from concatenated subject and relation phrase s and r. the set of relations r include temporal relations 'is before' and 'is after'. comet is pretrained on approximately 50,000 of such temporal triplets along with other commonsense relations from atomic 20 20 . the pretraining on commonsense knowledge ensures that comet is capable of distinguishing cause-effects (causality), past-future (temporality), and other eventcentered commonsense knowledge. recently integer sequence generation methods have achieved impressive performance @xcite . however, there exist some key differences that make the generative approaches fundamentally different and incomparable from the family of pair-wise (sentence pair classification-based) approaches: ## methodology an overall illustration of the proposed stack framework is shown in fig. 2 . we represent document d consisting of n sentences as a directed graph @xmath0, e, r), with nodes v @xmath1, r ij , v j ) ∈ e, where r @xmath2. initial node embeddings are denoted as g i . ## conclusion in this work, we presented stack, a framework that uses relational graph convolutional network to model document-level contextual information and temporal commonsense knowledge for sentence order prediction. in the graph network, the edge classification objective was applied for pairwise relative order prediction of the sentence pairs. this was followed by a topological sorting for the final order prediction of the sentences. stack achieves state-of-the-art results in several benchmark datasets.