Improving Graph-based Sentence Ordering with Iteratively Predicted Pairwise Orderings (2021)

Abstract: Dominant sentence ordering models can be classified into pairwise ordering models and set-to-sequence models. However, there is little attempt to combine these two types of models, which inituitively possess complementary advantages. In this paper, we propose a novel sentence ordering framework which introduces two classifiers to make better use of pairwise orderings for graph-based sentence ordering (Yin et al. 2019, 2021). Specially, given an initial sentence-entity graph, we first introduce a graph-based classifier to predict pairwise orderings between linked sentences. Then, in an iterative manner, based on the graph updated by previously predicted high-confident pairwise orderings, another classifier is used to predict the remaining uncertain pairwise orderings. At last, we adapt a GRN-based sentence ordering model (Yin et al. 2019, 2021) on the basis of final graph. Experiments on five commonly-used datasets demonstrate the effectiveness and generality of our model. Particularly, when equipped with BERT (Devlin et al. 2019) and FHDecoder (Yin et al. 2020), our model achieves state-of-the-art performance. Our code is available at https://github.com/DeepLearnXMU/IRSEG .

## introduction with the rapid development and increasing applications of natural language processing (nlp), modeling text coherence has become a significant task, since it can provide beneficial information for understanding, evaluating and generating multi-sentence texts. as an important subtask, sentence ordering aims at recovering unordered sentences back to naturally coherent paragraphs. it is required to deal with logic and syntactic consistency, and has increasingly attracted attention due to its wide applications on several tasks such as text generation @xcite and extractive summarization @xcite . recently, inspired by the great success of deep learning in other nlp tasks, researchers have resorted to neural sentence ordering models, which can be classified into: pairwise ordering models @xcite @xcite @xcite @xcite and set-to-sequence models @xcite @xcite @xcite @xcite @xcite . generally, the former predicts the relative orderings between pairwise sentences, which are then leveraged to produce the final ordered sentence sequence. its advantage lies in the lightweight pairwise ordering predictions, since the predictions only depend on the semantic representations of involved sentences. by contrast, the latter is mainly based on an encoder-decoder framework, where an encoder is first used to learn contexualized sentence representations by considering other sentences, and then a decoder, such as pointer network @xcite , outputs ordered sentences. overall, these two kinds of models have their own strengths, which are complementary to each other. to combine their advantages, @xcite propose fhdecoder that is equipped with three pairwise ordering prediction modules to enhance the pointer network decoder. along this line, @xcite introduce bert to exploit the deep semantic connection and relative orderings between sentences and achieve sota performance when equipped with fhdecoder. however, there still exist two drawbacks: 1) their pairwise ordering predictions only depend on involved sentence pairs, without considering other sentences in the same set; 2) their one-pass pairwise ordering predictions are relatively rough, ignoring distinct difficulties in predicting different sentence pairs. therefore, we believe that the potential of pairwise orderings in neural sentence ordering models has not been fully exploited. in this paper, we propose a novel iterative pairwise ordering prediction framework which introduces two classifiers to make better use of pairwise orderings for graph-based sentence ordering @xcite . as an extension of sentence-enity graph recurrent network (se-grn) @xcite , our framework enriches the graph representation with iteratively predicted orderings between pairwise sentences, which further benefits the subsequent generation of ordered sentences. the basic intuitions behind our work are two-fold. first, learning contextual sentence representations is helpful to predict pairwise orderings. second, difficulties of predicting ordering vary with respect to different sentence pairs. thus, it is more reasonable to first predict the orderings of pairwise sentences easily to be predicted, and then leverage these predicted orderings to refine the predictions for other pairwise sentences. concretely, we propose two graph-based classifiers to iteratively conduct ordering predictions for pairwise sentences. the first classifier takes the sentence-entity graph (se-graph) @xcite as input and yields relative orderings of linked sentences via corresponding probabilities. next, in an iterative manner, the second classifier enriches the previous graph representation by converting high-value probabilities into the weights of the corresponding edges, and then reconduct graph encoding to predict orderings for the other pairwise sentences. based on the final weighted graph representation, we adapt se-grn to construct a graph-based sentence ordering model, of which the decoder is also a pointer network. to the best of our knowledge, our work is the first to exploit pairwise orderings to enhance the graph encoding for graph-based set-to-squence sentence ordering. to investigate the effectiveness of our framework, we conduct extensive experiments on several commonly-used datasets. experimental results and in-depth analyses show that our model enhanced with some proposed technologies @xcite achieves the state-of-the-art performance. ## related work early studies mainly focused on exploring humandesigned features for sentence ordering @xcite @xcite @xcite . recently, neural network based sentence ordering models have become dominant , consisting of the following two kinds of models: with similar motivation with ours, that is, to combine advantages of above-mentioned two kinds of models, @xcite introduced three pairwise ordering predicting modules (fhdecoder) to enhance the pointer network decoder of attorder-net. recently, @xcite proposed berson that is also equipped with fhdecoder and utilizes bert to exploit the deep semantic connection and relative ordering between sentences. however, significantly different from them, we borrow the idea from the mask-predict framework @xcite @xcite to progressively incorporate pairwise ordering information into se-graph, which is the basis of our graph-based sentence ordering model. to the best of our knowledge, our work is the first attempt to explore iteratively refined gnn for sentence ordering. ## background in this section, we give a brief introduction to the se-grn @xcite , which is selected as our baseline due to its competitive performance. as shown in figure 1 , se-grn is composed of a bi-lstm sentence encoder, grn @xcite paragraph encoder, and a pointer network @xcite decoder. ## our framework in this section, we give a detailed description to our framework. as shown in figure 2 , we first introduce two graph-based classifiers to construct an iteratively refined sentence-entity graph (irse-graph). it is a weighted version of se-graph, where pairwise ordering inforamtion is iteratively incorporated to update ss-edge weights. then, we adapt the conventional grn to establish a neural sentence ordering model based on the final irse-graph. ## conclusion in this work, we propose a novel sentence ordering framework that makes better use of pairwise orderings for graph-based sentence ordering. specifically, we introduce two classifiers to iteratively predict pairwise orderings, which are gradually incorporated into the graph as edge weights. then, based on this refined graph, we construct a graph-based sentence ordering model. experiments on five datasets demonstrate not only the superiority of our model over baselines, but also the compatibility to other modules utilizing pairwise ordering information. moreover, when equipped with bert and fhdecoder, our enhanced model achieves sota performance across datasets. in the future, we plan to explore more effective gnn for sentence ordering. in particular, we will improve our model by iteratively merging nodes to refine the graph representation.