Sentence Ordering with a Coherence Verifier (2023)

Abstract: This paper presents a novel sentence ordering method by plugging a coherence verifier (CoVer) into pair-wise ranking-based and sequence generation-based methods. It does not change the model parameters of the baseline, and only verifies the coherence of candidate (partial) orders produced by the baseline and reranks them in beam search. We also propose a coherence model as CoVer with a novel graph formulation and a novel data construction strategy for contrastive pre-training independently of the sentence ordering task. Experimental results on four benchmarks demonstrate the effectiveness of our method with topological sorting-based and pointer network-based methods as the baselines. Detailed analyses illustrate how CoVer improves the baselines and confirm the importance of its graph formulation and training strategy. Our code is available at https://github.com/SN-Jia/SO_with_CoVer .

## introduction coherence is essential for effective communication. the correct order of sentences is a necessary attribute of text coherence. sentence ordering aims to organize a set of possibly unordered sentences into a coherent text. it is closely associated with coherence modeling. on one hand, it has been used as an objective for learning coherence models. on the other hand, it can be viewed as a follow-up module of coherence evaluation, e.g., for improving texts with low coherence scores. so sentence ordering has highly practical value in downstream tasks for evaluating and improving the quality of human writing @xcite or machine-generated content @xcite @xcite . recent sentence ordering studies can be classified into 2 categories: pair-wise ranking-based and sequence generation-based methods. pair-wise ranking-based methods first model the relative order of each sentence pair and then integrate all the predicted relative orders with some ranking methods to get the final order @xcite @xcite . for example, b-tsort (prabhumoye et al., 2020) uses bert for pair-wise classification, builds a constraint graph to integrate pair-wise predictions, and adopts the topological sorting algorithm for sentence ranking. sequence generation-based methods are mainly based on the pointer networks @xcite . an encoder encodes all unordered sentences in various ways to capture the paragraph-level contextual information @xcite @xcite @xcite , then a decoder iteratively selects the next one from the set of unordered sentences conditioned on the states of the encoder and the already ordered sentence sequence. however, both categories of methods have a shortcoming in that the coherence of ordered sentences is not directly optimized but is approximated by optimizing auxiliary tasks, e.g., pair-wise ordering and ranking algorithms, or optimizing a series of conditional decisions, e.g., iterative sentence selection by a pointer network. these sub-optimal objects have a misalignment with the purpose of finding an order with maximal global coherence. in this paper, we propose a simple sentence ordering method by introducing a coherence verifier (cover). it can be plugged into the ranking-based and sequence generation-based models. figure 1 shows an example of how cover works together with a sequence generation baseline. cover only intervenes in the generation process. at each inference step, we let the baseline provide top candidates as the next sentence (e.g., s 4 and s 3 ) and use cover to verify the coherence of the sentence sequence candidates (e.g., s 1 , s 2 , s 4 and s 1 , s 2 , s 3 ) and re-rank the candidates for future generations. as a result, our method combines local conditional evidence and global coherence. cover is trained to measure coherence independently of the sentence ordering task. this is reasonable and important since the input of a coherence model is an ordered sentence sequence rather than a set of unordered sentences, and the model can be pre-trained with multi-domain datasets. we propose a novel coherence model, with a new graph formulation to model sentence pair orders, sequence order, and paragraph-to-sentence relations, and a novel gradual permutation-based data construction strategy for effective contrastive pretraining from pairs of sentence orders with different coherence degrees. we evaluate the effectiveness of cover by letting it work with a topological sorting-based baseline b-tsort @xcite and a pointer network-based sequence generation baseline berson @xcite . experimental results on four benchmarks demonstrate that our method improves both baselines and especially, obtains a large gain for the topological sorting-based baseline. it also outperforms other recent methods. we conduct a series of in-depth analyses showing that our method can correct a large ratio of sentence pair classification errors made by b-tsort and improve ordering accuracy at the early decoding stage for berson, which alleviates the gap between training and inference, and reduces error propagation. these effects come from the key designs of our coherence model. moreover, the cover pre-trained with larger cross-domain datasets obtains better performance than the mod-els trained with domain-specific datasets. the results verify the importance of pre-training the independent coherence model and also indicate that sentence ordering and coherence modeling can cooperate and interact well. ## cover: the coherence model we propose a new graph-based coherence model as cover. specially, we propose a new graph formulation and model it with gnns for coherence evaluation ( §4.1). we also propose a new data construction strategy for contrastive pre-training of the coherence model ( §4.2). ## conclusion this paper has presented a novel sentence ordering method by incorporating a coherence verifier (cover). we show that cover works well with pair-wise ranking-based and sequence generationbased baselines. our framework combines local evidence from the baselines and larger context coherence from cover and can gradually improve partial orderings. the coherence verifier is independent of the sentence ordering task but can be optimized for sentence ordering (e.g., via gradual permutation), and can be pre-trained with multidomain datasets, obtaining superior performance compared with domain-specific models. so it is effective and easy to maintain and transfer. sentence ordering is often used as a training task for coherence modeling. this paper, however, suggests that coherence models can also support sentence ordering methods to correct incoherent texts. coherence models are able to identify sentences that are not well-connected. sentence ordering models can then be used to reorder these sentences to improves the coherence of the text with the assistance of the coherence models.