BERT -enhanced Relational Sentence Ordering Network (2020)

Abstract: In this paper, we introduce a novel BERT-enhanced Relational Sentence Ordering Network (referred to as BRSON) by leveraging BERT for capturing better dependency relationship among sentences to enhance the coherence modeling for the entire paragraph. In particular, we develop a new Relational Pointer Decoder (referred as RPD) by incorporating the relative ordering information into the pointer network with a Deep Relational Module (referred as DRM), which utilizes BERT to exploit the deep semantic connection and relative ordering between sentences. This enables us to strengthen both local and global dependencies among sentences. Extensive evaluations are conducted on six public datasets. The experimental results demonstrate the effectiveness and promise of our BRSON, showing a significant improvement over the state-of-the-art by a wide margin.

## introduction coherence modeling is one of the essential aspects of natural language processing @xcite @xcite . a coherent text can facilitate understanding and avoid the confusion for reading comprehension. the sentence ordering task @xcite aims to reconstruct a coherent paragraph from an unordered set of sentences and has shown to be beneficial to improve the coherence in many nlp tasks including multi-document summarization @xcite , conversational analysis @xcite , and text generation @xcite . on the other hand, to better capture the global coherence, pointer network @xcite has been gradually used for the decoder of the ordering model. it is able to capture the paragraphlevel contextual information for generating an ordered sequence with the highest coherence probability @xcite @xcite . further, han @xcite and tgcm @xcite introduce the attention mechanism @xcite , and fudecoder @xcite proposes pairwise ordering prediction modules to enhance the traditional pointer network. despite having achieved great successes, pairwise ranking and pointer network-based ordering approaches have a few problems. the former focuses on learning the local relationship between sentence pairs, but may have trouble in capturing the global interactions among all the sentences. the latter overlooks the importance of learning relative order between sentence pairs through the encoder-decoder, and lacks enough local interactions among sentences. to address the above limitations, in this paper, we propose a novel bert-enhanced relational sentence ordering network (referred to as berson) by integrating bert @xcite with the pointer network to fully exploit the pairwise relationships between sentences for a better coherence modeling. specifically, we first introduce a bert-based hierarchical relational sentence encoder, which uses sentence pairs as the input to the model and learns the high-level representation for each sentence. next, a self-attention based paragraph encoder is adopted for paragraph encoding. building upon the above pairwise sentence and paragraph encoding, a novel relational pointer decoder (referred to as rpd) is developed by incorporating the informative relative ordering information into the pointer network with a deep relational module (referred to as drm). this module leverages the next sentence prediction objective of bert to learn the relative ordering between sentences and constructs a pairwise relationship representation for each sentence pair, which helps rpd not only exploit the global orientation information among unordered sentences but also consider the local coherence between the candidate sentence and the previously sorted ones. thus, rpd is able to generate a more coherent order assignment for the input sentences. in addition, the pairwise ordering prediction loss is also added as the auxiliary objective to guide the coherence modeling in the training procedure. the overall architecture of our model is presented in figure 1 . extensive experiments are conducted on six public datasets in different domains to evaluate the performances of berson. the results show that berson significantly outperforms the existing approaches by a wide margin and achieves a state-of-the-art performance on all the datasets and under all the evaluation measurements. ## relational sentence ordering network in this section, we start by formulating the sentence ordering problem and then present the proposed model berson, which is composed of a bert-based hierarchical relational sentence encoder, a self-attention based paragraph encoder, and a relational pointer decoder enhanced by a new deep relational module to model the text coherence in a more effective way. ## experiments in this section, we empirically evaluate the effectiveness of berson in the sentence ordering task.