Centering-based Neural Coherence Modeling with Hierarchical Discourse Segments (2020)

Abstract: Previous neural coherence models have focused on identifying semantic relations between adjacent sentences. However, they do not have the means to exploit structural information. In this work, we propose a coherence model which takes discourse structural information into account without relying on human annotations. We approximate a linguistic theory of coherence, Centering theory, which we use to track the changes of focus between discourse segments. Our model first identifies the focus of each sentence, recognized with regards to the context, and constructs the structural relationship for discourse segments by tracking the changes of the focus. The model then incorporates this structural information into a structure-aware transformer. We evaluate our model on two tasks, automated essay scoring and assessing writing quality. Our results demonstrate that our model, built on top of a pretrained language model, achieves state-of-the-art performance on both tasks. We next statistically examine the identified trees of texts assigned to different quality scores. Finally, we investigate what our model learns in terms of theoretical claims.

## introduction coherence describes the semantic relation between elements of a text. it identifies a text passage as either a unified whole or a collection of unrelated sentences. the most well-known formal theory, centering theory, determines the most salient item in each sentence, the center or focus, and tracks the changes of the focus @xcite . prior studies of coherence have mainly focused on modeling local coherence in centering theory @xcite . they aim to identify the semantic relations between adjacent sentences. however, coherence arises not only at the local level, but also at the document level giving insight into the structure of the discourse. discourse structure represents the semantic organization of a text. incorporating structural information into the model has been beneficial for diverse downstream tasks including text summarization @xcite , translation @xcite , sentiment analysis @xcite , and text classification @xcite . to identify discourse structure, earlier work adopts a supervised approach, relying on human annotations @xcite . however, annotating discourse structure is time consuming and costly. it requires annotators to understand not only the local context surrounding the target sentence but also higher level relations. learning latent structure has been proposed to alleviate this limitation. this approach induces the discourse structure from a text without annotations using an attention layer @xcite . recent work argues that, however, the learned trees have mostly little to no structure at the document level, and the model relies on specific linguistic cues @xcite . in this paper, we propose a coherence model inspired by centering theory which takes structural information into consideration. our model does not rely on human annotations to identify this information. our model consists of two components: the discourse segments parser first identifies the hierarchical discourse segments of a text, building upon an approximation of centering theory @xcite . this theory first defines three data structures to describe the focus of a sentence, a list of forward-looking centers (cf ), the preferred center (cp), and a single backward-looking center (cb). cf indicates the salient items of the sentence, that are candidates of the focus in the next sentence, and cp indicates the most preferred item of cf. cb describes the focus of a sentence with regards to the previous context. the theory also defines centering transitions to describe the changes of focus by comparing two centers, cp and cb. we propose an algorithm to approximate this theory using a pretrained language model. our algorithm first identifies the focus of sentences using multihead attention scores provided by the pretrained language model and semantic similarity between vector representations. our algorithm then constructs hierarchical discourse segments using a focus stack -inspired by the concept of @xcite -to track the changes of the focus between discourse segments. secondly, we propose a structure-aware transformer to account for structural information. @xcite introduce the transformer, a model solely based on a self-attention mechanism. this mechanism relates all items to capture semantic relations in a sequence. in contrast, the self-attention of our transformer is restricted to considering sentences with regards to the identified hierarchical discourse segments. we first calculate document structure priors to allow self-attention to relate sentences connected in the identified structure. then, the document structure attention is calculated by element-wise multiplication of the document structure priors and the self-attention of a naive transformer. we evaluate our model on two tasks: automated essay scoring (aes) and assessing writing quality (awq). aes is the task of assigning a score for a given essay, aiming to replicate human scoring results @xcite . this task has been used to evaluate coherence models @xcite . secondly, awq is the task of assigning labels of text quality recognized by human annotators. coherence is one of the most essential aspects of text quality @xcite . we first show that a simple fine-tuned model, relying on a pretrained language model, outperforms the state of the art on both tasks. we then demonstrate that our model achieves state-of-the-art performance on both tasks. our results indicate that the identified trees let the model assess text quality better by structure-aware coherence modeling. we then examine the identified trees to investigate differences of texts in writing quality. we finally inspect identified centers to investigate what our model learns in terms of theoretical claims. ## related work while unsupervised approaches for discourse parser have been developed @xcite , earlier work mostly adopted a supervised approach to identify discourse structure relying on human annotations. @xcite incorporate various linguistic features, including compositional semantics and part-of-speech information, to propose a discourse parser based on inductive logic programming. @xcite introduce a discourse parser which constructs discourse structure from a full input text. they train classifiers to identify discourse relations, and use them to build a tree structure of an input text. feng and hirst (2012) improve the tree building algorithm of this system by incorporating more linguistic features. @xcite introduce an svm-based model that consists of two stages, one identifying discourse structure, and the other classifying types of relations between units. more recently, neural models have been developed to recognize discourse structure. @xcite present a simple model based on a recursive neural network. @xcite claim that this model suffers from a vanishing gradient problem caused by long sequences, and propose an attentionbased hierarchical neural network model. to alleviate the shortage of human annotations, braud previous models of discourse parsing are mostly based on rhetorical structure theory @xcite . this theory represents a document as a tree structure built by connecting discourse units recursively through predefined discourse relations. another line of work is based on the penn discourse treebank @xcite , which annotates discourse structure in a lexically-grounded approach. these studies represent discourse structure with discourse relations. unlike these studies, our model does not consider discourse relations but we investigate centering theory to take structural relationships for discourse segments into account. a supervised approach requires annotations for each task. to overcome the lack of a labeled dataset, recent work has investigated to learn latent structures, which induce the tree structure directly from a text. while yogatama et al. @xcite and @xcite induce structure at the sentence level to learn syntax, @xcite propose a neural model which induces structural information without a labeled resource. they induce the non-projective dependency structure from a text by structured attention. more recently, however, @xcite claim that induced document-level structures do neither match human intuitions nor align with linguistic theories. unlike latent structure learning, we identify hierarchical discourse segments using a pretrained language model. it lets our model identify the focus of a sentence by comparing semantic similarities between representations of sentences without relying on a resource of manually labeled discourse structure. ## our model figure 1 presents the architecture of our coherence model. we first introduce input representations at the sentence level using a pretrained language model. we then describe the algorithm of the discourse segments parser. finally, we present a structure-aware transformer and the document representation created. ## conclusions we propose a neural model of coherence inspired by centering theory. the intuition is that it describes coherence by tracking the changes of the focus between discourse segments. our model identifies the hierarchy of discourse segments without human annotations, and incorporates structural information into the model. we demonstrate that the identified hierarchical discourse segments improve performance of the model on two tasks, automated essay scoring and assessing writing quality. interestingly, we find statistical differences of trees generated from texts of different quality. dani yogatama, phil blunsom, chris dyer, edward grefenstette, and wang ling. 2017. learning to compose words into sentences with reinforcement learning. in proceedings of the iclr conference. nan yu, meishan zhang, and guohong fu. 2018. transition-based neural rst parsing with implicit syntax features. in proceedings of the 27th international conference on computational linguistics, pages 559-570, santa fe, new mexico, usa. association for computational linguistics.