Modeling Structural Similarities between Documents for Coherence Assessment with Graph Convolutional Networks (2023)

Abstract: Coherence is an important aspect of text quality, and various approaches have been applied to coherence modeling. However, existing methods solely focus on a single document’s coherence patterns, ignoring the underlying correlation between documents. We investigate a GCN-based coherence model that is capable of capturing structural similarities between documents. Our model first creates a graph structure for each document, from where we mine different subgraph patterns. We then construct a heterogeneous graph for the training corpus, connecting documents based on their shared subgraphs. Finally, a GCN is applied to the heterogeneous graph to model the connectivity relationships. We evaluate our method on two tasks, assessing discourse coherence and automated essay scoring. Results show that our GCN-based model outperforms all baselines, achieving a new state-of-the-art on both tasks.

## introduction coherence describes the relationship between sentences that makes a group of sentences logically connected rather than just a random collection of them @xcite . it is an important aspect of text quality @xcite , and its modeling has been applied in many downstream tasks, including summarization @xcite , dialogue generation @xcite , machine translation @xcite and document-level text generation @xcite . given the importance of the task, there is a long line of methods proposed for coherence modeling. previous models leverage linguistic features to solve the problem. for example, entity grid-based methods @xcite capture the entity transition between adjacent sentences of a text to model local to fill this gap, we investigate a graph-based approach to model the correlation between docu-ments from the perspective of structural similarity. the main idea is to connect structurally similar documents through a graph and capture those connectivity relationships using graph convolutional networks (gcn) @xcite . in particular, inspired by @xcite , we first represent a document as a sentence graph, where nodes are sentences and two nodes will be connected if they contain semantically related nouns. our method further converts each sentence graph into a subgraph set as it proves to be an efficient approach for measuring the topological similarity between graphs @xcite . then, we construct a heterogeneous graph for the training corpus, containing document and subgraph nodes, based on subgraphs shared between documents. in this way, structurally-similar documents are explicitly linked through the subgraphs (shown in figure 1 ). finally, a gcn is applied to the heterogeneous graph to learn the representation of document nodes while considering the connections between them. we evaluate our method on two benchmark tasks foot_0 : assessing discourse coherence and automatic essay scoring. experimental results show that our method significantly outperforms a baseline model that does not consider structural similarities between documents, achieving a new state-ofthe-art performance on both tasks. in addition, we provide a comprehensive comparison and detailed analysis, which empirically confirm that structural similarity information helps to mitigate the effects of uneven label distributions in datasets and improve the model's robustness across documents with different lengths. ## related work our work is related to text coherence modeling and graph neural networks (gnn)-based methods for natural language processing (nlp). coherence modeling. inspired by centering theory @xcite , @xcite propose an entity-based approach (the entity grid) to assess coherence by considering entity transitions between adjacent sentences of a text. the entity grid model has been improved by grouping entities based on their semantic relatedness @xcite , incorporating entity-specific features @xcite , replacing grammatical roles with discourse roles @xcite . on the other hand, @xcite propose an entity graph to capture entity transitions between not only adjacent sentences but also non-adjacent ones. motivated by the functional sentence perspective of text coherence @xcite , @xcite improve the entity graph with graphbased features extracted from text structures. similarly, we also leverage the structural features of texts. however, instead of feeding individual documents' structure into the model as coherence patterns, we use them to capture the underlying correlation between documents. with the advent of deep learning, neural networks have been applied to coherence modeling. li and hovy (2014); @xcite learn to assess coherence by training a model to distinguish coherent texts from incoherent ones using different neural encoders. tien nguyen and joty (2017); joty et al. (2018) extend the entity grid model with a convolutional neural network. @xcite propose to enrich the coherence features of a document by considering discourse relations and syntactic patterns within it. jeon and strube (2020a) design structure-guided inter-sentence attention to learn a document's local and global coherence patterns. inspired by human reading habits, jeon and strube (2020b) investigate a model to measure a document's coherence by incrementally interpreting sentences. our work is in line with the above approaches to learning a coherence model based on neural networks. the main difference is that the above neural models focus on extracting features within a single document, whereas our graph-based approach aims to study the effectiveness of correlations between documents. specifically, rooting on the linguistic definition of text coherence, we model the correlation from the perspective of structural similarities between documents. ## method figure 2 shows an overview of our proposed method. we describe step-by-step how to capture the structural similarities between documents, including i) identifying the structure of a document (section 3.1); ii) representing the sentence graph of a document as a subgraph set (section 3.2); iii) building a corpus-level heterogeneous graph to connect structurally similar documents based on the shared subgraphs (section 3.3); iv) applying a gcn encoder to capture connectivity relationships between document nodes (section 3.4). ## conclusion in this paper, we investigated the effectiveness of structural similarity information between documents for coherence modeling. we proposed a graph-based method to connect structurally similar documents based on shared subgraphs, and model the connectivity relations with a gcn. experiments on two benchmark tasks show that our method consistently outperforms the baseline model, achieving state-of-the-art results on both tasks.