A Neural Graph-based Local Coherence Model (2021)

Abstract: Entity grids and entity graphs are two frameworks for modeling local coherence. These frameworks represent entity relations between sentences and then extract features from such representations to encode coherence. The benefits of convolutional neural models for extracting informative features from entity grids have been recently studied. In this work, we study the benefits of Relational Graph Convolutional Networks (RGCN) to encode entity graphs for measuring local coherence. We evaluate our neural graph-based model for two benchmark coherence evaluation tasks: sentence ordering (SO) and summary coherence rating (SCR). The results show that our neural graph-based model consistently outperforms the neural grid-based model for both tasks. Our model performs competitively with a strong baseline coherence model, while our model uses 50% fewer parameters. Our work defines a new, efficient, and effective baseline for local coherence modeling.

## introduction local coherence is a discourse property that distinguishes a high-quality text from a random sequence of sentences. modeling local coherence is crucial for various downstream nlp applications, e.g., summary evaluation and generation @xcite , readability assessment @xcite , essay scoring @xcite , dialogue evaluation and generation @xcite , and machine translation @xcite . motivated by the centering theory @xcite , many approaches to local coherence modeling rely on entity relations between sentences. the entity grid (barzilay and lapata, 2005, 1 https://github.com/ukplab/emnlp2021neural-graph-based-coherence-model 2008) and the entity graph @xcite are two well-studied frameworks for representing entity relations in a text. entity grid-based models use grids while entity graph-based models use graphs to capture entity relations between sentences. several methods have been proposed to enrich these representations and also to extract features from these representations to model local coherence. recent work shows the effectiveness of convolutional neural networks (cnns) for extracting features from entity grids to encode coherence @xcite . pre-trained transformer-based encoders can also capture relations between tokens in a text @xcite . however, these encoders are potentially incapable of capturing long-distance relations @xcite , specifically where the text length is greater than the maximum input length in these encoders. in this work, we revisit graph-based coherence assessment by introducing a neural graph-based coherence model. to do so, we represent a text via a graph (figure 1 ) since a graph can capture longdistance relations in a text. such a graph contains two types of edges: (1) edges that capture entitybased relations between sentences, and (2) edges that capture the linear order of sentences in the text. to encode such graphs, we adapt relational graph convolutional networks (rgcns) @xcite . rgcns encode nodes of a graph into vectors using the graph's connectivity structure and any feature information captured in the graph, such as edge types. we then apply a self-attention layer to these node vectors to capture to what extent each sentence of the text is crucial for estimating the coherence of the entire text. we finally use an output layer to transform the outputs of the self-attention layer to a score, which estimates the coherence degree of the text. figure 2 depicts an overview of our model. we evaluate our model for two benchmark co-s1: ldi crop., cleveland, said it will offer $50 million in commercial paper backed by lease-rental receivables. s2: the program matches funds raised from the sale of the commercial paper with small to medium-sized leases. s3: ldi leases and sells data-processing telecommunications and other high-tech equipment. s4: ldi termed the paper 'non-resource financing', meaning that investors would be repaid from the lease receivables, rather than directly by ldi corp. s1 s2 s3 ... sn v1 v2 v3 ... vn adj adj ent ent ent ent rgcn layer v1 v2 v3 ... vn self-attention layer output layer c figure 2: an overview of our coherence model. herence evaluation tasks: (1) sentence ordering (so) on the wall street journal (wsj) corpus, and (2) summary coherence rating (scr) on the document understanding conference (duc 2003) corpus. the results of our experiments confirm that our model consistently outperforms the neural gridbased coherence models (tien nguyen and joty, 2017; joty et al., 2018) by about 3.10% for so and 1.2% for scr. our model performs on par with a recent coherence model (moon et al., 2019), while our model uses 50% fewer parameters. ## experiments we evaluate our model for two benchmark tasks for coherence modeling: sentence ordering (so) and summary coherence rating (scr). in so, a text is compared with random permutations of its sentences @xcite . a coherence model should ideally rank a text higher than its permutations concerning coherence. in scr, we deal with ranking summary texts, where each summary text comes with a coherence rating assigned by human judges @xcite . given a pair of summary texts with different coherence ratings, a coherence model is expected to rank them properly with respect to their coherence ratings. datasets for so, we follow prior work @xcite @xcite and use the wall street journal (wsj) english news corpus. we use the same data splits and text permutations as used by @xcite . sections 00-13 of wsj are used for training and sections 14-24 for testing ( ## results and discussion our model performs substantially better than entgraph. similar to entgraph, we use graphs to represent relations between sentences. however, entgraph relies on merely entity-based relations to construct graphs and uses a heuristically-defined feature (i.e., the average outdegree of nodes in a graph) to estimate the text coherence. our model performs better because our graphs contain edges for capturing linear order of sentences as well as entity-based relations. moreover, our model adapts rgcn to extract features for estimating coherence. our model also outperforms the examined entity grid-based models. the neural entgrid and lex. neural entgrid models represent entity relations in text by entity grids and then apply cnns to these grids to extract features for modeling the text coherence. differently, our model uses graphs to represent relations between sentences and applies rgcn to learn features from graphs. our model slightly outperforms the model proposed by @xcite . we note that the best results for m&m are 92.93 for so and 83.8 for scr, achieved with elmo as word embeddings. we compare with their word2vec setting to study the influence of our models, not word embeddings. moon et al. (2019) 's model uses no explicit representations of text structure (neither graphs nor grids). it captures linear relations between adjacent sentences using a neural bilinear layer, and their relations with a global representation of a text using a cnn-based module. this model is trained by a language model loss together with a ranking loss specifically designed for so. our model achieves scores similar to those of ( moon et al., 2019 note that the neural entgrid's score for scr is its best performing results, where the model is first pretrained for so and then fine-tuned on the training set of the scr's dataset. our model outperforms the neural entgrid model while our model is trained for scr from scratch, i.e., without pretraining. it is worth noting that the size of the test split used for scr is small (80 text pairs). the improvements achieved by our model translates into the fact that our model makes 10 and 6 out of 80 correct rankings more than what neural entgrid and the (moon et al., 2019)'s model make, respectively. however, such improvements on the scr's dataset are important as texts in this dataset are associated with human-provided coherence ratings. ## conclusions we introduced a neural graph-based model for local coherence assessment. we construct a graph of relations among sentences in a text using entitybased and linear relations between sentences. we apply relational graph convolutional networks to such graphs to extract features encoding coherence. our model outperforms its counterparts for sentence ordering and summary coherence rating. the high performance of current coherence models on tasks with synthetic data possibly being not representative of real-life performance @xcite . so, we aim to further study the performance of our model for tasks with natural data.