Entity-based Neural Local Coherence Modeling (2022)

Abstract: In this paper, we propose an entity-based neural local coherence model which is linguistically more sound than previously proposed neural coherence models. Recent neural coherence models encode the input document using large-scale pretrained language models. Hence their basis for computing local coherence are words and even sub-words. The analysis of their output shows that these models frequently compute coherence on the basis of connections between (sub-)words which, from a linguistic perspective, should not play a role. Still, these models achieve state-of-the-art performance in several end applications. In contrast to these models, we compute coherence on the basis of entities by constraining the input to noun phrases and proper names. This provides us with an explicit representation of the most important items in sentences leading to the notion of focus. This brings our model linguistically in line with pre-neural models of computing coherence. It also gives us better insight into the behaviour of the model thus leading to better explainability. Our approach is also in accord with a recent study (O’Connor and Andreas, 2021), which shows that most usable information is captured by nouns and verbs in transformer-based language models. We evaluate our model on three downstream tasks showing that it is not only linguistically more sound than previous models but also that it outperforms them in end applications.

## introduction coherence describes the semantic relation between elements of a text. it recognizes how well a text is organized to convey the information to the reader effectively. modeling coherence can be beneficial to any system which needs to process a text. example sentence 1 mr. specter, seeming exasperated, said in an interview thursday. focus candidates captured by xlnet "_said", "_in", "day", "_interview", "_,", "er", "_an", "th", "s", "_exasperated", ..., "spect" example sentence 2 at the same time, unadvertised products may have almost identical ingredients but less namerecognition. focus candidates captured by xlnet "_name", "ition", "_products", "-", "_un", "_may", "_less", "_ingredients", "_have", ..., "_same" recent neural coherence models @xcite encode the input document using large-scale pretrained language models @xcite . these neural models compute local coherence, semantic relations between items in adjacent sentences, on the basis of words and even sub-words. however, it has been unclear on which basis these models compute local coherence. @xcite present a neural coherence model, which allows to interpret focus information for the first time. their investigation reveals that neural models, adopting large-scale pretrained language models, compute coherence on the basis of connections between any (sub-)words or function words @xcite . in these cases, the model might capture the focus based on spurious information. while such a model might reach or set the state of the art in some end applications, it will do so for the wrong reasons from a linguistic perspective. this problem did not appear with pre-neural models, since they compute coherence on the basis of entities. early work about pronoun and anaphora resolution by @xcite assumes that there is one single salient entity in a sentence, its focus, which serves as a preferred antecedent for anaphoric expressions. centering theory @xcite builds on these insights and introduces an algorithm for tracking changes in focus. centering theory serves as basis for many researchers to develop systems computing local coherence by approximating entities @xcite @xcite . in this paper, we propose a neural coherence model which is linguistically more sound than previously proposed neural coherence models. we compute coherence on the basis of entities by constraining our model to capture focus on noun phrases and proper names. this provides us with an explicit representation of the most important items in sentences, leading to the notion of focus. this brings our model linguistically in line with pre-neural models of coherence. our approach is not only linguistically more sound but also is in accord with a recent empirical study by @xcite who investigate what contextual information contributes to accurate predictions in transformer-based language models. their experiments show that most usable information is captured by nouns and verbs. their findings suggest that we can design better neural models by focusing on specific context words. our work follows their findings by modeling entitybased coherence in an end-to-end framework to improve a neural coherence model. our model integrates a local coherence module with a component which takes context into account. our model first encodes a document using a pretrained language model and identifies entities using a linguistic parser. the local coherence module captures the most related representations of entities between adjacent sentences, the local focus. then it tracks the changes of local foci. the second component captures the context of a text by averaging sentence representations. we evaluate our model on three downstream tasks: automated essay scoring (aes), assessing writing quality (awq), and assessing discourse coherence (adc). aes and awq determine text quality for a given text, aiming to replicate human scoring results. since coherence is an essential factor in assessing text quality, many previous coherence models are evaluated on aes and awq. adc evaluates coherence models on informal texts such as emails and online reviews. in our evaluation, our model achieves state-of-the-art performance. we also perform a series of analyses to investigate how our model works. our analyses show that capturing focus on entities gives us better insight into the behaviour of the model, leading to better explainability. using this information, we examine statistical differences of texts assigned to different qualities. from the perspective of local coherence, we find that texts of higher quality are neither semantically too consistent nor too variant. finally, we inspect error cases to examine how our model works differently compared to previous models. ## related work entity-based modeling has been the prevailing approach to model coherence in pre-neural models. the entity grid is its most well-known implementation @xcite . it represents entities in a two-dimensional array to track their transitions between sentences. many variations have been proposed to improve this model, e.g., projecting the grid into a graph representation @xcite or converting the grid to a neural model @xcite . however, the neural version of the entity grid (tien nguyen and joty, 2017) has two limitations. first, @xcite state that entity grids applied to downstream tasks are often extremely sparse. in their evaluation, it is difficult to find meaningful entity transitions between sentences in the grids. accordingly, this model performs worse than other neural models. more importantly, this neural model cannot provide any clues of how this model works since tien nguyen and joty (2017) apply a convolutional layer on the entity grid. the feature map of the convolutional layer is not interpretable. they cannot examine which entity is assigned more importance than others by their model. in contrast, we constrain our model to capture focus on entities using noun phrases. then our model tracks the changes of focus. hence, it provides us with an interpretable focus (section 5). more recently, @xcite propose a neural coherence model to exploit both local and structural aspects. they evaluate their model on an arti-ficial task only, the shuffle test, which determines whether sentences in a document are shuffled or not. however, recent studies @xcite claim that this artificial task is not suitable to evaluate coherence models. @xcite show that the neural coherence models, which achieve the best performance on this task, do not outperform non-neural models on downstream tasks. more recently, @xcite find a weak correlation between the model performance in artificial tasks and downstream tasks. in our evaluation, we compare @xcite with ours in an artificial task as well as in three downstream tasks. @xcite perform the best in the artificial task, but do not outperform our model in three downstream tasks (section 4). ## our model figure 1 presents the architecture of our model. we first introduce our entity representation and sentence encoding using a pretrained language model. next, we describe a novel local coherence model. we then combine the two representations of local coherence and the context vector, simply averaged sentence representations. finally, we apply a feedforward network to produce a score label. ## conclusions we propose a neural coherence model based on entities by constraining the input to noun phrases. this makes our model better explainable and sets a new state of the art in end applications. it also allows us to reveal that texts of higher quality are neither semantically too consistent nor too variant. our findings suggest a few interesting directions for future work. our analysis shows that pretrained language models frequently exploit coreference relations to capture semantic relations. we could design an advanced neural model which exploits these relations explicitly. lastly, our work could be extended to a multilingual setup. our model is not tied to a specific pretrained language model but connect a language model with linguistic insights. it can employ a multilingual model @xcite , and our datasets can be translated to other languages.