How coherent are neural models of coherence? (2020)

Abstract: Despite the recent advances in coherence modelling, most such models including state-of-the-art neural ones, are evaluated on either contrived proxy tasks such as the standard order discrimination benchmark, or tasks that require special expert annotation. Moreover, most evaluations are conducted on small newswire corpora. To address these shortcomings, in this paper we propose four generic evaluation tasks that draw on different aspects of coherence at both the lexical and document levels, and can be applied to any corpora. In designing these tasks, we aim at capturing coherence-specific properties, such as the correct use of discourse connectives, lexical cohesion, as well as the overall temporal and causal consistency among events and participants in a story. Importantly, our proposed tasks either rely on automatically-generated data, or data annotated for other purposes, hence alleviating the need for annotation specifically targeted to the task of coherence modelling. We perform experiments with several existing state-of-the-art neural models of coherence on these tasks, across large corpora from different domains, including newswire, dialogue, as well as narrative and instructional text. Our findings point to a strong need for revisiting the common practices in the development and evaluation of coherence models.

## introduction a variety of research has focused on modelling textual coherence, with the goal of distinguishing coherent from incoherent documents. this distinction has important repercussions for a variety of downstream tasks, including automatic essay scoring, document summarization, and natural language generation @xcite @xcite . given the importance of the task, there is a long line of approaches proposed for modelling coherence. most earlier models leverage linguistic features to discriminate between coherent and incoherent documents: the classic work of @xcite , and several follow-up studies @xcite evaluated using a benchmark known as the order discrimination task, where the goal is to discriminate between an original coherent document and its variants in which sentences are randomly shuffled. while this method allows to train and evaluate models of coherence without the need for expensive human annotations, an important question left unanswered is what aspects of coherence this evaluation captures. high performance on such a proxy task does not necessarily mean that a model is capable of identifying naturally-incoherent documents, as noted by many researchers, and empirically demonstrated in a recent study by @xcite . despite this important limitation, a majority of coherence models focus on optimizing performance on the order discrimination task, and mainly on news corpora (such as the wall street journal). although such standardization of task and data makes comparison across models easy, it also undesirably encourages the community to gradually move away from the real goal of modelling coherence, and towards optimizing performance on the specific test itself. to address the above concerns, we propose four new tasks for evaluating different aspects of coherence, including appropriate use of connectives, consistency in topics, as well as overall temporal and causal cohesion among events and participants. we perform extensive evaluation of several existing state-of-theart (sota) models on these tasks, across a variety of corpora from different domains, including news, dialogue, as well as narrative and instructional text. importantly, our proposed evaluation tasks do not require specialized human annotations, and instead levarage automatic methods or pre-existing annotated corpora. similar to the standard order discrimination task, our proposed tasks also aim at distinguishing an original coherent document from an incoherent variant generated by manipulating certain elements of the original document. our results show a complex landscape of model performances that vary across tasks and corpora. ## background models of coherence can be divided into two main general approaches, with implications on how these models are evaluated. one group is concerned with directly measuring human ratings of coherence and/or text quality, with applications in language learning and automatic essay scoring @xcite @xcite @xcite @xcite . a second group of studies focuses on modeling coherence for the purpose of learning better representations that can improve downstream natural language processing applications, such as summarization and generation @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . the former group evaluates models by examining their correlation with human ratings of coherence and readability, whereas the latter develops proxies of coherence, such as the sentence order discrimination task. this second group is the most relevant to our study, and as such we further elaborate on several proxy tasks that have been commonly used for evaluating and comparing models of coherence. by far the most common evaluation technique is the order discrimination task, where a coherence model is required to distinguish between an original document and a set of incoherent variants automatically generated by randomly shuffling the sentences in the original text. earlier studies simply assumed that a shuffled variant of a document is less coherent than the original, an assumption later verified by @xcite . this task was initially proposed by @xcite , and has since become a standard benchmark for developing and evaluating coherence models. however, as noted by @xcite , "the synthetic data used in the [order discrimination] task only partially approximates coherence violations that human readers encounter". others have also noted that the order discrimination may be much easier than the actual task of coherence rating, hence proposing improved variants of this task. for example, @xcite suggest the insertion task, which evaluates a coherence model based on how well it can predict the original position of each sentence within a document. @xcite evaluate their model on a more challenging local discrimination task, where the random shuffling is performed locally (e.g., within a 3-sentence window) to generate incoherent variants of an original document. @xcite propose next sentence prediction that aims to predict which of the next 5 sentences in a document is the correct continuation given a history/context -a task they use as a discourse-based objective to learn text representations that also encode coherence. recognizing the limitations of the order discrimination task, many evaluate their models on additional tasks that aim at rating the overall quality of text, including summary coherence rating, readability assessment, and essay scoring @xcite @xcite @xcite . in the summary coherence rating task, the goal is to rank a human-generated document summary higher than a machine-generated summary of an original document, whereas for readability assessment a model must discriminate between an original difficult-to-read document and its easy-to-read simplified variant. although such evaluation tasks complement the order discrimination task and its variants, they still lack full interpretability since the overall quality and/or readability of a document depends on many factors of which coherence is one. @xcite address this limitation by compiling a new annotated dataset to evaluate coherence "in the wild". their dataset contains "real-world" user-generated text, such as emails and online reviews, each annotated by experts to reflect its level of coherence. we take a significant step further from the above work and propose four new evaluation tasks that each capture a different aspect of coherence, without requiring coherence-specific annotations. ## our proposed tasks we propose four tasks that capture various aspects of coherence, by manipulating a text at the local lexical and global document levels. at the lexical level, we focus on the role of discourse connectives (connective substitution), and overall lexical cohesion (sentence cloze). at the document level, we assess whether coherence models can detect a sudden shift in topic (topic switching), and whether they can predict the correct last sentence of a document to create a coherent story (story cloze). ## selected coherence models we select four existing neural coherence models to carry out evaluations on, based on two criteria: performance-wise, they were shown to yield good performance on the order discrimination task; modelwise, we wanted models that differ in their underlying architecture to assess whether this has an impact on the different coherence tasks. specifically, we focus on models that build upon and are extensions of raw text encoders; this choice is mainly dictated by the nature of some of our tasks where we require a model to make a local choice (at the word or sentence level) depending on the surrounding context. we do not consider models that are based on entity grids since they require an added level of abstraction (i.e., grammatical roles extracted from constituency trees), and encode context as a simple chain of noun-specific grammatical roles. we briefly describe the four models next. ## discussion and conclusions our results point to a first important conclusion: the task of order discrimination paints only a partial picture of what is sota in modelling coherence, as also noted by other researchers @xcite inter alia). this conclusion is evident from the high variability in performance of models across tasks and corpora. particularly, we observe that some models exhibit clear advantage over others on certain tasks and/or corpora, e.g., cohlstm does particularly well on the sentence cloze task which manipulates the degree of lexical cohesion in a document. in other words, different models seem to fit some tasks and corpora better than the others. in addition, certain properties of the corpora affect different models differently. we observe that document length is of particular importance for models that represent the overall coherence of an entire document either via convolution (unc), or lstm (parseq), as attested by their results on document-level tasks, namely topic switching and story cloze. for topic mixing, this is observed in the two corpora with long documents (fisher and pdtb), and for story cloze, by the poor performance on hellaswag. finally, all models still struggle with common-sense reasoning, especially when there is little context available. we believe this kind of common-sense reasoning over short stories is an integral aspect of overall coherence, and as such we see this relevant to coherence modeling. results comparing randomlygenerated incorrect endings vs. human generated ones strengthen our initial claim: the fact that random alternatives/distractors can be easily spotted by existing sota models does not necessarily mean that these models are capturing true aspects of coherence that human readers are sensitive to. all in all, our findings have important implications, both for the design of coherence models, and for their proper evaluation. specifically, the two should not be seen as independent goals, but rather they should inform each other: by designing appropriate and diverse evaluation tasks, we can design multi-faceted models that capture all aspects of coherence. at the same time better models require better evaluation techniques that clearly showcase their strengths and shortcomings. future work should aim at extending this line of work to other domains. for instance, these tasks could be easily applied to measure the coherence of texts produced by l2 learners or on machine translation output. furthermore, the tasks we propose could be further refined. for instance, in the sentence cloze task we pooled hypernyms, hyponyms and antonyms together to generate distractors but models might be more sensitive to one of these classes; similarly, the topic switching did not look at the similarity between topics which could also affect model performance. finally, one could also experiment with even lower levels of distortion than just half the sentences in the document.