Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks (2021)

Abstract: Although coherence modeling has come a long way in developing novel models, their evaluation on downstream applications for which they are purportedly developed has largely been neglected. With the advancements made by neural approaches in applications such as machine translation (MT), summarization and dialog systems, the need for coherence evaluation of these tasks is now more crucial than ever. However, coherence models are typically evaluated only on synthetic tasks, which may not be representative of their performance in downstream applications. To investigate how representative the synthetic tasks are of downstream use cases, we conduct experiments on benchmarking well-known traditional and neural coherence models on synthetic sentence ordering tasks, and contrast this with their performance on three downstream applications: coherence evaluation for MT and summarization, and next utterance prediction in retrieval-based dialog. Our results demonstrate a weak correlation between the model performances in the synthetic tasks and the downstream applications, motivating alternate training and evaluation methods for coherence models.

## introduction and related work coherence is an important aspect of discourse that distinguishes a well-written text from a poorlywritten one that is difficult to comprehend @xcite . computational models that can assess coherence have applications in text generation and ranking, such as summarization, machine translation, essay scoring and dialog systems. researchers have proposed a number of formal theories of discourse coherence, which have in-spired the development of many coherence models -both traditional and neural ones. inspired by the centering theory @xcite , the entity based local models @xcite ) formulate coherence in terms of syntactic roles (e.g., subject, object) of entities in nearby sentences. another branch of models @xcite @xcite use coherence relations between adjacent sentences to model local coherence, inspired by the discourse structure theories of @xcite and @xcite . other traditional methods include word cooccurrence based local models @xcite , topic based global models @xcite , and syntax based local and global models @xcite . despite continuous research efforts in developing novel coherence models, their usefulness in downstream applications has largely been ignored. they have been evaluated in mainly two ways. the most common approach has been to evaluate them on synthetic discrimination tasks that involve identifying the right order of the sentences at the local and global levels @xcite @xcite . the other (rather infrequent) way has been to assess the impact of coherence score as an additional feature in downstream tasks like readability assessment and essay scoring @xcite . but since the concept of coherence goes beyond these constrained tasks and domains, so should the models. given the recent advances in neural nlp methods, with claims of reaching human parity in machine translation @xcite , fluency in summarization @xcite , or context-consistent response generation @xcite , coherence modeling of machine-generated texts, par-ticularly at a document-level, is now more crucial than ever @xcite . traditional task-specific evaluation methods (e.g., bleu, rouge) may not be an accurate reflection of their real-world performance in terms of readability @xcite . however, it is unclear if existing coherence models are capable of this task, since their performance on downstream applications is rarely studied, even though that is one of the main motivations for their development. our main goal in this work is to assess the performance of the existing coherence models not only on standard, challenging synthetic tasks like global and local discrimination, but more importantly on real downstream text generation problems. specifically, we investigate the performance of coherence models in three different settings: we show through experiments that there is only a slight correlation between model performances on synthetic tasks and the real-world use cases. although models perform strongly in the synthetic tasks, they show poor performance and low correlations with human judgments on distinguishing coherent machine translations and system-generated summaries from incoherent ones. they also fail to perform well on the next utterance ranking task, which is similar to the synthetic task of insertion @xcite , even if re-trained with task-specific data. however, we show that re-training the coherence models with task-specific data for machine translation evaluation leads to improved results and agreements with human judgments. this leads us to conclude that there is a possible mismatch in the task setting that is used to train coherence models. models trained on traditional synthetic tasks do not seem to be learning features that are useful for downstream applications. we hope that our results will motivate the broadening of the standard of coherence model evaluations to include more downstream tasks, and also motivate the redesigning of the training paradigm for coherence models. ## coherence models advancements in deep learning have inspired researchers to neuralize many of the traditional models. @xcite model syntax and intersentence relations using a recurrent sentence encoder followed by a fully-connected layer. in a follow-up work, li and jurafsky (2017) use generative models to incorporate global topic information with an encoder-decoder architecture. @xcite propose a neural entity grid model using convolutions over distributed representations of entity transitions. @xcite model change patterns of salient semantic information between sentences. @xcite propose a local discriminative model that retains the advantages of generative models and uses a smaller negative sampling space that can learn against incorrect orderings. @xcite propose a unified model that incorporates sentence syntax, inter-sentence coherence relations, and global topic structures in a single siamese framework. we benchmark the performance of five representative coherence models on the tasks discussed above. our selected models comprise of both traditional and neural models. moreover, two models are currently the state-of-the-art at the time of submission (transferable and unified neural model). entity grid (egrid). @xcite introduced the popular entity-based model for representing and assessing text coherence motivated by the centering theory @xcite . this model represents a text with a twodimensional array called an entity grid, that captures transitions of discourse entities across sentences. these local entity transitions are used as deciding patterns for text coherence; a local entity transition of length k is a sequence of {s,o,x,-} k representing grammatical roles (subject, object, other, and absent, respectively) played by an entity in k consecutive sentences. the salience of the entities, quantified by the occurrence frequency, is also incorporated to identify transitions of important entities. @xcite improve the basic entity grid by including non-head nouns as entities (with the grammatical role x). instead of using a coreference resolver, they match the nouns to detect coreferent entities. in our work, we consider this version of the entity grid model. ## evaluation tasks and experiments in this section, we present the performance of the coherence models on standard synthetic tasks (i.e., global/local discrimination), followed by the experiments where we apply the coherence models trained on the global discrimination task to three downstream tasks (i.e.,, abstractive summarization, extractive summarization, and machine translation). we then present the results of the coherence models re-trained on the next utterance ranking task. for each of the coherence models, we conducted experiments with publicly available codes from the respective authors. the three recent methods sections # doc. # pairs train 00-13 1,378 26,422 test 14-24 1,053 20,411 ## task-specific training for mt as a special use case, we report the results of retraining the coherence models using machine translation data for coherence evaluation. the aim is to investigate whether changing the usual training setup, that uses negative documents which are only small variations of the positive documents, might help coherence models learn more useful task-specific features. setup. under the assumption that the reference translations are more coherent at the document level than the system translations, we train the coherence models with the reference text as the positive and the system translation as the negative document, forming a positive-negative document pair. we use the data from wmt-2011 to wmt-2015 for training (28,985 document-pairs), wmt-2016 for development (7,647 document-pairs) and the same test data (wmt-2017 to @xcite and study data as used for the previous experiment ( §3.2.1). results. ## discussion compared to the downstream tasks of coherence evaluation in mt and extractive and abstractive summarization, the traditional global discrimination task can be considered to be a simpler task on the next utterance ranking task, the models fail to generalize and perform quite poorly despite task-specific re-training. the best model performance for the synthetic task of insertion, which is similar, also barely reaches 26% @xcite . this indicates that the training procedures may not be providing the right setting to learn features that are generic enough to apply to tasks in a harder setup. in the synthetic tasks, the models' selfsupervision comes from distinguishing an original coherent document from its incoherent renderings generated by random permutations of its sentences. this permutation-based self-supervision tries to capture document-level language properties. however, it is quite likely that this is simply a poor approximation of real-world coherence problems. consider for example that mt systems mostly translate at the sentence-level. consecutive sentences may lack coherence, but if two system translations of a text are compared, the translations themselves will be in the same order for both. the coherence models are not trained for such (realworld) settings. another possibility is that outputs from downstream tasks have different error distributions that are captured to varying degrees by different models, since they are originally designed based on synthetic tasks. that is, models that perform very well on the permutation task might be overfitting on this task, and therefore failing to find coherence issues that are more subtle than shuffled text. thus, we conclude that the current self-supervision for coherence modeling is not suitable for downstream coherence problems. when re-trained on machine translation data, most of the model performances improve, implying that a different training setting may be required to make the models applicable to actual downstream tasks. this is not apparent from the evaluation results that are usually reported, which show performances crossing the 90% mark. elsner and charniak (2011a) show a similar lack of generalizability and applicability of coherence models to the downstream task of chat disentanglement. our results suggest that despite nearly a decade of research since, the standard training and testing paradigm for coherence modeling continues to be inadequate in its capability to generalize to real-world use-cases and even to similar task settings, and also fails in being indicative of realworld task performance. ## conclusions we benchmark the performance of representative traditional and neural coherence models on standard synthetic discrimination tasks, and contrast this with their performance on various downstream application tasks in nlp. we show that higher accuracies on synthetic tasks do not translate into better performance on downstream tasks. we demonstrate this for real-world tasks like mt and summarization coherence evaluation, and next utterance ranking. our results signal a need for change in the way coherence models are typically trained and evaluated. other downstream applications like coherence evaluation of language model generated text and tasks such as chat disentanglement are also good candidates for testing coherence models. it would be worthwhile to build a coherence testset that is independent of the training tasks and similar to downstream applications, which could be used by the community to test the generalization ability of their models. in future work, we also hope to investigate the possible training scenarios that will result in more generalizable coherence models which can be used for evaluating downstream tasks.