A Novel Computational Modeling Foundation for Automatic Coherence Assessment (2025)

Abstract: Coherence is an essential property of well-written texts, that refers to the way textual units relate to one another. In the era of generative AI, coherence assessment is essential for many NLP tasks such as summarization, long-form question-answering, and more.Current NLP approaches for modeling coherence often rely on a proxy task, specifically, sentence reordering . However, such an approach may not capture the full range of factors contributing to coherence.To remedy this, in this work we employ the formal linguistic definition by Reinhart:1980 of what makes a discourse coherent, consisting of three conditions, cohesion, consistency and relevance , and formalize these conditions as respective computational tasks, which are in turn jointly trained. We evaluate this modeling approach on two human-rated coherence benchmarks: one of automatically-generated stories and one of real-world texts.Our experiments show that jointly training on the proposed tasks leads to better performance on each task compared with task-specific models, and to better performance on assessing coherence overall.Our proposed computational framework thus paves the way for a more advanced, broad-coverage coherence assessment.

## introduction the concept of coherence refers to the quality of textual flow, where sentences and paragraphs are logically connected, enabling a clear and understandable progression of ideas. coherence is critical for many nlp tasks like summarization and question answering, directly impacting output accuracy and naturalness @xcite . as generative large language models (llms) become more prominent, coherence assessment has become essential for evaluating and improving text quality @xcite @xcite . this study introduces a novel approach to coherence assessment by leveraging linguistic theory alongside established nlp tasks. developing a robust model capable of accurately evaluating text coherence has been challenging due to several reasons. first, the scarcity of large-scale datasets specifically designed for training coherence assessment models, which are very difficult to produce @xcite . additionally, existing methods for coherence detection rely on a proxy task, and have not been able to accurately capture the multifaceted nature of coherence @xcite . our goal is to address these challenges by developing a model that assesses coherence effectively. our vantage point is the theoretical foundation provided by reinhart's linguistic framework, which defines coherence through three essential conditions: cohesion, consistency, and relevance ( §2.2). guided by these principles, we designed our model to integrate five carefully chosen nlp tasks that reflect these conditions. specifically, the model incorporates sentence reordering @xcite , implicit discourse relation detection @xcite , natural language inference @xcite , np enrichment @xcite , and irrelevant-sentence detection ( §3). each task is supported by a large-scale dataset, and an associated multi-task model thereof enables capturing the multifaceted nature of coherence. we hypothesize that jointly fine-tuning a model on these tasks will produce a model that effectively captures the core properties of coherence as delineated by the theoretical framework ( §3). to test this, we develop a unified model jointly fine-tuned on the selected tasks that act proxies for the coherence conditions ( §4). we then evaluate the model's coherence assessment capabilities through an additional fine-tuning stage on coherence-specific tasks ( §5). we use two human-annotated benchmarks: the grammarly corpus of discourse coherence (gcdc) @xcite for realworld texts across domains, and cohesentia (maimon and tsarfaty, 2023) for synthetic texts. our results ( §6) demonstrate that models jointly trained on our selected coherence proxy tasks outperform models without such training, achieving new sota performances on various coherence assessment benchmarks. we also show that training on tasks unrelated to coherence offers little to no value for coherence evaluation, validating our task selection. finally, given shared knowledge across tasks, we expect the unified model to enhance individual proxy-task performances, and our experiments indeed confirm this for most of the tasks. all in all, our approach presents a promising direction for future research to enable the development of tools that not only detect (in)coherence but also offer actionable insights into its causes. by integrating coherence assessment into text generation, we envision significant improvements in the quality and coherence of machine-generated texts. ## research hypotheses and tasks to design the coherence assessment model, we start by mapping reinhart's coherence conditions to computational tasks, employing a minimal set of nlp tasks designed to capture the key features of cohesion, consistency, and relevance. our hypothesis is that a model jointly fine-tuned on all of these tasks will effectively learn to capture coherence features. additionally, we anticipate that this unified architecture, when further fine-tuned on coherence assessment tasks, will outperform models fine-tuned on those coherence assessment tasks directly. to validate these hypotheses, we define five tasks that reflect the coherence conditions: the sentence reordering (sro) task this task, proposed by @xcite , involves reordering shuffled sentences to restore their original coherent form. for example, given the following input: "(1) finally, the parser is evaluated. (2) we develop a useful parser. (3) then we present our parser. (4) we first describe the older one." the correct order is (2) → (4) → (3) → (1). a model excelling at paragraph reconstruction should capture syntactic and semantic relationships between sentences, reflecting both cohesion and consistency (cf. @xcite ). the implicit discourse-relation recognition (idrr) task given a pair of sentences (discourse units; dus), the aim is to predict the discourse relation between them, reflecting notions such as cause and effect, comparison, and contrast @xcite . for example, with the following input: "john worked all night. he slept all day today." the model is expected to detect a discourse relation reflecting contingency (e.g., 'so', 'hence'). the discourse relation identification task enhances the model's ability to connect sentences, addressing the second sub-condition of cohesion. the np enrichment (npe) task introduced by @xcite , the npe task identifies implicit prepositional links between noun phrase (np) entities. given two np mentions in a text, it determines the existence of a prepositional relation and identifies the best preposition describing it p(n p 1 n p 2 ) (or none is no relation exists). for example, in the paragraph: "[crown princess mary] of @xcite gives [birth] to a [male child]." @xcite there are 4 nps (marked by @xcite ) and thus 12 potential np pairs. two examples of valid prepositional pairs and their relations are (1) in(birth, denmark) and (2) of(birth, male child). a model trained on this task captures referential links between sentences, serving as a proxy for the referential-linking sub-condition of cohesion. the natural language inference (nli) task the nli task @xcite aims to determine the semantic relation between a premisehypothesis pair as entailment/contradiction/neutral. for example, given the premise: "john inspects the uniform of a figure in some east asian country." and the hypothesis: "john is sleeping." the output will be a contradiction. nli evaluates nlp models' ability to capture logical relationships between sentences, serving as a proxy for the consistency condition. ## joint models of coherence this section presents the proposed coherence models, which are based on the proxy tasks described in section 3. we introduce both classification-based (bert) and generation-based (t5) variants, along with the datasets and evaluation metrics used for assessment. ## conclusion and future work in this paper we propose a new coherence modeling method, based on reinhart (1980)'s theory, which reflects the conditions needed for coherence: cohesion, consistency, and relevance. we use five nlp tasks as proxies of these conditions, and train an mtl model on them jointly, in both classificationbased and generation-based architectures. in both cased, our unified coherence model achieves sota results on these individual tasks, and excels in coherence scoring for both natural and generated texts. we propose this framework to enhance nlp systems' ability to evaluate text quality automatically. future follow-up research will focus on using these conditions for improved coherent-text generation, and for automatically detecting particular causes of incoherence. our code and models are publicly available to encourage further research on broadcoverage coherence scoring and coherence reasoning.