COHESENTIA : A Novel Benchmark of Incremental versus Holistic Assessment of Coherence in Generated Texts (2023)

Abstract: Coherence is a linguistic term that refers to the relations between small textual units (sentences, propositions), which make the text logically consistent and meaningful to the reader. With the advances of generative foundational models in NLP, there is a pressing need to automatically assess the human-perceived coherence of automatically generated texts. Up until now, little work has been done on explicitly assessing the coherence of generated texts and analyzing the factors contributing to (in)coherence. Previous work on the topic used other tasks, e.g., sentence reordering, as proxies of coherence, rather than approaching coherence detection heads on. In this paper, we introduce CoheSentia, a novel benchmark of human-perceived coherence of automatically generated texts. Our annotation protocol reflects two perspectives; one is global, assigning a single coherence score, and the other is incremental, scoring sentence by sentence. The incremental method produces an (in)coherence score for each text fragment and also pinpoints reasons for incoherence at that point. Our benchmark contains 500 automatically-generated and human-annotated paragraphs, each annotated in both methods, by multiple raters. Our analysis shows that the inter-annotator agreement in the incremental mode is higher than in the holistic alternative, and our experiments show that standard LMs fine-tuned for coherence detection show varied performance on the different factors contributing to (in)coherence. All in all, these models yield unsatisfactory performance, emphasizing the need for developing more reliable methods for coherence assessment.

## introduction coherence is an essential property of well-written texts that refers to the way different textual units relate to one another. thanks to these relations, the text appears to be logically consistent and semantically meaningful to human readers. coherence detection plays a pivotal role in nlp tasks or applications that are required to generate human-like extended texts, or otherwise measure the text quality; these include text summarization, story generation, long-form question-answering, and automatic essay scoring, to name just a few. in this era of large language models (llms), generating texts is at the forefront of nlp research, and the generation capacity of llms has an unprecedented celebrated success. however, using llm-generated texts blindly is not always a viable option since texts may ultimately be rendered incoherent, inconsistent, or both. these texts typically require human judgment regarding the meaningfulness and quality of the generated text. to reduce the substantial manual effort by humans, required for validating the quality of the text in order to use it, ideally an automatic model for coherence assessment will be made available. this would reduce the reliance on time-consuming and expensive human judgments, allowing for faster and more reproducible evaluations. however, no standard metric or automatic model currently exist or are readily available for this task. in the pre-neural era, a large body of work on models for coherence was inspired by linguistic theories. for instance, entity-based local models @xcite consider syntactic realization of entities in adjacent sentences, inspired by the centering theory @xcite . another line of work, inspired by discourse theories such as rhetorical structure theory (rst) @xcite , is using discourse relations between sentences to predict local coherence @xcite . other notable methods include word co-occurrence-based local models @xcite , content (or topic distribution)based global models @xcite , and syntax-based local and global models @xcite . due to the fact that properties of coherence remain ambiguous and hard to formalize, much nlp work on detecting coherence is done by creating models that are based on proxy tasks such as summarization, essay scoring @xcite and, most commonly, the sentence reordering task @xcite . such proxies, while related to coherence, do not necessarily capture coherence fully, resulting in coherence models that are sensitive to task-specific artefacts and can be misleading. so far, to the best of our knowledge, the only dataset annotated explicitly for humanrated coherence is gcdc @xcite which contains coherence scores for real-world texts in four domains. however, with the increased use of automatically-generated text by e.g., the gpt models' family, a coherence scoring benchmark based on generated texts is called for. in this work, we set out to fill the gap and explicitly address the human-based coherence of automatically generated texts. concretely, we present cohesentia, a new benchmark for human assessment of automatically generated texts, which are generated by a generative llm, annotated by human workers for rating the coherence score of the artificially-generated text. crucially, instead of using a single holistic score of coherence for each text, we propose a novel incremental, sentence-bysentence, labeling scheme that manifests the conceived coherence of the text fragments, shedding light on both the specific factors that tend to break the coherence for human readers, and the specific points where the coherence broke. this benchmark and scheme provide insights into discourse coherence, allowing to dissect and investigate the specific factors contributing to coherence -towards advanced computational modeling of coherence. we compare the standard holistic coherence scoring with our incremental scheme and show that the incremental method does not only enable us to pinpoint the incoherence causes and positions where it breaks, but also, the global score obtained for via the incremental methods shows higher interannotator agreement than when using the holistic method. in addition, for llms fine-tuned on this task, we show that different size lms have similar f1 score, corroborating our assumption that while llms capture many important aspects they still lack the ability to assess coherence. we also assess the detection of different coherence factors by llms. their performance is, at best, varied. in sum, the contribution of this paper is threefold. first, we propose a novel coherence-annotation protocol that obtains higher agreement on human judgments than previous rating practices. second, we present a benchmark labeled with this scheme, which is to the best of our knowledge the first benchmark for assessing the quality of models for detecting coherence of generated texts. finally, we provide a first glimpse into the coherence factors that can be learned or identified by contemporary llms, versus factors that are harder to detect. ## data creation and curation the purpose of creating the coherence benchmark was twofold. first, due to the increased use of encoder-decoder models for generating text automatically, we aim to assess the coherence of such generated texts, as these texts are often considered with high quality and are used blindly but in reality may lack coherence. second, we aimed to enable a comparison between human annotations for coherence scoring, when employing holistic versus incremental annotation approaches, in order to provide an optimal validated protocol for future use. the stages of benchmark creation have been as follows: ## results figure 3 shows the results of the classificationbased and generation-based architectures for the different models, for both the holistic protocol and the incremental protocol. we can see that the results for all methods are relatively low, thus corroborating our hypothesis that while such models learn important text features, coherence scoring contains intricate features that are much harder for such a model to capture. interestingly, the outcomes for larger llms exhibit only small improvements compared with their smaller counterparts, suggesting that both models have limitations in comprehending and identifying coherence, and that size and scaling are not major factors contributing to models' ability of capturing coherence. additionally, results on the incremental data are higher than the ones from the holistic method, which suggests that the model learns better representations of coherence from the incremental coherence scoring, possibly due decomposing the large context to the shorter contexts included in the task. interestingly, when we finetuned gpt-3, the model used to generate the text in the first place, on coherence scoring, the results were not superior to the other models. ## conclusion this work addresses the foundation of automatic assessment of discourse coherence, by presenting a new benchmark, cohesentia, containing humanrated generated stories and presented alongside a linguistically-motivated taxonomy of possible reasons for incoherence. we further present two annotation protocols, holistic versus incremental. through the incremental method, we get a better inter-annotator agreement, as well as valuable insights into the fundamental components that contribute to coherence and key challenges of generated text. consequently, we are better equipped to explore more advanced methods for generating texts in a more refined manner, ultimately leading to more coherent generated texts. we make this benchmark, annotation guidelines, and code, publicly available, to encourage future work on the topic of automatic coherence detection. 3