Coherent or Not? Stressing a Neural Language Model for Discourse Coherence in Multiple Languages (2023)

Abstract: In this study, we investigate the capability of a Neural Language Model (NLM) to distinguish between coherent and incoherent text, where the latter has been artificially created to gradually undermine local coherence within text. While previous research on coherence assessment using NLMs has primarily focused on English, we extend our investigation to multiple languages. We employ a consistent evaluation framework to compare the performance of monolingual and multilingual models in both in-domain and out-domain settings. Additionally, we explore the model’s performance in a cross-language scenario.

## introduction coherence is a fundamental aspect of a wellorganized text and it can be defined as "a semantic property of discourse, based on the interpretation of each individual sentence relative to the interpretation of other sentences" @xcite . in order to be fully coherent, a discourse must exhibit both a local and a global coherence, where the former concerns mainly the relationships between adjacent or nearby sentences whereas the latter focuses on the discourse-level relations connecting remote sentences. modeling discourse coherence has a long history in the nlp community, particularly in the "pre-deep-learning" era, where a great deal of studies was inspired to the centering theory framework @xcite , such as the popular entity-grid approach for measuring local coherence @xcite . the long-standing interest for coherence modeling has been also motivated by the large variety of downstream applications which can benefit by measuring coherence in text, such as automatic essay scoring in language learning scenarios @xcite , language assessment in clinical settings @xcite , readability assess-ment @xcite . a further emerging scenario, which is closer to our study, is related to research on the interpretability of modern deep neural networks. in this respect, while the majority of existing tasks and benchmarks on which nlms are evaluated focuses on properties acquired from stand-alone sentences, their ability to model discourse and pragmatic phenomena is still unclear. few exceptions are represented by recent works such as @xcite @xcite , which introduced dedicated test suites aimed at measuring if neural sentence representations show sensitivity to discourse phenomena spanning across sentences. our paper intends to provide a novel contribution to the current body of literature by investigating whether and to what extent nlms in multiple languages are able to distinguish a coherent piece of text from an incoherent one, where the latter has been artificially created to undermine local coherence within text, at gradual levels of difficulty. while all previous work on coherence assessment using nlms has been focused on english, we probed these models for multiple languages using the same evaluation framework and compared the performance achieved by monolingual and multilingual models both in a in-domain and out-domain setting, as well as in a cross-language scenario. our contributions this paper makes the following contributions: i) we devised a new task to model discourse coherence understanding; ii) we compiled two new multilingual datasets (freely available) representative of two different domains and levels of complexity, containing coherent and incoherent passages (artifically manipulated); iii) we assessed how a multilingual nlm, xlm-roberta-base, performs over the task and compared the performance against the model without pretraining in order to measure the impact of pretraining on the task at hand; iv) we evaluated the task performance in a cross-domain and cross-prompt target coherence in 1998, an intense flare was observed. the star has also been a target of plans for interstellar travel such as project daedalus. in 2005, astronomers using data from the green bank telescope discovered a superbubble so large that it extends beyond the plane of the galaxy. it is called the ophiuchus superbubble. ## coherence evaluation framework we formulated the task of coherence modeling as a binary classification problem, that is: given a short piece of text (hereafter referred to as prompt) along with an individual sentence (the target), the model is asked to predict whether the target is contiguous or not, thus joining it to the prompt gives out a coherent or incoherent text. see by systematically manipulating the distance from the prompt we had intended to introduce incremental degrees of complexity in approaching the task, assuming that candidates closer to the prompt would pose a higher likelihood of being misleading. we tested our approach on the following languages: english, french, italian, portuguese, and spanish. ## results figure 1 reports the results of the multilingual model in the in-domain setting across languages for both written (top) and spoken (bottom) domain. as a general remark, we observe that the baseline model reports chance-level performance or even below across all task configurations and languages. this suggests that the knowledge acquired in the pre-training phase enables the model to capture information that is involved in local coherence. such an impact is particularly beneficial when the negative target is sourced from a different text, especially evident in the wikipedia data, where the pretrained model achieves an average f-score of 0.94 across all languages, compared to the 0.83 achieved on the ted data. conversely, although still performing better than the baseline, the model's performance significantly decreases when the negative target belongs to the same document of the prompt. this effect becomes more pronounced as the target gets closer to the prompt (either preceding or following it). this suggests that the model tends to rely more on explicit lexical clues to detect incoherent passages and may be more confounded when the target and the prompt share the same topic. this observation is particularly relevant in ted speeches, where clear topic distinctions are less prominent, and the discourse structure is less defined compared to encyclopedic written articles. in this case, the "easiest" scenario of out-domain negative targets becomes more challenging, indicating that the model struggles to grasp coherence-related cues beyond lexical or semantic ones. taken overall, these results highlight that the out-document negative items are extremely easy to detect, whereas similar scores are obtained despite the configuration and prompt-pair distance. based on this, we decided to conduct the cross-language and cross-domain experiments exclusively using the forward configuration, where the negative targets correspond the 10th sentence following the prompt. shifting our focus to the cross-domain classification ( ## conclusion in this study we carried out a comprehensive series of experiments to evaluate the ability of xlm-roberta base, one of the leading neural language models (nlms), in distinguishing coherent text from incoherent text, where the latter has been artificially created to gradually undermine local coherence within text. our findings indicate that nlms still face challenges in modeling discourse coherence, and the linguistic knowledge acquired during the pre-training phase provides limited assistance when coherence relies on information not directly related to the topic. as expected, the crossdomain experiments highlighted that the model performances degrade with respect to the in-domain classification scenario, particularly when tested on data with a less defined structure, such as ted talks. interestingly, the generalisation ability of the multilingual model holds across different languages, showing competitive results with the monolingual ones.