D isco S core: Evaluating Text Generation with BERT and Discourse Coherence (2023)

Abstract: Recently, there has been a growing interest in designing text generation systems from a discourse coherence perspective, e.g., modeling the interdependence between sentences. Still, recent BERT-based evaluation metrics are weak in recognizing coherence, and thus are not reliable in a way to spot the discourse-level improvements of those text generation systems. In this work, we introduce DiscoScore, a parametrized discourse metric, which uses BERT to model discourse coherence from different perspectives, driven by Centering theory. Our experiments encompass 16 non-discourse and discourse metrics, including DiscoScore and popular coherence models, evaluated on summarization and document-level machine translation (MT). We find that (i) the majority of BERT-based metrics correlate much worse with human rated coherence than early discourse metrics, invented a decade ago; (ii) the recent state-of-the-art BARTScore is weak when operated at system level—which is particularly problematic as systems are typically compared in this manner. DiscoScore, in contrast, achieves strong system-level correlation with human ratings, not only in coherence but also in factual consistency and other aspects, and surpasses BARTScore by over 10 correlation points on average. Further, aiming to understand DiscoScore, we provide justifications to the importance of discourse coherence for evaluation metrics, and explain the superiority of one variant over another. Our code is available at https://github.com/AIPHES/DiscoScore .

## introduction in discourse, coherence refers to the continuity of semantics in text. often, discourse relations and lexical cohesion devices, such as repetition and coreference, are employed to connect text spans, aiming to ensure text coherence. popular theories in the linguistics community on discourse were pro-vided by @xcite and @xcite . they formulate coherence through the lens of readers' focus of attention, and rhetorical discourse structures over sentences. later on, coherence models as computational approaches of these theories emerged to judge text coherence in discourse tasks such as sentence ordering and essay scoring @xcite @xcite . while humans also often use text planning at discourse level prior to writing and speaking, up until recently, the majority of natural language generation (nlg) systems, be it text summarization or document-level mt, has performed sequential word prediction without considering text coherence. for instance, mt systems mostly do not model the interdependence between sentences and translate a document at sentence level, and thus produce many incoherent elements such as coreference mistakes in system outputs @xcite . only more recently has there been a surge of interest towards discourse based summarization and mt systems, aiming to model inter-sentence context, with a focus on pronominal anaphora @xcite and discouse relations @xcite . however, there appears a mismatch between discourse based nlg systems and non-discourse nlg evaluation metrics such as moverscore @xcite and bertscore @xcite which have recently become popular for mt and summarization evaluation. as these metrics base their judgment on semantic similarity (and lexical overlap @xcite ) between hypotheses and references-which by design does not target text coherence-it is not surprising that they do not correlate well with human rated coherence @xcite @xcite . recently, bartscore @xcite receives increasingly attention, which uses sequence-tosequence language models to measure the likeli-chelsea have made an offer for fc tokyo forward yoshinori muto. the 22year-old will join chelsea 's dutch partner club vitesse arnhem on loan next season if he completes a move to stamford bridge. chelsea signed a £200million sponsorship deal with japanese company yokohama rubber in february. ## our approach in the following, we elaborate on the two variants of discoscore, focusdiff and sentgraph, which we refer to as ds-focus and ds-sent. focus difference. in discourse, there have been many corpus-based studies towards modeling focus transitions over sentences, showing that focus transition patterns are indicative of text coherence @xcite . when reading a document, readers may have multiple focus of attention, each associated to a group of expressions: (i) referring expressions such as pronouns and (ii) semantically related elements such as @xcite . here, we assume two focus based conditions that a coherent hypothesis should meet in referencebased evaluation scenarios: in the following, we present focus modeling towards semantics and frequency, according to which we compare hypothesis with reference. for a hypothesis, we introduce a bipartite graph g @xmath0, s, a hyp ), where v and s are two sets of vertices corresponding to a set of foci and all tokens (per occurrence a word is a separate token) within a hypothesis. let @xmath1, 1} n×m be an adjacency matrix where n and m are the number of foci and tokens respectively, and a ij equals 1 if and only if the i-th focus associates to the j-th token. let f @xmath2. similarly, we use notation g ref , f ref and z ref for a human reference. we use contextualized encoders such as bert to produce token embeddings z hyp and z ref . we use a simple approach to model both semantics and frequency of a focus. that is, we assign per focus v an embedding by summing token embeddings that a focus is associated to: next, we measure the distance between a common set of foci ω in a hypothesis and reference pair based on their embeddings: sentence graph. few contextualized encoders can produce high-quality sentence embeddings in the document context, as they do not model interdependence between sentences. according to centering theory @xcite , two sentences are marked continuous in meaning when they share at least one focus, on the one hand; one marks a meaning shift for two sentences when no focus appears in common, on the other hand. from this, one can aggregate sentence embeddings for which corresponding sentences are considered continuous. in the following, we present a graph-based approach to do so. for a hypothesis foot_1 , let s @xmath3. we introduce a graph g @xmath4, a hyp ) where v is a set of sentences and a hyp is an adjacency matrix weighted according to the number of foci shared between sentences and the distance between sentences as listed below to depict two variants of a hyp : analyses by @xcite indicate that global statistics (e.g., average) over such adjacency matrices can distinguish incoherent from coherent text to some degree. here we depict adjacency matrices as a form of sentence connectivity derived from focus transitions over sentences. we use them to aggregate sentence embeddings from hypothesis and from reference: next, we derive per graph an embedding with simple statistics from ŝhyp and ŝref , i.e., the concatenation of mean-max-min-sum embeddings. finally, we compute the cosine similarity between two graph-level embeddings: choice of focus. in discourse, often four popular choices are used to describe a focus: (i) a noun that heads a np @xcite , (ii) a noun @xcite , (iii) a coreferent entity associated with a set of referring expressions @xcite and (iv) a semantic entity associated with a set of lexical related words @xcite . in this work, we investigate two focus choices: noun (nn) and semantic entity (entity). linguistically speaking, the latter is a lexical cohesion device in the form of repetition. from this, nn as focus yields few useful coherence signals but a lot of noise, while entity as focus uses 'signal compression' by means of aggregation to produce better signals. to produce entities, we first extract all nouns in hypothesis (or reference), and aggregate them into different semantic entities if their cosine similarities based on dep2vec word embeddings @xcite is greater than a threshold-assuming that nouns with high similarity refer to the same semantic entity. ## results we first examine the importance of discourse for evaluation metrics-which underpins the usefulness of discourse metrics, and then benchmark dis-coscore on summarization and mt datasets. importance of discourse. ds-focus and ds-sent concern the modeling of discourse coherence on two different levels: (i) the occurrences of foci, and (ii) the interdependence between sentences driven by focus transitions, both reflecting the discourse characteristics of a text. in the following, we describe these discourse features, and examine their importance for assessing system outputs by contrasting the discourse patterns of hypothesis and reference. figure 2 shows that the scales on freq(ref) and freq(hyp) in summarization differ by a large amount, i.e., from 0.5 to 2.5 on y-axis and up to 6 on x-axis. this means that hypothesis and reference can be strongly distinguished by freq(x), which underpins the usefulness of including such discourse signals in the assessment of system outputs when references are available. further, the larger scale on freq(hyp) indicates that foci in hypothesis are more repetitive than in reference, as a result of needless repetition in poor summariesin line with previous studies on incoherent machine translations @xcite . the results for other discourse features are similar, we provide them in figure 6 (appendix). overall, these results show discourse features can separate hypothesis from reference. ## conclusions given the recent growth in discourse based nlg systems, evaluation metrics targeting the assessment of text coherence are essential next steps for properly tracking the progress of these systems. although there have been several attempts made towards discourse metrics, they all do not leverage strong contextualized encoders which have been held responsible for the recent success story of nlp. in this work, we introduced discoscore that uses bert to model discourse coherence from two perspectives of readers' focus: (i) frequencies and semantics of foci and (ii) focus transitions over sentences used to predict interdependence between sentences. we find that bert-based non-discourse metrics cannot address text coherence, even much worse than early feature-based discourse metrics invented a decade ago. we also find that the recent state-of-the-art bartscore correlates weakly with human ratings on system level. discoscore, on the other hand, performs consistently best in both single-and multi-reference settings, equally strong in coherence and several other aspects such as factual consistency, despite that we have not targeted them. more prominently, we provide understanding on the importance of discourse for evaluation metrics, and explain the superiority of one metric over another with simple features, in line with recent work on explainability for evaluation metrics @xcite . scope for future research is huge, e.g., developing reference-free discourse metrics comparing source text to hypothesis, improving discourse metrics on fine-grained levels foot_2 , and ranking nlg systems via discourse metrics and rigorous approaches @xcite .