Unlocking Structure Measuring: Introducing PDD , an Automatic Metric for Positional Discourse Coherence (2024)

Abstract: Recent large language models (LLMs) have shown remarkable performance in aligning generated text with user intentions across various tasks. When it comes to long-form text generation, there has been a growing interest in generation from a discourse coherence perspective.However, existing lexical or semantic metrics such as BLEU, ROUGE, BertScore cannot effectively capture the discourse coherence.The development of discourse-specific automatic evaluation methods for assessing the output of LLMs warrants greater focus and exploration. In this paper, we present a novel automatic metric designed to quantify the discourse divergence between two long-form articles.Extensive experiments on three datasets from representative domains demonstrate that our metric aligns more closely with human preferences and GPT-4 coherence evaluation, outperforming existing evaluation methods.

## introduction real-life texts often exhibit underlying structures. news articles, for instance, adhere to a specific narrative order, as illustrated in fig. 1 , employed by journalists to efficiently convey messages and improve reader experience. despite recent advances in generation of fluent text, @xcite demonstrate that transformer-based models struggle to effectively capture and learn the underlying latent transition structure of coherent text. consequently, generating structurally coherent text remains an under-explored area of research. follow the theory of functional discourse structure, elaborated in appendix a.1, we leverage the discourse structure to model the coherence of long-form texts. several recent works @xcite have addressed the problems of generating long-form text while following specific indomain discourse schema. the united kingdom envisions a futuristic landscape in the year 2100, featuring state-of-the-art spaceports and innovative sky farms as part of its evolving infrastructure. the integration of spaceports and sky farms into britain's infrastructure in 2100 is expected to revolutionize the country's economy and environmental sustainability, leading to increased job opportunities, advanced agricultural practices, and reduced carbon emissions. the united kingdom has been steadily investing in research and development for space exploration and vertical farming in the years leading up to 2100, laying the foundation for the implementation of spaceports and sky farms in the future. looking back at the history of british innovation, it is evident that the united kingdom has a rich legacy of pioneering breakthroughs, from the industrial revolution to the modern computing era. while established automatic metrics such as bleu @xcite , rouge-l @xcite and bertscore @xcite exist for natural language generation evaluation, they predominantly measure lexical n-gram overlaps or semantic similarities. the evaluation of structural coherence has been a long-existing challenge @xcite @xcite . a common baseline metric for measuring functional discourse structure is the exact match, which compares structure elements one-to-one at each exact position. however, this metric is notably sensitive to local variations and differences in the lengths of articles. to address this gap, we propose a novel automatic model-free metric, positional discourse divergence (pdd), specifically designed to evaluate the underlying discourse structure of articles in comparison to references. pdd partitions the sentences of an article into multiple position bins and calculates the divergence in discourse structures within each bin. this approach renders pdd re-silient to various challenges encountered in longform text generation, such as accommodating local variations and handling misaligned numbers of sentences. to validate the effectiveness and generalizability of the pdd, we evaluate the inter-agreement with human evaluations and gpt-4 coherence evaluations on three representative datasets with different discourse schema: news discourse (choubey et al., 2020), long-form question answering @xcite and recipe1m+ @xcite . across all three domains, pdd demonstrates the highest agreement with human judgements on coherence. ## positional discourse divergence texts within a specific genre often exhibit similar patterns in their discourse sequences, albeit with some variations at a local level. in other words, the distribution of discourse roles is inherently tied to their approximate positions within the articles. for instance, news reports commonly present main events and their consequences at the beginning to capture the reader's interest, even though the precise order can differ. likewise, recipes tend to follow a predictable structure, where the preparation of ingredients is generally mentioned first, followed by cooking actions towards the middle or end of the text. despite the fluency achieved by (large) language models, they struggle to organize discourse structures like humans. in fig. 2 , we observe disparities between the discourse distributions of model predictions and human-written references when the news articles are divided into 5 positional bins. to quantitatively capture these gaps, we introduce the positional discourse divergence (pdd), denoted as d pos , as an automatic metric. equation 1 outlines the calculation for applying pdd to compare a predicted article against its corresponding reference: firstly, both articles are segmented into n positional bins. note the number of bin, n , should be smaller than the number of sentences in both the reference and the prediction. we denote p n (r) to represent the distribution of discourse role r for the reference in the n-th position bin, and q n (r) to represent the distribution for the generated article. these discourse distributions are calculated by the frequency density of the discourse roles within each bin. these discourse distributions are calculated by the frequency density of the discourse roles within each bin. then, for each bin n, the kl divergence between the discourse distributions is calculated. 2 to address the sparsity in the discourse distribution of a single article, small-value terms, denoted as ϵ, are introduced. this addition helps in avoiding instances of zero probabilities in the distribution. to compute pdd or other discourse measurements like exact match, it is inevitable to employ a discourse role classifier for labeling both prediction and reference articles. an off-the-shelf discourse classifier, trained on human-annotated data with a defined schema (e.g., the news discourse dataset for news domain), can serve this purpose. further information regarding the discourse classifiers is provided in appendix b. ## metric validation to validate the efficacy of the positional discourse divergence metric, we evaluate its agreement with human assessments, and gpt-4 on article coherence. additionally, we compare pdd against baseline automatic metrics, such as exact match, bleu, and bertscore. to assess generalizability, we conduct this validation across three different domains, each characterised by distinct human annotated, sentence-level discourse schemas: for further information regarding dataset details and schema definitions, please refer to appendix e. ## conclusion in conclusion, the exploration of text generation with natural underlying structure remains a significantly under-explored domain. addressing this gap, we introduced pdd, a simple and model-free metric designed to assess discourse structure. by quantifying the divergence between discourse distributions within position bins, pdd exhibits robust agreement with human and gpt-4 coherence evalu-ations across three representative domains, outperforming a range of baseline metrics. our hope is that pdd will stimulate future research endeavors focused on unraveling the intricacies of underlying structure in text generation.