C o UDA : Coherence Evaluation via Unified Data Augmentation (2024)

Abstract: Coherence evaluation aims to assess the organization and structure of a discourse, which remains challenging even in the era of large language models. Due to the scarcity of annotated data, data augmentation is commonly used for training coherence evaluation models. However, previous augmentations for this task primarily rely on heuristic rules, lacking designing criteria as guidance.In this paper, we take inspiration from linguistic theory of discourse structure, and propose a data augmentation framework named CoUDA. CoUDA breaks down discourse coherence into global and local aspects, and designs augmentation strategies for both aspects, respectively.Especially for local coherence, we propose a novel generative strategy for constructing augmentation samples, which involves post-pretraining a generative model and applying two controlling mechanisms to control the difficulty of generated samples. During inference, CoUDA also jointly evaluates both global and local aspects to comprehensively assess the overall coherence of a discourse.Extensive experiments in coherence evaluation show that, with only 233M parameters, CoUDA achieves state-of-the-art performance in both pointwise scoring and pairwise ranking tasks, even surpassing recent GPT-3.5 and GPT-4 based metrics.

## introduction coherence is a vital aspect of communication that evaluates the structure and organization of discourse @xcite . consequently, models capable of evaluating coherence of the given text are widely applicable in both discourse generation and assessment. while recent large language models show strong performance in various tasks @xcite , they have not presented superiority in coherence opening: adding regular exercise to your daily routine has many benefits. ## couda framework in this section, we introduce our couda framework, as illustrated in figure 2 . first, we use global and local augmentation to create negative samples that have relatively poor global coherence and local coherence, respectively. to be specific, we use sentence shuffling for global augmentation, and design a generative strategy for local augmentation. our generative strategy involves post-pretraining a generative model, and applying two controlling mechanisms to control the difficulty of generated samples. then we combine the constructed negative samples with the original discourses, which serves as positive samples, to train our metric model for coherence/incoherence classification. in inference phase, we utilize a unified scoring strategy to incorporate global and local coherence for overall assessment. ## results in this section, we show that couda framework achieves impressive coherence evaluation results on pointwise scoring and pairwise ranking tasks, even when compared with gpt-4 based models. we report average scores across 3 runs with different random seeds. ## comparison of augmentation methods in this section, we validate the advantage of our unified data augmentation strategy for coherence scoring over previous data augmentation strategies. compared data augmentation methods. coherence evaluation emphasizes the sentence structure and organization of a discourse. due to this special focus, data augmentation strategies designed for other tasks, e.g. eda @xcite , are not directly applicable. instead, we compare following data augmentation strategies for generating negative samples: 1) g: global augmentation via sentence shuffling @xcite , which is also adopted in our framework. global vs. local vs. unified. in generative vs. rule-based. further, we compare the result of generative augmentation vs. rulebased augmentation for modeling local coherence. first, metric model trained with l g outperforms that of l r by a large margin on both sample-level correlation (+3.8/+3.0/+3.0 in ρ, r, τ ) and datasetlevel correlation (+3.5/+1.5/+2.4 in ρ, r, τ ). second, when combined with global augmentation, g + l g yields significantly superior performance than g + l r . based on these two aspects, we can conclude that our generative strategy is more effective than rule-based methods. ## conclusion we propose a unified data augmentation framework called couda, with the designing criteria to unify both global and local aspects of coherence, as inspired by linguistic theory of discourse structure. this data framework includes global and local augmentation, a classification paradigm for training and a unified scoring strategy for inference. we specifically propose a novel generative augmentation strategy, which involves post-pretraining a generative model, and applying two controlling mechanisms to control the difficulty of generated samples. with only 233m parameters, our framework achieves remarkable improvement over previous methods, including gpt-4 based metrics.