How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation (2022)

Abstract: Automatically evaluating the coherence of summaries is of great significance both to enable cost-efficient summarizer evaluation and as a tool for improving coherence by selecting high-scoring candidate summaries. While many different approaches have been suggested to model summary coherence, they are often evaluated using disparate datasets and metrics. This makes it difficult to understand their relative performance and identify ways forward towards better summary coherence modelling. In this work, we conduct a large-scale investigation of various methods for summary coherence modelling on an even playing field. Additionally, we introduce two novel analysis measures, _intra-system correlation_ and _bias matrices_, that help identify biases in coherence measures and provide robustness against system-level confounders. While none of the currently available automatic coherence measures are able to assign reliable coherence scores to system summaries across all evaluation metrics, large-scale language models fine-tuned on self-supervised tasks show promising results, as long as fine-tuning takes into account that they need to generalize across different summary lengths.

## introduction automatically generated summaries should not only be informative, but also well-written and coherent. while informativeness is routinely evaluated automatically with rouge @xcite , there is no agreement on how to evaluate summary coherence. however, automatic evaluation is highly desirable to reduce evaluation costs and as a tool for improving summarizer output, e.g. as reranker. many coherence measures (cms) have been suggested for automatically assigning a coherence score to a summary, including learning from human judgements @xcite @xcite , learning from the shuffle task @xcite , where models are trained to discriminate original documents from documents with randomized sentence order @xcite , using next sentence prediction as a proxy task @xcite , and finally unsupervised measures that exploit heuristics @xcite or large-scale lms @xcite . cm performance is then evaluated by comparing the automatic scores to human coherence scores on a set of system summaries. however, this evaluation is often conducted on disparate datasets. it also often uses system outputs from duc conferences @xcite @xcite , which do not necessarily represent recent advances in text summarizers. in addition, there is no agreement on how the cm scores should be compared to human scores. system-level correlation @xcite , pairwise ranking accuracy @xcite @xcite and summary-level correlation @xcite have all been suggested as evaluation metrics (ems). this makes it hard to ascertain the state of summary coherence modelling and to identify promising directions for future research. we attack this problem by making the following contributions: using these insights, we conduct a large-scale comparison of cms on the recent summeval dataset @xcite . we find that: code and data for our experiments are available at https://github.com/julmaxi/ summary_coherence_evaluation . ## dataset we evaluate cms on the expert annotations in the summeval dataset 2 (fabbri et al., 2021) which is, to the best of our knowledge, the largest dataset that includes such coherence annotations for a variety of state-of-the-art summarizers. it contains annotations on a 1-5 scale for outputs of 17 systems for 100 documents from the cnn/dm dataset @xcite by three annotators each. figure 1 highlights two important properties. firstly, there is a large gap in average performance between different summarizers, and secondly, most summarizers exhibit considerable variance in scores. ## evaluating cms cm performance is typically assessed on a set of summaries generated on document set d by a set of summarizers s using the agreement of predicted scores @xmath0,s) |@xmath1, @xmath2,s) |@xmath3, @xmath4. however, this agreement can be computed in different ways. we identify the following common ems: system-level correlation τ sys assesses cm performance by correlating the mean human and mean cm scores of the individual summarizers. pairwise accuracy acc pair assesses cm performance by comparing scores on outputs of two different systems on the same document. ## coherence measures we identify the following families of reference-free cms for summarization and include representatives of each in our study: supervised cms trained on human coherence ratings of summaries, selfsupervised cms trained on the shuffle task and unsupervised cms. for the supervised setting, we investigate measures trained on data from duc03 @xcite as well as the duc05-07 dataset used by @xcite . while the duc03 dataset is set up as a pairwise ranking dataset, the duc05-07 dataset is used in a regression setting. the extended entity grid (eeg) @xcite is an extension of the entity grid of @xcite . it represents texts using occurrence patterns of the mentioned entities across sentences. the model uses a generative approach that models the probability of an entity appearing in a specific role in a sentence, given its eeg egr neg unf gra ccl sqe gru bas unsupervised role in the two preceding sentences. the entity graph (egr) @xcite constructs a sentence graph of a document by identifying entity overlap between sentences. two sentences are connected if they share at least one entity, with edge weights decreasing when they are further apart. the score of a document is the average outdegree of sentences, with higher outdegree indicating better coherence. the neural entity grid (neg) (tien nguyen and joty, 2017) applies a convolutional network to the entity grid. the model is trained on a pairwise ranking loss. the unified model (unf) (moon et al., 2019) is a lexical cm that uses a convolutional network to build sentence representations from raw text. the model uses an adapted version of the ranking loss for the shuffle task that is computed only for three sentence windows in which shuffled and original documents differ. we use the model based on elmo @xcite , as it performs best in the original paper. the graph-based neural coherence model (gra) @xcite is a recent cm that combines entity-based representation with lexical information in a graph nn. like the previous two models, it employs a pairwise ranking loss. recently, @xcite have shown that a roberta-based @xcite classifier can easily achieve near-perfect results on the shuffling task on wsj. however, they did not test whether this model can predict coherence on non-artificial tasks. we thus include a simple roberta model that is trained to classify shuffled vs. unshuffled summaries, naming it coherence classifier (ccl). 6 sumqe (sqe) @xcite predicts five linguistic quality scores via multi-head regression on human scores. we use the coherence head of the model trained on all three duc datasets. foot_4 gruen (grn) (zhu and bhat, 2020) is a collection of unsupervised measures for linguistic quality that combines multiple unsupervised heuristics. bartscore (bas) @xcite uses the probability of a summary under a pre-trained bart model as a score. we use the variant fine-tuned on cnn/dm summaries in the source-to-summary configuration, as suggested by the authors. finally, we include an upper and lower bound: rnd assigns each summary a uniformly chosen score between 0 and 1. for hum, we use the summ-eval human annotations and select the annotator with the worst overall correlation to the remaining annotators and use their scores as predictions. 8 we train all shuffling models on the wsj corpus of newswire articles, which is frequently used in coherence modelling @xcite @xcite . we also train models using the same technique on reference summaries from the train portion of cnn/dm. for eeg we also estimate model parameters on both datasets. for wsj, we follow the original implementations regarding the number of shuffled samples. for cnn/dm, we only use a single shuffled instance per summary, as it is larger by two orders of magnitude (wsj: 1,400; cnn/dm: 287,113 documents before shuffling). detailed accounts of our experiments with each cm are found in appendix a. ## results we present the correlation of all cms with human coherence ratings in focusing on τ sys first, we find that ccl, bas, grn and to a lesser extent sqe achieve relatively qe/bert_duc_all_q5_multi%20task-5.h5 8 we note that unlike automatic measures, humans may only differentiate among five classes. we might thus underestimate actual human performance. metric τintra τpair τsum τsys acc.pair hum +0.75 (+0.70 +0.79) +0.81 (0.76, 0.85) +0.81 (+0.77 +0.84) +0.91 (+0.71 +1.00) +0.77 (+0.71 +0.81) rnd -0.00 (-0.06 +0.05) -0.00 (-0.07 +0.06) +0.00 (-0.05 +0.05) +0.09 (-0.41 +0.53) +0.50 (+0.46 +0.54) egr -0.04 (-0.12 +0.04) -0.11 (-0.19, -0.02) -0.09 (-0.16 -0.01) -0.25 (-0.59 +0.10) +0.40 (+0.36 +0.44) eeg c/d +0.02 (-0.07 +0.10) +0.04 (-0.10 +0.18) +0.06 (-0.06 +0.17) -0.19 (-0.68 +0.26) +0.52 (+0.45 +0.59) eeg wsj +0.02 (-0.06 +0.10) +0.00 (-0.09 +0.11) +0.03 (-0.06 +0.11) -0.19 (-0.60 +0.26) +0.50 (+0.44 +0.55) neg c/d -0.07 (-0.14 -0.00) -0.05 (-0.14 +0.07) -0.06 (-0.15 +0.03) -0.15 (-0.61 +0.32) +0.47 (+0.42 +0.53) neg duc -0.08 (-0.16 +0.01) -0.06 (-0.18 +0.06) -0.07 (-0.17 +0.04) -0.06 (-0.49 +0.31) +0.47 (+0.40 +0.53) neg wsj -0.02 (-0.08 +0.05) -0.08 (-0.17 +0.00) -0.07 (-0.15 +0.02) -0.43 (-0.69 -0.05) +0.45 (+0.41 +0.50) unf c/d +0.04 (-0.03 +0.11) +0.05 (-0.05 +0.14) +0.06 (-0.01 +0.13) +0.13 (-0.33 +0.59) +0.53 (+0.48 +0.57) unf wsj +0.02 (-0.05 +0.09) -0.11 (-0.26 +0.03) -0.04 (-0.15 +0.05) -0.09 (-0.51 +0.39) +0.44 (+0.36 +0.52) gra duc -0.04 (-0.12 +0.03) -0.05 (-0.16 +0.03) -0.06 (-0.13 +0.01) -0.19 (-0.65 +0.25) +0.47 (+0.43 +0.52) gra c/d +0.08 (+0.02 +0.15) +0.09 (-0.02 +0.19) +0.11 (+0.01 +0.18) +0.37 (-0.07 +0.69) +0.55 (+0.49 +0.60) gra wsj +0.08 (+0.01 +0.15) -0.01 (-0.11 +0.10) +0.02 (-0.06 +0.12) -0.09 (-0.47 +0.37) +0.49 (+0.44 +0.55) ccl c/d +0.26 (+0.19 +0.33) +0.40 (+0.31 +0.49) +0.39 (+0.31 +0.44) +0.62 (+0.30 +0.86) +0.71 (+0.66 +0.76) ccl wsj +0.20 (+0.12 +0.26) +0.35 (+0.25 +0.46) +0.33 (+0.24 +0.41) +0.74 (+0.40 +0.92) +0.69 (+0.63 +0.74) bas +0.17 (+0.08 +0.26) +0.37 (+0.23 +0.51) +0.32 (+0.20 +0.42) +0.72 (+0.42 +0.89) +0.69 (+0.62 +0.77) grn +0.18 (+0.12 +0.25) +0.26 (+0.17 +0.35) +0.27 (+0.19 +0.34) +0.72 (+0.38 +0.89) +0.63 (+0.58 +0.69) sqe +0.19 (+0.13 +0.26) +0.26 (+0.15 +0.36) +0.24 (+0.15 +0.32) +0.51 (+0.05 +0.80) +0.64 (+0.58 +0.69) ## cm analysis of the cms that perform best on coherence evaluation (see for the entity-based measures egr, eeg and neg, their difficulties on the summeval dataset are also reflected in the shuffle task. this suggests that these cms struggle generally on cnn/dm-style summaries. in section 7.3 we demonstrate that this is due to the overall lack of entity overlap in this dataset. finally, unf c/d and gra are outliers in that they show shuffle performance on cnn/dm that is similar or better than sqe but still perform near random on summeval coherence modelling. we investigate this in section 7.4. ## conclusion we have investigated the performance of a wide array of cms for summary evaluation that have not been previously systematically compared. our investigations show that cms must be carefully evaluated in order to avoid rewarding the modelling of shallow, system-level confounders, that are unlikely to generalize. we thus recommend researchers report our newly suggested intra-system correlation alongside other ems and use bias matrices to understand unexpected drops when going from system-level to intra-system correlation. there is considerable need to improve cms before they become practical for summary coherence modelling. our results point towards the following lessons for future work. firstly, cnn/dm summaries are not amenable to entity-based analysis without considerable additional work to improve entity detection. secondly, self-supervised training via the shuffle task shows the greatest promise for future improvements. however, we show that good shuffle performance does not naturally transfer to coherence evaluation for settings with documents of different lengths and contents. training in a classification setup instead of the more common pairwise setup provides an effective fix for this. wanzheng zhu and suma bhat. 2020. gruen for evaluating linguistic quality of generated text. in findings of the association for computational linguistics: emnlp 2020, pages 94-108, online. association for computational linguistics.