HOLMS : Alternative Summary Evaluation with Large Language Models (2020)

Abstract: Efficient document summarization requires evaluation measures that can not only rank a set of systems based on an average score, but also highlight which individual summary is better than another. However, despite the very active research on summarization approaches, few works have proposed new evaluation measures in the recent years. The standard measures relied upon for the development of summarization systems are most often ROUGE and BLEU which, despite being efficient in overall system ranking, remain lexical in nature and have a limited potential when it comes to training neural networks. In this paper, we present a new hybrid evaluation measure for summarization, called HOLMS, that combines both language models pre-trained on large corpora and lexical similarity measures. Through several experiments, we show that HOLMS outperforms ROUGE and BLEU substantially in its correlation with human judgments on several extractive summarization datasets for both linguistic quality and pyramid scores.

## introduction generating human readable summaries of textual documents is posed to remain a key technology in our information era. whether summarizing news, scientific articles, encyclopedias, or social media posts, the demand for a faster consumption of the most relevant information is expected to grow hand in hand with the amount of information available online. a current bottleneck to a wider adoption of automatic summarizers is the lack of efficient solutions addressing both the relevance of the generated summaries and their linguistic quality. one component of this current limitation is the lack of efficient evaluation measures that address both aspects. the two most cited and widely adopted measures in various summarization datasets and summarization challenges are rouge @xcite and bleu @xcite . while both measures have been shown to be highly efficient baselines in ranking summarization systems based on their average score @xcite @xcite @xcite , fewer studies have examined their relevance for individual summary ranking and their adequacy with linguistic quality and fluency. with the new levels of performance achieved by neural language models in a variety of natural language processing tasks, several insights point towards the high potential of their contextual language encoding for language representation. most of the state-of-the-art models such as t5 @xcite , bert @xcite , gpt @xcite and the universal sentence encoder @xcite , are built from very large corpora, which reduces substantially the potential bias from using them to evaluate summaries in a restricted document set or benchmark. on the other hand, the shallow lexical features of the original texts play a key role in extractive summarization and in pointer-generator approaches. it can also be argued that lexical similarities with gold summaries are implicitly capturing an important portion of the relevant semantics, especially at high similarity values. in this paper, we propose a new evaluation measure for summarization, called holms, relying on both contextual neural representations and lexical similarities. to capture the salient indicators from each side more efficiently, holms relies on a multi-dimensional gaussian function combining a sequential similarity measure based on neural embeddings and rouge. we motivate and present each component in more details in section 3. we evaluate holms on five different summarization datasets by computing its correlation with human judgements for content relevance and linguistic quality, and show that holms has higher pearson, spearman and kendall correlations than state-of-the-art measures on both aspects. in the next section, we discuss related works on summarization evaluation. we present holms in details in section 3 and our experiments in section 4. finally we discuss the results in section 5 before concluding. ## related work summary evaluation was studied from several perspectives, including the similarity of the candidate summaries with reference human summaries, their intrinsic linguistic quality and coherence, and their relevance w.r.t. the original document @xcite . in this paper, we focus on extrinsic evaluations when human generated summaries are available as they offer both more specific parameters for the task, and available benchmarks with human-generated scores for automatically generated summaries. 1 , we present several summarization evaluation measures and their main characteristics. in terms of usage, rouge and bleu stand out as the most cited measures, albeit a big portion of bleu citations are likely from language translation papers, and not from research works on summarization. rouge stands for recall-oriented understudy of gisting evaluation @xcite . it allows evaluating system-generated summaries by comparing them with (ideal) summaries created by humans. it relies on computing the ratio of overlapping units between the two summaries and has several variants according to the unit type: e.g., unigrams, n-grams or skip-grams. bleu was designed for the evaluation of language translation systems @xcite . it is a precision metric that counts a unigram as correct if it occurs in a reference translation and tackles redundancies by clipping the maximum number that a candidate word can be matched to its maximum number of occurrences in the gold/reference translation. other metrics have been proposed, mostly in the first decade of the twenty first century, in the context of the duc and tac challenges, with a few recent exceptions. for instance, rouge 2.0 was proposed in 2018 as an extension of rouge that relies on wordnet synonyms @xcite and main topics in addition to the original tokens. autosummeng is a lexical method relying on computing similarities between the n-gram graphs of candidate summaries and gold summaries. edges represent proximity between two n-grams and are weighted by the number of co-occurrences of their vertices (connected n-grams) in a specified window of text. one of the closest approaches to our work considered applying the rouge metric using word embeddings @xcite . instead of computing word matching in a binary fashion as in rouge, the authors consider a word or n-gram similarity to be either 0 if the candidate word is out of vocabulary, or equal to the dot-product of their embeddings otherwise. while this approach made use of contextual embeddings, it still required the words to be present in both the gold summary and the candidate summary, which makes it subject to the same limitation of rouge, i.e., the lack of semantic generalization that would allow matching synonyms and paraphrases. results-wise, their approach was tested only on one dataset, and under-performed substantially rouge-su4 and other rouge variants in terms of pearson's correlation. in contrast, holms does not restrict the coverage of neural representations with lexical constraints, but explores new associative ways to get the best of both worlds. the embeddings component of holms also relies on a sequential similarity function that grants increasingly more weight to (matching) sub-sequences with higher embeddings similarity. as far as we know, holms is the first summarization evaluation measure that (a) takes into account embeddings similarity with a sequential perspective, and (b) uses a gaussian function to combine lexical and embeddings-based similarities. ## holms holms stands for hybrid lexical and model-based evaluation of summaries. it relies on both deep contextual embeddings and lexical similarities. the embeddings-based similarity (es) used in holms is computed in a sequential method. the intuition behind es is that embeddings similarity between a word in a system summary and a word in a reference summary should be more pronounced if the contiguous words also have high embeddings similarity. such perspective can also be extended to consecutive n-grams to capture conceptual similarity between sequences. concretely, to compute es, a first step is to transform the input texts from both the gold and system summaries into two sets of consecutive n-grams: @xmath0, . . . , g l } for the gold summary, and @xmath1, . . . , a m } for a system summary. an embedding vector v l (x) is then generated for each n-gram x from a given language model l. a second step is to compute the best distance value for a given gold n-gram g i , dist a (g i ), with a filtering method where each system n-gram @xmath2. with euc being the euclidean distance, and a k the dynamic set computed by removing the previously matched n-grams in a. the last step is inspired from textflow @xcite , and consists in using the (consecutive) distance values as coordinates on a curve and computing the area under curve as the overall distance. this is best shown with an example as in figure 1. the two example sentences, s, and g are from a news article foot_1 , and were slightly modified for ease of presentation: drawing the curve connecting the distance values as coordinates allows adding more weight to highly matching sub-sequences and less weight to weakly matched sub-sequences. the final value of es is then computed as the complementary of the area under curve normalized by |g| × m ax euc to obtain values in the [0, 1] range. the lexical similarity used in holms is the rouge-1 recall. there is no restriction on the lexical similarity measure, or ensemble of measures, that could be used for the lexical component of holms. we picked rouge-1 for our first proof-of-concept as it has shown a good potential in ranking systems based on their average performance and is also widely used in the community. for the remainder of the paper, we will refer to rouge-1, rouge-2 ,and rouge-su4 as r 1 , r 2 , and r su 4 . the combination of both aspects (lexical and neural) needs to take into account the implicit and relative links existing between shallow lexical similarities and embeddings-based similarity. to this effect, we build holms using a bound three-dimensional gaussian function that highlights further the summary pairs on which both measures agree, by promoting or downgrading exponentially strong agreements and strong disagreements. the function has its peak at 1 for perfect agreement on the quality of a summary and a low at 0 for total disagreement between the two measures (cf. figure 2 ). ## evaluation we used 4-grams as our units to compute the es and holms values. the peaks and spreads are set to 1. to compare holms with a hybrid baseline, we also compute the correlation results for linear, a simple equal-weighted linear combination of es and r 1 . we tested several variations of es using three different sources for neural embeddings: in our preliminary experiment on the tac 2011 dataset @xcite , we observed that both the bert embeddings and glove embeddings underperformed substantially the embeddings from the universal sentence encoder (use) by more than 51%. we experimented with both the cls class embeddings and the sum of the tokens embeddings from bert, and with different vector dimensions for glove. subsequently, we used only the universal sentence encoder for our extended experiments on the 5 datasets. in future works, we plan to test more recent language models such as t5 @xcite and scibert @xcite . we compute the correlations of holms with human judgments of automatic summaries on 5 datasets from the summarization benchmarks introduced in the duc and tac challenges from 2007 to 2011 @xcite @xcite @xcite . we use the three standard correlation measures: pearson correlation (p), spearman rank coefficient (s), and kendall's rank coefficient (k). each dataset consists of a set of source documents, a set of human-generated summaries for each document (called models in the challenges), and manual annotations of the relevance score for each candidate summary generated by the participating systems. the score for each candidate summary is computed as the average similarity between the system summary and the set of reference summaries. for content relevance, human assessors first selected the important content units (scus) then used the pyramid method @xcite to score automatically the system-generated summaries based on the scus. in addition to content relevance scoring, the assessors also annotated the system-summaries with linguistic quality scores ranging from 1 to 5 for the duc 2007, tac 2008, and tac 2011 datasets. several scores ranging from 1 to 5 were first assigned to assess the system summaries according to five questions or aspects: grammaticality, non-redundancy, referential clarity, focus, structure, and coherence. the final linguistic quality score is then obtained by averaging the answers/scores given to all sub-questions. further details about the data collection and annotation methods are described in the challenges overview papers and websites @xcite @xcite @xcite . in the duc and tac editions, the rouge measure achieved state-of-the-art performance on its correlation with human judgments for both content relevance and linguistic quality. however, most evaluations of the tac and duc tracks relied only on average system scores and did not focus on ranking individual summaries. tables 2 and 3 present the correlations of bleu, rouge, and holms, with the linguistic quality scores computed manually by the nist assessors 2 . tables 4 and 5 present the correlations with pyramid scores computed manually by nist assessors to measure the content relevance of the system summaries. common observations. bleu underperformed substantially all other measures in our experiments, except for the correlation on the average systems linguistic score in duc 2007, which is likely due to shorter summaries. this could be caused mostly by the design of bleu that was aimed at sentence-level machine translation. to capture only the relative performance of each summary and each system, we normalized the bleu scores by the maximum score obtained by a system/summary for a given document. this improved the results (presented in section 4) compared with the correlation of the raw bleu scores but was not enough to close the gap with the other evaluation measures. another common observation is the relatively low level of correlation of all measures on linguistic quality when compared to content relevance correlations (pyramid scores). this can be explained in part by a higher level of subjectivity for linguistic quality inducing a higher assessor bias. evaluation of the linguistic quality of individual summaries. holms outperformed the rouge variants, its embeddings-based component es and the linear baseline in terms of pearson correlation with a relative improvement ranging from 3% to 10.1%. the fact that holms led to an improvement over both of its components (es and r 1 ) and over the linear baseline suggests that: in terms of spearman's and kendall's correlation factors, the picture was slightly more nuanced, with es performing better on tac 2011, holms performing better on tac 2008, and rouge-2 performing better on duc2007. the macro-average is less prone to dataset-specific bias and showed that holms performed better than all other measures. ## conclusions we presented a new summarization evaluation measure, called holms , based on a sequential n-gram embeddings similarity and rouge. in our experiments on 5 summarization evaluation benchmarks, holms performed consistently better on average than its individual components, bleu, and a linear combination baseline. holms can also be used as a framework, as many more variations can be tested, including the use of consecutive skip-grams as the input instead of n-grams, and the combination of sentence level similarity and n/skip-gram similarity. moving forward, we think that summarization systems should be evaluated on a combination of discrete and contextual or semantic evaluation measures. such extension fits naturally in holms through the insertion of additional dimensions in its gaussian function. our results suggest that such combination is likely to bring a higher level of correlation with human assessments of both linguistic quality and content relevance.