RISE : Leveraging Retrieval Techniques for Summarization Evaluation (2023)

Abstract: Evaluating automatically-generated text summaries is a challenging task. While there have been many interesting approaches, they still fall short of human evaluations. We present RISE, a new approach for evaluating summaries by leveraging techniques from information retrieval. RISE is first trained as a retrieval task using a dual-encoder retrieval setup, and can then be subsequently utilized for evaluating a generated summary given an input document, without gold reference summaries. RISE is especially well suited when working on new datasets where one may not have reference summaries available for evaluation. We conduct comprehensive experiments on the SummEval benchmark (Fabbri et al., 2021) and a long document summarization benchmark. The results show that RISE consistently achieves higher correlation with human evaluations compared to many past approaches to summarization evaluation. Furthermore, RISE also demonstrates data-efficiency and generalizability across languages.

## introduction summarization evaluation has became a topic of interest in recent years. in the past, many summarization approaches have relied on rouge @xcite for evaluating generated summaries. yet, as reported by @xcite , rouge and other automated metrics tend to fall short when compared to human evaluations. to overcome this, many new approaches have been developed leveraging pretrained language models, showing various degrees of success. we present our new approach to summarization evaluation called retrieval-inspired summarization evaluation (rise). as with recent approaches, rise leverages pre-trained language models. but unlike past approaches, we treat evaluation as a retrieval task, leveraging techniques from information retrieval. this is done by using a dual-encoder approach @xcite , feeding in the source document and the summary to be evaluated, in order to get a final score for evaluation. the benefits of rise are as follows: ## related work there has been a lot of work in recent years on evaluation of summarization. these approaches have been diverse, and there are different ways of categorizing them. for example, @xcite grouped evaluation metrics into four categories: matching, regressions, ranking and generation. in this case, rise falls under ranking. one of the idealistic benefits of creating such a ranking approach is that while we need references during training of such a model, for evaluation we can focus on just the source document and the generated summary to be evaluated. ## model figure 1 shows how rise works. as previously mentioned, rise leverages pre-trained language models via a dual-encoder network. to do so, rise builds upon t5x retrieval foot_1 as the framework for the dual-encoder model. each encoder in the model is the encoder of a pre-trained t5 model @xcite . t5 is an encoder-decoder model, thus we are only using the encoder half of the model. to evaluate a summary, rise will feed in the source document d i into one of the encoders (the left encoder in figure 1 ), and the summary s i to be evaluated into the other encoder. both documents are encoded, and finally we take the dot product of the resultant output encodings in order to get the final score used for evaluation. to train the dual-encoder models, we apply contrastive learning and use the positive pairs of (document, summary). assume s + i is considered as a positive summary for document d i . during training, all other summaries in a batch are considered as negatives. the models are trained using an inbatch sampled softmax @xcite : the similarity scoring function sim is the cosine distance in our experiments foot_2 . at inference time, the similarity scores are used to estimate the quality of generated summaries. b is a mini-batch of examples and τ is the softmax temperature. during training, we could prepare additional negatives s - i for each document d i , and the loss can be computed as: ## results we use summeval @xcite to evaluate how well our approach correlates with human evaluations. summeval is a collection of human annotations on the quality of 16 models and their outputs for 100 examples from the cnn / daily mail task @xcite . as with past approaches, we focus on the annotations made by expert annotators, and use kendall's tau for systemlevel correlation. as described in their work, there are four criteria used for human judgements: ## conclusion we have presented our new model rise for evaluating text summaries. as the results show, rise has strong correlation with human evaluations. being a reference-free metric, it can be used in new domains where generating golden summaries may be prohibitive. and while rise shows strong correlation with human evaluations, we do not view rise as a replacement of other metrics. instead, we view it as complementary, especially to referencedependent metrics such as rouge, chrf, and smart. summarization evaluation continues to be a challenging problem. leveraging data from within the domain can help though with calibration of evaluation metrics. bartscore had earlier touched upon this, and rise further helps show the importance of this. we hope this will help spur future research in how we can use domain data to improve such metrics. one of the benefits of rise not explored but left for future work is how rise can be tailored to address specific needs. since rise depends on contrastive learning, one can create negatives that reflect characteristics they want to specifically evaluate. another area of future work is looking at the possibility of using rise in other generative domains outside of summarization, such as dialogue systems. many of the techniques here for training can easily be applied to other domains, including creation of lexical and model negatives.