Accuracy is not enough: Evaluating Personalization in Summarizers (2023)

Abstract: Text summarization models are evaluated in terms of their accuracy and quality using various measures such as ROUGE, BLEU, METEOR, BERTScore, PYRAMID, readability, and several other recently proposed ones. The central objective of all accuracy measures is to evaluate the model’s ability to capture \textit{saliency} accurately. Since saliency is subjective w.r.t the readers’ preferences, there cannot be a fit-all summary for a given document. This means that in many use-cases, summarization models need to be personalized w.r.t user-profiles. However, to our knowledge, there is no measure to evaluate the \textit{degree-of-personalization} of a summarization model. In this paper, we first establish that existing accuracy measures cannot evaluate the degree of personalization of any summarization model, and then propose a novel measure, called EGISES , for automatically computing the same. Using the PENS dataset released by Microsoft Research, we analyze the degree of personalization of ten different state-of-the-art summarization models (both extractive and abstractive), five of which are explicitly trained for personalized summarization, and the remaining are appropriated to exhibit personalization. We conclude by proposing a generalized accuracy measure, called P - Accuracy , for designing accuracy measures that should also take personalization into account and demonstrate the robustness and reliability of the measure through meta-evaluation.

## introduction the growing availability of large-scale text data has led to an increasing demand for automated text summarization systems that aim to compress a lengthy document into a short paragraph, which includes salient information about the document. evaluation of such summarization systems (and the underlying models) is performed either in terms of their accuracy or quality. to date, several accuracy *all authors have equal contributions. measures have been proposed such as the widely adopted rouge variants (e.g., rouge-n/l/su4 etc.) @xcite , bleu @xcite , meteor @xcite , bertscore @xcite , pyramid @xcite and the more recently proposed ones such as supert @xcite , widar @xcite , and infolm @xcite . at the same time, several meta-evaluation studies on the reliability of these measures based on the human judgment correlation have been proposed @xcite @xcite @xcite @xcite . within the context of all these developments on novel metrics and meta-evaluation showing varied results on different datasets and summarization systems, the central idea remains the same for all accuracy measures, i.e., they all need to measure the ability of summarization models to capture saliency accurately. however, saliency is subjective to a reader's prior reading history and evolving preferences (aka attention drift). in other words, in many use cases, models have to consider user reading patterns and then generate personalized summaries instead of a fit-all generic summary. this calls for summarizers to be personalized. although, relatively speaking, personalized summarization needs much research attention, there have been a few noteworthy studies in this direction @xcite . the evaluation of these models has been mostly based on the accuracy measures (e.g., rouge-l). however, in this paper, we establish that we cannot measure the degree of personalization of summarization models using accuracy. we theoretically prove that personalization is an independent characteristic compared to accuracy, and it is possible that while a model performs fairly reasonably w.r.t any accuracy measure, it may have a poor degree of personalization. we, thereby, propose a novel measure, called the effective degree of insensitivity w.r.t subjectivity (egises) that conversely measures how insensitive a model is if there is a significant difference in the preferences (or expected summaries) of two readers for the same original document, where an insensitive model would not have much change in the generated summaries for the two readers. to the best of our knowledge, this notion of degree of personalization and its corresponding measure would be the first of its kind. we evaluate egises on ten stateof-the-art summarization models (both abstractive and extractive), including the best-performing ones proposed by @xcite using the pens test dataset @xcite , and show that the leaderboard so generated does not have any consistent correlation with those generated by standard accuracy measures, thereby empirically showing that accuracy is not enough. a key step for any new accuracy metric is to establish its robustness and reliability. the standard method to do so is human-judgment-based meta-evaluation using correlation statistics such as pearson's coefficient, spearman rank correlation, and kendall τ rank correlation. since direct methods are practically infeasible (we will discuss that in section 6.1) we adopt an indirect metaevaluation method that establishes a high humanjudgment vs. egises correlation. we then propose a generic accuracy measure based on egises, called p -accuracy, for calculating the realistic accuracy of models that need to exhibit personalization, and have empirically shown that the measure is stable w.r.t its invariance to the original accuracy leaderboard. ## personalization of existing models one of the key objectives of this paper is to analyze the degree of personalization of different state-ofthe-art summarization models using egises. for this purpose, we chose ten different models, including abstractive and extractive summarizers. in this section, we provide the framework for evaluating these models. ## personalized accuracy in this paper, we primarily focus on proving theoretically and establishing empirically that personalization is a separate attribute from accuracy (and therefore, the leaderboards of both do not correlate). having said that, in this section, we show that even when the primary objective of an evaluator is to measure the accuracy of a model, one should also take into account the ability of personalization of the model to get a realistic accuracy judgment. we call such an accuracy measure as p -acc. however, we would like to emphasize that the key objective of p -acc is not to improve accuracy ranking but rather to achieve a more reliable accuracy value (or score) so that we can benchmark our personalized sota summarization models better and understand the scope of further improvement. therefore, it is rather desirable that p -acc should have a high correlation with corresponding acc. considering this objective, this section aims to establish that egisescan be a good plugin for p -acc. ## related work personalization in summarization models can be broadly categorized into two types -iterative human feedback based models (ihf models), and user-profile based models. in ihf-based models, the reader keeps giving feedback on the summary iteratively till the reader is satisfied with the modelgenerated summary. @xcite proposed a personalized summarization approach in which exdos @xcite is used as a base model for extractive summarization model to rank sentences of news body, and then concepts are extracted and shown to readers for their feedback. the system uses this feedback to iteratively generate a summary that includes the most important concepts till no further negative feedback. however, the evaluation metric used was rouge variants as accuracy metrics, and as established in this paper, cannot be used for measuring personalization. they also measured the change in rouge as the iterations increased, but that still is not a measure of subjective deviation. on the other hand, user-profile-based models such as those that were designed using the pens framework @xcite , which we studied extensively, need significant improvement in personalization. these models, too, were evaluated using rouge variants w.r.t accuracy. in the area of personalized recommendation and search, we can find jaccard index-based measures, order edit distance-based measures @xcite , and the popular ndcg (normalized discounted cumulative gain)-based measures @xcite . however, these are not directly applicable to text summarization and are also extrinsic, relying on human feedback (clicks, likes, etc.), and therefore, cannot be automatic intrinsic measures. ## conclusion in this paper, we establish theoretically and empirically that accuracy measures are unsuitable to evaluate the degree-of-personalization of summarization models, specifically when saliency is fairly subjective. we introduce a new measure, called egises, and show that egises is both stable w.r.t bias and variance and reliable w.r.t humanjudgment correlation when we analyzed the degreeof-personalization of ten summarization models. as an extension, egises needs to be appropriated for capturing the sensitivity of models to the timevariant evolving interests of readers.