An Anchor-Based Automatic Evaluation Metric for Document Summarization (2020)

Abstract: The widespread adoption of reference-based automatic evaluation metrics such as ROUGE has promoted the development of document summarization. In this paper, we consider a new protocol for designing reference-based metrics that require the endorsement of source document(s). Following protocol, we propose an anchored ROUGE metric fixing each summary particle on source document, which bases the computation on more solid ground. Empirical results on benchmark datasets validate that source document helps to induce a higher correlation with human judgments for ROUGE metric. Being self-explanatory and easy-to-implement, the protocol can naturally foster various effective designs of reference-based metrics besides the anchored ROUGE introduced here.

## introduction automatic evaluation metric plays a vital role in evaluating system performance for the task of document summarization. challenges remain in the design of an ideal evaluation metric and the off-the-shelf metrics have their own drawbacks @xcite . the widely adopted metrics, e.g. rouge @xcite , are reference-based in that they compare the output of some summarizer (namely peer summary) with one or multiple human-authored summaries (namely reference/model summary). the reference-free metrics are still not mature enough to be utilized for evaluation in a real-world setting since their correlations with human judgments have been reported to fall far behind reference-based metrics, especially for multi-document summarization @xcite 1 . in this paper, we consider a new protocol of reference-based summarization metrics by rethinking the role of source document which is indeed a lost treasure neglected by most previous works. furthermore, a specific implementation of the protocol (i.e. anchored version of rouge) will be discussed. the reference-based metrics that already exist typically pursue a kinda computation of overlap between the peer and reference summary either at a lexical level @xcite or at a semantic level @xcite @xcite . however, to our knowledge, few of them consider the impact of source document (or documents in multi-document summarization) on the computation. this goes against common sense as source document is the true information source of both summaries and can be utilized to boost the discriminative power of metrics. therefore, we advance a new protocol of reference-based metrics for the evaluation of document summarization. more specifically, the direct participation of source document is a necessity to compute any reference-based metric for document summarization. this makes source document endorse a certain metric and the advantage lies with the ability to fact-check the information of peer summary based on the information pool (i.e. source document). the protocol change is illustrated in figure *: the authors contribute equally to this paper. 1 reference-free evaluation metrics inherently are more suitably used as the reward function in a reinforcement-learningbased summarizer @xcite . col are called "active metrics" since they will be able to refer to the source. in a word, the new protocol has introduced a key dimension that can nurture reference-based summarization metrics. for a verification purpose, we propose an anchored version of rouge metric under the new protocol. the anchors here mean a set of lexical items (called particles) in source document corresponding to a certain particle in the summary (peer or reference). utilizing anchor set in the computation of rouge can introduce a weighted scheme that focuses more on the link to source document, as will be detailed in the next section. ## a specific implementation: anchored rouge following the new protocol, the rouge metric can be revised by introducing the anchor set for each particle (i.e. lexical item such as n-gram and skipping bigram) in both peer and reference summaries. the anchor set for a particle in the summary comprises k particles in source document, each of which is a good match for the summary particle. in other words, anchor sets serve as the ground of summary particles. we build the anchor set a s for summary particle s following the two steps: (1) compute the cosine similarity of embedding vectors of s and d with d being any arbitrary document particle (s and d should be of the same lexical form such as bigram); (2) extract top-k document particles based on similarity to form the anchor set, i.e. a @xmath0, d s2 , ..., d sk }. also, we record the similarity as the strength of anchor and denote the strength between s and d si as q si (1 ≤ @xmath1. the embedding vector of the particle in this paper is obtained by averaging the contextualized embeddings of all tokens occurring in the particle. specifically, in the following experiment, we will sum the last four hidden layers of the pretrained uncased bert base model foot_0 @xcite to get the embedding for each token (dimension of embedding vector is 768). an example of anchor set can be found in figure 2 . -------------------------great success------------------------- -------------------------------------- remarkable achievement anchor set (@xmath2. the anchored version of rouge can be defined as follows once all the anchor sets for summary particles (both in peer and reference) have been built. we calculate the union of anchor sets for all particles in a reference summary and denote it as c ref . ## evaluation efficacy of anchored rouge datasets. we select two datasets of topic-focused multi-document summarization (mds), i.e. tac 2008 foot_1 and tac 2009 foot_2 , for two main reasons: (1) mds is more challenging than single document summarization and summarizers tend to behave more differently for evaluation, which fits the purpose to examine various metrics; (2) multiple reference summaries are offered, which makes it possible to perform robustness test (see ## related work there are various reference-based automatic evaluation metrics for the task of document summarization. the widely accepted metric is rouge @xcite ) that focuses primarily on n-gram co-occurrence statistics. some strategies are proposed to replace the "hard matching" of rouge, such as the adoption of wordnet @xcite and the fusion of @xcite . another promising method of designing metrics is to directly compute the semantic similarity of peer and reference summary, including the metrics utilizing various word embeddings such as elmo @xcite and bert @xcite . furthermore, @xcite proposes a metric computing factual correctness based on information extraction. however, none of the above metrics fall into the newly-introduced protocol. the anchored rouge proposed by us is a refined metric that has followed the new protocol and enjoyed higher correlations with human judgments. ## conclusion we propose a new protocol to foster the development of reference-based automatic metrics to evaluate document summarization. the protocol features the endorsement of source document and can be implemented as an anchored version of the rouge metric fixing each summary particle on the ground of source document. experiments demonstrate that anchored rouge has a higher correlation with human judgments as compared to other metrics. also, our metric is robust to the number of reference summaries, which can be applied to the challenging low-resource setting. future works include extending the new protocol to get various workable evaluation metrics besides anchored rouge.