G e SERA : General-domain Summary Evaluation by Relevance Analysis (2021)

Abstract: We present GeSERA, an open-source improved version of SERA for evaluating automatic extractive and abstractive summaries from the general domain. SERA is based on a search engine that compares candidate and reference summaries (called queries) against an information retrieval document base (called index). SERA was originally designed for the biomedical domain only, where it showed a better correlation with manual methods than the widely used lexical-based ROUGE method. In this paper, we take out SERA from the biomedical domain to the general one by adapting its content-based method to successfully evaluate summaries from the general domain. First, we improve the query reformulation strategy with POS Tags analysis of general-domain corpora. Second, we replace the biomedical index used in SERA with two article collections from AQUAINT-2 and Wikipedia. We conduct experiments with TAC2008, TAC2009, and CNNDM datasets. Results show that, in most cases, GeSERA achieves higher correlations with manual evaluation methods than SERA, while it reduces its gap with ROUGE for general-domain summary evaluation. GeSERA even surpasses ROUGE in two cases of TAC2009. Finally, we conduct extensive experiments and provide a comprehensive study of the impact of human annotators and the index size on summary evaluation with SERA and GeSERA.

## introduction automatic summary evaluation is a challenging task in natural language processing (nlp). evaluation is usually done by humans, but manual evaluation is subjective, costly and time expensive @xcite . automatic evaluation methods @xcite @xcite are an alternative to save time for users who extract the most relevant content from the web using automatic text summarization systems (ats). there exist two types of evaluation approaches: (1) manual evaluation methods like pyramid @xcite and responsiveness, where human intervention is mandatory, and (2) automatic evaluation methods, where human intervention can be needed as a ground-truth reference @xcite or not @xcite . summary evaluation by relevance analysis (sera) @xcite is an automatic evaluation method that partially relies on human references to evaluate abstractive summaries from the biomedical domain. it was proposed as an alternative to rouge @xcite , the widely used automatic metric, that is based on lexical overlaps between candidate and reference summaries. rouge is unfair to evaluate abstractive summaries where the ats paraphrases the text instead of just copying-pasting chunks of it @xcite . sera is based on content relevance and is fairer to evaluate abstractive summaries because it attributes high scores to summaries that are lexically different but semantically related. however, it surpasses rouge on the biomedical domain only. in this paper, we modify sera to make it usable for generic collections. we propose the following contributions: 1. implement an open-source version of sera from scratch. 2. propose gesera (general-domain sera), an improved version of sera that is domainindependent. 3. conduct extensive experiments with two large indexes (aquaint-2 @xcite and wikipedia) and three summarization datasets (tac 1 2008, tac2009, cnn-daily mail (bhandari et al., 2020)). these datasets are well-suited for general-domain and news abstractive summary evaluation. gesera achieves competitive results compared to a range of state-of-the-art evaluation approaches on both abstractive and extractive summaries. 4. make the code and our wikipedia dataset publicly available to help future researchers. foot_1 5. conduct extensive experiments on the impact of the index size and human annotators on summary evaluation with sera and gesera. sera is defined in equation 1. another variant of sera is called sera-dis. it takes the order of retrieved documents into consideration (equation 2 ). the intuition behind sera is that a summary context is represented by its most related articles. thus, two summaries related to the same documents are semantically related, even if they are lexically different. consequently, sera is fairer to evaluate abstractive summaries contrarily to the lexical-based rouge. however, sera suffers from a series of limitations: (1) the code is not open-source, (2) no information was provided concerning the subset of pubmed used as an index, and (3) pubmed is specialized in the biomedical domain only. the first two drawbacks make sera unusable by the community, while the third restricts its usage to the biomedical domain. ## results and discussion in subsection 4.1, we vary the index size in sera and gesera and study the variation of their performance on tac datasets by averaging the score of all four manual annotators a 1 , a 2 , a 3 , and a 4 . once we determine the best index size, we present in subsection 4.2 the correlations using the best index size of each method. ## related work evaluation methods are fundamental techniques to assess if summaries generated by an automatic system capture the original document's idea. different evaluation methods have been developed in the last decade for the evaluation of automatically-generated summaries. there exist two types of evaluation methods: (1) manual evaluation methods, and (2) automatic evaluation methods. the first group of methods requires human intervention as ground-truth references. pyramid @xcite and responsiveness are the most popular such methods. the second group of methods is divided itself into two subsets: (1) methods that need human intervention like rouge @xcite and sera @xcite , and (2) methods that do not need any human reference like summtriver (cabrera-diego and torres-moreno, 2018) and fresa @xcite . the most popular automatic metric used by the community is rouge @xcite . it needs reference summaries, and is based on their lexical overlaps with candidate summaries. that is why it is more useful to evaluate extractive summaries where chunks of the text are copied and pasted to form the summary. however, in the case of abstractive summaries where the ats paraphrases the text with possibly new vocabulary, the rouge metric becomes unfair. to overcome this issue, researchers have been proposing in the last few years other automatic metrics to fairly evaluate both extractive and abstractive summaries. the first type of automatic evaluation methods relies partially on human judgment as rouge does. the simplest method is based on a jensen-shannon (js-2) @xcite divergence between bi-gram's distribution of candidate and reference summaries. more sophisticated systems include moverscore @xcite that is based on fine-tuning the bert model and combining contextualized representations with earth mover distance (emd) from @xcite . bertscore @xcite also is based on bert model. unlike rouge @xcite , bertscore makes use of contextual embeddings that are effective for paraphrase detection. similarly to bertscore, semantic similarity for abstractive summarization (ssas) @xcite is based on semantic matching between candidate and reference summaries. the second type of automatic evaluation methods does not need any human intervention. for instance, framework for evaluating summaries automatically (fresa) @xcite is based on divergences among probability distributions between the summary to evaluate and its source document. another well-known metric is summtriver (cabrera-diego and torres-moreno, 2018). it is based on trivergences between the summary to evaluate, its source document(s), and a set of summaries related to the same source document(s) but generated with other ats systems. ## conclusion and perspectives we introduced gesera, an open-source system for general-domain summary evaluation. we rede-fine query reformulation of sera based on pos tags analysis of datasets from different domains, and replace the biomedical index with documents from aquaint-2 and wikipedia. gesera achieves competitive results compared to state-ofthe-art approaches. overall, gesera surpasses sera and reduces its gap with rouge, and in two cases, it even surpasses rouge, the lexicalbased method. unsurprisingly, the comparison with evaluation methods that do not rely on human references reveals a large gap in favor of gesera since it relies on human references while the others do not. extensive experiments show that the index size has a considerable effect on the performance of sera and gesera that tend to perform better with small-size indexes. finally, the study of human annotators shows their impact on the performance of automatic evaluation methods that rely on human intervention. our code is publicly available to facilitate reproducibility. we will: