SEM -F1: an Automatic Way for Semantic Evaluation of Multi-Narrative Overlap Summaries at Scale (2022)

Abstract: Recent work has introduced an important yet relatively under-explored NLP task called Semantic Overlap Summarization (SOS) that entails generating a summary from multiple alternative narratives which conveys the common information provided by those narratives. Previous work also published a benchmark dataset for this task by collecting 2,925 alternative narrative pairs from the web and manually annotating 411 different reference summaries by engaging human annotators. In this paper, we exclusively focus on the automated evaluation of the SOS task using the benchmark dataset. More specifically, we first use the popular ROUGE metric from text-summarization literature and conduct a systematic study to evaluate the SOS task. Our experiments discover that ROUGE is not suitable for this novel task and therefore, we propose a new sentence-level precision-recall style automated evaluation metric, called SEM-F1 (Semantic F1). It is inspired by the benefits of the sentence-wise annotation technique using overlap labels reported by the previous work. Our experiments show that the proposed SEM-F1 metric yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.

## introduction human beings can be viewed as subjective sensors who observe real word events and report relevant information through their narratives @xcite . thus, multiple alternative narratives provide a robust way to comprehend the complete picture of an event being reported and verify corresponding facts and opinions from different perspectives. despite great progress in nlp research in recent years, computers are still far from being able to accurately interpret multiple alternative narratives, which remains an open problem @xcite . in this paper, we study this challenging area of automatic summarization of multiple alternative narratives from different perspectives. more precisely, we exclusively focus on the automated evaluation of a new nlp task called semantic overlap summarization (sos) from multiple alternative narratives. the sos task has been introduced very recently by @xcite , where they conducted a systematic study of this task by creating a benchmark dataset as well as exploring how to manually evaluate this task. sos essentially means the task of summarizing the overlapping information present in multiple alternate narratives by cross-verifying their information contents against each other. computationally, the sos task is defined as follows: given two distinct narratives n 1 and n 2 of an event e, how can we automatically generate a single summary about e which conveys the common information provided by both n 1 and n 2 ? multiple-perspective alternative narratives are frequent in a variety of domains, including education, the health sector, military intelligence, content analysis and privacy. therefore, automatic summarization of multiple-perspective narratives has become a pressing need in this information explosion era and can be highly useful for digesting such multi-narratives at scale and speed. figure 1 presents an example of the sos task, where two human agents are reporting about the potential hiding location of a terrorist and the military general in charge of the mission wants to get a concise summary of the common information (reported by both parties) from both narratives. as shown in figure 1 , both agents report that terrorist leader y has been located (semantic overlap). however, agent 342 reports the hiding location to be san francisco (represented by blue text), whereas agent 463 reports the location to be portland (oregon) (represented by red text). agent 342 suspects that the target is wearing a suicide vest (represented by blue text), while agent 463 mentions that the target is hiding in a tunnel (represented by red text). here two human agents are reporting about the potential hiding location of a terrorist and the military general in charge of the mission wants to get a concise summary of the common information (reported by both parties) from both narratives. "green" text denotes the common information from both reports (semantic overlap), while "blue" and "red" text denotes the unique perspectives of each report. the goal of sos task is to generate a summary that conveys the common/overlapping information provided by the green text, i.e., the terrorist leader has been located. at first glance, the sos task may appear similar to a traditional multi-document summarization task where the goal is to provide an overall summary of the (multiple) input documents; however, the difference is that, for sos, the goal is to provide summarized content with an additional constraint, i.e., the commonality criteria. there is no current baseline method that exactly matches our task; more importantly, it is unclear how to properly evaluate this task in an automated fashion. therefore, as a starting point, we frame the sos task as a constrained seq-to-seq problem where the goal is to generate a summary from two input documents that convey the overlapping information present in both input text documents. however, the bigger challenge we need to first address is the evaluation of the task. to address these challenges, we make the following contributions in this paper. ## related works as sos can be viewed as a multi-document summarization task with additional commonality constraint, text summarization literature is the most relevant to our work. over the years, many paradigms for document summarization have been explored @xcite . the two most popular among them are extractive approaches @xcite @xcite and abstractive approaches @xcite @xcite . some researchers have also tried combining extractive and abstractive approaches @xcite @xcite . recently, encoder-decoder-based neural models have become really popular for abstractive summarization @xcite @xcite . it has become prevalent to train a general language model on a huge corpus of data and then transfer/fine-tune it for the summarization task @xcite @xcite @xcite @xcite . summary length control for abstractive summarization has also been studied @xcite @xcite @xcite . in general, multiple document summarization @xcite @xcite @xcite is more challenging than single document summarization. however, the sos task is different from traditional multi-document summarization tasks in that the goal here is to summarize content with an overlap constraint, i.e., the output should only contain the common information from both input narratives. alternatively, one could aim to recover verb predicate-alignment structure @xcite @xcite from a sentence and further, use this structure to compute the overlapping information @xcite . sentence fusion is another related area which aims to combine the information from two given sentences with some additional constraints @xcite @xcite . a related but simpler task is to retrieve parallel sentences @xcite @xcite without performing an actual overlap summary generation. however, these approaches are more targeted towards individual sentences and do not directly translate to arbitrarily long documents. thus, the sos task is still an open problem and there is no existing dataset, method or evaluation metric that has been systematically studied @xcite . recently, @xcite conducted an initial exploration of the semantic overlap summarization problem and created a benchmark dataset for further research in this area. along the evaluation dimension, rouge @xcite is perhaps the most commonly used metric today for evaluating automated summarization techniques; due to its simplicity and automation. however, rouge has been criticized a lot for primarily relying on lexical overlap @xcite @xcite of n-grams. as of today, around 192 variants of rouge are available @xcite including rouge with word embedding @xcite and synonym @xcite , graph-based lexical measurement @xcite , vanilla rouge @xcite and highlight-based rouge @xcite . a recent study by @xcite showed that the rouge metric is not appropriate for evaluating the sos task. however, there has been no study yet on what can be an alternative to the rouge metric which is automatic and scalable, which is one of the central goals of our work. ## background here we first provide a brief description of the sos task and the benchmark dataset that was introduced by @xcite . ## conclusions in this work, we proposed a more accurate metric, called sem-f 1 , for evaluating the sos task. this metric compares the model-generated overlap summaries with the reference summary on a per-sentence basis using overlap labels and com-bines them to generate f 1 scores. our experiments show that sem-f 1 is more robust and yields higher agreement with human judgement and most importantly, can be computed automatically making it suitable for large-scale evaluation.