Semantic Overlap Summarization among Multiple Alternative Narratives: An Exploratory Study (2022)

Abstract: In this paper, we introduce an important yet relatively unexplored NLP task called Semantic Overlap Summarization (SOS), which entails generating a single summary from multiple alternative narratives which can convey the common information provided by those narratives. As no benchmark dataset is readily available for this task, we created one by collecting 2,925 alternative narrative pairs from the web and then, went through the tedious process of manually creating 411 different reference summaries by engaging human annotators. As a way to evaluate this novel task, we first conducted a systematic study by borrowing the popular ROUGE metric from text-summarization literature and discovered that ROUGE is not suitable for our task. Subsequently, we conducted further human annotations to create 200 document-level and 1,518 sentence-level ground-truth overlap labels. Our experiments show that the sentence-wise annotation technique with three overlap labels, i.e., Absent (A), Partially-Present (PP), and Present (P), yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.

## introduction in this paper, we look deeper into the challenging yet relatively under-explored area of automatic summarization of multiple alternative narratives with different perspectives. to be more specific, we formally introduce a new nlp task called semantic overlap summarization (sos) from multiple alternative narratives and conduct a systematic study of this task by creating a benchmark dataset as well as exploring how to accurately evaluate this task. sos essentially means the task of summarizing the overlapping information present in multiple alternate narratives by cross-verifying their information contents against each other. computationally, our research question is the following: given two distinct narratives n 1 and n 2 of an event e, how can we automatically generate a sin-gle summary about e which conveys the common information provided by both n 1 and n 2 ? multiple alternative narratives appear frequently in a variety of domains, including education @xcite , the health sector @xcite , businesses intelligence @xcite , content analysis @xcite and privacy @xcite . therefore, automatic summarization of multiple-perspective narratives has become a pressing need in this information explosion era and can be highly useful for digesting such multinarratives at scale and speed. figure 1 shows a toy example of the sos task, where both articles cover the same event related to "abortion". however, they report from different political perspectives, i.e., one from the left wing and the other from the right wing. for greater visibility, "left" and "right" wing reporting biases are represented by blue and red text, respectively. green text denotes the common information in both news articles. the goal of the sos task is to generate a summary that conveys the common/overlapping information provided by the green text. at first glance, the sos task may appear similar to a traditional multi-document summarization task where the goal is to provide an overall summary of the (multiple) input documents. however, the difference is that, for sos, the goal is to provide summarized content with an additional constraint, i.e., the commonality criteria. there is no current baseline method or an existing dataset that exactly matches our task; more importantly, it is unclear which one is the right evaluation metric to properly evaluate this task. as a starting point, we frame sos as a constrained seq-to-seq task where the goal is to generate a summary from two input documents that conveys the overlapping information present in both input text documents. however, the bigger challenge we need to address first is the following: 1) how can we evaluate this task? and 2) how here, an abortion issue-related event has been reported by two news media (left-wing and right-wing). "green" text denotes the common information from both news media, while "blue" and "red" text denotes the unique perspectives of left and right wing. some real examples from the benchmark dataset are provided in the to create a benchmark dataset for this task? to address these challenges, we make the following contributions in this paper. ## related works as sos can be viewed as a multi-document summarization task with additional commonality constraints, text summarization literature is the most relevant to our work. over the years, many paradigms for document summarization have been explored @xcite . the two most popular among them are extractive approaches @xcite @xcite and abstractive approaches @xcite @xcite . some researchers have also tried combining extractive and abstractive approaches @xcite @xcite . recently, encoder-decoder-based neural models have become really popular for abstractive summarization @xcite @xcite . it has become even more prevalent to train a general language model on a huge corpus of data and then transfer/fine-tune it for the summarization task @xcite @xcite @xcite @xcite . summary length control for abstractive summarization has also been studied @xcite @xcite @xcite . in general, multiple document summarization @xcite @xcite @xcite is more challenging than single document summarization. however, the sos task is different from traditional multidocument summarization tasks in that the goal here is to summarize content with an overlap constraint, i.e., the output should only contain the common information from both input narratives. alternatively, one could aim to recover verbpredicate alignment structure @xcite @xcite from a sentence and further, use this structure to compute the overlapping information @xcite . sentence fusion is another related area that aims to combine the information from two given sentences with some additional constraints @xcite @xcite . a related but simpler task is to retrieve parallel sentences @xcite @xcite without performing an actual overlap summary generation. however, these approaches are more targeted toward individual sentences and do not directly translate to arbitrarily long documents. thus, the sos task is still an open problem and there is no existing dataset, method, or evaluation metric that has been systematically studied. along the evaluation dimension, rouge @xcite is perhaps the most commonly used metric today for evaluating automated summarization techniques; due to its simplicity and automation. however, rouge has been criticized a lot for primarily relying on lexical overlap @xcite @xcite of n-grams. as of today, around 192 variants of rouge are available @xcite including rouge with word embedding @xcite and synonym @xcite , graph-based lexical measurement @xcite , vanilla rouge @xcite and highlightbased rouge @xcite . however, there has been no study yet as to whether the rouge metric is appropriate for evaluating the sos task, which is one of the central goals of our work. ## motivation and applications multiple alternative narratives are frequent in a variety of domains and, therefore, have direct implications in technical areas such as information retrieval/search engines, question answering, machine translation, etc. below are a few examples of use cases. peer-reviewing: given two peer-review narratives for an article, the sos task can generate a summary of portions of the narratives that agree with each other, which can help prepare a meta-review quickly. security and privacy: sos task can enable realworld users to quickly conduct a comparative analysis of multiple privacy policies by mining and summarizing overlapping clauses from those poli-cies. thus, it can help users make informed decisions while choosing from various alternative web services. health sector: sos can be used to compare clinical notes in patient records to perform a comparative analysis of patients with the same diagnosis/treatment. for example, sos can be applied to the clinical notes of two (or more) patients who went through the same treatments to assess the effectiveness of the treatment. ## problem formulation a human would mostly express the output in the form of natural language, and this is why we frame the sos task as a constrained multi-seq-to-seq (text generation) task, where the output text only contains information that is present in both the input documents. we also argue that brevity (minimal repetition) is a desired property of semantic overlap summarization. for example, if a particular piece of information or quote is repeated twice in both the documents, we don't necessarily want it to be present in output overlap summary two times. the output can either be extractive summary or abstractive summary or a mixture of both, as per the use case. this task is inspired by the set-like intersection operator as envisioned by @xcite and the aim of this work is to summarize the overlapping information in an abstract fashion. additionally, sos should follow the commutative property, i.e. narrative pair example # 1 narrative 1: n1 narrative 2: n2 washington -u.s. intelligence and law enforcement agencies have confirmed that president donald trump's campaign aides and associates had constant contact with russian intelligence officials before the election, directly contradicting public statements made by top administration officials. on jan. 15, shortly before trump took office, vice president mike pence repeatedly said on television that there were zero contacts between the campaign and russian officials. . . . pence also answered "of course not" when asked a similar question that day by "fox news sunday" host chris wallace . . . trump himself also denied these interactions . . . "there's nothing that would conclude me that anything different has changed with respect to that time period," spicer said. . . . president trump said wednesday that new reports saying his associates had contact with russian officials during last year's campaign are "non-sense" and accused the u.s. intelligence community of illegally leaking information to news outlets. "this russian connection non-sense is merely an attempt to cover-up the many mistakes made in hillary clinton's losing campaign," mr. trump tweeted. . . . among those supposedly communicating with russian nationals was former trump campaign chairman paul manafort, the report said. mr. manafort denied that he ever knowingly talked to any intelligence official "or anyone reference overlap summaries a1 a2 a3 allsides president trump and the trump administration deny allegations that advisers close to trump were in constant communication during the campaign with russians known to us intelligence. trump denied climas that advisers close to him were in "constant communication during the campaign with russians known to us intelligence. donald trump and his group claimed that there is no contact with russian officials during his last year's campaign. russian intelligence officials made repeated contact with members of president trump's campaign staff, according to new reports that cite anonymous u.s. officials. american agencies were concerned about the contacts but haven't seen proof of collusion between the campaign and the russian security apparatus. narrative pair example # 2 narrative 1: n1 narrative 2: n2 john mccain is out of mcconnell's clutches for a week or two. while sen. john mccain remains in arizona recovering from friday's craniotomy, surgery to remove a 5 cm blood clot from above his left eye, business will not go on as usual in washington. majority leader mitch mcconnell, who has to have every republican senator voting to have a prayer of passing trumpcare, has postponed the vote for the week or two (more likely two) that mccain's recovery will take. that means there's more time for opponents to fight this thing, from the side of all of us trying to keep 22 million people from losing insurance and from the other side. . . . with both paul and sen. susan collins (r-me) solid "no" votes on the bill, opponents only need one more out of the eight or so who've expressed reservations about the bill and the secretive, exclusive process mcconnell washington -the republican effort to repeal and replace obamacare faces a major setback as sen. john mccain, r-ariz., left the nation's capital for surgery on his eye. over the weekend, senate majority leader mitch mcconnell, r-ky., announced the scheduled better care act vote would be delayed indefinitely because of mccain's absence. subsequently, the congressional budget office (cbo) also delayed its analysis of the bill. with two republican senators opposed to the measure, mcconnell needs as least 50 "yes" votes to pass it. sen. rand paul, r-ky., says the bill, which keeps taxes on investments and other pieces of obamacare, doesn't go far enough. moderate sen. susan collins, r-maine, is also withholding her support because it would slow the rate of growth in spending on medicaid. . . . reference overlap summaries a1 a2 a3 allsides sen. john mccain remains in arizona recovering from eye surgery. senate majority leader mitch mcconnell postponed the vote due to mccain's absence. two republican senators opposed to the bill. possibility of bill failing. sen. john mccain remains unavailable because of the surgery on his eye. senate majority leader mitch mcconnell delayed the vote in his absence. sen. rand paul and sen. susan collins said "no" votes on the bill. senate majority leader mitch mcconnell, r-ky., announced the scheduled health care vote would be delayed indefinitely because of mccain's absence. senate majority leader mitch mcconnell, r-ky., announced the scheduled better care act vote would be delayed indefinitely because of mccain's absence. from the "right" wing, thus 300 narratives in total) and then asked 3 humans to write a summary of common information present in both narratives describing each of the 150 events. after the first round of annotations, we immediately observed a discrepancy among the three annotators in terms of the real definition of "common/overlapping information". for example, one annotator argued that the reference summary should be non-empty as long as there is an overlap between two narratives along at least one of the 5w1h facets (who, what, when, where, why, and how), while another annotator argued that overlap in only one facet is not enough to decide whether there is indeed enough semantic overlap between the two narratives and reference summary should be left empty in such cases. as an example, one of the annotators wrote only "donald trump" as the reference summary for a couple of cases where the actual narratives were substantially different except for "donald trump" being the only common entity, while others had those cases marked as "empty". to mitigate this issue, we only retained the narrative pairs where at least two of the annotators wrote a minimum of 15 words as their reference summaries, assuming that a human-written summary will contain 15 words or more only in cases where there is indeed a significant overlap between the two original narratives. this filtering step gave us a test set with 137 narrative pairs where each sample had 4 reference summaries, one from allsides and three from human annotators, resulting in a total of 548 reference summaries. a couple of sample narrative pairs are shown in for evaluation, we first evaluated the machinegenerated overlap summaries for the 137 manually annotated testing samples using the rouge metric by following the procedure mentioned in @xcite to compute the rouge-f 1 scores against multiple reference summaries. more precisely, since we have 4 reference summaries, we got 4 precision, recall pairs which are used to compute the corresponding f 1 scores. for each sample, we took the max of these four f 1 scores and averaged them out across the test dataset (see ## limitations and future work one particular limitation of this work is that we have used pre-trained abstractive summarization models as naive baselines / proxies for semantic overlap summarizer and did not attempt to develop a custom method that optimizes for the overlap constraint. however, the primary focus of this paper is to define the sos task, as well as establish the first benchmark dataset and a suitable evaluation approach for the task. therefore, the design and optimization of methods is an orthogonal task to this paper, which we will pursue as our immediate future work. another limitation of our work is that the test set is not big (∼ 150 examples), which makes it difficult to do a rigorous evaluation. however, while the number 150 may initially appear to be small; cleaning and annotating the dataset required significant time and resources. to elaborate further, our test dataset contains 137 samples, where each sample consists of two alternative narratives along with 4 ground truth references. out of these 4 references, 3 of them were manually written by 3 human annotators. thus, we manually created 3 * 137 = 411 reference summaries in total. additionally, for each sample (narrative pair), each annotator first had to carefully read through two alternative narratives several times, digest the semantic overlap between them and then summarize the overlap in their own words. this process took on average 40 minutes per annotator per sample, which means we spent around 411 * 40 = 16, 440 minutes of human efforts in one round of the annotation process. next, we had to resolve conflicts among annotators by going through each of their annotated summaries (a very painstaking process) and figuring out the reasons for the conflicts. based on follow-up discussions, we revised the guidelines for annotation and went through the entire annotation process again. in total, we needed 4 iterations (16, 440 * 4 =∼ 65, 760 minutes) to resolve most of the conflicts. the whole process took more than 8 months for us. finally, we agree that having more samples in the test dataset would definitely help. but this is both time and money-consuming. we are working towards it and would like to increase the sample size in the future. ## conclusion in this work, we introduced a new nlp task, called semantic overlap summarization (sos) and created a benchmark dataset through meticulous human efforts to initiate a new research direction. as a starting point, we framed the problem as a constrained summarization task and showed that rouge is not a reliable evaluation metric for this task. further human experiments show that sentence-wise evaluation leads to higher agreement with human judgment, therefore, an evaluation metric that aggregates sentence-wise overlap labels should be used while evaluating the sos task.