Learning to Generate Overlap Summaries through Noisy Synthetic Data (2022)

Abstract: Semantic Overlap Summarization (SOS) is a novel and relatively under-explored seq-to-seq task which entails summarizing common information from multiple alternate narratives. One of the major challenges for solving this task is the lack of existing datasets for supervised training. To address this challenge, we propose a novel data augmentation technique, which allows us to create large amount of synthetic data for training a seq-to-seq model that can perform the SOS task. Through extensive experiments using narratives from the news domain, we show that the models fine-tuned using the synthetic dataset provide significant performance improvements over the pre-trained vanilla summarization techniques and are close to the models fine-tuned on the golden training data; which essentially demonstrates the effectiveness of out proposed data augmentation technique for training seq-to-seq models on the SOS task.

## introduction semantic overlap summarization (sos) is a novel and relatively under-explored seq-to-seq task. it refers to the process of generating a summary of common information from multiple alternative narratives originating from various sources. figure 1 depicts a toy use-case of sos task on two alternative narratives @xcite . here, both articles cover the same event related to a supreme court ruling on abortion in kentucky, but from two opposite political viewpoints; one from the left wing and the other from the right wing (color coded). the goal of the sos task is to generate the overlapping information (in green text). in this sense, the sos task can be defined as a multi-seq-to-seq task with an additional overlap constraint. multiple alternative narratives are frequent in a variety of domains, including education, health, and privacy, and summarizing the common information from these alternative narratives can be highly useful for digesting those multi-narratives at scale and speed @xcite . however, one of the major challenges associated with implementing a seq-to-seq model which can perform the sos task is the lack of readily available training data for supervised learning. one may manually create a training corpus for a particular domain (e.g., news, health etc.) by spending a significant amount of time and money, yet it is unclear how much it will generalize for other domains. therefore, an unsupervised approach is desired to address this problem. in this paper, we propose a new unsupervised data generation technique which can generate an arbitrarily large number of synthetic training examples for the sos task. more specifically, given an arbitrary text corpus from a particular domain, our data generation algorithm can produce an infinite number of sos examples of the form {{d a , d b }, (d a ∩ o d b )}, where, d a and d b are two narratives (in text) and d a ∩ o d b is the desired reference summary of semantic overlap. although the reference overlap summaries in our synthetic examples are noisy and do not ensure the high quality of human-written summaries, they can at least help us train an sos model in a weakly supervised fashion and allow us to leverage the powerful yet data-hungry seq-to-seq deep learning architectures. noteworthy, our main focus in this paper is to propose an intelligent way to create a synthetic dataset for training existing seq-to-seq models for the sos task rather than proposing a new model specifically customized for it. therefore, finding the best model to solve the sos task is an orthogonal goal to our work and hence, out of scope for this paper. rather, the goal of this work is to leverage existing pre-trained seq-to-seq summarization models as an approximation of the overlap summary generator and create artificial examples to further fine-tune such seq-to-seq models. as such, it is important to validate whether fine-tuning with artificial examples are indeed useful for improving the accuracy of the seq-to-seq models. to achieve this, we use the human annotated and verified dataset @xcite to show the efficacy of seq-toseq models fine-tuned on our synthetic examples as compared to the pre-trained baseline models with no fine-tuning. through extensive experiments using narratives from the news domain, we show that the models fine-tuned using our synthetic dataset provide significant performance improvements over the pretrained-only baselines and are close to the models fine-tuned on the golden training data; which essentially demonstrates the effectiveness of the proposed data augmentation technique. overall, we make the following contributions in this paper. ## background here we first provide a brief description of the sos task and the benchmark dataset that was introduced by @xcite . ## challenges for supervised training one of the major challenges with deep seq-to-seq models is that they require a huge amount of data samples to train the model. even the largest pretrained language models require at least a small amount of data to fine-tune them for a new domain @xcite . in our case, we could fine-tune a large pre-trained language model on the allsides data we collected with readily available reference summaries, which particularly covers narratives from the news domain; however, one may not find such annotated data in other domains like health, legal, education etc. in other words, while one may find multiple alternative narratives across many different domains, a dataset with the groundtruth overlap summaries of the narratives will not be available in most cases. to address this challenge, we need to come up with an automatic technique for generating synthetic training data at a large scale. one naive idea is to divide a document into two segments such that some sentences are repeated in both segments and the repeated sentences can be regarded as the desired overlap. however, this approach will not work since the model will simply learn to copy the repeated sentences and ignore the actual semantics entirely. this is where our new synthetic data generation approach can help us circumvent this hurdle, the details of which are discussed in the following section. ## synthetic training data generation our basic idea is to divide a given document d into two parts d 1 and d 2 such that there is a nonempty overlap between d 1 and d 2 in terms of the sentences they contain, i.e., d 1 ∩ d 2 = d o ( = φ) and d 1 ∪ d 2 = d (constraint i). here the ∩ and ∪ operators are classical set operators, i.e., they simply mean intersection and union in terms of the set of sentences and should not be confused with sos output (∩ o ). now, consider {{d 1 , d 2 }, d o } as our training sample where the unordered pair {d 1 , d 2 } is the input to our sos model and d o is the target overlap summary. if we naively train a model on such samples, it will simply learn to copy the repeated sentences (d o ) and would fail terribly in a real testing scenario. also, identifying repeated sentences is a trivial task and training a seq-to-seq model for this has no practical value. indeed, true semantic overlap should be written in an abstract fashion, which is a much harder computational task than identifying repeated sentences. now, assume we have a perfect abstractive summarizer m s and using it, we generate summaries for each of the documents d 1 , d 2 and d o . more specifically, we generate summaries s 1 , s 2 and s o which would contain the core information content of the original documents d 1 , d 2 and d o , respectively. although d 1 , d 2 and d o have some repeated sentences between them by definition, assuming a perfect abstractive summarizer, one can expect that s 1 , s 2 and s o will most likely have no repeated sentence as they have been transformed through an abstractive summarizer. the assumption of a perfect abstractive summarizer also means that s o will only have the common information present in both s 1 and s 2 . in other words, s o can be regarded as a true semantic overlap of s 1 and s 2 and at the same time, s 1 , s 2 and s o will have minimal lexical overlap. thus, having {{s 1 , s 2 }, s o } as our synthetic sample will be perfect for training a seq-to-seq model with {s 1 , s 2 } being the input and s o as the target semantic overlap. however, in the absence of a perfect summarizer, we hypothesize that a reasonable abstractive summarizer pre-trained on a particular domain will be able to generate a large number of noisy synthetic training examples in the form of {{s 1 , s 2 }, s o } and subsequently, fine-tuning a seq-to-seq model using such noisy data will still help us in learning to generate overlap summaries. one nice benefit of our data generation technique is that a large number of synthetic samples can easily be generated from a domain-specific corpus of documents. by partitioning a single document into two overlapping segments and then introducing non-linearity through an abstractive summarization model, we propose a simple yet effective synthetic data generation technique for training the sos task. algorithm 1 generate synthetic data. 1: given document d, abstractive summarization model ms, overlap percentage p, split type spt 2: d1, d2, do ← split(spt, d, p) 3: s1, s2, so ← ms(d1), ms(d2), ms(do) 4: return {s1, s2}, so 1: procedure split(spt, doc, p) 2: ## experiments and results as we mentioned in section 1, we aim is to show the efficacy of our synthetic data generation technique rather than proposing a new specialized solution for the sos task. thus, we leverage off-the-shelf abstractive summarization models as a proxy for sos models and simply, fine-tune them using our synthetic examples.