The Power of Summary-Source Alignments (2024)

Abstract: Multi-document summarization (MDS) is a challenging task, often decomposed to subtasks of salience and redundancy detection, followed by text generation.In this context, alignment of corresponding sentences between a reference summary and its source documents has been leveraged to generate training data for some of the component tasks. Yet, this enabling alignment step has usually been applied heuristically on the sentence level on a limited number of subtasks.In this paper, we propose extending the summary-source alignment framework by (1) applying it at the more fine-grained proposition span level, (2) annotating alignment manually in a multi-document setup, and (3) revealing the great potential of summary-source alignments to yield several datasets for at least six different tasks. Specifically, for each of the tasks, we release a manually annotated test set that was derived automatically from the alignment annotation. We also release development and train sets in the same way, but from automatically derived alignments.Using the datasets, each task is demonstrated with baseline models and corresponding evaluation metrics to spur future research on this broad challenge.

## introduction common information needs are most often satisfied by multiple texts rather than by a single text. processing multiple texts is a challenging feat due to the wealth of content they possess, as well as the anticipated redundancy of information. to address this challenge, various tasks support multi-text processing needs, such as information selection, consolidation and fusion. a prominent application that aims to respond to user information needs is the multi-document summarization (mds) task. inherently, mds either explicitly or implicitly prescribes * work was done as an intern at amazon. a florida starbucks has made it easy for deaf customers to order at the drive -thru -and video of one deaf woman 's experience has been viewed more than 6.5 million times since she posted it on facebook tuesday . i think it's awesome because it gives the deaf opportunity to order in a different way through the window," dr. fellows said. the video was just posted tuesday, and it has more than 1 million views and over 65,000 shares. sub-tasks like those listed above @xcite . notably, many works showed the superiority of decomposing a complex task, and more specifically summarization, into its natural subtasks @xcite @xcite . however, in most cases there is no training data for each of the subtasks, nor gold test data to evaluate them. that is, while summarization datasets contain gold reference summaries, they do not publish, e.g., gold document salient spans for the salience detection task, or gold sentences that fuse information from a few salient document spans for the fusion task. in this paper, we unveil a simple approach to obtain high-quality datasets for a wide variety of multi-text related tasks, via a single annotation process of summary-source alignments. specifically, we match summary propositions with their proposition-level supporting evidence in the source over an existing mds dataset (multi-news; @xcite . an example for alignments is presented in figure 1 . aligning all the information segments between a reference summary and its paired document set reveals the underlying sub-tasks constructing the summarization process, as illustrated in figure 2 . for instance, the aligned spans in the document set constitute the salient information within them, since they collectively embody the corresponding summary. this characteristic captures a salience detection task (figure 2 (b) ). similarly, a summary segment can be viewed as a fused version of all its aligned document-set mentions, representing a sentence fusion task (figure 2 (f) ). overall, with these alignments we automatically derive datasets for six tasks: (1) salience detection, (2) proposition coreference clustering, (3) evidence detection, (4) text planning, (5) sentence fusion, and (6) in-context passage fusion. in essence, this procedure "reverse engineers" the human summarization process by which the reference summaries were originally created. the resulting data enriches the current inventory of datasets for these individual tasks, while being derived automatically solely from the high-quality alignment annotations. while the tasks addressed here stand on their own merits, such tasks have also been shown to benefit the overall summarization process when used within a pipeline, as mentioned before (more on this in §2). our high-quality alignments test set was obtained through a controlled crowdsourcing procedure @xcite , with annotators diligently trained for this task, and contains 100 topics (document-set/summary pairs) with 2256 alignments. we also created large-scale training and development sets by extracting alignments automatically, using the superpal alignment model @xcite , from the multi-news train and dev sets ( §3). we automatically derive and release train and test datasets from the alignment data for the six mentioned tasks ( §4). for each of the tasks, and using its respective dataset, we develop and evaluate two baseline models explicitly targeting the task: one is a trained model while the other is a non-finetuned execution of a chatgpt llm (ope-nai, 2023) ( §5). over the six tasks, we generally find that smaller trained models yield better results than the gpt counterpart, leaving room for future advances using our task and dataset suite. overall, this work showcases that alignments from an mds dataset empower a rich collection of multi-document related tasks. these tasks are appealing on their own, and are additionally advan-tageous as sub-components of mds solutions. foot_0 ## background aligning information between source and reference texts has been previously addressed in the realm of summarization. an early effort was conducted for the purpose of summary evaluation through the pyramid method @xcite ). it's effectiveness, albeit burden of annotation, triggered the pursuit of automatic procedures that mimic the content extraction and alignment @xcite @xcite , generally via proposition extraction and matching. the alignment approach and manual annotation process for our dataset is reminiscent of the pyramid method's, however it is more scalable thanks to controlled crowdsourcing @xcite . while shapira et al. (2019) also applied crowdsourcing for easing manual alignment, the summary is not exhaustively aligned and high quality is not guaranteed. besides summary evaluation, alignments have also been useful for the summarization process itself. to acquire such alignments, the rouge metric @xcite was leveraged to match between summary and source sentences @xcite . this approach was also taken for components of the summarization pipeline, such as detection of salient sentences @xcite and sentence fusion @xcite . the heuristic nature of this pairing approach yields noisy alignments since it is both on the sentence level and based on lexical matching. many additional works have shown the benefit of decomposed summarization pipelines. @xcite and @xcite split the summarization task to planning and realization phases (corresponding to our derived tasks) showing improved summary outputs. @xcite find that decomposition of the summarization process can improve faithfulness of the output summary to its source documents. @xcite shows the benefit of salience detection as an initial phase, both for improving summary quality, and to provide attribution to summary segments. information alignment has also been treated as a standalone task. @xcite designed a supervised model that far exceeds the abilities of lexical-based aligners. they also released a high quality test set of sub-sentence-level alignments. collecting data required expert cleaning of crowdsourced annotations. our more scalable approach yielded a dataset that is an order of magnitude larger (11 vs 100 topics), and further improves alignment accuracy through refined guidelines and removed ui constraints in the annotation tool @xcite . recently, @xcite released usb, a benchmark for summarization-related tasks, also derived from alignments. alignment was conducted between the leading section of a wikipedia article and its body, via controlled crowdsourcing, and required editing text spans in the leading section to remain faithful to the article body. in contrast to their sentence level alignments, our proposition-level alignments eliminate non-aligning noise. moreover, aligning in the multi-document setting, as opposed the single-document in usb, introduces challenges arising from cross-document information sharing and size. the differences listed above induce tasks in our work that are mostly different from those addressed in usb, and that can be treated as standalone tasks. ## collecting alignments data our alignments data consists of a high quality test set, collected through careful manual annotation ( §3.1), as well as large-scale training and development sets that were automatically compiled ( §3.2). all alignments were extracted from the respective data split of multi-news @xcite , a mds dataset of sets of news articles with professionally prepared summaries. an instance in our alignment dataset is based upon a document set d and a corresponding reference summary s. ## the task suite out of the summary-source alignments, annotated manually or automatically, we derive six new datasets for six different tasks, as elaborated below. the tasks are illustrated in figure 2 and an example topic from our manual dataset is presented in appendix f. we denote this data suite as "spark", for summary proposition alignment for reconstructive knowledgebases. ## baseline experiments we next examine the performance of current technology on the six aforementioned datasets. for each task, we consider a dedicated trained model and an execution of gpt-3.5-turbo, once in zero-shot mode and once with an in-context example (prompts in appendix b). since the large size of a multi-document set limits model architecture options, we resorted to models devised for the multi-document setting. we used our train set to train the dedicated models ( §3.2). in addition, the large input sizes afforded us to execute gpt with only one in-context example. we explain the finetuned models in §5.1 and discuss the results in §5.2. ## source dataset characteristics the alignments extracted from the mds dataset (multi-news) sheds light on various characteristics of the source dataset. it helps understand the amount of information redundancy, level of abstractivness, and spread of content within documents. ## conclusion we advocate the potential utility of propositionlevel summary-source information alignment, particularly in the multi-document setting, for exposing a wide range of summarization-related tasks. specifically, we reveal that these alignments induce datasets for a broad range of appealing tasks arising in summarization, and applicable as standalone tasks. we annotated a high-quality test dataset of alignments, and automatically compiled large-scale train and dev sets. from the alignments data, we automatically derived datasets for six distinct tasks. our released dataset collection, along with our baselines and analyses, promotes future research on a challenging multi-text task suite.