Extending Multi-Text Sentence Fusion Resources via Pyramid Annotations (2022)

Abstract: NLP models that process multiple texts often struggle in recognizing corresponding and salient information that is often differently phrased, and consolidating the redundancies across texts. To facilitate research of such challenges, the sentence fusion task was proposed, yet previous datasets for this task were very limited in their size and scope. In this paper, we revisit and substantially extend previous dataset creation efforts. With careful modifications, relabeling, and employing complementing data sources, we were able to more than triple the size of a notable earlier dataset. Moreover, we show that our extended version uses more representative texts for multi-document tasks and provides a more diverse training set, which substantially improves model performance.

## introduction despite recent advances reported in nlu benchmarks for single document tasks, cross-document tasks, such as multi-document summarization (mds) have not progressed with the same pace. the handling of information across documents requires effective measures for identifying overlapping content. moreover, generative tasks require consolidating the relevant and redundant content into a coherent utterance. in light of this, several works proposed a focused sentence-level task, called sentence fusion, which focuses on summarizing multiple sentences with overlapping content into a non-redundant one. this allows a finegrained analysis of which information units are shared among the input sentences, as well as control over different degrees of information inclusion and exclusion. however, the available datasets for fusing sentences which exhibit significant content overlap are still lacking, with the most recent containing only several hundreds of examples @xcite fisheries in parts of the philippines have been decimated by the use of cyanide in fishing. ## background the sentence fusion task deals with combining multiple sentences with overlapping content into a single summary sentence that represents the shared information across the inputs @xcite @xcite @xcite . several other variants of sentence fusion have also been explored, such as sentence union -fusing the union of information in the input @xcite . in another variant, "disparate" sentence fusion @xcite @xcite , the input sentences do not exhibit considerable content overlap but are rather related in discourse. such sentences often originate in a single document and pose a different kind of challenge to generate the right discourse structure that will fluently connect the inputs. for pragmatic purposes, a "loose" variant of sentence intersection is desired, since redundant content is most likely salient, yet additional important but non-overlapping information may be relevant for a final summarized sentence. for this reason, our extended dataset follows the fusion as "loose" intersection approach applied by @xcite , @xcite and statoil's internal investigation acknowledged inadequate planning and a lack of risk appreciation led to the leak. ## data collection after carefully analyzing @xcite 's pre-processing pipeline described in subsection 3.1, we decided to substantially modify it (subsection 3.2), recovering significantly more data. we proceed with relabeling some of the targets, and adding samples from source documents (i.e. not just from expertly written summaries) that were mapped to scus but overlooked in the past. ## data analysis we suggest that the additional instances previously skipped would more closely resemble challenges in a multi-document setting. to show that, we compare our extended dataset pyrfus++ to its predecessor pyrfus, that uses closely knit sentence clusters, and to disparate @xcite , that contains mostly non-overlapping within document sentences. the latter allows to estimate a lower bound for overlap for document sentences with little shared content that still relate to each other, making the bound tighter than for randomly picked sentences (some examples are shown in appendix d). we denote by ∆-pyrfus the instances that we added exclusively as part of our extension. 5 to assess content overlap empirically, we calculate the micro-average of rouge @xcite word-train data dev test test++ pyrfus 36.4 40.9 28.5 pyrfus++ 42 45.4 32.5 as expected, pyrfus contains a much higher label to sentence content overlap (r @xmath0, given that the original pre-processing explicitly removed instances with less overlap between the scu output and the source sentences. in fact, our analysis revealed that in pyrfus, extractive target labels, where the target sentence is an approximate copy of one of the input sentences (up to two words), account for 29% of the clusters, while in our extended dataset they account only for 11%. overall our new fusion clusters express high relatedness between the source sentences and their label, while exhibiting higher diversity. ## baselines and data effectiveness we implement a modern baseline (see appendix e for details) for pyrfus @xcite , which outperforms their pre-neural one. 6 to that end, we employ the pre-trained auto-encoder bart @xcite as our end-to-end generation model due to its demonstrated performance on summarization tasks. results, as shown in ## conclusion in this work we extended a sentence fusion dataset by almost four times its original size, while relabeling some of the data. the new dataset includes more complex and relevant training instances, better reflecting those that could be found in "the wild", and thus facilitates further research on data consolidation in multi-text tasks. in addition, we train baseline fusion models and show that when trained on our extended data we achieve notably better performance on the original available fusion test set, while also generating qualitatively better ("loose") sentence intersections. filter filtered scu labels 1 sl saudi arabia urged withdrawal 2 nv resignation of prime minister karami and his government 3 nv | sl murder in boulder, colorado 4 nv confirmed bird flu cases in hong kong 5 sl farc commits slaughters 6 sl water being diverted scus. these include discarding all clusters that: we found that certain filters were safe to remove. we discard filter [2] since the majority of scu labels without a verb use a nominalization (affecting 601 instances). similarly, we ease the length requirement of scus to be between 4-100, as they were found to be coherent and descriptive, affecting 497 instances. additionally, we allow scu clusters that have low overlap between their label and their marked contributing spans, discarding filters [4] and [5], affecting 2410 instances (see