Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline (2021)

Abstract: Aligning sentences in a reference summary with their counterparts in source documents was shown as a useful auxiliary summarization task, notably for generating training data for salience detection. Despite its assessed utility, the alignment step was mostly approached with heuristic unsupervised methods, typically ROUGE-based, and was never independently optimized or evaluated. In this paper, we propose establishing summary-source alignment as an explicit task, while introducing two major novelties: (1) applying it at the more accurate proposition span level, and (2) approaching it as a supervised classification task. To that end, we created a novel training dataset for proposition-level alignment, derived automatically from available summarization evaluation data. In addition, we crowdsourced dev and test datasets, enabling model development and proper evaluation. Utilizing these data, we present a supervised proposition alignment baseline model, showing improved alignment-quality over the unsupervised approach.

## introduction text summarization aims to extract the salient information out of a single document or a set of topically-related documents, and to generate a coherent summary. inherently, summarization needs to address, either explicitly or implicitly, several embedded subtasks, such as salience detection, redundancy removal, and text generation. attempting to cope with this encompassing challenge, summarization research often involves developing models for specific summarization subtasks, utilized either as system components or for auxiliary purposes such as training data creation. in this paper, we draw attention to the particular auxiliary task of summary-source alignment, which was utilized as a supporting data generation step in the summarization literature, for both single and multi document summarization. given a gold summarization dataset, the task aligns information pieces in a reference summary with corresponding information in the source documents. such alignments were generated automatically over large summarization datasets, most typically at the full sentence level. then, (noisy) training data for certain summarization subtasks was automatically derived from these alignments ( §2), notably for salience detection @xcite @xcite , but also for redundancy recognition @xcite ) and text rephrasing and fusion @xcite . even though the quality of the subsequent trained models relies on alignment quality, the intermediate alignment methods were neither optimized nor evaluated explicitly, making them difficult to compare and improve. in this paper, we establish summary-source alignment as a stand-alone auxiliary task, with corresponding novel datasets, model, and intrinsic evaluation. as a major contribution, aiming to yield more precise alignments, we propose aligning source-summary information at the more finegrained level of propositions, rather than at the common sentence level. specifically, we align information at the level of individual proposition spans, termed information units (ius), as exemplified in fig. 1 . this level provides much tighter alignments compared to the sentence level used in prior work, since aligned full sentences would typically include both matching propositions as well as non-matching ones. an illustration for the tighter alignment at the proposition-level, over random examples, is shown in ## background and related work as mentioned above, several methods leveraged automatically generated reference-source sentence alignments, to derive (noisy) training sets for summarization subtasks @xcite . for example, training datasets for sentence salience detection were derived from referencesource alignments, by marking as salient those source sentences that were aligned with a summary sentence (e.g. chen and bansal (2018)). as another example, @xcite al. (2019 ) leveraged 1 all corresponding datasets and code are publicly available at https://github.com/oriern/superpal . the underlying sentence alignments, from which the above training datasets were derived, were extracted automatically from large summarization datasets. alignments were detected using unsupervised sentence similarity measures, typically based on rouge score @xcite (see §6 for more details). typically, models trained over the alignmentbased datasets were evaluated only on the final summary. yet, alignment quality, which determines the quality of the utilized training datasets, was never optimized or assessed explicitly, as we do in this paper. notably, alignment of matching pieces of information provided the basis for the prominent pyramid method for summarization evaluation @xcite , capturing information overlap between system summaries and reference summaries. alignments were performed at the level of individual propositions, termed summary content units (scus) (similar to the information units marked in fig. 1 ). matching information at the proposition level was favored over the more coarse sentence level, since a system summary sentence may include some propositions that match the reference and some that don't. later works attempted to automate the pyramid procedure, using open ie @xcite or elementary discourse units (edu) @xcite , to extract proposition-level units. as propositional units (like scus) may share their arguments (such as in conjunctions and other constructions, e.g. "the boy went home and ate dinner"), and may be discontiguous ("the boy...ate dinner"), open ie output, which satisfies these requirements, is best suitable for extracting such units (while edu format ## dev and test alignment datasets this section presents the manually-annotated development and test datasets for reference-source alignments, including their structure ( §3.1), source data ( §3.2) and annotation process ( §3.3). these datasets allow direct tuning and evaluation of alignment algorithms, lacking in prior work ( §1). ## pyramid-based training dataset to obtain larger amounts of alignments for training supervised models, we derive them automatically from existing mds evaluation data. while these alignments are less exhaustive, compared to the manual processes in §3, they are of fairly high quality and proved useful for model training. we follow copeck et al.'s line of work @xcite @xcite , which established a sentence alignment dataset based on the pyramid evaluation method @xcite , applied for the duc 2005-2007 and tac 2008 summarization benchmarks. as many of the systems participating in these benchmarks were extractive, system summary sentences directly link to document sentences. these links, along with the pyramid's expert mapping between reference summary spans and system summary spans, enabled @xcite to transitively align reference summary sentences to document sentences, yielding a sentence-level alignment dataset. we take this process one step further, aligning in a similar manner iu spans in the reference summaries (rather than full sentences) to corresponding iu spans in documents. this is possible thanks to the pyramid's annotated mappings, which link spans between reference and system summaries. such derived alignments are not exhaustive, since the pyramid-based alignments cover only sentences included in the evaluated system summaries. nevertheless, this provides a novel large-scale alignment dataset at the proposition-level, consisting of 18,505 alignments, sufficient for training neural alignment models. the alignments obtained thus far involve spans that were detected manually by the pyramid an-notators. yet, our alignment model (section 5) needs to align ius that were extracted automatically, a step that we implemented using openie (oie) @xcite . in order to train our alignment model appropriately, we create an additional version of the training set, in which the aligned ius were detected automatically by our oie-based extraction (section 5). to that end, we first apply the iu extractor over the summaries and source documents. then, we consider a candidate pair of (oie-based) summary iu and source iu as matching if each of these ius matches the corresponding ius in a pyramid-based alignment, with a jaccard similarity score above the @xmath0.25 threshold (based on section 3). this process yielded 23,492 oie-based alignments, as some of the original spans matched several oie spans, while others did not match any. next, we add negative instances to our dataset. first, we include challenging negative instances by selecting all non-aligning source-summary oie combinations that have bertscore (zhang et al., 2019) above 0.89. in addition we include naturallydistributed negative instances, by taking a summary iu that has an alignment in a certain document and combining it with all non-aligned candidate ius in that document. since our training dataset is based on alignment to extractive system summaries, we consider in this process only document sentences that are included in these system summaries. overall, we sampled 219,772 negative examples, where 22% of them were selected for high bertscore. we estimated our data quality by manually counting errors in random samples, including 80 positive examples and 80 negative examples of each of the two negative example types. the respective error rates were 11% in the positive sample and 5% in each of the negative samples, suggesting the highquality of our automatically-derived data. its actual effectiveness for model training is assessed next. a few samples from the data are presented in ## iu alignment methods and evaluation this section describes two baseline methods ( §5.1, §5.2) and our proposed supervised method ( §5.3) for summary-source iu-level alignment. we compare their performance ( §5.4) against our manuallycreated test set, showing the substantial advantage of the supervised aligner, trained over our automatically-derived training dataset ( §4). the presented alignment methods require a pre-orig summary span oie summary span orig document span oie document span (a) european union (eu) nations agreed that a single currency (the euro) will go into effect on january 1, 1999 that a single currency ... euro) will go into effect on january 1, 1999. the 1999 introduction of the single european currency, the euro the mass printing of the banknotes of the single european currency ... euro ... would be started at the beginning of 1999 (b) president's line-item veto...of 38 military construction projects in october 1997 congress overrode the president's line-item veto against 36 of 38 military construction projects. u.s. president bill clinton used his line-item veto...to strike out...projects...from a military construction bill his line-item veto power to strike out 38 projects worth 287 million u.s. dollars (c) n/a 500 anti-government activists surrendered in march 1999. n/a a total of 15 antigovernment ethnic armed groups have made peace with the government (d) n/a the suit filed by the recording industry. n/a metallica...alleging that napster's software encouraged users to freely trade the band's songs without permission. (e) n/a us tobacco companies appear capable of sustaining strong momentum. n/a yesterday reported a more modest 10.9 per cent advance in net earnings ## iu-enhanced sentence alignment as mentioned in §2, most previous alignment approaches operate on the sentence-level. in this section, to fairly compare to sentence-level methods, we apply our superpal aligner to extract salient sentences, based on the predicted iu alignments (denoted as "superpal sent "). we first show, intrinsically, that our method yields a more accurate set of salient sentences than those derived by common sentence-level rouge-based alignments summary 'the splc's most outstanding successes.. ## conclusion and discussion we advocate the potential of summary-source proposition-level alignment to extract cleaner and more accurate alignments than through the common sentence-level approach. to that end, we establish a new proposition-level alignment task by releasing high-quality dev and test datasets, and an automatically-derived training set. our proposed supervised baseline alignment model, trained on the released data, outperforms existing lexical and semantic similarity methods. notably, it exhibits an excessive ability to yield abstract alignments. these resources provide fertile ground for developing improved proposition-based alignment methods that, similar to sentence-level aligners, can supply training datasets for several summarization subtasks. a proposition-level salience dataset, for example, can be derived by marking each aligned iu in a source document as salient. accordingly, such datasets can be used to train proposition-based models for various summarization components. in future work, proposition-based extractive summarization has the potential to yield bulletstyle summaries with optimized content (similar to cnn/dailymail @xcite ), albeit somewhat less coherent. an example of such potential summary, illustrated by an oracle-system summary derived from our supervised aligner predictions on cnn/dailymail, is shown in overall, we suggest that our released resources would encourage appealing future research on proposition-based summarization approaches, as well as on developing improved alignment models, addressing a challenging semantic matching task. gabriel stanovsky, julian michael, luke zettlemoyer, and ido dagan. 2018. supervised open information extraction. in naacl-hlt. md arafat sultan, steven bethard, and tamara sumner. 2014. back to basics for monolingual alignment: exploiting word similarity and contextual evidence. transactions of the association for computational linguistics, 2:219-230. qian yang, rebecca j. passonneau, and gerard de melo. 2016. peak: pyramid evaluation via automated knowledge extraction. in aaai. tianyi zhang, varsha kishore, felix wu, kilian q weinberger, and yoav artzi. 2019. bertscore: evaluating text generation with bert. arxiv preprint arxiv:1904.09675. xingxing zhang, mirella lapata, furu wei, and ming zhou. 2018. neural latent extractive document summarization. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 779-784, brussels, belgium. association for computational linguistics.