Towards Summary Candidates Fusion (2022)

Abstract: Sequence-to-sequence deep neural models fine-tuned for abstractive summarization can achieve great performance on datasets with enough human annotations. Yet, it has been shown that they have not reached their full potential, with a wide gap between the top beam search output and the oracle beam. Recently, re-ranking methods have been proposed, to learn to select a better summary candidate. However, such methods are limited by the summary quality aspects captured by the first-stage candidates. To bypass this limitation, we propose a new paradigm in second-stage abstractive summarization called SummaFusion that fuses several summary candidates to produce a novel abstractive second-stage summary. Our method works well on several summarization datasets, improving both the ROUGE scores and qualitative properties of fused summaries. It is especially good when the candidates to fuse are worse, such as in the few-shot setup where we set a new state-of-the art. We will make our code and checkpoints available at https://github.com/ntunlp/SummaFusion/.

## introduction leading abstractive summarization methods typically rely on transfer learning and the pre-trainthen-finetune paradigm. in this approach, a deep sequence-to-sequence neural model is first pretrained on a very large text corpus, either with a general-purpose text generation objective like masked span generation as in t5 @xcite , bart @xcite and prophetnet @xcite , or with a pre-training objective specific to summarization as in pegasus @xcite and ted @xcite . then, the model is fine-tuned on the downstream summarization dataset(s) of interest, which can have a wildly varying amount of human labels, from a few thousands to hundreds of thousands. ## related work second-stage methods have recently enabled strong progress in the state-of-the-art of abstractive summarization research. gsum @xcite uses additional discrete guidance signal such as salient sentences predicted by an extractive model to better guide the abstractive system. while abstractive summarization models are trained to maximize mle at the token-level, second-stage methods usually work at the sequence-level. consum @xcite and seqco @xcite fine-tune the model with a different, contrastive loss to assign more confidence to higher-quality summary candidates. refsum @xcite uses metalearning to re-rank summary candidates produced by several base systems. summareranker @xcite and simcls @xcite train a roberta to re-rank candidates, the former with multi-label binary cross-entopy, the latter with con-trastive learning and a ranking loss. brio @xcite re-uses the base model for a secondround of fine-tuning with both the cross-entropy loss and a candidate-level ranking loss. existing fusion work in summarization focuses on sentence fusion. fusing several sentences for the purpose of summarization was first proposed by @xcite , paving the way for more abstractive summaries. @xcite later proposed a much larger dataset for sentence fusion in multi-document abstractive summarization, driving up model performance. through a thorough human evaluation, @xcite ask annotators to label which type of fusion of sentences is taking place while also rating the sentence properties for sentences generated by several abstractive systems. in a follow-up work @xcite , the authors build a transformer model @xcite enriched with sentence structure information for the explicit goal of fusing sentences, and evaluate the model on a dataset dedicated to sentence fusion. the same authors also introduce a cascade model @xcite for abstractive summarization which contains a fusion mechanism based on combining highlighted phrases of the source text. to the best of our knowledge, there is no method yet to fuse or combine in an abstractive manner entire summary candidates (not just sentences). ## model if x i is a source document and y i its associated target summary, b is the first stage model generating m candidates c @xmath0, . . . , c m }, the secondstage fusion model θ is trained to maximize the likelihood of the target given x i and c i : this formulation is essentially the same as the one typically used to train the base model b: autoregressive cross-entropy loss with teacher forcing. the only difference is that in the second stage, the model can also condition on the first stage candidates c i on top of the source x i . fig. 1 shows the overall architecture of our fusion model. ## evaluation we compare summafusion outputs with our own pegasus with 15 diverse beams baseline (pe-gasus (ours)), as well as pegasus-random, a baseline consisting in randomly selecting a summary candidate. we also include the oracle for reference (pegasus-oracle) and compare with pegasus reported results @xcite . we use rouge-1, rouge-2, and rouge-l @xcite and their mean as quantitative metrics to assess summary closeness to the target. ## analysis 6.1 when is summary fusion helpful? the previous section suggested that summafusion is better on lower-quality base candidates, such as in reddit-tifu or few-shot setups. to verify this hypothesis and better characterize the fixing behavior of summafusion, we split the test set across four different features: results are shown in fig. 3 , with one subplot for each of the four features and for each dataset. there are several important takeaways from this figure: there is no clear trend over all datasets for this feature. it is also not intuitive which case is harder to learn, as a short compression ratio means a higher level of summarizing, while a longer one corresponds to longer output summaries, which is also more prone to decoding errors.