Unsupervised Summarization Re-ranking (2023)

Abstract: With the rise of task-specific pre-training objectives, abstractive summarization models like PEGASUS offer appealing zero-shot performance on downstream summarization tasks. However, the performance of such unsupervised models still lags significantly behind their supervised counterparts. Similarly to the supervised setup, we notice a very high variance in quality among summary candidates from these models while only one candidate is kept as the summary output. In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models. Our approach improves the unsupervised PEGASUS by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks ; and achieves relative gains of 7.51% (up to 23.73% from XSum to WikiHow) averaged over 30 zero-shot transfer setups (finetuning on a dataset, evaluating on another).

## introduction transformer-based encoder-decoder language models have achieved great success in abstractive summarization in the last few years, and produce fluent summaries which can be quite abstractive @xcite @xcite . these models follow the pre-train then fine-tune paradigm: they are first pre-trained with a selfsupervised objective on a large text corpus; then they are fine-tuned on the downstream dataset of interest, using the available supervision, which may be very scarce. finding a better pre-training objective remains an active research area. some models like t5 @xcite and @xcite adopt a more general language modeling objective (e.g., masked span generation), while recently, in the supervised setup, second-stage approaches have gathered interest in abstractive summarization research. while the base encoderdecoder model is trained with maximum-likelihood estimation (mle) to predict each token of the ground-truth summary in an autoregressive manner, second-stage methods work with a global view at the whole sequence level. @xcite and @xcite propose to train another neural model to rank summary candidates generated by decoding methods like beam search @xcite or diverse beam search @xcite . @xcite bypasses the need for another model, and re-uses the fine-tuned model for another fine-tuning stage in which the model also learns to rank candidates in the correct order. summafusion @xcite encodes each summary candidate separately and decodes into a new, abstractive secondstage summary. such second-stage methods have improved rouge-1 state-of-the-art on cnn/dm by more than 3 points @xcite . in this paper, we propose to re-rank summary candidates in the unsupervised setup. following observations made by second-stage summarization studies in the supervised setup @xcite , we also observe large variance in performance among summary candidates in the unsupervised setup. in the main challenge lies in the fact that the reranker must also not access any supervision. our proposed model does not train any neural model, but simply computes features indicative of summary quality to score each summary candidate, some of them which also leverage the source document. a weighted average of these features is used for candidate re-ranking, and we explore several methods to estimate the feature weights. our method, named summscore, is lightweight, fast and easy to use as it does not rely on a neural network. since it is purely unsupervised, the re-ranked results can provide more refined self-supervision to the pre-trained models, complementing the pretraining with rounds of self-training. our contributions in this paper are threefold: ## related work unsupervised abstractive summarization in unsupervised abstractive summarization, summae @xcite proposes to auto-encode paragraphs with a sequence-to-sequence model and decode single-sentence summaries from the latent embeddings. seq3 @xcite also uses an auto-encoder to compress the input then reconstruct it into a differentiable manner, the encoder output serving as a summary. however, both methods stick to unsupervised sentence summarization. more recent approaches typically rely on language models being pre-trained, then used in a zero-shot fashion. pegasus (zhang et al., 2020) treats salient sentences as pseudo abstractive targets to build a pre-training objective. ted (yang et al., 2020) exploits the lead bias in news articles and takes out the first sentences of the document as pseudo summary targets for pre-training. due to their pre-training objective built for summary generation, these pre-trained models can be directly used for unsupervised summarization. the summary loop (laban et al., 2020) uses reinforcement learning to train a model to fill-in deleted important words from the source document using the summary generated so far, then refines this summary. re-ranking in abstractive summarization second-stage or sequence-level methods are gaining traction recently in supervised summarization. among such methods, re-ranking consists in selecting a better summary candidate out of several of them produced by a base model (which has already been fine-tuned). refsum (liu et al., 2021) uses a meta-learning approach to learn how to rank summaries coming from multiple systems. simcls (liu and liu, 2021) trains a roberta (liu et al., 2019b) model with a ranking loss to learn how to rank summary candidates generated by a base bart or pegasus in their target metric order. summareranker (ravaut et al., 2022a) also trains a roberta re-ranker, but this time in a multi-label binary classification manner to predict whether each summary candidate maximizes each of the metrics of interest. to avoid using another neural network for re-ranking, brio (liu et al., 2022b) performs a second fine-tuning stage with the re-ranking loss built in the base summarization system. each of the four models above improves the sota on the cnn/dm benchmark, reaching 47.78 rouge-1 for brio. to the best of our knowledge, there is no work on sequence-level unsupervised abstractive summarization. concurrently to our work, mbrd (suzgun et al., 2022) proposes to rank generated candidates in several generation tasks using majority voting based on bertscore (zhang et al., 2019). ## conclusion we introduced summscore, the first unsupervised abstractive summarization re-ranking system. summscore does not rely on a neural network: instead, it builds features for each summary candidate, some of them using the source as well, and aggregates them into a final re-ranking score. feature coefficients are estimated through tuning against a pseudo-label derived from the source document. it is a simple framework which easily supports the addition of new features. summscore significantly improves the performance of the base summarization model, in terms of rouge, bertscore, factual consistency, and human preference ; in both unsupervised and zeroshot transfer scenarios. moreover, summscore selected summary candidate naturally extends into a self-training objective for abstractive summarization, which improves unsupervised summarization.