C o L o: A Contrastive Learning Based Re-ranking Framework for One-Stage Summarization (2022)

Abstract: Traditional training paradigms for extractive and abstractive summarization systems always only use token-level or sentence-level training objectives. However, the output summary is always evaluated from summary-level which leads to the inconsistency in training and evaluation. In this paper, we propose a Contrastive Learning based re-ranking framework for one-stage summarization called CoLo. By modeling a contrastive objective, we show that the summarization model is able to directly generate summaries according to the summary-level score without additional modules and parameters. Extensive experiments demonstrate that CoLo boosts the extractive and abstractive results of one-stage systems on CNN/DailyMail benchmark to 44.58 and 46.33 ROUGE-1 score while preserving the parameter efficiency and inference efficiency. Compared with state-of-the-art multi-stage systems, we save more than 100 GPU training hours and obtaining 3x 8x speed-up ratio during inference while maintaining comparable results.

## introduction in general, there are two main paradigms to do text summarization: abstractive @xcite @xcite and extractive @xcite @xcite methods. for extractive summarization, previous studies @xcite formulate it as a sentence-level sequence labeling task. however, there is an inherent gap between the sentence-level scoring and the summary-level evaluation @xcite .this means that some high-scoring sentences may share the same meaning, making them not a qualified summary when combined. similarly, the previous training paradigm for abstractive summarization models can be viewed as a token-level scoring process ⇤ corresponding author. to tackle this problem, state-of-the-art summarization systems @xcite are enhanced with an additional module (called re-ranker) and follow a two-stage paradigm. they first train a summarizer to model the conditional distribution p(y |x) where x is the document and y is the output summary. then the re-ranker is trained to re-score candidates sampled from the pre-trained summarizer in the second stage. however, this paradigm trades efficiency for accuracy, the auxiliary re-ranking greatly harms the inference efficiency especially for the highly efficient extractive systems. experimentally, the decoding speed of two-stage re-ranking models is only ~7.0 samples/s while removing the re-ranker module will greatly boost the decoding speed to ~42.0 samples/s foot_0 . this makes two-stage summarization systems may be unacceptable in real-world scenarios that require timely feedback. the limitations of the existing work motivate us to build a one-stage summarization system that can 1) replace previous naive sentence/token-level score with a summary-level score and 2) do not sacrifice the parameter and inference efficiency. in this paper, we propose a contrastive learning based re-ranking framework for one-stage summarization called colo for both extractive and abstractive approach. contrastive learning has been explored in summarization @xcite and generation @xcite . colo uses a contrastive re-ranking training objective. we first present a novel sampling method that can be equipped to any one-stage summarization systems so that it can re-score candidates without the second stage. the existing two-stage models use offline sampling to preprocess samples for training of re-ranker where candidate samples are drawn from a fixed model distribution. this is a huge obstacle to turning summarize-then-rerank two-stage framework into an efficient end-to-end model. to solve this issue, we propose an online sampling approach. concretely, instead of sampling from a fixed distribution, we draw positive and negative samples from a dynamic distribution of model outputs during training, which ultimately eliminates the requirement for additional modules in the overall framework. we then introduce a summary-level optimization strategy in addition to the traditional sentence-level (for extractive systems) or token-level loss (for abstractive systems). as a result, as a one-stage model, colo achieves comparable performance to two-stage systems, and greatly improves decoding speed to meet the needs of real-world applications. we summarize our contributions as follows: ## results we denote the model without contrastive learning as the baseline system. since the backbone of our extractive model is bart encoder so that we call the baseline model bartext. the baseline model for abstractive system is bart. our extractive model is called colo ext and its abstractive version is denoted as colo abs . ## limitations and future work compared with the most well-known contrastive learning framework simclr @xcite which propose to construct positive and negative pairs from training samples in the same batch, drawing negative-positive pairs from the summarization model requires more training time. ideally, providing more positive and negative samples will benefit the performance of colo . however, decoding with very large beam size in training mode will cost more gpu memory and training time. future work can search for an efficient way to construct these positive-negative pairs to perform reranking during training. ## conclusion we introduce colo, a contrastive learning based summarization framework for one-stage summarization where positive-negative pairs are generated directly from the summaizer with online sampling. colo can be both easily applied on extractive and abstractive methods. results show that we greatly exceed previous stage-of-the art one-stage systems with no additional parameters and obivious decline of the inference efficiency.