BRIO : Bringing Order to Abstractive Summarization (2022)

Abstract: Abstractive summarization models are commonly trained using maximum likelihood estimation, which assumes a deterministic (one-point) target distribution in which an ideal model will assign all the probability mass to the reference summary. This assumption may lead to performance degradation during inference, where the model needs to compare several system-generated (candidate) summaries that have deviated from the reference summary. To address this problem, we propose a novel training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality. Our method achieves a new state-of-the-art result on the CNN/DailyMail (47.78 ROUGE-1) and XSum (49.07 ROUGE-1) datasets. Further analysis also shows that our model can estimate probabilities of candidate summaries that are more correlated with their level of quality.

## introduction neural methods for abstractive summarization @xcite @xcite @xcite formulate summarization as a sequenceto-sequence (seq2seq) problem @xcite , learning to generate the summary in an autoregressive manner. such models are commonly trained with maximum likelihood estimation (mle), maximizing predictive probability of the reference output given the gold sub-sequence before it. however, during inference the model must also generate the output based on possibly erroneous previous steps. this can hurt model performance, a phenomenon often called exposure bias @xcite . to maintain reasonable performance even in the case of a sub-sequence with errors, we argue that the 1 we have made our code, results, and trained models publicly available at https://github.com/yixinl7/brio . model must accurately estimate relative quality of different generated outputs, since effective inference requires comparison among these candidates. to understand whether existing models can accurately perform such relative comparisons, we conducted a preliminary study on pre-trained bart @xcite , first generating two candidate summaries from the model and observing whether a higher probability is assigned to the candidate with a higher rouge @xcite score. as tab. 1 shows, the accuracy is far from ideal. this is likely due to the fact that mle training only encourages the model to assign high probability to the reference summary, and is agnostic about any relative comparison between non-reference summaries. however, we argue that it is also important for the order of model scores to be coordinated with the actual quality metrics by which the summaries will be evaluated -higher model scores should indicate better quality summaries. in the following we will refer to models that have such scores as "coordinated" for conciseness. we introduce a training paradigm which requires the abstractive model to be able to be accurate with respect to predicting the tokens in the reference summaries and coordinated with respect to encoder decoder 𝒑 𝟏 𝒑 𝟐 𝒑 𝟑 𝒑 𝟒 source input reference output ℒ 𝑀𝐿𝐸 encoder decoder 𝒑 𝟏 (𝒃) 𝒑 𝟐 (𝒃) 𝒑 𝟑 (𝒃) 𝒑 𝟒 (𝒃) source input candidate output 𝐵 decoder 𝒑 𝟏 (𝒂) 𝒑 𝟐 (𝒂) 𝒑 𝟑 (𝒂) 𝒑 𝟒 (𝒂) candidate output 𝐴 ℒ 𝐶𝑡𝑟 𝑀(𝐴) 𝑀(𝐵) -> seq2seq generation model reference-free evaluation model 𝜃 𝜃 figure 1: comparison of mle loss (lmle) and the contrastive loss (lctr) in our method. mle assumes a deterministic (one-point) distribution, in which the reference summary receives all the probability mass. our method assumes a nondeterministic distribution in which system-generated summaries also receive probability mass according to their quality. the contrastive loss encourages the order of model-predicted probabilities of candidate summaries to be coordinated with the actual quality metric m by which the summaries will be evaluated. we assign the abstractive model a dual role -a single model could be used both as a generation model and a reference-free evaluation model. the candidate summaries. in other words, we give the abstractive model a dual role: as a generation model, it generates the output summaries in an autoregressive way; as an evaluation model, it can be used to score the quality of candidate summaries by estimating a probability distribution over candidate outputs. the generation model is trained using the standard mle loss, but to train the evaluation model we introduce a contrastive loss @xcite defined over different candidate summaries generated by pre-trained abstractive models (fig. 1 ), following previous work on ranking-based or contrastive learning @xcite @xcite . our main contribution is to change the target distribution of abstractive models from a one-point deterministic distribution assumed by mle training to a non-deterministic distribution in which candidate summaries are also assigned probability mass according to their quality. the new sota performance on cnn/dailymail @xcite and xsum @xcite datasets demonstrated the effectiveness of our method. our in-depth analysis also found that the abstractive models trained using our method can estimate the candidate summary quality more accurately, in concert with the the objective of our training paradigm. ## neural abstractive summarization the goal of abstractive summarization is to create a function g that takes a source document d and generates an appropriate summary s s ← g(d) training objective neural abstractive summarization models aim to learn a neural model g that results in good summaries. maximum likelihood estimation (mle) is the standard training algorithm. it aims to maximize the likelihood of the reference summary s * , i.e., in practice, label smoothing @xcite is a widely used and effective technique that modifies the target distribution in eq. 4 to a "soft" label by assigning probability mass β to other tokens: one important step in search is estimating the probability of the next word s t given the previous predicted sequence @xmath0: comparing eq. 6 with eq. 3, the major difference is that during inference the model makes new predictions based on its own previous predictions @xmath1. as a result, even if the generation model g achieves very high accuracy w.r.t. eq. 3, once @xmath2, there is the risk that the performance of g will significantly degrade. this problem has been identified as the exposure bias @xcite . ## coordinating abstractive models eq. 6 implies that the abstractive model g should be able to assign higher estimated probability to the better candidate summary during inference. however, this intuition is not directly captured in the standard mle objective used in training -a model obtaining zero mle loss would assign zero probability to any candidate summary different from the reference. this is obviously improper for any task where multiple reasonable generations may exist @xcite , and also does not say anything about the ordering of two imperfect references. we therefore advocate for making the alternative assumption that the probability of one candidate should be well-correlated with its quality as evaluated by an automatic metric m . since it is intractable to enumerate all the possible candidate outputs, we only require our model to be able to accurately predict the ranking order of a set of the most probable candidate summaries ŝ, which are its own beam search results. in order to achieve this objective, we slightly modify the conditions of eq. 5, maintaining the general functional form, but instead specifying the marginal probability of the non-reference candidates s to be β, and encouraging coordination of probabilities and qualities among non-reference candidates as follows: we next describe precisely how we encourage coordination through contrastive learning. contrastive learning for coordination the candidate quality measure m can be defined in many ways. in this work we define it as the rouge @xcite ) score of a candidate summary s i given the reference summary s * . to coordinate a pre-trained abstractive model, we 1) use it to generate different candidate summaries with various levels of quality, foot_0 then 2) encourage the model to assign higher estimated probabilities to better candidates by fine-tuning the model with a contrastive loss, following the previous work @xcite : ## related work training methods of seq2seq models in order to align the training objective and evaluation metric, structured losses have been used for the seq2seq model training. among them, marginbased losses @xcite @xcite , which require the model to assign higher probability to the better output, are a major category. many margin-based losses used in modern seq2seq models @xcite assume a deterministic (one-point) distribution: a model can achieve zero loss if it can assign a much higher probability to the (pseudo)-reference, regardless of relative comparisons of other candidate summaries. by contrast, our method has a non-deterministic assumption (eq. 7), which focuses on the pair-wise ranking of a set of candidate summaries. one main challenge of directly optimizing a seq2seq model with quality scores of the output is that the discrete sampling process makes the loss non-differentiable. to circumvent this problem, reinforcement learning has been used to reformulate the conditional text generation tasks @xcite @xcite @xcite . compared to this school of methods, our method is based on supervised learning, and it is more stable and less sensitive to the design choices (e.g. reward shaping), which are well-known challenges of reinforcement learning methods. minimum risk training @xcite and other online sampling based methods @xcite @xcite belong to another school of methods used to circumvent the problem of non-differentiability. however, they also exhibit similar problems of stability as reinforcement learning. contrastive learning recently, contrastive learning @xcite has been introduced into several conditional text generation tasks, such as machine translation @xcite ), text summarization @xcite @xcite , and other tasks @xcite @xcite . among these application scenarios, most work deployed contrastive learning in the latent representation space, following the framework proposed in @xcite . however, in this work we adopt contrastive learning over the discrete space of the generated texts. besides, instead of constructing the contrastive learning examples by rule-based methods (e.g. perturbing the reference output), we use the generation models to construct the examples, which makes the contrastive learning task closer to the generation task. sun and li (2021) also adopted contrastive learning on the generated texts. however, their formulation belongs to the margin-based losses. we have discussed the difference between our method and the margin-based losses in the previous paragraphs. discriminative reranking discriminative reranking has been widely studied for conditional generation tasks @xcite @xcite . some recent works @xcite have also explored discriminative reranking of candidates from neural natural language generation models, which adopt large pre-trained language models (e.g. bert @xcite ) as the reranker. in this work, we factorize the seq2seq model (e.g., bart) trained on the same dataset as the reranking model, which maximizes the parameter sharing across two stages. besides, our approach contributes an instance of leveraging large pre-trained seq2seq models as a quality estimation model @xcite . ## conclusion and future work in this work, we presented a new training paradigm that assigns candidate outputs probability mass according to their quality using contrastive learning. while our method has achieved significant improvement on abstractive summarization, we note several directions for the future work to explore. first, since our method makes no assumptions specifically about the summarization task, it can be extended to other conditional text generation tasks such as machine translation. second, it is possible to apply our method in a reinforcement learning setting, where the candidate summaries are dynamically generated. finally, in experiments we only used diverse beam search to generate the candidate summaries, but it is likely that other candidate generation methods could yield further improvements.