Balancing Lexical and Semantic Quality in Abstractive Summarization (2023)

Abstract: An important problem of the sequence-to-sequence neural models widely used in abstractive summarization is exposure bias. To alleviate this problem, re-ranking systems have been applied in recent years. Despite some performance improvements, this approach remains underexplored. Previous works have mostly specified the rank through the ROUGE score and aligned candidate summaries, but there can be quite a large gap between the lexical overlap metric and semantic similarity. In this paper, we propose a novel training method in which a re-ranker balances the lexical and semantic quality. We further newly define false positives in ranking and present a strategy to reduce their influence. Experiments on the CNN/DailyMail and XSum datasets show that our method can estimate the meaning of summaries without seriously degrading the lexical aspect. More specifically, it achieves an 89.67 BERTScore on the CNN/DailyMail dataset, reaching new state-of-the-art performance. Our code is publicly available at https://github.com/jeewoo1025/BalSum .

## introduction the performance of sequence-to-sequence (seq2seq) neural models for abstractive summarization @xcite @xcite ) has improved significantly. the dominant training paradigm of seq2seq models is that of maximum likelihood estimation (mle), maximizing the likelihood of each output given the gold history of target sequences during training. however, since the models generate the sequence in an auto-regressive manner at inference, the errors made in the previous steps accumulate in the next step thereby affecting the entire sequence. this phenomenon is known as exposure bias @xcite . to mitigate this problem, re-ranking systems @xcite @xcite have recently been introduced to generate a more appropriate summary. there are two training objectives for applying reranking to abstractive summarization: contrastive learning and multi-task learning. the contrastive learning-based approaches deploy margin-based losses. simcls @xcite and brio-ctr @xcite train a large pre-trained model, such as roberta @xcite and bart @xcite , to align the candidate summaries according to the quality. the authors use the rouge @xcite score as a quality measurement. the multi-task learning-based approaches combine at least two losses that perform different roles. summareranker @xcite minimizes the average over the binary cross-entropy losses optimized for each evaluation metric. in addition, brio-mul @xcite demonstrates that the combination of the contrastive and cross-entropy loss works complementarily and has better performance. in this paper, we analyze the three main drawbacks of existing re-ranking approaches. first, we argue that current methods focus excessively on ranking summaries in terms of lexical overlap. inspired by zhong et al. (2020), we conduct a preliminary study, by sorting candidate summaries in descending order based on the rouge score and then defining z as the rank index of the highest bertscore summary. as demonstrated in fig. 1 , we can observe that there is a large gap between lexical overlap and semantic similarity. in a majority (52%) of cases @xmath0. second, despite more than half of the candidates with the same rouge score, previous studies do not accurately reflect quality measurements as they are trained with different ranks even if they have equal scores (appendix f). lastly, for the first time, we find summaries with high lexical overlap but low semantic similarity as false positives (appendix g). they can be noises during training phrase, which are not considered substantially in the prior works. to address these issues, we propose a novel training method in which a re-ranker balances lexical and semantic quality. based on a two-stage framework, our model, named balsum, is trained on multi-task learning. we directly reflect the rouge score difference on a ranking loss to preserve the lexical quality as much as possible. then, we use a contrastive loss with instance weighting to identify summaries whose meanings are close to the document. specifically, we define novel false positives (semantic mistakes) and present a strategy to reduce their influence in ranking. experiments on cnn/dm and xsum datasets demonstrate the effectiveness of our method. notably, balsum achieves an 89.67 bertscore on cnn/dm, reaching a new state-of-the-art performance. ## method our method follows the two-stage framework. given a source document d, a function g is to generate a pool of candidate summaries @xmath1, c 2 , ..., c m } at the first stage: then, a function f is to assign scores to each candidate and select the best summary c * with the highest score at the second stage: our goal is to train the ranking model f that identifies the correct summary from the outputs of the generation model g. ## conclusion in this work, we propose balsum which aims to evaluate summaries by considering the balance between lexical and semantic quality. to achieve this, we perform a multi-task learning, which aligns summaries according to their lexical overlap qualities and identifies whether they are similar to the document. in addition, to our best knowledge, our method is the first attempt to present a new perspective of false positives (semantic mistakes) in ranking and creating the model to reduce their in-fluence. our experimental results and fine-grained analyses validate that our model achieves consistent improvements over competitive baselines.