Demoting the Lead Bias in News Summarization via Alternating Adversarial Learning (2021)

Abstract: In news articles the lead bias is a common phenomenon that usually dominates the learning signals for neural extractive summarizers, severely limiting their performance on data with different or even no bias. In this paper, we introduce a novel technique to demote lead bias and make the summarizer focus more on the content semantics. Experiments on two news corpora with different degrees of lead bias show that our method can effectively demote the model’s learned lead bias and improve its generality on out-of-distribution data, with little to no performance loss on in-distribution data.

## introduction neural extractive summarization, which produces a short summary for a document by selecting a set of representative sentences, has shown great potential in real-world applications, including news @xcite and scientific paper summarization @xcite . typically, a generalpurpose extractive summarizer learns to select the most important sentences from a document to form the summary by considering their content salience, informativeness and redundancy. however, when restricted to a specific domain, the summarizer can learn to exploit particular biases in the data, the most famous of which is the lead bias in news @xcite ; namely that sentences at the beginning of a news article are more likely to contain summary-worthy information. as a result, not surprisingly, such bias is strongly captured by neural extractive summarizers for news, for which the sentence positional information tends to dominate the actual content of the sentence in model prediction @xcite @xcite . while learning a summarizer reflecting the biases in the training dataset is completely fine when the summarizer is going to be deployed to summarize documents having similar biases, it would be problematic when the model was applied to deal with documents coming from a mixture of datasets with different degrees of such biases. in this paper, we address this problem in the context of the lead bias in the news domain by exploring ways in which an extractive summarizer for news can be trained so that it learns to balance the lead bias with the content of the sentences, resulting in a model that can be applied more effectively when the target documents belong to news datasets in which the lead bias is present in rather different degrees. recently, @xcite proposes two preliminary solutions. one is to pretrain the summarizer on an automatic generated "unbiased" corpus where the document sentences are randomly shuffled, which however has the negative effects of preventing the learning of inter-sentential information. the other, which can be only applied to rl-based summarizers, is to add an explicit auxiliary loss to directly balance position with content. alternatively, @xcite and @xcite investigate strategies to train the summarizer on multiple news datasets with different degrees of lead bias, but this may still be problematic when we apply the trained summarizer to the documents with lead bias not covered in the training data. outside the summarization area, methods have also been proposed to eliminate data biases for other nlp tasks like text classification or entailment @xcite @xcite . inspired by @xcite , we have developed an alternating adversarial learning technique to demote the summarizer lead bias, but also maintain the performance on the in-distribution data. we introduce a position prediction component as an adversary, and optimize it along with the neural extractive summarizer in an alternating manner. furthermore, in contrast with @xcite al. (2019) and wang et al. (2019) , our proposal is model-independent and only requires one type of news dataset as training input. in this paper, we apply our proposed method to a biased transformer-based extractive summarizer @xcite trained on cnn/dm training set @xcite and conduct experiments on two test sets with different degrees of lead bias: cnn/dm and xsum @xcite , for in-distribution and generality evaluation respectively. the experimental results indicate that our proposed "debiasing" method can effectively demote the lead bias learned by the neural news summarizer and improve its generalizability, while still mostly maintaining the model's performance on the data with a similar lead bias. ## proposed debiasing method our method aims to demote the lead bias learned by the summarizer and encourage it to select content based more on the semantics covered in sentences. as shown in figure 1 , our method comprises two components: one for summarization (red) and the other for sentence position prediction (green). ## conclusion and future work we propose a lead bias demoting method to make news extractive summarizers more robust across datastets, by optimizing a position prediction and a summarization component in an alternating way. experiments indicate that our method improves model's generality on out-of-distribution data, while still largely maintaining its performance on in-distribution data. as such, it represents the best viable solution when at inference time input documents may come from an unknown mixture of datasets with different degrees of position bias. for the future, we plan to explore more sophisticated and effective methods (e.g., adjusting the lead bias online) and infuse them together with neural abstractive summarization models, known to generate more succinct and natural summaries. another interesting direction for future work can be exploring the potential of applying our proposed bias demoting strategy to other tasks, which can also be framed as the sequence labeling problem and possibly troubled by biases in the training data (e.g., topic segmentation @xcite and semantic role labeling @xcite ).