Rewards with Negative Examples for Reinforced Topic-Focused Abstractive Summarization (2021)

Abstract: We consider the problem of topic-focused abstractive summarization, where the goal is to generate an abstractive summary focused on a particular topic, a phrase of one or multiple words. We hypothesize that the task of generating topic-focused summaries can be improved by showing the model what it must not focus on. We introduce a deep reinforcement learning approach to topic-focused abstractive summarization, trained on rewards with a novel negative example baseline. We define the input in this problem as the source text preceded by the topic. We adapt the CNN-Daily Mail and New York Times summarization datasets for this task. We then show through experiments on existing rewards that the use of a negative example baseline can outperform the use of a self-critical baseline, in Rouge, BERTScore, and human evaluation metrics.

## introduction topic-focused summarization is the task of generating a summary given a source text and a specific query or topic. approaches to topic-focused summarization include query relevance and importance @xcite , multi-modality manifold ranking @xcite @xcite , and query attention @xcite . @xcite @xcite are examples of datasets that are widely used for this task. these datasets are much smaller than benchmark datasets for generic summarization, resulting in fewer research work to train topicfocused summarization on state-of-the-art systems @xcite . in parallel, there has been growing work in recent years on reinforcement learning approaches to (generic) abstractive summarization. proposed rewards aim to optimize non-differentiable summarization metrics like rouge @xcite and bertscore @xcite , or to encourage desirable summary aspects like semantic cohe-sion @xcite and entity coherence @xcite . many reinforced abstractive summarization methods use the self-critical baseline or @xcite to cap their rewards. this self-critical baseline is obtained by greedily searching for a sequence that maximizes the likelihood probability of the current model. in this work, we propose a reinforcement learning-based approach to topic-focused summarization. first, we adapt widely used generic summarization benchmarks to this task, such that we aim to generate only one out of three summary sentences, given a corresponding topic. then, instead of using the self-critical baseline, we introduce a novel baseline that uses negative examples: a sentence that contains information that the summarization model should not focus on. we run experiments on two existing generic summarization datasets adapted to our task: cnn-daily mail @xcite and new york times @xcite . our experiments span two existing rewards: the popular rouge-l reward and the distributed semantic reward (dsr) of @xcite , inspired by bertscore @xcite . our results show that using our negative example baseline outperforms the self-critical baseline across both datasets and both rewards. we obtain improvements on both datasets in rouge and bertscore metrics, and human annotators find that summaries generated with our negative baseline for rewards are generally more relevant to the given topic. ## problem statement we tackle the task of topic-focused abstractive summarization as the problem of producing an abstractive summary focused on a given topic, a phrase of one or multiple words. the generated summary should include information from the input text that is related to the topic, and exclude all other infor-mation. consequently, different topics with the same input text should yield different summaries. more formally, given an input text x, a topic t and a corresponding reference summary y, we aim to maximize the probability that we generate the right summary: ## rewards with a negative example in self-critical sequence learning @xcite , the rl loss formula is as follows for our task: in our particular task, we propose to use a baseline with a negative example. the intuition is that we encourage the model to generate summaries that are more similar to the reference summary than the negative example. this negative example is an independent sentence from the summary of the corresponding source text, but which does not contain the topic. the negative example acts as a sample of undesirable information, and helps the summarization model learn what kind of information to exclude. given a reference summary y, and a negative summary ȳ, our rl loss with negative examples is defined as in equation 2, where we define the rl baseline: @xmath0. ## conclusions we propose a deep reinforcement learning approach to topic-focused abstractive summarization, where we aim to generate summaries focused on a given phrase of one or multiple words. we introduce a new baseline for rewards, based on negative examples collected from independent summary sentences. we show through experiments that our proposed approach outperforms the baseline of selfcritical reinforcement learning in the optimized reward metric, and human annotators find our model generates summaries that are more relevant to the topic.