R ewards O f S um: Exploring Reinforcement Learning Rewards for Summarisation (2021)

Abstract: To date, most abstractive summarisation models have relied on variants of the negative log-likelihood (NLL) as their training objective. In some cases, reinforcement learning has been added to train the models with an objective that is closer to their evaluation measures (e.g. ROUGE). However, the reward function to be used within the reinforcement learning approach can play a key role for performance and is still partially unexplored. For this reason, in this paper, we propose two reward functions for the task of abstractive summarisation: the first function, referred to as RwB-Hinge, dynamically selects the samples for the gradient update. The second function, nicknamed RISK, leverages a small pool of strong candidates to inform the reward. In the experiments, we probe the proposed approach by fine-tuning an NLL pre-trained model over nine summarisation datasets of diverse size and nature. The experimental results show a consistent improvement over the negative log-likelihood baselines.

## introduction the current state-of-the-art neural text summarisation models have been refined to excel at either the extractive or abstractive styles, or even both @xcite @xcite . along with contemporary summarisation datasets @xcite @xcite , the advent of large pre-trained language models, and their subsequent derivations @xcite , has allowed summarisation to become a more practical and reasonable task to implement, without compromising, and often improving, the accuracy. however, these models usually employ the standard negative loglikelihood (nll) as their training objective, which aims to maximise the likelihood of each token in a given ground-truth reference. despite its efficacy, the nll fails to account for synonymous tokens and other potentially valid variations, and strongly biases the model towards the ground-truth reference @xcite . furthermore, the nll operates as a token-level objective during training, which promotes an inconsistent comparison with sequence-level evaluation metrics, such as rouge @xcite . in order to address the inconsistency between token-level training and sequence-level evaluation, reinforcement learning (rl) has been adopted in summarisation and other language generation tasks to afford the optimization of sequence-level metrics during training @xcite . reinforcement learning has proved successful at improving the accuracy of language generation tasks, such as summarisation @xcite @xcite and machine translation @xcite . however, balancing exploration and exploitation remains imperative to the successful choice of an effective reward. when standard rl techniques, such as reinforce @xcite , are implemented in natural language generation tasks, the required expectation becomes intractable due to large vocabulary sizes. therefore, the application of reinforce is typically reduced to calculating the approximate expectation with respect to only a single predicted sequence. to teach the model to understand the importance of sample variation among synonymous tokens, we instead choose to implement an objective function which includes multiple predicted sequences, allowing for a scenario in which several valid candidate summaries can be considered. another consideration is that the success of techniques such as reinforce strongly depends on the use of an effective and appropriate reward. designing such a reward, one which enables the model to manipulate multiple sequences and yet provides a positive and informative outcome in the process, is therefore necessary for producing better results. this allows us to modify the reinforcement learning framework in such a way that enforces only a higher weighting to those predicted sequences which obtain a higher reward. as such, we apply two techniques to summarisation; rwb-hinge, which applies a hinge-loss modification to the classical reinforce with baseline @xcite to selectively apply the model gradients, and expected risk minimization (risk) @xcite , which leverages a small pool of strong sampled candidates to smartly inform the reward function. we aptly refer to our framework as rewardsofsum, to hint at the exploration of suitable reward functions for summarisation. empirically, we show that the two proposed variants perform better than standard negative log-likelihood baselines over a range of datasets of diverse size and nature. ## related work in recent years, there has been some work in summarisation to separate from the traditional negative log-likelihood (nll) objective function, and mollify its dependency on ground-truth references. several implementations of reinforcement learning in summarisation involved optimizing discrete metrics, such as the standard rouge @xcite . others have introduced novel rewards into the reinforcement learning framework, such as question-focused rewards (arumae and @xcite , saliency and entailment rewards @xcite , and even distributional semantic rewards @xcite . @xcite also present a novel unsupervised metric for summarisation which correlates highly with discrete evaluation metrics if adopted in a reinforcement learning approach. on the other hand, there has been much work in leveraging large, pre-trained language models (lm) @xcite @xcite to improve the quality and performance of summarisation models. utilizing pretrained language models requires significantly less engineering effort to continually improve over stateof-the-art baselines. typically, these approaches include using novel pre-training objectives @xcite @xcite or implementing successful reinforcement learning techniques @xcite . @xcite found that optimizing semantic rewards in reinforcement learning, using bertscore @xcite , does not necessarily correlate with the rouge score at test time. as such, the choice of reward in a reinforcement learning approach should attempt to carefully align with the evaluation metric. how best to inform the reward via the reward function, is critical to the performance of models in an rl framework. in our work, we aim to stray from the typical sole nll objective, and by leveraging a pre-trained language model in a reinforcement learning framework, explore different rl-based reward functions for summarisation. ## proposed reinforcement learning training in order to improve over the negative log-likelihood baseline models, we aim to implement a reinforcement learning framework that adopts the standard evaluation metric, rouge, as a reward during training. we aim to keep consistent with previous implementations of reinforcement learning in summarisation, and assume rouge-l f1 to be the reward metric in the following work. in sections 3.1 and 3.2, we consider the following standard notations: x is defined as an input source document, y * , ŷ, and y s are referred to as the ground-truth reference, argmax prediction, and sampled sequence, respectively, and r(y) refers to the reward of sequence y, computed with respect to the ground-truth reference, y * . by exploiting a combination of sampling and predictions, we aim to enhance training diversity in the vein of the work of @xcite ; @xcite ; holtzman et al. (2020). ## results figure 1 compares the effect that each finetuning method has had over the production of novel n-grams during test time (a property nicknamed as n-gram novelty). for medium sized datasets in particular, the reinforcement learning approaches appear to, on average, facilitate the production of more distinct uni-, bi-, and tri-grams at test time, compared to the nll baseline. whilst n-gram novelty is typically used in summarisation to showcase test-time summary abstractiveness, the results in figure 1 highlight that training with objectives that promote sample variation leads to models capable of producing more novel n-grams (up to 13.8 pp in tri-gram novelty over cnn/dm). this is supported by the qualitative example in in addition, figure 2 shows a performance comparison with respect to the length of the reference summaries for the full-data approach over a medium size dataset (cnn/dm). we see that our fine-tuning methods have led, on average, to higher rouge-l scores for the longer summaries (up to 2.3 rouge-l points for summaries between 80-100 tokens, and up to 6.2 points for summaries over 100 tokens). likely, the proposed methods have been able to amend the reported tendency of the nll models to curtail the prediction of long summaries. ## conclusion in this paper, we have proposed two variants to the reinforcement learning approaches typically used in sequence-to-sequence learning tasks. the two proposed approaches -nicknamed rwb-hinge and risk -have been designed to improve the reinforcement learning rewards by selecting and diversifying the predictions used during the fine-tuning of the model. in a set of automated summarisation experiments over nine, diverse datasets, the approaches have consistently led to improved performance, and also diversified the generated summaries. we note that, despite its commonplace use for summarisation evaluation, utilizing rouge as reinforcement learning reward does not easily translate into improved performance. for this reason, in the near future we plan to explore other contemporary score functions, such as bertscore @xcite , in an attempt to build more effective rewards.