A Multi-Document Coverage Reward for RELAX ed Multi-Document Summarization (2022)

Abstract: Multi-document summarization (MDS) has made significant progress in recent years, in part facilitated by the availability of new, dedicated datasets and capacious language models. However, a standing limitation of these models is that they are trained against limited references and with plain maximum-likelihood objectives. As for many other generative tasks, reinforcement learning (RL) offers the potential to improve the training of MDS models; yet, it requires a carefully-designed reward that can ensure appropriate leverage of both the reference summaries and the input documents. For this reason, in this paper we propose fine-tuning an MDS baseline with a reward that balances a reference-based metric such as ROUGE with coverage of the input documents. To implement the approach, we utilize RELAX (Grathwohl et al., 2018), a contemporary gradient estimator which is both low-variance and unbiased, and we fine-tune the baseline in a few-shot style for both stability and computational efficiency. Experimental results over the Multi-News and WCEP MDS datasets show significant improvements of up to +0.95 pp average ROUGE score and +3.17 pp METEOR score over the baseline, and competitive results with the literature. In addition, they show that the coverage of the input documents is increased, and evenly across all documents.

## introduction multi-document summarization (mds) aims to consolidate salient points of information across a set of documents into a concise summary. the main requirement for the summary is that it adequately represent the document set, with low redundancy and high coverage across all documents, while at the same time being readable and fluent. combined with this, is the need to develop techniques that can handle the significant memory complexity required to tackle mds. recently, the re-lease of dedicated datasets @xcite , and intelligently designed transformer models @xcite @xcite , have helped drive advancements in multi-document summarization, generally improving the accuracy and fluency of the predicted summaries. however, aspects such as the requirement to cover as much salient information from the input documents as possible, whilst still maintaining low repetition and low redundancy, have certainly been less explored to date @xcite . within the sphere of contemporary neural mds models, two main lines of investigation can be identified: graph-based approaches @xcite , and concatenation approaches @xcite . the former are approaches that rely on the construction of graphs to capture the inter-and intra-document relations. while powerful, they need to elicit the relations explicitly. the latter instead assume that all the input documents within a document set can be simply concatenated, possibly with document separators and tags, such that the relations can be "discovered" by the model. like ordinary summarization, also mds comes in two remarkably different styles: extractive, where the generated summaries consist of verbatim sentences from the original input documents @xcite , and abstractive, where the model is instead encouraged to generate a paraphrased understanding of the input documents. the intrinsic appeal of abstractive summaries and the advent of sequenceto-sequence models have increasingly shifted the trend toward abstractive summarization @xcite @xcite @xcite . as for what models are concerned, abstractive mds has made increasing use of transformers, both "conventional" @xcite and modified to accommodate the characteristic input length of multi-document sets @xcite . similarly to general summarization, the majority of mds models are trained using the negative log-likelihood (nll) as training objective, which aims to maximize the conditional log-likelihood of the tokens of a given reference summary. despite its speed and efficacy, the nll exhibits both the wrong-objective problem @xcite , where the model is trained on a convenient objective rather than a desirable one, and the wellknown exposure bias problem @xcite . to alleviate these issues, reinforcement learning has been adopted in summarization, as in other language generation tasks, to train the model with a more appropriate objective @xcite . however, its effective use for mds requires a reward function that can appropriately balance the reference summary and the multiple input documents in the document set. for this reason, in this paper we propose exploring a reward that combines a referencebased metric such as rouge with a coverage term over the input documents. to implement the reinforcement learning approach, we employ a contemporary gradient estimator of the policy gradient, relax @xcite , which is both low-variance and unbiased. in addition, to limit the computation and the risk of parameter drift, we apply the objective to fine-tune an nll-pretrained model in a few-shot manner. in light of the above, this paper makes the following contributions: the rest of this paper is organized as follows: first the related work is reviewed in section 2, and then the proposed approach is introduced in section 3. section 4 describes the experimental set-up and main results, while section 5 presents a more detailed analysis of the main components of the proposed approach. eventually, section 6 summarizes our findings and concludes the paper. ## related work early work in multi-document summarization (mds) that pre-dates the neural era @xcite @xcite was shaped around the notion of mds as a collection of graph structures. as approaches in language generation naturally evolved into neural-based @xcite , later improved with the emergence of large, pre-trained language models @xcite @xcite , the effort shifted to integrating these graph structures into the models, often building on top of strong single-document summarization (sds) baselines @xcite . concurrently, the growing interest in multidocument summarization has led to the development of dedicated, multi-document datasets such as wikisum @xcite , multi-news @xcite , wikipedia current events portal (wcep) @xcite and others. the typical amount of input data that comes with these datasets has increased the pressure on the models to be able to handle larger inputs. for instance, wcep has up to 100 documents in each document set, and 63.7 on average. as such, the standard transformers used to develop successful sds models such as bart @xcite and pegasus @xcite have proved inadequate for mds due to their limited maximum input length (in the order of 10 3 tokens) and quadratic memory complexity @xcite . in turn, this has prompted the development of long transformer models such as longformer @xcite (built upon bart) and bigbird @xcite (built upon pegasus) which, thanks to their smart attention layers that scale linearly with the input length, have opened up the possibility of presenting the input documents "at once", allowing these re-designed attention mechanisms to discover both inter-and intra-document relations. document summarization, as have other language generation tasks, has often been criticized for using maximum-likelihood training objectives that may prove limitative for the eventual performance of the models @xcite . for this reason, reinforcement learning has been employed as an alternative, to directly optimize the models over evaluation metrics and explicitly reward the quality of the model's predictions. reinforcement learning approaches have used metrics such as rouge-1, rouge-2 and rouge-l f1 @xcite , and also more contemporary scoring functions such as bertscore @xcite as rewards, often mixed with maximum-likelihood objectives. when applying reinforcement learning to mds, we contend that the reward should not simply be a rouge score against the reference summary, since this would dismiss key characteristics of the task such as inter-document information transfer. for instance, @xcite have leveraged maximal marginal relevance @xcite to mollify higher-order information redundancy between the input documents. several other performance measures could potentially be included in the reward, such as extractive fragment coverage and density @xcite and mint @xcite , but to the best of our knowledge they have never been utilized as, or for, training objectives. to address this gap, in this paper we propose leveraging a modified coverage reward to improve information coverage across all the documents in the input set, jointly with a principled policy gradient estimator (relax) and a performing long transformer model (the bart longformer encoder-decoder, or bart-led), in the hope of benefiting from the synergy between these components. ## proposed approach in this section, we present the details of the proposed approach, including the reinforcement learning framework (section 3.1), the multi-document coverage reward (section 3.2), and the overall training objective (section 3.3). ## analysis in this section, we present a more detailed analysis of the impact of the coverage term, the few-shot fine-tuning, and the relax gradient estimator using the multi-news validation set as reference. for a further insight into the coverage reward, we also include an analysis of its trajectory during training. all the selected hyperparameters are listed in appendix a.1. ## conclusion in this paper, we have proposed fine-tuning a multidocument summarization model with a reward that balances the use of the reference summaries with the coverage of the input documents within a reinforcement learning framework. the rationale for the proposed reward is that the reference summaries alone may not be sufficient for an effective fine-tuning of the model in the presence of very large inputs such as those typical of mds datasets. another key component of the proposed approach is the use of a modern gradient estimator of the policy gradient, relax. the experimental results over two news-based mds datasets, multi-news and wcep, have shown that the proposed approach has been able to achieve a marked improvement of rouge and meteor scores compared to its nll-pretrained baseline, and prove competitive against most existing approaches. in addition, the proposed approach has been able to increase the coverage of the input documents, and evenly across the entire document set. as future work, we aim to explore ways to prevent or mollify the model's drift with larger number of training steps, and explore alternative architectures and configurations for the control variate of the relax estimator.