Multi-document Summarization with Maximal Marginal Relevance-guided Reinforcement Learning (2020)

Abstract: While neural sequence learning methods have made significant progress in single-document summarization (SDS), they produce unsatisfactory results on multi-document summarization (MDS). We observe two major challenges when adapting SDS advances to MDS: (1) MDS involves larger search space and yet more limited training data, setting obstacles for neural methods to learn adequate representations; (2) MDS needs to resolve higher information redundancy among the source documents, which SDS methods are less effective to handle. To close the gap, we present RL-MMR, Maximal Margin Relevance-guided Reinforcement Learning for MDS, which unifies advanced neural SDS methods and statistical measures used in classical MDS. RL-MMR casts MMR guidance on fewer promising candidates, which restrains the search space and thus leads to better representation learning. Additionally, the explicit redundancy measure in MMR helps the neural representation of the summary to better capture redundancy. Extensive experiments demonstrate that RL-MMR achieves state-of-the-art performance on benchmark MDS datasets. In particular, we show the benefits of incorporating MMR into end-to-end learning when adapting SDS to MDS in terms of both learning effectiveness and efficiency.

## introduction text summarization aims to produce condensed summaries covering salient and non-redundant information in the source documents. recent studies on single-document summarization (sds) benefit from the advances in neural sequence learning @xcite @xcite as well as pretrained language models (liu and lapata, 2019; 1 code can be found at https://github.com/ morningmoni/rl-mmr . @xcite and make great progress. however, in multi-document summarization (mds) tasks, neural models are still facing challenges and often underperform classical statistical methods built upon handcrafted features @xcite . we observe two major challenges when adapting advanced neural sds methods to mds: (1) large search space. mds aims at producing summaries from multiple source documents, which exceeds the capacity of neural sds models @xcite and sets learning obstacles for adequate representations, especially considering that mds labeled data is more limited. for example, there are 287k training samples (687 words on average) on the cnn/daily mail sds dataset @xcite and only 30 on the duc 2003 mds dataset (6,831 words). (2) high redundancy. in mds, the same statement or even sentence can spread across different documents. although sds models adopt attention mechanisms as implicit measures to reduce redundancy @xcite , they fail to handle the much higher redundancy of mds effectively (sec. 4.2.3). there have been attempts to solve the aforementioned challenges in mds. regarding the large search space, prior studies @xcite perform sentence filtering using a sentence ranker and only take top-ranked k sentences. however, such a hard cutoff of the search space makes these approaches insufficient in the exploration of the (already scarce) labeled data and limited by the ranker since most sentences are discarded, foot_0 albeit the discarded sentences are important and could have been favored. as a result, although these studies perform better than directly applying their base sds models @xcite to mds, they do not outperform state-of-the-art mds methods @xcite . regarding the high redundancy, various redundancy measures have been proposed, including heuristic postprocessing such as counting new bi-grams @xcite and cosine similarity @xcite , or dynamic scoring that compares each source sentence with the current summary like maximal marginal relevance (mmr) @xcite . nevertheless, these methods still use lexical features without semantic representation learning. one extension @xcite of these studies uses capsule networks @xcite to improve redundancy measures. however, its capsule networks are pre-trained on sds and fixed as feature inputs of classical methods without end-to-end representation learning. in this paper, we present a deep rl framework, mmr-guided reinforcement learning (rl-mmr) for mds, which unifies advances in sds and one classical mds approach, mmr @xcite through end-to-end learning. rl-mmr addresses the mds challenges as follows: (1) rl-mmr overcomes the large search space through soft attention. compared to hard cutoff, our soft attention favors top-ranked candidates of the sentence ranker (mmr). however, it does not discard low-ranked ones, as the ranker is imperfect, and those sentences ranked low may also contribute to a high-quality summary. soft attention restrains the search space while allowing more exploration of the limited labeled data, leading to better representation learning. specifically, rl-mmr infuses the entire prediction of mmr into its neural module by attending (restraining) to important sentences and downplaying the rest instead of completely discarding them. (2) rl-mmr resolves the high redundancy of mds in a unified way: the explicit redundancy measure in mmr is incorporated into the neural representation of the current state, and the two modules are coordinated by rl reward optimization, which encourages non-redundant summaries. we conduct extensive experiments and ablation studies to examine the effectiveness of rl-mmr. experimental results show that rl-mmr achieves state-of-the-art performance on the duc 2004 @xcite and tac 2011 @xcite datasets (sec. 4.2.1). a comparison between various com-bination mechanisms demonstrates the benefits of soft attention in the large search space of mds (sec. 4.2.2). in addition, ablation and manual studies confirm that rl-mmr is superior to applying either rl or mmr to mds alone, and mmr guidance is effective for redundancy avoidance (sec. 4.2.3). contributions. (1) we present an rl-based mds framework that combines the advances of classical mds and neural sds methods via end-to-end learning. (2) we show that our proposed soft attention is better than the hard cutoff of previous methods for learning adequate neural representations. also, infusing the neural representation of the current summary with explicit mmr measures significantly reduces summary redundancy. (3) we demonstrate that rl-mmr achieves new state-ofthe-art results on benchmark mds datasets. ## problem formulation we define @xmath0, d 2 , ..., d n } as a set of documents on the same topic. each document set d is paired with a set of (human-written) reference summaries r. for the convenience of notation, we denote the j-th sentence in d as s j when concatenating the documents in d. we focus on extractive summarization where a subset of sentences in d are extracted as the system summary e. a desired system summary e covers salient and non-redundant information in d. e is compared with the reference summaries r for evaluation. ## the rl-mmr framework overview. at a high level, rl-mmr infuses mmr guidance into end-to-end training of the neural summarization model. rl-mmr uses hierarchical encoding to efficiently encode the sentences in multiple documents and obtains the neural sentence representation a j . rl-mmr models salience by combining mmr and sentence representation a j , and measures redundancy by infusing mmr with neural summary representation z t , which together form the state representation g t . at each time step, one sentence is extracted based on the mmr-guided sentence representation and state representation, and compared with the reference, the result (reward) of which is then back-propagated for the learning of both neural representation and mmr guidance. an illustrative figure of rl-mmr is shown in fig. 1 . in the following, we first describe mmr and the neural sentence representation. we then introduce the neural sentence extraction module and how we incorporate mmr guidance into it for better mds performance. finally, we illustrate how neural representation and mmr guidance are jointly learned via end-to-end reinforcement learning. ## experiments we conduct extensive experiments to examine rl-mmr with several key questions: (q1) how does rl-mmr perform compared to state-of-the-art methods? (q2) what are the advantages of soft attention over hard cutoff in learning adequate neural representations under the large search space? (q3) how crucial is the guidance of mmr for adapting sds to mds in the face of high redundancy? ## conclusion we present a reinforcement learning framework for mds that unifies neural sds advances and maximal marginal relevance (mmr) through end-toend learning. the proposed framework leverages the benefits of both neural sequence learning and statistical measures, bridging the gap between sds and mds. we conduct extensive experiments on benchmark mds datasets and demonstrate the superior performance of the proposed framework, especially in handling the large search space and high redundancy of mds. in the future, we will investigate the feasibility of incorporating classical mds guidance to abstractive models with large-scale pre-training @xcite and more challenging settings where each document set may contain hundreds or even thousands of documents.