Multi-Granularity Interaction Network for Extractive and Abstractive Multi-Document Summarization (2020)

Abstract: In this paper, we propose a multi-granularity interaction network for extractive and abstractive multi-document summarization, which jointly learn semantic representations for words, sentences, and documents. The word representations are used to generate an abstractive summary while the sentence representations are used to produce an extractive summary. We employ attention mechanisms to interact between different granularity of semantic representations, which helps to capture multi-granularity key information and improves the performance of both abstractive and extractive summarization. Experiment results show that our proposed model substantially outperforms all strong baseline methods and achieves the best results on the Multi-News dataset.

## introduction document summarization aims at producing a fluent, condensed summary for given documents. single document summarization has shown promising results with sequence-to-sequence models that encode a source document and then decode it into a summary @xcite @xcite . multi-document summarization requires producing a summary from a cluster of thematically related documents, where the given documents complement and overlap each other. multi-document summarization involves identifying important information and filtering out redundant information from multiple input sources. there are two primary methodologies for multidocument summarization: extractive and abstractive. extractive methods directly select important sentences from the original, which are relatively simple. @xcite rank sentences with a recursive neural network. @xcite employ a graph convolutional network (gcn) to incorporate sentence relation graphs to improve the performance for the extractive summarization. abstractive methods can generate new words and new sentences, but it is technically more difficult than extractive methods. some works on multidocument summarization simply concatenate multiple source documents into a long flat sequence and model multi-document summarization as a long sequence-to-sequence task @xcite . however, these approaches don't take the hierarchical structure of document clusters into account, while the too-long input often leads to the degradation in document summarization @xcite . recently, hierarchical frameworks have shown their effectiveness on multi-document summarization @xcite . these approaches usually use multiple encoders to model hierarchical relationships in the discourse structure, but other methods to incorporate the structural semantic knowledge have not been explored. the combination of extractive and abstractive has been explored in single document summarization. @xcite use the extracted sentences as the input of the abstractive summarization. @xcite concatenate the extracted summary to the original document as the input of the abstractive summarization. in this work, we treat documents, sentences, and words as the different granularity of semantic units, and connect these semantic units within a three-granularity hierarchical relation graph. with the multi-granularity hierarchical structure, we can unify extractive and abstractive summarization into one architecture simultaneously. extractive summarization operates on sentence-granularity and directly supervises the sentence representations while abstractive summarization operates on wordgranularity and directly supervises the word repre-sentations. we propose a novel multi-granularity interaction network to enable the supervisions to promote the learning of all granularity representations. we employ the attention mechanism to encode the relationships between the same semantic granularity and hierarchical relationships between the different semantic granularity, respectively. and we use a fusion gate to integrate the various relationships for updating the semantic representations. the decoding part consists of a sentence extractor and a summary generator. the sentence extractor utilizes the sentence representations to select sentences, while the summary generator utilizes the word representations to generate a summary. the two tasks are trained in a unified architecture to promote the recognition of important information simultaneously. we evaluate our model on the recently released multi-news dataset and our proposed architecture brings substantial improvements over several strong baselines. we explore the influence of semantic units with different granularity, and the ablation study shows that joint learning of extractive and abstractive summarization in a unified architecture improves the performance. in summary, we make the following contributions in this paper: ## related work the methods for multi-document summarization can generally be categorized to extractive and abstractive. the extractive methods produce a summary by extracting and merging sentences from the input documents, while the abstractive methods generate a summary using arbitrary words and expressions based on the understanding of the documents. due to the lack of available training data, most previous multi-document summarization methods were extractive @xcite @xcite . since the neural abstractive models have achieved promising results on single-document summarization @xcite @xcite , some works trained abstractive summarization models on a single document dataset and adjusted the model to adapt the multi-document summarization task. @xcite added a document set encoder into the single document summarization framework and tuned the pre-trained model on the multi-document summarization dataset. @xcite combined an extractive summarization algorithm (mmr) for sentence extraction to reweight the original sentence importance distribution learned in the single document abstractive summarization model. recently, two large scale multi-document summarization datasets have been proposed, one for very long input, aimed at generating wikipedia @xcite and another dedicated to generating a comprehensive summarization of multiple real-time news @xcite . @xcite concatenated multiple source documents into a long flat text and introduced a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder-decoder architectures. @xcite introduced intermediate document representations and simply add the document representations to word representations for modeling the cross-document relationships. compared with our proposed multi-granularity method, @xcite inclined to the traditional bottomup hierarchical method and don't effectively utilize the hierarchical representations while ignoring the hierarchical relationships of sentences. @xcite incorporated mmr into a hierarchical pointer-generator network to address the information redundancy in multi-document summarization. ## our approach our model consists of a multi-granularity encoder, a sentence extractor, and a summary generator. firstly, the multi-granularity encoder reads multiple input documents and learns the multi-granularity representations for words, sentences, and documents. self-attention mechanisms are employed for capturing semantic relationships of the representations with same granularity, while crossattention mechanisms are employed for the information interaction between representations with different granularity. fusion gates are used for integrating the information from different attention mechanisms. then the sentence extractor scores sentences according to the learned sentence representations. meanwhile, the summary generator produces the abstractive summary by attending to the word representations. in the following sections, we will describe the multi-granularity encoder, the sentence extractor, and the summary generator, respectively. ## conclusion and future work in this work, we propose a novel multi-granularity interaction network to encode semantic representations for documents, sentences, and words. it can unify the extractive and abstractive summarization by utilizing the word representations to generate the abstractive summary and the sentence representations to extract sentences. experiment results show that the proposed method significantly outperforms all strong baseline methods and achieves the best result on the multi-news dataset. in the future, we will introduce more tasks like document ranking to supervise the learning of the multi-granularity representations for further improvement.