Neural Abstractive Multi-Document Summarization: Hierarchical or Flat Structure? (2020)

Abstract: With regards to WikiSum (CITATION) that empowers applicative explorations of Neural Multi-Document Summarization (MDS) to learn from large scale dataset, this study develops two hierarchical Transformers (HT) that describe both the cross-token and cross-document dependencies, at the same time allow extended length of input documents. By incorporating word- and paragraph-level multi-head attentions in the decoder based on the parallel and vertical architectures, the proposed parallel and vertical hierarchical Transformers (PHT &VHT) generate summaries utilizing context-aware word embeddings together with static and dynamics paragraph embeddings, respectively. A comprehensive evaluation is conducted on WikiSum to compare PHT &VHT with established models and to answer the question whether hierarchical structures offer more promising performances than flat structures in the MDS task. The results suggest that our hierarchical models generate summaries of higher quality by better capturing cross-document relationships, and save more memory spaces in comparison to flat-structure models. Moreover, we recommend PHT given its practical value of higher inference speed and greater memory-saving capacity.

## introduction with the promising results achieved by neural abstractive summarization on single documents @xcite @xcite , an increasing number of attempts are made to study abstractive multidocument summarization (mds) using seq2seq models @xcite @xcite . compared with the single-document summarization, multi-document summarization places challenges in two primary aspects, that is representing large source documents and capturing cross-document relationships. to address the former issue, @xcite adopts a two-stage approach by first selecting a list of important paragraphs from all documents in an extractive framework. then a modified language model based on the transformer-decoder with memory compressed attention (t-dmca) is developed to conduct abstractive summarization after concatenating the extracted paragraphs to a flat sequence. although the proposed flat structure of t-dmca demonstrates both theoretical and practical soundness to learn long-term dependencies, it fails to implant the cross-document relationship in its summaries. on the other hand, the encoderdecoder structure that allows hierarchical inputs of multiple documents offers not only another solution to the long-text summarization problem @xcite @xcite but also allows cross-document information exchange in the produced summaries. in particular, @xcite proposes a hierarchical transformer with local and global encoder layers to represent cross-token and cross-paragraph information, which are both utilized later to enrich token embeddings. summaries are then generated based on a vanilla transformer @xcite by concatenating enriched token embeddings from different documents to a flat sequence. such hierarchical transformer though captures crossdocument relationships, the essentially-flat transformer it adopts fails to learn dependencies of sequences longer than 2000 tokens according to @xcite . in this paper, we develop two novel hierarchical transformers to address both the text-length and cross-document linkage problems in mds. by introducing the word-level and paragraph-level multi-head attention mechanisms, our models are designed to learn both cross-token and crossdocument relationships. the word-and paragraphlevel context vectors are then jointly used to generate target sequences in order to abandon the flat structure, thus to mitigate the long-dependency problem. in detail, both of the proposed hierarchical architectures are based on the transformer encoder-decoder model @xcite , with context-aware word embeddings obtained from a shared encoder and cross-token linkages described by the word-level multi-head attention mechanism in the decoder. the difference lies in the way that the document-level information is handled. based on the static foot_1 paragraph embeddings computed from the context-aware word embeddings, the parallel hierarchical transformer (pht) models crossdocument relationships with paragraph-level multihead attention parallel to the word-level multi-head attention. the paragraph attentions are then used to normalize the word attentions. on the other hand, the vertical hierarchical transformer (vht) stacks the paragraph-level attention layer on top of the word-level attention layer in order to learn the latent relationship between paragraphs with dynamic foot_2 paragraph embeddings from the previous layer. to evaluate the performance of the proposed models as well as to compare flat and hierarchical structures in the mds task, we select several strong baselines covering abstractive models of flat strucuture (t-dmca @xcite and transformer-xl @xcite ) and of hierarchical structure @xcite ). a systematic analysis is conducted on the wikisum dataset according to four criteria including the models' abilities of capturing cross-document relationships, rouge evaluation, human evaluation and computational efficiency. the results show that pht&vht outperform other baselines significantly with memory space. hierarchical neural network hierarchical neural document models are applied in various fields of nlp such as document auto-encoder @xcite or text classification @xcite . in the area of abstractive summarization, @xcite extends a hierarchical rnn encoderdecoder @xcite with the hybrid sentenceword attention. instead of trainable attention machanisms, @xcite hires a hierarchical rnn with maximal marginal relevance (mmr) @xcite to represent the relationship between sentences. @xcite proposes a hierarchical transformer by incorporating a global self-attention to represent cross-document relationships. moreover, @xcite constructs a hierarchical bert @xcite to learn the context relationships among sentences by using other sentences to generate the masked sentence. ## hierarchical transformer this paper proposes two hierarchical transformers with parallel & vertical architectures, respectively. section 3.1 explicitly explains the construction of the parallel hierarchical transformer (pht) and its application in mds, whereas section 3.2 places emphasis on explaining the structural differences between the vertical hierarchical transformer (vht) and pht.