Abstractive Summarization Guided by Latent Hierarchical Document Structure (2022)

Abstract: Sequential abstractive neural summarizers often do not use the underlying structure in the input article or dependencies between the input sentences. This structure is essential to integrate and consolidate information from different parts of the text. To address this shortcoming, we propose a hierarchy-aware graph neural network (HierGNN) which captures such dependencies through three main steps: 1) learning a hierarchical document structure through a latent structure tree learned by a sparse matrix-tree computation; 2) propagating sentence information over this structure using a novel message-passing node propagation mechanism to identify salient information; 3) using graph-level attention to concentrate the decoder on salient information. Experiments confirm HierGNN improves strong sequence models such as BART, with a 0.55 and 0.75 margin in average ROUGE-1/2/L for CNN/DM and XSum. Further human evaluation demonstrates that summaries produced by our model are more relevant and less redundant than the baselines, into which HierGNN is incorporated. We also find HierGNN synthesizes summaries by fusing multiple source sentences more, rather than compressing a single source sentence, and that it processes long inputs more effectively.

## introduction sequential neural network architectures in their various forms have become the mainstay in abstractive summarization @xcite . however, the quality of machine-produced summaries still lags far behind the quality of human summaries @xcite @xcite . due to their sequential nature, a challenge with neural summarizers is to capture hierarchical and inter-sentential dependencies in the summmarized document. article sentences: 1. the town is home to the prestigious leander club, which has trained more than 100 olympic medal-winning rowers. ## hierarchy-aware graph neural encoder hiergnn learns the document structure in an endto-end fashion without any direct structure supervision, and does not need an external parser to construct the structure, unlike previous work @xcite @xcite . in addition, it empirically improves over supervised graph construction, which has been a challenge @xcite . sequential summarizers encode an n -token article, @xmath0, • • • , x n ) as d-dimensional latent vectors using an encoding function h enc (x t ) ∈ r d and then decodes them into the target summary y . (we denote by h enc (x) the sequence of x t encodings for @xmath1.) our model includes four modules in addition to this architecture: 1) a sparse matrix-tree computation for inferring the document hierarchical structure, ii) a novel message-passing layer to identify inter-sentence dependencies, iii) a reasoning fusion layer aggregating the outputs of the message-passing module; and vi) a graphselection attention module to leverage the encoded structural information. ## discussion coverage and copy length. we report two metrics introduced by @xcite in layer-shared or layer-independent reasoning? in tables 2 and 3 , we observe that the layer-shared reasoning (lsr) architecture for hiergnn-pgn on cnn/dm outperforms the layer-independent reasoning (lir) architecture, with the opposite being true for xsum. we attribute this difference to the inductive bias of the base model and the essential difference between the cnn/dm and xsum datasets. pgn-based models tend to copy and degenerate the model into an extractive summarizer @xcite . with a more extractive dataset like cnn/dm, a complex reasoning procedure for the pgn-based model may not be necessary; instead, learning a single hierarchical structure and selecting the sentences to be copied accordingly is sufficient. however, xsum summaries are abstractive, and the dataset emphasizes combining information from multiple document sites (see discussion by @xcite ). lir then shows its advantage by learning separate hierarchical structure in each layer. for an abstractive l1 r oo t l2 r oo t l1 e dg es l2 e dg es 0.00 0.02 0.04 0.06 0.08 intra-layer diversity cnndm model sparse dense l1 r oo t l2 r oo t l1 e dg es l2 e dg es 0.00 0.02 0.04 0.06 0.08 0.10 xsum model sparse dense r oo t e dg es 0.00 0.05 0.10 0.15 0.20 0.25 inter-layer diversity cnndm model sparse dense r oo t e dg es 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 xsum model sparse dense base model (bart), lir consistently outperforms lsr on both cnn/dm and xsum. compression or fusion? to assess whether sentence fusion happens often, we quantify the ratio of sentence compression and sentence fusion that the model uses to generate summaries in effectiveness for longer sequence. the performance of sequence-to-sequence models decays as the length of the input sequence increases @xcite because they do not capture long-range dependencies. we hypothesize that hiergnn has a better capability in capturing such dependencies via its learned document hierarchical structure, thus enhancing the performance for long-sequence inputs. top-3 sentences with highest root probabilities (our sparse mtc): 8th sent. (9.77%): a lunar eclipse happens when the sun, earth and moon form a straight line in space, with the earth smack in the middle. 6th sent. (9.40%): the sun shines on the earth and creates a shadow. 10th sent. (7.79%): parts of south america, india, china and russia also will be able to see the eclipse, but it won't be visible in greenland, iceland, europe, africa or the middle east. top-3 sentences with lowest root probabilities (our sparse mtc): 20th sent. (sparsified): share your photos with cnn ireport. 18th sent. (sparsified): if you want to learn more about the eclipse, nasa astronomer mitzi adams will take questions on twitter nasa_marshall. 19th sent. (0.02%): did you see the total lunar eclipse? reference: the total eclipse will only last 4 minutes and 43 seconds. people west of the mississippi river will have the best view. parts of south america, india, china and russia also will see the eclipse. ## conclusion we propose hiergnn that can be used in tandem with existing generation models. the module learns the document hierarchical structure while being able to integrate information from different parts of the text as a form of reasoning. our experiments verify that hiergnn is effective in improving the plain sequential summarization models.