S truct S um: Summarization via Structured Representations (2021)

Abstract: Abstractive text summarization aims at compressing the information of a long source document into a rephrased, condensed summary. Despite advances in modeling techniques, abstractive summarization models still suffer from several key challenges: (i) layout bias: they overfit to the style of training corpora; (ii) limited abstractiveness: they are optimized to copying n-grams from the source rather than generating novel abstractive summaries; (iii) lack of transparency: they are not interpretable. In this work, we propose a framework based on document-level structure induction for summarization to address these challenges. To this end, we propose incorporating latent and explicit dependencies across sentences in the source document into end-to-end single-document summarization models. Our framework complements standard encoder-decoder summarization models by augmenting them with rich structure-aware document representations based on implicitly learned (latent) structures and externally-derived linguistic (explicit) structures. We show that our summarization framework, trained on the CNN/DM dataset, improves the coverage of content in the source documents, generates more abstractive summaries by generating more novel n-grams, and incorporates interpretable sentence-level structures, while performing on par with standard baselines.

## introduction text summarization aims at identifying important information in long source documents and expressing it in human readable summaries. two prominent methods of generating summaries are extractive @xcite , where important sentences in the source article are selected to form a summary, and abstractive @xcite , where the model restructures and rephrases essential content into a paraphrased summary. state of the art approaches to abstractive summarization employ neural encoder-decoder methods that encode the source document as a sequence of tokens producing latent document representations and decode the summary conditioned on the representations. recent studies suggest that these models suffer from several key challenges. first, since standard training datasets are derived from news articles, model outputs are strongly affected by the layout bias of the articles, with models relying on the leading sentences of source documents @xcite . second, although they aim to generate paraphrased summaries, abstractive summarization systems often copy long sequences from the source, causing their outputs to resemble extractive summaries @xcite . finally, current methods do not lend themselves easily to interpretation via intermediate structures @xcite , which could be useful for identifying major bottlenecks in summarization models. to address these challenges, we introduce struct-sum: a framework that incorporates structured document representations into summarization models. structsum complements a standard encoderdecoder architecture with two novel components: encoders with induced latent structures have been shown to benefit several tasks including document classification, natural language inference @xcite , and machine translation @xcite . our latent structure attention module builds upon @xcite to model the dependencies between sentences in a document. it uses a variant of kirchhoff's matrix-tree theorem @xcite to model such dependencies as non-projective tree structures( §2.2). the explicit attention module is linguistically-motivated and aims to incorporate inter-sentence links from externally annotated document structures. we incorporate a coreference based dependency graph across sentences, which is then combined with the output of the latent structure attention module to produce a hybrid structureaware sentence representation ( §2.3). we test our framework using the cnn/dm dataset @xcite and show in §4.1 that it outperforms the base pointer-generator model @xcite by up to 1.1 rouge-l. we find that the latent and explicit structures are complementary, both contributing to the final performance improvement. our modules are also orthogonal to the choice of an underlying encoder-decoder architecture, rendering them flexible to be incorporated into other advanced models. quantitative and qualitative analyses of summaries generated by structsum and baselines ( §4), reveal that structure-aware summarization mitigates the news corpora layout bias by improving the coverage of source document sentences. additionally, structsum reduces the bias of copying large sequences from the source, inherently making the summaries more abstractive by generating ∼15% more novel n-grams than a competitive baseline. we also show examples of the learned interpretable sentence dependency structures, motivating further research for structure-aware modeling. ## structsum framework consider a source document x consisting of n sentences {s} where each sentence s i is composed of a sequence of words. document summarization aims to map the source document to a target summary y of m words {y}. a typical neural abstractive summarization system is an attentional sequence-to-sequence model that encodes the input sequence x as a continuous sequence of tokens {w} using a standard encoder @xcite . the encoder produces a set of hidden representations {h}. a decoder maps the previously generated token y t-1 to a hidden state and computes a soft attention probability distribution p(a t | x, y 1:t-1 ) over encoder hidden states. a distribution p over the vocabulary is computed at every time step t and the network is trained using the negative log likelihood loss: loss @xmath0. structsum modifies the above architecture as follows. we aggregate the token representations from the encoder to form sentence representations as in hierarchical encoders @xcite . we then use implicit-and explicit-structure attention modules to augment the sentence representations with sentence dependency information, leveraging both a learned latent structure and an external structure from other nlp modules. the attended vectors are then passed to the decoder, which produces the output abstractive summary. in the rest of this section, we describe our framework architecture, shown in figure 1 , in detail. ## evaluation a standard rouge metric does not shed meaningful light into the quality of summaries across important dimensions. as a recall-based metric it is not suitable for assessing the abstractiveness of summarization; it is also agnostic to layout biases and does not facilitate intrepretability of model decisions. we thus adopt automatic metrics tailored to evaluating separately each of these aspects. we compare structsum to our base model, the pointergenerator network with coverage @xcite and the reference. ## analysis of induced document structures similar to @xcite , we also look at the quality of the intermediate structures learned by the model. we use the chu-liu-edmonds algorithm @xcite to extract the maximum spanning tree from the attention score matrix as our sentence structure. the results are shown in figure 4 shows qualitative examples of induced structures along with summaries from the struct-sum. the first example shows a tree with sentence 3 chosen as root, which was the key sentence mentioned in the reference. in both examples, the sentences in the lower level of the dependency tree contribute less to the generated summary. similarly, in the examples source sentences used to generate summaries tend to be closer to the root node. in the first summary, all source content sentences used in the summary are either the root node or within depth 1 of the root node. in the second example, 4 out of 5 source sentences were at @xmath1. in both examples, generated summaries diverged from the reference by omitting certain sentences used in the reference. these sentences are in the lower section of the tree, providing insights on which sentences were preferred for the ## related work data-driven neural summarization falls into extractive @xcite or abstractive @xcite @xcite . pointer-generator @xcite learns to either generate novel in-vocabulary words or copy from the source. it has been the foundation for much work on abstractive summarization @xcite @xcite . our model extends it by incorporating latent/explicit structure, but these extensions are applicable to any other encoder-decoder architecture. for example, a follow-up study has already shown benefits of our method in multi-document summarization @xcite . in pre-neural era, document structure played a critical role in summarization @xcite @xcite @xcite . more re-cently @xcite infuse source syntactic structure into the pointer-generator using wordlevel syntactic features and augmenting them to decoder copy mechanism. in contrast, we model sentence dependencies as latent structures and explicit coreference structures; we do not use heuristics or salient features. @xcite propose structural compression and coverage regularizers incorporating structural bias of target summaries while we model the structure of the source document. @xcite klementiev (2019 ## conclusion and future work in this work, we propose the framework struct-sum for incorporating latent and explicit document structure in neural abstractive summarization. we introduce a novel explicit-attention module which incorporates external linguistic structures, instantiating it with coreference links. we show that our framework improves the abstractiveness and coverage of generated summaries, and helps mitigate layout biases associated with prior models. we present an extensive evaluation of structsumalong abstractiveness, coverage, and layout quantitatively. future work will investigate the role of document structures in pretrained language models @xcite .