HIBRIDS : Attention with Hierarchical Biases for Structure-aware Long Document Summarization (2022)

Abstract: Document structure is critical for efficient information consumption. However, it is challenging to encode it efficiently into the modern Transformer architecture. In this work, we present HIBRIDS, which injects Hierarchical Biases foR Incorporating Document Structure into attention score calculation. We further present a new task, hierarchical question-summary generation, for summarizing salient content in the source document into a hierarchy of questions and summaries, where each follow-up question inquires about the content of its parent question-summary pair. We also annotate a new dataset with 6,153 question-summary hierarchies labeled on government reports. Experiment results show that our model produces better question-summary hierarchies than comparisons on both hierarchy quality and content coverage, a finding also echoed by human judges. Additionally, our model improves the generation of long-form summaries from long government reports and Wikipedia articles, as measured by ROUGE scores.

## introduction document structure facilitates information searching, reading comprehension, and knowledge acquisition by providing an informative overview of the content @xcite @xcite @xcite . specifically, for summarization, its utility is twofold: (1) source document structures, such as sections and paragraphs, can be instructive for summary generation @xcite @xcite ; nonetheless, state-of-the-art abstractive summarization systems, all built on the transformer architecture @xcite , use attentions to estimate relations between pairwise tokens and largely ignore document structures. while hierarchical encoding has been investigated @xcite , its need for training large amounts of additional parameters leads to increased memory footprint and thus limits the allowed input length. as for the output, the structure of single document summaries remains largely "flat", such as a list of aspects @xcite . we argue that it is imperative to develop systems that can output summaries with rich structures to support knowledge acquisition, which is especially critical for long documents that cover numerous subjects with varying details @xcite . this work consists of two main objectives: (1) effectively informing summarization models of the source document's structure, and (2) presenting a new summarization task that produces hierarchically organized question-summary pairs to facilitate information consumption. to this end, we propose hibrids (hierarchical biases for incorporating document structure). 1 we design learnable hierarchical biases, as part of the transformer attention calculation, to adjust attention weights based on tokens' relative positions with regard to the document structure, inspired by the relative position method that modifies attention calculation @xcite . concretely, we leverage the natural structure of a document, i.e., section levels, to construct a document structure tree (figure 2 ). each learnable bias corresponds to the relation between a pair of sections, based on the distance between them in the structure tree. intuitively, hierarchical biases adjust attention weights between tokens based on how conceptually close/distant their corresponding sections are, and they also enable summarizers to capture long-range federal land management agencies have taken … q1: what have federal land management agencies done in light of the epact 2005? q1.1: what did the blm do specifically that was intended to streamline the permitting process? q1.2: what is the purpose of regularly established meetings for these agencies? ## hibrids with hierarchical biases in this section, we first introduce how relative positions are defined over the document structure tree. then we present hibrids, which can be included in encoder self-attentions or decoder crossattentions to adjust the attention scores based on tokens' relative positions. ## a new task: hierarchical question-summary generation we introduce a new summarization task in this section: given a document or several sections of a document, we aim to generate question-summary (qs) pairs that are organized hierarchically. as shown in figure 1 , this qs hierarchy lays out details for topics at multiple levels, with each child qs pair expanding the content of its parent. our task is motivated by how human learns knowledge in a top-down fashion, where general knowledge is acquired first and details and in-depth content are explored later @xcite . this hierarchy proactively highlights the document structure, to further promote content engagement and comprehension @xcite .