Discourse-Aware Unsupervised Summarization for Long Scientific Documents (2021)

Abstract: We propose an unsupervised graph-based ranking model for extractive summarization of long scientific documents. Our method assumes a two-level hierarchical graph representation of the source document, and exploits asymmetrical positional cues to determine sentence importance. Results on the PubMed and arXiv datasets show that our approach outperforms strong unsupervised baselines by wide margins in automatic metrics and human evaluation. In addition, it achieves performance comparable to many state-of-the-art supervised approaches which are trained on hundreds of thousands of examples. These results suggest that patterns in the discourse structure are a strong signal for determining importance in scientific articles.

## introduction single document summarization aims at shortening a text and preserving the most important ideas of the source document. while abstractive strategies generate summaries with novel words, extractive strategies select sentences from the source to form a summary @xcite . despite recent advances in abstractive summarization, extractive models are still attractive in cases where faithfully preserving the original text is the priority. for example, legal arguments can hinge on the exact wording of a contract @xcite , and ensuring the factual correctness of a summary can be critical in the health or scientific domains, which is a known weakness of current abstractive methods @xcite . supervised neural-based models have been the dominant paradigm in recent extractive systems, at least for short news summarization @xcite introduction anxiety affects quality of life in those living with parkinson's disease (pd) more so than overall cognitive status, motor deficits, apathy, and depression. introduction although anxiety and depression are often related and coexist in pd patients, recent research suggests that anxiety rather than depression is the most prominent and prevalent mood disorder in pd. related work furthermore, since previous work, albeit limited, has focused on the influence of symptom laterality on anxiety and cognition, we also explored this relationship . methodology this study is the first to directly compare cognition between pd patients with and without anxiety. result the findings confirmed our hypothesis that anxiety negatively influences attentional setshifting and working memory in pd. result moreover, anxiety has been suggested to play a key role in freezing of gait (fog), which is also related to attentional set-shifting. future work s. future research should examine the link between anxiety, set-shifting, and fog, in order to determine whether treating anxiety might be a potential therapy for improving fog. however, these models cannot easily be adapted to out-of-domain data that have greater length and fewer training examples such as scientific article summarization @xcite due to two significant limitations. first, they require large domain-specific training pairs of source documents and gold-standard summaries, which are often not available or feasible to create @xcite . second, the typical setup of using a tokenlevel encoder-decoder with an attention mechanism does not scale well to longer documents @xcite , as the number of attention computations is quadratic with respect to the number of tokens in the input document. we instead explore unsupervised approaches to address these challenges on long document summarization. we show that a simple unsupervised graph-based ranking model combined with proper sophisticated modelling of discourse information as an inductive bias can achieve unreasonable effectiveness in selecting important sentences from long scientific documents. for the choice of unsupervised graph-based ranking model, we follow the paradigm of lexrank @xcite and pacsum @xcite . in these methods, sentences are nodes and weighted edges represent the degree of similarity between sentences. summary generation is formulated as a node selection problem, in which nodes (i.e., sentences) that are semantically similar to other nodes are chosen to be included in the final summary. in other words, they determine node importance by defining a notion of centrality in the graph. in addition, we augment the document graph with directionality and hierarchy to reflect the rich discourse structure of long scientific documents. in particular, our method relies on two insights about the discourse structure of long scientific documents. the first is that important information typically occurs at the start and end of sections; i.e., they tend to appear near section boundaries @xcite @xcite . we implement this using an asymmetric edge weighting function in a directed graph which considers the distance of a sentence to a boundary. the second is that most sentences across section boundaries are unlikely to interact significantly with each other @xcite . we implement this insight by injecting hierarchies into our model, introducing section-level representations as graph nodes in addition to sentence nodes. by doing so, we convert a flat graph into a hierarchical non-fully-connected graph, which has two advantages: 1) reduced computational cost and 2) pruning of distracting weak connections between sentences across different sections. we call our approach hierarchical and positional ranking model (hiporank) and evaluate it on summarizing long scientific articles from pubmed and arxiv @xcite . empirical results show that our method significantly improves performance over previous unsupervised models @xcite in both automatic and human evaluation. in addition, our simple unsupervised approach achieves performance comparable to many expensive state-of-the-art supervised neural models that are trained on hundreds of thousands of examples of long document pairs @xcite . this suggests that patterns in the discourse structure are highly useful for determining sentence importance in long scientific articles, and that explicitly building in biases inspired by this structure is a viable strategy for building summarization systems. ## method our proposed method combines simple graphbased ranking algorithms with a two-level hierarchical model of the rich discourse structures of long scientific documents @xcite . we incorporate this discourse information into the graph as inductive biases through the construction of a directed hierarchical graph for document representation (figure 1 and section 3.2) and through the asymmetric edge weighting of edges with boundary functions (section 3.3). ## experimental setup this section describes the datasets, the hyperparameter choices, the baseline models, and the evaluation metrics used in the experiments. ## conclusion we presented an unsupervised graph-based model for long scientific document summarization. the proposed approach augments the measure of sentence centrality by inserting directionality and hierarchy in the graph with boundary positional functions and hierarchical topic information grounded in discourse structure. our simple unsupervised approach with rich discourse modelling outperforms previous unsupervised graph-based summarization models by wide margins and achieves comparable performance to state-of-the-art supervised neural models. this makes our model a lightweight but strong baseline for assessing the performance of expensive supervised approaches for long scientific document summarization.