Leveraging Information Bottleneck for Scientific Document Summarization (2021)

Abstract: This paper presents an unsupervised extractive approach to summarize scientific long documents based on the Information Bottleneck principle. Inspired by previous work which uses the Information Bottleneck principle for sentence compression, we extend it to document level summarization with two separate steps. In the first step, we use signal(s) as queries to retrieve the key content from the source document. Then, a pre-trained language model conducts further sentence search and edit to return the final extracted summaries. Importantly, our work can be flexibly extended to a multi-view framework by different signals. Automatic evaluation on three scientific document datasets verifies the effectiveness of the proposed framework. The further human evaluation suggests that the extracted summaries cover more content aspects than previous systems.

## introduction automatic text summarization is a challenging task of condensing the salient information from the source document into a shorter format. two main categories are typically involved in the text summarization task, one is extractive approach @xcite @xcite which directly extracts salient sentences from the input text as the summary, and the other is abstractive approach @xcite @xcite @xcite which imitates human behaviour to produce new sentences based on the extracted information from the source document. traditional extractive summarization methods are mostly unsupervised, extracting sentences based on n-grams overlap @xcite , relying on graph-based methods for sentence ranking @xcite , or identifying important sentences with a latent semantic analysis technique @xcite . these unsupervised systems have been surpassed by neural-based models @xcite in respect of performance and popularity, their encoder-decoder structures use either recurrent neural networks @xcite or transformer @xcite . @xcite developed an unsupervised auto-encoder model which attempts to encode and then reconstructs the documents with some properly designed reconstruction loss. however, as it tries to preserves every detail that helps to reconstruct the original documents, it is not applicable to long-document summarization settings. recently, @xcite proposes an unsupervised non-neural approach for long document by building graphs to blend sentences from different text spans and leverage correlations among them. nevertheless, none of the aforementioned works utilize explicit guidance to aid the model in summarizing a source text. to this end, some works @xcite @xcite @xcite explore the use of guided signals extracted from the input source document such as keywords, highlighted sentences and others to aid the model architecture in summarizing the input document. these works only utilize a single signal, and @xcite empirically showed that if multiple guided signals can be optimally exploited, the model could achieve even greater improvement to its summary outputs in the supervised neural summarization research space. based on this finding, we propose a multi-view information bottleneck framework that can effectively incorporate multiple guided signals for the scientific document summarization task. the original idea of information bottleneck (ib) principle @xcite in information theory is to compress a signal under the guidance of another correlated signal. bottlesum @xcite successfully applied ib to the summarization task for short document. their model generates summary merely by removing words in each sentence while preserving all sentences without considering the importance of the sentences at a document level. it is not suitable for long scientific document summarization as it would preserve all sentences, resulting in significant redundancy. in contrast, our framework applies ib principle on document level rather than sentence level, where pruning unrelated information will only work on the selected important sentences. in particular, at the content selection stage as shown in figure 1 , the signal that we seek to compress is the source document and the correlated signals are extracted from the source document using state-of-the-art language models. followed by the text realization step where our proposed architecture conducts sentence search based on fluency to return the final extracted summaries. this framework can be flexibly extended to multi-view architecture by incorporating more selfdefined correlated signals. our experiments on arxiv, pubmed @xcite and @xcite show that our framework yields competitive performance compared to previous extractive summarizers. despite the less satisfactory results for multi-view framework in our experiment, we believe it has fruitful potential for further study since the experiments from the work of @xcite have empirically shown that summarization through multiple guided signals can achieve significant improvements to the system with a single signal. ## conclusion and future work in this paper, we proposed an unsupervised framework based on ib principle for long document summarization. our framework employs a two-steps system where content selection is guided by defined signal(s) and is followed by a text realization step where a pre-trained language model conducts sentence search to return final summaries. experiments on three scientific show the effectiveness of our framework. further human analysis suggests that the extracted summaries exhibit more coverage. despite the less satisfactory results for multi-view framework in our experiment, we believe it has fruitful potential for further study.