Unsupervised document summarization using pre-trained sentence embeddings and graph centrality (2021)

Abstract: This paper describes our submission for the LongSumm task in SDP 2021. We propose a method for incorporating sentence embeddings produced by deep language models into extractive summarization techniques based on graph centrality in an unsupervised manner. The proposed method is simple, fast, can summarize any kind of document of any size and can satisfy any length constraints for the summaries produced. The method offers competitive performance to more sophisticated supervised methods and can serve as a proxy for abstractive summarization techniques

## introduction automatic text summarization is a very old and important task in natural language processing (nlp) that has received continued attention since the creation of the field in the late 50's @xcite , mainly because of the ever-increasing size of collections of text. the objective of the task is, given a document, to produce a shorter text with maximum information content, fluency and coherence. the summarization task can be classified into extractive and abstractive. extractive summarization means that the summary is composed exclusively of passages present in the original document and abstractive summarization means that there can be words in the summary that did not appear in the original document. since the creation of the first neural language models @xcite , vector representations of text that encode meaning (called embeddings) have played a significant role in nlp. they allow the application of statistical and geometrical methods to words, sentences and documents ( @xcite , @xcite , @xcite ), leading to stateof-the-art performance on several nlp tasks like information retrieval, question answering or paraphrase identification. among these neural language models, very deep pre-trained neural language models, like bert @xcite , t5 @xcite , and gpt-3 @xcite have shown impressive performance in tasks like language modelling and text generation or benchmarks like glue @xcite . an important variation of extractive summarization that goes back as far as the late 90's @xcite utilizes graphs, where the nodes represent text units and the links represent some measure of semantic similarity. these early graphbased summarization techniques involved creating a graph where the nodes were the sentences or paragraphs of a document and two nodes were connected if the corresponding text units had a similar vocabulary. after creating the document graph, the system created a summary by starting at the first paragraph and following random walks defined by different algorithms that tried to cover as much of the graph as possible. a more evolved approach was the creation of lexical centrality @xcite @xcite , which is a measure of the importance of a passage in a text where the sentences of the document are connected by the similarity of their vocabularies. the current state of the art in automatic summarization with graphs is mainly based on algorithms like pagerank @xcite enhanced with statistical information of the terms in the document (like in @xcite ) or graph neural networks (kipf and welling, 2016) on top of deep language models (like in @xcite ). only two systems from the previous scholarly document processing workshop held in 2020 are based on graphs: cist-bupt and monash-summ. in cist-bupt @xcite , they used recurrent neural networks to create sentence embeddings that can be used to build a graph which is then fed into a graph convolutional network (kipf and welling, 2016) and a graph attention network @xcite to create extractive summaries. to generate abstractive summaries, they used the gap-sentence method of @xcite to fine-tune t5 @xcite . in monash-summ @xcite , they propose an unsupervised approach that leverages linguistic knowledge to construct a sentence graph like in summpip @xcite . the graph nodes, which represent sentences, are further clustered to control the summary length, while the final abstractive summary is created from the key phrases and discourse from each cluster. this work focuses on extractive summarization using graphs leveraging sentence embeddings produced by pre-trained language models. the essential idea is that, while the sentence embeddings produced by sbert @xcite are not well suited for clustering algorithms like hierarchical clustering or dbscan @xcite , they produce excellent results in paraphrase identification or semantic textual similarity when compared with cosine similarity, which implies that they can be used along with graph centrality methods. the text summarization method proposed in this paper has the following contributions: ## methodology in this section, we describe how the system works. the system is composed of three main steps: first, we use sbert to produce sentence embeddings for every sentence in the document to summarize; next, we form a graph by comparing all the pairs of sentence embeddings obtained and finally, we rank the sentences by their degree centrality in this graph. fig. 1 gives an overview of the whole method. ## results overall, we observed that the 600-word constraint of the task prevented our method from performing better, but we also observed that the best summaries produced by our method are too long (around 1,000 words or more). ## conclusion and future work