G en C ompare S um: a hybrid unsupervised summarization method using salience (2022)

Abstract: Text summarization (TS) is an important NLP task. Pre-trained Language Models (PLMs) have been used to improve the performance of TS. However, PLMs are limited by their need of labelled training data and by their attention mechanism, which often makes them unsuitable for use on long documents. To this end, we propose a hybrid, unsupervised, abstractive-extractive approach, in which we walk through a document, generating salient textual fragments representing its key points. We then select the most important sentences of the document by choosing the most similar sentences to the generated texts, calculated using BERTScore. We evaluate the efficacy of generating and using salient textual fragments to guide extractive summarization on documents from the biomedical and general scientific domains. We compare the performance between long and short documents using different generative text models, which are finetuned to generate relevant queries or document titles. We show that our hybrid approach out-performs existing unsupervised methods, as well as state-of-the-art supervised methods, despite not needing a vast amount of labelled training data.

## introduction recent advancements in transformer-based architectures have enabled improvements in natural language processing (nlp) tasks. the use of encoder-decoder models, such as the t5 language model @xcite in generative linguistic tasks, such as abstractive summarization @xcite and query generation @xcite , have been shown to significantly improve performance over existing methods. bidirectional-encoder transformer architectures, namely @xcite have also proven to be powerful for a broad range of nlp tasks, including text summarization @xcite . whilst transformers have made great advancements in their ability at capturing semantic knowledge, they have also introduced new limitations. firstly, they are restricted by the number of tokens that they can process at any one time. another issue is the computational cost of finetuning the attention mechanisms embedded in transformers. these constraints are challenging for recent text summarization methods, often resulting in analysis being done on a truncated version of a document @xcite @xcite @xcite . since summarization should be able to succinctly capture the meaning of very long documents in a few sentences, the requirement to truncate a document before summarization is a major disadvantage. as a result, recent works have shifted their attention towards addressing the issue of long document summarization @xcite @xcite . however, these are mostly supervised methods, requiring large amounts of labelled training data, which are often unavailable or time-consuming and costly to produce. we address the challenges of supervised methods by adopting a hybrid unsupervised approach, where the plms are required only to act on short sections of the document at any time, meaning that our method can be extended to any document length. furthermore, by nature of it being an unsupervised approach, it does not require manually labelled training data for the extractive summarization task. to-date, unsupervised methods for text summarization have generally used graph-based methods @xcite @xcite @xcite , the more recent of these using transformer-based embeddings to calculate weights between the nodes in the graph @xcite . we differ from these previous approaches as we do not use a graph-based model and instead evaluate the effectiveness of a novel approach -generating and using salient textual fragments to guide the extractive summarization. moreover, earlier unsupervised, graph-based methods have been criticised in their ability to effectively represent documents which present multiple facts @xcite . our method addresses this by generating multiple salient texts per document, thus enabling it to represent multiple facts per document. text summarization methods are divided into extractive and abstractive groupings. extractive methods select the most relevant sentences from a document and abstractive methods consider the most relevant pieces of information to produce new textual fragments which convey the core message. although abstractive summarization has the potential to be more succinct and readable, in its current state it cannot be trusted to be factually consistent @xcite , making it unsuitable in many practical applications, such as summarization of biomedical articles for use by clinicians. furthermore, huang (2020) showed extractive techniques to outperform their abstractive counterparts in human evaluation. see (2017) recognises the advantage of hybrid extractiveabstractive summarization methods and uses a pointer-generator approach, where the model is mostly abstractive, but identifies and copies key facts directly from the source document to try to reduce factual inconsistency. we consider these factors and choose also to opt for a hybrid approach; however, we differ from see (2017) in our use of abstractive models. specifically, we use transformer-based models for the generation of salient points, but ultimately, we generate an extractive summarization to ensure factual consistency. our method, gencomparesum is a two-step hybrid summarization approach. gencomparesum first splits a document into sections of several sentences and walks through them, generating salient textual fragments which represent each section. we experiment with different generative models, which are finetuned 1 https://github.com/jbshp/gencomparesum to predict either queries or document titles, that best represent a section of the document. our method then uses these generated textual fragments to guide an unsupervised extractive summarization by calculating the bertscore similarity between each of the generated texts and each of the sentences in the source document. we then select the sentences with the highest scores to form the extractive summarization. we evaluate our approach on short and long versions of data sets from the biomedical and scientific domains. furthermore, we compare the use of different plms for generating salient textual fragments. our main contributions are as follows: ## methods we propose gencomparesum, a hybrid abstractive-extractive model, which makes use of transformer-based architectures but is extendable to any document length, can represent multiple facts, and does not require vast amounts of training data. the method is comprised of two steps: first, using a generative model to produce salient textual fragments, i.e., queries or document titles, which represent key points from across a document, then a comparison between these salient fragments and each sentence, to select the most important sentences from across the document. a representation of our method can be seen in figure 1 . we make our code publicly available 1 . ## future work in this section we suggest future directions for our research. firstly, we highlight that our method is generalizable and not restricted to t5based architectures for the generation of salient text fragments. therefore, we would like to experiment with different models for this step, e.g., bart-based (lewis, 2019) models, or models trained with different data. another interesting direction would be the inclusion of zoning into the method. as mentioned previously, we chose not to use an article's pre-defined sections as they are often not available. however, it would be interesting to predict a classification for a sentence within the text (e.g., 'results' for a scientific article), and to incorporate this into the model. we would also like to evaluate our models on other data sets and domains in future research, e.g., clinical notes, and would like to carry out a human evaluation to validate the results, ideally with experts from the same domain as the data being summarized. human evaluation would allow for aspects such as fluency, factual consistency, and coherence to be assessed, which have been shown not to necessarily align well with rouge evaluation in previous works @xcite . lastly, our analysis (details in appendix d) showed that bertscore was the best performing method for calculating text similarity for the extractive summarization task, outperforming methods using sentence embeddings. we hypothesise that this could be due to our evaluation metric being rouge scoring, which favours methods that produce summaries containing exactly the same words as the gold summary, rather than semantically similar sentences. experimentation into different evaluation metrics for extractive summarization, including human evaluation, and how they correlate to the performance of our methods when using different models for calculating text similarity, is also an interesting direction for future work. ## conclusion in this work we propose gencomparesum, a novel two-step unsupervised hybrid abstractiveextractive method for text summarization. we evaluate the efficacy of using plms to generate salient textual fragments which represent the key points of a document -experimenting with generation of both queries and document titlesand using them to guide the second step, extractive summarization. we show that that our unsupervised method, which can be extended to any length of document and does not require a corpus of annotated training data, outperforms over both strong supervised and unsupervised baselines on long and short documents. furthermore, we show that our best-performing model uses title-document pairs for the generative task, which are readily available across many domains without the need for manual labelling effort.