OTE xt S um: E xtractive T ext S ummarisation with O ptimal T ransport (2022)

Abstract: Extractive text summarisation aims to select salient sentences from a document to form a short yet informative summary. While learning-based methods have achieved promising results, they have several limitations, such as dependence on expensive training and lack of interpretability. Therefore, in this paper, we propose a novel non-learning-based method by for the first time formulating text summarisation as an Optimal Transport (OT) problem, namely Optimal Transport Extractive Summariser (OTExtSum). Optimal sentence extraction is conceptualised as obtaining an optimal summary that minimises the transportation cost to a given document regarding their semantic distributions. Such a cost is defined by the Wasserstein distance and used to measure the summary’s semantic coverage of the original document. Comprehensive experiments on four challenging and widely used datasets - MultiNews, PubMed, BillSum, and CNN/DM demonstrate that our proposed method outperforms the state-of-the-art non-learning-based methods and several recent learning-based methods in terms of the ROUGE metric.

## introduction text summarisation aims to condense a given document into a short and succinct summary that best covers the semantics of the document with the least redundancy. it helps users quickly browse and understand long documents by focusing on their most important sections @xcite . a common practice for text summarisation is extractive summarisation which aims to select the salient sentences of a given document to form its summary. extractive summarisation ensures the production of grammatically and factually 1 our code is publicly available for research purpose in https://github.com/peggypytang/otextsum/ figure 1 : illustration of optimal transport extractive summariser (otextsum): the formulation of extractive summarisation as an optimal transport (ot) problem. optimal sentence extraction is conceptualised as obtaining the optimal extraction vector m * , which achieves an ot plan from a document d to its optimal summary s * that has the minimum transportation cost. such a cost is defined as the wasserstein distance between the document's semantic distribution tf d and the summary's semantic distribution tf s and is used to measure the summary's semantic coverage. existing methods @xcite often first score the importance of individual sentences of a given document and then combine the top-ranked ones to form a summary. however, the sentences with high importance scores may not well represent the document from a global perspective, which results in a sub-optimal summary. recently, learning-based methods, especially those based on supervised and unsupervised deep learning techniques @xcite @xcite @xcite @xcite can significantly improve summarisation performance. however, training deep learning models is computationally expensive, and it can be difficult to apply those models learned from a particular domain to other domains with different distributions. moreover, deep learning methods generally lack interpretability for the summarisation process. motivated by these issues, we propose a novel non-learning based extractive summarisation method, namely optimal transport extractive summariser (otextsum). as illustrated in figure 1 , we formulate extractive summarisation based on the optimal transport (ot) theory @xcite . a candidate summary can be evaluated by an ot plan regarding the optimal cost to transport between the semantic distributions of the summary and its original document. then a wasserstein distance can be obtained with this optimal plan to measure the discrepancy between the two distributions. to this end, it can be expected that a summary of high quality minimizes this wasserstein distance. moreover, a common assumption in the formulations of the ot problem is that the source and target distributions are fixed. in otextsum problem formulation, we relax this assumption by adding an extraction vector m * to indicate which document sentences would be extracted to form the summary's semantic distribution, thus making the target distribution variable. the semantic distributions of a given document and its candidate summary can be formulated in line with the frequency of their tokens. inspired by word mover's distance @xcite , summarisation can be conceptualized as moving the "semantics" of a given document to its summary, and the ideal summary is obtained at the minimal transportation cost. this ensures the highest semantic coverage of the given document and the least redundancy in the summary without explicitly modelling conventional criteria such as relevance and redundancy. thus, under the ot plan, the wasserstein distance indicates the candidate summary's semantic coverage of the given document. we design two optimisation strategies to approximate the extraction vector m * , namely beam search strategy @xcite , which itera-tively evaluates the semantic coverage scores of a set of candidate summaries to obtain the optimal extraction, and binary integer programming strategy, which approximates the optimal extraction given the constraints of the wasserstein distance and extraction budget. as a non-learning based method, otextsum does not require any training and is applicable to different document domains. furthermore, it provides explainable results in terms of the semantic coverage of the summary. there have been some studies on ot in nlp, such as document distance @xcite ), text generation @xcite ), text matching @xcite , and machine translation @xcite . these methods generally focus on deriving similarities between words, sentences, and documents. on the contrary, we for the first time formulate text summarisation as an ot problem that optimally transports the semantic distributions between two texts (e.g., source document and summary candidate). overall, the key contributions of this paper are: ## related work generally, text summarisation methods can be categorized as extractive, abstractive, and hybrid ones. while abstractive and hybrid summarisation methods @xcite aim to mimic human beings in summarisation by paraphrasing a given document, extractive summarisation generally produces more factual summaries. in this section, we review existing extractive sum-marisation methods in two categories: non-learning based and learning-based methods. ## methodology as shown in figure 1 , otextsum utilizes a text ot approximation to obtain the optimal extraction vector m * = [m 1 , ..., m n ] t , where m @xmath0, 1} denotes whether the i-th sentence is to be extracted (denoted by 1) or not (denoted by 0). the optimal extraction vector m * achieves an ot plan from the semantic distribution of the document to that of its optimal candidate summary which has the minimum total transportation cost. the ot approximation consists of four components: 1) a tokeniser & embedding procedure that formulates token level representations and a semantic distribution estimation that computes the frequency of each token within a summary or a document ; 2) a transportation cost matrix that measures the cost using one token to represent another based on their euclidean distances; 3) an ot solver that approximates wasserstein distance and semantic coverage of the candidate summaries; and 4) an optimisation strategy that obtains the optimal extraction vector by choosing the summary with the minimum wasserstein distance, and thus with the highest semantic coverage of the source document. ## conclusion in this paper, we have presented otextsum, the first optimal transport-based optimisation method for extractive text summarisation. it aims to identify an optimal subset of sentences for producing a summary that achieves high semantic coverage of the document by minimising the wasserstein distance between the semantic distributions of the document and the summary. it helps obtain a summary from a global perspective and provides an interpretable visualisation of extraction results. in addition, otextsum does not require computationally expensive training. the comprehensive experiments demonstrate the effectiveness of otextsum, which is generalisable over various document domains. in our future work, we will explore other ot solvers for extractive summarisation.