Sparse Optimization for Unsupervised Extractive Summarization of Long Documents with the Frank-Wolfe Algorithm (2020)

Abstract: We address the problem of unsupervised extractive document summarization, especially for long documents. We model the unsupervised problem as a sparse auto-regression one and approximate the resulting combinatorial problem via a convex, norm-constrained problem. We solve it using a dedicated Frank-Wolfe algorithm. To generate a summary with k sentences, the algorithm only needs to execute approximately k iterations, making it very efficient for a long document. We evaluate our approach against two other unsupervised methods using both lexical (standard) ROUGE scores, as well as semantic (embedding-based) ones. Our method achieves better results with both datasets and works especially well when combined with embeddings for highly paraphrased summaries.

## introduction with the overwhelming increase of digital information, automatic text summarization has become important for many applications such as financial reviews, medical articles, etc. manually summarizing this amount of information takes a considerable amount of time and effort. this has motivated the study of efficient and reliable automatic text summarization methods. the task of automatic summarization is the process of generating a condensed version of a text that best describes the original one @xcite . the two mainstream approaches in the field of automatic summarization are extractive and abstractive. extractive approaches generate summaries by selecting a subset of informative words, phrases, or sentences directly from the source text. in contrast, abstractive approaches use linguistic methods to decompose and build a semantic representation of the text and use natural language generation techniques to generate a summary @xcite @xcite . in recent years, neural network architectures have made abstractive summarization popular. however, abstractive approaches are generally harder to develop as they require high performing natural language generation techniques, which is also an active research field. besides these two categories, mixed strategies that combine both extractive and abstractive approaches have also been proposed in recent literature @xcite @xcite . previous work in extractive approaches to summarization include statistical @xcite @xcite @xcite , graph-based and optimizationbased ones. the graph-based approaches treat text as a network instead of as a simple bag of words and use graph-based ranking methods to generate a summary @xcite @xcite . optimizationbased methods use techniques such as sparse optimization @xcite , integer linear programming (ilp) @xcite @xcite and constraint optimization @xcite to reconstruct the summary. in this work, we focus on extractive summarization for long documents. performing automatic text summarization for long documents is especially challenging as obtaining high quality human summaries for long documents is often quite costly and time consuming. recent works on extractive summarization have been focusing on neural network architectures @xcite . although these methods are successful in generating summaries for short documents, they often have difficulties with long input sequences @xcite . most recent works have started to investigate neural extractive summarization methods for long documents @xcite . however, these methods are supervised and require high quality training data in order to train the neural network models. this creates challenges for domains that do not have massive training datasets. @xcite compared recent neural extractive summarization models across different domains including news, personal stories, and medical articles. they found that many sophisticated neural extractive summarizers do not have better performance than those consisting of simpler models, and that word embedding averaging performs equally or better than rnns or cnns for sentence embedding. this suggests that a simpler model combined with pre-trained word embeddings show promise for summarizing long documents in domains that have few or no training data. in this work, we propose an unsupervised method for extracting long documents based on a sparse optimization framework and solve it using a dedicated frank-wolfe algorithm, which can be combined with pre-trained word embeddings to construct a distributive input representation. our work is based on the previous work of @xcite but designed specifically for the summarization task. the proposed framework is an unsupervised model that is efficient and does not require a training corpora, as typical supervised solutions would require. we test our method on two datasets that contain long documents, 2019 financial outlooks and classical literature, and compare it against two baselines: sparse subspace clustering (ssc) and textrank. the experimental results show that our method gives a higher rouge score than our baseline for both datasets. in particular, when combined with sentence embedding, our method gives a higher semantic rouge score when evaluated on paraphrased summaries. moreover, we show that our method is computationally more efficient compared to others. ## results and analysis in our experiment, we set the number of selected sentences k to be the same as the length of reference summary for all methods. fwsum-bw25 performs especially well when evaluated with lexical rouge, highlighting its capabilities of capturing lexical information (measured by unigram and bigram). when evaluated on the financial outlook data, fwsum-bw25 and textrank generally outperform fwsum-sif, with fwsum-bm25 being the best performing method. presumably, this is due to the fact that the gold summaries of the financial outlook data are taken directly from the source document without much paraphrasing, favoring sentence scoring functions that directly measure the content overlaps. however, when evaluated by the semantic rouge on the classical literature data, fwsum-sif start to show promises. the gold summaries of the classical literature data are written by human writers and are highly paraphrased and condensed. as a result, semantic rouge is a better measurement for this dataset. as shown in the table, fwsum-sif starts to outper-form other methods by a significant amount. the improvement over the other methods suggests that using embedding in the sentence scoring function allows for comparisons based on the semantics of words sequences. this results show that different sentence scoring functions may be used based on the nature of the summary. for summaries that are mostly taken from the source document without much paraphrasing, a lexical overlap or tf-idf-like kernel matrix may be used. for summaries that are highly paraphrased, an embedding-like kernel matrix may be more suitable. our method is able to work with both. ## conclusion unsupervised document summarization has been a challenging task, especially on long documents. in this work, we propose an efficient unsupervised extractive summarization model that is suitable for long documents by employing a dedicated frank-wolfe algorithm. our method allows one to incorporate sentence embedding or any sentence scoring functions that is best suited for the dataset or the application. we evaluate our method and compare it with two other unsupervised extractive summarization methods on two datasets that are much longer than other summarization corpora used in the past. we evaluate the methods on both lexical and semantic rouge in order to overcome the shortcoming of lexical rouge and to provide a better assessment of the quality of the summaries. we observed that our methods (both fwsum-bm25 and fwsum-sif) achieve the best results for both datasets and that fwsum-sif works especially well with summaries that are paraphrased. our results also motivate the exploration of different kernel functions or embedding methods, which is left as a future work.