Unsupervised Extractive Summarization using Pointwise Mutual Information (2021)

Abstract: Unsupervised approaches to extractive summarization usually rely on a notion of sentence importance defined by the semantic similarity between a sentence and the document. We propose new metrics of relevance and redundancy using pointwise mutual information (PMI) between sentences, which can be easily computed by a pre-trained language model. Intuitively, a relevant sentence allows readers to infer the document content (high PMI with the document), and a redundant sentence can be inferred from the summary (high PMI with the summary). We then develop a greedy sentence selection algorithm to maximize relevance and minimize redundancy of extracted sentences. We show that our method outperforms similarity-based methods on datasets in a range of domains including news, medical journal articles, and personal anecdotes.

## introduction modern neural network-based approaches to summarization require a large amount of documentsummary pairs that are usually unavailable outside of the news domain. for example, summarization datasets of personal narratives and office meetings contain only a few hundred examples @xcite . in this work, we tackle the problem of unsupervised extractive summarization which aims to select important sentences from the document. while there exists extensive prior work @xcite @xcite , most approaches rely on the assumption that important sentences are similar to other sentences in the document. however, it is unclear if similarity-based features lead to meaningful content selection @xcite . inspired by recent work on formalizing the notion of importance in summarization @xcite , we propose metrics for relevance and redundancy based on pointwise mutual information (pmi). intuitively, a relevant summary allows the reader to maximally infer the document content, and a summary has minimal redundancy if each sentence in it provides additional information. therefore, we measure the relevance of a summary by its pmi with the document. high relevance means that the probability of the document increases conditioning on the summary. similarly, we measure redundancy by pmi of sentence pairs within the summary. a sentence is redundant if seeing other sentences significantly increases its probability. based on the new metrics, we design a simple sentence extraction algorithm. we estimate the pmi of sentence pairs by a pre-trained language model fine-tuned on in-domain documents. we then use a simple sequential sentence selection algorithm for extractive summarization, which greedily maximizes relevance and minimizes redundancy. experimental results show that our algorithm outperforms similarity-based methods across multiple domains, including news, personal stories, and medical articles. 1 ## relevance and redundancy we begin by formalizing relevance and redundancy for summarization. consider a document of n sentences @xmath0, . . . , d n } and a summary of m sentences @xmath1, . . . , s m }. we would like to measure the relevance of s to d and the redundancy of s. relevance. relevance measures how well the summary condenses the original text such that we can infer its key content. specifically, a summary sentence is relevant if observing it reduces our uncertainty about (unseen) sentences in the document. for example, the summary may contain the main link in the thread of conversation in the document. we thus quantify the relevance of a summary sentence s to a document sentence d by their pmi: a positive score means that s and d are very likely to co-occur, thus seeing one implies the other. a zero score means that s and d are independent. a negative score means that s and d are unlikely to co-occur, e.g. contradicting sentences, thus such a summary sentence is discouraged. we further define the relevance of a summary s to the document d by the sum of sentence-level relevance: redundancy. redundancy measures how much overlap exists among the summary sentences. it is typically measured by the semantic similarity between two sentences. however, even if two sentences express different meanings, there is redundancy if one is entailed by the other. for example, consider: though expressing different information, both imply michelle's weight loss. given a summary sentence, we want to assign a score proportional to the amount of information in the sentence which is already present in the rest of the summary. therefore, we quantify the redundancy of a sentence s given another sentence s by their dependence in terms of pmi: similarly, the redundancy of a summary s is defined by the total redundancy of all sentence pairs: estimate pmi. by definition, pmi(s; d) = log p(s|d) p(d) = log p(d|s) p(s) . since both s and d are sentences, we use a language model, p lm , to estimate the probabilities. conditional probabilities are comoputed by considering the condition sentence as the prefix. note that while pmi can be computed in two equivalent ways according to the definition, the estimates from a language model do not guarantee that p lm (d|s) p lm (d) = p lm (s|d) p lm (s) . thus we choose to condition on the summary sentence: this is consistent with our definition of relevance: seeing the summary, how well we can estimate the document content. for redundancy, we condition on the earlier sentence: ## sequential sentence extraction given relevance and redundancy defined above, we aim to select important sentences from the document that maximize relevance and minimizes redundancy. we consider a weighted combination of the two criteria: ## experiments datasets. we evaluate on a diverse range of domains: (i) news articles: cnn-dailymail (cnn-dm) @xcite and xsum @xcite ; (ii) personal anecdotes: reddit (ouyang algorithm 1: extractsentences(d, k) s := ∅ for j ← 1 to k do for @xmath2:= λ 1 rel({s} , d) + λ 2 @xmath3, s) end s j := argmax @xmath4:= s ∪ {s j } end return s et al., 2017) and reddit-tifu @xcite ; (iii) scientific articles: pubmed @xcite . further, these datasets allow us to evaluate on highly abstractive summarization (xsum), small data (reddit), and long documents (pubmed). ## conclusion and discussion we propose metrics for relevance and redundancy in summarization based on pointwise mutual information, and an unsupervised extractive summarization algorithm using pre-trained language models. we demonstrate the effectiveness of our method on both news and non-news domains. supervised models often learn the lead bias in the datasets and degrade significantly when such hues are absent @xcite . furthermore, even human evaluation of content selection has large variance @xcite . our work is a first step towards formalizing a notion of importance that informs algorithm design in summarization. we believe it is important to have a better formalization of content importance in terms of both task definition/evaluation and modeling. we hope our results will spur more work in this direction.