S up MMD : A Sentence Importance Model for Extractive Summarization using Maximum Mean Discrepancy (2020)

Abstract: Most work on multi-document summarization has focused on generic summarization of information present in each individual document set. However, the under-explored setting of update summarization, where the goal is to identify the new information present in each set, is of equal practical interest (e.g., presenting readers with updates on an evolving news topic). In this work, we present SupMMD, a novel technique for generic and update summarization based on the maximum mean discrepancy from kernel two-sample testing. SupMMD combines both supervised learning for salience and unsupervised learning for coverage and diversity. Further, we adapt multiple kernel learning to make use of similarity across multiple information sources (e.g., text features and knowledge based concepts). We show the efficacy of SupMMD in both generic and update summarization tasks by meeting or exceeding the current state-of-the-art on the DUC-2004 and TAC-2009 datasets.

## introduction multi-document summarization is the problem of producing condensed digests of salient information from multiple sources, such as articles. concretely, suppose we are given two sets of articles (denoted set a and set b) on a related topic (e.g., climate change, the covid-19 pandemic), separated by publication timestamp or geographic region. we may then identify three possible instantiations of multi-document summarization (see figure 1 most existing work on this topic has focused on the generic summarization task. however, update summarization is of equal practical interest. intuitively, the comparative aspect of this setting aims to inform a user of new information on a topic they are already familiar with. multi-document extractive summarization methods can be unsupervised or supervised. unsupervised methods typically define salience (or coverage) using a global model of sentence-sentence similarity. methods based on retrieval @xcite , centroids @xcite , graph centrality @xcite , or utility maximization @xcite @xcite have been well explored. however, sentence salience also depends on surface features (e.g., position, length, presence of cue words); effectively capturing these requires supervised models specific to the dataset and task. a body of work has incorporated such information through supervised learning, for example based on point processes @xcite , learning important words @xcite , graph neural networks @xcite , and support vector regression @xcite . these supervised methods have either a separate model for learning and inference, leading to a disconnect between learning sentence salience and sentence selection @xcite @xcite , or are designed specifically for generic summarization @xcite . in this work, we propose supmmd, which has a single model of learning salience and inference and can be applied to generic and comparative summarization. we make the following contributions: ## literature review multi-document summarization can be extractive, where salient pieces of the original text such as sentences are selected to form the summary; or abstractive, where a new text is generated by paraphrasing important information. the former is popular as it often creates semantically and grammatically correct summaries @xcite . in this work, we focus on generic and update multidocument summarization in the extractive setting. most extractive summarizers have two components: sentence scoring and selection. a variety of unsupervised and supervised methods have been developed for the former. unsupervised sentence scorers are based on centroids @xcite , graph centrality @xcite , retrieval relevance @xcite , word statistics @xcite , topic models @xcite , or concept coverage @xcite . supervised techniques include: using a graph-based neural network @xcite , learning sentence quality from point processes @xcite , combining word importances @xcite , combining sentence and phrase importances @xcite , or employing a mixture of submodular functions @xcite . sentence selection methods can be broadly categorized as greedy methods @xcite @xcite @xcite @xcite @xcite , which produce approximate solutions by iteratively selecting the sentences with the maximal score, or exact integer linear programming (ilp) based methods @xcite . some greedy methods use an objective which belongs to a special class of set functions called submodular functions @xcite @xcite , which have good approximation guarantees under greedy optimization @xcite . there has been limited research into update and comparative summarization. notable prior work includes maximizing concept coverage using ilp @xcite , learning sentence scores using a support vector regressor @xcite , and temporal content filtering @xcite . @xcite cast the comparative summarization problem as classification, and use mmd @xcite . in this work, we adapt their method to learn sentence importances driven by surface features. ## summarization as classification we review a perspective introduced by @xcite , where summarization is viewed as classification, and provide a brief introduction to maximum mean discrepancy (mmd). both these ideas form the basis of our subsequent method. ## the supmmd method we start by developing a technique for incorporating sentence importance into mmd for the purpose of generic multi-document extractive summarization. we then extend this method to comparative summarization, and incorporate multiple different kernels to use a diverse sets of features. ## experimental setup we include guidance on applying supmmd and the details required to reproduce our experiments. ## experimental results we compare our methods with the baselines on @xcite @xcite we present several variants of our method to analyze the effects of different components and modeling choices. we report the performance of unsupervised mmd (unsupmmd) which does not explicitly consider sentence importance. for our supervised method supmmd, we report the performance with a bigram kernel (supmmd) and combined kernels (supmmd + mkl). we also evaluated the impact of our oracle extraction method by replacing it with the extraction method suggested by @xcite in supmmd + alt oracles. meanwhile, supmmd + mkl + compress presents the result of applying sentence compression @xcite to our model. ## conclusion in this work, we present supmmd, a novel technique for update summarization based on the maximum mean discrepancy. supmmd combines supervised learning for salience, and unsupervised learning for coverage and diversity. further, we adapt multiple kernel learning to exploit multiple for future work, we leave the task of incorporating embeddings features such as bert @xcite , and evaluating with large generic multi-document summarization dataset multinews @xcite .