Modeling Content Importance for Summarization with Pre-trained Language Models (2020)

Abstract: Modeling content importance is an essential yet challenging task for summarization. Previous work is mostly based on statistical methods that estimate word-level salience, which does not consider semantics and larger context when quantifying importance. It is thus hard for these methods to generalize to semantic units of longer text spans. In this work, we apply information theory on top of pre-trained language models and define the concept of importance from the perspective of information amount. It considers both the semantics and context when evaluating the importance of each semantic unit. With the help of pre-trained language models, it can easily generalize to different kinds of semantic units n-grams or sentences. Experiments on CNN/Daily Mail and New York Times datasets demonstrate that our method can better model the importance of content than prior work based on F1 and ROUGE scores.

## introduction and related work text summarization aims to compress long document(s) into a concise summary while maintaining the salient information. it often consists of two critical subtasks, important information identification and natural language generation (for abstractive summarization). with the advancements of large pre-trained language models (pretlms) @xcite , state-of-the-art results are achieved on both natural language understanding and generation. however, it is still unclear how well these large models can estimate "content importance" for a given document. previous studies for modeling importance are either empirical-based, which implicitly encode importance during document summarization, or theory-based, which often lacks support by empirical experiments @xcite . benefiting from * corresponding author the large-scale summarization datasets @xcite , data-driven approaches @xcite @xcite have made significant progress. yet most of them conduct the information selection implicitly while generating the summaries. it lacks theory support and is hard to be applied to low-resource domains. in another line of work, structure features @xcite , such as centrality, position, and title, are employed as proxies for importance. however, the information captured by these features can vary in texts of different genres. to overcome this problem, theory-based methods @xcite @xcite aim to formalize the concept of importance, and develop general-purpose systems by modeling the background knowledge of readers. this is based on the intuition that humans are good at identifying important content by using their own interpretation of the world knowledge. theoretical models usually rely on information theory (it) @xcite . @xcite uses dirichlet distribution to represent the background knowledge and employs bayesian surprise to find novel information. @xcite instead models the importance with entropy, assuming the important words should be frequent in the given document but rare in the background. however, statistical method is only a rough evaluation for informativity, which largely ignores the effect of semantic and context. in fact, the information amount of units is not only determined by frequency, but also by its semantic meaning, context, as well as reader's background knowledge. in addition, bag-of-words approaches are difficult to generalize beyond unigrams due to the sparsity of n-grams when n is large. in this paper, we propose a novel and generalpurpose approach to model content importance for summarization. we employ information theory on top of pre-trained language models, which are expected to better capture the information amount of semantic units by leveraging their meanings and context. we argue that important content contains information that cannot be directly inferred from context and background knowledge. large pretrained language models are suitable for our study since they are trained from large-scaled datasets consisting of diverse documents and thus containing a wide range of knowledge. we conduct experiments on popular summarization benchmarks of cnn/daily mail and new york times corpora, where we show that our proposed method can outperform prior importance estimation models. we further demonstrate that our method can be adapted to model semantic units of different scales (n-grams and sentences). ## methodology in this section, we first estimate the amount of information by using information theory with pretrained language models ( §2.1 and §2.2), where we consider both the context and semantic meaning of a given text unit. we then propose a formal definition of importance for text summarization from a perspective of information amount ( §2.3). ## results we conduct extensive experiments with pre-trained models foot_2 in all three types of language models, including alm: gpt-2 @xcite ; mlms: bert @xcite , and distill-bert @xcite ; plms: xlnet @xcite . as shown in surprisingly, our method can also generalize to sentence-level semantic units and serve as an unsupervised extract-based model for summarization. our models achieve significantly higher rouge scores than previous work by average 2.02. this observation inspires a potential future direction for sentence-level importance modeling based on background knowledge as well as context information. we also compare the performance of pretlms in different categories. mlms, including bert and distillbert, have the best overall performance, since they are able to encode bidirectional context. plm, i.e. xlnet, is slightly inferior to mlms because the probabilities of the words are related to the order of their permutation, which may hurt importance estimation by our method. ## future work in the future work, we would like to fine-tune the current language models on the target of max p (x i |x -x i ) to better align with the interpretation of information theory. currently, the pretlms mostly mask the text randomly, which still differ from our current method's objective. background knowledge also deserves further investigation. the background knowledge of our methods comes from the pre-training process of language models, suggesting that the information distribution largely depends on the training data. meanwhile, most pretlms are trained with wikipedia or books, which may affect the determination of content importance from text with different styles. so domain-specific knowledge, such as genres or topics, can be included in the future work. ## conclusion we propose to use large pre-trained language models to estimate the information amount of given text units, by filtering out the background knowledge as encoded in the large models. we show that the large pre-trained models can be used as unsupervised methods for content importance estimation, where significant improvement over nontrivial baselines is achieved on both keyphrase extraction and sentence-level extractive summarization tasks.