TED : A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising (2020)

Abstract: Text summarization aims to extract essential information from a piece of text and transform the text into a concise version. Existing unsupervised abstractive summarization models leverage recurrent neural networks framework while the recently proposed transformer exhibits much more capability. Moreover, most of previous summarization models ignore abundant unlabeled corpora resources available for pretraining. In order to address these issues, we propose TED, a transformer-based unsupervised abstractive summarization system with pretraining on large-scale data. We first leverage the lead bias in news articles to pretrain the model on millions of unlabeled corpora. Next, we finetune TED on target domains through theme modeling and a denoising autoencoder to enhance the quality of generated summaries. Notably, TED outperforms all unsupervised abstractive baselines on NYT, CNN/DM and English Gigaword datasets with various document styles. Further analysis shows that the summaries generated by TED are highly abstractive, and each component in the objective function of TED is highly effective.

## introduction summarization refers to the task of condensing a document into a shorter version without losing the key information. summarization models can be categorized into two types: abstractive and extractive. extractive models select sentences from the input article as the summary. such process ensures a basic level of grammaticality and accuracy, but also limits the model ability to copying. in contrast, abstractive models summarize a document using newly generated tokens and phrases that may not be found in the original article, which involves ⇤ equal contribution. work was done during first author's internship at microsoft. a process requiring an advanced ability to refine, paraphrase and re-organize language information @xcite @xcite . like most machine learning algorithms, summarization models can also be divided into supervised and unsupervised categories. supervised approaches require in-domain parallel data, i.e. both input articles and corresponding reference summaries must be present for the teacher-forcing training @xcite . unfortunately, high-quality paired data are not always available across different text domains and styles. moreover, considering the fact that summarization is not an easy task even for people, reliable human-labeled data are also difficult to obtain. therefore, several unsupervised summarization approaches have been proposed, which do not require reference summaries for the target domain. we introduce these methods as follows. unsupervised extractive models. textrank @xcite encodes sentences in the article as nodes in an undirected graph. the weights of edges are measured by sentences similarity. the centrality of a node (sentence) is computed by @xcite to decide whether a sentence should be included in the final summary. @xcite advances upon textrank by encoding sentences with bert representation @xcite to compute pairs similarity and build graphs with directed edges decided by the relative positions of sentences. unsupervised abstractive models. @xcite leverages differentiable sampling and optimizes by re-constructing the input article from the generated summary. chu and @xcite proposes a similar idea in the multi-document summarization setting. @xcite uses adversarial training and reinforcement learning to make the summary human-readable. @xcite adopts denoising autoencoders originally used in sentence compression. however, most of these models are only tested on datasets with considerably small article/summary length. also, previous models usually utilize the recurrent neural networks (rnns). however, transformers @xcite have shown superior performances over rnns on various nlp tasks, including machine translation, reading comprehension, sentiment analysis, etc. few efforts have been made to leverage transformers in unsupervised abstractive summarizations. pretraining language model. in recent years, pretraining language models have proved to be quite powerful in solving numerous nlp tasks. the state-of-the-art pretrained models include cove @xcite , @xcite , gpt @xcite , bert @xcite and unilm @xcite . taking advantage of corpora with billions of tokens, the pretrained language models learn universal and robust representations for various semantic structures and linguistic relationships. as a result, pretrained models have been widely used with considerable success in applications such as question answering @xcite , sentiment analysis @xcite and passage reranking @xcite . furthermore, unilm @xcite leverages its sequence-to-sequence capability for abstractive summarization; the bert model has been employed as an encoder in bertsum @xcite for supervised extractive and abstractive summarization. in this paper, we present ted, a pretrained unsupervised abstractive summarization model which is finetuned with theme modeling and denoising on in-domain data. ted utilizes a transformer-based encoder-decoder structure and the pretraining leverages large-scale corpora containing millions of unlabeled articles. our primary contributions are twofold as follows. first, we leverage the lead bias in news articles to pretrain ted. the lead bias is introduced by the journalistic convention of writing using an inverted pyramid structure, placing the most important information in the beginning of an article. we propose to use the leading sentences as the target summary and train the model to predict it during pretraining. in this way, we pretrain a summarization model on a large-scale corpus with 21.4m news articles. the model yields better performance than most existing unsupervised methods. second, to finetune on specific datasets, ted is further trained with a theme modeling loss and a denoising autoencoder. the role of the theme modeling module is to make the generated summary semantically close to the article. the module uses a semantic classifier trained using a discriminative objective function. furthermore, to optimize on the generated summary tokens, we adopt the gumbel-softmax @xcite estimator to replace the non-differentiable arg max. the denoising autoencoder has been previously used in unsupervised machine translation @xcite and sentence compression @xcite , and we employ it to help the model extract salient information from corrupted text. instead of classical word tokenization, we adopt the sentencepiece tokenization @xcite to alleviates the long-standing out-ofvocabulary (oov) problem in language generation tasks @xcite . we test ted on several benchmark datasets. the experimental results show that ted outperforms all unsupervised abstractive baselines on all datasets. for example, on the cnn/dm dataset, it outperforms the state-of-the-art unsupervised abstractive model by more than 9 rouge-1 points and compares favorably with most unsupervised extractive models. we further show that ted is capable of generating novel words and phrases in summaries and is a highly abstractive system even compared with supervised systems. ## methodology in this section, we will go through the model structure of ted, i.e. the transformer encoder and decoder. then we introduce the pretraining method and two in-domain finetuning objectives: theme modelling and the denoising autoencoder. the overall architecture of ted is illustrated in fig. 1 . ## conclusion in this paper, we propose ted, an unsupervised abstractive summarization model. first, we introduce an effective large-scale pretraining approach leveraging the lead bias in news articles. the pretraining employs automatic filtering mechanism and does require any human-labeled data. we then develop a finetuning scheme to induce the semantic similarity between summaries and input articles, together with a denoising autoencoder to improve the quality of generated summaries. experiments across three datasets show that ted significantly outperforms unsupervised abstractive baselines.