On Extractive and Abstractive Neural Document Summarization with Transformer Language Models (2020)

Abstract: We present a method to produce abstractive summaries of long documents that exceed several thousand words via neural abstractive summarization. We perform a simple extractive step before generating a summary, which is then used to condition the transformer language model on relevant information before being tasked with generating a summary. We also show that this approach produces more abstractive summaries compared to prior work that employs a copy mechanism while still achieving higher ROUGE scores. We provide extensive comparisons with strong baseline methods, prior state of the art work as well as multiple variants of our approach including those using only transformers, only extractive techniques and combinations of the two. We examine these models using four different summarization tasks and datasets: arXiv papers, PubMed papers, the Newsroom and BigPatent datasets. We find that transformer based methods produce summaries with fewer n-gram copies, leading to n-gram copying statistics that are more similar to human generated abstracts. We include a human evaluation, finding that transformers are ranked highly for coherence and fluency, but purely extractive methods score higher for informativeness and relevance. We hope that these architectures and experiments may serve as strong points of comparison for future work. Note: The abstract above was collaboratively written by the authors and one of the models presented in this paper based on an earlier draft of this paper.

## introduction automatic text summarization is the process of compressing a document while preserving key information content and meaning. this process is often achieved through extractive or abstractive 1 introduction language models (lms) are trained to estimate the joint probability of an arbitrary sequence of words or characters using a large corpus of text. they typically factorize the joint distribution of tokens p(x1,x2 ...xn) into a product of conditional probabilities qn i p(xi|@xmath0. it is possible to use n-gram based models to estimate these conditional probabilities via counts, relying on markovian assumptions. however, markovian assumptions and the curse of dimensionality make it harder for n-gram lms to model long range dependencies and learn smooth functions that can learn similarities between words in the vocabulary. this has led to a preference for recurrent or feed-forward neural language models @xcite in recent years due to to their ability to learn expressive conditional probability distributions @xcite . the sequence-to-sequence (seq2seq) paradigm @xcite uses language models that learn the conditional probability of one sequence given another. here, a language model serves as a "decoder" that is typically conditioned on a representation of an input sequence produced by an encoder neural network. these types of encoder-decoder architectures have been particularly successful when applied to problems such as machine translation @xcite and abstractive summarization @xcite . the encoder and conditional decoder language models are often paramaterized as recurrent neural networks (rnns). attention mechanisms @xcite are used in the decoder to provide more informative conditioning on the representations produced by the encoder and to ease gradient flow into the encoder. rnns however, are limited by their sequential nature, making them 1) difficult to optimize and learn for long sequences with long range dependencies @xcite , and 2) hard to parallelize on modern hardware like gpus, limiting their scalability. ## introduction language models (lms) are trained to estimate the joint probability of an arbitrary sequence of words or characters using a large corpus of text. they typically factorize the joint distribution of tokens p(x1,x2 ...xn) into a product of conditional probabilities qn i p(xi|@xmath1. it is possible to use n-gram based models to estimate these conditional probabilities via counts, relying on markovian assumptions. however, markovian assumptions and the curse of dimensionality make it harder for n-gram lms to model long range dependencies and learn smooth functions that can learn similarities between words in the vocabulary. this has led to a preference for recurrent or feed-forward neural language models @xcite in recent years due to to their ability to learn expressive conditional probability distributions @xcite . the sequence-to-sequence (seq2seq) paradigm @xcite uses language models that learn the conditional probability of one sequence given another. here, a language model serves as a "decoder" that is typically conditioned on a representation of an input sequence produced by an encoder neural network. these types of encoder-decoder architectures have been particularly successful when applied to problems such as machine translation @xcite and abstractive summarization @xcite . the encoder and conditional decoder language models are often paramaterized as recurrent neural networks (rnns). attention mechanisms @xcite are used in the decoder to provide more informative conditioning on the representations produced by the encoder and to ease gradient flow into the encoder. rnns however, are limited by their sequential nature, making them 1) difficult to optimize and learn for long sequences with long range dependencies @xcite , and 2) hard to parallelize on modern hardware like gpus, limiting their scalability. figure 1 : our approach for abstractive summarization of a scientific article. an older version of this paper is shown as the reference document. first, a sentence pointer network extracts important sentences from the paper. next, these sentences are provided along with the whole scientific article to be arranged in the following order: introduction, extracted sentences, abstract & the rest of the paper. a transformer language model is trained on articles organized in this format. during inference, the introduction and the extracted sentences are given to the language model as context to generate a summary. in domains like news and patent documents, the introduction can be replaced by the entire document. recent work by @xcite (gpt-2) has demonstrated that transformer language models (tlms) trained on web text can inadvertently learn to perform abstractive summarization, since a large crawl of web documents may contain some documents which have a "tl;dr" token followed by a summary. we are interested here in explicitly configuring autoregressive transformer models to generate summaries in an intentional and focused manner. since summaries or abstracts typically appear at the beginning of a document, a model trained from such web-crawl data does not enforce strong conditioning on the text to be summarized. our tests using models naively trained on web-crawl data yielded summarization quality far below baseline methods. however, in this paper we explore what can be achieved through simply ordering the passages of an input text, correctly structuring the task definition and training procedure. we also examine the impact of combining this approach with simple but high quality extractive techniques. while pure language models can be applied to short input documents, memory considerations make it difficult to scale to long documents. further, as high quality extractive summarization methods illustrate, much of the content of a long document is not needed to create a summary. for these reasons we also explore a hybrid approach which combines an extractive and abstractive approach. we achieve this by stepping away from the classical end-to-end sequence-to-sequence paradigm, using an initial extractive step that reduces the amount of context for a subsequent abstractive step (see figure 1 ). such an approach could be thought of as a form of hard attention. moreover, we show that such a paradigm works even for datasets where the entire input can fit in memory, i.e. see in general, as we shall detail in our experiments below, we find that tlms are surprisingly effective at summarizing long documents, outperforming typical seq2seq approaches, even without using copying/pointing mechanisms, an encoder or additional losses. our contribution consists of an extensive set of large scale experiments comparing our hybrid extractive and abstractive approach to long document summarization with different variants of our model, strong and simple baselines as well as with state-of-the-art summarization models (see section 3.2 for a complete description of comparisons). we examine these models through rouge scores, through a study of the amount of n-gram copying performed by different models, as well as through a human evaluation using a standard protocol. we find that our hybrid approach yields results that surpass current state-of-the-art results on several metrics of these evaluations. we see our extensive experimentation and the wide variety of evaluation protocols provided here as being a key part of the contribution provided by this work and we hope that the analysis, insights and models here will serve as strong yet simple baselines for future comparison and research. ## related work the earliest attempts at automatic summarization focused on extractive techniques, which find words or sentences in a document that capture its most salient content. recently, with advances in distributed representations of words, phrases and sentences, researchers have proposed to use these to compute similarity scores. such techniques were further refined by @xcite ; cheng and lapata (2016); chen and bansal (2018) with encoder-decoder architectures -the representations learned by the encoder are used to choose the most salient sentences. @xcite and @xcite trained encoder-decoder neural networks as a binary classifier to determine if each sentence in a document should belong to the extractive summary or not. @xcite use a pointer network @xcite to sequentially pick sentences from the document that comprise its extractive summary. such techniques however heavily rely on the span of words from the input document. human summarizers have four common characteristics. they are able to (1) interpret a source document, (2) prioritize the most important parts of the input text, (3) paraphrase key concepts into coherent paragraphs and (4) generate diverse output summaries. while extractive methods are arguably well suited for identifying the most relevant information, such techniques may lack the fluency and coherency of human generated summaries. abstractive summarization has shown the most promise towards addressing points (3) and ( 4 ) above. abstractive generation may produce sentences not seen in the original input document. motivated by neural network success in machine translation experiments, the attention-based encoder decoder paradigm has recently been widely studied in abstractive summarization @xcite @xcite . the advantages of extractive, abstractive and attention-based models were first combined in @xcite with a copy mechanism for out-of-vocabulary words present in the source document. similarly, @xcite used the attention scores to calculate the probability of generating vs copying a word. the most similar approach to our hybrid extractive and abstractive technique is that of @xcite ; @xcite . in such set-ups, an extractor first selects salient sentences from the input. then, an abstractive summarizer rewrites extracted sentences into a final summary. our framework has a few advantages over previous methods. 1), we explore high capacity transformer lms akin to @xcite as our abstractive summarizer, which results in grammatical and fluent generations 2), our language modeling formulation of the problem allows us to easily "recycle" the input document and use it additional in-domain data for lm training. 3) we improve over previous approaches without the use of a copy mechanism, which results in fewer n-gram copies from the input document. @xcite generate wikipedia articles given references to source material and extracted sentences. they rank the importance of paragraphs found in the reference material based on techniques such as textrank @xcite , a graph based ranking technique. in contrast, the extractive methods we use here are trained discriminatively using an extractive abstract as the target that is generated using an oracle. wikipedia article synthesis also necessarily combines potentially redundant information from multiple documents that is relatively specific and less abstractive compared to the task of writing the abstract of a scientific paper. as seen in figure 2 , human generated (ground-truth) abstractive summaries in our datasets actually have very little word overlap with the source document. ## framework our model comprises two distinct trainable components: 1) an extractive model, comprising a hierarchical encoder that outputs sentence representations, used to either point to or classify sentences in the input, and 2) a transformer language model, conditioned on the extracted sentences as well as a part of or the entire input document. ## results and analysis datasets we experiment with four different large-scale and long document summarization datasets -arxiv, pubmed @xcite , bigpatent @xcite and newsroom @xcite data preprocessing both our extractive and abstractive models use sub-word units computed using byte pair encoding @xcite with 40, 000 replacements. to address memory issues in the sentence pointer network, we only keep 300 sentences per article, and 35 tokens per sentence. evaluation we evaluate our method using fulllength f-1 rouge scores @xcite and re-used the code from @xcite for this purpose. all rouge numbers reported in this work have a 95% confidence interval of at most 0.24. comparison we compare our results to several previously proposed extractive, abstractive and mixed summarization models on rouge scores. rouge scores tend to measure lexical overlap @xcite which favors extractive methods of summarization. since rouge scores do not capture system summary fluency and readability (which typically does not favor abstractive summarization), we also include a human evaluation. for this reason, tables 2 , 3 , 4, 5 have a "type" column to inform the reader on the type model evaluated (@xmath2, @xmath3. all prior results reported on the arxiv and pubmed benchmark are obtained from @xcite , except for the bottom-up model foot_0 @xcite . similarly, prior results for the bigpatent dataset are obtained from @xcite and newsroom from @xcite and @xcite . these methods include lexrank @xcite , sumbasic ( @xcite , lsa @xcite , attention-seq2seq @xcite , pointer-generator seq2seq @xcite , discourse-aware, which is a hierarchical extension to the pointer generator model, @xcite , sent-rewriting (chen and bansal, 2018), rnn-ext (chen and bansal, 2018), exconsumm @xcite . we present our main results on summarizing arxiv and pubmed papers in tables 2, 3. tlm+i+e (g,m) sets a new state-of-the-art on arxiv, pubmed and bigpatent datasets on abstractive summarization rouge scores. our extractive models are able to outperform previous extractive baselines on both the arxiv and pubmed datasets. our extractive techniques also score higher than our abstractive techniques on arxiv and pubmed. again, rouge does not capture all aspects of a summary's quality such as fluency and coherence. for instance, previous work that have used rl to maximize rouge scores have concluded that "rl has the highest rouge-1 and rouge-l scores, it produces the least readable summaries" @xcite . our tlm conditioned on the extractive summary produced by our best extractive model (tlm-i+e (g,m)) outperforms prior abstractive/mixed results on the arxiv, pubmed and bigpatent datasets, except on rouge-l. on newsroom, our tlm model performs close to 7 times better than the other purely abstractive model (seq2seq with attention). we achieve better performance than the pointer generator even on the figure 2 further supports this hypothesis, it is evident that a model with a copy mechanism is often able to copy even upto 25-grams from the article. further, @xcite finds that rouge-l is poorly correlated with human judgements when compared to rouge-1,2,3. in figure 2 : n-gram overlaps between the abstracts generated by different models and the input article on the arxiv dataset. we show in detail which part of the input was copied for our tlm conditioned on intro + extract. as the number of grams increases to 25-grams, 4/5th of the words copied are from the extracted sentences. ## conclusion we have demonstrated that transformer language models can generate high-quality summaries of long sequences of text via an extractive step followed by an abstractive step. we quantitatively measure the positive impact of the extractive step, by comparing it to a abstractive model variant that only sees the input text itself. our approach outperforms previous extractive and abstractive summarization methods on the arxiv, pubmed and big-patent datasets and is less prone to copying entire phrases or sentences from the input text. as with other problem domains, we have observed that abstractive summaries generated by transformers can generate imaginary content. we advise that such evaluations should probe multiple aspects of the summarization results including both factual correctness and coherency. we also note that for evaluating the correctness of the summaries of scientific articles and patents one must have highly trained evaluators who are willing to invest significant amounts of time to read the underlying papers and patents. such studies could therefore require significant investments of resources. while we believe that this work is a step forward towards generating more abstractive summaries, it remains an open challenge to develop abstactive models that respect the underlying facts of the content being summarized while matching the creative ability of humans to coherently and concisely synthesize summaries. however, the tlms we use here are of moderate size compared to what is now possible.