We present a method to produce abstractive summaries of long documents that exceed several thousand words via neural abstractive summarization. We perform a simple extractive step before generating a summary, which is then used to condition the transformer language model on relevant information before being tasked with generating a summary. We show that this extractive step significantly improves summarization results. We also show that this approach produces more abstractive summaries compared to prior work that employs a copy mechanism while still achieving higher rouge scores. Note: The abstract above was not written by the authors, it was generated by one of the models presented in this paper. Language models (LMs) are trained to estimate the joint probability of an arbitrary sequence of words or characters using a large corpus of text. They typically factorize the joint distribution of tokens p(x 1 , x 2 . . . x n ) into a product of conditional probabilities n i p(x i |x <i ). It is possible to use ngram based models to estimate these conditional probabilities via counts, relying on Markovian assumptions. However, Markovian assumptions and the curse of dimensionality make it harder for n-gram LMs to model long range dependencies and learn smooth functions that can learn similarities between words in the vocabulary. This has led to a preference for recurrent or feed-forward neural language models (Bengio et al. 2003; Mikolov et al. 2010 ) in recent years due to to their ability to learn expressive conditional probability distributions (Radford et al. 2019) . The sequence-to-sequence (seq2seq) paradigm (Sutskever, Vinyals, and Le 2014) uses language models that learn the conditional probability of one sequence given another. Here, a language model serves as a "decoder" that is typically conditioned on a representation of an input sequence produced by an encoder neural network.  We demonstrate that Transformer language models are extremely promising at summarizing long texts, and provide a new approach to deep summarization that can be used to generate more "abstractive" summaries. We show that our approach produces more abstractive summaries than state-of-the-art methods without a copy mechanism. We provide an application to text summarization of the arXiv and PubMed datasets, and show that our model outperforms other popular summarization techniques. We also discuss a simple neural extractive model based on pointers networks trained on documents and their salient sentences. We show that this model can be used to augment Transformer language models to generate better summarization results. Note: The abstract above was generated by one of the models presented in this paper, as a summary of this paper.
REF , use a two step approach and employ transformer based models for abstractive summarization.
On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
ArXiv
ArXiv
Computer Science