We propose a new framework for abstractive text summarization based on a sequence-to-sequence oriented encoderdecoder model equipped with a deep recurrent generative decoder ( DRGN ) . Latent structure information implied in the target summaries is learned based on a recurrent latent random model for improving the summarization quality . Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables . Abstractive summaries are generated based on both the generative latent variables and the discriminative deterministic states . Extensive experiments on some benchmark datasets in different languages show that DRGN achieves improvements over the state-ofthe-art methods . Automatic summarization is the process of automatically generating a summary that retains the most important content of the original text document ( Edmundson , 1969 ; Luhn , 1958 ; Nenkova and McKeown , 2012 ) . Different from the common extraction-based and compression-based methods , abstraction-based methods aim at constructing new sentences as summaries , thus they require a deeper understanding of the text and the capability of generating new sentences , which provide an obvious advantage in improving the focus of a summary , reducing the redundancy , and keeping a good compression rate ( Bing et al . , 2015 ; Rush et al . , 2015 ; Nallapati et al . , 2016 ) . * The work described in this paper is supported by a grant from the Grant Council of the Hong Kong Special Administrative Region , China ( Project Code : 14203414 ) . Some previous research works show that human-written summaries are more abstractive ( Jing and McKeown , 2000 ) . Moreover , our investigation reveals that people may naturally follow some inherent structures when they write the abstractive summaries . To illustrate this observation , we show some examples in Figure 1 , which are some top story summaries or headlines from the channel " Technology " of CNN . After analyzing the summaries carefully , we can find some common structures from them , such as " What " , " What-Happened " , " Who Action What " , etc . For example , the summary " Apple sues Qualcomm for nearly $ 1 billion " can be structuralized as " Who ( Apple ) Action ( sues ) What ( Qualcomm ) " . Similarly , the summaries " [ Twitter ] [ fixes ] [ botched @POTUS account transfer ] " , " [ Uber ] [ to pay ] [ $ 20 million ] for misleading drivers " , and " [ Bipartisan bill ] aims to [ reform ] [ H-1B visa system ] " also follow the structure of " Who Action What " . The summary " The emergence of the ' cyber cold war " ' matches with the structure of " What " , and the summary " St. Louis ' public library computers hacked " follows the structure of " What-Happened " . Intuitively , if we can incorporate the latent structure information of summaries into the abstractive summarization model , it will improve the quality of the generated summaries . However , very few existing works specifically consider the latent structure information of summaries in their summarization models . Although a very popular neural network based sequence-to-sequence ( seq2seq ) framework has been proposed to tackle the abstractive summarization problem ( Lopyrev , 2015 ; Rush et al . , 2015 ; Nallapati et al . , 2016 ) , the calculation of the internal decoding states is entirely deterministic . The deterministic transformations in these discriminative models lead to limitations on the representation ability of the latent structure information . Miao and Blunsom ( 2016 ) extended the seq2seq framework and proposed a generative model to capture the latent summary information , but they did not consider the recurrent dependencies in their generative model leading to limited representation ability . To tackle the above mentioned problems , we design a new framework based on sequenceto-sequence oriented encoder-decoder model equipped with a latent structure modeling component . We employ Variational Auto-Encoders ( VAEs ) ( Kingma and Welling , 2013 ; Rezende et al . , 2014 ) as the base model for our generative framework which can handle the inference problem associated with complex generative modeling . However , the standard framework of VAEs is not designed for sequence modeling related tasks . Inspired by ( Chung et al . , 2015 ) , we add historical dependencies on the latent variables of VAEs and propose a deep recurrent generative decoder ( DRGD ) for latent structure modeling . Then the standard discriminative deterministic decoder and the recurrent generative decoder are integrated into a unified decoding framework . The target summaries will be decoded based on both the discriminative deterministic variables and the generative latent structural information . All the neural parameters are learned by back-propagation in an end-to-end training paradigm . The main contributions of our framework are summarized as follows : ( 1 ) We propose a sequence-to-sequence oriented encoder-decoder model equipped with a deep recurrent generative decoder ( DRGD ) to model and learn the latent structure information implied in the target summaries of the training data . Neural variational inference is employed to address the intractable posterior inference for the recurrent latent variables . ( 2 ) Both the generative latent structural information and the discriminative deterministic variables are jointly considered in the generation process of the abstractive summaries . ( 3 ) Experimental results on some benchmark datasets in different languages show that our framework achieves better performance than the state-of-the-art models . We propose a deep recurrent generative decoder ( DRGD ) to improve the abstractive summarization performance . The model is a sequenceto-sequence oriented encoder-decoder framework equipped with a latent structure modeling component . Abstractive summaries are generated based on both the latent variables and the deterministic states . Extensive experiments on benchmark

Challenge: Although humans follow inherent structures in summary writing, currently there are no abstractive summarization models which take latent structure information and recurrent dependencies into account.
Approach: They propose a Variational Auto-Encoder-based sequence-to-sequence oriented encoder-decoder model with a deep recurrent generative decoder which learns latent structure information implied in the target summaries.
Outcome: The proposed model outperforms the state-of-the-art models on some datasets in different languages.