L EARN TO E NCODE T EXT AS C OMPREHENSIBLE S UMMARY BY G ENERATIVE A DVERSARIAL N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Auto-encoders compress the input data into latent-space representation and reconstruct the original data from the representation.
The latent representation cannot be easily interpreted by human.
In this paper, we propose a new idea to train an auto-encoder that encodes input text into comprehensible sentences.
The autoencoder is composed of a generator and a reconstructor.
The generator encodes the input text into a shorter word sequence, and the reconstructor recovers the input of generator from the output of generator.
To make the generator output comprehensible by human, a discriminator restricts the output of the generator to look like the summaries written by human.
By taking the generator output as the summary of the input text, abstractive summarization can be achieved without document-summery pairs as training data.
Promising results were obtained on both English and Chinese corpora.
I NTRODUCTION
When it comes to learning representation of data, one of the popular approaches is auto-encoder architecture which compresses the data into latent representation without supervision.
This paper focuses on learning representation of text.
Because text is a sequence of words, to encode a sequence, sequence-to-sequence (seq2seq) auto-encoder (Li et al., 2015; Kiros et al., 2015) is usually used, in which a RNN is used to encode the input sequence into a ﬁxed length representation, and then another RNN to decode this input sequence out of that representation.
The latent representation can be used in the downstream applications, but they are usually not explanatory to human.
Is it possible for machine to use human-readable language as latent representation of text?
In this work, we show that it is absolutely possible for auto-encoder to use comprehensible natural language as latent representation of text if machine is given some example summaries written by humans.
The comprehensible latent representation can be considered as the summaries of text.
In this way, unsupervised abstractive summarization is achieved.
The idea of using human languages as latent representation has been explored (Miao & Blunsom, 2016), but only in semi-supervised scenario.
The previous work uses a prior distribution from pretrained language model to constrain the generated sequence to natural language.
However, they trained their model on some labeled data to teach their compressor network generate text summary.
In this paper, we do not need any labeled data to learn such representation.
The proposed model is composed of three components: generator, discriminator and reconstructor.
The generator and reconstructor together form an auto-encoder of text.
The generator generates latent representation from input text as encoder.
Instead of using a vector as latent representation, the generator generates a word sequence much shorter than the input text.
From the shorter text, the reconstrutor generates the original input of the generator.
By minimizing the reconstruction errors, the generator learns to generate short text containing the main information of the original input.
We choose seq2seq model as the models of generator and reconstructor because both have input and output sequences with different lengths 1 .
However, it is very possible that the output word sequence of generator can be processed by the reconstructor, but not comprehensible by humans.
In this paper, instead of regularizing the generator output by pretraining language model (Miao & Blunsom, 2016), we use the third component, discriminator, to regularize the output word sequence of generator.
The discriminator and the gener- ator form a generative adversarial network (GAN) (Goodfellow et al., 2014).
GANs are generative models composed of a generator and a discriminator.
The discriminator tries to discriminate between the output of generator and real data, and the generator produces outputs as similar as real data as possible to confuse discriminator.
In this paper, we feed document summaries to the discriminator as real data 2 .
The summaries do not have to be paired with the documents used to train generator and reconstructor.
With GAN framework, discriminator teaches generator how to create human-like summary sentences as latent representation, but it only guarantees the generator to generate grammatically correct sentences but not necessarily represent the input text.
On the other hand, reconstructor teaches generator how to produce sentence capturing the core idea of source text.
However, generating discrete distribution with GAN is extremely challenging, since it is difﬁcult to evaluate the distance between continuous distribution from generator and discrete distribution of real sample.
In addition, if we feed sampled words from generator output distribution to discriminator, the process of word selection is non-differentiable, which makes gradient from discriminator unable to back-propagate to generator.
There are two kinds of ways on language generation with GAN: training with adversarial REINFORCE regarding words as actions or directly feeding output layer of generator to discriminator making gradient able to back-propagate to generator.
In this work, on language generation with GAN, we explore both methods in the experiments.
We evaluate the results on abstractive text summarization task in which machine generates text summary by its own words.
The model is learned from a set of unpaired documents and summaries.
The summaries can come from another set of documents not related to the training documents.
Hence, the training is unsupervised.
We take the output word sequence of the generator as the summaries of the input text.
The results show that the generator can generate summaries with reasonable quality on both English and Chinese corpora.
R ELATED W ORK GAN ON L ANGUAGE G ENERATION
The major challenge for applying GAN on sentence generation is the discrete nature of natural language.
To generate a word sequence, the generator usually has non-differential part like argmax or other sample function, which makes the original GAN fails.
Therefore, new kinds of GANs are proposed for sentence generation.
SeqGAN (Yu et al., 2017) tackles the sequence generation problem with reinforcement learning.
In this paper, we refer this kind of approaches as adversarial REINFORCE.
In adversarial REINFORCE, the generator is regarded as agent, the generated sequence of words is viewed as sequence of actions, and the current state is deﬁned on generated sequence so far and input prior.
However, the discriminator only measures the quality of whole sentence, and thus the reward is extremely sparse and the rewards assigned to all actions in sequence are all the same.
To tackle this problem, they propose MC search to evaluate approximate rewards at each timestep, but this method suffers from high time complexity.
Following this idea, (Li et al., 2017) uses another way to evaluate expected reward at each timestep.
They break both generated sequence and real sequence into partial sequence, and discriminator discriminates between generated and real partial sequence.
Inspiring by this idea, we introduce another way to evaluate expected reward at each timestep in our proposed self-critic in adversarial REINFORCE.
On the other hand, in Improved Training of Wasserstein GANs paper (Gulrajani et al., 2017), instead of feeding discrete word sequence, they directly feed generator output layer to discriminator.
The main reason this method works is that they use earth mover’s distance on GAN proposed in (Arjovsky et al., 2017) which is capable to evaluate the distance between discrete and continuous distribution.
In order to satisfy the requirement of earth mover’s distance, they use gradient penalty trick to conﬁne complexity of discriminator function.
Their method achieves an amazing result that it’s the ﬁrst paper training GAN on language generation without pretraining.
In our work, we also make experiment on this method with discriminator setting almost same as original paper.
, the structure of summary sentence is concise, and they usually contain some speciﬁc words like country or human name, or some key verbs in source text.
Because the following application is document summarization, using document summaries as real data is very helpful.
It is possible to use text with different styles as real data for discriminator to make the output of the generator have different styles.
A BSTRACTIVE T EXT S UMMARIZATION
Recently, the model architectures of abstractive text summarization basically use the sequence-tosequence (Sutskever et al., 2014) framework combined with some extra novel mechanism.
One popular mechanism is attention (Bahdanau et al., 2015), which has shown to be helpful on summarization (Nallapati et al., 2016; Rush et al., 2015).
It’s also possible to directly optimize evaluation metric such as ROUGE (Lin, 2004) with reinforcement learning (Ranzato et al., 2016; Paulus et al., 2017; Bahdanau et al., 2016).
Hybrid pointer-generator network (See et al., 2017) selects words from original text with pointer (Vinyals et al., 2015) or from the whole vocabulary with a trained weight.
In order to eliminate repetition, it uses coverage vector (Tu et al., 2016) keeping track of attended words and use coverage loss to encourage model focus on diverse words.
While most papers focus on supervised learning with novel mechanism, we explore on training model with unsupervised manner.
P ROPOSED M ETHOD
The overview of the proposed model is shown in ﬁgure 1.
The model is composed of three components: generator G , discriminator D and reconstructor R .
Both G and R are seq2seq hybrid pointergenerator network (See et al., 2017).
They take a word sequence as input and output a sequence of word distributions.
On the other hand, D takes a sequence as input and outputs a scalar.
The model is learned from a set of documents x and summaries y real , but the documents and summaries are unpaired .
To train the model, a training document x = { x 1 , x 2 , ..., x t , ..., x T } , where x t represents a word, is fed to G , and it outputs a sequence of word distributions G ( x ) =
{ y 1 , y 2 , ..., y n , ..., y N } , where y n is a distribution over all the words in the lexicon.
Then we sample a word y sn from each distribution y n , and a word sequence y s = { y s 1 , y s 2 , ..., y sN } is obtained according to G ( x ) .
We feed the sampled word sequence y s to reconstuctor R , and R outputs another sequence of word distributions ˆ x .
The reconstuctor R manages to reconstruct the original text x from y s .
That is, we want the output of reconstuctor ˆ x as close to the original text x as possible, so the loss for training the reconstuctor R ,
R loss , is deﬁned as below.
[Equation]
where reconstruction loss l s ( x, ˆ x ) is cross-entropy loss computed between reconstructor output sequence ˆ x and source text x .
The subscript s in l s ( x, ˆ x ) means that ˆ x is reconstructed from y s .
K is the number of training examples (documents), and (1) is the summation of the cross-entropy loss over all the training documents x .
In the proposed model, the generator G and reconstuctor R form an auto-encoder.
However, the reconstuctor R does not directly take generator output distribution G ( x ) as input 3 .
Instead, the reconstuctor takes sampled discrete sequence y s as input.
Due to the non-differentiable property of discrete sequence, we apply the REINFORCE algorithm, which will be described in section 4.
In addition to reconstruction, we need discriminator D to discriminate between the real summary sequence y real and the generated sequence y s to regularize generated sequence satisfying summary distribution.
D learns to give y real larger scores while y s smaller scores.
The loss for training the discriminator D is denoted as D loss , which will be further described in section 5.
G learns to minimize the reconstruction error R loss , whereas maximize the loss of the discriminator
D to generate summary sequence y s that can not be tell real or generated by D .
The loss for training the generator G , G loss , is shown as below:
[Equation]
where D (cid:48) loss is highly related to D loss , but not necessary the same, and α is a hyper-parameter.
After obtaining the optimal generator G ∗ by minimizing (2), it can be used to generate summaries.
The the generator G and discriminator D form a GAN.
We use two different adversarial training methods to train D and G , and these two methods have their own discriminators 1 and 2 as shown in ﬁgure 1.
Discriminator 1 takes generator output layer G ( x ) as input, while discriminator 2 takes sampled discrete word sequence y s as input.
The two methods will be described respectively in sections 5.1 and 5.2.
M INIMIZING R ECONSTRUCTION E RROR
Due to the non-differentiable property of discrete sequence, we apply REINFORCE algorithm.
The generator is considered as an agent whose reward given source text x is − l s ( x, ˆ x ) .
Maximizing the reward is equivalent to minimizing the reconstruction loss R loss in (1).
However, the reconstruction loss differs a lot from sample to sample, and thus the rewards to generator is not stable.
We should add a baseline to reduce their difference.
We apply self-critical sequence training (Rennie et al., 2017), and the modiﬁed reward r R ( x ) from reconstructor R with baseline for generator is shown as below,
[Equation]
where − l a ( x, y a ) − b is the baseline.
l a ( x, ˆ x ) is also the cross-entropy reconstruction loss as l s ( x, ˆ x ) , except that ˆ x is obtained from y a instead of y s .
y a is a word sequence { y a 1 , y a 2 , ..., y an , ..., y aN } , where y an is selected by argmax function from the output distribution of generator y n .
Because in early stage of training sequence y s barely gets higher reward than sequence y
a
, we introduce the second baseline score b gradually decreases to zero to encourage exploration.
Then, the generator is updated by REINFORCE algorithm with reward r R ( x ) to minimize R loss .
GAN T RAINING
We feed summary sentence to discriminator as real sample, with adversarial training, generator manages to produce sentence as similar to summary as possible.
In this paper, we make experiment on two kinds of methods on language generation with GAN.
In section 5.1 we directly feed generator output probability distributions to discriminator and apply Wasserstein GAN (WGAN) with gradient penalty.
In section 5.2, we explore adversarial REINFORCE that feeds sampled discrete word sequence to discriminator and uses the evaluation of quality of sequence from discriminator as reward signal to generator.
5. D ISCRIMINATOR 1: W ASSERSTEIN GAN
The model of discriminator is shown as discriminator1 in lower-left of ﬁgure 1.
The loss D loss of discriminator is:
[Equation]
where K denoted the number of training examples, and k denoted k’th example in batch.
The last term in (3) is gradient penalty (Gulrajani et al., 2017).
We interpolate between generator output layer G ( x ) and real sample y real , and apply gradient penalty on interpolated sequence y i .
β decides the scale of gradient penalty.
In equation (2), we add the reconstruction loss for generator R loss and loss related to discriminator
D (cid:48) loss together to formulate total generator loss.
For WGAN, D (cid:48) loss is the score of the generated example,
[Equation]
5. S ELF -C RITIC A DVERSARIAL REINFORCE
In this section, we describe in detail our proposed adversarial REINFORCE method.
The core idea is we make LSTM discriminator evaluate the current quality of generated sequence { y s 1 , y s 2 , ..., y si } at each time step i .
Hence, generator will know that compared to last time step, the generated sentence becomes worse or better, so it can easily ﬁnd the problematic generation step in a long sequence, and ﬁx the problem easily.
5.2. D ISCRIMINATOR
As discriminator2 shown in ﬁgure 1, the model of discriminator is a one-way LSTM network which takes discrete word sequence as input.
At time step i , given input word y si it will predict the current score s i based on the sequence { y 1 , y 2 , ..., y i } .
The score is viewed as the quality of current sequence.
In order to compute the loss of discriminator, we sum all the scores { s 1 , s 2 , ..., s N } of the whole sequence y s and the loss of discriminator D loss is:
[Equation]
where K and N denote to the number of training examples and generated sequence length respectively.
With the loss mentioned above, the discriminator tries to determine whether the current sequence is real or false as early as possible.
The earlier timestep discriminator determines current sequence is real or false, the lower loss it get.
We show a real example on how it works in ﬁgure 2.
5.2. S ELF -C RITICAL G ENERATOR
Since we feed discrete sequence y s to discriminator, the gradient from discriminator can not directly back-propagate to generator.
Here, we apply policy gradient method.
At timestep i , we use the i − 1 timestep score
s i − 1 from discriminator as its self critical baseline.
The reward r Di evaluates compared to timestep i − 1 the quality of sequence in timestep i is worse or better.
The generator reward r Di from D is:
[Equation]
However, some sentence may be judged as bad sentence at previous timestep, but judged as good sentence at latter and vice versa.
To tackle this problem, we use discounted expected reward d with discounted factor γ , the discounted reward d i at time step i is:
[Equation]
The adversarial REINFORCE score related to discriminator D (cid:48) loss in equation (2) is:
[Equation]
We apply likelihood ratio trick to approximate gradient.
E XPERIMENT 6. E XPERIMENTAL S ETUP
Our model is evaluated on two datasets: Chinese Gigaword and English Gigaword.
The seq2seq models of generator and reconstructor are all composed of two one-layer LSTM with hidden layer size 600.
In section 5.1 which feeds generator output layer to discriminator, we used 4 residual block with hidden dimension 512 as our discriminator, set the weight β of gradient penalty to 10, and used RMSPropOptimizer with learning rate 0.00001 and 0.001 on generator and discriminator respectively.
In section 5.2, we applied weight clipping to approximate function required by Wasserstein distance.
We only used one hidden layer one-way LSTM with hidden size 512 and clipped the value of its weights to ± 0 . 15 , and used RMSPropOptimizer with learning rate 0.0001 and 0.001 on generator and discriminator respectively.
During testing, when using the generator to generate summary, we simply took the words with highest probability without beam-search, and we eliminated repetition.
Before jointly training whole model, we pretrained three major components, generator, discriminator and decoder, in our model separately.
First, we pretrained generator with unsupervised approach to make generator grasp some semantic meaning of source text, and roughly initialized with a language model.
We found that the different pretraining methods on generator inﬂuenced ﬁnal performance dramatically in all of our experiment, and thus it’s crucial to ﬁnd a proper unsupervised pretraining method to help machine understand semantic meaning.
We use different pretraining strategies which will be further described in the following discussion.
We pretrained discriminator and decoder respectively with pretrained generator’s outputs to ensure these two critic networks provide good feedback to generator.
6. C HINESE G IGA W ORD Methods
[Table]
(A) Training with Paired Data (Supervised)
[Table]
(B) Trivial Baselines: lead -
[Table]
(C) Unsupervised (B-1)
Pretrained Generator
[Table]
(C-2)
WGAN
[Table]
(C-3)
Adversarial REINFORCE
[Table]
The Chinese Gigaword corpus is composed of 2.2M paired data of headlines and news.
We preprocessed the raw data as following.
First, we selected the most frequent 4K Chinese characters as our vocabulary.
We ﬁltered out headline-news pairs containing too many Chinese characters out from our vocabulary or the length of news was extremely long or short.
After ﬁltering, we obtained 1.1M headline-news pairs.
We randomly selected 5K headline-news pairs from ﬁltered data as our testing set, 5K headline-news pairs as our validation set and the remaining pairs as our training set.
During training or testing, the generator only took the ﬁrst 80 Chinese characters of the source texts as input.
Before jointly training whole model, we pretrained generator to predict the next sentence of part of source text.
The next sentence was not included in given text to generator.
If the next sentence of given text contained more than 50% words not appearing in given text, we ﬁltered out this pretraining sample pair.
This pretraining method allows generator to capture the important semantic meaning of source text and also initialize with a crude language model.
The results are shown in table 1.
Row (A) is the results using 1.1 million document-summary pairs to directly train the generator without reconstructor and discriminator, which is the upper bound of the proposed approach.
In row (B), we simply took the ﬁrst ﬁfteen words in a document as summary 4 .
Part (C) are the results obtained in the unsupervised scenario without paired data.
We show the results of the pre-trained generator in row (C-1), and rows (C-2) and (C-3) are the results for two GAN training methods respectively.
We found that although there is still a performance gap between the unsupervised methods and supervised methods (rows (C-2), (C-3) v.s. (A)), the proposed method obtained much better performance than the trivial baselines (rows (C-2), (C-3)
v.s.
(B)).
In the above experiments, we set the weight α in 2 controlling R loss to be 10.
6. E NGLISH G IGA W ORD Methods
[Table]
Data
[Table]
Supervised
[Table]
Trivial
[Table]
Learning
[Table]
Learning
[Table]
in part (
On English Gigaword, we set our vocabulary size to 15k, and we used the dataset preprocessed by (Rush et al., 2015) for training and testing.
We used 3.8M unpaired training data for our training, and we used full 200k ﬁltered data in validation set for testing.
The length of source texts in English Giga Word dataset is comparatively short, and it’s difﬁcult to split the last sentence from source text, hence the previous pretraining method on Chinese Gigaword does not ﬁt for this dataset.
In order to get a proper initialization, we randomly selected consecutive 6-11 words in source text, then we randomly swapped 70 % words in source text.
Given text with incorrect words arrangement, generator predicted the selected words in correct arrangement.
The reason we pretrained with this approach is that we expect the generator to initialize with a rough language model and focus on diverse part of text 5 .
The results on English Giga Word are shown in table 2.
Compared with the trivial baselines (row (B)), the proposed approach (row (C-2)and (C-3)) improved a lot in terms of ROUGE-1.As shown in ﬁgure 3, unsupervised method is capable to select key words in source text and generate text summary.
However, as shown in table 2, although both unsupervised methods get ROUGE-1 score close to supervised training, they get lower score on ROUGE-2 especially on training GAN with reinforcement learning.
The main reason is they are able to extract key words from source text, but sometimes fail to arrange these words in correct order.
As shown in ﬁgure 4 part (C-3), the words in sentence generated by unsupervised manner are put in incorrect arrangement that the Italian prime minister, Berlusconi, should not visit himself.
6. T RANSFER L EARNING
In this subsection, we study transfer learning.
We used CNN/Daily Mail dataset Hermann et al.
; Nallapati et al.
2016 preprocessed by the script provided by See et al.
as our source domain
S , and English Giga Word as our target domain T .
The data distributions among two datasets are quite different.
In English Giga Word, the articles consist of 32 words in average and the summary consists of one sentence with 8 words in average.
While in CNN/Daily Mail dataset the articles are composed of 790 words and the summary are composed of several sentences.
We preprocessed the source data into three parts: 50K source articles
S a , 50K source summaries S t and 240K source real sentences S r .
We only took ﬁrst 30-45 words in original source domain articles as our new source article S a .
However, unlike English Giga Word, the summary in S is composed of few sentences.
We split the summary into several sentences and viewed all split sentences as ground truth S t to S a .
Hence, one S a has several ground truth S t .
Then, all the sentences in original source summary were viewed as source real summary sentences S r .
We ﬁltered out the samples containing so many unknown words or the bad summary sentences.
The transfer learning is applied on two direction.
The ﬁrst direction is we pretrained our generator on the data from source dataset.
We pretrained our generator with S a as input, and generator predicted
S t .
We pretrained generator in this manner in all of the transfer learning experiments.
Then, after pretraining, the generator was further learned jointly with reconstructor and discriminator on the data from target domain.
The results are shown in the part (D) of Table 2.
We found that pretraining on the source data set did not degrade the performance, and even improved the performance in some cases (parts (D)
v.s. (C)).
The second direction is we used the summary from source domain as our real data.
The experiments we made so far required unpaired data of summary and text in target domain.
In this experiment, we made our task more challenging that we used summary S r from source domain S as real data for discriminator, and generator took text of target domain as input.
In fact, each summarization task dataset has it own data distribution, for example writing style or preferred words of summary or content of articles.
In this experiment, we set the weight alpha to be 50.
We used larger α to prevent overﬁtting to S r .
If the weight α is too small, as training going on, the summary produced by generator becomes more and more unrelated to article, and ROUGE score becomes lower and lower.
The results without using summaries in the target domain are shown in the part (E).
We found that using title from another dataset gets lower ROUGE score on target testing set (parts (E) v.s.
(D))
due to the mismatch between the summaries of the source and target domains.
However, the discriminator still roughly regularizes the language model of generated word sequence.
After training, the model still greatly enhanced ROUGE score of the pre-trained model (rows (E-1), (E-2) v.s. (D-1)).
The results in part (E) are just comparable with the trivial baselines in row (B), but from the real examples, we found that the results in part (E) were better, which is not reﬂected in the ROUGE scores.
In ﬁgure 3, the result in part (E) is better than leading 8 words and the result of pretrained generator in part (D-1).
To support this idea, we provide more examples in appendix.
6. GAN T RAINING
The two methods for training GAN are not comparable since their settings are almost different, but we can still discuss the pros and cons of these two methods.
In fact, both methods are sensitive to initialization of generator especially for adversarial REINFORCE, since it requires a good exploration direction.
In Chinese Gigaword, as shown in table 1, since generator was initialized with a better parameters, the adversarial REINFORCE got better performance than its performance in
English Giga Word.
In addition, compared with English Gigaword, the vocabulary size of Chinese Gigaword is smaller, which means it’s easier for generator to select a proper word as action.
On the other hand, training with feeding output layer converges faster.
It manages to make the distribution as sharp as possible at early stage of training because it directly evaluates the distance between continuous distribution from generator and discrete distribution of real data.
C ONCLUSION AND F UTURE W ORK
By GAN, we propose a model to encode text as comprehensible summary, which is learned without document-summary pairs.
Promising results were obtained on both Chinese and English corpora.
In future work, we are going to explore more techniques for natural language generation by GAN.
Moreover, we will use extra discriminators to control the style and sentiment of the generated summaries.