R EL GAN: R ELATIONAL G ENERATIVE A DVERSARIAL N ETWORKS FOR T EXT G ENERATION
A BSTRACT
Generative adversarial networks (GANs) have achieved great success at generating realistic images.
However, the text generation still remains a challenging task for modern GAN architectures.
In this work, we propose RelGAN, a new
GAN architecture for text generation, consisting of three main components: a relational memory based generator for the long-distance dependency modeling,
the
Gumbel-Softmax relaxation for training GANs on discrete data, and multiple embedded representations in the discriminator to provide a more informative signal for the generator updates.
Our experiments show that RelGAN outperforms current state-of-the-art models in terms of sample quality and diversity, and we also reveal via ablation studies that each component of RelGAN contributes critically to its performance improvements.
Moreover, a key advantage of our method, that distinguishes it from other GANs, is the ability to control the trade-off between sample quality and diversity via the use of a single adjustable parameter.
Finally,
RelGAN is the ﬁrst architecture that makes GANs with Gumbel-Softmax relaxation succeed in generating realistic text.
I NTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) were originally designed to generate continuous data and have achieved a lot of success at generating continuous samples, such as images.
Recently, GANs were extended to generate discrete data, in particular text sequences (Kusner & Hern ´ andez-Lobato, 2016; Yu et al., 2017; Zhang et al., 2017; Lin et al., 2017; Guo et al., 2017;
Fedus et al., 2018).
However, this extension is not straightforward.
The main issue is that outputs of GANs for the discrete data generation are not differentiable and thus the standard gradient-based techniques cannot be applied directly in these settings.
To overcome this, most state-of-the-art GANs have used the REINFORCE algorithm
(Williams, 1992) and its variants that originate from the reinforcement learning (RL) community to train the generator while the discriminator is still a classiﬁer to discriminate real and generated text and provides reward signals for the generator updates.
A detailed description of the related work is deferred to Appendix A.
Although these state-of-the-art GANs have shown some promising results in text generation as compared to the conventional maximum likelihood estimation (MLE) method, they also suffer from some fundamental issues, including training instability and mode collapse.
First, their performance is quite sensitive to random parameter initializations and hyperparameter choices (Semeniuta et al., 2018).
Moreover, many GANs heavily employ RL heuristics, such as Monte Carlo search (Yu et al., 2017) and hierarchical RL
(Guo et al., 2017), making the already difﬁcult-to-train GANs more complicated and thus the individual role of adversarial training unclear.
The second issue is mode collapse as the generated text sentences tend to be less diverse (Semeniuta et al., 2018; Fedus et al., 2018) and it becomes more severe when generating longer sentences.
The mode collapse issue can be caused either by a lack of expressive power in the generator (since it may not be capable of covering many more complex modes in data distribution), or by a less informative guiding signal in the discriminator (as it may constrain the generator updates to within certain modes).
In this work, we propose a new GAN architecture – Relational GAN (RelGAN), whose design is motivated by the issues identiﬁed above.
The RelGAN architecture mainly consists of three parts: 1) a relational memory (Santoro et al., 2018)
based generator, which promises more expressive power and better ability of modeling longer-range dependencies in text; 2) Gumbel-Softmax relaxation (Jang et al., 2016; Maddison et al., 2016) for training GANs on discrete data, which simpliﬁes our model, enabling us to stay within a classical GAN framework without intensive RL heuristics; 3) multiple embedded representations in the discriminator, enabling a more diverse and informative signal for the generator updates.
We experimentally demonstrate that RelGAN outperforms most current models in terms of sample quality and diversity.
Furthermore, we show via ablation studies that each part of RelGAN plays an important role in its performance improvements.
A key advantage of our method, that distinguishes it from other GANs, is the ability to control the trade-off between sample quality and diversity, via the use of a single adjustable parameter .
Finally, to the best of our knowledge, RelGAN is the ﬁrst architecture to demonstrate that GANs with Gumbel-Softmax relaxation are capable of generating realistic text.
R EL GAN 2. R ELATIONAL M EMORY BASED G ENERATOR
Current dominant GANs for text generation, such as Kusner & Hern ´ andez-Lobato (2016); Yu et al.
; Lin et al.
; Guo et al.
(2017); Fedus et al.
(2018) are built using LSTM (Hochreiter & Schmidhuber, 1997) as the generator architecture.
However, the LSTM-based generator might be the bottleneck of GANs from the following experimental observations: 1)
The discriminator’s loss value very quickly goes to near its minimum after few iterations of adversarial training.
It means that the discriminator may be much more powerful than the generator and can easily distinguish between real and fake samples.
Mode collapse in current GANs (Fedus et al., 2018) may also partly indicate the incapacity of generator, as it may not be expressive enough to ﬁt all the modes of data distribution.
3) Current GANs perform poorly at long sentence generation (Guo et al., 2017), and we know that LSTM packs all information about the previous text sequences into a common hidden vector, potentially limiting its ability of modeling the long-distance dependency.
Therefore, we propose to use the more powerful module – relational memory (Santoro et al., 2018) – as the generator architecture for text generation.
The basic idea of relational memory is to consider a ﬁxed set of memory slots (e.g. memory matrix) and allow for interactions between memory slots by using the self-attention mechanism (Vaswani et al., 2017).
The empirical ﬁndings by Santoro et al.
(2018) showed that relational memory performs better in the language modeling compared to
LSTM.
Intuitively, the use of multiple memory slots and the attention across these memories can increase the expressive power of generator and its ability of generating longer text sentences.
Formally, we assume each row of the memory M t represents a memory slot and Figure 1 shows how self-attention updates M t by incorporating new observation x t at time t .
Given H heads, we have
H sets of queries, keys and values via three linear transformations, respectively:
For each head, we get query Q ( h ) t
[Equation]
where [; ] denotes the row-wise concatenation.
Thus, the updated memory ˜
M t +1 is given by
[Equation]
where σ ( · ) denotes the softmax function which is performed on each row, d k is the column dimension of the key K ( h ) t and [:] denotes the column-wise concatenation.
By following the same idea of Santoro et al.
(2018), the next memory M t +1 and output (logits)
o t of the generator at time t are given by
[Equation]
respectively, where the two parametrized functions f θ 1 and f θ 2 are combinations of skip connections, multi-layer perceptron (MLP), gated operations and/or pre-softmax linear transformations. [Figure]
2. T RAINING WITH D ISCRETE D ATA 2.2. G UMBEL -S OFTMAX R ELAXATION
Before the introduction of Gumbel-Softmax relaxation, we ﬁrst show why training GANs with discrete data is a critical issue.
Assuming the vocabulary size is V , for the output logits o t ∈ R V of the generator in (2), the next generated one-hot token y t +1 ∈ R V will be obtained by sampling:
[Equation]
where similarly σ ( · ) denotes the softmax function which is performed on o t element-wisely and we use σ ( o t ) to represent the multinomial distribution on the set of all possible tokens.
As we know, the sampling operations in (3) on the multinomial distribution of the generator output are not differentiable, which implies a step function at the end of the generator.
Since the derivative of a step function is 0 almost everywhere, we have ∂y t
+1 ∂θ G = 0 a.e. for t = 0 , · · · , T − 1 where θ G denotes the generator parameters.
By chain rule, the gradients of the generator loss l G w.r.t.
θ G will be
[Equation]
So the gradients of the generator loss cannot pass back to the generator via the discriminator.
This is the notorious “non-differentiability issue” of GANs in discrete data generation.
To deal with the non-differentiablity issue, we apply the Gumbel-Softmax relaxation technique which deﬁnes a continuous distribution over the simplex that can approximate samples from a categorical distribution (Jang et al., 2016; Maddison et al., 2016).
Formally, the Gumbel-Softmax relaxation includes two parts: 1) The Gumbel-Max trick.
According to Jang et al.
; Maddison et al.
(2016), the sampling in (3) can be reparametrized as
[Equation]
where o ( i ) t denotes the i -th entry of o t and g ( i ) t is from the i.i.d.
standard Gumbel distribution, i.e. g ( i ) t
= − log( − log U ( i ) t ) with U ( i ) t ∼ Uniform (0 , 1) .
2) Relaxing the discreteness.
As the arg max operation in (5) is still non-differentiable, we need further approximate the “one-hot with arg max ” by softmax, which yields
[Equation]
where β > 0 is a tunable parameter called inverse temperature .
As ˆ y t +1 in (6) is differentiable with respect to o t , we can use ˆ y t +1 instead of the one-hot token y t +1 as the input of the discriminator.
Also, note that the new observation x t +1 of the generator to be concatenated with M t +1 in next time t + 1 is given by x t
+1 =
f θ 3 ( y t +1 ) , where the parametrized function f θ 3 is composed of an embedding layer that maps y t +1 to an embedded input and a linear layer that makes x t +1 match the row dimension of M t +1 (the embedded input and the linear layer are shown in Figure 1) . D D D
D loss
D loss
D loss ... T one-hot (or softmax) ... T : sharing weights 2.2. T EMPERATURE C ONTROL
With the larger inverse temperature β , ˆ y t +1 in (6) will become a better approximation of y t +1 in (3) and asymptotically as β → ∞ , ˆ y t +1 →
y t +1 .
However, the issue is that the variance of gradients will be very large as we have Var ( ∂ ˆ y t
+1 ∂o t )
∝ β 2 , and thus the parameter updates will become very sensitive to the input noise.
Intuitively, this might cause poor sample quality.
On the other hand, with the smaller inverse temperature β , the generator will pay more attention to making a sharp distribution of entries in ˆ y i +
due to the larger (initial) approximation gap between ˆ y t
+1 and y t
+1 , which implicitly discourages its possible “exploration”.
Intuitively, this might be one factor that contributes to mode collapse of RelGAN on text generation.
Therefore, the larger β encourages more exploration for better sample diversity while the smaller β encourages more exploitation for better sample quality.
We thus propose to increase the inverse temperature β over iterations via an exponential policy: β n = β n/N max , where β max denotes the maximum inverse temperature, N is the maximum number of training iterations and n denotes the current iteration.
In the exponential policy, as the increase rate of inverse temperature depends on β max , β max will decide the transition time from the exploitation phase to the exploration phase .
In such sense, RelGAN provides a ﬂexibility of either generating more diverse samples with a large β max or generating better quality samples with a small β max while most current GANs cannot provide.
2. M ULTIPLE R EPRESENTATIONS IN D ISCRIMINATOR
A commonly used discriminator for text generation is a CNN-based classiﬁer (Kim, 2014) that employs a convolutional layer with multiple ﬁlters of different sizes to capture relations of various word lengths and a max-pooling layer over the entire input sentence for each feature map (Yu et al., 2017; Zhang et al., 2017; Lin et al., 2017; Guo et al., 2017).
In this discriminator architecture, the input of the CNN-based classiﬁer is a sentence of length T represented by a single embedded matrix ˜
X ∈ R d × T where each column ˜ x t ∈ R
d is a d -dimensional embedded vector of each word.
In this work, we propose a new discriminator framework that applies multiple embedded representations for each sentence, with each representation independently passed through the above CNNbased classiﬁer to get an individual score.
The average of these individual scores will serve as the ﬁnal guiding information to update the generator.
Our hypothesis is that each embedded representation may capture a speciﬁc aspect of the input sentence and the discriminator which compares real and generated sentences from these different perspectives can provide more diverse and comprehensive guiding information for the generator updates.
This idea resembles the use of multiple discriminators to improve GANs on image generation (Durugkar et al., 2016), but the difference is that we only use multiple different representations of the input while still keeping a single or weight-sharing CNN-based classiﬁer, which presumably has much less computational cost.
Formally, we assume that r t denotes the t -th one-hot real token and ˆ y t from (6) denotes the tth softmax-like generated token, and Figure 2 shows the proposed discriminator framework with multiple embedded representations where either the real input [ r 1 : · · · : r T ] ∈ R V × T or the generated input [ˆ y 1 : · · · : ˆ y T ] ∈ R V × T will be mapped into S embedded representations by
S distinct embedding matrices { W ( s ) e } S s =1 with W ( s ) e ∈ R d × V .
Let ˜ X
( s ) r and ˜ X ( s )
y be the s -th embedded representation of the real and generated input, respectively.
Thus, we have
[Equation]
and the ﬁnal discriminator loss is given by
[Equation]
where the expectation is taken w.r.t.
both real sentence distribution P r and generated sentence distribution P θ , and the loss function f is determined by the speciﬁc GAN loss, such as vanilla
GAN (Goodfellow et al., 2014), f -GAN (Nowozin et al., 2016) and WGAN (Arjovsky et al., 2017).
Throughout this paper, the generator loss can be simply set to be l G = − l D .
2. T RAINING T ECHNIQUES
Choice of Loss Function.
Empirically, we ﬁrst compared three different standard GAN losses: standard GAN (the non-saturating version) (Goodfellow et al., 2014), hinge loss (Nowozin et al., 2016; Zhang et al., 2018) and Relativistic standard GAN (RSGAN)
(Jolicoeur-Martineau, 2018) on the synthetic data (shown in next section) and then chose the best one – RSGAN for the rest of all experiments.
Note that it does not mean RelGAN only works with the RSGAN loss and please see
Appendix C for training curves of RelGAN with different loss functions.
Formally, the function f in (8) for RSGAN is f ( a, b )
= log sigmoid ( a − b ) for a, b ∈ R , and thus (8) becomes
[Equation]
Intuitively, the loss function in (9) is to directly estimate the average probability that real sentences are more realistic than generated sentences in terms of different embedded representations.
Generator Pre-training.
Most current GANs for text generation need the pre-training for both generator and discriminator, such as SeqGAN (Yu et al., 2017), and some may further heavily rely on the exclusive pre-training techniques, such as TextGAN (Zhang et al., 2017),
LeakGAN (Guo et al., 2017) and MaskGAN (Fedus et al., 2018).
Instead, the proposed RelGAN only need to pre-train the generator simply via the standard MLE training for several epochs before starting the adversarial training.
Experimentally, we ﬁnd that a good initialization for generator provided by the
MLE pre-training is necessary for a good convergence behavior of adversarial training.
E XPERIMENTS
We test RelGAN on both synthetic and real data, where the synthetic data are 10,000 discrete sequences generated by an oracle-LSTM with ﬁxed parameters (Yu et al., 2017) and the real data include the COCO image captions (Chen et al., 2015) and EMNLP2017 WMT News, ﬁrst used by
Guo et al.
(2017) for text generation.
The experimental settings are given in Appendix B.
Evaluation Metrics.
How to properly evaluate generative models remains an open research question (Theis et al., 2015; Semeniuta et al., 2018).
The key issue plaging current evaluation metrics for
GANs is that they cannot measure sample quality and sample diversity simultaneously.
Therefore, we use two distinct metrics: for synthetic data, we use both negative log-likelihood (called NLL gen ) and its counterpart (called NLL oracle ), deﬁned as: [Equation]
[Equation]
where the generated sentence distribution P θ and the real sentence distribution P r are both known by evaluating the generator and oracle-LSTM, respectively.
Generally, NLL gen measures sample diversity while NLL oracle is more sensitive to sample quality (Theis et al., 2015; Arjovsky & Bottou, 2017).
For the real dataset, we also apply NLL gen to measure the sample diversity, similar to Lu et al.
However, since NLL oracle cannot be evaluated without an oracle, we instead apply the commonly-used BLEU scores (Papineni et al., 2002) to measure the sample quality and compare with the MLE baseline, along with other start-of-the-art GANs, including SeqGAN (Yu et al., 2017), RankGAN (Lin et al., 2017) and LeakGAN (Guo et al., 2017). Note that for BLEU score evaluation, we follow the strategy in (Yu et al., 2017; Zhu et al., 2018) by using the test data as the reference.
3. S YNTHETIC D ATA
We run the synthetic data experiments with sequence length 20 and 40, respectively.
The NLL oracle results of RelGAN and other models are shown in Table 1 where we set β max = 1 for length 20 and β max
= 2 for length 40.
Note that “MLE” in Table 1 denotes the baseline model where LSTMs are trained with the teacher-forcing algorithm to maximize the likelihood (same with Table 2 and 3).
We can see that RelGAN outperforms other models in both cases, and its lead in performance becomes larger with longer sequence length, demonstrating the log-distance dependency modeling ability of the proposed generator.
[Table]
We also evaluate the trade-off between sample quality and diversity as a function of the maximum inverse temperature β max and the results are shown in Figure 3.
As β max increases, NLL gen decreases, which implies better sample diversity, but NLL oracle increases, which implies worse sample quality.
Especially when β max ∈ { 10 , 100 } , the best NLL gen score of 4.4 for RelGAN is very close to the best
NLL gen score of 4.2 for MLE pre-training, implying that RelGAN with a sufﬁciently large inverse temperature suffers little mode collapse on synthetic data.
3. COCO I MAGE C APTIONS D ATASET
In order to test RelGAN on real-world data, we ﬁrst run experiments using the COCO Image Captions dataset.
By applying the same preprocessing as in LeakGAN (Guo et al., 2017), the dataset includes 4,682 unique words with the maximum sentence length 37.
Both the training and test data contain 10,000 text sentences.
The BLEU scores of RelGAN compared with previous models are shown in Table 2 where we set β max = 100 and 1000, respectively.
We can see that RelGAN is signiﬁcantly and consistently better than other models in terms of all the BLEU scores, which means its ability of generating high-quality sentences of COCO Image Captions.
Furthermore, the NLL gen scores of RelGAN and previous models are also shown in Table 2, where RelGAN also achieves the state-of-the-art results in terms of sample diversity.
For example, we do not see obvious mode collapse with β max =by looking at the generated samples (see Appendix D.1 for more details). [Table]
3. EMNLP2017 WMT N EWS D ATASET
The EMNLP2017 WMT News dataset consists of 5,255 unique words with the maximum sentence length 51 after applying the same preprocessing as in LeakGAN (Guo et al., 2017).
Similarly, both training contain 10,000 sentences
[Table]
The BLEU scores of RelGAN compared with previous models are shown in Table 3 where we set β max = 100 and 1000, respectively.
We can see that RelGAN also consistently outperforms previous models in terms of all the BLEU scores, demonstrating its ability of generating high-quality sentences on EMNLP2017 WMT News.
Moreover, the sample diversity metric NLL gen scores of
RelGAN and previous models are also shown in Table 3.
Similarly, RelGAN achieves the stateof-the-art results in terms of sample diversity.
Upon visually examining generated samples (See
Appendix D.2 for more details), we do not observe obvious mode collapse for β max ∈ { 100 , 1000 } . Finally
[Table] from Tables 2 and [Table] we can see that the sample quality and diversity trade-off with different values of maximum inverse tempereture β max also exists on the real data.
That is, RelGAN with β max = 100 achieves better sample quality while RelGAN with β max = 1000 achieves better sample diversity.
Depending on what the underlying applications of text generation via RelGAN are, we can adjust β max properly to get either better quality or better diversity.
To further evaluate the sample quality of RelGAN and othe models on EMNNLP2017 WMT News, we also perform the human evaluation by using Amazon Mechnical Turk.
We randomly sampled 100 sentences for each model and the real dataset, and asked 10 different people to score each sentence on a scale of 1-5.
Please see Appendix B.2 for more details of human evaluation.
[Table]
The human score results are provided in Table 4, where we can see that RelAGN generates better human-looking samples than other GANs and the MLE baseline model [Table]
3. A BLATION S TUDY 3.4. I MPACT OF R ELATIONAL M EMORY
To show the impact of relational memory in RelGAN, we propose to replace relational memory by
LSTM-32 and LSTM-512 as the generator architecture, respectively, and see how the performance differs.
Here LSTM- k represents the LSTM with hidden dimension being k .
We choose k =because most previous GANs (Yu et al., 2017; Guo et al., 2017) have used this architecture for text generation, and also choose k = 512 because for more fair comparison, we want to keep the total memory size of LSTM to be the same with the relational memory we have used.
The results on the COCO Image Captions dataset are shown in Figure 4 (Left), where we provide the
BLEU-4 score (See Appendix E.1 for all the BLEU scores).
We can see that the BLEU scores of relational memory are consistently better than those of LSTM-32 and LSTM-512, which demonstrates the advantages of using relational memory as generator in RelGAN.
3.4. I MPACT OF G UMBEL -S OFTMAX R ELAXATION
To show the impact of Gumbel-Softmax relaxation in RelGAN, we can instead apply the vanilla
REINFORCE method to deal with the non-differentiable issue of RelGAN on text generation.
In this experiment, we keep all other hyperparamters in RelGAN ﬁxed and compare the performance of Gumbel-Softmax relaxation and the vanilla REINFORCE method.
The results are shown in Figure 4 (Right), where the BLEU-2 score is provided (See Appendix E. for all the BLEU scores).
We can see that under the proposed RelGAN framework, Gumbel-Softmax relaxation performs much better than the vanilla REINFORCE method.
During experiments, we ﬁnd that the variance of generator gradients in the vanilla REINFORCE method is too large to provide any useful update for generator, which may explain why the performance of vanilla REINFORCE does not improve after the pre-training, as observed in Figure 4 (Right).
The exploration of various variance reduction techniques for the REINFORCE method in RelGAN is out of scope of this paper.
3.4. I MPACT OF M ULTIPLE R EPRESENTATIONS IN D ISCRIMINATOR
To show the impact of multiple embedded representations in discriminator while keeping the expressive power of discriminator the same for fair comparison, we propose to apply S embedded presentations with each embedded vector of length
d = d max S where d max denotes the total length of representations.
In this experiment, we set d max = 64 , and thus for instance, if S = 1 then d =for each embedded vector, and if S = 2 then d = 32 for each embedded vector, and so on.
We ﬁrst test RelGAN on the synthetic data with S ∈ { 1 , 2 , 4 , 8 , 16 , 32 , 64 } and the results are shown in Figure 5 (Left).
We can see that as the number of embedded representations S increases, the best
NLL oracle score tends to keep decreasing, yielding better sample quality.
Furthermore, we test RelGAN on COCO Image Captions with S ∈ { 1 , 64 } and the BLEU-3 score is shown in Figure(Right).
Still, we can see that the BLEU scores of RelGAN with S = 64 are consistently better than those of RelGAN with S = 1 (see Appendix E.3 for all the BLEU scores).
Note that in both experi- ments, we do not see an obvious sign of mode collapse with varying number of representations.
For example, the NLL gen score on the synthetic data stays around 4 .
4 for different values of S (close to the best NLL gen 4.2 for MLE shown in Figure 3 (Left)).
Thus, these experiments demonstrate the advantages of using multiple embedded representations for discriminator in RelGAN.
C ONCLUSIONS
We proposed a new GAN architecture called RelGAN for text generation, that outperforms most current models in terms of sample quality and diversity on both synthetic and real data.
Furthermore, the trade-off between the generated sample diversity and quality can be adjusted properly in RelGAN by controlling the inverse temperature.
In RelGAN, we used the relational memory based generator to improve its ability of modeling long distance dependencies and also applied multiple embedded representations in discriminator such that it can provide more diverse and informative guiding signal for generator.
By applying Gumbel-Softmax relaxation to deal with the non-differentiable issue, our architecture is simple to implement without employing intensive RL heuristics.
For the future directions, since we have demonstrated that GANs with Gumbel-Softmax relaxation is very promising for text generation, we would like to explore further in this direction.
For example, it is interesting to make RelGAN work better without any pre-training.
Also, extending RelGAN to a conditional model for many text generation related applications is another interesting direction.
A R ELATED W ORK
Since GANs are originally proposed for continuous data, extending GAN training to discrete data generation has been an active research topic.
Current works focus on dealing with the nondiferentiable issue brought by the discrete data nature either by considering the RL methods or by reformulating the problem in continuous space.
A large class of GANs for text generation relies on the RL algorithm.
SeqGAN ((Yu et al., 2017)) models the text generation as a sequential decision making process and trains the generator with policy gradient methods (Sutton et al., 2000).
MaliGAN (Che et al., 2017) proposes the co-training with a maximum-likelihood objective to reduce the gradient variance.
RankGAN (Lin et al., 2017) proposes a ranking model to replace the original binary classiﬁer as the discriminator.
LeakGAN (Guo et al., 2017) designs a mechanism to provide intermediate information about text generation for generator, where the discriminator can leak its features through a manager module.
MaskGAN (Fedus et al., 2018) introduces an actor-critic conditional GAN that ﬁlls in missing text conditioned on the surrounding context by resorting to a seq2seq model.
Other GANs without RL methods either approximate the discrete data or work in the continuous latent space.
TextGAN (Zhang et al., 2017) provides a feature matching mechanism that matches the latent features of real and generated sentences via a kernelized discrepancy metric to alleviate the mode collapse.
FM-GAN (Chen et al., 2018) proposes to match the latent feature distributions of real and synthetic sentences using the feature-movers distance.
Similar to our work, both textGAN and FM-GAN apply an annealed softmax to approximate the argmax in the generator.
However, they do not rely on the Gumbel-Max trick to reparametrize the sampling operations, which is the major difference with us in dealing with the non-differentiable issue.
ARAE (Zhao et al., 2018) applies an additional autoencoder to embed the discrete data into a continuous latent space in which
GANs can be trained properly.
As for approximating the categorical distribution with GumbelSoftmax relaxation, Gu et al.
(2017) has used it to improve the generation quality in neural machine translation.
More similarly, Kusner & Hern´andez-Lobato (2016) provides some initial experiments of training GANs with Gumbel-Softmax relaxation on a synthetic task, but scaling them to work on real text dataset remains a challenging open problem.
Attention mechanisms, in particular self-attention (Vaswani et al., 2017), have gradually become a building block of many novel neural network architectures (Vaswani et al., 2017; Parmar et al., 2018;
Santoro et al., 2018) due to its ability of capturing long or global dependencies and reducing computational cost via parallelization.
In the context of GANs, self-attention have not been fully explored.
SAGAN (Zhang et al., 2018) applies self-attention in GANs to model long range dependencies in images and get the state-of-the-art results on conditional image generation.
In contrast, we employ self-attention in GANs for text generation by using relational memory as generator.
B E XPERIMENTAL S ETTINGS B. T RAINING D ETAILS
Unless stated otherwise, for the CNN-based discriminator architecture, we use ﬁlter windows of sizes { 3,4,5 } and 300 feature maps each.
For relational memory, we set memory size to be 256, memory slots to be 1, number of heads to be 2.
The batch size is set to be 64.
For embedding dimensions, we set the embedding dimension of the input token for generator to be 32 and that for discriminator to be 1 with the number of embedded representations S = 64 .
We use Adam (Kingma & Ba, 2014) with β 1 = 0 .
9 and β 2 = 0 .
999 and gradient clipping is applied if the norm of gradients exceeds 5.
We ﬁrst pre-train the generator via MLE with learning rate of 1e-2 for 150 epochs and then start adversarial training with learning rate of 1e-4 for both discriminator and generator.
For adversarial training, we set the maximum number of iterations N =
and we perform 5 gradient descent steps on the discriminator for every step on the generator.
B. H UMAN E VALUATION D ETAILS
On Amazon Mechanical Turk, our instructions are given as follows:
The text quality evaluation is based on grammatical correctness and meaningfulness (i.e. if a sentence makes sense).
Please ignore any text formatting problems (e.g., capitalization, punctuation, spelling errors, extra spaces between words and punctuations).
Note: A very short sentence (less than 10 words) should be penalized with its score minus 1.
Please see below for the detailed criteria.
[Table]
5 - Excellent Its grammatically correct and makes sense.
For example, “ if England wins the World Cup next year , it will be the most signiﬁcant result the sport has seen in more than a decade . ”
4 - Good It has some small grammatical errors and mostly makes sense.
For example, “ it is useful to have had a doctor who forced her to release him a couple of days before she was cleared . ”
3 - Fair It has major grammatical errors but the whole still conveys
[Table]
C R EL GAN WITH V ARYING H YPERPARAMETERS
[Figure]
D G ENERATED S AMPLES ON R EAL D ATASET D. G ENERATED S AMPLES ON COCO I MAGE C APTIONS D ATASET
a man is sitting on a bench next to a bicycle .
a man ﬁxing a motor cycle in a race .
a rectangle shaped wooden sitting on a lush green ﬁeld .
a man standing in a picture of a kitchen with a dog watching him .
a man is carving some meat in a park .
a kitchen with a black window and a large white stove wall .
a cat is looking on a man in a bathroom .
a train is covered in the air in the city scene .
a bathroom has a toilet , and bathroom rug for a urinal or a urinal on the side .
a home kitchen with a double oven and table while a table and chairs .
a small airplane ﬂying above an airport covered with wood .
a man in a kitchen preparing food on a table .
a large passenger jet ﬂying in a clear blue sky .
a group of people riding motorcycles on a street .
a woman in a kitchen with her hands clasped .
a smiling woman is sitting on a green bench .
people are hiding under colorful umbrellas on a rainy day .
a photo of a small restroom in a kitchen .
many sheep graze are shown in front of a group of people .
a plane is parked next to an airplane on a runway near a control tower with two back .
a woman walking past a straw shower holding a corner .
an ofﬁce desk with a row of books in the kitchen .
a cat sitting on top of a kitchen counter .
a person riding a bike through a lush green park .
a bathroom with a sink , toilet and toilet paper dispenser .
a man is on a motorcycle with a woman on the back of it .
two giraffes in a wild , lightly wooded ﬁeld .
a metal tin pan ﬁlled with two different kitchen appliances .
a white airplane ﬂying in the sky over a runway .
a car driving on a busy street at night from an airport .
D. G ENERATED S AMPLES ON EMNLP2017 WMT N EWS D ATASET
he is also watching closely on staying in the plan , but sometimes i still don ’ t think that ’ s going to happen on saturday .
and so , they didn ’ t want to be in their position , and that was something they would do to my wife .
i would like to assess whether you should be able to take that into that issue than you did on a sunday .
“ he ’ s a young lady , so i don ’ t want to show up for them , ” she said .
that ’ s not yet about what you want to say about reality , but probably don ’ t think about that would just make me comfortable in my life .
ofﬁcials have been also a member of nato against the us , and for the ﬁrst time that so many other countries are on the rise , and that will be the only way to stay here .
he has always vowed to give him a little more on for him , and i think he is very willing for the division .
we ’ ve had to try and get into the coming down to the today ’ s end .
i don ’ t think that ’ s why we did not have to score the last two .
meanwhile , it was never been in the past , but it was not known until the coalition was given the support of the rebels on the terms of the claims .
he said : “ i don ’ t think we should be better at what we would do to the good .
“ we had to get on with that , ” she said at a news conference .
by contrast , but this is a turning point at her age .
but i will do that , which i have to do with the city , and my new hopes is from all over the world .
“ this is a very bad , and it ’ s not clear that this is a danger of this , ” he said .
at a detroit press with reporters on the ﬂight , gave informed operators time to interview mr . cox that he was struggling with all of his treatments .
“ in the past ﬁve days , we ’ re going to enjoy maybe that and after that , you ’ re going to stay to dinner with friends and family , ” he said .
the union has indicated that there are no restrictions on the long - standing alliance in any key areas .
i never thought i could put over the line but
i couldn ’ t quite lose my job .
since then , while the number of people stood by the wall street banks fell by 2 percent over the past 10 years , there ’ s no need to say that .
“ they had to ﬂy in the ﬁeld , ” he said , adding that it didn ’ t miss it .
“ trump ’ s voice will be a positive one for mr . trump ’ s transition team , ” he said .
he was still working on a training camp friday with a small into a new manager .
the 15 - year - old man has been reported missing by a falling from the city centre .
“ it ’ s a process that can take a little while , ” one resident said .
a well - meaning - predicted or very public policy , seeking to work with us .
“ i ’ m not to have made me a bit more than anything , but i ’ ve never done that , ” he said .
they were waiting to see how many changes could come from us and that ’ s why it has made it leave facing .
she initially noted that some of the other victims began to come from being more than prepared to stand for .
“ i ’ ve never heard of the abuse , because we need to work with him , ” he added
E M ORE RESULTS ON A BLATION S TUDY E. I MPACT OF R ELATIONAL M EMORY
E. I MPACT OF G UMBEL -S OFTMAX R ELAXATION
F D IVERSITY -Q UALITY T RANSITION D URING T RAINING
As we can observe from Figures 7-9 in the above appendices, the BLEU scores of RelGAN (denoted by the blue curves in each subﬁgure) ﬁrst increase over iterations and then keep decreasing after around 800 iterations.
In other words, its sample quality ﬁrst increases and then decreases during the adversarial training.
To see what happens in the training dynamics of RelGAN, we also provide the training curve of the diversity metric – NLL gen in RelGAN as shown in Figure 10.
Interestingly, Figure 10 shows that during the adversarial training, the sample diversity (measured by
NLL gen ) of RelGAN ﬁrst decreases and then increases, and its turning point matches well with that of the sample quality (measured by BLEU scores) of RelGAN.
These training dynamics illustrate a diversity-quality transition over iterations in RelGAN: Early on in training, it learns to aggressively improve sample quality while sacriﬁcing diversity.
Later on, it turns instead to maximizing sample diversity while gradually decreasing sample quality.
Intuitively, it seems to be much easier for the generator to just produce realistic samples – regardless of their diversity – to fool the discriminator in the early stage of training.
As the discriminator becomes better at distinguishing samples with less diversity over iterations, the generator has to focus more on producing more diverse samples to fool the discriminator.
G R EL GAN WITHOUT P RE - TRAINING
In this section
[Table]
we want to test the performance of RelGAN without pre-training for different loss functions, including standard GAN (the non-satuarating version) (Goodfellow et al., 2014), WGANGP (Gulrajani et al., 2017), hinge loss (Nowozin et al., 2016; Zhang et al., 2018) and RSGAN (Jolicoeur-Martineau, 2018).
The BLEU and NLL gen scores evaluated on COCO Image Captions are shown in Table
[Table]
We can see that without pre-training, there is still a signiﬁcant improvement for RelGAN compared with the random generation, in particular for the standard GAN loss, even though the improvement is inferior to the case with pre-training.
In contrast, without pre-training, previous RL-based GANs for text generation, such as SeqGAN and RankGAN, always get stuck around their initialization points and are not able to improve their performance at all.
This demonstrates that RelGAN may be a more promising GAN architecture to explore in order to completely get rid of the pre-training for GANs on text generation.
Besides, Table 8 also shows that the evaluation results of RelGAN without pretraining vary with different loss functions and values of β max .
We leave an extensive hyperparameter search to further improve the performance of RelGAN without pre-training for future work.
H M ORE E XPLORATION ON T UNABLE H YPERPARAMETER β max
For real data experiments, we have showed the advantages of RelGAN over other models by setting the maximum inverse temperature β max ∈ { 100 , 1000 } , which are carefully chosen for a good tradeoff between sample quality and diversity.
A natural question will be to explore the two extremes: what happens with the real data if β max is either too large or too small?
Does it behave similarly to the synthetic data experiments in terms of the trade-off between sample diversity and quality?
To this end, we choose a broad range of β max ∈ { 1 , 10 , 10 2 , 10 3 , 10 4 , 10 5 , 10 6 , 10 7 } and test its impact in RelGAN on the COCO Image Captions dataset.
The BLEU and NLL gen scores are given in Table 9, where we can see that both BLEU and NLL gen scores increase with the decrease of β max , and the variance of each score also consistently becomes larger for a smaller β max .
For better illustration, we also plot BLEU-4 and NLL gen scores with error bars in Figure 11.
It ﬁrst conﬁrms that similar to the synthetic data experiments, there also exists a consistent trade-off between sample quality and diversity in real data, controlled by the tunable hyperparameter β max .
Besides, it reveals the failing cases at the two extremes: On the one hand, if β max is too small, i.e. β max = 1 , RelGAN suffers from severe mode collapse (denoted by the large NLL gen ) and training instability (denoted by high variances of scores) issues.
On the other hand, if β max is too large, i.e. β max = 10 7 , the sample quality improvement of RelGAN becomes marginal (denoted by the low
BLEU scores).
Therefore, we have chosen the two intermediate values { 100 , 1000 } of β max in the
[Table]
gen scores increase with the decrease of β max .
Besides, the variance of each score also consistently becomes larger for a smaller β max .
main text to show the advantages of RelGAN over other models while still demonstrating its ability to control the trade-off between sample quality and diversity.