Original: A Q UALITY -D IVERSITY C ONTROLLABLE GAN FOR T EXT G ENERATION
Revised: A Q UALITY -D IVERSITY C ONTROLLABLE GAN FOR T EXT G ENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Text generation is a critical and difﬁcult natural language processing task.
Revised: Text generation is a critical and difﬁcult natural language processing task.

Original: Maximum likelihood estimate (MLE) based models have been arguably suffered from exposure bias in the inference stage and thus varieties of language generative adversarial networks (GANs) bypassing this problem have emerged.
Revised: Maximum likelihood estimate (MLE) based models have been arguably suffered from exposure bias in the inference stage and thus varieties of language generative adversarial networks (GANs) bypassing this problem have emerged.

Original: However, recent study has demonstrated that MLE models can constantly outperform GANs models over quality-diversity space under several metrics.
Revised: However, recent study has demonstrated that MLE models can constantly outperform GANs models over quality-diversity space under several metrics.

Original: In this paper, we propose a quality-diversity controllable language GAN.
Revised: In this paper, we propose a quality-diversity controllable language GAN.

Original: We illustrate an intimate connection between forward/reverse KL divergences and our principled approach, and demonstrate that our proposed model can achieve quality-diversity control by adjusting a hyper-parameter.
Revised: We illustrate an intimate connection between forward/reverse KL divergences and our principled approach, and demonstrate that our proposed model can achieve quality-diversity control by adjusting a hyper-parameter.

Original: Through extensive experiments, we show that the proposed model outperforms MLE models and other GANs variants on synthetic data and is more competitive on real-world tasks such as COCO Image Captions and EMNLP2017 WMT News datasets
Revised: Through extensive experiments, we show that the proposed model outperforms MLE models and other GANs variants on synthetic data and is more competitive on real-world tasks such as COCO Image Captions and EMNLP2017 WMT News datasets

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Text generation is one of the most challenging and widely used natural language processing task and it serve as a critical module on numerous practical applications.
Revised: Text generation is one of the most challenging and widely used natural language processing task and it serve as a critical module on numerous practical applications.

Original: In recent years, neural autoregressive models have shown superiority on various natural language generation (NLG) tasks (Zhang &
Revised: In recent years, neural autoregressive models have shown superiority on various natural language generation (NLG) tasks (Zhang &

Original: Lapata, 2014; Gal & Ghahramani, 2016; Du et al., 2017; Fiorini & Lu, 2018).
Revised: Lapata, 2014; Gal & Ghahramani, 2016; Du et al., 2017; Fiorini & Lu, 2018).

Original: The most common method of training these models is the maximum likelihood estimation (MLE) which maximize the log predictive likelihood of each true token in the training sequence given the previous observed tokens.
Revised: The most common method of training these models is the maximum likelihood estimation (MLE) which maximize the log predictive likelihood of each true token in the training sequence given the previous observed tokens.

Original: However, MLE models have been arguably suffered from exposure bias (Bengio et al., 2015;
Revised: However, MLE models have been arguably suffered from exposure bias (Bengio et al., 2015;

Original: Ranzato et al., 2016) which means a discrepancy between training and inference stages due to the fact that at inference time, the model predicts next token based on the sequence generated by itself which may not occur in training set.
Revised: Ranzato et al., 2016) which means a discrepancy between training and inference stages due to the fact that at inference time, the model predicts next token based on the sequence generated by itself which may not occur in training set.

Original: In order to overcome this problem, scheduled sampling (Bengio et al., 2015) and professor forcing (Lamb et al., 2016) are proposed, but scheduled sampling is proven to be fundamentally inconsistent (Husz ´ ar, 2015) and professor forcing use an adversarial method similar to GANs (Goodfellow et al., 2014).
Revised: In order to overcome this problem, scheduled sampling (Bengio et al., 2015) and professor forcing (Lamb et al., 2016) are proposed, but scheduled sampling is proven to be fundamentally inconsistent (Husz ´ ar, 2015) and professor forcing use an adversarial method similar to GANs (Goodfellow et al., 2014).

Original: Varieties of generative adversarial networks (GANs) (Goodfellow et al., 2014) models for text generation have emerged as alternatives to MLE models with the hope of not suffering from exposure bias.
Revised: Varieties of generative adversarial networks (GANs) (Goodfellow et al., 2014) models for text generation have emerged as alternatives to MLE models with the hope of not suffering from exposure bias.

Original: However, due to the discrete nature, GANs have difﬁculty in text generation since the gradients from discriminator can not be passed to generator explicitly and thus most these GANs based models (Yu et al., 2017; Guo et al., 2018; Lin et al., 2017; Fedus et al., 2018; Xu et al., 2018) treat text generation as a sequential decision making process (Bachman & Precup, 2015) and utilize policy gradient (Williams, 1992) to overcome this difﬁculty.
Revised: However, due to the discrete nature, GANs have difﬁculty in text generation since the gradients from discriminator can not be passed to generator explicitly and thus most these GANs based models (Yu et al., 2017; Guo et al., 2018; Lin et al., 2017; Fedus et al., 2018; Xu et al., 2018) treat text generation as a sequential decision making process (Bachman & Precup, 2015) and utilize policy gradient (Williams, 1992) to overcome this difﬁculty.

Original: Despite the impressive achievements, they inevitably have a problem of high variance due to the use of reinforcement learning (RL) strategies.
Revised: Despite the impressive achievements, they inevitably have a problem of high variance due to the use of reinforcement learning (RL) strategies.

Original: Caccia et al.
Revised: Caccia et al.

Original: (2018) have made several surprising observations and argued that, through simple temperature sweeping at inference stage, MLE models can constantly outperform these RL-based
Revised: (2018) have made several surprising observations and argued that, through simple temperature sweeping at inference stage, MLE models can constantly outperform these RL-based

Original: GANs models by a large margin over quality-diversity space under several metrics.
Revised: GANs models by a large margin over quality-diversity space under several metrics.

Original: They attribute this to the fact that the negative impact coming with RL is severer than so-called exposure bias caused by MLE.
Revised: They attribute this to the fact that the negative impact coming with RL is severer than so-called exposure bias caused by MLE.

Original: A RL-free approach to train GANs for text generation without resorting to an explicit neural network as the discriminator, named No Neural Network as the Discriminator GAN (N3DGAN), is proposed by (Li et al., 2019) and shows state-of-the-art results on both synthetic and real datasets under quality-only metrics.
Revised: A RL-free approach to train GANs for text generation without resorting to an explicit neural network as the discriminator, named No Neural Network as the Discriminator GAN (N3DGAN), is proposed by (Li et al., 2019) and shows state-of-the-art results on both synthetic and real datasets under quality-only metrics.

Original: We propose a quality-diversity controllable GAN through a principle approach which is a generalization of N3DGAN (Li et al., 2019).
Revised: We propose a quality-diversity controllable GAN through a principle approach which is a generalization of N3DGAN (Li et al., 2019).

Original: This generalization has intimate connections with direct optimization of both forward and inverse KullbackLeibler (KL) divergences with respect to empirical distribution and model distribution.
Revised: This generalization has intimate connections with direct optimization of both forward and inverse KullbackLeibler (KL) divergences with respect to empirical distribution and model distribution.

Original: As shown in (Husz´ar, 2015), minimizing forward KL divergence often leads to models that overgeneralize, and sometimes produce samples that are very unlikely under true distribution and thus it is good at diversity.
Revised: As shown in (Husz´ar, 2015), minimizing forward KL divergence often leads to models that overgeneralize, and sometimes produce samples that are very unlikely under true distribution and thus it is good at diversity.

Original: Minimizing the reverse KL divergence instead will try to avoid any behaviour that is unlikely under true distribution at the cost of ignoring modes of true distribution completely, thus it is good at quality.
Revised: Minimizing the reverse KL divergence instead will try to avoid any behaviour that is unlikely under true distribution at the cost of ignoring modes of true distribution completely, thus it is good at quality.

Original: Through this principled approach, we demonstrate that our proposed model can control the quality-diversity trade-off, that is, the degree to which the model depends on forward and reverse KL divergences respectively through a single hyper-parameter.
Revised: Through this principled approach, we demonstrate that our proposed model can control the quality-diversity trade-off, that is, the degree to which the model depends on forward and reverse KL divergences respectively through a single hyper-parameter.

Original: The key contributions of our work are three-folds: (1) We propose a simple but effective GANs model for text generation which can control the quality-diversity trade-off through a single hyperparameter at training stage.
Revised: The key contributions of our work are three-folds: (1) We propose a simple but effective GANs model for text generation which can control the quality-diversity trade-off through a single hyperparameter at training stage.

Original: (2) Different from most existing GANs variants, our proposed model, as a generalization of N3DGAN, does not need to rely on MLE pre-training and is RL-free.
Revised: (2) Different from most existing GANs variants, our proposed model, as a generalization of N3DGAN, does not need to rely on MLE pre-training and is RL-free.

Original: (3) Through extensive experiments, we show that our proposed model outperform MLE based model and other
Revised: (3) Through extensive experiments, we show that our proposed model outperform MLE based model and other

Original: GANs variants over quality-diversity space in the synthetic-data tasks and are more competitive to
Revised: GANs variants over quality-diversity space in the synthetic-data tasks and are more competitive to

Original: MLE model than other GANs variants in the real-world tasks.
Revised: MLE model than other GANs variants in the real-world tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Husz ´ ar (2015) analyzes the difference between optimizing forward and reverse KL divergences, and suggests to use a generalized Jensen-Shannon divergence to replace MLE, but he doesn’t realize its relation to the mini-max game’s value function as well as tractability of directly optimizing the generalized Jensen-Shannon divergence on training data, and moreover he doesn’t conduct any experiments.
Revised: Husz ´ ar (2015) analyzes the difference between optimizing forward and reverse KL divergences, and suggests to use a generalized Jensen-Shannon divergence to replace MLE, but he doesn’t realize its relation to the mini-max game’s value function as well as tractability of directly optimizing the generalized Jensen-Shannon divergence on training data, and moreover he doesn’t conduct any experiments.

Original: Caccia et al.
Revised: Caccia et al.

Original: (2018) try to navigate the quality-diversity space through temperature sweep, beam search and generator rejection sampling (Ott et al., 2018) methods, unfortunately all these methods can only navigate the quality-diversity space at inference stage.
Revised: (2018) try to navigate the quality-diversity space through temperature sweep, beam search and generator rejection sampling (Ott et al., 2018) methods, unfortunately all these methods can only navigate the quality-diversity space at inference stage.

Original: Li et al.
Revised: Li et al.

Original: (2019) propose N3DGAN which is RL-free and doesn’t need additional maximum likelihood pre-training, but they only report results on quality metrics but not on diversity.
Revised: (2019) propose N3DGAN which is RL-free and doesn’t need additional maximum likelihood pre-training, but they only report results on quality metrics but not on diversity.

Original: Lu et al.
Revised: Lu et al.

Original: (2019) propose a cooperative training (CoT) approach for text generation which dont have to rely on RL and MLE for pre-training, instead they optimize an approximate Jensen-Shannon divergence.
Revised: (2019) propose a cooperative training (CoT) approach for text generation which dont have to rely on RL and MLE for pre-training, instead they optimize an approximate Jensen-Shannon divergence.

Original: It is done by introducing a mediator and dropping the term in the gradient of generator which behaves like RL.
Revised: It is done by introducing a mediator and dropping the term in the gradient of generator which behaves like RL.

Original: In the synthetic-data experiments, they perform similarly but slightly worse than MLE model (Caccia et al., 2018).
Revised: In the synthetic-data experiments, they perform similarly but slightly worse than MLE model (Caccia et al., 2018).

Original: de Masson d’Autume et al.
Revised: de Masson d’Autume et al.

Original: (2019) combine existing techniques such as large batch size, dense rewards and discriminator regularization to train language GANs from scratch without the need for
Revised: (2019) combine existing techniques such as large batch size, dense rewards and discriminator regularization to train language GANs from scratch without the need for

Original: MLE pre-training.
Revised: MLE pre-training.

Original: But they still have to use RL which is essentially the same as most language
Revised: But they still have to use RL which is essentially the same as most language

Original: GANs (Yu et al., 2017; Guo et al., 2018; Lin et al., 2017; Fedus et al., 2018; Xu et al., 2018), and they don’t conduct any experiment on synthetic data which has oracle metrics to validate its effectiveness.
Revised: GANs (Yu et al., 2017; Guo et al., 2018; Lin et al., 2017; Fedus et al., 2018; Xu et al., 2018), and they don’t conduct any experiment on synthetic data which has oracle metrics to validate its effectiveness.

Original:  A Q UALITY -D IVERSITY C ONTROLLABLE GAN
Revised:  A Q UALITY -D IVERSITY C ONTROLLABLE GAN

Original: In the well-known adversarial modeling framework Goodfellow et al.
Revised: In the well-known adversarial modeling framework Goodfellow et al.

Original: (2014), there are two players, one is called generator represented by a differentiable function G and another one is called discriminator represented by a differentiable function D .
Revised: (2014), there are two players, one is called generator represented by a differentiable function G and another one is called discriminator represented by a differentiable function D .

Original: They approximate the data distribution p data through an alternative mini-max game between these two components.
Revised: They approximate the data distribution p data through an alternative mini-max game between these two components.

Original: The discriminator D is trained to discriminate the samples drawn from the generator distribution p G from the real data, while the generator is trained to fool the discriminator so that it cannot distinguish between the samples from p G and p data .
Revised: The discriminator D is trained to discriminate the samples drawn from the generator distribution p G from the real data, while the generator is trained to fool the discriminator so that it cannot distinguish between the samples from p G and p data .

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2014) proved that if the generator G has enough capacity to cover data distribution p data , then optimizing the mini-max game is equivalent to directly optimizing the Jensen-Shannon divergence.
Revised: (2014) proved that if the generator G has enough capacity to cover data distribution p data , then optimizing the mini-max game is equivalent to directly optimizing the Jensen-Shannon divergence.

Original: Unfortunately Goodfellow et al.
Revised: Unfortunately Goodfellow et al.

Original: (2014) didn’t distinguish the real data as either population ( true but unknown ) distribution or empirical distribution.
Revised: (2014) didn’t distinguish the real data as either population ( true but unknown ) distribution or empirical distribution.

Original: Li et al.
Revised: Li et al.

Original: (2019) notice the distinction and they explicitly use the empirical distribution in their approach.
Revised: (2019) notice the distinction and they explicitly use the empirical distribution in their approach.

Original: They argue that when an alternative mini-max optimization procedure is performed for the value function, a closed form solution for the discriminator exists in the maximization step, thus an explicit neural network model for the discriminator is not needed.
Revised: They argue that when an alternative mini-max optimization procedure is performed for the value function, a closed form solution for the discriminator exists in the maximization step, thus an explicit neural network model for the discriminator is not needed.

Original: Moreover optimizing the mini-max game is equivalent to directly optimizing the Jensen-Shannon divergence (JSD) between the generator’s distribution and the empirical distribution over the training data, where the corresponding KL divergence related to the generator’s distribution and the empirical distribution over the training data is reduced to the sum over the training data.
Revised: Moreover optimizing the mini-max game is equivalent to directly optimizing the Jensen-Shannon divergence (JSD) between the generator’s distribution and the empirical distribution over the training data, where the corresponding KL divergence related to the generator’s distribution and the empirical distribution over the training data is reduced to the sum over the training data.

Original: This means that sampling from the generator over the entire space as well as the policy-gradient algorithm to compute the gradient of the generator can be eliminated.
Revised: This means that sampling from the generator over the entire space as well as the policy-gradient algorithm to compute the gradient of the generator can be eliminated.

Original: Optimizing the JSD becomes computationally tractable to train the generator for text generation.
Revised: Optimizing the JSD becomes computationally tractable to train the generator for text generation.

Original: We now propose a quality-diversity controllable GAN which is a generalization of N3DGAN (Li et al., 2019) and illustrate its connections to the forward KL divergence and the reverse KL divergence as stated below.
Revised: We now propose a quality-diversity controllable GAN which is a generalization of N3DGAN (Li et al., 2019) and illustrate its connections to the forward KL divergence and the reverse KL divergence as stated below.

Original: Since the samples in the training data are limited, we use π ∈ [0 , 1] to denote the ratio of labeled samples discriminator receives from the generator and the training data, and let the discriminator D and the generator G play the following two-player minimax game with a generalized distinguishability game value function V π ( G, D ) : 
Revised: Since the samples in the training data are limited, we use π ∈ [0 , 1] to denote the ratio of labeled samples discriminator receives from the generator and the training data, and let the discriminator D and the generator G play the following two-player minimax game with a generalized distinguishability game value function V π ( G, D ) : [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ˜ p data ( x ) denotes the empirical distribution over training set C
Revised: where ˜ p data ( x ) denotes the empirical distribution over training set C

Original: = { x 1 , · · · , x N } with N samples, and
Revised: = { x 1 , · · · , x N } with N samples, and

Original: [Equation]
Revised: [Equation]

Original: if x ∈ C otherwise 
Revised: otherwise We will then show that optimize (1) is equivalent to optimize generalized Jensen-Shannon divergence (Husz ´ ar, 2015) between ˜ p data and p G , which is formulated as follows [Equation] and we can control the quality-diversity trade-off through a single parameter π .

Original: Proposition 1.
Revised: Proposition 1.

Original: When optimizing the value function V π ( G, D ) , for any given generator G , the optimal discriminator D ∗ G ( x ) is
Revised: When optimizing the value function V π ( G, D ) , for any given generator G , the optimal discriminator D ∗ G ( x ) is

Original: [Equation]
Revised: [Equation]

Original: if x ∈ C[Equation] otherwise and the value function at the optimal discriminator D ∗ G ( x ) becomes
Revised: if x ∈ C otherwise and the value function at the optimal discriminator D ∗ G ( x ) becomes

Original: [Equation]
Revised: [Equation]

Original: where JSD π is the generalized Jensen-Shannon divergence (Husz´ar, 2015) between two distributions ˜ p data and p G , and is not symmetric.

Original: Proof: Given any generator G ( x )

Original: [Equation]
Revised: [Equation]

Original:  the training criterion for the discriminator D ( x ) is to maximize the value function V π ( G, D ) in (1) and could be re-written as.
Revised: Proof: Given any generator G ( x ) , the training criterion for the discriminator D ( x ) is to maximize the value function V π ( G, D ) in (1) and could be re-written as.

Original: [Equation]
Revised: [Equation]

Original: Taking derivative with respect to D ( x ) 
Revised: Taking derivative with respect to D ( x ) , we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: otherwise

Original: [Equation]
Revised: [Equation]

Original: )) otherwise (4)

Original: Thus, the optimal solution for D ( x ) is
Revised: Thus [Equation] the optimal solution for D ( x ) is

Original: [Equation]
Revised: [Equation]

Original: Plug (5) into (3), we have
Revised: Plug (5) into (3), we have

Original: [Equation]
Revised: [Equation]

Original: The generalized JSD can be written as
Revised: The generalized JSD can be written as

Original: [Equation]
Revised: [Equation]

Original: Because the last terms on the right side of Equation (6) and Equation (7) both are constant, thus equation (2) is established.
Revised: Because the last terms on the right side of Equation (6) and Equation (7) both are constant, thus equation (2) is established.

Original: Proposition 1 shows that optimizing the mini-max game of equation (1) is equivalent to optimizing equation (2) which is over the training data, thus it is tractable and the model parameter θ can be updated directly using stochastic gradient descent (SGD) or its variants.
Revised: Proposition 1 shows that optimizing the mini-max game of equation (1) is equivalent to optimizing equation (2) which is over the training data, thus it is tractable and the model parameter θ can be updated directly using stochastic gradient descent (SGD) or its variants.

Original: Proposition 2.
Revised: Proposition 2.

Original: When π is converging to 0, optimizing V π ( G,D ) π is equivalent to optimizing the forward KL divergence D KL (˜ p data ( x ) (cid:107) p G ( x )) , i.e. optimizing maximum log-likelihood; and when π is converging to 1, optimizing V π ( G,D ) 1 − π is equivalent to optimizing the reverse KL divergence
Revised: When π is converging to 0, optimizing V π ( G,D ) π is equivalent to optimizing the forward KL divergence D KL (˜ p data ( x ) (cid:107) p G ( x )) , i.e. optimizing maximum log-likelihood; and when π is converging to 1, optimizing V π ( G,D ) 1 − π is equivalent to optimizing the reverse KL divergence

Original: D KL ( p G ( x ) (cid:107) ˜
Revised: D KL ( p G ( x ) (cid:107) ˜

Original: p data ( x )) .
Revised: p data ( x )) .

Original: Proof: Consider the Taylor expansion KL [ Q || Q + a ] ≈ a T Ha  and proposition 1, the limitation of V π ( G,D ) π can be written as
Revised: Proof: Consider the Taylor expansion KL [ Q || Q + a ] ≈ a T Ha wher H is the positive deﬁnite Hessian and proposition 1, the limitation of V π ( G,D ) π can be written as

Original: [Equation]
Revised: [Equation]

Original: thus when π is converging to 0, optimizing V π ( G,D ) π is equivalent to optimize the forward KL divergence D KL (˜ p data ( x ) (cid:107) p G ( x ))
Revised: thus when π is converging to 0, optimizing V π ( G,D ) π is equivalent to optimize the forward KL divergence D KL (˜ p data ( x ) (cid:107) p G ( x ))

Original: On the other hand, we can also explain this proposition from the perspective of the gradient.
Revised: On the other hand, we can also explain this proposition from the perspective of the gradient.

Original: Taking derivative with respect to the parameters θ of generator, we have
Revised: Taking derivative with respect to the parameters θ of generator, we have

Original: [Equation]
Revised: [Equation]

Original: Since the term log
Revised: Since the term log

Original: [Equation]
Revised: [Equation]

Original: converges to − ˜ p data ( x ) p G [Equation] x ) in the limit of π → 0 , thus we can get
Revised: converges to − ˜ p data ( x ) p G ( x ) in the limit of π → 0 , thus we can get

Original: [Equation]
Revised: [Equation]

Original:  where term ∂ log P G ( x ) ∂θ is the gradient of log-likelihood.
Revised: ) where term ∂ log P G ( x ) ∂θ is the gradient of log-likelihood.

Original: The same can be proved that in the limit of π → 1 , V π ( G,D ) 1 − π is equivalent to reverse KL divergence
Revised: The same can be proved that in the limit of π → 1 , V π ( G,D ) 1 − π is equivalent to reverse KL divergence

Original: DKL ( p G ( x ) (cid:107) ˜
Revised: D

Original: p data ( x )) .
Revised: [Equation]

Original: This proposition shows an intimate connection of optimizing the objective V π ( G, D ) to those of the forward and reserve KL divergences.
Revised: This proposition shows an intimate connection of optimizing the objective V π ( G, D ) to those of the forward and reserve KL divergences.

Original: Thus we can control the quality-diversity trade-off for text generation through a single hyper-parameter π .
Revised: Thus we can control the quality-diversity trade-off for text generation through a single hyper-parameter π .

Original:  E XPERIMENTAL S TUDY 4. D ATASETS
Revised:  E XPERIMENTAL S TUDY 4. D ATASETS

Original: The experiments consist of two parts: synthetic-data experiments and real-world experiments.
Revised: The experiments consist of two parts: synthetic-data experiments and real-world experiments.

Original: We use two datasets, COCO Image Captions dataset and EMNLP2017 WMT News dataset in our realworld experiments since they have become common benchmarks for text generation (Guo et al., 2018; Caccia et al., 2018; Montahaei et al., 2019).
Revised: We use two datasets, COCO Image Captions dataset and EMNLP2017 WMT News dataset in our realworld experiments since they have become common benchmarks for text generation (Guo et al., 2018; Caccia et al., 2018; Montahaei et al., 2019).

Original: 4.1. S YNTHETIC D ATASET
Revised: 4.1. S YNTHETIC D ATASET

Original: In synthetic-data experiments, in order to test the validity of our proposed model we conduct a simulated test similar to Yu et al.
Revised: In synthetic-data experiments, in order to test the validity of our proposed model we conduct a simulated test similar to Yu et al.

Original: Following them, we use a random initialized LSTM as the oracle model to generate the real data distribution P oracle ( x t | x 1 , · · · , x t − 1 ) to stimulate the realworld structured sequences.
Revised: Following them, we use a random initialized LSTM as the oracle model to generate the real data distribution P oracle ( x t | x 1 , · · · , x t − 1 ) to stimulate the realworld structured sequences.

Original: We use the oracle model to generate 10,000 sequences of length 20 and 40 respectively as our training set and another two 5,000 sequences are generated as our validation set and test set.
Revised: We use the oracle model to generate 10,000 sequences of length 20 and 40 respectively as our training set and another two 5,000 sequences are generated as our validation set and test set.

Original: Both of them have a vocab size of 5,000.
Revised: Both of them have a vocab size of 5,000.

Original: 4.1. COCO I MAGE C APTIONS D ATASET
Revised: 4.1. COCO I MAGE C APTIONS D ATASET

Original: For COCO Image Captions dataset (Chen et al., 2015), we use the same pre-processing method as (Guo et al., 2018), but we randomly select 5,000 samples from training set as our validation set.
Revised: For COCO Image Captions dataset (Chen et al., 2015), we use the same pre-processing method as (Guo et al., 2018), but we randomly select 5,000 samples from training set as our validation set.

Original: The resulted dataset consist of a training set with 75,000 samples, a validation set with 5,000 samples and a test set with 5,000 samples.
Revised: The resulted dataset consist of a training set with 75,000 samples, a validation set with 5,000 samples and a test set with 5,000 samples.

Original: The vocab size is 4,838 and the maximum length of captions is32.
Revised: The vocab size is 4,838 and the maximum length of captions is

Original:  4.1. EMNLP2017 WMT N EWS D ATASET
Revised: 32. 4.1. EMNLP2017 WMT N EWS D ATASET

Original: EMNLP2017 WMT News (Bojar et al., 2017) is a collection of news texts for the machine translations task, among a version of this dataset for English Corpus containing more than 600,000 sentences, we drop the sentences having more than 3 words with less than 400 frequency and keep only the sentences that have between 20 and 40 words.
Revised: EMNLP2017 WMT News (Bojar et al., 2017) is a collection of news texts for the machine translations task, among a version of this dataset for English Corpus containing more than 600,000 sentences, we drop the sentences having more than 3 words with less than 400 frequency and keep only the sentences that have between 20 and 40 words.

Original: We replace these most infrequent words with token < unk > .
Revised: We replace these most infrequent words with token < unk > .

Original: Next, we randomly sample 200,000 sentences as the training set, 10,000 sentences as the validation set and 10,000 as the test set.
Revised: Next, we randomly sample 200,000 sentences as the training set, 10,000 sentences as the validation set and 10,000 as the test set.

Original: The resulted vocab size is 3,980.
Revised: The resulted vocab size is 3,980.

Original: 4. E VALUATION M ETRICS
Revised: 4. E VALUATION M ETRICS

Original: It is still difﬁcult to accurately evaluate the effects of text generation models using existing metrics since the effects consist of several aspect such as consistency, diversity and ﬂuency.
Revised: It is still difﬁcult to accurately evaluate the effects of text generation models using existing metrics since the effects consist of several aspect such as consistency, diversity and ﬂuency.

Original: Thus, in order to be consistent, we choose evaluation metrics according to previous work (Semeniuta et al., 2018;
Revised: Thus, in order to be consistent, we choose evaluation metrics according to previous work (Semeniuta et al., 2018;

Original: Caccia et al., 2018; de Masson d’Autume et al., 2019; Montahaei et al., 2019).
Revised: Caccia et al., 2018; de Masson d’Autume et al., 2019; Montahaei et al., 2019).

Original: We use negative log likelihood for synthetic-data experiments and use n-gram based metrics and Fr´echet Distance (FD) for real-data experiments.
Revised: We use negative log likelihood for synthetic-data experiments and use n-gram based metrics and Fr´echet Distance (FD) for real-data experiments.

Original: 4.2. N EGATIVE L OG L IKELIHOOD
Revised: 4.2. N EGATIVE L OG L IKELIHOOD

Original: In the work of (Yu et al., 2017), the authors recommend to use the oracle negative log likelihood (NLL oracle ) which is an oracle quality metric for synthetic-data experiments.
Revised: In the work of (Yu et al., 2017), the authors recommend to use the oracle negative log likelihood (NLL oracle ) which is an oracle quality metric for synthetic-data experiments.

Original: However, only use this metric will be misleading since it is blind to diversity.
Revised: However, only use this metric will be misleading since it is blind to diversity.

Original: Thus, similar to Caccia et al.
Revised: Thus, similar to Caccia et al.

Original: (2018), we additionally use a metric NLL test to represent diversity that is calculated by taking as input the sentences generated by oracle model to the language model we’re interested in.
Revised: (2018), we additionally use a metric NLL test to represent diversity that is calculated by taking as input the sentences generated by oracle model to the language model we’re interested in.

Original: 4.2. N- GRAM B ASED M ETRICS
Revised: 4.2. N- GRAM B ASED M ETRICS

Original: We use the local metrics BLEU-5 (Papineni et al., 2002) and Self BLEU-5 (Zhu et al., 2018) to capture quality and diversity respectively.
Revised: We use the local metrics BLEU-5 (Papineni et al., 2002) and Self BLEU-5 (Zhu et al., 2018) to capture quality and diversity respectively.

Original: While BLEU based metrics is considered to have limitations that only capture local consistency and fail to capture semantic variations (Semeniuta et al., 2018;
Revised: While BLEU based metrics is considered to have limitations that only capture local consistency and fail to capture semantic variations (Semeniuta et al., 2018;

Original: Caccia et al., 2018; de Masson d’Autume et al., 2019), they are still one of the most widely used and simple metrics for text generation tasks.
Revised: Caccia et al., 2018; de Masson d’Autume et al., 2019), they are still one of the most widely used and simple metrics for text generation tasks

Original:  4.2. F R ´ ECHET BERT D ISTANCE
Revised: . 4.2. F R ´ ECHET BERT D ISTANCE

Original: Fr´echet Inception Distance (FID) is originally proposed to computes the distance between two Gaussian distributions of the features of real images and generated images extracted by Inception network (Heusel et al., 2017).
Revised: Fr ´ echet Inception Distance (FID) is originally proposed to computes the distance between two Gaussian distributions of the features of real images and generated images extracted by Inception network (Heusel et al., 2017).

Original: We follow Montahaei et al.
Revised: We follow Montahaei et al.

Original: (2019) to utilize BERT (Devlin et al., 2019), a powerful unsupervised pre-training model for text, to replace the Inception network to capture quality and diversity of generated sentences.
Revised: (2019) to utilize BERT (Devlin et al., 2019), a powerful unsupervised pre-training model for text, to replace the Inception network to capture quality and diversity of generated sentences.

Original: This metric is named as Fr´echet BERT Distance (FBD).
Revised: This metric is named as Fr´echet BERT Distance (FBD).

Original: We do not use MS-Jaccard and Language Model Score (LMS) because MS-Jaccard is also an ngram based metric like BLEU and it is highly correlated with FBD (Montahaei et al., 2019), and
Revised: We do not use MS-Jaccard and Language Model Score (LMS) because MS-Jaccard is also an ngram based metric like BLEU and it is highly correlated with FBD (Montahaei et al., 2019), and

Original: LMS relies on training new models and have an inherent bias favoring MLE language models, since they are trained using the same criteria (de Masson d’Autume et al., 2019).
Revised: LMS relies on training new models and have an inherent bias favoring MLE language models, since they are trained using the same criteria (de Masson d’Autume et al., 2019).

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: We choose our hyper-parameters according to the result on validation set and report all metrics on test set.
Revised: We choose our hyper-parameters according to the result on validation set and report all metrics on test set.

Original: Our models are implemented with Tensorﬂow (Abadi et al., 2016) using an NVIDIA V
Revised: Our models are implemented with Tensorﬂow (Abadi et al., 2016) using an NVIDIA V

Original: GPU.
Revised: GPU.

Original: The hidden size, embedding size and batch size we used are 32, 32 and 64 respectively which are kept the same as Texygen framework (Zhu et al., 2018) for fair comparison.
Revised: The hidden size, embedding size and batch size we used are 32, 32 and 64 respectively which are kept the same as Texygen framework (Zhu et al., 2018) for fair comparison.

Original: We terminate the training of all models based on NLL oracle in synthetic-data experiments and based on their BLEUscores on validation set in real-data experiments.[Figure]
Revised: We terminate the training of all models based on NLL oracle in synthetic-data experiments and based on their BLEUscores on validation set in real-data experiments.

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: In this section, we report the experimental results on synthetic-data experiments and real-world experiments.
Revised: In this section, we report the experimental results on synthetic-data experiments and real-world experiments.

Original: We compare our proposed models with three baseline models: MLE, SeqGAN (Yu et al., 2017),
Revised: We compare our proposed models with three baseline models: MLE, SeqGAN (Yu et al., 2017),

Original: LeakGAN (Guo et al., 2018) in all the experiments.
Revised: LeakGAN (Guo et al., 2018) in all the experiments.

Original: Similar to Caccia et al.
Revised: Similar to Caccia et al.

Original: (2018), we conduct sweep of temperatures in inference stage to evaluate the trained models over qualitydiversity space.
Revised: (2018), we conduct sweep of temperatures in inference stage to evaluate the trained models over qualitydiversity space.

Original: We ﬁxed the hyper-parameter π and sweep the temperatures so that we can get different temperature curves for our proposed models trained with different π . 
Revised: We ﬁxed the hyper-parameter π and sweep the temperatures so that we can get different temperature curves for our proposed models trained with different π . Samples from our proposed model and baseline models are shown in Appendix A.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  S YNTHETIC E XPERIMENTS
Revised: 4.4. S YNTHETIC E XPERIMENTS

Original: In the synthetic-data experiments, we use NLL oracle and NLL test as our quality and diversity metrics respectively.
Revised: In the synthetic-data experiments, we use NLL oracle and NLL test as our quality and diversity metrics respectively.

Original: Both of them are the lower the better.
Revised: Both of them are the lower the better.

Original: Lowering the temperature make the model tend to produce high quality but similar sentences, while raising the temperature makes the model tend to generate diverse but relatively random sentences.
Revised: Lowering the temperature make the model tend to produce high quality but similar sentences, while raising the temperature makes the model tend to generate diverse but relatively random sentences.

Original: We compute all metrics at different temperatures for each model to draw sweep curves in quality-diversity space.
Revised: We compute all metrics at different temperatures for each model to draw sweep curves in quality-diversity space.

Original: Figure 1 depicts the temperature sweep curves of sequence length 20 and 40 respectively over
Revised: Figure 1 depicts the temperature sweep curves of sequence length 20 and 40 respectively over

Original: NLL test vs.NLL oracle space.
Revised: NLL test vs.NLL oracle space.

Original: From this ﬁgure, we can observe that when π → 0 , the curve of our proposed model basically overlaps with the curve of the MLE model which shows the correctness of proposition 2.
Revised: From this ﬁgure, we can observe that when π → 0 , the curve of our proposed model basically overlaps with the curve of the MLE model which shows the correctness of proposition 2.

Original: Besides, we can also observe that our models outperform other GANs variants by a large margin and perform much better than MLE on quality metric at the cost of relatively small losses on diversity metric.
Revised: Besides, we can also observe that our models outperform other GANs variants by a large margin and perform much better than MLE on quality metric at the cost of relatively small losses on diversity metric.

Original: It is worth noting that our models do not need additional maximum likelihood pre-training and reinforcement learning and thus it is more faster in training than traditional
Revised: It is worth noting that our models do not need additional maximum likelihood pre-training and reinforcement learning and thus it is more faster in training than traditional

Original: GANs variants.
Revised: GANs variants.

Original: As can be seen from Figure 2, the frontier curve obtained from a family of temperature sweep curves of our models outperforms the curves of MLE, SeqGAN and LeakGAN in most cases.
Revised: As can be seen from Figure 2, the frontier curve obtained from a family of temperature sweep curves of our models outperforms the curves of MLE, SeqGAN and LeakGAN in most cases.

Original: [Table]
Revised: [Figure]

Original: TRAINING SET
Revised: Models Temperature 0. MLE 4. SEQGAN 4.

Original: [Table]
Revised: LEAKGAN 5. OURS π →4. OURS π = 0 .4. OURS π = 0 .4.

Original: [Table]
Revised: Models Temperature 0. MLE 6.

Original: TRAINING SET
Revised: SEQGAN 7. LEAKGAN 7. OURS π →6. OURS π = 0 .6.

Original: [Table]
Revised: [Table]

Original: scores on EMNLP2017 WMT News test set at different temperatures.

Original: 4.4. R EAL - DATA E XPERIMENTS
Revised: 4.4. R EAL - DATA E XPERIMENTS

Original: Figure 3 reports negative BLEU-5 versus Self BLEU-5 metrics on COCO Image Captions and
Revised: Figure 3 reports negative BLEU-5 versus Self BLEU-5 metrics on COCO Image Captions and

Original: EMNLP2017 WMT News test sets for the models trained via our proposed objective, MLE model and other GANs variants.
Revised: EMNLP2017 WMT News test sets for the models trained via our proposed objective, MLE model and other GANs variants.

Original: Table 1 and 2 show the FBD scores on these two test sets respectively for the models and training set.
Revised: Table 1 and 2 show the FBD scores on these two test sets respectively for the models and training set.

Original: For COCO Image Captions task,  when hyper-parameter π = 0 .
Revised: For COCO Image Captions task, as can be seen from Figure 3(a) and Table 1, when hyper-parameter π = 0 .

Original: 1 and π = 0 .
Revised: 1 and π = 0 .

Original: 5 , our models outperform other GANs variants under quality-diversity space and outperform MLE model in terms of quality metric which may be because compared with maximum likelihood, our models can properly optimize both forward KL divergence D KL (˜ p data ( x ) (cid:107) p G ( x )) and reverse KL divergence D KL ( p G ( x ) (cid:107) ˜
Revised: 5 , our models outperform other GANs variants under different qualitydiversity spaces and outperform MLE model in terms of quality metric, which may be because compared with maximum likelihood, our models can properly optimize both forward KL divergence D KL (˜ p data ( x ) (cid:107) p G ( x )) and reverse KL divergence D KL ( p G ( x ) (cid:107) ˜

Original: p data ( x )) and compared with other GANs variants, we do not rely on reinforcement learning.
Revised: p data ( x )) and compared with other GANs variants, we do not rely on reinforcement learning.

Original: When hyper-parameter π → 0 , the  performance of our model is almost identical to the MLE model which is consistent with proposition 2 and better than other GANs variants .
Revised: When hyper-parameter π → 0 , the result shown in Figure 3(a) and Table 1 demonstrate that the performance of our model is al- most identical to the MLE model which is consistent with proposition 2 and better than other GANs variants .

Original: For EMNLP2017 WMT News task, as shown in Figure 3(b) and Table 2,  our models show much more competitive results to MLE  than other GANs variants.
Revised: For EMNLP2017 WMT News task, as shown in Figure 3(b) and Table 2, when π = 0 . 1 is and π = 0 . 5 , although our models does not perform as well as the MLE model, they are signiﬁcantly better than other GANs variants and more competitive to MLE model. Consistent with the conclusions obtained from the synthetic data experiments and the COCO experiment, the performances of our proposed model and the MLE model are almost equal when π tends to zero.

Original: Compared with the COCO task, the difference between the FBD scores of the models on the News task and training set is slightly large which may be because EMNLP2017 WMT News is a relatively more difﬁcult task.
Revised: Compared with the COCO task, the difference between the FBD scores of the models on the News task and training set is slightly large which may be because EMNLP2017 WMT News is a relatively more difﬁcult task.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This research proposes a quality-diversity controllable GAN which is a generalization of N3DGAN and it can achieve quality-diversity control through a single hyper-parameter π .
Revised: This research proposes a quality-diversity controllable GAN which is a generalization of N3DGAN and it can achieve quality-diversity control through a single hyper-parameter π .

Original: When π converges to 0, it is equivalent to directly optimizing forward KL divergence; and when π converges to 1, it is equivalent to directly optimizing reverse KL divergence.
Revised: When π converges to 0, it is equivalent to directly optimizing forward KL divergence; and when π converges to 1, it is equivalent to directly optimizing reverse KL divergence.

Original: Through experiments in syntheticdata and real-data scenarios, we verify the convergence and the controllability, and we show that our proposed model outperforms MLE model and other GANs variants over quality-diversity space in the synthetic-data experiments, and in the real-data experiments, it is more competitive to MLE model than other GANs variants under several metrics.
Revised: Through experiments in syntheticdata and real-data scenarios, we verify the convergence and the controllability, and we show that our proposed model outperforms MLE model and other GANs variants over quality-diversity space in the synthetic-data experiments, and in the real-data experiments, it is more competitive to MLE model than other GANs variants under several metrics.

Original: As can be seen from the experimental results, depending on the task and purpose, such as to get higher quality text or more diverse text, we should choose different hyper-parameter π to get better results on what we are interested in. 
Revised: As can be seen from the experimental results, depending on the task and purpose, such as to get higher quality text or more diverse text, we should choose different hyper-parameter π to get better results on what we are interested in. LeakGAN A television brushing in a stair mat at the table .

Revised: beautiful baseball player holding a bat is traveling in the ocean

Revised: and water .

Revised: Two

Revised: young men standing near a phone at ﬁve coffee

Revised: next to sidewalk .

Revised: A blue dog sitting on top of a kite at a living room .

Revised: A group of people standing with her kite in the beach .

Revised: A big train with a checkered cake , keyboard and mouse .

Revised: A person wearing a hat holding a wii boat holding two chairs .

Revised: A man sitting next to a window .

Revised: An old man holding a tennis ball while upside his

Revised: racket in the

Revised: sky

Revised: by the sky .

Revised: here is a boat on a living room on a table

Revised: with

Revised: a

Revised: tennis

Revised: ball .

Revised: [Table]

Revised: Example

Revised: OURS π →in fact , this is in the past , at least in the european union , given the public nuclear program as a result of the macroeconomic ¡unk¿ .

Revised: in the world , he lost an annual funds rate is even less than the government ’ s single bank planning .

Revised: the main factors view : preserving the us and central banks will be ¡unk¿ to the current global ﬁnancial crisis , and is no longer .

Revised: at this time , the us and the eurozone ’ s ¡unk¿ would reduce the majority of policies that is still driven by both sides for ¡unk¿ .

Revised: the democratic party will be able to ¡unk¿ the ¡unk¿ of the american and political and cultural opportunity to ﬁx the security council .

Revised: there has become a time for iran , and damaging only in the region where the us ¡unk¿ us economy would be competitive .

Revised: after all , this is the ¡unk¿ of the lessons about the ¡unk¿ , would become the region seeking to challenge to the national culture .

Revised: however , the budget deﬁcits is not a true economic crisis in a growing global scale so much less than they have beneﬁted from emerging economies .

Revised: but they are in the crisis in other words , and it will have ¡unk¿ the emergence of a ¡unk¿ of the eu and north america .

Revised: in the wake of the transformation of the media , there is a question who would be ¡unk¿ in the past 30 years .

Revised: OURS

Revised: [Table]

Revised: in the rest of the election , however the us will not at the difﬁculty of climate change .

Revised: more important should be the ¡unk¿ of europe and used in modern history .

Revised: and all the chinese central banks was shown to ¡unk¿ the european union and the global ﬁnancial crisis .

Revised: with the global crisis , the us administration leaders must act fast in the ¡unk¿ of the global ﬁnancial system .

Revised: how many of the europeans could bring it to be in the common way to dominate the ﬁnancial crisis .

Revised: in fact , it is not at the same time , there is ¡unk¿ - by the west , and the global ﬁnancial crisis .

Revised: however , it is particularly important to explain the chinese government .

Revised: this would be able to ¡unk¿ its debts , it is supposed to ¡unk¿ the international economy and ¡unk¿ violence .

Revised: the costs of the ¡unk¿ will be increasing over the ﬁnancial crisis .

Revised: the world politics is likely to accept the expense of others in the crisis .

Revised: Sources

Revised: Example

Revised: MLE they pushing for long as 18 % of chinese without the currency support of the new middle east ’ s strengthening their imports .

Revised: the government ’ s relationship with american power was ¡unk¿ to become a crucial tool to create a serious threat from israel and the ¡unk¿ of the way .

Revised: for example , however , the rest of the world is we are the ¡unk¿ of the ¡unk¿ that markets must be able to success .

Revised: in the fourth quarter of 2011 , the largest of the world s periphery economies created the world peace in the ¡unk¿ of 2007 .

Revised: the us was unable to its willingness to tackle the ﬁnancial crisis , the new government must be addressed by the start to ¡unk¿ and the future .

Revised: indeed , the integrity of the world is not about whether the eu has to be ¡unk¿ by ¡unk¿ , as well as the iranian constitution , as well as their problem .

Revised: the us policy is , that would have not seen the dollar as the us to ¡unk¿ asia and the world economy .

Revised: moreover , the eu s inﬂation is more likely to take more improved than about the us , and the beneﬁts of economic recovery .

Revised: a key question is that a ¡unk¿ should be on a new framework for a ¡unk¿ , from the american government .

Revised: this is that the russian economy is just one-third of the value of $ 1 billion annually a source of global trade .

Revised: SeqGAN like the ¡unk¿ of the center for the most of the russian crisis .

Revised: the ¡unk¿ of the uk is right to ensure that the eu will ¡unk¿ .

Revised: it said that it says ¡unk¿ and could not take a signiﬁcant impact on oil .

Revised: in asia , political has been wrong to the world bank , like what is the most important time .

Revised: in the recent crisis , is so that something else about most of the parties .

Revised: what is important for observers that high and how he could ¡unk¿ to support the education system .

Revised: we could develop a relatively little chance to fundamental classes , it is likely to ¡unk¿ its response to the yen .

Revised: on the asian ﬁnancial system , like all of the west bank , it is what could not ¡unk¿ to put up its foreign ¡unk¿ .

Revised: in short , an unprecedented increase in the ecb would allow ¡unk¿ opportunity to rise to ¡unk¿ for the euro .

Revised: to manage the ¡unk¿ of the economic crisis , and the administration should increase their lives .

Revised: LeakGAN but , gradually , the candidates must have to the excessive ¡unk¿ and that the old conﬂict can be made for the necessary housing .

Revised: both of the tax increase are on an opportunity to come to wonder that iraq is not little scenario .

Revised: ¡unk¿ to recognize the need for greater responsibility for ¡unk¿ concerning its foreign program .

Revised: ﬁnally , ¡unk¿ much of there is no more risky to achieve the goal .

Revised: it is likely to be reduced to make domestic risk to stimulate inﬂation in the national cooperation .

Revised: the relationship between a ﬁnancial system is associated to raise income inequality .

Revised: but any recent problem is that made such events as much by ¡unk¿ up between global investment and strategic costs .

Revised: for radical institutions to keep the access to minimize the biggest impact of an independent world .

Revised: the ¡unk¿ process of capital requirements must be given that , or ¡unk¿ , and from the decision to guide global security .

Revised: it is nothing that hope that , lack of trust , such as a freedom of law .