L OCAL GAN: M ODELING L OCAL D ISTRIBUTIONS FOR A DVERSARIAL R ESPONSE G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper presents a new methodology for modeling the local semantic distribution of responses to a given query in the human-conversation corpus, and on this basis, explores a speciﬁed adversarial learning mechanism for training Neural Response Generation (NRG) models to build conversational agents.
The proposed mechanism aims to address the training instability problem and improve the quality of generated results of Generative Adversarial Nets (GAN) in their utilizations in the response generation scenario.
Our investigation begins with the thorough discussions upon the objective function brought by general GAN architectures to NRG models, and the training instability problem is proved to be ascribed to the special local distributions of conversational corpora.
Consequently, an energy function is employed to estimate the status of a local area restricted by the query and its responses in the semantic space, and the mathematical approximation of this energy-based distribution is ﬁnally found.
Building on this foundation, a local distribution oriented objective is proposed and combined with the original objective, working as a hybrid loss for the adversarial training of response generation models, named as LocalGAN.
Our experimental results demonstrate that the reasonable local distribution modeling of the query-response corpus is of great importance to adversarial NRG, and our proposed LocalGAN is promising for improving both the training stability and the quality of generated results.
I NTRODUCTION
End-to-End generative conversational agents (a.k.a., generative Chat-bots) are believed to be practicable on the basis of the Sequence-to-Sequence (Seq2Seq) architecture (Sutskever et al., 2014) trained with large amounts of human-generated conversation sessions (Shang et al., 2015a;
Sordoni et al., 2015), and this task is named as Neural Response Generation (NRG).
Similar to the Neural
Machine Translation (NMT) approaches (Bahdanau et al., 2014; Wu et al., 2016), the deep Seq2Seq models are expected to directly generate appropriate and meaningful responses according to the input query.
Compared to the success of NMT systems, the application progress of NRG models is not satisfying at present due to the “safe response” problem (Li et al., 2016).
That is, most of the generated responses are boring and meaningless, which blocks the continuation of conversations.
Indeed, eliminating “safe responses” is the essential task of NRG models.
Thus, various methods have been considered to address this problem (Li et al., 2016; Xu et al., 2017; Li et al., 2017;
Xing et al., 2017; Pandey et al., 2018; Zhang et al., 2018a; Du et al., 2018).
More recently, Generative Adversarial Nets (GAN) (Goodfellow et al., 2014) have been introduced to eliminate “safe responses” (Li et al., 2017; Xu et al., 2017).
Basically, this methodology is reasonable since the GAN framework involves an adversarial discriminator that helps NRG models leap out of the shortsighted state of minimizing the empirical risk on word distribution, by providing feedback on real samples from the model generated ones.
Despite the improvement on the diversity, the adversarial training process of GAN based response generation models is generally unstable and sensitive to the training strategy (Yu et al., 2017).
The unstable convergence problem is largely ascribed to the complicated data distribution in practical scenarios (Arora et al., 2017; Arora & Zhang, 2017).
For the response generation oriented GAN models, in particular, the data distribution appears to be much more complicated.
Fundamentally, an essential characteristic of conversation data is that, for each given query, there always exists a group of semantically-diverse responses, rather than the semantically-uniﬁed ones.
Furthermore, the response groups of two different queries tend to keep great divergences in the semantic space.
In this scenario, the discriminator needs to consider the distributions of the generated result in the semantic space, rather than simply examining whether one single sample comes from the generator or the original dataset, so as to make the generator sense the distribution of conversation dataset in the adversarial training procedure.
This paper aims at presenting a speciﬁc adversarial training schema for neural response generation.
Beginning with the investigation on the reason for the unsatisfying performance of GANs on the
NRG task, we ﬁnd the upper-bound of the current GAN learning strategies taking query-response pairs as the independent training samples.
On this basis, we claim that the training schema, including the actual adversarial strategy and the overall loss function, should be re-deﬁned to agree with the distribution of NRG training samples in the semantic space, rather than roughly adopting the GAN framework designed for generating images.
Consequently, we describe the distributional state of the given query and the corresponding responses with the free energy deﬁned on the basis of Deep Boltzmann Machines (DBM) (Salakhutdinov & Hinton, 2009).
In this way, we can quantify the formation process of generating the response set with the topic restriction of the given query.
From the perspective of free energy, this paper proposes a new cost function to measure the expansion degree of the responses in the local area of the real-valued semantic space.
Cooperating with the traditional implicit density discriminating loss of GAN, the proposed cost actually provides an explicit density approximation for the local distribution of each response cluster.
Thus, the adversarial learning procedure can be expected to be more stable with better response generation results obtained 1 .
T HE LIMITATION OF G ENERAL GAN IN THE NRG S CENARIO
According to (Goodfellow et al., 2014), the standard GAN framework contains a generator G and a discriminator D , which are trained by an iterative adversarial learning procedure based on the following objective function:
[Equation]
where p z denotes the prior on input noise variables, and p d is the true data distribution.
It should be noted that the GAN tries to learn the manifold of a given dataset (Khayatkhoei et al., 2018; Kumar et al., 2017), and the discriminator D actually provides a metric for judging whether the results generated according to z ﬁts the expected manifold or not.
In the NRG scenario, a naive Seq2Seq model without the guidance signal from D can not capture the data manifold of the real query-response corpus, which is one of the major facts the safe-response problem can be ascribed to.
Assuming that there exists an oracle discriminator with the ability of distinguishing the generated fake samples from the ground-truth ones, by mapping each query-response pair ( q, r ) 2 to a conﬁdence score s , and meanwhile, it can be assume that any practically existing discriminator of GAN gives the conﬁdence ˜ s to ( q, r ) .
If the practical discriminator can make ˜ s → s , the generator will obtain more meaningful guidance for the better generation.
That is, to improve the capability of GAN-NRG, it is wise to construct more powerful discriminators for more reasonable J ( G ) .
Now let’s pay attention to the actual change of NRG models brought by GAN.
In the generative conversation agent scenario, G ( z ) is corresponding to a generated response ˜ r to a given query q .
Thus, the objective of the generator in the GAN based NRG model can be simply formulated as:
[Equation]
The training of the generator is actually the procedure to maximize J ( G ) toward E ( q,r ) ∼ p
d [log D ( q, r )] , so as to generate realistic responses according to given queries.
Thus, in the context of adversarial learning, J ( G ) should satisfy the following inequality:
[Equation]
However, it is well known that a conversational dataset should not be simply taken as a collection { ( q, r ) } composed of independent query-response pairs.
Instead, to each given query q , there exists a ﬁnite set of corresponding responses R q = { r i } .
In this case, it is of great necessity to consider the whole training dataset as a collection of R q , which takes the form of a number of clusters with their own local distributions in the semantic space.
And we can rewrite the joint distribution p d in (4) as p ( q ) p
( r | q ) and assume every corresponding response to a query follows equal-probability distribution, which means that p ( r | q ) = 1 | R q | .
Thus, in the real NRG scenario, on the basis of
Equation 3 and Inequation 4, J ( G ) follows the inequality as below:
[Equation]
where p ( q ) denotes the probability of the query q , and R q is deﬁned above.
According to Equation 5, the upper bound of J ( G ) is obtained, in which the log (cid:104) 1 | R q | (cid:80) r ∈ R q D ( q, r ) (cid:105) part is the essence.
Apparently, the expression
[Equation]
indicates the mean value of the conﬁdence scores given by the discriminator to each member of the response set R q to a given query q .
Moreover, it should be noted that current studies tend to utilize semantic relevance oriented models to build the discriminators of GAN (Xu et al., 2017; Li et al., 2017).
Consequently, D ( q, r ) can be actually considered as the spatial relationship of q and r in the semantic space.
Thus, 1 | R q | (cid:80) r ∈ R q D ( q, r ) stands for the spatial center of all the responses within
R q .
That is, the optimization process of adversarial learning upon conversational datasets will make the generated responses approach to the center of each local distribution of R q corresponding to each given dependent query.
The practical value of this change lies in that, intuitively, the GAN architecture forces the generation to pay attention to the local distributions of the individual response clusters, rather than taking the ( q, r ) -pairs as an entirety.
According to the thorough studies on the safe responses of NRG models (Li et al., 2016; Xu et al., 2017; Zhang et al., 2018a; Pandey et al., 2018), it can be inferred that the general Seq2Seq will fall into the divergence state of generating the patterns with the maximum probabilities taking account of the entire dataset, ignoring the individual-difference of each query.
By introducing the implicit loss focusing on the response clusters, GAN makes the divergence of the generator much closer to the ‘local patterns’ rather than the general patterns, and thus the higher diversity can be expected and observed (Xu et al., 2017; Li et al., 2017).
The problem turns to: Is the upper bound in Equation 5 powerful enough?
Apparently, there exists an obvious gap between the ‘local patterns’ and the vivid and interesting generated responses.
The upper bound only focuses on the mean of each response set, but the local distribution (or the actual ‘shape’) of each cluster has not been taken into account.
This situation does not change when the cost function of adversarial training is deﬁned by Wasserstein GANs (WGAN)
(Arjovsky et al., 2017) with 1-Lipschitz function f :
[Equation]
[Equation] Intuitively, it is of paramount importance to estimate both the ‘location’ and the ‘shape’ of the response set R q in the semantic space (indeed, 1 | R q | (cid:80) q,r D ( q, r ) is only relative with ‘location’), so as to determine the optimization objective of adversarial training.
Consequently, we have two critical problems to be discussed and addressed in the following sections:
• How to describe the state of the response set R q with a given query q in the semantic space?
• Taking account of the reasonable state modeling of ( q, R q ) , what is the loss function for adversarial training to generate responses?
M ODELING THE S TATE OF THE L OCAL D ISTRIBUTION FOR R ESPONSES
As mentioned above, the semantic one-to-many relationship between queries and responses makes it necessary to model the local distribution of the response cluster R q corresponding to each query in the space, and it is paramount to turn to the ﬁtting of each local distribution in the adversarial learning procedure, rather than considering each ( q, r ) -pair as an independent sample.
Basically, this issue equals to the task of reasonably modeling the state of ( q, R q ) in the semantic space, by considering each ( q, R q ) as a systematic entirety and assigning the state of the entirety with probabilistic distribution.
The additional major challenge of this task is, indeed, we have to infer the state of a local area in the semantic space from a group of ﬁnite samples, since it is impossible to sample all the possible responses to a query from the given corpus, regardless of the corpus size.
3. R EPRESENTING L OCAL D ISTRIBUTIONS WITH Q UERY -R ESPONSE ORIENTED F REE E NERGY
In this part, we typically take an energy based statistical model, the Average Free Energy (Hinton &
Zemel, 1994; Friston et al., 2006; Ngiam et al., 2011; Friston et al., 2012), to describe the state of the local distribution of ( q, R q ) in the semantic space, for the reasons that: a) energy based models are considered as a promising avenue towards learning explicit generative models (LeCun et al., 2006;
Le Roux & Bengio, 2008), by representing data distributions without any prior assumptions; and b) energy based models can be trained in the unsupervised way, and the energy functions of such models have the potential to estimate the state of generative models (Zhao et al., 2016).
At ﬁrst, the free energy of a given query-response pair ( q, r ) can be deﬁned as following:
[Equation]
where E ( q, r, H ) stands for the energy function deﬁned according to the relationship of the query q and its response r via the hidden variable H .
[Equation]
We employ the
Deep
Boltzmann
Machine (DBM) (Salakhutdinov & Larochelle, 2010; Smolensky, 1986; Hinton & Salakhutdinov, 2006) to implement
E ( q, r, H ) , as illustrated by Figure 1.
The reason for this choice lies in that, from the view of conversational agents, the meaningful query and the corresponding response are generally considered to maintain strong semantic relevance.
Thus, the query and response can be mutually transformed into each other, which is supported by the considerable amount of studies on response generation (Shang et al., 2015b;
Shao et al., 2017; Zhang et al., 2018a; Baheti et al., 2018) and question generation (Du et al., 2017; Zhao et al., 2018; Sun et al., 2018).
Without loss of generality, the pairwise semantic relationship of the query q and the corresponding response r can be modeled by a two-layer DBM.
The bottom layer is actually an abstract version of Seq2Seq models, in which a response r can be generated based on a hidden variable h q , and h q depends on the given query q theoretically.
In the top layer, h q conditionally depends on a hyper hidden variable h .
Figure 1 illustrates the Deep Boltzmann Machine for modeling the semantic relationship of a query and its responses.
Following (Salakhutdinov & Hinton, 2009), on the basis of the DBM in Figure 1, the energy of the state { ( q, r ) , H } is deﬁned as:
[Equation]
where q denotes the query, r stands for the response and H = { h q , h } represents the hidden units.
W qr , W qq and W qh stand for the weights on the corresponding connections of the query, response and the hidden variables respectively in the graph model shown by Figure 1.
Consequently, we can deﬁne the average free energy of the query q and its response set R q as follows:
[Equation]
For better conducting the following discussion, we further deﬁne the energy difference between response r i and r j as:
Deﬁnition 3.1 ( Scaled Energy Difference ) .
[Equation]
Meanwhile, it is necessary to assign a spatial intuition to R q in the semantic space by deﬁning:
Deﬁnition 3.2 ( Response Cluster ) .
In the semantic space, the meaningful responses to the given query q lie in a restricted region (e.g., a hyper sphere), which can be named as the Response Cluster , in which F ( q, r ) can be taken as the distance from a response r to the cluster center.
3. E STIMATION OF F ( q, R q )
Basically, the DBM in Figure 1 provides the deﬁnition of the energy function E ( q, r, H ) of the free energy given in Equation 8.
Thus, the local distribution state of the responses R q with the given query q can be mathematically described by F ( q, R q ) based on Equation 7 - 9.
In practice, however, this procedure is not operable yet because the computation of F ( q, R q ) requires all the response in
R q , and it is impractical to perform the exhaustive enumeration over all the possible responses of the given query, regardless of the amount of the training query-response pairs.
Consequently, it is highly necessary to approximate F ( q, R q ) under some reasonable assumptions.
According to the response cluster deﬁned in Deﬁnition 3.2, by taking the response r as a random variable, we can assume that the response follows multivariate normal distribution with mean r c and covariance matrix Σ .
It is obvious that the sample mean 1 | R q | (cid:80) i r
i is an estimator for the population mean r c .
First, we propose the following two lemmas 3 :
Lemma 1.
If the possible responses are sufﬁcient, F ( q, R q ) approximates E [ F ( q, r )] .
Lemma 2.
If the expected Euclidean distance between random variable r and constant variable r c is small enough, F ( q, r c ) can be taken as the approximation of E [ F ( q, r )] .
Based on the lemmas we have:
Theorem 1.
Assume that r c can be estimated with ˆr c properly and the expected Euclidean distance between r and r c is small enough, we can take F ( q, ˆr c ) as the approximation of F ( q, R q ) .
proof of theorem 1 .
[Equation]
First, the implicit assumption that | R q | is large enough can be met easily.
Following lemma 1, we have | F ( q, R ) − E
[ F ( q, r )] | ≤ (cid:15) .
Second, the assumption that the expected Euclidean distance between r and r c is small enough can yield | E [ F ( q, r )] − F ( q, r c ) | ≤ (cid:15) according to lemma 2.
Finally, since the function F ( q, r ) is the composition and combination of simple continuous functions, F ( q, r ) is continuous as well.
Thus, according to the assumption that r c can be estimated with ˆ r c properly, we have | F ( q, r c ) − F ( q, ˆr c ) | ≤ (cid:15) .
To sum up, under these assumptions,
[Equation]
Therefore, F ( q, ˆr c ) is proved to be the approximation of F ( q, R q ) .
T HE H YBRID L OSS OF A DVERSARIAL R ESPONSE G ENERATION
As discussed in Section 2, the ability of the response generator in the general GAN architecture is limited to learning the dense distribution around 1 | R q | (cid:80) r ∈ R q D ( q, r ) (see Equation 5), which is composed of the most frequent patterns in the semantic space.
By contrast, it is difﬁcult for the general architecture to sense the remaining sparse space containing high-quality diverse responses.
Therefore, reasonably describing the local distribution of the responses to a given query is highly necessary.
According to the analysis in Section 3, the average free energy can be taken to model the state of the local area of the responses to a query, and such energy can be reasonably approximated via the DBM deﬁned on the query-response pairs.
On the basis of the previous sections, this section will ﬁnally propose the new hybrid loss function to force the generator to produce responses with better diversity through the more stable adversarial training process.
4. T HE R ADIAL D ISTRIBUTION F UNCTION OF THE R ESPONSE
The analysis in Section 3 have shown that the local distribution state of the responses R q to the given query q can be modeled by the average free energy F ( q, R q ) .
On this basis, it is possible to propose the description of the spatial state of a single response r in the semantic space, and consequently, we can give a new adversarial loss indicating the cost of simulating the local distribution of R q .
According to Deﬁnition 3.1 and 3.2, in each response cluster R q , the distance from a response r to the cluster center r c is actually equivalent to the scaled energy difference between them, that is,
[Equation]
Meanwhile, on the basis of Theorem 1, F ( q, R q ) can be approximated by F ( q, ˆ r c ) , and ˆ r c is modeled from training data, and thus we have:
[Equation]
Here we approximate ∆ q,r,r c with α ( q,r ) , and formally call α ( q,r ) as the Radial Distribution Function (RDF), indicating the relative cost ratio to F ( q, r c ) for obtaining r from a given q (also the distinctiveness of r , actually).
4. T HE H YBRID O BJECTIVE F UNCTION
Based on the previous discussions, for the adversarial response generation methodology, the essence is to reasonably describe the state of the local distribution of the response cluster given by Deﬁnition 3.2, and further more, to take this important element into account in the ﬁnal optimization progress.
Especially, in Subsection 4.1, we have deﬁned the Radial Distribution Function in Equation 14 to quantify the distinctiveness of a response, the very basis of which is the description of the local state
F ( q, R q ) in the semantic space.
Thus, we can further build a mechanism to quantify the difference between the generated response and the golden response as follows:
[Equation]
15) where ˜ r is the generated response given by the generator and r comes from the original data.
If δα moves toward zero, α ( q, ˜ r ) would be close to α ( q,r ) sharing the same F ( q, r c ) .
Consequently, a new expectation comes out.
That is, the generator needs to provide results that can minimize δα , so as to ﬁt the local distribution of the existing responses to a given query.
Thus, a hybrid objective of the generator can be ﬁnally deﬁned as:
[Equation]
It should be noted that a hinge loss, conducted by the ReLU function ReLU ( δα ) = max(0 , δα ) , is especially introduced to reform δα .
The primary reason of this operation is that the ReLU function has positive output only if δα (cid:62) 0 , according to the deﬁnition of ReLU ( δα ) .
Apparently, δα <indicates that the generated response ˜ r is too far from the center of the response cluster in the semantic space, so that its relevance may be highly questionable.
Meanwhile, minimizing a negative variable is against the optimization direction.
After the ReLU transformation, there remains valid loss only when δα (cid:62) 0 , and thus both the diversity and the relevance of generated results are taken into account.
4. T HE P HASE - WISE O PTIMIZATION
According to the analysis in Section 2, the trivial adversarial training directed by − E [log D ( q, r )] can only determine the form of general responses to a given query.
From the spatial perspective in the semantic space, the original adversarial objective is helpful to roughly locate the response cluster to be generated.
However, the local distribution can not be captured by this procedure.
By contrast, according to the discussions above, the proposed hybrid objective actually provides a way to force the generated responses, originally gathering around the general form, to expand into the expected local shape described by the golden truth, by conducting a phase-wise optimizing operation.
This mechanism can be detailed in an intuitive way:
Foundation : Once the DBM in Figure 1 is well-trained with the query-response corpus, the semantic center r c of a Response Cluster can be determined by the given query q .
Phase-1 :
In the early stage of the adversarial training, a generated response ˜ r is not semantically relevant to the query q .
Thus, it can be inferred that ˜ r is radially farther from the cluster center r c than the golden response r .
In this situation, according to Equation 14 and Equation 15, we can claim that δα ≤ 0 .
In this phase, the hyper objective goes back to the general adversarial objective due to the ReLU function.
Thus, the model is trying to force the generated samples to approach the center of each cluster, ignoring local distributions.
Phase-2 :
During the adversarial training in Phase-1 , the generated result ˜ r will go approaching to the cluster center r c , which means α ( q, ˜ r ) → 0 .
It should be noted that, for any meaningful existing training sample r , α ( q,r ) > 0 .
Therefore, at some point, it turns to δα >
and the right part of the hybrid objective in Equation 16 takes effect.
Consequently, for each given query, the distribution of the generated results will expand to ﬁt the local distribution of the golden samples.
E XPERIMENTS 5. E XPERIMENTAL S ETUPS
Datasets.
Our experiments are conducted on two main stream open-access conversation corpora:
The Opensubtitles corpus and the Sina Weibo corpus.
The OpenSubtitles dataset contains 5,200, movie dialogues, where we extract query-response pairs following (Xu et al., 2018; Li et al., 2016).
The Sina Weibo Corpus (Shang et al., 2015a) contains 2,500,000 single-turn Chinese dialogues, in which the length of the query and response ranges from 4 to 30.
We sample 100,000, and 2, unique query-response pairs as validation and testing dataset respectively from both of the corpora 4 .
Baselines.
For meaningful comparison, we introduce the following models as the baselines:
(1) Seq2Seq : a sequence-to-sequence model trained with maximum likelihood estimation (MLE); (2) Seq2Seq-MMI : the NRG model with a Maximum Mutual Information (MMI) criterion (Li et al., 2016); (3) Adver-REGS : the NRG model trained using adversarial framework, in which the policy gradient was employed to transfer the reward of the discriminator to the generator (Li et al., 2017); (4) GAN-AEL : an adversarial framework with an approximate embedding layer for connecting the generator with the discriminator directly (Xu et al., 2017).
[Table]
5) AIM / DAIM : the adversarial training strategy allowing distributional matching of synthetic and real responses and explicitly optimizing a variational lower bound on pairwise mutual information between the query and response, so as to improve the informativeness and diversity of generated responses (Zhang et al., 2018b) 5 .
Evaluation Metrics.
To evaluate diversity of results, we adopt three widely-applied metrics: Distinct-1 ( Dist-1 ), Distinct-2 ( Dist-2 ), and Entropy ( Ent4 ) (Li et al., 2016; Zhang et al., 2018b; Jost, 2006).
Besides, the relevance ( Rel. ) is measured by summing three embedding-based similarities (greedy, average, extreme) (Liu et al., 2016) upon the ground-truth and generated responses.
Training Details.
The details of the model training are given as follows:
The vocabulary size of both datasets is 40,000.
The embedding layer of OpenSubtitles and Sina Weibo is initialized using 200dimensional Glove vectors (Pennington et al., 2014) and 300-dimensional Weibo vectors (Li et al., 2018) respectively.
All the models are ﬁrst pre-trained by MLE, and then the models including
Adver-REGS, GAN-AEL, AIM, DAIM and LocalGAN are trained with adversarial learning.
The discriminator of Adver-REGS and GAN-AEL are based on CNN following (Yu et al., 2017; Xu et al., 2017), in which the ﬁlter sizes are set to (1,2,3,4) and the ﬁlter number is 128, while that of
LocalGAN adopts DBM with (2 × embedding size, 128, 128) to represent the semantic of queries and responses.
The hidden size of the generator is set to 256 and 512 in GAN-based models and
Seq2Seq respectively.
To guarantee the performance consistency of AIM and DAIM, we adopt the recommended parameter settings given by Zhang et al.
(2018b).
The experiments are conducted on the Tesla K80 GPU.
5. R ESULTS & A NALYSIS
Table 1 lists quantitative results on the diversity and relevance of generated responses on both datasets.
As shown by the results, compared to Seq2Seq and Seq2Seq-MMI, the GAN-based methods give better results on the diversity oriented metrics, including Dist-1, Dist-2 and Ent4.
This observation indicates that adversarial learning does provide the meaningful guidance to NRG models to avoid some of the safe-responses.
It can be observed that LocalGAN outperforms the baselines with adversarial learning architecture (Adver-REGS, GAN-AEL, AIM and DAIM) on both the diversity metrics and the relevance metrics.
Generally, a notable improvement on diversity may lead to some negative inﬂuence on relevance, and thus promoting the diversity of generated response while maintaining their relevance is essentially desired for any methodologies, which has been achieved by our LocalGAN.
The performances of
LocalGAN can be attributed to the fact that LocalGAN has taken the local distribution of responses to a given query into account.
By adopting the hybrid objective function, the proposed adversarial model gets to capture the spatial characteristics of response clusters, and the generation process is consequently forced to ﬁt the semantic distributions of response clusters.
The training stability is a tough issue to be addressed for adversarial learning (Yu et al., 2017), and as discussed in the previous sections, one of the motivations of our LocalGAN is to make adversarial [Figure] learning more stable.
To verify this aspect, we track the changing of the Entropy (Ent4) of results given by GAN-AEL, Adver-REGS, AIM, DAIM and LocalGAN, as shown in Figure 2.
It can be observed that the training of LocalGAN and Adver-REGS is stable.
By contrast, there exist obvious ﬂuctuations on the curves of AIM and DAIM, and GAN-AEL rapidly gets out of control after 1000 batch, which makes it rather difﬁcult to grasp the models with the best status.
This group of results indicates the necessity of introducing additional restrictions into adversarial learning processes.
For this purpose, Adver-REGS introduces a teacher-forcing loss (Li et al., 2017), while AIM and DAIM have taken the informativeness oriented constraints to partially control the stability (Zhang et al., 2018b).
However, GAN-AEL only takes the Wasserstein distance as the objective (Xu et al., 2017), and thus the entropy goes down rapidly.
Compared to the Adver-REGS, our LocalGAN achieves better diversity with even a more smooth entropy curve.
The training of
LocalGAN beneﬁts from the phase-wise optimization driven by the hybrid loss, and its stability also indicates the meaningfulness of modeling and utilizing local distributions of responses.
5. H UMAN E VALUATION
[Table]
To further conduct intuitive comparisons among the NRG models, we perform human evaluations on 500 testing samples.
Five annotators are asked to judge whether a response is relevant to the given query and whether the response is informative or not respectively.
Both human metrics “Relevance” and “Informativeness” solely have two labels 0 and 1 (0: irrelevant or non-informative; 1: relevant or informative) for simplicity.
The human evaluation results are listed in Table 2, and the results basically match the observations in Table 1.
The
Kappa of relevance and informativeness annotations are 0.734 and 0.561 respectively.
5. C ASE S TUDY
The effectiveness of the NRG models can be illustrated by the responses generated by them intuitively.
Table 3 lists the sampled results given by all the baselines and our proposed LocalGAN, trained on the OpenSubtitle corpus (The samples based on the Sina Weibo dataset are given in Appendix C).
It can be observed that, compared to the basic Seq2Seq and Seq2Seq-MMI, the models incorporating adversarial learning have generated results with varying degrees of semantic diversity, as expected.
Among the baselines, DAIM produces more diverse responses, which matches the quantiﬁed evaluation results in Table 1, however, little improvement has been found in the cases from this model.
By contrast, the samples from LocalGAN have shown better semantic relevance [Table] and meanwhile the diversity and informativeness are also guaranteed. [Table]
[Table]
C ONCLUSIONS
This paper has given the theoretical proof of the upper bound of the adversarial training leveraged models on the Seq2Seq-based neural response generation task.
The proof indicates that, due to the local distribution nature of query-response corpora, the GAN based NRG models will converge to the states mostly generating specialized patterns corresponding to given queries.
To address this issue, we proposed to model the local distribution of queries and their response in the semantic space by adopting energy-based function, and found the approximation of this function.
According to this approximated distribution representation, a new loss function describing the local expansion cost in the ﬁtting of response distribution is presented and ﬁnally combined with the traditional
GAN loss to form a hybrid training objective for the GAN based NRG model.
This paper provides a reasonable explanation to the unstable training process and unsatisfying results of GAN based NRG approaches, and meanwhile gives a different perspective to leverage the local data distribution to enhance classic GAN approaches.
A T HE G RAPHICAL M ODEL FOR R ESPONSE D ISTRIBUTION M ODELING
Different from the computer vision related scenario, training a DBM on a set of text vectors is not trivial since the training procedure is difﬁcult to converge due to the value scale of text vectors is much larger than image vectors.
Moreover, the simple scaling methods are not effective enough for this issue.
For this purpose, this paper adopts standard-scaler 6 to remove the mean and scale to unit variance.
To valid the effectiveness of standard-scaler, We conduct experiments on the queryresponse matching task using normalized vectors.
The experiment result show that the matching performance based normalized vectors is similar to that of CNN based architecture (Kim, 2014).
B D ETAILED P ROOF OF L EMMAS
Proof of Lemma 1 .
As mentioned before, we assume that r ∼ N ( r c , Σ) .
It is a reasonable assumption that response
r
i is independent from other response r j , that is, a possible response to one query does not rely on other possible responses.
More speciﬁc, F ( q, r i ) can be seen as an independent identically-distributed random variable.
Based on the law of large numbers,
[Equation]
Thus, when the possible responses are sufﬁcient, which means R q is large enough, we can say that for any small term (cid:15) >
[Equation]
Proof of Lemma 2 .
For a ﬁxed query, F ( q, r ) can be seen as the the scalar function of vector r .
For simplicity, we denote F ( q, r ) as f ( r ) .
Taylor expansions for the ﬁrst moment of function of random variables are as follows.
[Equation]
(18) where Df ( r c ) is the gradient of f evaluated at r = r c and D 2 f ( r c ) is the Hessian matrix. Since E r = r c , the second term ( r
− r c [Equation] T Df ( r c ) disappears.
Now, simplify the third term.
• According to (Petersen & Pedersen, 2012), assume A is symmetric, c =
E
[ x ] and Σ = Var[ x ] , then
[Equation]
• (Mirsky, 1975) states following theorem:
If A, B are complex n × n matrices with singular values α 1 ≥ α 2 ≥ · · · ≥ α n and β 1 ≥ β 2 ≥ · · · ≥ β n respectively, then
[Equation]
• A symmetric matrix is positive semi-deﬁnite if and only if all eigenvalues are non-negative.
On the one hand, since r − r c ∼ N (0 , Σ) and Σ is positive semi-deﬁnite matrix, the third term can be simpliﬁed as follows.
[Equation]
where α 1 ≥ α 2 ≥ · · · ≥ α n and β 1 ≥ β 2 ≥ · · · ≥ β n denote the singular value of matrix { D 2 f ( r c ) } and Σ respectively.
On the other hand, the expected Euclidean distance between random variable r and constant variable r c can be written as
[Equation]
Thus, for a small value (cid:15) > 0 ,
[Equation]
Hence, under the assumption that the Euclidean distance between random variable r and constant variable r c is small enough, we have
[Equation]
To observe the maximum singular value α 1 of hessian matrix of f ( r ) , we ﬁrst substitute the deﬁnition of E ( q, r, H ) into F ( q, r ) and have following equation:
[Equation]
After that, its ﬁrst-order and second-order partial derivative are calculated as follows:
[Equation]
where a ( h q , h ) =
exp ( h Tq [Equation] + h Tq
[Equation]
q + h [Equation] q [Equation]
˜ hq ˜ h exp (˜ h
Tq W
qr r +˜ h
Tq W qq q +˜ h T W qh ˜
h
q ) and b ( h q , i ) =
( h Tq W qr )
i representing the i -th element of vector h
Tq W qr .
Meanwhile, according to the deﬁnition of a ( h q , h ) , it is obvious that a ( h q , h ) > 0 and (cid:80) h q ,h a ( h q , h ) = 1 .
Denoting max h q ,i | b ( h q , i ) | as M ( h q follows multinomial distribution), the following inequality holds:
[Equation]
Therefore, we have
[Equation]
where n is the vector dimension.
As mentioned in Appendix A, DBM inputs are normalized by standard-scaler, and thus M is less than 1.
Consequently, 2 nM 2 can be approximate to a controllable constant.
To conclude, if the Euclidean distance between random variable r and constant variable r c is small enough, | F ( q, r c ) − E [ F ( q, r )] | is small enough as well.
C A DDITIONAL C ASES
[Table] [Figure]